Plandek Academy logo

Ultimate Guide – 5

Download the Ultimate Guide

Table of Contents

Metrics

5. Delivery and Agile Metrics

Figure 6 below shows a summary of the end-to-end software delivery process, the core toolsets that underpin that process; and the key categories of metrics that are derived from the various data sources.

Figure 6: High-level view of end-to-end software delivery metrics by data source
Figure 6: High-level view of end-to-end software delivery metrics by data source

Delivery and Agile metrics track the effectiveness of the end-to-end delivery process and, therefore, draw data from all the toolsets that sit across that process (see Figure 6 above).

In this section, we have divided delivery and agile metrics into five categories:

  • Value delivered
  • Delivery efficiency
  • Dependability (covered under the Sprint Metrics section)
  • Backlog health
  • Delivery quality

In the ‘Delivery quality’ section, we have added the four Flow Metrics popularised in the Flow Framework as relevant in this context.

 

Value Delivered Metrics

Figure 7 below shows our choice of ‘value delivered’ metrics. Delivery of value is always tricky to measure, and many organisations end up using the proxy measure of story points delivered (throughput), though some organisations may use ‘value points’ as an alternative to story points. Typical metrics include therefore:

 

Stories Delivered by Epic

A core metric that simply tracks stories delivered by Epic. An Epic is designed to reflect a (valuable) increment of work that is meaningful to the organisation. This metric is, therefore, a decent proxy of value delivered by relevant increment of work (epic). You may also want to consider Stories Delivered by Sprint and Stories Delivered by Benefit (if defined as a custom field).

 

Delivered Value Points

For those organisations using value points, this is clearly a critical metric and may also be viewed by Epic or Sprint, for example.

Clearly, value is not delivered until the software is deployed to live. Hence, value delivered metrics should also measure total Lead Time (from the time a ticket leaves the backlog until the time deployed to live) rather than Cycle Time, which typically just looks at the time a ticket spends in development (pre-integration test and deployment).

 

Lead Time for Epics, Lead Time for Stories

Time is a critical factor in software delivery, so being able to understand Lead/Cycle Time and potential bottlenecks makes this a must-have. Lead Time is calculated by looking at completed tickets over time and adding up the elapsed time in calendar days (including weekends) of all statuses of those tickets. You then divide the total time by the number of completed tickets to derive the Lead Time.

As the Integration-test-deployment phase of the delivery process is a key potential bottleneck in value delivery, two other metrics we tend to include under the Value Delivered umbrella are Mean Build Time and Deployments by Pipeline.

 

Mean Build Time

This is a helpful metric to identify slow-build processes that affect the ability of the team to deliver software. A steadily increasing mean workflow time will want to be addressed and will drive longer cycle times. You can filter this metric by status to help keep an eye on slow builds, which ultimately end in failure.

 

Deployments by Pipeline

As the name suggests, this is an important measure of your ability to deploy increments of software to live rapidly. It becomes more powerful when filters and breakdowns are applied by project or workflow name, for example.

Figure 7: Plandek screenshot showing example Value Delivered dashboard
Figure 7: Plandek screenshot showing example Value Delivered dashboard

Delivery Efficiency Metrics

Measuring throughput (or value delivered) is a very good starting point, but it is clearly also very important to understand how efficiently you are delivering that value in order to understand how you might increase velocity and deliver value more often with your existing resource.

Figure 8 below summarises some key Delivery Efficiency metrics.

 

Speeding Transitions Rate

We have put this metric first as it may be a critical start-point for many delivery teams. If teams are not accurately using their workflow management tool (e.g. Jira), then the visibility of the delivery process declines very rapidly. As such, it is very difficult to make meaningful efficiency improvements. This metric, therefore, tracks how accurately teams are using ticket statuses in Jira. Engineers may forget to update ticket status in real-time and ‘shepherd’ tickets through multiple statuses at the end of a sprint to ‘tidy up’. As such, you have no idea of the real duration of each status and when it really occurred. The Speeding transitions rate shows the proportion of tickets (by individual and team) that have been ‘sped through the process’ rather than updated in real-time. This is, therefore, a critical place to start – to encourage teams to use Jira correctly so that the delivery process can then be analysed and improved.

 

Flow Efficiency

Flow Efficiency is one of the Flow Metrics and is a critical measure of efficiency that all delivery teams should track and act upon. It analyses ticket statuses to calculate the proportion of time tickets remain in an ‘active’ status versus an ‘inactive’ status. As such, it shows a very helpful overall measure of the efficiency of your end-to-end delivery process. It is not uncommon for Flow Efficiency percentages to be as low as 10% – hence, for 90% of the total Lead Time, tickets are sitting in an ‘inactive’ status (such as ‘awaiting’ or ‘queuing’ at a certain point in the process).

Figure 8: Plandek screenshot showing example Delivery Efficiency dashboard
Figure 8: Plandek screenshot showing example Delivery Efficiency dashboard

First-Time Pass Rate

First Time Pass Rate is one of our favourite metrics as it is a very good proxy metric for overall delivery process health (rather than a reflection of the capabilities of the individual engineer). The metric refers to tickets successfully passing through QA without returning for additional work. A high FTPR reflects teams that are working well together, with well-defined requirements, a carefully prepared ticket backlog and highly engaged engineers – so that tickets have the highest probability of passing the first time. Conversely, a low FTPR greatly reduces velocity and drains morale.

 

Cycle Time for Stories and Code Cycle Time

These metrics examine the efficiency of two key elements of the end-to-end delivery process. Cycle Time for Stories looks at the development time elapsed and time spent in each status within the development process in order to identify bottlenecks. Code Cycle Time looks at all your completed Pull Requests (e.g. closed, merged, declined, etc.) within the specified time range and shows the average hours to complete from when the PR was opened. Not only that, but it provides full insight into the different stages that a PR goes through (time to review, time to approve, time to merge/close, time to deploy).

 

Backlog Health

The software delivery process relies heavily on a prepared backlog of well-defined tickets waiting to be worked on. Clearly, if this backlog of tickets is too small or poorly defined, the effectiveness of the whole delivery process is put at risk. Hence, Backlog Health metrics are another critical category of Agile Delivery Metrics.

 

Story Points Ready for Dev

As the name suggests, this is the core Backlog Health metric. It shows the available backlog of defined tickets expressed in story points and should be considered relative to the velocity of teams to ensure that there are always 1-2 sprints of backlog ‘cushion’. This ensures that teams always have well-defined tickets to work on and never wait for tickets to be defined and/or are forced to work on poorly defined tickets.

 

Time to Design Stories

Tracks the time taken to design and prepare tickets. This is an important consideration when analysing the time it is likely to take to recover if there is an insufficient backlog.

 

Stories in Backlog and Story Points in Backlog by Team

These are two further metrics to understand better the size and allocation of available backlog by the team.

 

Story Backlog Distribution

This is a very useful metric to manage the backlog management process. It shows tickets in the backlog by status (e.g. ‘in design’, ‘in copy’, ‘to be refined’) so that you can see where effort is required to maintain your backlog health and hence overall delivery health.

 

Orphaned Stories

All stories should be linked to a ‘parent’ (e.g. an epic) rather than unassigned or ‘orphaned’. This process should take place at the outset when a story is prepared and placed in the backlog. Hence this metric is another useful measure of backlog health.

Figure 9: Plandek screenshot showing example Backlog Health dashboard
Figure 9: Plandek screenshot showing example Backlog Health dashboard

Delivery Quality

There are very many Delivery Quality metrics that can be applied. We have selected three simple quality metrics.

 

Escaped Defects

This is a good summary quality metric that tracks the number of Escaped Defects (bugs) identified over time. The defects can be tracked by severity and remain a fundamental measure of the quality of software output. It can be expressed as Defect Density for a more representative view of quality, which is often defined as the number of defects per 1,000 lines of code or function points.

 

Unresolved P1 and P2 Bugs

This metric is useful in tracking the impact of poor quality on the delivery process. An increased backlog of priority 1 and 2 bugs is not only bad for the end-user but also bad for the delivery organisation, which will struggle to deliver new features with the bug backlog hanging over the teams.

 

P1 Resolution Time

This is related to the Unresolved Bugs metric as it allows managers to understand the impact of the backlog of unresolved bugs both for the end-user and the delivery team. A long P1 Resolution Time suggests that customers will be negatively impacted by the bug for a prolonged period, whilst symptomatic of a delivery team/ process that is unable to operate effectively.

Figure 10: Plandek screenshot showing example Delivery Quality dashboard
Figure 10: Plandek screenshot showing example Delivery Quality dashboard

Flow Metrics

The Flow Metrics are popularised in the Flow Framework and are aligned with the concept of Value Stream Management as they are designed to measure the rate of value delivery for software products from a customer perspective.

 

Flow Efficiency

Flow Efficiency is a core Flow Metric and is a critical measure of efficiency that all delivery teams should track and act upon. It analyses ticket statuses to calculate the proportion of time tickets remain in an ‘active’ status versus an ‘inactive’ status. As such, it shows a very helpful overall measure of the efficiency of your end-to-end delivery process. It is not uncommon for Flow Efficiency percentages to be as low as 10% – hence, for 90% of the total Lead Time, tickets are sitting in an ‘inactive’ status (such as ‘awaiting’ or ‘queuing’ at a certain point in the process).

 

Flow Velocity

Flow Velocity tracks the speed of value delivery and measures the number of ‘Flow items’ of each type (i.e. by value stream) over time.

 

Flow Time

Flow Time is a similar metric to Time to Value. It measures the time taken for Flow Items to progress from ‘work start’ to ‘work complete’.

 

Flow Load

Flow Load is a measure of WIP and is designed to measure the utilisation of teams across value streams. Flow Load measures the number of Flow Items currently in progress across a team or value stream.

 

Flow Distribution

Flow Distribution considers work types underway across specific timeframes and is therefore, helpful when prioritising resources/work. Flow Distribution tracks the ratio between Flow item types over a period of time.

 

The Flow Metrics are popular and helpful, but in our view, they form only part of a holistic view of delivery capability and Agile DevOps maturity.

In addition, they can prove hard to operationalise at the team level, so may prove helpful for tech leadership, but less so for the teams themselves, who may find them in isolation quite far removed from the day-to-day engineering and delivery challenges.