Metrics
5. Delivery and Agile Metrics
Figure 6 below shows a summary of the end-to-end software delivery process, the core toolsets that underpin that process; and the key categories of metrics that are derived from the various data sources.
Delivery and Agile metrics track the effectiveness of the end-to-end delivery process and, therefore, draw data from all the toolsets that sit across that process (see Figure 6 above).
In this section, we have divided delivery and agile metrics into five categories:
- Value delivered
- Delivery efficiency
- Dependability (covered under the Sprint Metrics section)
- Backlog health
- Delivery quality
In the ‘Delivery quality’ section, we have added the four Flow Metrics popularised in the Flow Framework as relevant in this context.
Value Delivered Metrics
Figure 7 below shows our choice of ‘value delivered’ metrics. Delivery of value is always tricky to measure, and many organisations end up using the proxy measure of story points delivered (throughput), though some organisations may use ‘value points’ as an alternative to story points. Typical metrics include therefore:
Stories Delivered by Epic
A core metric that simply tracks stories delivered by Epic. An Epic is designed to reflect a (valuable) increment of work that is meaningful to the organisation. This metric is, therefore, a decent proxy of value delivered by relevant increment of work (epic). You may also want to consider Stories Delivered by Sprint and Stories Delivered by Benefit (if defined as a custom field).
Delivered Value Points
For those organisations using value points, this is clearly a critical metric and may also be viewed by Epic or Sprint, for example.
Clearly, value is not delivered until the software is deployed to live. Hence, value delivered metrics should also measure total Lead Time (from the time a ticket leaves the backlog until the time deployed to live) rather than Cycle Time, which typically just looks at the time a ticket spends in development (pre-integration test and deployment).
Lead Time for Epics, Lead Time for Stories
Time is a critical factor in software delivery, so being able to understand Lead/Cycle Time and potential bottlenecks makes this a must-have. Lead Time is calculated by looking at completed tickets over time and adding up the elapsed time in calendar days (including weekends) of all statuses of those tickets. You then divide the total time by the number of completed tickets to derive the Lead Time.
As the Integration-test-deployment phase of the delivery process is a key potential bottleneck in value delivery, two other metrics we tend to include under the Value Delivered umbrella are Mean Build Time and Deployments by Pipeline.
Mean Build Time
This is a helpful metric to identify slow-build processes that affect the ability of the team to deliver software. A steadily increasing mean workflow time will want to be addressed and will drive longer cycle times. You can filter this metric by status to help keep an eye on slow builds, which ultimately end in failure.
Deployments by Pipeline
As the name suggests, this is an important measure of your ability to deploy increments of software to live rapidly. It becomes more powerful when filters and breakdowns are applied by project or workflow name, for example.
Delivery Efficiency Metrics
Measuring throughput (or value delivered) is a very good starting point, but it is clearly also very important to understand how efficiently you are delivering that value in order to understand how you might increase velocity and deliver value more often with your existing resource.
Figure 8 below summarises some key Delivery Efficiency metrics.
Speeding Transitions Rate
We have put this metric first as it may be a critical start-point for many delivery teams. If teams are not accurately using their workflow management tool (e.g. Jira), then the visibility of the delivery process declines very rapidly. As such, it is very difficult to make meaningful efficiency improvements. This metric, therefore, tracks how accurately teams are using ticket statuses in Jira. Engineers may forget to update ticket status in real-time and ‘shepherd’ tickets through multiple statuses at the end of a sprint to ‘tidy up’. As such, you have no idea of the real duration of each status and when it really occurred. The Speeding transitions rate shows the proportion of tickets (by individual and team) that have been ‘sped through the process’ rather than updated in real-time. This is, therefore, a critical place to start – to encourage teams to use Jira correctly so that the delivery process can then be analysed and improved.
Flow Efficiency
Flow Efficiency is one of the Flow Metrics and is a critical measure of efficiency that all delivery teams should track and act upon. It analyses ticket statuses to calculate the proportion of time tickets remain in an ‘active’ status versus an ‘inactive’ status. As such, it shows a very helpful overall measure of the efficiency of your end-to-end delivery process. It is not uncommon for Flow Efficiency percentages to be as low as 10% – hence, for 90% of the total Lead Time, tickets are sitting in an ‘inactive’ status (such as ‘awaiting’ or ‘queuing’ at a certain point in the process).
First-Time Pass Rate
First Time Pass Rate is one of our favourite metrics as it is a very good proxy metric for overall delivery process health (rather than a reflection of the capabilities of the individual engineer). The metric refers to tickets successfully passing through QA without returning for additional work. A high FTPR reflects teams that are working well together, with well-defined requirements, a carefully prepared ticket backlog and highly engaged engineers – so that tickets have the highest probability of passing the first time. Conversely, a low FTPR greatly reduces velocity and drains morale.
Cycle Time for Stories and Code Cycle Time
These metrics examine the efficiency of two key elements of the end-to-end delivery process. Cycle Time for Stories looks at the development time elapsed and time spent in each status within the development process in order to identify bottlenecks. Code Cycle Time looks at all your completed Pull Requests (e.g. closed, merged, declined, etc.) within the specified time range and shows the average hours to complete from when the PR was opened. Not only that, but it provides full insight into the different stages that a PR goes through (time to review, time to approve, time to merge/close, time to deploy).
Backlog Health
The software delivery process relies heavily on a prepared backlog of well-defined tickets waiting to be worked on. Clearly, if this backlog of tickets is too small or poorly defined, the effectiveness of the whole delivery process is put at risk. Hence, Backlog Health metrics are another critical category of Agile Delivery Metrics.
Story Points Ready for Dev
As the name suggests, this is the core Backlog Health metric. It shows the available backlog of defined tickets expressed in story points and should be considered relative to the velocity of teams to ensure that there are always 1-2 sprints of backlog ‘cushion’. This ensures that teams always have well-defined tickets to work on and never wait for tickets to be defined and/or are forced to work on poorly defined tickets.
Time to Design Stories
Tracks the time taken to design and prepare tickets. This is an important consideration when analysing the time it is likely to take to recover if there is an insufficient backlog.
Stories in Backlog and Story Points in Backlog by Team
These are two further metrics to understand better the size and allocation of available backlog by the team.
Story Backlog Distribution
This is a very useful metric to manage the backlog management process. It shows tickets in the backlog by status (e.g. ‘in design’, ‘in copy’, ‘to be refined’) so that you can see where effort is required to maintain your backlog health and hence overall delivery health.
Orphaned Stories
All stories should be linked to a ‘parent’ (e.g. an epic) rather than unassigned or ‘orphaned’. This process should take place at the outset when a story is prepared and placed in the backlog. Hence this metric is another useful measure of backlog health.
Delivery Quality
There are very many Delivery Quality metrics that can be applied. We have selected three simple quality metrics.
Escaped Defects
This is a good summary quality metric that tracks the number of Escaped Defects (bugs) identified over time. The defects can be tracked by severity and remain a fundamental measure of the quality of software output. It can be expressed as Defect Density for a more representative view of quality, which is often defined as the number of defects per 1,000 lines of code or function points.
Unresolved P1 and P2 Bugs
This metric is useful in tracking the impact of poor quality on the delivery process. An increased backlog of priority 1 and 2 bugs is not only bad for the end-user but also bad for the delivery organisation, which will struggle to deliver new features with the bug backlog hanging over the teams.
P1 Resolution Time
This is related to the Unresolved Bugs metric as it allows managers to understand the impact of the backlog of unresolved bugs both for the end-user and the delivery team. A long P1 Resolution Time suggests that customers will be negatively impacted by the bug for a prolonged period, whilst symptomatic of a delivery team/ process that is unable to operate effectively.
Flow Metrics
The Flow Metrics are popularised in the Flow Framework and are aligned with the concept of Value Stream Management as they are designed to measure the rate of value delivery for software products from a customer perspective.
Flow Efficiency
Flow Efficiency is a core Flow Metric and is a critical measure of efficiency that all delivery teams should track and act upon. It analyses ticket statuses to calculate the proportion of time tickets remain in an ‘active’ status versus an ‘inactive’ status. As such, it shows a very helpful overall measure of the efficiency of your end-to-end delivery process. It is not uncommon for Flow Efficiency percentages to be as low as 10% – hence, for 90% of the total Lead Time, tickets are sitting in an ‘inactive’ status (such as ‘awaiting’ or ‘queuing’ at a certain point in the process).
Flow Velocity
Flow Velocity tracks the speed of value delivery and measures the number of ‘Flow items’ of each type (i.e. by value stream) over time.
Flow Time
Flow Time is a similar metric to Time to Value. It measures the time taken for Flow Items to progress from ‘work start’ to ‘work complete’.
Flow Load
Flow Load is a measure of WIP and is designed to measure the utilisation of teams across value streams. Flow Load measures the number of Flow Items currently in progress across a team or value stream.
Flow Distribution
Flow Distribution considers work types underway across specific timeframes and is therefore, helpful when prioritising resources/work. Flow Distribution tracks the ratio between Flow item types over a period of time.
The Flow Metrics are popular and helpful, but in our view, they form only part of a holistic view of delivery capability and Agile DevOps maturity.
In addition, they can prove hard to operationalise at the team level, so may prove helpful for tech leadership, but less so for the teams themselves, who may find them in isolation quite far removed from the day-to-day engineering and delivery challenges.
