Critical Metrics to Keep Delivering Software Effectively in the “New Normal” World

Key takeaways

• It is vital that technology leaders understand the health of their delivery capability in the ‘new normal’ world of remote working, uncertainty and cost pressure.
• This requires the ability to track a set of critical metrics.  For organisations delivering software in an Agile way, a sensible place to start is a hierarchy of metrics that tie back to the core Agile principle of “the early and continuous delivery of valuable software”.
• Our five overall delivery health metrics for the ‘new normal’ world, which are meaningful when tracked over time at an aggregate level and give your whole organisation a simple set of metrics around which to align, are Time to Value, Deployment Frequency, Throughput, Defect Density, Team Engagement.
• Our top five cascaded delivery metrics for managers and teams, which drive the five over-riding metrics at the top of the metrics hierarchy, are Deployment Frequency, Flow Efficiency, Cycle Time and Lead Time, Completion Rate, and Engineer Morale Score.
• We hope that many organisations will take the metrics suggested as a good place to start.  However, you may prefer to build your bespoke metrics.  Whichever metrics you choose, In our view, it is the discipline of tracking and managing metrics (that reflect core Agile principles) that is critical in the ‘new normal’ world.

The world has changed dramatically, and a “new normal” has appeared almost overnight – a time of remote working, great uncertainty, changing priorities and dramatic cost pressures.

Software delivery teams sit at the heart of this challenging new environment as organisations look to them to deliver more for less in strategically critical areas.

Metrics, visibility, and risk management were already increasing priorities in Agile software delivery – particularly in large-scale organisations. But recent events have seen these catapulted from important to essential, as the ‘new normal’ world presents a whole new set of challenges.

This article describes how one company (which is involved in data-driven software delivery) is practising what it preaches by not losing sight of a few key metrics during this period of intense change and uncertainty.

Using metrics to understand the health of your delivery capability in the ‘new normal’ world

The ‘new normal’ world creates three sets of immediate challenges for delivery teams:

• It requires an immediate change to well-established ways of working – most notably due to forced remote working – so that daily routines and face-to-face communication are changed in both obvious and more subtle ways;
• Demands on delivery teams remain the same and are, in many of our client’s cases, increasing – with more to do, changing priorities and fewer available resources of all types and
• The very people upon whom success depends will never have felt more unsettled and unsure about the future.

It is vital that technology leadership fully understand the health of their delivery capability.  How are teams feeling?   How is the delivery process being affected by remote working and rapidly changing priorities?  And is Time to Value slowing dramatically, and if so, how do we mitigate it?

This, in turn, means better visibility of processes and teams and the ability to track a set of critical metrics that objectively answer questions such as these.

For organisations delivering software in an Agile way, a sensible place to start is a set of metrics that tie back to core Agile principles – so that everyone is focused on the ultimate Agile goal of increasing customer satisfaction through “the early and continuous delivery of valuable software” – despite the challenges thrown up by the ‘new normal’ world.

As Reuben Sutton, Plandek’s VP of Engineering, notes:

“We have had to move to a fully remote working environment overnight during one of the most intense software delivery periods our company has ever known.  The Agile delivery metrics that our teams track and understand have been our ‘North star’.  We know that we are still going in the right direction, as we can see it objectively in the metrics.”

Creating a hierarchy of simple metrics that everyone understands

The world has changed dramatically, and a “new normal” has appeared almost overnight – a time of remote working, great uncertainty, changing priorities and dramatic cost pressures.

Software delivery teams sit at the heart of this challenging new environment as organisations look to them to deliver more for less in strategically critical areas.

Metrics, visibility, and risk management were already increasing priorities in Agile software delivery – particularly in large-scale organisations. But recent events have seen these catapulted from important to essential, as the ‘new normal’ world presents a whole new set of challenges.

This article describes how one company (which is involved in data-driven software delivery) is practising what it preaches by not losing sight of a few key metrics during this period of intense change and uncertainty.

Figure 1. The principle of a metrics hierarchy supporting the core objective of Agile software delivery

Critically, the key is also to adopt leading metrics that are deterministic of improving the process (and predicting likely outcomes) – rather than lagging metrics that simply “look in the rear-view mirror”.  So, what is a sensible set of metrics to adopt?

The top of the metrics hierarchy – five overall delivery health metrics for the ‘new normal world

We have chosen five simple metrics that capture the overall health of your delivery capability as it navigates through the tricky road ahead.  These metrics have three characteristics:

• They are well-known “Agile metrics” that ensure you are still meeting your core Agile commitment of delivering value as quickly and efficiently as possible;
• They are meaningful when aggregated; and
• They are simple.

The metrics that we have selected are shown below in Figure 2.

They need to be sponsored and viewed at the aggregate level by technology leadership.  They are supported by a set of sub-metrics that are cascaded down the organisation to the team level so that teams work to continuously improve them and hence deliver to the technology leader’s overall delivery goals.

Figure 2. The five core metrics that underpin delivery health in the ‘new world’

Together, these five metrics summarise your “Agile health” – your ability to deliver software effectively despite the constraints.  They are meaningful when tracked over time at an aggregate level – and give your whole organisation a simple set of metrics around which to align.

1. Time to Value – the core Agile metric tracking how “early” you are delivering value for customers.  Measured from the beginning of the development process through to deployment.
2. Deployment Frequency – a key metric of how “continuously” you are delivering
3. Throughput – how much value are you delivering?  There are many ways Agile teams will measure this, but Story Points, Value Points, or Tickets Completed are a common place to start
4. Defect Density – a key measure of your ability to continue to deliver high-quality software, commonly measured by the ratio of stories delivered to Escaped Defects (or production defects)
5. Team Engagement – our favourite metric in the ‘new normal’ world of remote working.  Best collected via polling on collaboration hubs like Slack and, in our view, the key leading indicator of delivery health, as software delivery is dependent on your team struggling with the pressures of the current environment.

As Reuben Sutton comments: “These are the metrics that I have held close as the level of uncertainty has increased in recent weeks.  We all understand them, it’s pointless to try and gamify them – and assuming they remain in relatively good shape, I can stay confident that we will continue to deliver effectively going forward.”

Cascaded metrics – our top 5 delivery metrics for managers and teams in the ‘new normal’ world

The power of metrics is realised if they are vocally sponsored by leadership and are then cascaded across the organisation to the key functions, programs and Agile teams (squads) responsible for software delivery.

As shown in Figure 3, our preferred cascade of metrics falls into easily understood areas, all of which drive the five overriding Agile metrics shown in Figure 2.

The metrics fall into seven logical groups, which are measurable (and actionable) at the program and team level and which drive the delivery team’s health and effectiveness.  These are:

1. DevOps process effectiveness metrics – include metrics such as Deployment Frequency and Lead Time for Changes and remain key measures of the team’s ability to deploy effectively despite the new pressures.
2. Backlog analysis – backlog size and health metrics are always important leading indicators of program health and are even more salient at a time of remote working and constantly changing priorities.
3. Development process efficiency – Flow Efficiency is a little measured but is a critical metric, especially at a time like this. It looks at the proportion of time that a team is actively working on a ticket (i.e. active vs inactive states of delivery). Ensuring that collaboration remains high and members are effectively working together during this period is a critical measure of your delivery health.
4. Throughput and Time to Value – the core Agile metrics of Velocity, Cycle Time and Lead Time have never been more important as the measure of throughput.
5. Delivery Accuracy – evidence shows that if teams (working in a Scrum Agile environment) are unable to deliver their Sprint goals accurately, then the predictability of output over longer periods and broader team sizes is seriously compromised.  Predictable delivery will be a highly valued capability in the ‘new normal’ world. Hence, the leading indicator of the Completion Rate is very important. Completion Rate shows the percentage of planned story points completed over a time period.
6. Quality – this is always a ‘north star’ Agile metric and can be measured in many different ways. We like Defect Density as an expression of defects created per unit of code but also feel strongly that First Time Pass Rate is an important measure at the individual engineer level (but only when viewed in context within the team).
7. Talent – we can’t emphasise enough how important we believe it is to understand how your team is feeling in the ‘new normal’ world.   We think you should track a measure of morale and also a measure of how well people view their team to be functioning in the new remote working environment.

Figure 3. Suggested programme and team-level metrics in the ‘new normal’ software delivery world

There are a lot of metrics here to consider (and too many to realistically collect without an Agile metric BI platform).  So, if we had to choose five simple metrics to start within the ‘new normal’ world, the top five that we would choose are:

As Alice Dunn, Senior Developer at Plandek, notes: “It’s been pretty crazy, and the move to home working took some getting used to.  So, at the team level, we have focused on our Cycle Time and our Completion Rate.  We know that if we can keep hitting our sprint goals consistently and our Cycle Time remains tight, we won’t miss a beat in our delivery…”

Choosing metrics that reflect your particular objectives

We hope that many organisations will take the metrics suggested in this article as a good place to start.  And we would agree, as they would ensure that you continue to deliver against the most central Agile principles at a time of rapid change and stress.

However, you may prefer to build your bespoke metrics set that more closely mirrors your specific objectives.

With this in mind, there are a variety of commonly used metrics setting approaches – such as OKRs (Objectives and Key Results) or GQMs (Goal, Metric, Question) as popularised by Victor Basili, which can be applied to define a bespoke metric set.

In our view, whichever route you take is very much up to you, but it is the discipline of tracking and managing metrics (that reflect core Agile principles) that is critical in the ‘new normal’ world – so that when the chips are down, everyone across your organisation is focused on the things that matter.

As Reuben Sutton, Plandek’s VP of Engineering, concludes, “Aligning around shared goals and metrics is so critical in difficult times.  We have a simple set of Agile metrics that we track very closely, and so far, our delivery teams seem able to more than meet the challenges being thrown at us.”