Get started

The start point

Organisations increasingly rely on their technology ‘department’ to deliver technology initiatives (e.g. websites, apps, billing systems etc.) that are critical to their future.

Despite this reliance on technology teams to deliver, most senior managers outside technology have a limited understanding of the process of delivering technology projects (i.e. software). As a result, they don’t know what questions to ask, dialogue can remain constrained and misunderstanding and frustrations may proliferate.

This short paper aims to open the little understood ‘black box’ that is technology delivery to enable a better relationship between technology teams and their senior internal stakeholders. As such, it raises two key questions:

  1. How can senior leadership outside technology better understand the technology delivery process?
  2. What simple metrics can be shared, that help internal customers outside technology better understand the effectiveness of their software delivery colleagues?

This short paper answers both these questions and should help improve the relationship between technology teams and their internal customers which is critical in a world where effective technology delivery is increasingly critical for success.

Some key background – ‘Agile’ software delivery

In the ‘old days’ technology teams delivered software projects in a similar way to construction teams. This methodology is known as ‘Waterfall’. Owing to its similarity to a traditional house build methodology, it is an approach that is easy for non-technology folks to understand and is summarised in the graphic below.

Waterfall Methodology

The problem is that the waterfall methodology has some major drawbacks for software delivery – the key one being that projects are often delivered months/years after they were first designed and hence often do not meet the constantly changing needs of the internal customer. Indeed the people who signed off the original spec may no longer even work in the business…

As a result, the technology world looked for a better way and the ‘Agile Manifesto’ was born in 2001. It was conceived in a ski resort in Utah and was written down as a 12 point Manifesto. https://agilemanifesto.org/

It aims to address the key drawbacks of waterfall and has been a hugely successful concept and is now adopted (in some shape) by around 90% of all organisations globally. As such it is the de facto standard for software delivery.

It is critical that non-technology senior execs understand it, as without that understanding it is virtually impossible to have a sensible discussion with a technology delivery team

The key principles of Agile are very different to the waterfall methodology:

  1. Instead of one big team, we will split into efficient and motivated teams or ‘squads’ of 6-10 people
  2. Instead of delivering in one long increment of many weeks, we will work in short ‘sprints’ of around 2 weeks (so called ‘Scrum Agile’)
  3. Instead of little contact with the client and testing at the end, we will test as we go and constantly engage with the client; and crucially
  4. Instead of defining everything in detail up front, we will work more incrementally, taking on board changing circumstances and customer needs and only define a small increment upfront and keep defining additional increments as we go
  5. As such instead of releasing everything to live in one go at the end, we will release ‘little and often’ – regular increments as we progress.

Hence the core commitment in the Agile Manifesto which is “..the early and continuous delivery of valuable software.”

Scrum Agile Delivery Diagram

Agile – The key watch-out

Agile software delivery has brought some huge benefits (hence its global adoption) – but it is not without its drawbacks (particularly at scale) – and its very nature means it can breed frustration with internal customers used to the perceived and often illusory clarity of the waterfall methodology, that in theory delivered a finished product on a particular date.

So the key watch-out that internal stakeholders need to understand is that:

‘In Agile, projects are not scoped in detail upfront, so committing to a delivery date of a ‘finished product’ is very difficult – instead internal clients should expect regular delivery of small increments of progress.’

As a result, if a CMO asks “Will we get the new app live in time for the start of Christmas trading?” – many Agile software teams will struggle to answer.

So what is a sensible set of metrics (KPIs) that internal clients should expect from Agile delivery teams so that they better understand each other?

Simple metrics for shared understanding – the metrics the C-Suite should ask for

Fortunately the Agile delivery process is easily measurable as there is a rich digital footprint in the tool-sets used across the process – from pre-development; development; integration & deployment; and out into live software management.

As such, it is possible with Value Stream Management tools like Plandek to surface a set of metrics that track the end-to-end software delivery process and give the internal client a much clearer understanding of likely delivery timing and confidence in the effectiveness of the technology delivery team.

Our six selected metrics focus on simple measures that reflect the core aims of Agile software delivery and are easy to understand inside and outside technology.

6 Agile Delivery Metrics for Leadership

Lead Time

Lead Time is a core agile software delivery metric which tracks an organisation’s ability to delivery software early and often. The concept of Lead Time is borrowed from lean manufacturing.

Lead Time refers to the overall time to deliver an increment of software from initial idea through to deployment to live – i.e. the complete end-to-end software delivery life cycle (SDLC). As such it is probably the first metric that the C-Suite should ask for to better understand how effectively a technology team is delivering.

The shorter the Lead Time, the higher the ‘velocity’ of the delivery team and hence the quicker the organisation is going to receive new features….

Deployment Frequency

Deployment Frequency is another fundamental measure of Agile software delivery. A core objective of Agile delivery is the ability to develop and deploy to live small software increments rapidly.

Deployment Frequency tracks that base competence and is a powerful metric around which to focus effort at all levels in the delivery organisation. Hence it is another key KPI for the C-Suite.

Flow Efficiency

Flow Efficiency looks at the proportion of time tickets (being worked on by the technology team) spend in an ‘active’ versus ‘inactive’ status. Clearly, the less time they spend in an ‘inactive’ status the more efficient the end-to-end process and the quicker software will be delivered. Typical opportunities to remove inactive bottlenecks include time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA (testing).

Delivered Story Points

Delivered Story Points is often considered a problematic metric due to the potential inconsistencies in the calculation of story points and how much effort they represent. However, as a basic measure of output and how that is changing over time, it is the most effective measure of ‘throughput’ of the technology team – i.e. how much work they are completing over a time period.

Escaped defects

Escaped Defects is a simple but effective measure of overall software delivery quality. It can be tracked in a number of ways, but most involve tracking defects by criticality/priority.

When these simple Agile delivery metrics are viewed together, the C-Suite can get a good balanced view of how effectively the technology team is delivering. The key is to see improvements over time – as continuous improvement is another key principle of the Agile Manifesto.

Sprint Target Completion

‘Scrum Teams’ (also known as squads) and ‘Sprints’ are the basic building blocks of Scrum Agile software delivery. If Scrum Teams consistently deliver their Sprint goals (a ‘Sprint’ typically involving a two-week increment of work), Agile software delivery becomes relatively predictable.

On the other hand, if Scrum teams fail to deliver their planned sprint goals, then it becomes impossible to predict delivery outcomes across multiple teams and longer time periods. Scrum team predictability (often referred to as ‘dependability’) is therefore a critical success criterion in Agile software delivery.

Sprint Target Completion is the basic measure of a Scrum Team’s ability to hit their self-imposed sprint goals – and hence their dependability. It is a simple metric calculated as the percentage of tickets completed within a sprint from the tickets that started the sprint.

High performing Scrum teams will consistently have Sprint Target Completion rates in excess of 85%.

Conclusion

Technology teams have many other responsibilities that these metrics do not cover (e.g. managing the resilience of live systems) – but in terms of understanding their core raison d’etre which is ‘building new stuff’ (i.e. building software) – these metrics are an excellent place to start.

They represent a balanced scorecard that addresses the key elements of Agile software delivery and just discussing them starts to open a meaningful dialogue between stakeholder and technology team in an area that is almost always vital for the future success of the organisation.

About Plandek

Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics, analytics and reporting to optimise software delivery predictability, risk management and process improvement.

Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).

Plandek is based in London and works with clients globally to apply predictive data analytics to deliver software more effectively.

Metric Definition

Commits without a Ticket Reference is an often-overlooked engineering quality metric.

As the name suggest it tracks the percentage of code commits made to any branch that do not have a related issue management (e.g. Jira) ticket reference present in the commit to enable a full audit history of each commit.

Mature Agile DevOps organisations recognise the need to view Agile software delivery as an end-to-end process – hence also recognise the need to be able to track code commits back to the team/feature/ticket to which they relate.

Example Commits without a Ticket Reference summary chart – Plandek Delivery and Engineering Quality dashboard

Commits Without a Ticket Ref Chart

Analysis of Commits without a Ticket Reference – Breakdown and Filtering Options

Engineering metric dashboards like Plandek enable you to analyse delivery and engineering metrics in a number of different ways. In this case, Plandek will link to a ticket reference based in it being found in any of the below:

Plandek then cross-references the ticket reference with the data gathered from your workflow management system (e.g. Jira) to validate that the ticket is genuine. Once validated, the commit will then be considered as linked to a ticket.

Based on the date of commit the default over time chart (below) plots this percentage based on your date range selection. Breakdown/Filter options include Committer and Repository.

Example Commits without a Ticket Reference drill-down chart – Plandek Delivery and Engineering Quality dashboard

Commits Without a Ticket Ref Drill Down Chart

Related metrics

Commits without a Ticket Reference is one of many delivery and engineering quality metrics. As such it is often used is part of a ‘balanced scorecard’ of engineering quality metrics surfaced in real time at team and programme level.

Other such engineering quality metrics include:

Key use cases

Commits without a Ticket Reference is an important (but often ignored) engineering quality metric. It is particularly important for:
those delivery organisations introducing DevSecOps principles
large scale delivery capabilities with distributed teams (on shore, offshore, contractor, inhouse) and potentially a higher turnover of engineering talent
teams involved in strategically critical software delivery projects
security-sensitive and tightly regulated sectors such as fintech and medtech.

Expected outcomes

Maintaining very low levels of Commits without a Ticket Reference increases the traceability of code commits hence ultimately reducing defects and speeding up issue resolution. As such it is an important ‘hygiene’ metric to ensure the visibility and security of your end-to-end delivery process.

About Plandek

Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery predictability, risk management and process improvement.

Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).

Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.

Agile DevOps metrics are an increasingly hot topic. Up until recently DevOps metrics (sometimes known as DevOps KPIs) have been constrained by the analytical capabilities of the underlying DevOps toolsets.

But with a new generation of Value Stream Management tools, it is possible to surface and synthesise data from multiple DevOps tools (e.g. workflow management tools, code repos, CI/CD tools and APM/ITSM tools). As a result, a new wave of more powerful DevOps metrics and DevOps analytics are now helping DevOps practitioners to deliver valuable software much more effectively.

Selecting the top 5 DevOps Metrics

We have selected the most popular DevOps metrics among our clients and have focused on those DevOps metrics that give you ‘the biggest bang for your buck’ whatever level of Agile DevOps maturity you are currently at.

The DevOps metrics selected are similar to the DORA metrics as popularised by the DevOps Research and Assessments (DORA) group. But we have based our top 5 DevOps metrics selection based on actual feedback (and usage) from a wide variety of Plandek clients at all levels of Agile DevOps maturity. As such, our Top 5 DevOps metrics reflects what we actually see being used to drive value among our clients!

Our Top 5 DevOps metrics

In no particular order, here are our Top 5 DevOps metrics that make an immediate impact on your software delivery.

Deployment Frequency

Deployment Frequency is perhaps the most fundamental DevOps metric as it measures the ability of an organisation to develop and deploy to live small software increments rapidly.
More specifically it tracks the frequency with which increments of code are deployed to staging, testing and production.
As the core objective of Agile software delivery is ‘..the early and continuous delivery of valuable software..’, Deployment Frequency is a core Agile metric and represents a core objective of effective DevOps. Deployment Frequency is also one of the four DORA metrics.

Deployment Frequency Chart

Example Deployment Frequency chart – Plandek DevOps dashboard

The calculation of Deployment Frequency requires surfacing data from CI/CD tools (e.g. Jenkins, CircleCI). This is typically done via reporting plug-ins to such tools or via an end-to-end delivery metrics dashboard like Plandek (www.plandek.com). Delivery metric dashboards like Plandek enable you to analyse delivery, engineering and DevOps metrics in a number of different ways. For example, you can analyse the frequency of deployments by branch, portfolio, programme, repository or team. As such targets can be set and practical actions taken to increase deployment frequency.

Deployment Frequency Drill Down Chart
Example Deployment Frequency drill-down chart – Plandek DevOps dashboard

Code Cycle Time

Code Cycle Time is a key DevOps metric as it tracks the efficiency of the Pull Request process – which itself is a critical element of the end-to-end software delivery process.
More specifically, Code Cycle Time analyses all completed Pull Requests (PRs) (e.g. closed, merged, declined etc) within the specified time range and shows the average hours to complete, from when the PR was opened. Not only that but Code Cycle Time provides full insight into the different stages that a PR goes through (e.g. Time to Review, Time to Approve, Time to Merge/Close and Time to Deploy)

The method of calculation of Code Cycle Time is shown below:

Code Cycle Time Calculation: Code Cycle Time = Elapsed time of all stages on completed pull request / Count of completed pull requests

 

Code Cycle Time Chart

Example Code Cycle Time chart – Plandek DevOps metrics dashboard

A powerful DevOps analytics tools like Plandek enables you to analyse Code Cycle Time by a number of different filters in order to identify bottlenecks and reduce it significantly. Such filters include analysis by PR author, PR participant, Code Cycle Time Stage, Respository and Ticket Issue Type.

Code Cycle Time Drill Down Chart

Example Code Cycle Time drill-down chart – Plandek DevOps metrics dashboard

Mean Build Time

Mean Build Time is a very impactful DevOps metric for organisations at all levels of Agile DevOps maturity. It analyses the time taken for a build and is typically analysed by workflow.
It is a very good indicator of core DevOps process health as a steadily increasing mean workflow build time will drive longer Cycle Times.

We particularly like filtering by status to help you keep an eye on slow builds which ultimately end in failure.

Mean Build Time Chart

Example Mean Build Time view by Workflow Name

Mean Time to Recover from Build Failures

Mean Time to Recover from Build Failures is a key DevOps KPI as build failure is such a common occurrence and the main source of friction in the deployment process. As such it needs to be tracked and managed.

The metric identifies how long it takes for the next successful workflow on a branch that has failed. Mature Agile DevOps organisations become very good at managing this recovery time and so improve their deployment frequency.

Mean Time to Recover from Build Failures Chart

Example Mean Time to Recover from Build Failures metric view

Flakiest Files

Flakiest Files is a really helpful DevOps metric aimed to identify fragile source code files in your codebase which can be targeted for refactoring in order to reduce your Build Failure Rate.
The darker the shade of red the more often a commit has resulted in a failed build. Plandek’s drill-down enables you to filter by file extension to help focus on a particular subset of files.

Flakiest Files Chart

Example Flakiest Files DevOps metric view

Build Failure Rate

Eagle-eyed readers will have noticed that this is the sixth metric in our Top 5 DevOps metrics guide, but we’ve included it as it’s another key DevOps KPI that really shouldn’t be left out!
Indeed, Build Failure Rate is an extremely helpful DevOps metric as it identifies the percentage of workflows which fail and the overall risk this poses to development.
As such it helps track and manage a significant source of risk both in day to day development and responding to incidents due to the delays. Taking all your workflows over a period it calculates the percentage that ended in failure.

Build Failure Rate Chart

Example Build Failure Rate DevOps metric view

About the author
Charlie Ponsonby is co-founder and Co-CEO of Plandek (www.plandek.com) based in London.

Plandek was recently recognised as a top nine global vendor by Gartner in the fast-growing ‘DevOps Value Stream Management Platforms’ space and is used by enterprise clients globally to improve the effectiveness of their software delivery capability.

Plandek provides a cloud-based analytics platform to help software delivery teams deliver quality software, faster and more predictably. It works by mining data from delivery teams’ toolsets (such as issue tracking, code repos and CI/CD tools), to provide actionable and intelligent insight (including Value Stream metrics) across the end-to-end software delivery process.
Selected Sources:

https://www.infoq.com/articles/devops-maturity-metrics/

As the name suggests, ticket complexity allows you to understand the relative complexity of your tickets using 3 different possible Units of Measurement, all based on the commits linked to your tickets.

Unit of Measurement

Definition

Changed Lines

Total inserted and deleted lines made in commits related to each ticket

Developers

Number of unique individuals active on the commits and pull requests related to each ticket

Repositories

Number of unique repositories affected by commits related to each ticket

This metric includes all completed tickets within the selected time range that are linked to any commits.

Ticket Complexity – as shown in Plandek Delivery Efficiency dashboard

Ticket Complexity Metric Chart

Ticket Complexity – examples

Ticket Complexity by Story Points

Below we look at the complexity of tickets based on the number of developers involved. When breaking down this complexity score (average developers) by Story Points we see a strong correlation between the story points value and complexity.

This, in turn, highlights very effective estimation by this team, where their estimates are able to correctly predict the actual complexity of the work.

Ticket Complexity by Story Points – example Plandek metric visualisation

Ticket Complexity Metric Chart

Ticket Complexity – Changed Lines for different Issue Types

Using the Changed Lines unit of measurement and then breaking down the metric by Issue Type we can start to understand the relative complexity of different types of change.

Below, when looking over time, we can see a fairly consistent complexity of the Feature issue type, compared to the more sporadic complexity of Technical Improvements.

Ticket Complexity – Changed Lines by Issue Type – example Plandek metric drill-down chart

Ticket Complexity Metric

Related metrics

Ticket Complexity is an important delivery efficiency metric. Other delivery efficiency metrics that are relevant to be viewed in tandem include:

About Plandek

Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery predictability, risk management and process improvement.

Plandek’s metrics, data visualisations and insights are not available from plug-ins (e.g. Jira plug-ins) as they require collation of data from multiple data sources across toolsets.

Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).

Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.

DevOps Value Stream Management is a rapidly growing field in Agile software delivery, as it recognises that the trick to delivering valuable software, is to treat value delivery as an end-to-end process that requires mapping and optimising for the best outcomes.

Indeed, Gartner only started covering Value Stream Management as a discipline in September 2020 with the publishing of its first Market Guide in the space: https://www.gartner.com/en/documents/3991130/market-guide-for-devops-value-stream-management-platform

As the name suggests, effective Value Stream Management requires identifying (mapping) and managing individual software delivery value streams. Metrics are central to effective management, to track and drive the desired value delivery.

But in keeping with Agile principles, it is vital that the Value Stream metrics chosen engage both teams and managers. Measurement in Agile delivery has always been a contentious area and as such should not be a top down exercise (imposed on teams) – rather it should be a bottom-up exercise driven by the teams themselves.

So what are the Top 5 Value Stream metrics that engage both teams and managers alike?

Criteria for choosing our Top 5 Value Stream metrics

We have chosen the following criteria to select our Top 5 Value Stream metrics:

  1. Firstly and most importantly, the Value Stream metrics chosen track the core underlying objective of Agile software delivery (and Value Stream Management) – “the early and continuous delivery of valuable software” (Agile Manifesto Principle Number 1), https://agilemanifesto.org/iso/en/principles.html.
  2. second, the Value Stream metrics selected are valuable to teams and managers alike so that Value Stream Management is an adopted principle by all – rather than a management concept separated from the day-to-day reality of software delivery within the teams themselves
  3. the metrics are meaningful when considered at team level and when aggregated across teams
  4. and finally, the Value Stream metrics are simple to understand and if tracked, very quickly drive significant improvement in Agile software delivery effectiveness.

Our Top 5 Value Stream metrics

In no particular order, here are our Top 5 Value Stream metrics that make an immediate impact on your Value Stream Management (i.e. effective Agile delivery of valuable software). As would be expected, they closely match commonly used Agile metrics.

Time to Value (Lead Time)

Lead Time is a core Value Stream metric as it is a basic measure of your organisation’s Agility and ability to deliver value at pace. It measures your velocity, or the time taken to develop an increment of software (value).

Lead Time refers to the overall time to deliver an increment of software from initial idea through to deployment to live – i.e. the complete end-to-end software delivery life cycle (SDLC). Whilst Cycle Time is a subset of the overall delivery time, typically measured as the time from the start of work (development) until deployment to live (traditional Cycle Time) or from code commit to production (sometimes referred to as Code Cycle Time).

A Lead Time can be calculated for any increment of work – e.g. story, task, bug.

The example below shows Lead Time for stories and so relates to the time taken to deliver new features (value). As the chart shows the total Lead Time is 11 days which is actually very good and represents the time taken for a ticket (story) to leave the backlog until deployment to live.

Example software delivery Lead Time summary chart – Plandek Value Stream metrics dashboard

Lead time for stories metric

Example Lead Time status review chart – Plandek Value Stream metrics dashboard

Lead time for stories metric

As shown in the expanded view of Lead Time in the Lead Time Status Review chart above, analysis of Lead Time should become an integral part of effective Value Stream Management.

The chart shows the active and inactive status groups and the time stories spend in each status across the total 11.3 day Lead Time.

In the example above, the inactive statuses such as ‘Ready for Development’ account for a relatively large proportion of the overall Lead Time (5.6 days), hence there is friction in the process and there is scope to reduce the overall Lead Time and thereby increase software delivery velocity (and speed of value delivery).

Deployment Frequency

Deployment Frequency is another fundamental measure of an organisation’s value delivery (when viewed alongside the other critical Value Stream metrics described here).

A core objective of Value Stream Management is the ability to develop and deploy live small software increments rapidly. Deployment Frequency is a Value Stream metric that tracks that base competence and is a powerful Value Stream metric around which to focus effort at all levels in the delivery organisation.

Example Deployment Frequency metric view

Deployment Frequency Metric

Delivered Story Points or Delivered Value Points

Delivered Story Points is often considered a problematic Value Stream metric due to the potential inconsistencies in the calculation of story points and how much effort they represent. However, as a basic measure of output and how that is changing over time, it is a powerful Value Stream metric around which to align.

Indeed, some organisations may have gone further and work in Value Points rather than story points – with each Value Point aligned with the perceived value of the story under development. In this instance the Value Stream metric becomes Delivered Value Points.

There may be concerns of teams ‘gaming’ the metric with story point/value point inflation, but as with all Value Stream metrics, they should be viewed in context by experienced folks who know the teams well. And if this is the case, they can still give an excellent view of how the delivery organisation is progressing over time.

Example Delivered Story Points metric view

Delivered story points

Escaped Defects

Escaped Defects is a simple but effective Value Stream metric of overall software delivery quality (which is inversely correlated with the delivery of value).

It can be tracked in a number of ways, but most involve tracking defects by criticality/priority as per the example below.

Example Escaped Defects metric view

Escaped defects metric

When these four simple Value Stream delivery metrics are viewed together, the Agile DevOps practitioner can get a good balanced view of how their Value Stream Management is progressing.

Importantly, the Value Stream metrics can be tracked over time, making sure that an improvement in one metric (e.g. Lead Time) does not lead to a detrimental effect on another metric (e.g. Escaped Defects).

In addition, the relationship between Lead Time and Deployment Frequency can be closely watched. Very often teams are able to reduce their Lead Time, but this does not translate into quicker value delivery, due to bottlenecks in the integration and deployment process.

Flow Efficiency

Our final Value Stream metric in our Top 5 is Flow Efficiency. Flow Efficiency looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status and is a great Value Stream metric for teams.

The Flow Efficiency analysis (see Figure 7 below), enables Team Leads to isolate and analyse each ‘inactive’ status in the workflow and consider if there is scope to reduce or eliminate it.

The analysis shows the relative size of each ‘inactive’ status opportunity in terms of time spent in the inactive state and volume of tickets affected.

Figure 7 – Example Flow Efficiency metric

Flow efficiency metric

Typical opportunities to remove inactive bottlenecks include time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA. Where waits for QA are considered excessive, Delivery Managers can reconsider QA resource allocation by team.

About the author
Charlie Ponsonby is co-founder and Co-CEO of Plandek (www.plandek.com) based in London.

Plandek was recently recognised as a top nine global vendor by Gartner in the fast-growing ‘DevOps Value Stream Management Platforms’ space and is used by enterprise clients globally to improve the effectiveness of their software delivery capability.

Plandek provides a cloud-based analytics platform to help software delivery teams deliver quality software, faster and more predictably. It works by mining data from delivery teams’ toolsets (such as issue tracking, code repos and CI/CD tools), to provide actionable and intelligent insight (including Value Stream metrics) across the end-to-end software delivery process.

Selected Sources:

https://www.infoq.com/articles/devops-maturity-metrics/

There’s an old adage – “what gets measured gets done”. But metrics are a contentious subject in Agile software delivery, with some Agile practitioners contending that Agile metrics are a bad thing, as at best they can be gamed and manipulated by teams – and at worst may instil a negative culture of “Big Brother is watching you” detrimental to Agile team wellbeing.

However, this negative view of Agile metrics is very much on the decline – with most more mature Agile DevOps organisations recognising the core role that Agile metrics play if carefully selected and therefore adopted enthusiastically by Agile teams.

So what are the Top 5 Agile metrics that engage both teams and managers alike and avoid the pitfalls outlined above?

Criteria for choosing our Top 5 Agile metrics

We have chosen the following criteria to select our Top 5 Agile metrics:

  1. Firstly and most importantly, the Agile metrics chosen track the core underlying objective of Agile software delivery – “the early and continuous delivery of valuable software” (Agile Manifesto Principle Number 1), https://agilemanifesto.org/iso/en/principles.html
  2. second, the metrics selected are valuable to teams and managers alike – indeed it is vital that teams adopt Agile metrics to drive their own Agile continuous improvement (as per Principle 12 in the Manifesto)
  3. the metrics are meaningful when considered at team level and when aggregated across teams
  4. and finally, the Agile metrics are simple to understand and if tracked, very quickly drive significant improvement in Agile software delivery effectiveness.

Our Top 5 Agile metrics

In no particular order, here are our Top 5 Agile metrics that make an immediate impact on your Agile software delivery.

Cycle Time

Cycle Time is a basic measure of your organisation’s Agility, in that it measures your velocity, or the time taken to develop an increment of software. Unlike the more comprehensive Agile metric of Lead Time (which measures the length of the entire end-to-end delivery process), Cycle Time is easier to measure as it looks only at the time taken (within a scrum team) to take a ticket from the backlog, code and test that ticket – in preparation for integration and deployment to live.

As per Figure 2 below, the Cycle Time metric view allows teams to understand time spent in each ticket status within the development cycle. Analytics tools that offer filtering enable analysis by Status, Issue Type, or Epic (and any other standard or custom ticket field) all plotted over any time range required.

Figure 2 – Example Cycle Time metric view

Cycle Time Metric

 

Deployment Frequency

Deployment Frequency is another fundamental measure of an organisation’s agility (when viewed alongside the other critical metrics described here).

A core objective of Agile delivery is the ability to develop and deploy live small software increments rapidly. Deployment Frequency is an Agile metric that tracks that base competence and is a powerful Agile metric around which to focus effort at all levels in the delivery organisation at the early stages of an Agile transformation.

Figure 3 – Example Deployment Frequency metric view

Number of Deployments

Delivered Story Points

Delivered Story Points is often considered a problematic Agile metric due to the potential inconsistencies in the calculation of story points and how much effort they represent. However, as a basic measure of output and how that is changing over time, it is a powerful Agile metric around which to align.

There may be concerns of teams ‘gaming’ the metric with story point inflation, but as with all Agile metrics, they should be viewed in context by experienced folks who know the teams well. And if this is the case, they can still give an excellent view of how the delivery organisation is progressing over time.

Figure 4 – Example Delivered Story Points metric view

 

Delivered Story Points Metric

Escaped Defects

Escaped Defects is a simple but effective Agile metric of overall software delivery quality. It can be tracked in a number of ways, but most involve tracking defects by criticality/priority as per the example below.

Figure 5 – Example Escaped Defects metric view

 

Escaped Defects Metric

When these four simple Agile delivery metrics are viewed together, the Agile DevOps practitioner can get a good balanced view of how their Agile DevOps maturity is progressing.

Importantly, the Agile metrics can be tracked over time, making sure that an improvement in one metric (e.g. Cycle Time) does not lead to a detrimental effect on another metric (e.g. Escaped Defects).

In addition, the relationship between Cycle Time and Deployment Frequency can be closely watched. Very often teams are able to reduce their Cycle Time, but this does not translate into quicker value delivery, due to bottlenecks in the integration and deployment process.

Flow Efficiency

Our final Agile metric in our Top 5 is Flow Efficiency. Flow Efficiency looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status) and is a great Agile metric for teams.

The Flow Efficiency analysis (see Figure 7 below), enables Team Leads to isolate and analyse each ‘inactive’ status in the workflow and consider if there is scope to reduce or eliminate it. The analysis shows the relative size of each ‘inactive’ status opportunity in terms of time spent in the inactive state and volume of tickets affected.

Figure 7 – Example Flow Efficiency metric

Flow Efficiency Metric

Typical opportunities to remove inactive bottlenecks include time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA. Where waits for QA are considered excessive, Delivery Managers can reconsider QA resource allocation by team.

About the author
Charlie Ponsonby is co-founder and Co-CEO of Plandek based in London.

Plandek was recently recognised as a top nine global vendor by Gartner in the fast-growing ‘DevOps Value Stream Management Platforms’ space and is used by enterprise clients globally to improve the effectiveness of their software delivery capability.

Plandek provides a cloud-based analytics platform to help software delivery teams deliver quality software, faster and more predictably. It works by mining data from delivery teams’ toolsets (such as issue tracking, code repos and CI/CD tools), to provide actionable and intelligent insight (including Agile metrics) across the end-to-end software delivery process.

Selected Sources:

https://www.infoq.com/articles/devops-maturity-metrics/
https://www.atlassian.com/agile/project-management/metrics

Plandek backlog health metric of the week: Story Points Ready for Development

Metric Definition
Story Points Ready for Dev is a key Agile software delivery metric as it is an important measure of backlog health.

For Agile software delivery to work effectively there must be a constant flow of work through development teams. As such sufficient tickets must be prepared and ‘Ready’ in the backlog, to ensure that teams are fully utillised (whether under a Scrum Agile or Kanban Agile methodology).

Tickets defined as ‘Ready’ should have four attributes:

Story Points Ready for Dev as the name suggests shows the total number of Story Points residing within tickets that are currently in the ‘Ready’ status. For teams not using Story Points, ticket count can be a useful alternative so long as the throughput of tickets is known and a reliable measure.

Typically, in a Scrum Agile environment teams would expect to have Story Points Ready for Dev equating to 2 times team velocity. Velocity is defined as the amount of Story Points typically completed within a sprint. As such a healthy backlog would be defined as one that contains tickets ready for development equating to 2 sprints of work.

Example Story Points Ready for Dev chart – Plandek Backlog Health metrics dashboard

Example Story Points Ready for Dev chart – Plandek Backlog Health metrics dashboard

The Story Points Ready for Dev drill-down chart below shows how Team Leaders would typically analyse the metric. It shows the total Story Points Ready for Dev as it changes over time and also shows the metric by Board.

Example Story Points Ready for Dev drill-down chart – Plandek Backlog Health metrics dashboard

Example Story Points Ready for Dev drill-down chart – Plandek Backlog Health metrics dashboard

Related metrics
Story Points Ready for Dev is a key backlog health metric. Other backlog health metrics that are relevant to be viewed in tandem include:

Key use cases
Story Points Ready for Dev is a very useful Backlog Health metric to help teams work effectively by ensuring that they always have a healthy backlog of tickets on which to start work.

As a rule of thumb Story Points Ready for Dev should equate to 2 times team Velocity and therefore represent 2 Sprints worth of work to do. As such, Story Points Ready for Dev is a core Agile metric and is often included as part of a broader ‘balanced scorecard’ of Agile delivery metrics as it is relevant both at Team level and when aggregated across Teams and shown relative to average Velocity.

Expected outcomes
Maintaining a healthy level of Story Points Ready for Dev is critical to effective Agile software delivery. The core objective of Agile delivery is after all “the early and continuous delivery of valuable software”. And this is simply not possible without a healthy and well-prepared backlog ready for the Teams to get stuck into.

Maintaining Story Points Ready for Dev at 2 times Velocity will ensure that teams can continuously work on new tickets and deliver value.

However, it is not optimal if Story Points Ready for Dev levels start to go much higher than 2 times Velocity, as this starts to impose inflexibility on Teams and Agile projects, contrary to the fundamental aims of Agile delivery.

About Plandek
Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery predictability, risk management and process improvement.

Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).

Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.

Kit FriendKit Friend

This guest post was written by Kit Friend – Agile Coach and Atlassian Strategic Partner Lead for EMEA at Accenture

The views, opinions and positions expressed within this guest post are those of the author alone and do not necessarily reflect those of Accenture or their clients.

Let’s recap briefly why this Value Stream thing is getting everyone excited (and not even particularly recently – SAFe’s value stream architecture credits key elements to 2004’s Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck, and much of the logic is far older). Though there’s a huge amount of complexity in actually mapping the things within an existing complex enterprise, the concept itself is fairly simple – effectively it’s far better to organise your efforts around creating and iterating a channel that links the raw ingredients your company can access, all the way to the people willing to pay for them (usually your customers). I like to describe this in terms of two competing companies running watermills – it doesn’t matter how impressively or hugely a company builds segregated ditches from their water source, the company that builds a thin channel all the way from source to destination is going to win every time:

 Value Stream Canal Image

…and on top of this, which company is going to discover a critical fault or efficiency in paddle wheel bearings first? It’ll be the one with something working in the market and the ability to make small changes quickly of course.

So far, so good – but rallying your large siloed enterprise around the concept of a better way is just the theory box ticked, to mobilise change (particularly working the pandemic-shaped world of 2020 and beyond) it’s essential to invest in tooling to turn value streams from being a (digital) whiteboard and slideware exercise, into a living part of how you organise teams. Gartner splits these toolsets into:

Value Stream Delivery PlatformsValue Stream Delivery Platforms
(VSDPs)
Value Stream Management PlatformsValue Stream Management Platforms
(VSMPs)
‘provide a fully integrated set of capabilities to enable continuous delivery of software. These capabilities may include project or product planning, build automation, continuous integration, test automation, continuous deployment and rollback, release orchestration, and automated security policy enforcement, and may provide visibility to key value stream metrics.’ ‘enable organizations to optimize end-to-end product delivery lead time. These platforms provide greater visibility and traceability into the flow of all product delivery processes, from ideation to release and operation’
E.g.
Atlassian
CloudBees
GitHub
GitLab
JFrog
Amazon Web Services
Microsoft Azure
E.g.
Digital.ai
Plandek
ServiceNow
TaskTop

…essentially the Management Platforms seek to provide some intelligence to help knit together the lakes of data and activity being generated by the people and tooling doing the Delivery. It’s enticing to think that we should be able to jumble this all together and point some sort of neural network at these kind of problems, but giving all the data meaning and enabling teams to take actions from it isn’t quite so simple. Common issues faced by teams trying to tackle this include:

The last can often be the most difficult – where large organisations are still struggling to understand how to implement agility consistently at various levels, a lot of data can be a dangerous thing.

How to avoid creating a (data) monster

As the Large-Scale Scrum Guide stipulates:

It is common for organizations to look for tools to solve their problems even though tools are rarely the cause of the problem. Avoid solving problems with tools unless you truly, deeply understand the problem and consider a tool to be the right solution for that particular problem. https://less.works/less/framework/product-backlog.html

Over-emphasising the role of tooling can distract organisations from making the investment in supporting the people and process changes they deeply need in order for the tooling to make any sense. Instead, particularly in a maturing organisation, it is wise to view of the role of tooling as provide visual clues as to the problems you need to tackle:

A cumulative flow chart might help identify where bottlenecks in your team’s flow are
BUT

A Gantt Chart and complex permissions on ticket movement won’t de-risk your deliveries

Faced with the opportunity to create hundreds of fascinating graphs, charts, and metrics in a few clicks, it’s tempting to overwhelm teams and stakeholders with screens full of data without focussing on what is actionable. This inevitably results in the “email newsletter problem” – people only read and digest the first couple of items. When curating your tooling to support a value stream it’s therefore important to come to terms with there not being a one-size-fits all approach – far better to have a flexible set of tools which allow users to easily create views which are relevant to them.

Outcome vs Output

So you’ve assembled your Value Stream, got some fancy tooling, and have a bundle of agile teams hard at work – what could go wrong?

The next piece in the puzzle is to ensure they aren’t busy creating chocolate teapots and fish bicycles terribly efficiently. Christophe Achouiantz sum this up neatly in his writing:

How Outputs generate Outcomes that generate Impacts by Christophe Achouiantz

How Outputs generate Outcomes that generate Impacts by Christophe Achouiantz

Whilst implementing both elements of the aforementioned value stream tooling is vital to continue to optimise the Output segment here, teams not adopting Product-led ways of working and introducing a feedback loop to measure and integrate the performance of their products in live will swiftly find themselves experiencing disappointment. The rise of OKRs and platforms like www.gtmhub.com reflects the challenge in meshing together these very different worlds of data, but the majority of organisations still struggle to follow through on tracing the benefits data which is locked in before delivery begins, once the products they were associated with make it into live.

(An aside: be careful with language when comparing the concepts of efficient and effective with international teams – Howard Johnson and I encountered an amusing problem whilst teaching in Sweden, where both terms translate to the single Swedish word ‘effektiv’ – thankfully this hasn’t stopped Swedish companies like Saab becoming icons of agility)

So what?

How can these host of challenges and vague warning translate into something workable? Many of the same tips to pragmatically mixed agile transformations apply:

  1. Engage your teams, and save them time
    Try to avoid anything which creates additional overhead for your teams – anything requiring manual time logging in particular should be viewed with suspicion. Engage your teams early, and find out what data would help them take action to fix existing pain points.
  2. Plan on it being messy, full of surprises, and… not what you plan
    Don’t make the mistake of waterfalling your agile tooling – embrace an approach where elements can be created and validated early, changes made, and customers engaged.
  3. Maximising the number of tools not used vs experimentation
    Remote working expert Molood Ceccarelli cites the importance in not flooding people with too many tools – a critical philosophy in a world where more remote working away from office dashboards is likely to continue. Balance this with not narrowing down your options too early – use pilots and trials to gives teams the opportunity to provide real feedback before you narrow down your choice of tooling.
  4. Embrace the cloud
    Many organisations still struggle with opening up their data – blocking their ability to make the most of a flexible range of SaaS and PaaS services. Tackle this early, challenge assumptions (there are examples of adoption in pretty much every country and industry), and get sponsorship for the journey to the cloud – spinning up large platforms within your own infrastructure is unlikely to produce the best outcomes for supporting your teams.

Metric Definition
Code Cycle Time is a key software delivery metric as it tracks the efficiency of the Pull Request process – which itself is a critical element of the end-to-end software delivery process.

More specifically, Code Cycle Time analyses all completed Pull Requests (PRs) (e.g. closed, merged, declined etc) within the specified time range and shows the average hours to complete, from when the PR was opened.

Not only that but Code Cycle Time provides full insight into the different stages that a PR goes through. These stages are defined as:

  1. Time to Review – From open to the first comment or review
  2. Time to Approve – From the previous stage to approved
  3. Time to Merge/Close – From the previous stage to merge/close
  4. Time to Deploy – From the previous stage to deployed

The method of calculation of Code Cycle Time is shown below:

Code Cycle Time Calculcation

Example Code Cycle Time chart – Plandek DevOps metrics dashboard

Example Code Cycle Time Chart

Analysis of Code Cycle Time – Breakdown and Filtering Options
Delivery metric dashboards like Plandek enable you to analyse delivery and engineering metrics at the team level in a number of different ways.

Below is a selection of the options available to analyse Code Cycle Time:

Author The individual who created the PR. Useful to filter down to a specific individual or individuals across many repos.
Participant Any individual who took action on the PR (not including the Author). Great way to see who are the go-to team members for reviews etc. 
Stage The specific stage the PR has been through (as defined above in the article). Essential to understand where time is being spent on average across your completed PRs.
Repository Allows you to see how Code Cycle Time compares across repos and if PRs in particular repos take longer than others.
Ticket Issue Type Based on ticket reference linking, you can gain a unique perspective on this metric based on any Ticket related field you have configured. In this case, you would be able to understand code cycle time based on the related ticket being a Story or Bug etc.

Example Code Cycle Time drill-down chart – Plandek DevOps metrics dashboard

Example Code Cycle Time Drill-down Chart

Related metrics
Code Cycle Time is one of many delivery and DevOps metrics. As such it is often used is part of a ‘balanced scorecard’ of software delivery metrics surfaced in real time at team and programme level.

Other such software delivery metrics and DevOps metrics include:

Key use cases
Code Cycle Time is a very useful DevOps metric to help teams reduce their Lead Time and increase the velocity of software delivery. The Pull Request process can become a key blocker in the overall delivery process as code awaits peer review. This may be because teams rely on a few key engineers to action the majority of Pull Requests and hence bottlenecks quickly develop.

Empirical data shows that Code Cycle Time typically accounts for circa 30% of Cycle Time – so the PR process is very often a key source of ‘hidden’ delay.

Code Cycle Time is particularly important for organisations looking to increase delivery velocity (reduce Lead Time) -e.g.:

Expected outcomes
Reducing Code Cycle Time will reduce overall Lead Time and increase software delivery velocity. Empirical evidence shows that it is not uncommon to reduce Lead Time by 5-10% through tracking and effective management of Code Cycle Time.

Detailed analysis will also reveal the engineers responsible for actioning the majority of PRs and can help Team Leads reduce the pressure on these key individuals – thereby reducing pressure on the individuals themselves and at the same time reducing the ‘key-person’ risk within the software delivery capability.

About Plandek
Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery predictability, risk management and process improvement.

Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).
Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.

Metric Definition

Speeding Transitions Rate is a useful Agile delivery metric. It looks at how effectively workflow management tools (e.g. Jira, Azure DevOps) are being used by the engineering teams.  Specifically, it tracks how accurately teams are updating ticket statuses as work moves through the software delivery process.

‘Speeding tickets’ are defined as those that spent less than 60 seconds in a particular ticket status.  As such, it shows where engineers are retrospectively updating ticket statuses (e.g. at the end of a sprint), having forgotten to do so in real time.

If this is happening, it is very difficult to analyse workflow and make delivery process improvements – as there is no record of actually when tickets where in a certain status and for how long they really remained in that status.

As such when the Speeding Transitions Rate is high it becomes impossible to analyse Cycle Time and Flow Efficiency and identify where bottlenecks may lie within the end-to-end delivery process.

Speeding Transitions Rate is a popular metric in Plandek for teams keen to improve their basic processes and usage of their DevOps toolset.  As per below, Plandek allows Team Leads to define different durations, for their own definition of ‘speeding’ (e.g. they may choose to track a time length longer than 60 seconds).

 

Example Speeding Transitions Rate Formula

Example Speeding Transitions Rate chart – Plandek Delivery Efficiency dashboard

Example Speeding Transitions Rate drill-down chart

 

Analysis of Speeding Transitions Rate – Breakdown and Filtering Options

Delivery metric dashboards like Plandek enable you to analyse delivery, engineering and DevOps metrics in a number of different ways.

For example, you can analyse the Speeding Transitions Rate, by team, by issue type (as per the graphic below), by transition and over time.

Example Speeding Transitions Rate drill-down chart – Plandek Delivery Efficiency dashboard

Example Speeding Transitions Rate chart

 

This enables teams to quickly identify where the problem lies and to coach the engineers responsible in the importance of using workflow management tools effectively – in order to provide real visibility of the workflow process.

Related metrics

Speeding Transitions Rate is a useful ‘process hygiene’ metric to improve DevOps tool usage in Agile teams.

In essence, it is a stepping-stone to achieving data-driven delivery through the analysis of critical delivery metrics such as Cycle Time, Lead Time and Flow Efficiency.

Key use cases

Organisations often think that they are unable to move to a metrics-led delivery culture, because their underlying usage of tools like Jira is limited/compromised.

Indeed, we often hear delivery managers who say… ‘we are not yet ready for an analytics tool like Plandek as our underlying processes and usage of Jira need tightening up first…’

However, Plandek’s ability to surface metrics like Speeding Transitions Rate – shows why this argument is flawed.  Indeed, Plandek is a useful analytics tool especially if there is concern about imperfect usage of Jira or related DevOps tools!

Because until teams start tracking metrics like Speeding Transitions Rate and tightening up on the usage of their workflow management tools – they will never be able to move to a data-driven, continuous improvement Agile culture.

Speeding Transitions Rate should therefore be analysed as a first step towards moving to a data-led delivery culture, where teams have total visibility of their own end-to-end delivery process so that they self-improve over time.

Expected outcomes

A high level of Speeding Transitions Rate is to be avoided.  It is symptomatic of poor systems usage and results in limited visibility of teams’ workflows.  It therefore prevents, effective analysis of Cycle Time and Flow Efficiency and as such may well conceal bottlenecks in the end-to-end delivery process.

On the other hand, reducing Speeding Transitions Rate results in immediate benefit, as it makes the effective analysis of teams’ workflows possible and is therefore a key stepping-stone to data-driven Agile delivery.

About Plandek

Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery predictability, risk management and process improvement.

Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).

Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.