Change Failure Rate is a DORA metric and as such a core DevOps metric (and more broadly a core Agile delivery metric).
There are four DORA metrics popularised by the DevOps Research and Assessments (DORA) group of which Change Failure Rate is one. The others are: Deployment Frequency, Lead Time for Changes and Time to Recovery.
Change Failure Rate is an extremely helpful metric which identifies the percentage of workflows which fail to enter production and the overall risk that this poses to development -a significant source of risk both in day-to-day development and responding to incidents due to the delays. Taking all your workflows over a period it calculates the percentage that ended in failure/require remediation (e.g., require a hotfix, rollback, fix forward, patch).
Example Change Failure Rate chart – Plandek DORA metrics dashboard
The calculation of Change Failure Rate requires surfacing data from CI/CD tools (e.g. Jenkins, CircleCI). This is done via an analytics plug-in or via an end-to-end delivery metrics dashboard like Plandek (www.plandek.com).
Example Change Failure Rate drill-down chart – Plandek DORA metrics dashboard
Plandek’s powerful filtering enables you to analyse your Change Failure Rate by a number of different dimensions in order to reduce it. These include: branch, pipeline, portfolio, programme, repository and team.
Related DORA metrics
Change Failure Rate is one of four DORA metrics. As such it is often used is part of a ‘balanced scorecard’ of agile delivery and DevOps metrics surfaced in real time.
The other DORA metrics often closely associated with Change Failure Rate are:
- Deployment Frequency
- Mean Time to Recover
- Lead Time for Changes.
Key use cases
Change Failure Rate is a very useful DevOps metric to help teams reduce their overall Lead Time and increase the velocity of software delivery. Deployment failures are a key source of friction in the end-to-end delivery process and waste time and resource – hence the focus on reducing the Failure Rate.
Change Failure Rate is particularly important for organisations looking to increase delivery velocity (reduce Lead Time) -e.g.:
- delivery organisations at an early stage of Agile DevOps maturity
- large scale delivery capabilities with distributed teams (onshore, offshore, contractor, inhouse) and potentially a higher turnover of engineering talent
- teams involved in strategically critical software delivery projects.
Reducing Change Failure Rate will reduce overall Lead Time and increase software delivery velocity and quality.
Google and the DevOps Research & Assessment organization put out a study each year based on a very large survey of DevOps professionals worldwide. Teams are tiered depending on their relative performance.
Below is the breakdown for Change Failure Rate in the 2019 data.
|Tier||Change Failure Rate||% of Teams in Tier|
Plandek is a global leader in software delivery metrics and analytics, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).
Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics & analytics to optimise software delivery predictability, risk management and process improvement.
Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.