Metrics
9. Metrics for different levels of Agile DevOps Maturity
Advanced or ‘Mature’ Agile practitioners tend to be many months into their Agile transformation, with well-established Agile methodologies (often at scale); an effective set of DevOps tooling; Delivery and Operations teams aligned to deliver in an Agile way; and business stakeholders who understand Agile principles. As a result, these ‘mature’ Agile businesses are highly proficient at delivering quality software dependably, early and often.
At the other end of the spectrum are those organisations very early in their Agile transformation – who are just starting to implement an Agile way of working across their delivery teams and are at the early stages of implementing the DevOps toolsets to underpin those processes. Much of their software delivery may still be ‘waterfall’ in nature and they may not have yet achieved an effective CI/CD (continuous integration/continuous deployment) methodology.
It stands to reason that effective measurement of the end-to-end Agile delivery process is critical in order to track and drive improvement over time – and ultimately to deliver the core agile goal of “the early and continuous delivery of valuable software” (Source: Agile Manifesto). As such, there are many agile, delivery and engineering metrics that can be used. But how do you choose sensible metrics for your organisation’s level of Agile maturity?
This section considers our recommended metrics for mature, intermediate and early-stage Agile delivery practitioners. It considers these metrics in a hierarchical way – with suitable metrics for technology leadership, delivery & engineering management, and the teams themselves.
Using metrics to understand the health of your delivery capability and your DevOps maturity
Agile software delivery is a complex process that very often hides very significant inefficiencies and bottlenecks.
Fortunately, the process is easily measurable as there is a rich digital footprint in the toolsets used across the process – from pre-development; development; integration & deployment; and out into live software management.
However, surfacing data from these myriad data sources (toolsets) and synthesising meaningful metrics that compare ‘apples with apples’ across complex Agile delivery environments is very tricky.
Hence until recently, software delivery metrics have been much discussed but little used.
This has changed very significantly with the arrival of BI solutions like Plandek that enable the surfacing of accurate end-to-end software delivery metrics for the first time. As a result, the field is moving centre stage, with Gartner and Forrester (to name but two examples) starting to advocate the importance of metrics and analytics in effective DevOps Value Stream Management (See Gartner 2020-3 Market Guides ‘DevOps Value Stream Management Platforms’).
Indeed, the end-to-end view provided by Plandek enables clients to very closely track the effectiveness and maturity of their Agile DevOps transformation.
Metrics for early-stage Agile DevOps practitioners
Plandek can surface a myriad of metrics – with dashboards customisable by users so that Team Leads, Managers and Leadership have their own dashboards reflecting their individual responsibilities and goals.
Here we select a few simple metrics that, in our experience, are ideal for organisations at the early stage of an Agile DevOps transformation. The metrics focus on simple measures that underpin the fundamentals of increased agility and are relatively easy to understand and act upon.
Cycle Time is an ideal delivery metric for early-stage practitioners. It simply measures the time taken to develop an increment of software. Unlike the more comprehensive measure of Lead Time (which measures the length of the entire end-to-end delivery process), Cycle Time is easier to measure as it looks only at the time taken (within a scrum team) to take a ticket from the backlog, code and test that ticket – in preparation for integration and deployment to live.
As per Figure 16, the Cycle Time metric view allows teams to understand the time spent on each ticket status within the development cycle. Plandek has flexible analytics capability and powerful filtering to allow analysis by Status, Issue Type, and Epic (and any other standard or custom ticket field) all plotted over any time range required.
Deployment Frequency is another fundamental measure of an organisation’s agility (when viewed alongside the other critical metrics described here). A core objective of Agile delivery is the ability to develop and deploy to live small software increments rapidly. Deployment Frequency tracks the basis of competence and is a powerful metric around which to focus effort at all levels in the delivery organisation at the early stages of an Agile transformation.
Delivered Story Points is often considered a problematic metric due to the potential inconsistencies in the calculation of story points and how much effort they represent. However, as a basic measure of output and how that is changing over time, it is a powerful metric around which to align.
There may be concerns about teams ‘gaming’ the metric with story point inflation, but as with all metrics, they should be viewed in context by experienced folks who know the teams well. And if this is the case, they can still give an excellent view of how the delivery organisation is progressing over time.
And finally, Escaped Defects is a simple but effective measure of overall software delivery quality. It can be tracked in a number of ways, but most involve tracking defects by criticality/priority as per the example below.
When these four simple Agile delivery metrics are viewed together, the early-stage Agile DevOps practitioner can get a good balanced view of how their Agile DevOps maturity is progressing.
The metrics can be tracked over time, making sure that an improvement in one metric (e.g. Cycle Time) does not lead to a detrimental effect on another metric (e.g. Escaped Defects).
In addition, the relationship between Cycle Time and Deployment Frequency can be closely watched. Very often, teams are able to reduce their Cycle Time, but this does not translate into quicker value delivery due to bottlenecks in the integration and deployment process.
Metrics for intermediate Agile DevOps practitioners
As organisations progress in their Agile transformation, they may start to consider a more complete set of Agile delivery and DevOps metrics.
We would suggest that you start to implement a hierarchy of cascading metrics. These can be separated into:
- ‘North Star’ leadership metrics – to be adopted by the leadership team to set the overall direction for the delivery organisation.
- Team and competence metrics – to be adopted by key players within the delivery organisation, such as Team Leads, Delivery Managers, Product Managers, Engineering Managers and DevOps Managers. These are relevant to the areas in question and help drive improvement in the aggregate ‘North Star’ metrics adopted by the technology leadership team.
We would suggest that the early-stage metrics (discussed above) provide a sensible starting point of ‘North Star’ metrics for the ‘intermediate’ Agile DevOps practitioner as they underpin the fundamentals of increased agility and are relatively easy to understand and act upon.
The Team and Competence metrics have been selected as powerful determinant metrics that directly drive the ‘North Star’ leadership metrics that track your core progress towards Agile DevOps maturity.
We have selected three metrics that, in our experience, most directly increase velocity (and hence reduce Lead and Cycle Time). These are:
- Flow Efficiency (which looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status)
- Mean Time to Resolve Pull Requests (hrs)
- First-Time Pass Rate (%)
Typically, these metrics are adopted by each scrum team and related Scrum Masters and Delivery Managers so that they are tracked and analysed in daily stand-ups, sprint retrospectives and management review meetings.
The Flow Efficiency analysis (see Figure 21 below) enables Team Leads to isolate and analyse each ‘inactive’ status in the workflow and consider if there is scope to reduce or eliminate it. The analysis shows the relative size of each ‘inactive’ status opportunity in terms of time spent in the inactive state and the volume of tickets affected.
Typical opportunities to remove inactive bottlenecks included time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA. Where waits for QA are considered excessive, Delivery Managers can reconsider QA resource allocation by the team.
Mean Time to Resolve Pull Requests (MTRPR) is also often found to be a key bottleneck and, hence, a potential area to save time and reduce overall Cycle Time. Very significant variations in time to resolve PRs are seen between teams and individuals, with waits of over 50 hours not uncommon.
Plandek enables drill-down to understand variances by code repository and destination branch (see Figure 22). This enables quick identification of the biggest bottlenecks and targeted intervention, with the result that MTRPR can be reduced dramatically. This has a very significant impact on overall Cycle Time.
In keeping with the ‘North Star’ metric of increasing Deployment Frequency, DevOps practitioners can track a range of metrics including Number of Builds, Build Failure Rate and Deployment Lead Time. All three are simple metrics which directly impact Deployment Frequency.
Our experience shows that typically, you can expect to increase deployments per day (per pipeline) by 15% through a better understanding of the root cause of Build Failures and Deployment Cycle Time using Plandek.
Delivery Team Leads and Managers can adopt a range of determinant metrics that help track and drive throughput and the delivery of value.
These included Stories Delivered by Epic, Lead Time for Stories and Epic, and Delivered Value Points.
And finally, quality should also be a consistent focus – both the security and quality of the delivery process itself and the quality of the software delivered.
We have selected a few simple Team and Competence metrics that directly impact defect rate – to reduce time spent fixing P1 (high priority) bugs, to improve the customer experience and reduce time diverted from feature development.
Plandek’s customisable dashboards enabled each team to focus on their own P1 resolution time and to manage better the backlog of Unresolved P1 and P2 bugs and time to resolve key hotfixes.
We recommend that organisations at an intermediate level of Agile maturity start to adopt a broad view of ‘quality’ to include both the software output and the quality and security of the delivery process itself. As such, teams may also track and manage Commits without a Pull Request and Commits without a Ticket Reference.
The former ensures that all code is peer-reviewed before being committed (an important security requirement) – and the latter ensures the clear linkage between committed code and Jira tickets for security compliance.
Metrics for advanced Agile DevOps practitioners
As organisations reach an advanced level of Agile capability, they will no doubt have their own favoured metrics and analytics. With that in mind, we have selected our favourite metrics for mature agile delivery organisations.
Again, we suggest that you maintain a hierarchy of cascading metrics, separated into ‘North Star’ leadership metrics to set the overall direction for the delivery organisation and Team and Competence metrics – to be adopted by key players within the delivery organisation such as Team Leads, Delivery Managers, Product Managers, Engineering Managers and DevOps Managers.
As an example, mature agile delivery organisations are often very focused on refining the CI/CD process to increase deployment frequency and, hence, regular delivery of value to the organisation. As such, additional metrics such as Failed Build Recovery Time and Mean Time for Failed Builds become popular as teams try to track and reduce the impact of build failures. Flakiest Files is another specialist DevOps metric developed by Plandek that enables DevOps Managers to identify fragile source code files in their codebase, which can be targeted for refactoring in order to reduce failed builds.
