Plandek Metric of the week: First Time Pass Rate
First Time Pass Rate (%) is an excellent measure of overall team health. As the name suggests, it measures the percentage of tickets that pass QA first time (without stimulating a return transition or defect sub-task). Too often this metric is seen as an engineering quality metric, when indeed it is a better reflection of how well a team is working together and supporting one another.
A pass first-time requires the inter-dependent elements of an agile development team to be working well:
- strong collaboration and communication between team members throughout the delivery process
- well-defined tickets that are clearly understood by developers and QA testers
- reduced WIP levels, allowing engineers to focus on a single ticket at one time
- a well-supported QA process that’s engaged throughout the process and clear on the functionality that needs to be tested
Only when all these things are in place, will First Time Pass Rate be optimised. As such, it is a good overall Agile software delivery KPI.
First Time Pass Rate is often considered in tandem with a similar quality metric, Return Rate (%). The exact definition of both these software delivery metrics are shown below:
As shown in the example First Time Pass Rate summary chart below, First Time Pass Rate can easily be tracked over time as part of an ongoing continuous improvement initiative.
It is a powerful software delivery metric as it is meaningful when aggregated (e.g. at tribe, programme, release train level). It is also critical at delivery team level and should become a core agile metric for review in team retrospectives and other team effectiveness and planning meetings.
A software delivery and engineering metrics platform like Plandek enables you to drill-down into a metric like First Time Pass Rate to see trends over time, the tickets themselves that regressed and variations in First Time Pass Rate at key stages in the delivery process (e.g. QA).
First Time Pass Rate when considered at individual engineer level should never be used as a performance review metric – rather it is a critical team metric as it is influenced by so many factors beyond the individual engineer’s control.
First Time Pass Rate is often considered with the similar and related quality metric Return Rate. Return Rate (%) tracks the overall percentage of tickets returned from QA. This may include tickets that have been returned multiple times.
As discussed above, First Time Pass Rate is most illuminating when viewed in tandem with other delivery metrics relating to the effectiveness of the team – as it is so affected by interdependencies within the team. Other relevant team metrics include:
- Measures of backlog health such as Story Points Ready For Dev
- WIP (the number of work items started but not completed at any one time)
- Sprint completion % – a measure of the overall dependability of the team and its ability to hit its sprint goals
Key use cases
First Time Pass Rate is typically used as an agile delivery quality metric, particularly at the team level. As described, it is an excellent measure of overall team and process health and acts like the ‘canary in the coalmine’ in that it is a good leading indicator of underlying issues in team and process health.
As such it is often used within a broader balanced scorecard of agile delivery and engineering metrics to track Agile DevOps maturity.
By improving First Time Pass Rate you will see an immediate improvement in throughput (reduced Cycle Time) and a likely indirect positive impact on sprint completion and team morale.
A First Time Pass Rate of over 90% is achievable by mature Agile DevOps practitioners and high-performing teams.
Conversely, teams with low average First Time Pass Rate will struggle to deliver effectively – with longer Cycle Times, larger QA resource requirements and frustrated engineers.
Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery predictability, risk management and process improvement.
Plandek is a global leader in this fast-growing field, recognised by Gartner as a top nine global vendor in their DevOps Value Stream Management Market Guide (published in Sept 2020).
Plandek is based in London and works with clients globally to apply predictive data analytics and machine learning to deliver software more effectively.