How to reduce software delivery Cycle Time by over 25% in an OKR framework (case study)

Case study: Implementing Plandek at a $40bn data and publishing company

The client context: This high profile, multi-national data and publishing business use Plandek as a key element of their Value Stream Management across their global software delivery teams with over 2,000 engineers in multiple locations. Plandek provides the metrics and reporting to underpin their OKR (Objectives and Key Results) process within the global delivery organisation. As an example, Cycle Time was identified using Plandek as a key opportunity area for improvement and a specific OKR was created to reduce Cycle Time by 25% in 6 months during 2020.

Cycle Time was deemed critical due to the increasing pressure to deliver new features rapidly in a sector where speed-to-market is a critical differentiator.

Using a variety of delivery and engineering metrics available within the Plandek platform, including ‘Mean Time to resolve Pull Requests’ and ‘Flow Efficiency’, the teams drove a number of process improvement initiatives and saw a month on month reductions in Cycle Time resulting in an average 25% reduction in Cycle Time between January and June 2020.

5 Key Takeaways

1. A continuous improvement initiative underpinned by a simple set of ‘North Star’ metrics that teams understand, and trust can deliver rapid, sustainable and significant improvement in software delivery outcomes at scale

2. In large scale delivery environments, OKR provides an effective framework to prioritise a set of simple targets for improvement (such as a 25% improvement in Cycle Time). And Plandek is an ideal BI tool to provide the necessary end-to-end software delivery metrics to underpin the collective effort to deliver the OKR targets set

3. Using Plandek, four metrics were found to directly impact Cycle Time across multiple teams: Flow Efficiency (which looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status); Mean Time to Resolve Pull Requests (hrs); First Time Pass Rate (%); and Story Points Ready for Development

4. Plandek was embedded in teams’ management processes (e.g. standups, sprint retros) to track and manage to these four determinant metrics, with the result that average Cycle Time across workstreams was reduced by over 25% over a 6 month period.

5. This was only possible as teams trust the quality of the metrics/analytics as Plandek enables them to see the ‘provenance’ of the metric (how it is calculated) and to configure metrics to match their precise team circumstances (via Filtering functionality)

Introduction – the client context

The client operates a large Agile software delivery capability with over 2,000 engineers distributed globally across multiple workstreams.

The company is metrics-led and adopted Plandek as the global solution to surface end-to-end software delivery metrics across all teams, in order to greatly improve visibility across teams and thereby:

• reduce software delivery risk (improve delivery dependability)
• improve software delivery productivity and quality
• demonstrate the success of their Agile transformation with a balanced scorecard of improving metrics over time.

Plandek was first adopted in 2018 and the Plandek Customer Success team have since worked closely with the client to help them create and embed customised dashboards for all teams (squads), Delivery Managers and technology leadership.

Teams have adopted a simple set of metrics around which to drive their continuous improvement effort – and Plandek has been embedded in the daily and weekly work practices (e.g. stand-ups and sprint retros).

Getting started with Plandek – using metrics to understand the health of your delivery capability

Plandek works by mining data from toolsets used by delivery teams (such as Jira, Git, CI/CD tools and Slack), to provide end-to-end delivery metrics/analytics to optimise software delivery dependability, risk management and process improvement.

Mining data from multiple toolsets used across the SDLC creates a unique perspective, enabling Plandek to identify bottlenecks and opportunities for improvement throughout the design, development, integration, test and deployment processes.

Creating a hierarchy of simple metrics that everyone understands

Plandek can surface a myriad of metrics. The Plandek Customer Success team worked closely with the client to identify a simple set of ‘North Star’ metrics, (selected from this broader potential metrics set), around which to set their delivery goals.

The ‘North Star’ metrics were carefully selected to be meaningful when aggregated and illustrative of effective Agile software delivery:

These North Star metrics were adopted by the technology leadership team as key priorities within an OKR (Objectives and Key Results) framework.

Setting Cycle Time as an OKR target

As Time to Value was identified as a key priority (and opportunity for improvement), an OKR target was agreed to reduce Cycle Time by 25% over 6 months in H1 2020.

The Plandek network of dashboards allowed each team to closely analyse their own Cycle Time and understand where in the Cycle there was an opportunity to drive down time to value.

As per figure 1 below, the Plandek Cycle Time metric view allowed teams to understand time spent in each ticket status within the development cycle. The flexible analytics capability and powerful filtering allows analysis by Status, Issue Type, Epic (and any other standard or custom ticket field) all plotted over any time range required.

Figure 1. Example Plandek Cycle Time metric view

Tracking and improving key metrics that drive Cycle Time, to deliver the OKR

Reducing Cycle Time by 25% is an aggressive target, which if delivered effectively, drives very significant business benefits as the software is delivered more rapidly without additional delivery resource allocation (or impact on quality).

Working with the Plandek Customer Success team, Plandek was used by scrum teams to identify key determinant metrics that would have the biggest impact on reducing Cycle Time without impacting quality or requiring additional resource allocation.

Analysis showed four metrics which could unlock significant shortening of Cycle Times across almost all scrum teams. These were:

1. Flow Efficiency (which looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status)

2. Mean Time to Resolve Pull Requests (hrs)

3. First Time Pass Rate (%).

4. Story Points Ready for Development.

Figure 2. Example Flow Efficiency metric within Plandek dashboard

Typical opportunities to remove inactive bottlenecks included time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA. Where waits for QA were considered excessive, Delivery Managers reconsidered QA resource allocation by team.

Mean Time to Resolve Pull Requests (MTRPR) was also found to be a key bottleneck and hence potential area to save time and reduce overall Cycle Time. Very significant variations in time to resolve PRs were seen between teams and individuals, with waits of over 100 hours not uncommon.

Plandek enables drill-down to understand variances by code repository and destination branch (see Figure 3 below). This enabled quick identification of the biggest bottlenecks and targeted intervention, with the result that MTRPR was reduced dramatically (by <80% in some squads) and by an average of c50%. This has a very significant impact on overall Cycle Time.

Figure 3. Example Mean Time to Resolve Pull Request metric within Plandek dashboard

First Time Pass Rate (FTPR) was another key metric in driving the 25% Cycle Time improvement achieved over the six month period. It proved to be a popular metric at team level as high FTPR not only increases velocity (and reduces QA burden), but is symptomatic of a productive relationship between BAs, engineers and QA – with well managed backlogs, well defined tickets/requirements and hence a smoother flow of tickets through the development process.

Drill-down within the Plandek “Explore” functionality shows variations in FTPR by Board, ticket and individual within the team.

Figure 4. First Time Pass Rate example metric within the Plandek dashboard

Effective analysis of teams’ backlog proved to be a fertile area for identifying bottlenecks that reduced velocity and adversely affected Cycle Time.

Teams with well managed backlogs (i.e. with at least 2 sprints worth of tickets prepared and ready to progress), significantly reduced their Cycle Times. As such, the simple metric of Story Points Ready for Dev was a key metric in increasing velocity across the majority of teams. (See Figure 5).

The powerful Filter functionality within Plandek enables teams to identify and track relevant ticket types to ensure accurate analysis.

Metrics led Continuous Improvement in software delivery – buy-in and trust

The experience at the client showed the power of applying a metrics-led philosophy across a scaled Agile software delivery capability. Cycle Time was reduced by just over 25% over a 6 month period in H1 2020, thereby meeting the OKR set by the technology leadership team.

Key factors in the success of the approach included:

1. The identification and communication of a simple delivery goal in keeping with the underlying Agile delivery approach (a reduction in Cycle Time)

2. The use of Plandek to surface that metric in real time at all levels within the delivery hierarchy (across Board, team, workstream, PI, tribe etc)

3. Collective buy-in and trust in the metrics from Team Leads upwards. This was critical and was made possible as a result of the total transparency of Plandek metric presentation.

Experience shows that if Team Leads cannot see exactly how metrics are calculated and that they reflect their team’s context – they will question and ultimately reject the metrics – especially if the metric appears erratic or heavily negative.

Plandek is unique in its ability to show the ‘provenance’ of each metric and to allow individual teams to configure each metric in the way that reflects their particular circumstances, using the powerful Filter functionality. This is ultimately critical in the overall success of the initiative.

Figure 5. Story Points ready for Dev – example metric within the Plandek dashboard

For enquiries please contact Darina Lysenko: Dlysenko@plandek.com