Case studies
14. Case Study 1: Reducing Cycle Time by 25% at a global data business
The client context: This high-profile, multi-national data and publishing business uses Plandek as a key element of their Value Stream Management across their global software delivery teams with over 2,000 engineers in multiple locations. Plandek provides the metrics and reporting to underpin their OKR (Objectives and Key Results) process within the global delivery organisation. As an example, Cycle Time was identified using Plandek as a key opportunity area for improvement, and a specific OKR was created to reduce Cycle Time by 25% in 6 months during 2020.
Using a variety of delivery and engineering metrics available within the Plandek platform, including ‘Mean Time to resolve Pull Requests’ and ‘Flow Efficiency’, the teams drove a number of process improvement initiatives and saw month-on-month reductions in Cycle Time, resulting in an average 25% reduction in Cycle Time between January and June 2020.
5 Key Takeaways
- A continuous improvement initiative underpinned by a simple set of ‘North Star’ metrics that teams understand and trust can deliver rapid, sustainable and significant improvement in software delivery outcomes at scale.
- In large-scale delivery environments, OKR provides an effective framework to prioritise a set of simple targets for improvement (such as a 25% improvement in Cycle Time). Plandek is an ideal BI tool to provide the necessary end-to-end software delivery metrics to underpin the collective effort to deliver the OKR targets set.
- Using Plandek, four metrics were found to impact Cycle Time across multiple teams directly: Flow Efficiency (which looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status), Mean Time to Resolve Pull Requests (hrs); First Time Pass Rate (%); and Story Points Ready for Development.
- Plandek was embedded in teams’ management processes (e.g. stand-ups, sprint retros) to track and manage these four-determinant metrics, with the result that average Cycle Time across workstreams was reduced by over 25% over a six month period.
- This was only possible as teams trusted the quality of the metrics/analytics as Plandek enabled them to see the ‘provenance’ of the metric (how it is calculated) and to configure metrics to match their precise team circumstances (via Filtering functionality).
Creating a hierarchy of simple metrics that everyone understands
Plandek can surface a myriad of metrics. The Plandek Customer Success team worked closely with the client to identify a simple set of ‘North Star’ metrics (selected from this broader potential metrics set) around which to set their delivery goals.
The ‘North Star’ metrics were carefully selected to be meaningful when aggregated and illustrative of effective Agile software delivery:
These North Star metrics were adopted by the technology leadership team as key priorities within an OKR (Objectives and Key Results) framework.
Setting Cycle Time as an OKR target
As Time to Value was identified as a key priority (and opportunity for improvement), an OKR target was agreed to reduce Cycle Time by 25% over six months in H1 2020.
The Plandek network of dashboards allowed each team to closely analyse their own Cycle Time and understand where in the Cycle there was an opportunity to drive down time to value.
As per Figure 36 below, the Plandek Cycle Time metric view allowed teams to understand the time spent in each ticket status within the development cycle. The flexible analytics capability and powerful filtering allow analysis by Status, Issue Type, and Epic (and any other standard or custom ticket field), all plotted over any time range required.
Tracking and improving key metrics that drive Cycle Time to deliver the OKR
Reducing Cycle Time by 25% is an aggressive target, which, if delivered effectively, drives very significant business benefits as software is delivered more rapidly without additional delivery resource allocation (or impact on quality).
Working with the Plandek Customer Success team, Plandek was used by scrum teams to identify key determinant metrics that would have the biggest impact on reducing Cycle Time without impacting quality or requiring additional resource allocation.
Analysis showed four metrics that could unlock significant shortening of Cycle Times across almost all scrum teams. These were:
- Flow Efficiency (which looks at the proportion of time tickets spend in an ‘active’ versus ‘inactive’ status)
- Mean Time to Resolve Pull Requests (hrs)
- First-Time Pass Rate (%)
- Story Points Ready for Development
Each scrum team and related Scrum Masters and Delivery Managers updated their Plandek dashboards to surface these critical metrics so that they could be tracked and analysed in daily stand-ups, sprint retrospectives and management review meetings.
The Flow Efficiency analysis enables Team Leads to isolate and analyse each ‘inactive’ status in the workflow and consider if there is scope to reduce or eliminate it. The analysis shows the relative size of each ‘inactive’ status opportunity in terms of time spent in the inactive state and the volume of tickets affected.
Typical opportunities to remove inactive bottlenecks included time spent with tickets awaiting definition (e.g. Sizing) and tickets awaiting QA. Where waits for QA were considered excessive, Delivery Managers reconsidered QA resource allocation by the team.
Mean Time to Resolve Pull Requests (MTRPR) was also found to be a key bottleneck and, hence, a potential area to save time and reduce overall Cycle Time. Very significant variations in time to resolve PRs were seen between teams and individuals, with waits of over 100 hours not uncommon.
Plandek enables drill-down to understand variances by code repository and destination branch (see Figure 37 below). This enabled quick identification of the biggest bottlenecks and targeted intervention, with the result that MTRPR was reduced dramatically (by <80% in some squads) and by an average of 50%. This has a very significant impact on overall Cycle Time.
First Time Pass Rate (FTPR) was another key metric in driving the 25% Cycle Time improvement achieved over the six-month period. It proved to be a popular metric at the team level as high FTPR not only increases velocity (and reduces QA burden) but is symptomatic of a productive relationship between BAs, engineers and QA – with well-managed backlogs, well-defined tickets/requirements and hence a smoother flow of tickets through the development process.
Drill-down within the Plandek “Explore” functionality shows variations in FTPR by Board, ticket and individual within the team.
Effective analysis of teams’ backlog proved to be a fertile area for identifying bottlenecks that reduced velocity and adversely affected Cycle Time.
Teams with well-managed backlogs (i.e. with at least two sprints worth of tickets prepared and ready to progress) significantly reduced their Cycle Times. As such, the simple metric of Story Points Ready for Dev was a key metric in increasing velocity across the majority of teams. The powerful Filter functionality within Plandek enables teams to identify and track relevant ticket types to ensure accurate analysis.
Metrics led Continuous Improvement in software delivery – buy-in and trust
The experience with the client showed the power of applying a metrics-led philosophy across a scaled Agile software delivery capability. Cycle Time was reduced by just over 25% over a 6-month period in H1 2020, thereby meeting the OKR set by the technology leadership team.
Key factors in the success of the approach included:
- The identification and communication of a simple delivery goal in keeping with the underlying Agile delivery approach (a reduction in Cycle Time);
- The use of Plandek to surface that metric in real-time at all levels within the delivery hierarchy (across Board, team, workstream, PI, tribe, etc.);
- Collective buy-in and trust in the metrics from the Team Lead upwards. This was critical and was made possible as a result of the total transparency of the Plandek metric presentation.
Experience shows that if Team Leads cannot see exactly how metrics are calculated and that they reflect their team’s context – they will question and ultimately reject the metrics – especially if the metric appears erratic or heavily negative.
Plandek is unique in its ability to show the ‘provenance’ of each metric and to allow individual teams to configure each metric in a way that reflects their circumstances, using the powerful Filter functionality. This is ultimately critical to the overall success of the initiative.
