Manage the Agile metrics that matter – reduce unplanned go-live delays by 50%
By Charlie Ponsonby, Co-CEO Plandek
We work with a great variety of organisations at various points in their Agile transformations. Whilst the move to Agile has driven tangible and lasting benefits in almost all of these organisations – the great majority have experienced problems and unintended consequences along the way.
One issue that we hear very often (particularly in large commercial organisations) is the difficulty of reconciling Agile’s decentralised, iterative approach with internal clients used to agreeing budgets (up-front) and expecting outcomes delivered at certain points in time.
Products over Projects
An important principle of Agile is to align stable teams around designing and building particular products – so that they have end-to-end responsibility for designing, building and maintaining the technology and become increasingly expert over time. This is inherently sensible and prevents temporary project-based teams from being thrown together to build something, only to be reassigned after “launch”.
Effective product-based teams aim to iterate and deploy improvements increasingly rapidly in keeping with the core Agile goal of the “early and continuous delivery of valuable software”.
As a result, forecasting becomes less important as the business expects small, frequent increments to an existing application over time. Rather like painting the Forth Bridge – the job that is famously never finished…
Projects over Products?
After the popular writing of Martin Fowler and others, product-based teams have become the desired option for many Agile organisations – but in certain situations, a product-based methodology can have major drawbacks.
A typical example would be the building of a new application that the business requires at a certain time and for an agreed budget (signed off upfront). This is a very common scenario in large organisations. Typical examples might include:
- regulatory changes forcing an upgrade or change to a legacy system by a certain date;
- the launch of a new app or digital platform required in time for a certain date or milestone (e.g. Christmas trading).
Under these circumstances a product-based Agile methodology can cause serious problems as the methodology:
- is designed for small, regular iterations – to deliver value increments early and often;
- does not, therefore, involve detailed project scoping upfront. Tasks may be T-shirt sized, but no detailed estimation of effort is likely to be undertaken upfront;
- as such forecasting cost and timing of delivery is highly problematic (and is not in keeping with the aims/purpose of the product-based methodology).
This inability to forecast cost and timing of delivery almost inevitably causes serious problems with stakeholders anxious to receive progress updates and working software at the planned point in time and at the planned budget.
So how can an Agile team better predict delivery timing and cost – even if the Agile methodology that they are adopting is not suited to accurate forecasting?
The concept of a Delivery Risk Profile
Forecasting techniques within Agile teams are often rudimentary. Burndown charts offer a linear extrapolation of current velocity, compared to the known backlog – to estimate the future completion date of effort outstanding. There are two basic problems with this:
- as Agile teams estimate ticket size/effort on an ongoing basis, there will inevitably be an un-estimated backlog that is not included in the forecast;
- extrapolating current velocity may be highly misleading (for any number of reasons affecting the team’s ability to deliver going forward).
However, the good news is that if we collect and track a set of delivery metrics over time, we can start to put together a more informed view of forecast delivery timing and cost, expressed within a Delivery Risk Profile.
Plandek is the world’s leading BI platform to surface and analyse end-to-end delivery metrics. It mines data from toolsets used by delivery teams (such as Jira, Git, and CI/CD tools), to provide end-to-end delivery metrics to optimise software delivery forecasting, risk management and delivery effectiveness.
Analysis of this balanced scorecard of metrics gives a more measured view of likely future delivery time and cost. This analysis is presented in a Delivery Risk Profile.
The Delivery Risk Profile is made up of metrics in five key areas:
- Backlog analysis – it’s key to understand the quality and age of the current backlog and to get a better feel of the likely size of the unestimated element of the backlog.
- Estimation and sprint accuracy – a critical determinant of timing accuracy. If teams are unable to estimate effectively and deliver their sprint goals, then longer-term delivery targets become far more unreliable.
- Process efficiency – often very poorly understood, but vital to understanding likely future velocity.
- Throughput and velocity – clearly key, but even more interesting when trends by individual teams are carefully considered.
- Talent availability and engagement – again often poorly understood but critical in maintaining quality and velocity – as delivery is ultimately all about the talent.
Key metrics within the Delivery Risk Profile
The graphic below shows key metrics that when considered together give a much more informed view of likely future delivery timing and hence cost – as they are heavily deterministic of future velocity.
The Plandek analytics platform collects these metrics across all teams and projects and enables Delivery Managers to get a much more complete picture of likely future velocity – expressed in a quantitative Delivery Risk Profile.
Typically trends in the metrics are analysed by the team to build the profile of risk. This Risk Profile then enables the Delivery Manager to refine:
- linear burndown chart forecasts; and
- verbal forecasts provided by the teams themselves.
A more accurate delivery forecast is therefore synthesised from three different sources – the teams themselves; linear burndown estimates; and the balanced scorecard of metrics known to directly impact future velocity.
Case study – applying a Delivery Risk Profile to reduce unplanned go-live delays by 50%
We work with a number of data companies, one of which has been particularly successful at applying the Plandek metrics set to build an effective risk profile, which they have used to very significantly improve their go-live forecasting.
Their teams are organised around a product-led strategy and therefore not ideally suited to delivering time-dependent “projects”. In this instance, stakeholders need careful management to ensure that there is as much visibility as possible as regards progress and forecast timing of delivery of major milestones.
Before the Plandek metrics set was tracked and analysed – unplanned go-live delays were common. Delivery Managers relied on linear burndown charts and word-of-mouth updates from scrum teams.
Plandek was implemented and delivery Team Leads and Delivery Managers started to track and manage a simple set of delivery metrics. Trends were analysed over time for each scrum team to build a Delivery Risk Profile (as described on page 3).
The immediate effect was a greater focus on:
- Delivering sprints more accurately (measured via Sprint Overall Completion (%) and Sprint Target completion (%)
- The forecasting process – with word-of-mouth forecasts from the teams more rigorously debated and refined in the light of trends in the key risk profile metrics.
The net result was a very significant improvement in forecasting accuracy. Over a 6 month period, unplanned go-live delays were reduced by 50%. This greatly strengthened the relationship between the technology delivery team and the business stakeholders.
The case is summarised in the table below.
A European data business – Understanding the delivery team’s risk profile to improve delivery forecasting accuracy
