Becoming a Data-Driven Engineering Leader
Delivering software is hard. I know as well as anyone just how difficult it can be. I run Engineering at a company called Plandek. Before this, I led many software teams delivering software for companies from startups to FTSE100 companies. Like most leaders of engineering teams, I’ve struggled to balance my desire to give teams a great environment to work in, including a high degree of autonomy and responsibility, with being accountable to internal and external stakeholders.
The expectations from stakeholders are often beyond the capacity available – but it is hard for us to know by how much and reset expectations at a realistic level. Most engineers have also worked on projects where managing tech debt has been sacrificed in the name of delivering software faster – then experienced the frustration of attempting to explain the need to slow down new development to stabilise the codebase. We’ve experienced difficulty clarifying to outside engineering colleagues why a project hasn’t been completed on schedule and when it might be released.
For most managers, addressing these challenges results in either deep frustration with engineering from your stakeholders or a reversion to command and control style management – which invariably alienates your top-performing engineers. However, there is a better way: using a transparent outcome-based measurement system.
Let’s first address the elephant in the room: “Is it possible to design a perfect, objective measurement system?”. Clearly, the answer is no. However, Gilb’s Law states, “Anything you need to quantify can be measured in some way that is superior to not measuring it at all”. Your instincts and good judgement will have helped you to rise to a leadership role, and there is no replacement for these talents. However, a well-thought-out set of metrics will help you to direct your attention and create a culture of greater responsibility in your team.
So, how do you design a measurement system? Imagine that you are an Engineering Manager at Phoenix Bank, leading a team which is tasked with improving the stability and quality of a long-neglected legacy component in the bank’s architecture.
First, you begin to identify the metrics that best represent the ultimate outcome. These metrics will likely be clear from the demands on you from your business stakeholders. In your one-on-ones with your manager, you have often heard that “important functionality of this component is often broken” and “it takes a long time for bug fixes to go from report into production”. This focuses your mind on tracking the number of Unresolved Bugs for your component and also tracking the Lead Time from a bug being reported to the time it’s deployed into production.
But it is hard to lead your team towards accomplishing an outcome by only having that end-state measurement defined – it is like trying to sail across an ocean by asking yourself if you’re on the other side yet. To solve this problem, first, you need to know where in the sea you are. That is, you have to leverage measuring at the midpoint of your process and identifying metrics that can be tracked to indicate progress towards your ultimate goal. These are known as leading indicators.
To find these leading indicators, you gather your team to explain why and how you will measure progress. This allows you to define the business context, which measurements you have selected to show progress against your business goals and allows your team to identify the factors contributing to the business problem, which can then be measured and focused on for improvement. This transparency and openness also help you to prevent your team from feeling like they are being monitored and checked up on.
In this meeting, your team tells you that you have a problem with technical debt in your component and that this causes a lot of regressions, so you identify that cognitive complexity is your first leading indicator. You can then also prove the business benefits of resolving technical debt by tracking changes in the number of created bugs. These two metrics then form two of your leading indicators.
A team member suggests a clear relationship between the number of bugs being fixed and the number of unresolved bugs, so you add this to your set of leading indicators for the number of unresolved bugs. You agree to set a target on the number of bugs fixed weekly and review regularly to find ways to increase this number.
One of the team members then complains that there is a high degree of context switching due to new high-priority bugs from business stakeholders. This demand is expected due to the stability issues with your business-critical component, however, this causes long cycle times because work is frequently started and then left as the biggest focus moves to a different issue, so you add a measurement of Work In Progress to your leading indicators. You also agree to meet with your business stakeholders to use the new measures to show the impact of demands for context switches and try to reduce them in future.
Once you return to your desk, you begin looking at your Lead Time and Unresolved Bugs dashboards. You can see the impacts of technical debt and context switching causing a situation where more bugs are being created than fixed, so you budget time to resolve tech debt and change the process to introduce WIP limits. Over the next few weeks, you will begin seeing gradual improvement in the stability of your component. Your one-on-ones with your manager become more accessible as the business impact reduces, and it becomes clear how well you have improved the state of your component.
Your team’s morale improves as they leave firefighting mode and create a codebase they’re proud to work on. They begin proactively identifying areas of concern in the codebase, and you give the team the responsibility to identify and resolve these issues without your intervention.
This process will finish by giving you a first iteration of a simple measurement system your team feels invested in, which helps you focus on achieving positive business outcomes. It’s very unlikely that you will get all of the measurements right the first time, so it is essential to revisit the chosen metrics regularly to ensure that they are still driving the right changes and that the theories about leading indicators remain solid.
This has shown you how to use data to inform decisions for one use case and given you some principles you can apply to your own problems. Using data to inform leadership decisions and align around objectives in engineering teams is in its infancy, but I believe it poses several interesting questions. How are you ever sure if management decisions have made a positive impact without data? How do you align people around goals? What does a high-performing team look like? How are you able to understand what is holding your team back?
