How to measure Copilot's impact on productivity

Most software delivery teams are now using AI in some form to help engineers accelerate delivery, improve efficiency and increase engineering capacity.

But how do you know whether AI is actually improving engineering productivity?

This is a more complex question than simply measuring AI usage. High adoption of AI tools does not necessarily translate into better delivery outcomes. 

AI adoption does not guarantee AI impact. So how should you use GitHub Copilot in your organization, and how should we measure the impact of Copilot on software development productivity?

GitHub Copilot and AI-augmented engineering

GitHub Copilot is only one part of a rapidly evolving ecosystem of AI tools that support software delivery.

We’re now applying AI across multiple stages of the SDLC, including:

• Code generation and code review (e.g. Devin, Claude Code)

• Documentation (e.g DeepWiki)

• Testing (e.g. Momentic)

• Technical debt management (e.g. Grit)

• Architecture analysis (e.g. Icepanel)

As investment in AI tooling increases, we need evidence that these tools are generating measurable value rather than simply increasing activity.

The challenge is that traditional measures such as commits, pull requests or AI usage statistics tell only part of the story.

What we need to do is understand whether AI is improving delivery outcomes.

A methodology for measuring the impact of GitHub Copilot

To robustly measure the impact of GitHub Copilot and similar AI engineering tools, the methodology must be:

• Quantitative – based on measurable delivery data

• Holistic – considering the end-to-end software delivery lifecycle

• Balanced – combining delivery metrics with team feedback and survey data

• Outcome-focused – measuring delivery performance rather than AI activity alone

Some organizations are still working out how to adopt AI, while many have successfully rolled out AI tools and achieved strong adoption metrics, but have seen little improvement in software delivery performance.

To solve this problem, use the RACER Framework.

RACER is a five-part model designed to help engineering leaders systematically move from AI rollout to measurable results:

• Rollout – Deploy the tools and drive adoption across teams

• Approach – Match the right AI approach to the right task

• Constraints – Identify and address the bottlenecks limiting AI effectiveness

• Engineering Impact – Measure the operational impact on software delivery

• Results – Translate engineering improvements into business outcomes and ROI

Rather than viewing AI transformation as a tooling exercise, RACER treats it as an engineering systems challenge. The framework recognizes that successful adoption requires not only access to AI tools, but also changes to workflows, process improvement and continuous measurement.

Track metrics over time for a representative sample of users and compare them against non-users or baseline performance. Analyze at least three sprint cycles (6+ weeks) to account for normal delivery variation.

1. Rollout: Deploy and use the tools

The first step is ensuring teams have access to AI tools like GitHub Copilot and are using it regularly.

Successful rollout is characterized not simply by tool availability, but by frequent usage, meaningful engagement and growing confidence in AI-generated outputs.

Useful indicators include:

• Daily Active Users (DAU)

• Weekly Active Users (WAU)

• % active licenses

• Number of suggestions, completions and tokens used

• Suggestion Acceptance rate

• Tab Acceptance Rate

The goal is to understand whether Copilot has become part of the team's normal way of working.

2. Approach: Pick the right approach for the task

There is no single "best practice" approach to using AI tools.

Different tasks benefit from different levels of AI involvement, ranging from simple assistance through to highly autonomous agentic workflows.

We see four broad patterns:

• Ask – Ask an LLM questions about a task or topic

• Assist – Use like a pair programmer, typically in an IDE

• Agent (Supervized) – Use for broader tasks, within an IDE or command line environment

• Autonomous Agent – Use like a virtual team member; assign tasks via workflow or collaboration tools

The primary use case for Copilot is to assist with coding tasks, but we’re increasingly combining it with more agentic approaches.

The most effective teams consciously match the right approach to the right task rather than applying a single pattern everywhere.

Expert tip: Many metrics required to track the success of each type of pattern requires metrics that go beyond what is possible with built-in GenAI tool admin consoles. They require correlation of information with workflow management systems and other tools. Learn how Plandek helps you track these metrics.

3. Constraints: Systematically eliminate barriers to success

This is often the most important stage of the framework. RACER is heavily influenced by the Theory of Constraints: the idea that overall system performance is limited by the biggest bottleneck in the process.

While GitHub Copilot can significantly accelerate coding and testing activities, they often expose bottlenecks elsewhere in the software delivery lifecycle.

Teams frequently see increased coding velocity but little improvement in overall delivery performance because the constraint has simply moved.

Common constraints include:

• Slow code review cycles

• Unclear requirements

• Poor documentation

• Environment instability

• Deployment bottlenecks

• Governance and compliance requirements

• Skills and training gaps

• Organizational resistance to change

There is a repeatable four step process to address SDLC constraints:

1. Identify the constraints using metrics

2. Categorize the constraints – typically tooling, codebase & architecture, documentation, workflow/process, governance/compliance, training, culture

3. Track, prioritize, and address constraints

4. Reassess and repeat

Identifying these constraints is essential because they often determine whether AI adoption especially around tools like Copilot translates into measurable engineering impact.

Read our Complete Guide to Identifying Software Engineering Bottlenecks

4. Engineering Impact

Once we have assessed rollout, approach and constraints, we can begin measuring the operational impact of AI on software delivery performance.

Use a balanced set of software engineering metrics and DevOps metrics to ensure improvements in one area are not masking deterioration elsewhere.

5. Results

The final stage of RACER is connecting engineering improvements enabled by GitHub Copilot to measurable business outcomes. The ultimate objective is to generate business value.

This typically comes from revenue acceleration, cost reduction and cost avoidance.

Assess these benefits against the total cost of GitHub Copilot adoption, including:

• Licensing costs

• Usage costs

• Implementation costs

• Management costs

• Governance and compliance costs

The organizations achieving the greatest returns from AI are typically those that systematically progress through all five stages of RACER, continuously refining their approaches, addressing constraints and measuring impact over time.

How to measure the impact of GitHub Copilot – both good and bad

Done right, AI adoption should increase focus, speed, predictability and quality; ultimately leading to improved value delivery.

These are what we call the Four Pillars of Engineering Productivity. Developed from data from over 2000 teams, these pillars give a balanced view of engineering productivity and help avoid over-reliance on narrow metrics such as coding output or AI usage.

This is important because GitHub Copilot primarily affects coding activities, but engineering productivity is determined by the performance of the entire delivery system. Measuring only coding output or Copilot usage can give a misleading picture of impact.

Focus

If GitHub Copilot creates additional engineering capacity, we need to know how that capacity is being redeployed.

Key metrics include:

• Value Delivery %

• Support & Maintenance %

• Technical Debt %

Speed

GitHub Copilot is designed to accelerate software development, but the key question is whether those gains translate into faster delivery outcomes.

Key metrics include:

• Lead Time to Value

• Cycle Time

• Time to Merge PRs

• Throughput Quotient

• PR Efficiency Quotient

• Merge Frequency per Author

Predictability

Productivity improvements are most valuable when they improve the reliability of delivery.

Key metrics include:

• Sprint Capacity Accuracy

• Sprint Target Completion

• Mid-Sprint Scope Change %

• Velocity Volatility

Quality

Any improvement in speed must be balanced against quality. Increased coding velocity should not result in more defects, rework or technical debt.

Key metrics include:

• Bug Resolution Time

• Stories Delivered : Bugs Raised

• Bugs Resolved : Bugs Raised

Productivity improvements from GitHub Copilot: what does the evidence say?

GitHub’s own research shows Copilot can improve developer productivity. Developers using Copilot completed a coding task 55% faster, with 88% saying they felt more productive and 73% saying they were more in flow, while pull requests increase by 8.69%.

GitHub Copilot absolutely can create value.

But it is something that requires a specific effort to get right.

Use the RACER framework to move from rollout, to adoption, to impact.

Use the metrics within the Four Pillars to spot bottlenecks and adverse consequences, and assess the impact across Focus, Speed, Predictability and Quality.

Together, RACER and the Four Pillars help organisations move beyond Copilot usage metrics and understand whether AI-assisted development is improving software delivery outcomes.

Measure the real impact of GitHub Copilot with Plandek

GitHub Copilot can accelerate coding activity, but coding is only one part of the software delivery lifecycle. To understand whether Copilot is actually improving engineering productivity, you need visibility into what happens across the entire system.

Plandek helps engineering leaders track AI adoption, usage, constraints, engineering impact, and business outcomes using the RACER framework. 

By combining quantitative data from your DevOps toolchain with qualitative insight from engineering teams, Plandek shows where AI is improving delivery, where new bottlenecks are emerging, and what needs attention to maximize results.

With Plandek, you can:

• Track adoption and usage of AI tools such as GitHub Copilot, Cursor, Claude Code, and Devin

• Measure AI impact across Focus, Speed, Predictability, and Quality using the Four Pillars of Engineering Productivity

• Identify bottlenecks exposed by AI-accelerated development across planning, coding, review, testing, and release

• Understand whether increased coding throughput is translating into faster delivery outcomes

• Surface risks, blockers, and improvement opportunities with Dekka, Plandek's AI Delivery Assistant

• Share clear, understandable KPIs with engineering leaders and business stakeholders

AI usage metrics tell you whether teams are using Copilot. Plandek helps you understand whether it's actually improving software delivery outcomes and where to focus next to maximize return on your AI investment.

See how Plandek works and book a free demo

Key Takeaways

• GitHub Copilot usage is not the same as productivity impact. Adoption metrics show whether teams are using the tool, not whether delivery outcomes are improving.

• Measure Copilot across the full SDLC. Coding may get faster, but review, testing, deployment or planning bottlenecks can still limit overall delivery performance.

• Use RACER to move from rollout to ROI. Track rollout, approach, constraints, engineering impact and business results to assess whether Copilot is creating measurable value.

• Balance speed with quality and predictability. Faster coding is only valuable if it does not increase defects, rework, delivery volatility or technical debt.

• Track Focus, Speed, Predictability and Quality. The Four Pillars help teams measure AI-assisted development without over-relying on narrow metrics like commits or pull requests.

• Copilot can create value, but it needs active management. The best results come when teams monitor impact, remove constraints and continuously improve how AI is used.

FAQs

How do you measure GitHub Copilot productivity?

Measure GitHub Copilot productivity using both adoption metrics and software delivery metrics. Track usage data such as active users and suggestion acceptance rates alongside delivery metrics like Lead Time to Value, Cycle Time, PR review time, defect rates, and delivery predictability.

Does GitHub Copilot improve developer productivity?

GitHub Copilot can improve developer productivity by accelerating coding tasks and reducing time spent on routine work. However, faster coding does not automatically translate into better software delivery outcomes, so impact should be measured across speed, quality, predictability, and business value.

What are the best metrics for measuring GitHub Copilot ROI?

The best metrics combine AI adoption data with engineering productivity and business outcomes. Common measures include DORA metrics, Lead Time to Value, Cycle Time, defect rates, Sprint Capacity Accuracy, Value Delivery %, and the cost savings or additional delivery capacity created by AI.

Can GitHub Copilot increase coding speed without improving delivery performance?

Yes. GitHub Copilot can increase coding throughput while bottlenecks in code review, testing, deployment, or governance prevent improvements in overall delivery performance. This is why engineering leaders should measure the entire SDLC rather than coding activity alone.

Are DORA metrics enough to measure GitHub Copilot impact?

DORA metrics provide useful insight into software delivery performance, but they do not explain why performance is changing. Combine DORA metrics with the Four Pillars of Engineer Productivity: Focus, Predictability, Quality and Speed to understand the full impact of GitHub Copilot.