From Activity to Impact: Why AI Adoption Metrics Don’t Tell the Whole Story

Rethinking How we Measure AI in Engineering

The stopwatch told a different story 

In July 2025, METR ran a randomized controlled trial with sixteen experienced open-source developers. Each was timed completing real tasks on their own repositories — half with AI assistance, half without. The developers using AI took nineteen percent longer than those who weren’t. 

The detail that made the study famous was the next one. When the same developers were asked how AI had affected their speed, they reported being twenty percent faster. 

A forty-percentage-point gap between perception and stopwatch. With experienced engineers. On their own code. Under controlled conditions.

The same dataset can tell very different stories depending on whether measurement focuses on activity or outcomes.

This is the question this third article in our series is about: when the dashboard says one thing and the system is doing another, which one does an organization actually believe? In nearly every AI rollout I see, the answer is the dashboard. And the dashboards are mostly measuring the wrong thing. 

The trap is structural, not accidental 

Article 01 in this series, The Bottleneck Just Moved: Why Faster Coding Doesn't Mean Faster Delivery, ended with a single-line directive in the six-step sequence: measure outcomes, not activity. This article is what that sentence actually means in practice, and why it is so much harder to do than to say. 

Adoption metrics are attractive for three reasons. They are immediately available — most AI coding tool vendors surface seat utilization, usage hours, and acceptance rates from day one. They respond visibly to investment — you buy licenses, you run onboarding sessions, the numbers move. And they are easy to socialize upward — a chart showing ninety-four percent seat activation looks like a story of successful deployment. 

None of this makes them wrong. Seat activation is a prerequisite for any value at all. Usage hours confirm engineers are engaging. Acceptance rates give a crude signal about whether the suggestions are useful. The problem is not that organizations track these metrics — it is that they stop there. 

Adoption metrics answer the question: did we deploy the tool? They do not answer: did deploying the tool change our delivery outcomes? These are different questions. They require different data. And conflating them is how organizations end up three months into a rollout believing they are succeeding while the delivery system is quietly degrading downstream. 

One dataset, two stories

What DORA found when it looked for signal 

In September 2025, DORA published its AI Capabilities Model — the product of more than one hundred hours of qualitative research and survey responses from nearly five thousand technology professionals. The research set out to answer a specific question: what conditions enable AI adoption to produce better organizational outcomes? 

DORA identified seven foundational capabilities that amplify AI’s positive impact on performance. Not one of them is an adoption metric. The seven are: a clear and communicated AI stance, a healthy data ecosystem, AI-accessible internal data, strong version control practices, working in small batches, a user-centric focus, and quality internal platforms. These are capability conditions — the structural prerequisites for AI to produce outcomes. They are, notably, exactly the kind of things that seat utilization dashboards cannot measure.

DORA AI Capabilities Model, 2025 

“AI’s primary role in software development is to amplify. It magnifies the strengths of high-performing organizations — and the dysfunctions of struggling ones.” 

Nearly 5,000 technology professionals surveyed. Seven foundational AI capabilities identified. Zero of those seven are adoption metrics. 

The amplifier finding is the one that should give engineering leaders pause. AI does not improve organizations independently of their existing capability level. It amplifies whatever conditions it is deployed into. Adoption metrics cannot distinguish between an organization amplifying strength and one amplifying dysfunction. That determination requires outcome data. 

Three tiers of signal 

The measurement gap is not binary — it is layered. Organizations typically progress through three distinct tiers of measurement maturity, and most never leave the first. 

Tier 1 is where most rollout dashboards live. Tier 3 is where delivery decisions need to be made. The gap between them is where unrecognized dysfunction accumulates.

Three Tiers of AI Measurement: Measurement maturity progresses from easy-to-collect adoption metrics to outcome metrics that provide the strongest signal of business impact.

Tier 1 — Activity metrics are vendor-provided and immediately available. Seats activated, usage hours, acceptance rates, completions used. They confirm deployment. They do not measure impact.

 Tier 2 — Flow metrics require internal instrumentation but are accessible with most engineering toolchains. PR cycle time, deployment frequency, build success rate, sprint velocity. These are meaningfully better signals than Tier 1. DORA’s research does offer a warning here, though: generating large volumes of code — which AI makes dramatically easier — increases individual perceptions of effectiveness, but DORA found this is not the primary metric worth optimizing. Volume is not throughput, and throughput is not outcomes. 

Tier 3 — Outcome metrics are the hardest to collect and carry the most signal. Lead time to change, change failure rate, MTTR, escaped defects per sprint, time-to-market for specific feature types, customer satisfaction delta. These metrics require deliberate instrumentation and, crucially, pre-AI baselines. Without a baseline, there is no comparison. Without a comparison, there is no story about whether the rollout worked. 

The organizations that discover their AI rollout has moved velocity but not delivery outcomes are almost always operating entirely in Tier 1. They have confirmation that the tool is being used. They have no measurement of what the tool is producing at the system level.

Lead time lives at the handoffs

Article 02 in this series, Widening the Circle: Why AI Success Depends on More Than Engineers, ended with a line worth carrying forward: the most revealing metric is not within a role — it is between roles. Lead time lives at the handoffs. 

This is the diagnostic insight that connects Tier 3 measurement to the practical question of where to look. Pull request count is a within-role metric. Sprint velocity is a within-role metric. Code lines committed is a within-role metric. None of them surfaces what happens when work moves between the eight or so functions that touch every feature — PM to design, design to engineering, engineering to QA, QA to security, security to release. 

The Faros number is worth restating in this frame. PR review time up ninety-one percent. That metric lives precisely at the handoff between the developer who wrote the code and the developer who has to review it. It is a handoff metric. It is not on any vendor adoption dashboard. And it is exactly where AI’s acceleration of code generation is currently producing its largest organizational cost. 

The DX 2025 Impact Report observed the corollary at the role level: Staff+ engineers — the people best positioned to multiply organizational throughput — have the lowest AI adoption rates of any seniority band. The arXiv 2507.21280 study of fifty-four developers across twenty-seven teams documented why: organizations expecting rapid productivity gains without investing in learning support undermine the very gains that motivated adoption. The researchers called it the Productivity Pressure Paradox. The leverage points in the organization are also the points where measurement is hardest and pressure is highest.

The user-centric finding 

There is one DORA finding that does more to reframe the measurement question than any other, and Article 02 introduced it in the context of who needs to be enabled. It also belongs in the measurement conversation. 

DORA found that adopting AI-assisted development tools can harm teams that do not have a user-centric focus. The mechanism is straightforward: AI accelerates the rate at which teams produce output. If the team lacks a strong north star for what the output should accomplish for users, AI accelerates movement in the wrong direction. Speed multiplied by misalignment produces more misalignment, faster. 

“A team with poor user focus that adopts AI is likely to be propelled even further in the wrong direction.” — DORA AI Capabilities Model, 2025 

Now consider what adoption metrics report in this scenario. Usage is up. Acceptance rates are healthy. Completions are high. Every Tier 1 metric is green. But the team is accelerating away from the right product decisions. The delivery system is degrading in exactly the way that adoption dashboards are designed not to see. The measurement system tells the organization the rollout is working. The customer outcomes say otherwise. Without Tier 3 instrumentation, the contradiction is invisible until it shows up in revenue. 

What the transition actually requires 

Moving from activity measurement to outcome measurement is not primarily a tooling problem. It is a framing problem. The decision to measure outcomes has to precede the rollout, not follow it three months later when someone in finance asks about time-to-market. 

There are three practical starting points. 

Establish pre-AI baselines before you expand the rollout. Lead time to change, change failure rate, and escaped defects per sprint should be captured for a representative period before AI coding tools reach a meaningful percentage of the team. These baselines become the reference point against which everything subsequent is measured. Without them, you are comparing the AI period to a hypothetical rather than a measured state — and the hypothetical will lose every argument with the vendor’s dashboard. 

Instrument the handoffs, not the role. Article 02 made the case structurally; here it is operationally. The most useful additions to existing engineering metrics are at the seams: PM sign-off to engineering pickup, code complete to QA sign-off, QA pass to deployment slot. CircleCI’s 2025 analysis of AI-driven development found that static CI pipelines cannot keep pace with the volume AI generates — meaning the platform seam is the next bottleneck most organizations will discover. Looking at lead time at each seam tells you where AI is helping, where it is queuing, and where it has actively made things worse. 

Add Tier 3 questions to existing rituals. Sprint retrospectives and quarterly engineering reviews already exist. The question is what gets asked in them. Adding three questions — has our lead time to change moved? Has our change failure rate moved? Has our downstream QA load moved? — does not require new tooling. It requires a decision to look at different data than the vendor dashboard provides.

Three Questions for Leaders This Week

1. What was your lead time to change in the three months before the AI rollout? If you cannot answer this, you have no comparison point — and no basis for a business case. 
2. Has your change failure rate moved since deployment? Faros’s data showed PR size grew 154% on average with AI. Larger PRs are a known change-failure risk. If you are not tracking this number, the risk is accumulating in a place you cannot see. 
3. What has happened to your QA team’s capacity and your reviewers’ queue? If AI is accelerating code generation and your review and testing functions are unchanged, the bottleneck has moved downstream — and your adoption dashboard cannot see it. 

The measurement trap is not about bad intentions. Most engineering leaders who report adoption metrics are reporting what they have, not what they want. The vendor gave them a dashboard. The board asked for evidence the investment is working. Green numbers solve the immediate problem. 

The harder question — whether the rollout is actually improving the delivery system — requires a different kind of data, collected deliberately, and asked before the investment is made rather than after. 

METR’s developers believed they were twenty percent faster while the stopwatch said they were nineteen percent slower. The gap between perception and measurement is not unique to individual engineers. It scales to entire organizations. The dashboards do the same thing the developers did — they tell a confident story about progress that the system itself has not yet confirmed. 

The shift from activity to impact is what closes that gap.

The AI Enablement Series

This is Article 03 of 05 in the Field Notes on AI Enablement series.
Article 01, The Bottleneck Just Moved introduced the Theory of Constraints framing for AI coding tool rollouts.
Article 02, Widening the Circle examined what enablement looks like for PMs, QA, designers, security, and SRE.
Article 04 Beyond the Launch looks at what happens at the ninety-day mark — and why most rollouts stall. What metrics is your organization tracking beyond the vendor dashboard? 

If you have moved from Tier 1 to Tier 3 — or are part-way through the transition — I would be genuinely interested to hear what worked and what surfaced surprises along the way. https://www.linkedin.com/in/vijayk-kloudlinq/

Sources

• DORA / Google Cloud, 2025 State of AI-Assisted Software Development Report and AI Capabilities Model (September 2025) 
• Faros AI, AI Productivity Paradox report and 2026 telemetry update (July 2025, January 2026) 
• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (RCT, July 2025) DX, AI-Assisted Engineering: Q4 Impact Report (November 2025) 
• CircleCI, 2025 State of Software Delivery — AI-driven pipeline analysis 
• arXiv 2507.21280, Productivity Pressure Paradox paired-interview study (2025) 
• Eliyahu Goldratt, The Goal (1984) — Theory of Constraints (referenced in Article 01)