The 20% feeling and the 19% fact

Ask a team that has adopted AI coding tools how it is going, and you will usually hear a version of the same answer: noticeably faster. Ask them to put a number on it and the number lands somewhere around twenty per cent. I hear it in almost every delivery review — and until last year I had no polite way to check it.

Then somebody ran the experiment properly. METR, a research group that evaluates AI capabilities, took sixteen experienced open-source maintainers and 246 real tasks from their own repositories — not exercises, their actual backlog — and randomised which tasks allowed AI tools. Before starting, the developers forecast that AI would speed them up by 24%. After finishing, they estimated it had sped them up by about 20%. The measurement said they had been 19% slower with AI allowed.

Sit with that for a moment. These were not sceptics or novices. They believed they were faster while being slower, and they still believed it after the fact. If your adoption report is built on developer self-assessment — and most are — that is the accuracy of the instrument you are using.

Both numbers are real

The obvious objection: there is a controlled study on the other side. GitHub's 2023 trial with 95 developers found that Copilot made them 55.8% faster. It did — at building a standalone HTTP server, from a clear specification, in a fresh codebase, with no reviewers waiting and no production to break.

That is not a flaw in either study. It is the finding. AI assistance is enormously sensitive to the shape of the task: greenfield, well-specified and isolated is one regime; embedded in a large codebase the developer already knows intimately, with implicit standards and a review gate, is another. The first regime is where demos live. The second is where your delivery happens.

If that chart feels familiar, it is because the AI failure-rate numbers do exactly the same thing — I took those apart in an earlier piece. Different studies, measuring different things, quoted as if they were one fact. The productivity numbers deserve the same discipline as the failure numbers: ask what was measured, on what work, by what method, before letting the number into a plan.

Where the time goes

The interesting question is not whether the tools generate code quickly — they do — but what happens to that code downstream. Two large-scale signals suggest the bottleneck does not disappear; it moves.

DORA's 2024 report, drawing on roughly 39,000 responses, estimated that a 25-point increase in AI adoption came with delivery throughput down 1.5% and delivery stability down 7.2%. Individual developers report moving faster; the pipeline, measured end to end, does not — and what ships breaks slightly more often.

GitClear, analysing hundreds of millions of changed lines year over year, reports code duplication rising sharply since assistants arrived — duplicated blocks up several-fold — while the share of moved code, the signature of refactoring, declines. Generated code gets pasted in; restructuring work that keeps a codebase cheap to change is happening less. That is velocity now, paid for out of maintenance later.

Typing was never the constraint. Review, verification, integration and ownership were — and a tool that accelerates typing feeds the constraint faster than the constraint can drain.

What this does to delivery management

None of this argues for banning the tools. I use them daily, and this site is partly built with them. It argues for managing them like any other change to a production system — with instrumentation instead of testimony. Concretely, before believing a speed-up:

• Measure the pipeline, not the keystrokes. Cycle time from first commit to merged, review queue depth, and time-in-review. If AI is helping, these move. If only “lines written” moves, the constraint has been fed, not relieved.
• Watch the quality exhaust. Rework share, defect escape rate, change failure rate, duplication trend. DORA's stability finding says these are where the bill arrives.
• Split the work by regime. Greenfield scaffolding, test generation and one-off scripts sit in the regime where the 55.8% lives. Deep changes to a mature system sit in the regime where the −19% lives. An adoption policy that treats them identically is wrong in one direction or the other.
• Distrust self-report in both directions. The METR developers were honestly wrong about their own speed. Your enthusiasts and your sceptics will be too. The argument is settled by the pipeline metrics or it is not settled.

The disclosure

As with the failure-rate piece: I benefit from both halves of this story. If AI makes your team dramatically faster, someone has to restructure delivery around it; if it quietly makes your delivery worse, someone has to find where. Consultants are not disinterested readers of productivity statistics — which is precisely why the numbers should be checked rather than quoted.

The honest summary: the tools are genuinely powerful, the feeling of speed is genuinely unreliable, and the only version of the truth that matters is the one measured on your pipeline, on your work, with the method written down.

• AI delivery
• SDLC
• Measurement
• Developer productivity