Where the 19% came from¶
Almost exactly a year ago, I wrote about a randomized controlled trial by METR that measured open source developers 19% slower with AI than without. As part of my observation that AI assistants are decimating the "Junior Engineer" role at companies, this piece of information didn't help my view.
I've seen this in my own hiring, and touched on improvements my teams have made utilizing AI assistants in a post a few months ago. The story is the same though, while I've continued to hire for roles on my teams, none of those have been junior engineering roles.
METR started running it again in August 2025, and in early 2026 published their results. Let's talk about this...
What the first study measured¶
16 developers worked 246 tasks. The repositories averaged more than 22,000 stars, a million lines of code, and the developers had been contributing to them for years. Tooling was Cursor Pro with Claude 3.5 and 3.7 Sonnet.
Every one of those variables impacts the result. 16 developers is a small sample. A repository you've worked in for years is similar to a senior engineer at a company working on the same code base for years. They already hold the context an assistant would otherwise supply. Inline code suggestions in early 2025 are a different way of working than an agent running on its own for 20 minutes in 2026.
What happened when they ran it again¶
The follow-up was a bigger study with 57 developers, 143 repositories, and more than 800 tasks.
For the 10 developers who had also been in the original study, the estimated effect was an 18% slowdown, with a confidence interval running from 38% slower to 9% faster. For the 47 newly recruited developers, 4% slower, with an interval from 15% slower to 9% faster.
18% against the original 19%. The first result came back almost exactly the same. The confidence interval around it changed though. The original ran from 2% slower to 39% slower and excluded zero. Both of these new results cross it, which means the study can no longer separate the effect from nothing at all.
METR explained why they no longer trust their own instrument, because their results no longer mean anything.
Why METR stopped trusting the measurement¶
So what went wrong? METR lists six problems. Most of them follow from AI getting better over the past year to year and a half.
Three look like a selection effect. Developers declined to participate once there was a chance they would be assigned to work without AI, so the study systematically missed the people most optimistic about it. From the other participants, 30% to 50% told METR they were holding back specific tasks because they didn't want to do those tasks without AI, so it systematically missed the work with the most to gain. Developers were less likely to finish a task at all once it was assigned to the no-AI condition, with one participan completing none of theirs. Pay dropped from $150 an hour to $50 an hour over the same stretch, which METR believes contributed, though they put developers' expectations about AI ahead of money as the cause.
A fourth problem is the clock. Developers told METR that timing a task got difficult once agents were involved, because a developer waiting on an agent tends to work on something unrelated while it runs.
The bias in the experiment is fairly obvious. Refusing to work without AI, withholding the tasks where AI helps most, and abandoning the ones that landed in the wrong condition all push the measured result toward slowdown.
METR's own conclusion says:
Based on conversations with study participants, we believe it is likely that developers are more sped up from AI tools now - in early 2026 - compared to our estimates from early 2025. However, because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase.
They are now redesigning around developer-level randomization, shorter experiments, and observational data.
What held up?¶
One result from the original study did hold up though.
Before starting, developers predicted AI would make them 24% faster. After finishing the tasks, having done the work, they still believed they had been sped up by about 20%. The clock said 19% slower.
Watch out for that difference though. It's calculated from the same bias I mentioned, and the selection effects that pushed the measured result toward slowdown would widen it. METR didn't formally re-measure perception in the follow-up.
Under each condition METR has tested, developers' estimates of their own speedup have sat above what measurement showed, and METR now warns about the difficulty of interpreting self-reported productivity estimates "and thus their potential biases." This also isn't surprising to me. I've sat in enough developer sprint cycles, planning sessions, retros, quarterly plannings and other agile ceremonies to know that estimating effort is an inexact science. It takes time, experience as a developer and more importantly, experience with the system to be able to accurately estimate how much time and effort a change will take.
A developer survey is not a measurement of developer productivity. Ask your team how much AI is helping and you'll get a real answer about morale and an unreliable one about throughput.
The METR developers were an unusual population. Deep existing context, mature code, high standards, work they had been doing for years. My thought has been that an assistant supplying context to someone who already has all of it isn't worth much. In short, they were "Senior Developers".
Daniotti and colleagues, published a study in Science in January, and ran a classifier over more than 30 million commits from 160,000 developers in six countries. It put AI-written Python functions in the US at 29% by the end of 2024, up from 5% in 2022, with a 3.6% rise in quarterly commit rates alongside it. That's positive, and much smaller than what marketing messages say or CEOs broadcast in the news. It also finds that the gain lands somewhere specific:
GenAI increases output and helps programmers expand into new domains - but only for senior-level developers. Early-career developers, despite being the most enthusiastic adopters, see no measurable gains.
METR timed a few hundred self-chosen tasks inside million-line repositories its participants had maintained for years. Daniotti counted commits across an entire population, where "senior" means a few years of history rather than maintainer of a 22,000-star project. On mature code, with people who already hold the context, measured in wall-clock time, the gain is not obviously there.
Inside my own organization I write down the conditions before I write down the number. Which teams, what kind of work, how it's measured, over what period, against what baseline. I learned this flagging spam a decade ago, where I published human accuracy at 85.7% before I let a classifier act on a live queue, because a machine that beats nothing in particular isn't an improvement.
An engineering leader who can state their AI gain but not the conditions that produced it is telling you a story. Ask how the rollout is going and you should get numbers back, the same way you should from any healthy organization.
The first study ran on Cursor with Claude 3.5 and 3.7 Sonnet. That was 18 months ago. If you're still citing the number that came out of it, read what METR has published since.