Nobody Knows How Much Faster AI Makes You

The single most-quoted number in the AI productivity argument now sits under an out-of-date warning written by the researchers who produced it. METR's finding that developers took 19% longer with AI was careful, narrow, and honest about its limits. Their 2026 follow-up estimates the opposite direction and can't establish it, because developers started refusing to do tasks without AI and stopped being able to report how long anything took. The measurement came apart. I don't have a number of my own either, and I'm going to explain why that's the honest position rather than a cop-out.

The AI coding productivity number you've been quoting has a warning label on it

You have seen this study. Sixteen experienced open-source developers, real issues from repositories they'd maintained for years, each issue randomly assigned to allow or forbid AI. The result travelled because it was so satisfying to repeat: when they were allowed to use AI, they took 19% longer.

Go and look at that page today. There's a banner across the top, put there by METR:

These results are out of date. We have released results that are current as of early 2026, in a continuation of this study. We believe these historical results no longer reflect the current impact of AI models on open-source developer productivity.

That banner went up after a follow-up published on 24 February 2026. In the six months since, I've watched the 19% get quoted in statistics roundups, conference talks and comment threads, always clean, never with the banner. It has become the load-bearing fact for one side of an argument that its own authors have stopped standing behind.

This isn't a post about whether AI coding tools are good. It's a post about what the numbers you keep getting hit with actually measured, and why they can't be stacked on top of each other. I've done this once before, reading a coding-agent paper against its own statistics, and the pattern turned out to be the same: the popular number and the careful number are rarely the same number.

What METR actually measured

Sixteen developers, 246 issues, two-hour tasks

The design was tight. Sixteen developers, recruited from large open-source repositories averaging 22,000+ stars and over a million lines of code, projects they had contributed to for years. They supplied 246 real issues, the sort of bug fixes and refactors that were already on their plate. Each issue was randomly assigned: AI allowed, or AI forbidden. Tasks averaged about two hours. Developers recorded their screens and self-reported implementation time. METR paid them $150 an hour.

The tooling matters more than most people repeating the result seem to realize. This was primarily Cursor Pro driving Claude 3.5 and 3.7 Sonnet, which METR describes as "frontier models at the time of the study." The data was collected between February and June 2025. If you have used a 2026 agent on a large codebase, you already know that isn't the same activity.

METR then went looking for reasons the slowdown might be an artifact. In their words: "We investigate 20 potential factors that might explain the slowdown, finding evidence that 5 likely contribute." They checked whether developers actually complied with their assignment, whether they quietly dropped hard tasks, whether PR quality differed. This was good work.

The generalization METR refused to make

Here is the part almost nobody quotes. The original write-up contains a table of conclusions the study does not support. The first row reads:

AI systems do not currently speed up many or most software developers

with the clarification: "We do not claim that our developers or repositories represent a majority or plurality of software development work."

Read that again, because it's the exact claim the 19% gets used to support. Sixteen long-tenured maintainers of large, mature, high-standards codebases are not a random sample of programmers, and METR said so in the original post, in a table, in advance. The study was careful. The quoting was not.

What happened when they ran it again

Two numbers, both indistinguishable from zero

METR started a second experiment in August 2025: ten developers from the original study, plus 47 newly recruited from a wider set of projects. Pay dropped from $150 an hour to $50.

The results, in their own notation:

Our early 2025 study found the use of AI causes tasks to take 19% longer, with a confidence interval between +2% and +39%. For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%. Among newly-recruited developers the estimated speedup is -4%, with a confidence interval between -15% and +9%.

The sign convention trips people up, so: positive means tasks took longer. The original +19% sat between +2% and +39%, entirely on the "slower" side of zero. The follow-up numbers point the other way, toward faster.

Cohort	Point estimate	Confidence interval	Contains zero?
Original study, 2025	+19% (slower)	+2% to +39%	No
Returning developers, 2026	-18% (faster)	-38% to +9%	Yes
Newly recruited, 2026	-4% (faster)	-15% to +9%	Yes

But look at where those intervals end. A confidence interval is the range of true values the data can't rule out. Both follow-up intervals run past zero into positive territory: -38% to +9%, and -15% to +9%. When an interval includes zero, the data hasn't established that there's any effect at all, in either direction. That's not my statistical opinion, it's arithmetic on the numbers METR published.

METR's own verdict is blunter than anything I'd write:

we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools.

They still think developers are probably faster now than in early 2025. They say so. They also say their data is "only very weak evidence for the size of this increase." Both halves of that sentence are load-bearing, and only the first half ever gets repeated.

Developers wouldn't do the work without AI

The reason the second study couldn't produce a clean signal is more interesting than the result itself.

Participation itself became selective. Developers weren't being difficult, they just didn't want to work without AI, and the design required exactly that. METR reports that "an increased share of developers say they would not want to do 50% of their work without AI, even though our study pays them $50/hour to work on tasks of their own choosing." And among those who did participate:

When surveyed, 30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.

Between a third and a half of participants were withholding exactly the tasks where AI would have helped most. METR is careful to say the pay cut probably contributed too, which is the kind of self-incriminating detail that makes me trust the rest of the write-up.

Notice which way that bias runs. METR's own read is that these effects make their estimate "a lower-bound on the true productivity effects of AI on these developers," while adding that the selection effects "seem to affect a minority share of developers and of tasks, which limits the degree of bias." So the researchers are simultaneously telling you the experiment now leans against AI and telling you not to over-correct for it. That is what careful looks like.

The measurement unit came apart

You can't time a task you're not sitting at

Buried in METR's list of secondary problems is the sentence that reframes this entire argument:

Some developers reported it was challenging to report time-spent in completing tasks when they used agentic tools, because they would often work an unrelated task while waiting for the agent to complete its work.

Task-completion time was always built on assumptions: one person, one task, attention on it continuously from start to finish. That's a fair model of a developer with an editor open. It is not a model of anything I've done this year. I'll start an agent, go read something else, come back. What number goes in the box? Wall-clock time punishes the approach that's actually winning. Attention time is unmeasurable without instrumentation nobody has.

The instrument didn't get a wrong answer. It stopped having a well-defined question to answer.

"Done" moved too

The other half of the problem showed up independently, from a working developer rather than a research lab. In July 2026, in a post called "2x, not 10x: coding with LLMs in 2026" that drew 280 points and 245 comments on Hacker News, the author describes his workflow and lands on this:

I still consistently underestimate how long that iteration is going to take. A working implementation used to mean a task was 80% done; now it's more like 20%.

If the finish line moves, time-to-finish isn't comparable across conditions. A developer without AI arrives at a working implementation late and is nearly done. A developer with AI arrives early and has most of the work left. Same milestone, different meaning, and a stopwatch can't tell them apart.

His explanation for why the ceiling is 2x and not 10x is the most convincing thing I've read on this:

LLMs' increased rate of adoption in 2026 is largely due to them becoming reliable enough to run effectively in automated feedback loops. Now that they've passed that threshold, further improvements in model performance will have a much smaller impact on productivity than they have had previously.

His analogy: to climb stairs you have to be tall enough to make one step, but being tall enough to take three at a time matters much less. What made them useful was crossing a reliability threshold, not raw capability, and you only cross a threshold once. Models keep getting better on benchmarks. The thing that made them useful in a loop already happened.

Honest measurement is hard even when the quantity is trivial. When I wanted to know what a JavaScript import actually costs, I had to build a tool that tree-shakes each import in isolation, because every simpler method gave a number that was confidently wrong. That was a quantity with an unambiguous definition, sitting on disk. Developer productivity has neither property.

Three numbers, three different questions

Most of the confusion is people quoting incompatible measurements at each other. Here is what the three big sources are actually doing:

Source	What it measures	Population	Finding
METR RCT	Per-task completion time, randomized within subject	16 long-tenured OSS maintainers (2025); 57 developers (2026)	+19% time in 2025; -18% and -4% in 2026, both intervals crossing zero
DORA 2025	Organizational delivery throughput and stability	~5,000 technology professionals	Throughput and product performance up, delivery stability down
Survey figures	What people believe about themselves	Practitioners at large	Near-universal adoption, self-reported gains around 10%

These are three different questions. Whether a given task takes you longer, whether your organization ships more and breaks more, and whether people feel faster. A result from any one of them tells you nothing about the other two, and the roundup posts stack them as though they're votes in the same election.

DORA's headline is the best one-line summary anyone has produced: "AI doesn't fix a team; it amplifies what's already there." Their 2025 report found a positive relationship between AI adoption and both delivery throughput and product performance, alongside a continuing negative relationship with delivery stability. Those are associations in survey data, not demonstrated cause. It's still a real finding about how organizations behave, and it still says nothing about whether your Tuesday afternoon went faster.

And then there's the belief number. DORA found that 90% of respondents use AI at work and more than 80% believe it has made them more productive, while 30% report little or no trust in the code it generates. Hold that against METR's original result: those developers predicted AI would speed them up 24%, came out 19% slower, and afterwards still believed they'd been sped up 20%. Between what happened and what they thought happened is a gap of roughly 39 percentage points, on their own work, with screen recordings running.

So when a survey tells you 80% of developers feel more productive, you have learned something real about how people feel. You have not learned that they are.

All of them measure the work you were already going to do

There's a hole underneath every study here, and the best statement of it was a comment on that Hacker News thread:

While I agree with the premise, I think this angle only applies on work one was going to do no matter what. The real power of these tools is that there are so many ideas people would like to try, but never have the time or motivation to pursue. So the comparison is not only "built with and without LLM" but "would you even build this if you didn't have the LLM?"

Every design in this argument is a within-subject comparison on pre-specified tasks. METR asked developers to nominate issues in advance, which is the only way to randomize them. That design can never see the project that exists because starting got cheap enough to bother. If your gains are concentrated in work you'd otherwise have skipped, a study built on tasks you'd already committed to will measure approximately none of it.

This is where I'd be tempted to declare victory for the optimists, so here's the reply that stopped me, from the same thread: "This can be a negative multiplier: code I thought I wanted that gets immediately abandoned is a net-negative if no one else wants it." Cheap starting produces abandonware as readily as it produces products. I've got repositories that prove it. A measurement that counted every project I began would flatter me considerably more than one that counted the projects still alive in six months.

I don't have a number either

Here's where a post like this usually turns, and the writer produces their own figure. I'm not going to, because I haven't measured it.

I ship most of my work through agents now, across a dozen repositories, and I could give you a confident-sounding number this afternoon. It would be worthless, for reasons I can name precisely, and they're the same reasons METR is redesigning their study:

I pick work based on what agents are good at. That's the selection effect, and I do it constantly, without noticing. I run more than one agent at a time, so wall-clock time on any single task is meaningless. I have no control condition, because I'm not going to spend a month building things the slow way to find out. And I'd be scoring my own homework, which METR demonstrated I'd get wrong by around 39 points.

I do publish numbers when I have them. When I wanted to know which extraction strategy actually worked, I ran a 28-URL benchmark across 15 strategies and published the losses along with the wins. The difference is that extraction accuracy has a definition. "How much faster am I" currently doesn't.

What I believe, stated as belief: the threshold argument is right, and my gains are real and concentrated in work I wouldn't have started otherwise. I'm not putting a multiplier on that, because I'd be making the multiplier up. It's an impression from someone with an obvious stake in the answer, and worth exactly what that's worth.

What I'd want measured instead

Four things would improve this argument more than another point estimate:

Report the interval, not the number. METR did. Everyone quoting METR didn't. "+19%, CI +2% to +39%" and "19% slower" are not the same statement, and only one of them survives contact with a follow-up study.

Measure surviving work, not task time. Count what's still running in six months. Task time is now unmeasurable for the reason METR found, and it was measuring the wrong thing even when it worked.

Count the projects that wouldn't exist. And separately, the ones abandoned in a week. Both are real effects and they partly cancel. Nobody has tried to size either.

Stop pooling "AI" across three years of tooling. METR's 2025 result is about Cursor with Sonnet 3.5 on mature repositories. Citing it as evidence about 2026 agents is a category error, which is precisely why METR put a banner on it.

The one finding that has survived all of this is the unflattering one: developers were badly wrong about their own speed. They expected a 24% speedup, came out 19% slower, and still reported a 20% gain afterwards, with their screens recording. Then the researchers went back, came to think the slowdown had probably reversed, and couldn't establish it, because the profession had moved past the experiment's ability to describe it.

If you have actually measured this on your own work, with a control condition you can defend, I want to read it. Reply on X at @AliFoundry, or come see the things I build and tell me I'm wrong.

FAQ

Did METR retract the 19% slowdown study? No. Nothing was retracted. METR marked the original results as out of date, published a follow-up in February 2026 with results pointing the other way, described that follow-up's signal as unreliable, and are now redesigning the experiment. The original study stands as a valid measurement of sixteen developers using early-2025 tooling.

So is AI making developers faster or slower in 2026? At the level of an individual task, the honest answer is that nobody has established it. At the organizational level, DORA's 2025 data shows AI adoption associated with higher delivery throughput and lower delivery stability. Those are different questions with different answers.

Why do all the studies disagree? Mostly they don't. They measure different quantities. Per-task completion time, organizational throughput and self-reported belief are three separate things, and a finding in one doesn't transfer to the others.

What does "the confidence interval crosses zero" mean? A confidence interval is the range of true values your data can't rule out. If that range includes zero, then "no effect at all" is still consistent with what you observed, so you haven't demonstrated an effect. METR's 2026 intervals were -38% to +9% and -15% to +9%. Both include zero.

Should I trust my own sense that I'm faster with AI? Directionally, maybe. Numerically, no. METR's developers reported a 20% speedup while measurably running 19% slower, with their screens being recorded. Personal impressions are evidence about how work feels, which is worth something, just not the thing people cite them for.