Coding Was Never a Bottleneck

What the AI productivity research is actually telling us.

By Archana Rao and Gaurav Savla

AI has taken software development by storm. Between the two of us, we build products for software engineers and consumer products for millions of everyday users, so we have skin in the game. We want the AI productivity story to be true. More output, tighter timelines, happier and more productive engineers. Who wouldn’t?

But when we look at the actual research and then look at what’s happening in the real world, we can’t make them agree. Or rather we can, but only if we’re willing to admit that “productive” doesn’t mean what most of the recent discourse thinks it means.

The most uncomfortable finding first

In early 2025, a research organization, METR, ran a controlled experiment with open source developers. They found that (in contrast of what the industry was expecting) engineers using AI tools took 19% longer than those working without them, with a confidence interval of +2% to +39%. The slowdown was statistically robust. This was a different time in the industry. Claude hadn’t released its Opus models, the industry was figuring out what AI can and can’t do, but what makes this remarkable isn’t the slowdown, it’s that engineers believed they were approximately 20% faster while the data indicated otherwise, uncovering a significant gap between perception and reality.

Consider this finding for a moment before we pile the rest of the evidence on top of it because it changes how you read everything else.

METR attempted a follow-up study starting in August 2025, and what happened to that study is arguably more revealing than the original result. In February 2026 they published a post explaining why they abandoned the experimental design. The problem was that too many developers refused to participate unless they could use AI for all their tasks. Between 30% and 50% of remaining participants reported selectively avoiding submitting tasks they didn’t want to do without AI. The sample became systematically biased toward the developers and tasks least likely to show the value of AI.

Data from the late 2025 study shows an improvement in trends. For the subset of original developers who returned, the estimated effect shifted to an 18% improvement in speed (confidence interval: -38% to +9%). Among newly recruited developers, there was a 4% improvement in speed (-15% to +9%). But METR flagged these numbers as likely a lower bound because many people self-selected out. Their conclusion: AI tools have gotten more useful since early 2025, but the selection effects are now so severe that controlled measurement is nearly impossible. The developers most enthusiastic about AI will no longer work without it to serve as a control group. That’s not a failure of METR’s methodology. It’s a signal about where we are and where we’re headed.

Three more data points

Several additional studies landed over the course of late 2025 and early 2026.

Anthropic surveyed 132 of its own engineers in late 2025, conducted 53 interviews, and analyzed 200,000 Claude Code transcripts. Employees reported achieving a 50% productivity boost. As the engineering organization and usage of Claude grew, they claimed that pull requests per engineer per day were up 67%. Anthropic engineers use Claude in 60% of daily work, and Claude performs more tasks autonomously.

CircleCI analyzed 28 million CI workflows across thousands of teams. Workflow throughput was up 59%, but main branch throughput for the median team declined 7%. Build success rates fell to 70.8%, which is a five-year low. More code exists than ever, but less of it reaches production, and the CI is becoming a chokepoint.

Harvard Business School researchers studied 78 workers using artificial intelligence to perform tasks outside their expertise. AI helped everyone brainstorm equally well, but on execution, workers whose skills were far from the domain underperformed domain experts by 13%. The gap that AI appeared to close in planning reemerged in delivery.

METR’s May 2026 survey of 349 technical workers—which was conducted after the experimental design broke down—found self-reported productivity value gains of 1.4x to 2x from artificial intelligence tools. But METR’s own research staff, the people most calibrated on the perception bias they documented in 2025, reported the lowest gains of any subgroup in that survey.

What this looks like in practice

Here’s a scenario that will feel familiar to some readers: Engineer activity metrics look great on the surface. Pull requests are increasing, code commits are up, velocity points are being closed at a pace the team hasn’t hit in years. The leadership team is happy, engineers feel more productive. Then someone—likely a PM—asks why the roadmap items marked “in progress” six weeks ago are still in progress.

Everyone comes to the same realization all at once: The feature timelines haven’t really changed. What’s happened is that AI has dramatically reduced the cost of starting work, but production-ready polish remains a challenge. First draft functions, boilerplate, scaffolding, and test writing explanations for unfamiliar code have all gotten significantly cheaper. But the bottlenecks on shipping were never those tasks. They were product decisions, design reviews, QA, compliance, infrastructure, release processes. When you speed up coding, you end up jamming more work-in-progress items against the same downstream chokepoints. The CircleCI data on 28 million workflows is, in part, a picture of what that looks like at scale: massive activity in feature branches with flat or declining throughput on main.

This isn’t just a pattern in aggregate data. As Fiona Fung, a director of engineering for Claude Code at Anthropic, explained at a June 2026 talk, writing code, writing tests, and refactoring rarely slows her team down anymore, but the bottlenecks didn’t disappear. Verification, code review, and security took their place. She flagged CI specifically. As teams generate more code, build systems and CI pipelines can struggle to keep up. That’s a team running one of the most AI-accelerated engineering orgs in the world hitting the same constraint wall the CircleCI data describes. The ceiling isn’t code authoring speed anymore; it actually never was.

Anthropic’s finding that 27% of AI-assisted work wouldn’t have happened otherwise cuts both ways. Some of that work is genuinely valuable, like prototype explorations that inform real decisions, documentation that actually gets written. Some of it is work nobody prioritized because it simply wasn’t important enough. Now it’s burning review cycles and CI resources because building it became nearly free, while reviewing, testing, and maintaining it didn’t.

The competence-confidence gap

The HBS study identifies a specific mechanism: AI closes the confidence gap between novices and experts. It gives everyone equal access to plans, explanations, and first drafts. But it doesn’t close the competence gap. When a backend engineer builds a frontend feature with AI assistance, they produce something that looks right. The problems are underneath, in the decisions they didn’t know to question and the edge cases they didn’t know to test.

The early METR result suggests this extends even to experienced practitioners working in their own domains. The AI doesn’t make them incompetent; it actually makes them feel more capable than their output justifies. And as METR’s follow-up collapse demonstrated, once developers integrate AI deeply enough, they lose the ability to work without it as a reference point in what researchers have called automation bias.

This is the part that should concern engineering leaders. You can’t fix what you can’t see. If every engineer on your team sincerely believes they’re 50% more productive and your ship dates haven’t moved, there’s a problem that nobody thinks exists.

What makes artificial intelligence native development sustainable

Make code review more rigorous, not faster. AI-generated code passes surface checks easily—clean formatting, consistent conventions, no linter complaints, etc.—which is exactly why it’s dangerous. The problems are the kind a reviewer won’t catch from skimming a diff.

I’ve been calling this “reasonable doubt review.” The practice is to start from skepticism rather than trust, asking, “What could be wrong here that I wouldn’t catch from the diff?” Specifically, what assumptions did the model make that aren’t visible in the output? What edge cases does this silently fail on? Where does this couple to something the author might not have been thinking about?

This is slower. That’s the point. It’s also not infinitely scalable, which is why it needs to be paired with automation on the things that don’t require judgment and human attention concentrated on where it does.

The Claude Code team’s approach is a good example: Let AI handle style, linting, bug-catching, and test generation as a first pass, but route security-sensitive code, trust boundaries, and anything touching legal risk directly to domain experts. The division isn’t “AI reviews smaller, low-risk changes and humans review bigger, higher-risk changes.” It’s “AI handles surface correctness, humans own consequential judgment.” That’s a meaningful distinction. A lot of teams are doing the first while thinking they’re doing the second.

Adapt your CI to the new failure modes. CircleCI’s build success rate hitting a five-year low while throughput exploded suggests most teams haven’t updated their pipelines to catch how AI-generated code breaks. AI-generated code fails differently than human-generated code. It’s more likely to be locally correct but architecturally inconsistent, pass unit tests and fail integration tests, and respect function signatures while violating the assumptions that those functions were built around. Integration tests, contract tests, and architecture fitness functions that enforce your system’s constraints in the pipeline will catch more of this than a linter or a type checker. If AI-generated code violates your patterns, the build should catch it before a reviewer opens the diff. This addresses what will become your review problem and your infrastructure problem.

Ship behind feature flags and monitor aggressively. Accept that you will not catch everything before deployment. Instead of betting entirely on premerge quality—which the evidence suggests is harder to assess than it feels—deploy to 1% of users, watch the dashboards, and roll back fast when something’s wrong. This approach also forces investment in observability, which pays for itself independently of the AI question.

Require human-written tests for AI-assisted code (until AI can confidently generate deterministic tests). Human-written tests, especially for edge cases and boundary conditions. The discipline of writing the test forces the developer to think through the behavior rather than accept the output at face value. If an engineer can’t write the test, they probably don’t understand the code well enough to ship it. That’s a useful signal, not a failure state.

Protect deliberate knowledge-sharing time. The Anthropic study found that mentorship was quietly eroding as Claude replaced the conversations engineers used to have with each other. This is the long-horizon risk in the data. Architecture decision records, rotating system walkthroughs, and pairing sessions where a senior and junior work through a problem together feel inefficient next to asking an AI, but they’re how teams build the shared understanding that prevents the same mistakes from being rebuilt in better-formatted code every six months.

The measurement problem

So does this mean we stop using AI? No. Use AI and use it aggressively where it clearly helps tedious tasks, prototyping, and exploratory work, anything you can verify quickly. The gains on well-scoped, independently verifiable work are real.

But if you’re trying to measure whether AI is actually helping your team ship, PR count and self-reported velocity are the wrong instruments. The four studies we evaluated taken together indicate that these aren’t just measurement problems, they are a warning sign that the feedback loops we’d normally rely on to detect whether something is working have changed significantly.

The harder question—the one that all the research studies raise without quite answering—is what the measurement would actually tell you. Cycle time from feature conception to delivery, or the rate at which merged code reaches production without rollback, might be better metrics. Or the gap between planned and actual scope at the end of a sprint. Or maybe a bit more abstracted: company revenue growth correlated with the AI investment (tooling, infrastructure, and OpEx).

None of these are easy to instrument. The question you should be asking of your teams isn’t “How productive do we feel?” It’s “What would we need to measure to know?”

Note: The research work pertaining to this article was done in a personal capacity. Views are our own and do not reflect the views of our employers in any way.