Nobody owned the design

Correctness is verifiable. Architecture is not.

The compiler makes the case better than any failure could, precisely because it is so impressive. Ninety-nine percent of a hostile test suite passed. Then one design decision that nobody actually made, the register allocation strategy, produced a runtime gap that no test suite would ever have caught, because no test suite was ever asked to look for it.

The teardown’s own verdict called it “a remarkable achievement” that is “not ready for real use.” Both of those can be true at the same time, and the second one is not a coding failure.

Models have architectural knowledge. They don't have architectural judgment.

The benchmarks separate cleanly, and that separation is the whole finding. Ask a frontier model about architecture and it performs near ceiling. SAKE, which draws 2,154 questions from Bass and the Gang of Four, scored eleven models between 89.3% and 94.2%. It was saturated on release day.³

Now ask them to do it. ProgramBench hands an agent a compiled binary and its documentation and asks it to infer the spec and design a replacement. Across two hundred tasks, 1,800 runs and nine frontier models, every model scored 0.0% resolved.⁴ What they produced is as telling as the score: a median of 3 files where the human reference had 15, and 1,173 lines against 3,068.

R2ABench gives models real PRDs and asks for an architecture. They identify the right components, then connect them wrong, leaving 12 to 33% of those components as disconnected islands.⁵ Strip the architecture section out of the PRD and component extraction barely moves while orphans spike, which says the models were transcribing an architecture rather than inferring one.

The sharpest version is CodeTaste, built from 100 real refactorings averaging 91.5 files each.⁶ Given a blueprint, the best model reached 69.6% alignment with the human patch, at a precision of 48.4% against the human’s own 48.5%. That is parity on scope discipline. Given only a focus area like “refactor the routing system,” alignment fell to 7.9%.

The gap isn't knowing what a hexagonal architecture is. It's choosing which one this system needs.

The benchmark authors volunteer the caveat themselves. SAKE’s team writes that multiple choice “does not measure the quality of open-ended design decisions on a concrete system, where context, competing drivers, and trade-off judgment dominate.” That concession runs against their own interest, which is why it is the strongest form the claim can take.

The bottleneck is intent, and it shows up in telemetry

A study of 20,574 real coding-agent sessions across 1,639 repositories categorised where agents go wrong.⁷ Constraint violations accounted for 38.3%. Misreading developer intent accounted for 27.0%, and 44% of those traced back to an underspecified instruction. Inaccurate self-reporting made up another 22.6%.

Of episodes that got resolved, 91.5% required explicit developer pushback. Only 3.0% were caught by the agent itself.

That isn’t a coding failure. A study of 1,794 trajectories across 21 model-and-scaffold combinations found that 57.9% of failures were epistemic, meaning the agent misused information it already had, against 32.8% that were failures of competence.⁸ False premises alone accounted for 30.7%, the largest single trigger. The decisive error tended to land at median step 7 of a 27-step run, the first observable signal appeared about ten steps after that, and 28% of failures gave off no signal at all.

Models also don’t ask. HumanEvalComm rewrote 164 problems to inject ambiguity and incompleteness, and over 60% of responses generated code anyway, with pass@1 falling by 35 to 52%.⁹ On HiL-Bench, pass rates of 75 to 89% with complete information collapse to somewhere between 4 and 24% once the agent has to ask for what is missing.¹⁰ A Cornell group found the mechanism behind this. Models score 60 to 80% when asked point-blank whether something is ambiguous, but they actually stop to clarify in under 3% of real runs.¹¹ When expressing uncertainty isn’t rewarded, the authors write, “models learn to hide that awareness.” The awareness is there. It just doesn’t surface.

What got faster, and what quietly got harder

The speedups are real and specific. A CLI-agent rollout across tens of thousands of Microsoft engineers measured +24.0% merged PRs per engineer per day (95% CI: +14.5% to +33.7%).¹³ Google’s Int32→Int64 migration across 500M+ lines was 80% AI-authored and roughly 50% faster end to end.¹⁴ Airbnb migrated about 3,500 test files in six weeks against a 1.5-year estimate.¹⁵

Note what those three have in common: in each case a human had already decided what the change was. Google's localization ran on cross-reference graphs and deterministic AST filters that humans built, the model only generated the edits, and reviewer throughput was left as the bottleneck on purpose.

Google’s CEO supplies both endpoints himself: 75% of new code AI-generated, and measured engineering velocity up “10%.” The work moved rather than vanished. Median PR size roughly doubled across more than 1,450 organisations between Q1 2025 and Q1 2026, while PR volume rose only about 20%.¹⁷ Review load, in other words, is a diff-size problem rather than a PR-count problem. And review is being skipped rather than scaled.

Agent-steering commands make up 25.9% of human comments on AI-authored PRs, against 1.6% on human ones.¹⁸ Humans shifted from reviewing to directing.

Four things that haven't moved

Where we'd change our mind

Read this with a date on it. METR’s time-horizon measurements have been doubling roughly every 89 days since 2024, and their own page now notes that “measurements above 16 hrs are unreliable with our current task suite.”²⁷ The instrument ran out of range in ten months. Claims about what AI can’t do in this area have a half-life of roughly a year, and this one is no exception.

So here is the falsifier. CodeTaste's open track asks a model to choose what to refactor rather than execute a blueprint. If an agent crosses 50% there, or if R2ABench edge F1 crosses 0.5, the argument above needs rewriting.

Until then: the models will write the code, and they'll write it well. Someone still has to decide what it should have been, and answer for it when it isn't.

Every figure above was checked against a fetched primary source on 27 Jul 2026. Where a claim could not be verified it is flagged here rather than dropped quietly.

1. 01Building a C compiler with Claude · Anthropic Engineering, Feb 2026. The builder reporting on its own buildvendor
2. 02CCC vs GCC: an independent teardown · harshanu.space, 8 Feb 2026independent
3. 03SAKE: software architecture knowledge evaluation · arXiv:2606.29520, Univ. of Southern Denmarkpreprint
4. 04ProgramBench: binary to replacement design · arXiv:2605.03546, Meta FAIR / Stanford / Harvardpreprint
5. 05R2ABench: requirements to architecture · arXiv:2604.06683, Beihang. N=17, directionalpreprint
6. 06CodeTaste: 100 real refactorings · arXiv:2603.04177, ETH Zurichpreprint
7. 0720,574 coding-agent sessions in the field · arXiv:2605.29442, Notre Dame / Vanderbilt / Googlepreprint
8. 08Failure as a Process: 1,794 trajectories · arXiv:2607.09510, UCL / Nanjingpreprint
9. 09HumanEvalComm: communication under ambiguity · ACM TOSEM, arXiv:2406.00215peer-reviewed
10. 10HiL-Bench: asking for what's missing · arXiv:2604.09408, Scale AI. Conflict of interest notedpreprint
11. 11Knowing but Not Showing · arXiv:2605.25284, Cornellpreprint
12. 12Ask or Assume? · arXiv:2603.26233, Univ. of Viennapreprint
13. 13CLI-agent rollout, tens of thousands of engineers · arXiv:2607.01418, Microsoft. Vendor-adjacentpreprint
14. 14How is Google using AI for internal code migrations · arXiv:2501.06972, Googlepreprint
15. 15Accelerating large-scale test migration with LLMs · Airbnb Engineering, self-reportedvendor
16. 16Agentic coding tools and developer output · NBER w35275, 100,000+ GitHub developerspreprint
17. 17Should humans still review code? · Swarmia, 1,450+ organisations. Vendor telemetryvendor
18. 18Review practices on 33,596 agent-authored PRs · arXiv:2605.02273preprint
19. 19What is agentic engineering · Simon Willison, 15 Mar 2026practitioner
20. 20Building to the Test · arXiv:2606.28430, Microsoft. 18 runs, one taskpreprint
21. 21Cheap Code, Costly Judgment · arXiv:2607.01087, Purdue. N=1 case studypreprint
22. 22Harness engineering memo · Birgitta Böckeler, martinfowler.com, 17 Feb 2026practitioner
23. 23NIST SP 800-218A · US federal secure-development profile, Jul 2024standard
24. 24Linux kernel coding-assistants policy · docs.kernel.org, 2025–26policy
25. 25Godot contribution policy · Godot Engine, 30 Jun 2026policy
26. 26Agentic code review · Addy Osmani, 15 Jun 2026practitioner
27. 27Measuring AI ability to complete long tasks · METR time horizons, live pagelive page