Lilith.
⌕
Editorial illustration: Anthropic sped up model development 1.9x, but key evidence remains hidden
Lilith illustration · editorial remix

METR received unusually broad access to Anthropic's internal data for 11 weeks to examine whether AI is accelerating the development of subsequent models. The report supports the intuition that practitioners can sometimes feel a model-quality jump before public benchmarks capture it. It also shows why that feeling is not enough.

Internal benchmark progress accelerated by 1.9x

METR worked from August 12 to October 29, 2026. Its AECI analysis found that internal model scores had grown at 1.9 times the earlier rate since October 2025, with a 95% confidence interval of 1.5 to 2.8. Robustness variants produced a wider range of 1.1 to 2.9.

The evaluators judge that most of the current acceleration, perhaps about 80%, came from using AI in research and development. Interviews suggest that assistance to people still contributes more than fully autonomous work on the most important parts of model training.

Vibe-testing can detect a break, not explain it

Ethan Mollick argued that the result validates experience-based model assessment and said Claude 3.7 Sonnet clearly exceeded expectations in 2025. Such judgment can flag a change in real-world usefulness early. It cannot separate a better model from improved agent scaffolding, a larger inference budget or a different task mix.

METR tried to make those distinctions through benchmarks, internal data and interviews. The practical lesson for product teams is simple: subjective experience is a useful alarm and a poor measuring instrument.

Anthropic redacted most of the decisive evidence

Anthropic could remove non-public information and, according to METR, redacted most of the evidence that most informed the risk conclusions. The public cannot independently inspect a crucial part of the argument. METR also lacked direct access to unreleased models and used Claude for much of its own analysis.

AECI focuses on well-specified, verifiable tasks. Its result cannot automatically be translated into equal acceleration on vague research work or into productivity gains for an ordinary company.

The next measurement must separate model, workflow and human labor

Repeated measurements on new models will matter, ideally with a preregistered method and less provider control over disclosure. Internal teams should pair experience reports with completed experiments, review time, rework and total cost. Only that combination can show whether research is truly faster or merely feels smoother.

Lilith's verdict

Vibe-testing can feel the train pull away. METR added a speedometer, but Anthropic taped over part of the dial.