2026-10-19 · ← News
An AI agent lifted econometric code accuracy from 79% to 97%. The errors left over are the invisible kind
Ethan Mollick says writing good Stata code, a skill that built his academic career, is becoming obsolete. He points to a study of 40 econometric tasks: when the model can run and revise its own code, correct results rise from 79% to 97%. The errors that remain are silent, though, and catching them is still the researcher's job.
The image could not be loaded.
Wharton's Ethan Mollick wrote on X that writing good Stata code was very important for his academic career and that the skill is now largely becoming obsolete. A new study backs him up with data: an agent writes econometric code almost without mistakes. The catch is that the mistakes it still makes never trigger an error message.
As an agent, Stata caught up with Python after trailing by 25 points
On October 19, Mollick linked to a VoxEU column by economists Sebastian Galiani, Federico A. López and Raul A. Sosa. The authors built 40 econometric tasks covering linear regression, difference-in-differences, instrumental variables, regression discontinuity, survival analysis and time series. Every task ran in Stata, R and Python, with and without an example in the prompt.
They tested two models, Claude Sonnet 4.6 in Claude Code and GPT-5.6 Luna in Codex, for 14,400 runs in total. As a chatbot, the model writes a script and never sees it run. As an agent, it runs the script, reads the output and revises until it decides it is done. The share of correct results rose from 79% to 97%. Claude Sonnet 4.6 went from 81.4% to 99.2%, GPT-5.6 Luna from 77.3% to 95.1%.
Stata, the most widely used software in empirical economics, moved the most. As a chatbot with no example, it trailed Python by 25.5 percentage points for Claude and 30.8 points for GPT. As an agent, the gap shrank to 1.3 and 6.8 points.
Syntax got cheap, and checking the result is now the expensive part
Mollick's obsolescence claim fits one layer of the work: turning a method into code. For Claude, the agent raised the cost of a run from 4.56 to 14.09 cents. Each extra correct result costs about $0.54, which the authors say pays for 0.6 to 2.2 minutes of a research assistant's time. At that price, nobody will keep hand-debugging Stata macros.
The study also changes which errors survive. The authors separate loud failures, where the script crashes or returns no result, from silent ones, where it runs and a number is wrong. Loud failures fell from 15.8 per 100 runs to zero. Silent ones only fell from 4.9 to 2.9. Under the agent, every remaining error is silent. Typically it is a wrong index or a wrong term in a formula inside an otherwise correct method. The worst task was shift-share instrumental variables, where the agent returned every estimate but a standard error outside the tolerance.
For PhD students and their advisers, that implies a different curriculum. Fewer hours on syntax, more on spotting a plausible-looking regression that returns nonsense.
Simulated data is a lab, and real research is messier
The benchmark has clear limits. Each task names the method and comes with simulated data and a reference answer computed in advance. In real research, nobody hands you the correct result upfront. Much of the work is also data cleaning and choosing a specification, where silent errors hide even better.
The models differ a lot, too. The drop in silent failures comes almost entirely from Claude, from 4.7 to 0.8 per 100 runs. GPT-5.6 Luna barely moved, from 5.1 to 4.9. Human code is not clean either. An audit of 110 studies in prestigious journals, cited by the authors, found coding errors in about a quarter of them.
Journals will decide the impact once they start asking for specification checks
The authors stress that simply rerunning code will not reveal silent errors. Code can reproduce the published numbers and still compute something other than intended. The thing to watch is whether journals start asking, beyond replication packages, for an independent check of what the script actually estimates.
The second signal is more practical. Self-repair, meaning at most three revisions after a run, delivered 68% to 73% of the full agent's improvement. Anyone still copying scripts out of a chat window does not need an autonomous agent right away. Letting the model see its own error message is enough to get most of the gain. That is exactly the point of Mollick's jab at academics who still think of AI as a chatbot.
Lilith's verdict
Stata macros written late at night are no longer a ticket into academia. The valuable person now is the PhD student who looks at a finished regression and says: that standard error is suspiciously small.