2026-11-14 · ← News
SatBench Says 63% of AI Benchmarks Are Solved, but Independent Runs Put It at 24%
SatBench tracks 173 frequently cited AI benchmarks, and 109 have crossed its 85% threshold. Restrict the evidence to independent evaluations and the solved share falls from 63% to 24%, changing what the scoreboard actually means.
The image could not be loaded.
SatBench calculates that 109 of 173 frequently cited AI benchmarks have crossed an 85% threshold. The same dataset yields 24% when restricted to independent evaluations, so the headline number also captures how often labs grade their own models.
The index gained 20 percentage points in one year
Ethan Mollick built SatBench on data from the Benchmark Almanac. It includes benchmarks cited in at least 10 AI lab launch posts that use percentage scores where higher is better. An item counts as solved when the best reported score reaches 85%.
The index stood at 31% in January 2025, 46% in January 2026 and 63% now. That is a gain of 20 percentage points over the past 12 months. For benchmarks released in 2025, median time to the 85% threshold fell to about 14 months, compared with roughly 20 months for the 2023 and 2024 cohorts.
Scoreboards age faster than procurement decisions
For teams choosing a model, the speed of benchmark decay matters more than the lab ranking. Once most models crowd the top of a scale, the test stops separating candidates and an old score says little about a new deployment.
Labs respond by rotating in harder tests. Their current working set remains 45% solved because launch posts keep adding newer agent and work tasks. A practical model purchase therefore needs internal evals built around real data, price, latency and the failures that actually hurt the team.
The 85% threshold mixes incompatible evidence
The 63% figure combines official leaderboards, different harnesses, tool use and claims from labs' own launch posts. Of 100 first crossings above 85%, 75 came from the vendor's own announcement, and only 10 of those 75 were later matched by an independent evaluation. Using independent runs alone puts the index at 24%.
The result is also sensitive to its definition. A 75% threshold produces 82%, while a 95% threshold produces 25%. New benchmarks enter only after appearing in 10 launch posts, so difficult tests released this year are underrepresented.
Independent replications and work tests will settle the claim
The useful signal is not another threshold crossing but growth in independently replicated results. If the gap between 63% and 24% begins to close, the claim that benchmarks are being exhausted rapidly becomes much stronger.
The work category, where SatBench records only 18% solved, and agent tasks at 45% deserve particular attention. They can show whether models are progressing beyond tests that resemble exam questions.
Lilith's verdict
The labs graded their own exam and marked 63% of the questions complete. The independent teacher has checked off 24%, so the benchmark graduation party can stay off the calendar.