Artificial Analysis hid its tests, and the open-weights gap jumped from 3 points to 12

Artificial Analysis released Intelligence Index v5 on October 27. For the first time, most of the weight, 60%, sits on evaluations with private test sets or held-out answers. The best open model now trails the leader by 12 points. In v4.3 the gap was 3.

The new index drops GDPval-AA and measures a gap four times wider

The company removed GDPval-AA v2.1, SciCode and AA-LCR. It added FrontierCode 2.0, Cognition's private benchmark of 132 tasks in real open source repositories, scored only when a maintainer would merge the change. The second addition is Terminal-Bench-Science, 70 scientific tasks run in a terminal. Humanity's Last Exam gave way to the refined 1,000-question HLE-Diamond. The index now has four equally weighted parts at 25% each: knowledge work, coding, science and general knowledge.

Claude Opus 5.5 (max) leads with 58, just ahead of GPT-6 Astra at 57 and GPT-6.1 Sol at 55. Anthropic and OpenAI hold the top five, and Gemini 4 Argon scores 50. The best open model, Kimi K3.1 Pro, scores 46, but its weights are not out yet. Among models you can actually download, DeepSeek V4.1 Flash 1017 and GLM-5.3 lead at 32. That is 26 points behind the leader and below the small closed Claude Haiku 5.5 at 36.

Open models fared worst on Terminal-Bench-Science. GPT-6 Astra solves 63% of tasks there, Kimi K3.1 Pro 34% and DeepSeek V4.1 Flash 13%. Most other open models stayed under 10%. Ethan Mollick said on X that the result should be taken with a grain of salt, but that it matches the general sense that no Fable-class open model has appeared yet.

If you counted on a cheap open substitute, recalculate the cost per task

Until recently the line was that open models trail the closed frontier by about three months. Nathan Lambert made that case in a lecture we covered, citing Artificial Analysis data. The new index weakens that comfort: in agentic tasks, where a model has to work through dozens of steps in a terminal, the gap is far larger than on knowledge tests.

For companies weighing self-hosting on cost grounds, another number matters more. GPT-6.1 Sol (max) trails Opus by 4 points and costs $1.27 per index task, against $10.04 for Opus. A cheap closed option takes away the main price argument for open models. Open weights still earn their place where data privacy, offline operation or custom fine-tuning decide. As a cheap copy of the frontier for agentic work, they do not hold up yet.

The index also shows that effort settings matter more than before. Opus 5.5 scores 35 at low effort and 58 at max, at ten times the cost ($0.98 versus $10.04). Median output tokens per task rose from 35k to 59k. Anyone comparing models by price per million tokens sees only half the bill.

A new ruler measures the gap, so part of the jump comes from methodology

Comparing 3 points with 12 compares two different rulers. Versions 4.3 and v5 use different tests, different weights and a different share of agentic work. Moving to private sets also limits cases where a model saw public questions during training. If open models benefited from that more than closed ones, part of the earlier small gap was an illusion. That is a hypothesis: Artificial Analysis did not test it and has not published a side-by-side breakdown of the same models on both versions that would show it.

Private sets have a downside too. Nobody outside can reproduce the results, and the whole picture rests on one company and its Stirrup harness. FrontierCode also belongs to Cognition, which sells a coding agent of its own. Open models are often tuned for other environments, so part of the drop may come from the harness, not the weights themselves.

Kimi K3.1 Pro's weights and a second independent measurement will settle it

The first test comes soon. According to its earlier announcement, Moonshot promised to release the full Kimi K3.1 Pro weights by November 4. Then we will see whether the best open model holds 46 points in other hands and whether the gap to the frontier really settles around 12 points.

The second signal is whether someone else measures a similar drop, for example Epoch AI, whose work Artificial Analysis built on for its new Progress Index. If the next DeepSeek or GLM lifts Terminal-Bench-Science out of single digits, the debate returns to months. If not, companies will buy open models for control over their data, not for performance.

Lilith's verdict

Artificial Analysis locked the exam in a safe, and overnight the open models fell four times further behind the top student. Until someone else walks out with their own set of questions, we cannot tell whether the students got worse or simply knew the questions before.