Change a few words in the prompt and an AI novelty judge drops from 93.5% to worse than a coin flip

Ai2 has released a benchmark that tests how reliably language models judge whether a research idea is new. The result is awkward for anyone building an “AI scientist”: the same model, looking at the same ideas, flips its verdict depending on a few words in the instructions, and costly purpose-built systems lose to a simple prompt.

The benchmark rests on ICLR 2026 reviews, and acceptance alone does not count

On 27 October Ai2 published the study Old Ideas, Novel Problems together with data, code and a pipeline for building new test sets. It targets so-called novelty judges, language models that decide whether an idea adds something beyond prior work. Six models were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.

The set is built from ICLR 2026 peer reviews. The authors mined the reviews for passages in which reviewers explicitly discussed originality. The strong pool holds 154 ideas from accepted papers in the top 10% of their area, where reviewers praised the novelty and nobody disputed it. The weak pool holds 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns.

A second set pairs the same 154 human ideas with 154 ideas from a simple generator that did no literature search. The authors assume the generated ideas are weaker, so these are weak labels. An expert reviewed 30 pairs, mostly cases the best judge got wrong, and preferred the human idea in all 30.

A few words in the instructions move accuracy by more than 50 points

The prompt caused the largest swings. GPT-5.4 saw the same pairs under two instructions. The first told it that reviewers had marked exactly one idea in each pair as novel. The second simply asked which idea was novel. Accuracy fell from 93.5% to 40.9%, below chance, and the model changed its verdict on more than half of the pairs. On human-only ideas the same change moved GPT-5.4 by only about seven points, but Claude Opus 4.6 by 21.

For teams evaluating their own idea generators, another number matters more. When judges compared ideas from Claude Opus 4.5 against Sonnet 4.5 and GPT-5.4 against GPT-5.1, a prompt change flipped a quarter to a third of verdicts on average. A leaderboard of which model proposes more original research therefore largely measures who wrote the judge's prompt.

Spending more did not help. GPT-5.6 Sol with an extra-high reasoning budget and live arXiv search scored 0.78 macro-F1, 11 points behind Claude Opus 4.6 at 0.89, at roughly ten times the cost per decision. A simple Opus 4.6 prompt with majority voting over three calls reached 0.854 for $2.59 across the whole evaluation. Idea Novelty Checker scored 0.642 for $30.66, and the novelty evaluator from Sakana AI's AI Scientist scored 0.772 for $82.05.

The winning prompt got a hint that a real judge never has

The 93.5% figure applies to an instruction that tells the model in advance that each pair contains exactly one novel idea. In real use nobody has that certainty. The more useful reading runs the other way: without the hint, GPT-5.4 lost its bearings on AI-generated ideas.

The benchmark has its own limits. Novelty here is defined by ICLR reviewers, a single machine learning community with its own habits. The idea generator was deliberately primitive, so the set says little about how judges handle agents that search the literature themselves. And the tested models are no longer the newest: most results concern generations such as GPT-5.4 and Opus 4.6.

The test is whether AI scientist builders start publishing their judges' sensitivity

Ai2 is releasing both the set and a pipeline that can be run on reviews from other conferences. Worth watching is whether teams that claim their systems produce new ideas adopt it. A fair result should report, alongside accuracy, how many verdicts flip under a different prompt.

The second signal is whether the finding holds for newer models and outside machine learning. If the next generation of judges still changes its mind with the wording, the claim “our system proposes original research” will keep needing a human reviewer.

Lilith's verdict

Ai2 turned six models into the jury of a science fair and found that rewording a few lines of the invitation sends different ideas to the final. Anyone letting one model invent research and another grade it is running a contest where the winner is mostly decided by whoever wrote the rules.