OpenAI found broken data in the benchmark meant to judge coding agents

OpenAI published an audit of SWE-Bench Pro, a benchmark for agentic coding. On its 731-task public split, OpenAI says frontier models improved from a 23.3% pass rate to 80.3% in eight months, which is exactly why the company checked whether the benchmark still measures the right thing.

SWE-Bench Pro was meant to be harder, but the audit found a broken third

OpenAI describes SWE-Bench Pro as a more realistic successor to SWE-bench Verified: longer horizons, tasks from public and private repositories and solutions that must pass new tests without breaking existing behavior.

The audit is uncomfortable. An automated datapoint pipeline flagged 200 tasks, or 27.4%, as broken. Human annotation by five experienced software engineers identified 249 broken tasks, or 34.1%. OpenAI therefore estimates that roughly 30% of SWE-Bench Pro tasks have serious issues.

Buyers of coding agents need to measure work, not dataset mood

For companies deploying Codex, Claude Code or similar tools, this is a practical problem. A benchmark is supposed to separate agent capability from marketing. If hidden tests enforce unstated details, or the prompt omits requirements that tests check, the score starts punishing correct solutions.

OpenAI lists four main failure modes: overly strict tests, underspecified prompts, low-coverage tests and misleading prompts. These are not academic blemishes. They are exactly the faults that can make a buyer overrate one tool or reject a model that would work in production.

Agents can audit the test, but they can also learn the wrong game

The second layer is the audit method. OpenAI used an agent-assisted pipeline, Codex-based investigator agents and then human review. Agents are no longer only the subject of evaluation. They are becoming tools for benchmark quality control.

That helps, but it does not solve the whole problem. If the industry leans too hard on public leaderboards, models and teams learn to optimize around their quirks. A hard benchmark without audit hygiene is just an expensive leaderboard game.

Trust needs a repaired split with traceable fixes

The next signal is whether SWE-Bench Pro gets a repaired split, clearly marked defective tasks and repeatable audit methods. A warning alone is not enough, because buyers need to know which number deserves trust.

For coding agents, local evaluation on a company’s own repository will matter more. A public benchmark can be a map, but the purchase decision has to pass through the actual codebase.

Lilith's verdict

A leaderboard is a shop window where models like to shine. If a third of the price tags are wrong, the careful buyer brings their own scale.