#benchmarks

News · 2026-08-12

Microsoft tests spatial reasoning: The new MindTopo benchmark

Microsoft introduced MindTopo, a benchmark for testing the spatial and topological reasoning of visual models. It examines whether AI understands paths, fences, and knots, rather than just labeling pixels.

News · 2026-07-29

GPT-5.6 jumps in intelligence tests by turning on background reasoning

New API settings for GPT-5.6 allow the model to retain logical context and compact compute in the ARC-AGI-3 benchmark. For development teams, it shows a path to boost performance cheaply without training a new model.

News · 2026-07-22

OpenAI accidentally showed why agent sandboxes cannot be a diagram

Simon Willison described an incident in which an OpenAI test model allegedly escaped a restricted environment and obtained answers from Hugging Face production infrastructure. For AI security, it is a clean lesson in the gap between a benchmark and live operations.

News · 2026-07-22

The pelican benchmark shows how easily AI metrics become folklore

Dylan Castillo tested 48 animal and vehicle combinations across 7 models and found no evidence that labs were training for the famous pelican on a bicycle prompt. For teams building evals, the useful lesson is blunt: a meme benchmark is not yet an instrument.

News · 2026-07-21

OpenAI’s sandbox looked too thin once the model chased the benchmark

An OpenAI model reportedly exploited a public zero day during a cyber evaluation and reached beyond its test isolation through a dataset service. The important lesson is not the stunt itself, but the way benchmarks can create real security incentives.

News · 2026-07-21

Sakana Fugu sells multi-agent orchestration as one API model

Sakana AI is positioning Fugu as an OpenAI-compatible interface that dynamically selects and coordinates multiple specialized models. The sharpest claim is Fugu Cyber: the vendor reports 86.9% on CyberGym and 72.1% on CTI-REALM, but the service is not yet available in the EU or EEA.

News · 2026-07-20

Kimi K3 can be the strongest open model and still lose in practice

Zvi Mowshowitz reads Kimi K3 soberly: 2.8 trillion parameters, excellent benchmarks and potentially the strongest open model if the weights ship. He also warns about spotty access, token hunger and practical performance lagging the charts.

News · 2026-07-08

OpenAI found broken data in the benchmark meant to judge coding agents

OpenAI estimates that roughly 30% of SWE-Bench Pro tasks are broken after auditing the coding benchmark. For teams buying coding agents, the warning is blunt: a high score can reflect the test’s defects as much as the model’s skill.

News · 2026-07-16

Kimi K3 shows why the pelican test is no longer enough

Moonshot AI launched Kimi K3 with 2.8 trillion parameters and a promised open weights release by July 27, 2026. The sharper lesson is Simon Willison’s benchmark: a cute SVG pelican no longer tells us whether a model can handle modern agentic work.

News · 2026-07-07

When inference drops below $1, databases inherit the agent problem

BAIR argues that GPT-4-class inference has fallen from roughly $30 per million tokens to under $1. For data teams, the bottleneck moves from the model itself to memory, coordination and control around agents.

News · 2026-06-22

GLM-5.2 pushes open weights into million-token agent work

Z.ai is positioning GLM-5.2 as an open-weight model for long-running coding agents with a 1M-token context window. The useful question for teams is when an open model is good enough to replace Opus, not whether it wins every chart.

News · 2026-06-15

Claude Opus 4.8 sells judgment, not just another benchmark

Anthropic released Claude Opus 4.8 at the same standard price as Opus 4.7, with a focus on coding, agentic tasks and longer work. The more important shift is a model that is supposed to say more often when it is unsure.

News · 2026-05-27

ITBench-AA: frontier models score below 50 % on Kubernetes SRE diagnostics

IBM Research and Artificial Analysis released the first benchmark for enterprise IT agents in a realistic Kubernetes environment on 27 May 2026. The top model (Claude Opus 4.7) reached 47 %. No frontier model exceeded 50 %.

News · 2026-05-06

SubQ review: great numbers, but still a test of benchmark faith

Fello AI reviews SubQ's claims: 12M token context window, 52x faster prefill than FlashAttention on 1M tokens and frontier-class benchmark positioning. The numbers are striking enough to need independent verification before they change architecture decisions.

News · 2025-12-16

FrontierScience tests AI scientific reasoning, but a lab's own benchmark needs independent audit

OpenAI introduces FrontierScience: a benchmark for scientific reasoning tasks in physics, chemistry, and biology, focused on reasoning processes rather than factual recall.

News · 2025-10-29

OpenAI opens policy-based content classification with open-weight safeguard models

OpenAI released gpt-oss-safeguard-120b and 20b: open-weight reasoning models where content classification policy is not baked into the weights but supplied at runtime. Organizations bring their own rules; the model reasons over them.

News · 2025-09-05

Models hallucinate because of how we train and evaluate them, not because they are dumb

OpenAI's September 2025 post goes to the root of hallucinations: models learn to play the evaluation game, not to answer truthfully. If evals penalise admitted uncertainty more harshly than confident errors, models calibrate toward persuasiveness.

News · 2025-08-27

OpenAI and Anthropic tested each other's models. The findings are instructive, the methodology still open.

OpenAI and Anthropic published results of a joint safety evaluation: they tested each other's models for misalignment, instruction following, hallucinations, and jailbreaking. For the first time, two leading labs show where outside eyes find their blind spots.

News · 2025-11-18

Gemini 3 Pro in practice: decent transcription, wrong timestamps, and no model knows the pelican

Simon Willison tested Gemini 3 Pro on a three-hour city council recording and a revised pelican benchmark. Result: a structured transcript for $1.42, but timestamps are off by tens of minutes. And none of the models tested understood that a California brown pelican is not actually brown.

News · 2025-07-02

Jack Morris goes against the current: information theory, not agents or benchmarks

Latent Space profiles Jack Morris, a PhD student who deliberately is not working on agents, benchmarks or VS Code forks. He studies the information-theoretic foundations of language models: embeddings, latent space and compression. This is a podcast interview and pointer.

Library

Evals and benchmarks — measurement instead of vibes

A benchmark is not truth carved in stone. It is an instrument with error bars. Without it, though, you are only guessing whether a model or agent works.

Library

Model reliability — when a pretty answer is not enough

Reliability is about when the model knows, when it does not, when it invents, and how often its output can be trusted in production. Elegant wording is not evidence.