StateBench shows that a long context window is not reliable memory

Ai2 has released StateBench, a benchmark for tracking changing state over long contexts. The task resembles reading code, a spreadsheet or a story: after many updates, the model must determine where a value, object or character ended up.

Four levels separate value retrieval from genuine state tracking

The benchmark starts with immutable values, moves to replacement by the latest value and then adds updates derived from prior state. At the hardest level, one variable can take another variable's value. The model must follow a chain of dependencies rather than merely find the latest mention.

Ai2 tested Gemini 3.8 Flash, Claude Opus 5.5 and two GPT-6 variants, Sol and Luna. They performed well on the easier levels. Accuracy then fell sharply on long tasks with self referential updates and dependencies between variables. All four reached 13% or lower on the longest tasks at the hardest level.

Agents need to preserve state, not merely accept a long prompt

A large context window describes how many tokens a model can accept. It does not show whether every change will be carried into the final state. That distinction matters for agents working with code, long contracts, customer histories or multistep plans.

The result also exposes a weakness in ordinary retrieval tests. A model may locate one fact in a long text while failing when the correct value results from a sequence of earlier operations. Product evals therefore need to test change over time, not only needle in a haystack retrieval.

Synthetic variables are neither a long contract nor a real agent

StateBench is a controlled benchmark with a fixed number of variables. Real documents continually introduce new people, objects and rules. The authors acknowledge that the current framework does not cover this case. Surface format matters too: small hybrid models were near perfect on the same simple problem in RULER format and near chance when it appeared as spreadsheet edits.

The architecture results are not a ready made product ranking. Gated DeltaNet and hybrid models trained directly on the task handled the first three levels almost perfectly out to 64,000 changes. They received tightly targeted training, an advantage the pretrained models did not have.

Product specific state changes will reveal the practical failure mode

Teams should turn their own workflow into a sequence of concrete changes and inspect the final state after dozens or hundreds of steps. For a coding agent, that may involve renamed symbols and shifting dependencies. For contracts, it may involve amendments that supersede earlier clauses.

The next useful signal is independent replication and evidence that newer hybrid architectures improve state tracking without narrow benchmark training. Only then can a general capability be separated from a well rehearsed trick.

Lilith's verdict

A model can have the whole novel in front of it and still lose the knife passed between characters three chapters earlier. A large context window widens the stage but does not keep track of the props.