Judging an agent by its final answer no longer works. A Princeton-led paper maps 33 trends in agent evaluation

Arvind Narayanan and Sayash Kapoor's group at Princeton, together with co-authors from Stanford, the UK AI Security Institute and Brookings, has published Emerging Trends in AI Agent Evaluation. The paper lays out 33 trends and gives evaluators, funders and the public a set of questions to ask whenever someone shows them an agent's score.

Benchmark scores say less and less about agents

The authors build on roughly three years of their own work on agent evaluation. They sort the trends into three dimensions: science and methods, the research agenda, and the ecosystem of people and institutions doing the evaluating.

The core problem is stated plainly. With a model, you could compare answers. With an agent, the evaluator has to scrutinize the entire trajectory: which tools it called, whether it took shortcuts, and which component actually deserves credit or blame. Evaluating some agents on the ExploitBench benchmark cost tens of thousands of dollars in credits, and that figure excludes the engineering work needed to build and maintain the evaluation infrastructure.

Agents also produce so many traces that humans cannot read them all. When evaluators investigated the OpenAI and Hugging Face incident after the fact, they used other agents to find harmful actions, and acknowledged that this approach has its own limits and biases.

Lab results do not carry over to a company on their own

The paper keeps returning to one point: evidence at one layer does not automatically hold at the next. An agent can handle a task in the lab and stall in a specific workplace. The authors cite METR's randomized controlled trial, in which experienced open-source developers using AI tools took 19% longer to finish tasks, despite expecting a speedup.

For a company buying agents, the practical lesson is simple. A leaderboard number is a starting point. If you want to know whether an agent saves your people time, you have to measure it inside your own process. The authors add a framing choice: evaluators can ask whether an agent automates a task or whether it helps a human do it better. If most evaluation asks the first question, we learn who can be replaced, but not how people can work better with agents.

Evaluator independence starts with money and access

The sharpest part of the paper is about who gets to evaluate at all. As frontier agent evaluation gets more expensive, the budget of independent teams decides whether they can take part. Model access matters too. Labs now release summarized chain of thought rather than raw traces, so outside evaluators see less than internal teams. The authors also mention fear of retaliation and the question of how findings can be shared openly.

This is also where the paper is weakest. Its recommendations for labs, funders, leaderboard operators and policymakers are framed as opportunities and needs. Nothing in it forces a lab to open raw traces to outside evaluators. Readers expecting a new benchmark or fresh measurements will get a map of the terrain instead.

Priced leaderboards and raw traces will show whether the field moves

Leaderboards are the first signal. The authors want aggregators to help the public understand the rising cost of evaluation. If a per-run price starts appearing next to the score, buyers will see what it costs to check a vendor's claim. The second signal is whether outside teams get access to raw agent trajectories.

The third test is in the public sector. The authors want evaluation built into government deployments from day one, and they want AI Security Institutes to bring new participants into the process. If a year from now agents are still mostly evaluated by the people selling them, the paper will remain a well-written warning.

Lilith's verdict

Agents today sit exams that their own makers often wrote and paid for, while outside examiners only get to see a summarized copy of their notebook. Until that changes, read every score like a report card signed by the parents.