IBM gave GPT-4.1 a cheat sheet built from its own mistakes, and the agent beat frontier models on τ²-bench retail

IBM Research shows that a customer service agent can improve without a new model and without fine-tuning. Its ALTK-Evolve tool had GPT-4.1 work through training tasks and distil short rules from its successes and failures, then appended them to the system prompt. With this cheat sheet, the agent reached 67.5% Pass^4 on the retail part of τ²-bench and, according to IBM, beat every entry on the October 12, 2026 leaderboard snapshot.

GPT-4.1 wrote itself 292 rules and passed all four attempts on 67.5% of tasks

τ²-bench, from Sierra, simulates customer conversations. The agent has tools, such as looking up an order or issuing a refund, and must follow a written company policy. A task counts only if the final database state matches the expected outcome exactly. Pass^4 means the agent solved the task in all four independent attempts.

The rules are a plain markdown list. For example: authenticate the customer before touching an order. Before any change, summarise what you will do and wait for explicit approval. The rules are extracted by the same model that then serves customers. No human annotation is involved and the model weights do not change.

On held-out test tasks, retail Pass^4 rose from 57.5% to 67.5% with the full set of 292 rules. Airline went from 35% to 45%, but there the best result came from only the 14 most frequent rules. On the full leaderboard split, the retail agent also beat RAFT-30B-A3B, a model fine-tuned on synthetic τ²-bench data, which scores 61.4%.

Teams can improve an agent by editing the prompt instead of swapping the model

For teams running agents on support lines, this is practical news. Instead of waiting for a new model or paying for fine-tuning, they can collect conversation logs and let the model extract operating rules from them. The rules are readable. A support lead can review them, strike nonsense and add what is missing. Fine-tuning offers nothing like that.

The cost rises where you would expect. The longer prompt pushed retail cost per conversation from $0.084 to $0.139, or 65%. Conversations actually got slightly shorter, by 0.3 to 0.4 turns. The rule block is identical on every call, so prompt caching recovers much of the cost in production. It also worked on a stronger model: GPT-5.5 improved airline Pass^4 from 55% to 70% and reached 82% on the leaderboard, against 66% to 72% for frontier entries.

Small test sets and a leaderboard where the agent partly saw its textbook

The test sets are small: 40 retail tasks and 20 airline tasks. In airline, one task moves Pass^4 by 5 percentage points, so a 10-point gain means two extra tasks. The results are interesting, but on such a small sample it is hard to separate a stable improvement from noise.

The second catch is the leaderboard. τ²-bench scores the full set, including the training tasks the agent learned its rules from. IBM says so openly and reports held-out numbers separately. The headline about winning “without a frontier model” only holds for retail. In airline, GPT-4.1 finished eighth and first place took GPT-5.5. GPT-5.5 also played the simulated customer in every run, so the agent was talking to a model, not an angry human.

The question is whether the cheat sheet survives a policy change

The real test happens outside the benchmark. Company policies change, promotions end and return windows move. When policy changes, the rules have to be relearned, or the agent will confidently apply yesterday's procedures. It will be telling whether anyone deploys ALTK-Evolve on real support and reports how quickly the cheat sheet can be updated.

The second signal is whether the official τ²-bench leaderboard accepts the results. IBM submitted them as pull requests #542 and #551. The ALTK-Evolve library is on GitHub, so anyone with API access can check the claims.

Lilith's verdict

IBM slipped a 292-rule cheat sheet into an older model's pocket, and it outran newer colleagues on the customer service line. But part of the exam came from the very questions it copied the sheet from, and on the airline desk it only won with GPT-5.5.