Lilith.
⌕
Editorial illustration: Runtime-specific LoRA raised coding scores while cutting agent work by half
Lilith illustration · editorial remix

Hugging Face trained three LoRA adapters for Qwen3.8-27B, each inside a different agent runtime: Pi, Hermes Agent and OpenCode. Every specialist beat the base model in its own environment on Terminal-Bench 2.1 and the hard subset of SWE-bench Pro.

Training happened inside the runtime used at deployment

The authors used RL on verifiable coding tasks with the real, unmodified agent loops. The dataset contained 871 accepted tasks from 35 repositories. They selected 171 tasks for GRPO that the base model solved only some of the time, providing a useful learning signal.

On SWE-bench Pro, Pi rose from 39.2% to 64.7%, Hermes from 29.4% to 43.1% and OpenCode from 35.3% to 51.0%. On Terminal-Bench 2.1, Pi gained 16.3 points, Hermes 14.1 and OpenCode 6.0. On tasks solved by both variants, the adapters made 53 to 72% fewer tool calls.

The agent runtime is becoming part of the model

The result shows that agent performance depends on more than base weights. History format, tool handling and decision cadence in a particular runtime also matter. An adapter learns that interaction and can remove expensive detours.

Hugging Face also merged the three updates into one rank 96 super-adapter. With thinking off, mean pass@1 across six runtime and benchmark pairs increased from 39.5% to 49.0%. Specialists still performed better in several cases, especially under Pi.

The benchmark resembles the world used for training

The authors warn against comparing absolute scores across runtimes. Pi, Hermes and OpenCode handle prior reasoning and context differently. The super-adapter weight was also selected on Terminal-Bench 2.1, making its numbers there slightly optimistic.

The advantage nearly disappeared outside coding. Across 300 AutomationBench tasks, the base scored 23.0%, the merged adapter 23.7%, Pi 27.0%, OpenCode 25.3% and Hermes 17.7%. A runtime adapter should not be assumed to improve general capabilities.

Private task sets and total cost will decide deployment

Teams need to compare an adapter with the base model on their repositories, in the same runtime and under identical limits. Alongside pass@1, they should measure total tokens, tool calls, latency, price and regressions outside the training mix.

The recipe, datasets, adapters and trajectories are public, so the experiment can be repeated. The next meaningful signal will come from independent reruns on other codebases and tests of whether tool-call savings survive routine prompt and runtime updates.

Lilith's verdict

This adapter is not teaching the model more programming. It is teaching the driver to stop fighting the wheel in one particular cockpit, and the bill for needless turns falls with it.