Local eval framework shows prompt engineering is moving toward standard developer habits

Simon Willison, in collaboration with the applied AI research lab Prime Radiant (led by Jesse Vincent), has released smevals. It is a new framework designed for running small evaluation suites against various models, configurations, and prompts. The tool is executed locally via the command line, explicitly separates the model execution from its subsequent automatic grading, and can generate a static HTML report of the results.

From guesswork to measurable control over prompts

Working with prompts and various models has long resembled alchemy—developers tweak a sentence and guess whether the output improved. Smevals brings standard engineering practices to the world of large language models. It establishes a vocabulary (eval, task, config, run, grade) that allows developers to easily define in YAML files what the model should do and how its output should be checked. This can range from verifying the presence of a specific string to validating generated XML code.

A framework for solo developers hits complexity walls

While large tech companies build their own massive evaluation systems, smevals targets independent developers and small teams. Its greatest strength lies in its simplicity and local execution. However, the vulnerable point remains the creation of meaningful tests (checks). If a developer cannot accurately and deterministically define what a "correct" answer is, even the best local framework won't help. Grading generated code or formats is straightforward, but semantic text checking will still require using other, often imprecise models as judges.

Suite quality will decide practical utility

The success of such local tools will not depend on how many models they can call via API, but on the community's ability to share ready-made, useful evaluation suites. The proof of maturity will be when YAML definitions for basic tasks (like data formatting or extraction) become a standard part of repositories alongside traditional unit tests.

Lilith's verdict

This is no longer about finding the right magic words. It is the moment developers started treating models like untrusted third-party libraries that need a rigid testing harness.