Lilith.
⌕
Editorial illustration: A model spec should define AI behavior. A new framework also calls for tests, versioning, and independent review
Lilith illustration · editorial remix

The Safe AI Forum has published Emerging International Best Practices for AI Model Specs. Authors from 13 organizations describe how companies can write, publish, implement, and evaluate rules for desired model behavior. The aim is to turn a public promise about AI behavior into a document that can guide training, testing, and disputes over specific decisions.

Six components turn values into rules that can be inspected

The framework divides a model spec into 6 components: scope, developer goals, constraints, principles, communication style, and guidance for conflicts between instructions. Scope should identify which models the document covers, how the company evaluates compliance, and whether the same principles apply to internal systems.

The paper draws on public documents from OpenAI, Anthropic, and Alibaba as well as interviews with developers. It describes 4 stages: writing the spec, publishing and updating it, incorporating it into training, and measuring actual behavior. It concludes with 12 recommendations for companies, governments, and research or civil-society organizations.

A public document should connect engineers, lawyers, and outside evaluators

Inside a company, a model spec can serve as a shared reference for engineering, product, legal, and policy teams. It helps distinguish two different failures: a flawed rule and a model that failed to follow a sound rule. The document can also inform training data and safeguards.

For the public, the more important benefit is the ability to compare stated behavior with results. The authors recommend public versioning, clearly marked changes, results broken down by provision and risk, and independent evaluator access to internally deployed models. They also note that Measure 7.1 of the voluntary GPAI Code of Practice safety chapter asks signatories to provide a report covering principles, instruction priorities, refusal topics, and the system prompt for models with systemic risk.

Written rules do not show what the model will actually do

The paper notes that, according to research it cites, leading models may comply with their specs only 70 to 80% of the time. A single aggregate score can also hide the crucial difference between polite wording and a failure during an irreversible action. The authors therefore call for provision-level results, documented exceptions, and limits on evaluation coverage.

There is also a power problem. The company selling the model usually writes the spec, while internal approval processes remain opaque. A public document can reveal intent, but without access to the model, staff, and relevant internal information, an outside evaluator cannot determine what reached training and production.

Public change logs and repeatable evals will determine whether specs matter

The framework will gain weight if companies identify the model versions it covers, flag material changes, and publish evals that independent researchers can repeat. Disclosure of deviations for internal models and records of incidents that changed a rule would provide stronger practical signals.

The next question is the shift from model specs to system specs. Product behavior also depends on system prompts, filters, tools, and permissions. For an agent that can act in another account or spend money, a document describing only the model remains just one part of the map.

Lilith's verdict

A model spec is a timetable still written and inspected by the operator itself. Trust begins when passengers can see delays, route changes, and the independent inspector's report.