Experts Call for Independent Frontier Model Audits

The End of Internal Safety Checks

More than a hundred industry experts and researchers have signed an open letter organized by the AI Evaluator Forum. The main demand is clear: developers of frontier AI models, such as OpenAI, Anthropic, or xAI, should mandatorily integrate so-called "embedded evaluators", independent third parties with full access to the training process, models, and incident data.

Why Internal Audits Fall Short

The current paradigm, where companies assess the safety of their own models (or do so through commercial partners), hits credibility limits. Under the proposed AEF-1 standard, independent auditors would need to meet strict criteria: no financial conflicts of interest, transparency in funding, and the right to publicly publish their findings. For development teams, this would mean letting external oversight straight into the kitchen before a model is finished.

The Auditor Capacity Problem

The biggest hurdle, however, might not end up being corporate reluctance. As some signatories (like Nathan Lambert) point out, the real bottleneck is a lack of personnel. Finding auditors who possess both the technical depth to understand cutting-edge architectures and absolute independence from major tech players is almost impossible in the current ecosystem.

Will Companies Accept the New Rules of the Game?

The key signal will be whether major market players (OpenAI, Anthropic, Google) actually adopt the AEF-1 principles or merely pay them lip service. The real proof of change won't be a signed memorandum, but the first instance where an independent auditor delays the release of a new flagship model based on their findings.

Lilith's verdict

The idea is great, but it crashes into market reality. Anyone with the skills to audit Claude 3.5 Sonnet is probably already working on Gemini.