Lilith.
⌕
Editorial illustration: OpenAI brings four outside teams inside, but their independence ends at the agreed scope
Lilith illustration · editorial remix

OpenAI is launching the first phase of embedded assessments with four outside organizations. Depending on each assignment, teams will receive access to model checkpoints, source code, training and evaluation data, monitoring systems and company staff.

Apollo will spend four months looking for attempts to subvert human control

A six person Apollo Research team will examine whether training incentivizes subversion, whether evaluations can elicit it, whether it occurs during internal deployment and whether relevant reasoning can be detected. Apollo plans two public reports covering methods, conclusions and limitations.

Goodfire will focus on interpretability, training run monitoring and debugging unwanted behavior. METR plans to search for dangerous behavior in a large reinforcement learning run and later assess monitoring. Transluce will review the quality and coverage of post training alignment evaluations, with a broader eight week follow up intended for early 2027.

Continuous access can expose failures that prerelease testing misses

Outside evaluators have typically received models after post training and shortly before release. Working during development lets them track how behavior emerges, compare checkpoints and test whether internal monitors catch the same problem as an independent evaluation. That offers more information than a one time safety grade at the end.

Splitting the work among four organizations also separates distinct questions: what the model does, what happens inside it, what monitoring catches and how good the evaluations are. The public will still see only part of the evidence. OpenAI's Foundation Safety and Security Committee is supposed to receive unredacted reports.

Apollo gets deep access without full control over scope or evidence

Apollo calls the pilot a significant first step that falls short of its three conditions for full independence: access to all relevant material, the right to disclose findings against a developer's wishes and transparency about the restrictions themselves. OpenAI may withhold material for security, technical feasibility or intellectual property reasons. The assessment also covers only agreed claims, models and periods.

Conversations are limited to authorized staff. OpenAI may redact confidential evidence from public materials and request disclosure delays while a safety issue is addressed. Apollo and Goodfire receive compensation that is not contingent on results. METR and Transluce take no payment but may accept compute or API credits.

Apollo's two reports will show whether outsiders can verify independence

The first test arrives with the public reports. They need to identify unavailable records, denied requests and the effect of redactions on conclusions. Apollo secured the right to describe those limits without prior OpenAI review, giving readers at least a map of the blind spots.

A second signal will be whether the scope can expand after an unexpected finding. An evaluator inside the building sees more than a conventional auditor. Its independence becomes real when it tries to open a door the original agreement left out.

Lilith's verdict

An auditor inside the building sees more than a visitor behind glass. Independence is measured by whether they may reach for the handle on a door the host wanted closed.