Anthropic's draft AI law lets only a judge stop a model, and only if 50+ lives are at risk

Anthropic is shopping its own federal AI safety bill around US lawmakers. According to Politico, whose reporting TechCrunch builds on, a model's release could be blocked if more than 50 people could die or damage could exceed $1 billion. Only a court order could stop the model, though. The full bill text is not public, so this piece relies on what the press has quoted from it.

Only a court can stop a model, after a paid evaluator raises the alarm

The draft centres on a “foreseeable, imminent risk”. The catastrophe must stem from the creation or release of a biological weapon, from the model evading its developer's controls, or from a cyberattack. The thresholds of 50 deaths and $1 billion in damage resemble California's SB 53, which uses them to define catastrophic risk.

Risk would be assessed by independent evaluators registered with the US Commerce Department. The labs themselves could pay them, as long as payment is not contingent on the results. When an evaluator finds an imminent catastrophic risk, it has 72 hours to notify Commerce. The department would then have to ask a Washington, D.C. judge to halt the model's release. According to Politico, the bill expressly forbids any federal agency from halting a model's distribution at its own discretion.

An Anthropic spokesperson confirmed the draft to Politico, saying it turns into legislation the Advanced AI Framework the company has been advocating with federal and state lawmakers since June.

A lab is drafting the rules the government will use to check it

A leaked version surfaced last week, and critics then focused on how much power it would take away from the states. Now the text is confirmed and attention shifts to the mechanics of intervention. Every step has its own gatekeeper: the evaluator must find the risk, the department must file a request, and a judge must grant it. The fast route, where an agency pauses a model on its own, is closed.

For labs, this is a predictable regime. They know which scenarios count, who may assess them and that a court decides on intervention. Part of the industry takes a harder line: Nvidia CEO Jensen Huang recently argued that AI needs no new laws at all. Anthropic's proposal goes further than most industry offers. It also defines exactly which route government oversight is allowed to take.

An evaluator paid by the lab and a short list of catastrophes

Independence is the weakest point. A rule that payment cannot depend on the outcome does not address the fact that the lab chooses the evaluator and may not hire it next time. TechCrunch notes that the draft does not explain how it would keep money from influencing evaluations. A similar debate already surrounds the evaluators Anthropic and OpenAI want to embed inside their own companies.

The second limit is scope. Bioweapons, loss of control and cyberattacks are serious scenarios, but an “imminent” risk with 50 deaths only captures extreme cases. The agentic hacking and models ignoring their operators that the industry has struggled with this year would struggle to clear that bar. And 72 hours to report, plus a court process, is a long time if the risk really is imminent.

What decides it is who files the bill in Congress and what survives

The first signal is a sponsor. Until a senator or representative introduces it, this is one company's lobbying document. The second is whether tougher rules on evaluator independence make it in, such as evaluators assigned by the government rather than picked by labs.

Watch how other labs respond too. If OpenAI or Google sign on, Anthropic's draft becomes the industry's opening position. If they reject it, it will mostly show how one company imagines oversight that will not slow it down.

Lilith's verdict

Only someone the lab pays may call the fire brigade, and only a Washington judge can turn on the hose. For a fire the bill expects to kill more than 50 people, that is a long corridor.