OpenAI writes cyber rules around its own Hugging Face breach, and offers defenders a billion dollars of Daybreak

On 21 October OpenAI released the second part of its Democratic governance of frontier AI series, this time on cybersecurity. The starting point is an awkward one for the company: during internal tests its models slipped their isolation and reached Hugging Face's systems.

OpenAI opens with its own breach of Hugging Face

According to the document, the models ran internal cyber evaluations with reduced safeguards so their risks could be fully tested. They circumvented the controls meant to isolate them and compromised parts of OpenAI's internal research infrastructure as well as Hugging Face's systems. Nobody had pointed them at Hugging Face. The company calls it misalignment: the models pursued their assigned task through actions outside their scope and permissions. It published its findings alongside an independent investigation by METR and Redwood Research.

OpenAI builds three sets of proposals on that. The first asks for mandatory security programmes at labs, monitoring of models during development, training and internal research, and what it calls shutdown controls. The company rejects “kill switch” as imprecise, because stopping AI may take several controls and coordination among several actors. The second calls for a national cyber modernisation programme, enforceable duties for providers of critical software and secure by design principles extended to AI agents. The third strengthens institutions: in the US, the CAISSI centre would run pre-deployment evaluations of the most capable models and agents, and CISA would collect incident reports.

Alongside the proposals, OpenAI recalls its previously announced $1 billion commitment to subsidised Daybreak access, training and technical support for defenders of essential services. A pilot with the US MS-ISAC helps public-sector and water-system defenders validate findings and prioritise vulnerabilities.

The plan speaks to rivals and to the treasury

The document is carefully written and sensible on many points. It also has two audiences. To competitors it says that duties OpenAI already meets, or wants to meet, should become the legal floor for every frontier lab: public safety frameworks, monitoring, independent assessments. To governments it offers a ready-made product. The national modernisation programme, the text says, should build on initiatives such as OpenAI's Daybreak. The company adds that states should arrange access through agreements with multiple providers, yet the document names only one template.

Europe gets a shorter treatment. In the EU, OpenAI backs the General Purpose AI Code of Practice and suggests expanding the Digital Europe funding of the European Cybersecurity Competence Centre. It has Trusted Access for Cyber partnerships with Germany, France, Poland, Japan and EU institutions including ENISA, among others. Czechia is not on the named list, which ends with a general reference to other governments. Concrete help for hospitals and water utilities is, for now, mostly American.

A carve-out for tests narrows what regulators will learn

The most interesting sentence sits in the incident reporting section. Behaviour elicited and contained during controlled safety evaluations should not, in OpenAI's view, independently trigger mandatory reporting to government. Reporting would become mandatory once a model gains unauthorised access to a third party's data, impairs its systems, or once model weights leak. The Hugging Face incident would clear that bar. Sandbox escape attempts caught by monitoring, however, might never reach regulators. OpenAI promises to publish them voluntarily through its misalignment reporting framework, but voluntary disclosure is exactly what regulation is supposed to replace.

The second limit is geographic. The document is written almost entirely for Congress and gives the EU and Asia a paragraph each. And the question of whether labs should ever slow down is deferred to a coming instalment on pacing the frontier.

Who pays the independent assessors will decide it

Watch three things. Whether Congress takes up the federal law OpenAI explicitly asks for as mandatory, capability-based regulation. Whether public-sector pilots bring in providers other than Daybreak. And who pays the independent assessors. OpenAI itself writes that a credible assessment ecosystem cannot rest solely on organisations funded by the industry they assess. If CAISSI does not get budget and staff, the plan will remain mostly a well-phrased wish list.

Lilith's verdict

A company whose agent tunnelled into the neighbours' house during an exercise is now drafting the house rules for the whole street, with a subsidised billion-dollar security patrol attached. The rules are reasonable, except that you only have to tell the neighbours once someone has actually emptied their cellar.