One robots.txt signal will tell multiple AI companies not to train on a site

OpenAI has announced support for a shared robots.txt signal covering preferences about generative AI training. A website operator would no longer need a separate rule for every participating provider's crawler. OpenAI aims to implement support in the coming months, but has not given a firm date or full implementation details.

OpenAI is unifying the training opt-out while keeping existing controls

The mechanism draws on work by the IETF AI Preferences group. The proposal uses a separate policy token inside the familiar robots.txt structure, allowing sites to express path-specific preferences without changing the file format. It is still an Internet-Draft, not an approved IETF standard.

OpenAI says its existing crawler controls will remain in effect. Site operators can therefore continue to separate model training from visibility in ChatGPT search. The shared signal is meant to reduce rule maintenance, not collapse every use of web content into one switch.

Publishers get less configuration, not an end to the consent dispute

The practical benefit for smaller publishers and website teams is clear: one preference can address several providers and does not need revision whenever a crawler name changes. Google has also announced plans to support the mechanism in 2027, giving it more weight than another vendor-specific label.

The signal mainly solves communication. It does not prove that a provider removed content from training, and it cannot automatically cover copies of the same work on other domains. Creators also do not always control the websites where their work appears.

A public file is neither a lock nor an audit trail

robots.txt depends on voluntary cooperation. A compliant crawler reads the rule, an abusive collector can ignore it, and a website operator cannot inspect a training dataset from the outside. The proposal also treats content-use preferences separately from crawler access controls.

Providers will therefore need clear rules for conflicts and a verifiable process for correcting mistakes. Without that process, publishers receive an easier way to state a preference, not evidence that their material is absent from training data.

Adoption and redress will determine whether the signal matters

The next useful indicators are OpenAI's exact implementation, the number of additional providers and agreement on the token's meaning. Google plans support in 2027. Only broad adoption can turn the draft into a routine part of operating a website.

Dispute handling matters just as much. Publishers need to know where to report an ignored preference, how a provider will investigate and what happens to content that has already been processed.

Lilith's verdict

One robots.txt rule can clear up the crawler thicket. Until publishers can see who honored the sign, it remains a notice on a fence with no camera.