Four AI Companies Back One Signal to Opt Out of Training Scrapes

OpenAI, Google, Microsoft and Anthropic have backed a shared policy token in robots.txt. A publisher can use one signal to refuse AI training use instead of configuring a separate rule for each company's crawler.

One token aims to replace a blocklist for every crawler

Kevin Kelley of Anthropic proposed the mechanism through work at the Internet Engineering Task Force. The token lives in robots.txt, the file websites have used for decades to communicate access rules to crawlers. The difference is that the preference describes a purpose for content use rather than only naming a particular bot.

All four companies say their crawlers will honor the signal. Anthropic has already implemented it. OpenAI says support will arrive in the coming months, while Google and Microsoft point to next year. Meta voiced support for the proposal but did not commit to honoring it. A common standard therefore does not yet mean a common launch date.

Microsoft also promises separate controls for AI training and grounding. A website should remain in ordinary Bing Search results regardless of that preference. Google already separates some AI uses from conventional search through related controls.

Publishers can finally separate discovery from training

The previous choice was often too blunt. A website wanted search traffic but did not want to supply text to a training corpus. When one mixed-use crawler served several purposes, blocking one use could jeopardize the other. A shared signal gives publishers a clearer switch and crawler operators one consistent place to read it.

Cloudflare said in September that mixed-use crawlers represented 36.6% of verified crawler traffic on its network. Fewer than 1% of sites blocked search crawlers, while 17% restricted AI training. The gap explains the demand: website owners do not want to disappear from the web, but they do want to separate different uses of the same page.

For product and legal teams, the immediate benefit is operational. One preference is easier to maintain than a growing list of bot names that may change. A standard also creates a clearer record of what a site communicated to a crawler operator.

Robots.txt expresses a preference but cannot enforce it

The policy token does not technically stop a crawler. It works only when an operator recognizes and honors it. Commitments from four large companies give the signal weight, but they do not solve unidentified scrapers, renamed bots or disputes over data collected before the preference existed.

The boundaries between training, grounding, an AI search answer and an agent fetching a page for a user also remain unsettled. Microsoft promises granular options, but websites will need equally precise implementations from other providers. One token can standardize vocabulary without standardizing every business model.

Google, Microsoft and OpenAI implementations will be the test

The first measurable signal will arrive when the other three companies publish documentation, dates and testable crawler behavior. Site operators need to verify that refusing training does not remove conventional search visibility and that companies do not interpret the same preference differently.

The second test will come from smaller operators and infrastructure providers such as Cloudflare. The standard gains practical force only when hosts expose it as a routine setting and crawlers beyond the four large companies explain how they handle it.

Lilith's verdict

Four large companies have finally agreed to read the same sign on a website's door. The real test comes when the most valuable training material sits behind it.