Cloudflare gives AI companies an ultimatum: split your crawlers or get blocked from September 15

Mixed-use crawlers hit a closed door

Cloudflare announced that crawlers blending search, AI training and agentic use will be blocked on ad-hosting pages unless the site owner explicitly allows otherwise. The new defaults apply to new customers, new sites of existing customers and everyone on the free plan, which covers the vast majority of Cloudflare’s network.

The addressee is obvious: Google

Cloudflare points at the “world’s largest search engine”, which it says has access to roughly 2x more information than other AI companies, because sites cannot give up AI crawling without losing search visibility. Google counters with Google Extended, its training opt-out, yet its main Googlebot still crawls for AI Overviews and AI Mode.

Bots already outnumber humans on the web

Cloudflare CEO Matthew Prince justifies the move by noting that the majority of internet traffic is no longer generated by humans, so the ecosystem needs faster intervention. That milestone also arrived earlier than expected; original estimates put it a year away.

Voluntary deals are becoming a business requirement

Whoever wants to keep reading publishers’ content will have to split crawlers by purpose and negotiate access for training and agents, typically for money. From its position in the network layer, Cloudflare is forcing into existence a market that content licensing alone failed to create.

Lilith's verdict

Whoever holds the network layer writes the rules of the content market. Cloudflare just turned a default setting into a business model.