AI content consent — who may read, train and act

One web, different purposes

It used to be enough, roughly, to decide whether a page could be indexed by a search engine. With AI, that is too crude. The same text can appear in a search result, a training dataset, a RAG answer, an agent that browses on behalf of a user, or a product that turns the original content into a substitute.

Content consent is therefore not one switch. A publisher may want search visibility while rejecting model training. A creator may allow previews and quotation but not mass extraction of an archive. A company may allow an assistant to read documentation but not customer data.

Here, “consent” means expressed preferences and permission to use content, not a claim that every use requires individual approval. For example, Article 4 of EU Directive 2019/790 provides a conditional text-and-data-mining exception and a mechanism for reserving rights. A technical signal alone does not resolve the legal question. Directive, Article 4.

Robots.txt is not enough for AI economics

Robots.txt tells cooperating crawlers which paths they may visit. It does not enforce access or grant a content licence; RFC 9309 explicitly says it is not a substitute for authentication. Separating purposes also depends on each operator's bot identities and policies. Robots Exclusion Protocol.

Anthropic provides a concrete example: it distinguishes ClaudeBot for potential training data, Claude-SearchBot for search, and Claude-User for user-directed requests. Their rules can be configured separately; the same names and behaviour cannot be assumed for another provider. Crawler documentation.

Technical signals, contracts and actual access

A practical setup distinguishes technical preferences, licensing terms and actual access control. Product teams need to know which content they can use, for what purpose and on what basis. Recording the source and applicable terms helps explain a decision; the record itself creates no permission.

The weak point is enforcement. If a crawler lies about its purpose, mixes search with training, or hides reading behind an ordinary user session, the rule becomes a request. Infrastructure matters too: CDN controls, access logs, bot identities and the ability to block by behavior, not only by name.

Why this matters for products with agents

Agentic browsers and assistants make the problem sharper. The issue is not only that a model reads content. An agent can browse under a user identity, summarize paid material, copy creators' work, or automate steps that the original website designed for a human.

For product teams, the discipline is simple: separate reading, quoting, training and action. Each layer should have its own permissions, logs and off switch. Without that, consent dissolves into fog, and the dispute starts only after the content has already been used.

Example: public help and a customer portal

A company wants search engines to find its public help pages, while contractual documents are available only to signed-in customers. It configures bot preferences for the help pages and protects the portal with authentication and account-level permissions. When a customer's assistant summarises a document, the application checks access to that document; signing in does not automatically permit exporting the whole archive or changing contracts.

Separating purposes may also affect search visibility. Blocking future collection does not automatically remove previously used content from existing models. First decide what behaviour you need, then verify what the particular service supports.

Sources

• RFC 9309 — robots.txt rules and security limitations.
• Anthropic crawler documentation — distinct purposes of individual bots.
• EU Directive 2019/790, Article 4 — text and data mining and reservation of rights.