Which AI opt-out signals do AI companies say they honor?
For each AI and search operator in the BOZOS crawler directory, whether its own documentation says it honors each machine-readable opt-out or preference signal. honors cells quote the operator. not-stated means the operator's pages listed in searched do not mention the signal at all; it is not evidence that the signal is ignored. Observed behavior is out of scope here; see the crawler directory's docs_vs_reports.
Record what the operator states, scoped to the pages checked. A signal's own specification is quoted for what it is, never as proof that anyone honors it.
Quick answers
- How do I stop AI companies training on my site?
- The mechanism operators actually document is a robots.txt rule for their training crawler or control token. 8 of 14 operators say so: OpenAI, Anthropic, Google, Apple, Meta, Common Crawl, Amazon, Mistral AI. Get that robots.txt →
- Does anyone honor noai, Content-Signal, TDMRep, the IETF draft, or RSL?
- None of the 12 operators whose pages could be read mentions any of them. That does not prove they are ignored, only that no operator documents honoring them. They can still express your preference, and TDMRep and Content-Signal are framed as reservations of rights under EU law.
- Do robots meta tags affect AI?
- For some operators, in specific ways: Google (nosnippet also keeps a page from being used as direct input for AI Overviews and AI Mode. It does not address training); Apple (nosnippet keeps content out of AI-generated answers in Apple products. Training is controlled by Applebot-Extended instead); Amazon (noarchive means do not use the page for model training).
- Which operators say they do not train on crawled content?
- Perplexity, DuckDuckGo, Diffbot.
Operator by signal.
Yes: the operator's documentation says it honors the signal (see the quote below). —: not mentioned on the pages checked. n/a: the operator says it does not do what the signal controls. ?: the page cannot be read by a script.
| Operator | robots.txt training opt-out | Robots meta | noai | Content-Signal | IETF aipref | TDMRep | RSL |
|---|---|---|---|---|---|---|---|
| OpenAI | Yes | — | — | — | — | — | — |
| Anthropic | Yes | — | — | — | — | — | — |
| Yes | Yes | — | — | — | — | — | |
| Microsoft | ? | ? | ? | ? | ? | ? | ? |
| Apple | Yes | Yes | — | — | — | — | — |
| Perplexity | n/a | — | — | — | — | — | — |
| Meta | Yes | ? | ? | ? | ? | ? | ? |
| Common Crawl | Yes | — | — | — | — | — | — |
| Amazon | Yes | Yes | — | — | — | — | — |
| Mistral AI | Yes | — | — | — | — | — | — |
| DuckDuckGo | n/a | — | — | — | — | — | — |
| Diffbot | n/a | — | — | — | — | — | — |
| You.com | — | — | — | — | — | — | — |
| Allen Institute for AI (Ai2) | — | — | — | — | — | — | — |
robots.txt opt-out from AI training
Whether a named crawler may fetch a path. Operators decide what blocking their token means; some offer a token only for AI training, or a control token that changes how already-crawled data is used.
User-agent: GPTBot
Disallow: /These rules are not a form of access authorization.
Spec ↗
Operators that say they honor it: OpenAI, Anthropic, Google, Apple, Meta, Common Crawl, Amazon, Mistral AI.
Robots meta tags as AI controls (nosnippet, noarchive)
Page-level rules originally written for search. Some operators state that a specific rule also limits AI use of the page; the meaning differs by operator.
<meta name="robots" content="nosnippet"> or X-Robots-Tag: noarchiveOperators that say they honor it: Google, Apple, Amazon.
noai and noimageai directives
That content is not authorized for use in AI datasets or as input to AI models.
<meta name="robots" content="noai, noimageai"> or X-Robots-Tag: noaiDeviantArt has introduced the noai and noimageai directives to unambiguously communicate absence of such authorization.
Spec ↗
Operators that say they honor it: none of the operators checked.
Content-Signal (Content Signals Policy)
Yes/no preferences for three uses: search, ai-input (for example retrieval-augmented generation and grounding), and ai-train (training or fine-tuning).
Content-Signal: search=yes, ai-input=yes, ai-train=noThis policy defines three content signals - search, ai-input, and ai-train - and their relevance to crawlers.
Spec ↗
Operators that say they honor it: none of the operators checked.
IETF AI preferences (Content-Usage)
A shared vocabulary for usage preferences, attached through robots.txt or an HTTP header. Not yet an RFC.
Content-Usage: train-ai=n (robots.txt rule or HTTP header; vocabulary still in draft)This document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields.
Spec ↗
It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
Spec ↗
Operators that say they honor it: none of the operators checked.
TDM Reservation Protocol (TDMRep)
A reservation of text-and-data-mining rights under Article 4 of the EU copyright directive (2019/790), optionally pointing to a licensing policy.
tdm-reservation: 1 (HTTP header), <meta name="tdm-reservation" content="1">, or /.well-known/tdmrep.jsonThis specification defines a simple and practical Web protocol, capable of expressing the reservation of rights relative to text & data mining (TDM) applied to lawfully accessible Web content
Spec ↗
Operators that say they honor it: none of the operators checked.
Really Simple Licensing (RSL)
Machine-readable licensing, payment, and legal terms for AI use, with discovery through robots.txt, HTTP, HTML, or RSS.
License: https://example.com/license.xml (robots.txt), or a Link header with rel="license"machine-readable usage, licensing, payment, and legal terms that govern how digital assets may be accessed or licensed by AI systems and automated agents.
Spec ↗
Operators that say they honor it: none of the operators checked.
What each operator says.
OpenAI
- robots.txt training opt-out: Honors. Disallow GPTBot in robots.txt.
Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
Anthropic
- robots.txt training opt-out: Honors. Disallow ClaudeBot in robots.txt.
Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt.
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
- robots.txt training opt-out: Honors. Disallow Google-Extended in robots.txt.
Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models
Source ↗ - Robots meta: Honors. nosnippet also keeps a page from being used as direct input for AI Overviews and AI Mode. It does not address training.
will also prevent the content from being used as a direct input for AI Overviews and AI Mode.
Source ↗ - Not stated: noai, Content-Signal, IETF aipref, TDMRep, RSL.
Microsoft
- Documentation that cannot be read by a script: www.bing.com ↗
- Not stated: none.
Apple
- robots.txt training opt-out: Honors. Disallow Applebot-Extended in robots.txt.
Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file.
Source ↗ - Robots meta: Honors. nosnippet keeps content out of AI-generated answers in Apple products. Training is controlled by Applebot-Extended instead.
Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services.
Source ↗ - Not stated: noai, Content-Signal, IETF aipref, TDMRep, RSL.
Perplexity
- robots.txt training opt-out: Not applicable. The operator says its crawlers are not used for AI training.
It is not used to crawl content for AI foundation models.
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
Meta
- robots.txt training opt-out: Honors. Disallow meta-externalagent in robots.txt.
- Documentation that cannot be read by a script: developers.facebook.com ↗
- Not stated: none.
Common Crawl
- robots.txt training opt-out: Honors. Disallow CCBot in robots.txt.
To prevent Common Crawl from crawling your website, include the following in your robots.txt
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
Amazon
- robots.txt training opt-out: Honors. Disallow Amazonbot in robots.txt.
may be used to train Amazon AI models.
Source ↗ - Robots meta: Honors. noarchive means do not use the page for model training.
page level robots meta tags of noarchive (do not use the page for model training)
Source ↗ - Not stated: noai, Content-Signal, IETF aipref, TDMRep, RSL.
Mistral AI
- robots.txt training opt-out: Honors. Disallow MistralAI-Training in robots.txt.
Webmasters can disallow this user agent in their robots.txt file.
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
DuckDuckGo
- robots.txt training opt-out: Not applicable. The operator says its crawlers are not used for AI training.
This data is not used in any way to train AI models.
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
Diffbot
- robots.txt training opt-out: Not applicable. The operator says its crawlers are not used for AI training.
It is not used for AI training.
Source ↗ - Not stated: Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
You.com
- Not stated: robots.txt training opt-out, Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
Allen Institute for AI (Ai2)
- Not stated: robots.txt training opt-out, Robots meta, noai, Content-Signal, IETF aipref, TDMRep, RSL.
Quotes and searches you can re-run.
Every Yes quotes the operator's own page, and a script checks that each quote still appears there. Every not-stated cell lists the pages and search terms used, and the same script fails if one of those pages starts mentioning the signal, so a new statement is caught at the next review.
Not included: ai.txt (Spawning) (The publisher's site was under maintenance on 2026-10-20, so its specification could not be read and quoted.) llms.txt (A guide for language models, not an opt-out or preference signal.)
To test a robots.txt against every documented crawler, use the robots.txt checker. Found a statement we missed? Tell us.