# AI Opt-out Signals: Who Honors What Canonical: https://bozos.si/reference/ai-optout-signals Data: https://bozos.si/ai-optout-signals.json Verified: 2026-10-20 ยท License: CC0-1.0 > For each AI and search operator in the BOZOS crawler directory, whether its own documentation says it honors each machine-readable opt-out or preference signal. honors cells quote the operator. not-stated means the operator's pages listed in searched do not mention the signal at all; it is not evidence that the signal is ignored. Observed behavior is out of scope here; see the crawler directory's docs_vs_reports. ## Stances - **honors**: The operator's documentation says it honors the signal, for the scope given. - **not-stated**: The pages checked do not mention the signal. The search terms and pages are listed so the absence can be re-checked. - **not-applicable**: The operator says it does not do the thing the signal controls (for example, it says it does not train on crawled content). - **unknown**: The operator's documentation cannot be read by a script, so neither a statement nor its absence could be verified. ## Grid | Operator | robots-txt-training-optout | robots-meta-ai | noai-meta | content-signal | aipref-content-usage | tdmrep | rsl | | --- | --- | --- | --- | --- | --- | --- | --- | | OpenAI | yes | not stated | not stated | not stated | not stated | not stated | not stated | | Anthropic | yes | not stated | not stated | not stated | not stated | not stated | not stated | | Google | yes | yes | not stated | not stated | not stated | not stated | not stated | | Microsoft | unknown | unknown | unknown | unknown | unknown | unknown | unknown | | Apple | yes | yes | not stated | not stated | not stated | not stated | not stated | | Perplexity | n/a | not stated | not stated | not stated | not stated | not stated | not stated | | Meta | yes | unknown | unknown | unknown | unknown | unknown | unknown | | Common Crawl | yes | not stated | not stated | not stated | not stated | not stated | not stated | | Amazon | yes | yes | not stated | not stated | not stated | not stated | not stated | | Mistral AI | yes | not stated | not stated | not stated | not stated | not stated | not stated | | DuckDuckGo | n/a | not stated | not stated | not stated | not stated | not stated | not stated | | Diffbot | n/a | not stated | not stated | not stated | not stated | not stated | not stated | | You.com | not stated | not stated | not stated | not stated | not stated | not stated | not stated | | Allen Institute for AI (Ai2) | not stated | not stated | not stated | not stated | not stated | not stated | not stated | ## Signals - **robots.txt opt-out from AI training** (ietf-standard; IETF (RFC 9309) plus each operator's own user-agent token). Whether a named crawler may fetch a path. Operators decide what blocking their token means; some offer a token only for AI training, or a control token that changes how already-crawled data is used. Syntax: `User-agent: GPTBot / Disallow: /`. Spec: https://www.rfc-editor.org/rfc/rfc9309.html Spec says: "These rules are not a form of access authorization." - **Robots meta tags as AI controls (nosnippet, noarchive)** (de-facto-standard; Search engines, individually). Page-level rules originally written for search. Some operators state that a specific rule also limits AI use of the page; the meaning differs by operator. Syntax: ` or X-Robots-Tag: noarchive`. Spec: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag - **noai and noimageai directives** (convention; DeviantArt (introduced November 2022)). That content is not authorized for use in AI datasets or as input to AI models. Syntax: ` or X-Robots-Tag: noai`. Spec: https://www.deviantart.com/team/journal/UPDATE-All-Deviations-Are-Opted-Out-of-AI-Datasets-934500371 Spec says: "DeviantArt has introduced the noai and noimageai directives to unambiguously communicate absence of such authorization." - **Content-Signal (Content Signals Policy)** (vendor-policy; Cloudflare). Yes/no preferences for three uses: search, ai-input (for example retrieval-augmented generation and grounding), and ai-train (training or fine-tuning). Syntax: `Content-Signal: search=yes, ai-input=yes, ai-train=no`. Spec: https://blog.cloudflare.com/content-signals-policy/ Spec says: "This policy defines three content signals - search, ai-input, and ai-train - and their relevance to crawlers." - **IETF AI preferences (Content-Usage)** (internet-draft; IETF AIPREF working group). A shared vocabulary for usage preferences, attached through robots.txt or an HTTP header. Not yet an RFC. Syntax: `Content-Usage: train-ai=n (robots.txt rule or HTTP header; vocabulary still in draft)`. Spec: https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ Spec says: "This document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields." Spec says: "It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."" - **TDM Reservation Protocol (TDMRep)** (w3c-community-group-report; W3C TDM Reservation Protocol Community Group). A reservation of text-and-data-mining rights under Article 4 of the EU copyright directive (2019/790), optionally pointing to a licensing policy. Syntax: `tdm-reservation: 1 (HTTP header), , or /.well-known/tdmrep.json`. Spec: https://w3c-cg.github.io/tdm-reservation-protocol/spec/ Spec says: "This specification defines a simple and practical Web protocol, capable of expressing the reservation of rights relative to text & data mining (TDM) applied to lawfully accessible Web content" - **Really Simple Licensing (RSL)** (industry-specification; RSL Collective). Machine-readable licensing, payment, and legal terms for AI use, with discovery through robots.txt, HTTP, HTML, or RSS. Syntax: `License: https://example.com/license.xml (robots.txt), or a Link header with rel="license"`. Spec: https://rslstandard.org/rsl Spec says: "machine-readable usage, licensing, payment, and legal terms that govern how digital assets may be accessed or licensed by AI systems and automated agents." ## What each operator says - **OpenAI** (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot) robots-txt-training-optout: honors. Disallow GPTBot in robots.txt. "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." (https://developers.openai.com/api/docs/bots) - **Anthropic** (ClaudeBot, Claude-SearchBot, Claude-User) robots-txt-training-optout: honors. Disallow ClaudeBot in robots.txt. "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt." (https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - **Google** (Googlebot, Google-Extended, GoogleOther, Google-CloudVertexBot) robots-txt-training-optout: honors. Disallow Google-Extended in robots.txt. "Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" (https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) robots-meta-ai: honors. nosnippet also keeps a page from being used as direct input for AI Overviews and AI Mode. It does not address training. "will also prevent the content from being used as a direct input for AI Overviews and AI Mode." (https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) - **Microsoft** (bingbot). Unreadable by script: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 - **Apple** (Applebot, Applebot-Extended) robots-txt-training-optout: honors. Disallow Applebot-Extended in robots.txt. "Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file." (https://support.apple.com/en-us/119829) robots-meta-ai: honors. nosnippet keeps content out of AI-generated answers in Apple products. Training is controlled by Applebot-Extended instead. "Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services." (https://support.apple.com/en-us/119829) - **Perplexity** (PerplexityBot, Perplexity-User) robots-txt-training-optout: not-applicable. The operator says its crawlers are not used for AI training. "It is not used to crawl content for AI foundation models." (https://docs.perplexity.ai/guides/bots) - **Meta** (meta-externalagent, meta-webindexer, meta-externalfetcher, facebookexternalhit). Unreadable by script: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers robots-txt-training-optout: honors. Disallow meta-externalagent in robots.txt. - **Common Crawl** (CCBot) robots-txt-training-optout: honors. Disallow CCBot in robots.txt. "To prevent Common Crawl from crawling your website, include the following in your robots.txt" (https://commoncrawl.org/ccbot) - **Amazon** (Amazonbot, Amzn-SearchBot, Amzn-User) robots-txt-training-optout: honors. Disallow Amazonbot in robots.txt. "may be used to train Amazon AI models." (https://developer.amazon.com/amazonbot) robots-meta-ai: honors. noarchive means do not use the page for model training. "page level robots meta tags of noarchive (do not use the page for model training)" (https://developer.amazon.com/amazonbot) - **Mistral AI** (MistralAI-Training, MistralAI-Index, MistralAI-User) robots-txt-training-optout: honors. Disallow MistralAI-Training in robots.txt. "Webmasters can disallow this user agent in their robots.txt file." (https://docs.mistral.ai/robots) - **DuckDuckGo** (DuckAssistBot) robots-txt-training-optout: not-applicable. The operator says its crawlers are not used for AI training. "This data is not used in any way to train AI models." (https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) - **Diffbot** (Diffbot, Diffbot-User) robots-txt-training-optout: not-applicable. The operator says its crawlers are not used for AI training. "It is not used for AI training." (https://www.diffbot.com/docs/crawl/faq/robots-txt) - **You.com** (YouBot) - **Allen Institute for AI (Ai2)** (AI2Bot) Not included: ai.txt (Spawning): The publisher's site was under maintenance on 2026-10-20, so its specification could not be read and quoted. llms.txt: A guide for language models, not an opt-out or preference signal. Check a robots.txt against every crawler: https://bozos.si/tools/fetch#robots-check. Corrections: https://bozos.si/contact