VERIFIED 2026-10-20 · 32 TOKENS · 14 OPERATORS · CC0
Which AI crawler is which, and what it actually does.
Crawlers and user agents that AI and search companies document themselves. Every entry in crawlers is taken from the operator's own documentation, read on the verified date. Tokens people often ask about but whose operators publish no documentation are listed separately in undocumented_tokens, with what BOZOS searched for and found; they are not described as fact.
Record what the operator states, not what observers report. Where the documentation is silent, say so (robots_txt: not-stated, used_for_ai_training: null) rather than infer.
Quick answers
- How do I opt out of AI training but stay in AI search and answers?
- Disallow the training crawlers, the training control tokens, and Common Crawl's open dataset crawler (
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,meta-externalagent,CCBot,Amazonbot,MistralAI-Training,AI2Bot), and allow the rest. Get that robots.txt → - Which tokens are not crawlers at all?
Google-Extended,Applebot-Extended. They only exist in robots.txt and decide how data their parent crawler already fetched may be used. Blocking them does not affect search.- Which fetchers say robots.txt may not apply?
ChatGPT-User,Perplexity-User,meta-externalfetcher,facebookexternalhit,Amzn-User. Their operators say robots.txt may not apply, usually because a person asked for the page (facebookexternalhit: during security checks). robots.txt is a request, not access control.- How do I tell a real crawler from a fake one?
- Check the request's IP against the operator's published list (linked in the table) or use reverse DNS where the operator documents it. User-agent strings are easy to spoof.
- I want to block every AI crawler.
- Get a robots.txt that disallows all of them → User-initiated fetchers may still visit, as above.
All documented tokens, by operator.
OpenAI
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
GPTBot | training | Obeys | May train models | Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot signals the site's content should not be used for training.In OpenAI's words (2)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
OAI-SearchBot | search | Obeys | Not stated | Surfaces sites in ChatGPT search features. Opted-out sites are not shown in ChatGPT search answers but can still appear as navigational links. Changes take about 24 hours.In OpenAI's words (2)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
ChatGPT-User | user-fetch | May ignore | Not stated | Visits a page when a ChatGPT or Custom GPT user asks a question, and for GPT Actions. Does not crawl automatically. OpenAI states robots.txt rules may not apply because the fetch is user-initiated.In OpenAI's words (2)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
OAI-AdsBot | other | Not stated | Not for training | Checks landing pages submitted as ChatGPT ads for safety and relevance. Visits only submitted ad pages. OpenAI's page does not say whether robots.txt applies.In OpenAI's words (2)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
Anthropic
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
ClaudeBot | training | Obeys | May train models | Collects web content that could contribute to training Anthropic's models. Also supports the non-standard Crawl-delay extension. Anthropic asks site owners to set rules on every subdomain.In Anthropic's words (3)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
Claude-SearchBot | search | Obeys | Not stated | Navigates the web to improve search result quality for Claude users. Covered by Anthropic's general statement that its bots honor robots.txt. Blocking may reduce visibility in Claude search results.In Anthropic's words (2)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
Claude-User | user-fetch | Obeys | Not stated | Retrieves pages when Claude users ask questions. Covered by Anthropic's general statement that its bots honor robots.txt. Blocking prevents retrieval in response to user queries.In Anthropic's words (2)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
Googlebot | search | Obeys | Not stated | Crawls for Google Search, including Discover, Images, Video, and News surfaces. Standard robots.txt rules apply.In Google's words (2)
| Docs ↗ · Archived 2026-10-20 ↗ · IPs ↗ |
Google-Extended | control-token | Control token | May train models | Not a crawler. Controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and the Vertex AI API. Set rules for Google-Extended in robots.txt. It does not affect inclusion in Google Search.In Google's words (3)
| Docs ↗ · Archived 2026-10-20 ↗ |
GoogleOther | other | Obeys | Not stated | Generic crawler used by Google product teams, for example for internal research. Standard robots.txt rules apply. Affects no specific product.In Google's words (2)
| Docs ↗ · Archived 2026-10-20 ↗ · IPs ↗ |
Google-CloudVertexBot | owner-requested | Obeys | Not stated | Crawls sites at the site owner's request to build Vertex AI Agents. No effect on Google Search.In Google's words (2)
| Docs ↗ · Archived 2026-10-20 ↗ · IPs ↗ |
Microsoft
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
bingbot medium confidence | search | Obeys | Not stated | Crawler for the Bing search index. Standard robots.txt rules apply. Microsoft's help page renders client-side; the IP list was verified directly.Quotes not machine-verifiable: the documentation page renders in the browser, so its text cannot be fetched and checked automatically. Details rest on the token name and the verified IP list. | Docs ↗ · Archived 2026-10-19 ↗ · IPs ↗ |
Apple
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
Applebot | search | Obeys | May train models | Powers search in Spotlight, Siri, and Safari. Apple says crawled data may also be used to train its foundation models. Apple says Applebot respects standard robots.txt directives in general search crawls targeted at Applebot, and falls back to Googlebot rules when no Applebot rules exist. Does not follow Crawl-delay.In Apple's words (4)
| Docs ↗ · Archived 2026-10-16 ↗ · IPs ↗ |
Applebot-Extended | control-token | Control token | May train models | Not a crawler. Controls whether data Applebot already crawled may be used to train Apple's general-purpose foundation models. Blocking it does not remove pages from Apple search results.In Apple's words (1)
| Docs ↗ · Archived 2026-10-16 ↗ |
Perplexity
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
PerplexityBot conflict reported | search | Obeys | Not for training | Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. Perplexity recommends allowing it in robots.txt; its page does not state compliance in so many words.In Perplexity's words (2)
| Docs ↗ · Archived 2026-10-20 ↗ · IPs ↗ |
Perplexity-User conflict reported | user-fetch | May ignore | Not for training | Visits a page when a Perplexity user asks a question, and links it in the answer. Not used for crawling or training. Perplexity's page says both that Perplexity-User "controls which sites these user requests can access" and that, since a user requested the fetch, the fetcher "generally ignores robots.txt rules".In Perplexity's words (4)
| Docs ↗ · Archived 2026-10-20 ↗ · IPs ↗ |
Meta
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
meta-externalagent | training | Obeys | May train models | Crawls for uses such as training foundation AI models or improving products by indexing content directly. Block with a disallow rule. Changes can take up to 24 hours.Quotes not machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. | Docs ↗ · Archived 2026-10-20 ↗ |
meta-webindexer | search | Not stated | Not stated | Improves Meta AI search results so Meta AI can cite and link content. Meta's page does not describe its robots.txt behavior.Quotes not machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. | Docs ↗ · Archived 2026-10-20 ↗ |
meta-externalfetcher | user-fetch | May ignore | Not stated | Fetches individual links at a user's request for agentic AI features. Meta states this crawler may bypass robots.txt rules.Quotes not machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. | Docs ↗ · Archived 2026-10-20 ↗ |
facebookexternalhit | link-preview | May ignore | Not stated | Fetches title, description, and thumbnail for links shared in Meta apps. Meta states it might bypass robots.txt for security or integrity checks.Quotes not machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. | Docs ↗ · Archived 2026-10-20 ↗ |
Common Crawl
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
CCBot | open-dataset | Obeys | Not stated | Builds Common Crawl's open web archive, which anyone may analyze and which is widely used in AI training datasets. Common Crawl documents blocking with User-agent: CCBot / Disallow: /. Spoofed CCBot user agents exist; verify by reverse DNS.In Common Crawl's words (2)
| Docs ↗ · Archived 2026-10-05 ↗ · IPs ↗ |
Amazon
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
Amazonbot | training | Obeys | May train models | Improves Amazon products and services. Amazon says content may be used to train Amazon AI models. Does not support Crawl-delay. May use a cached robots.txt up to 30 days old.In Amazon's words (3)
| Docs ↗ · Archived 2026-10-07 ↗ · IPs ↗ |
Amzn-SearchBot | search | Obeys | Not for training | Improves search in Amazon products such as Alexa. Not used for generative AI training. If not named, follows the rules given to other search bots.In Amazon's words (3)
| Docs ↗ · Archived 2026-10-07 ↗ · IPs ↗ |
Amzn-User | user-fetch | May ignore | Not for training | Handles user-initiated requests, for example fetching live data for Alexa. Not used for generative AI training. Amazon states it may not always follow robots.txt because it acts on a user's behalf.In Amazon's words (3)
| Docs ↗ · Archived 2026-10-07 ↗ · IPs ↗ |
Mistral AI
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
MistralAI-Training | training | Obeys | May train models | Collects web content to build training datasets for Mistral's generative models. Mistral says webmasters can disallow it in robots.txt.In Mistral AI's words (2)
| Docs ↗ · Archived 2026-10-10 ↗ |
MistralAI-Index | search | Not stated | Not for training | Crawls for indexing only, feeding Mistral search. Not used for generative AI training. Mistral's page does not give bot-specific robots.txt guidance.In Mistral AI's words (2)
| Docs ↗ · Archived 2026-10-10 ↗ · IPs ↗ |
MistralAI-User | user-fetch | Obeys | Not for training | Visits pages when a user asks Mistral's assistant a question. Not used for automatic crawling. Mistral says the MistralAI-User robots.txt tag governs which sites these user requests can be made to.In Mistral AI's words (3)
| Docs ↗ · Archived 2026-10-10 ↗ · IPs ↗ |
DuckDuckGo
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
DuckAssistBot | user-fetch | Obeys | Not for training | Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite sources. Not used to train AI models. Opt-outs take effect after 72 hours and do not affect organic search rankings.In DuckDuckGo's words (3)
| Docs ↗ · Archived 2026-10-13 ↗ · IPs ↗ |
Diffbot
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
Diffbot | search | Obeys | Not for training | General, proactive web crawling for Diffbot's Knowledge Graph and web search, so sites can be discovered and cited there. Diffbot says this user agent is not used for AI training. Honors disallow and crawl-delay by default. Diffbot says robots.txt can be overridden in specific cases, usually under a partnership or agreement with the site.In Diffbot's words (4)
| Docs ↗ · Archived 2026-10-20 ↗ |
Diffbot-User | user-fetch | Not stated | Not stated | Requests made on behalf of a person browsing a URL with Diffbot software. Diffbot's robots.txt statement covers its crawls; the page does not say how robots.txt applies to user-initiated Diffbot-User requests.In Diffbot's words (1)
| Docs ↗ · Archived 2026-10-20 ↗ |
You.com
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
YouBot | search | Obeys | Not stated | The web crawler that powers the You.com search engine. You.com says YouBot respects robots.txt, including user-agent-specific rules and crawl-delay, and caches robots.txt for 30 minutes.In You.com's words (4)
| Docs ↗ · Archived 2026-10-20 ↗ |
Allen Institute for AI (Ai2)
| Token | Category | robots.txt | Training | What it does | Sources |
|---|---|---|---|---|---|
AI2Bot | training | Not stated | May train models | Explores certain domains to find web content that Ai2 says is used to train open language models. The crawler page does not mention robots.txt. It says the user-agent string can be used to filter or reject the crawler's traffic.In Allen Institute for AI (Ai2)'s words (3)
| Docs ↗ · Archived 2026-10-02 ↗ |
DOCS VS. REPORTS
Where observers say a crawler does not do what its documentation says.
Recorded only when a named, primary source publishes its own observations. The operator's response is recorded next to it. Every other entry is marked none-recorded, which means no report is on file, not that behavior was verified.
Cloudflare, 2025-08-04: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives ↗. Cloudflare reported that when Perplexity's declared crawler was blocked, it used "a generic browser intended to impersonate Google Chrome on macOS" from IPs outside its published list, "ignoring — or sometimes failing to even fetch — robots.txt files". Cloudflare "de-listed them as a verified bot". Applies to: PerplexityBot, Perplexity-User.
Operator response, 2025-08-04: Agents or Bots? Making Sense of AI on the Open Web ↗: “Cloudflare fundamentally misattributed 3-6M daily requests from BrowserBase's automated browser service to Perplexity.”
NOT IN THE DIRECTORY
Often asked about, never documented by their operators.
These tokens appear in many blocklists, but BOZOS could not find documentation from the companies they are attributed to. Third-party claims about them disagree, so they are not described here as fact.
Bytespidersearched 2026-10-20- BOZOS found no ByteDance-published page documenting Bytespider's purpose, robots.txt behavior, or IP ranges. Third-party claims about it disagree, so they are not recorded here as fact. You can still name it in robots.txt (User-agent: Bytespider). Compliance cannot be confirmed from operator documentation, so enforce a block at your CDN or firewall if it matters.
cohere-aisearched 2026-10-20- BOZOS found no Cohere-published crawler documentation for cohere-ai (Cohere's own pages use the name for its SDK packages). Third-party descriptions of its purpose conflict. You can name it in robots.txt (User-agent: cohere-ai). Third parties also list a separate cohere-training-data-crawler token; neither is documented by Cohere.
METHOD
Operator documentation only.
Each entry was read from the operator's own crawler documentation on 2026-10-20, and every published IP list URL was fetched to confirm it resolves. Where documentation is silent, the field says not-stated or null instead of guessing.
Evidence: each record lists the operator's exact words behind its fields. A script fetches every source page and fails if any quote no longer appears, and each source has a Wayback Machine snapshot, so you can check a record without trusting BOZOS.
Categories: training: Collects content that may be used to train AI models. search: Builds a search index that AI answers or search results draw on. user-fetch: Fetches a page only when a person asks an assistant something. Often treated as outside robots.txt. control-token: Not a crawler. A robots.txt token that controls how another crawler's data is used. open-dataset: Builds a public archive that third parties, including AI developers, reuse. owner-requested: Crawls only sites whose owners asked for it. link-preview: Fetches metadata to render link previews. other: Documented purpose that fits none of the above.
Crawler documentation changes often. If something here is out of date, agents and people can report it; fixes go in the corrections log.