# AI Crawler Directory Canonical: https://bozos.si/reference/ai-crawlers Data: https://bozos.si/ai-crawlers.json Verified: 2026-10-20 · License: CC0-1.0 > Crawlers and user agents that AI and search companies document themselves. Every entry in crawlers is taken from the operator's own documentation, read on the verified date. Tokens people often ask about but whose operators publish no documentation are listed separately in undocumented_tokens, with what BOZOS searched for and found; they are not described as fact. ## Categories - **training**: Collects content that may be used to train AI models. - **search**: Builds a search index that AI answers or search results draw on. - **user-fetch**: Fetches a page only when a person asks an assistant something. Often treated as outside robots.txt. - **control-token**: Not a crawler. A robots.txt token that controls how another crawler's data is used. - **open-dataset**: Builds a public archive that third parties, including AI developers, reuse. - **owner-requested**: Crawls only sites whose owners asked for it. - **link-preview**: Fetches metadata to render link previews. - **other**: Documented purpose that fits none of the above. ## robots.txt behavior values - **obeys**: The operator states, or clearly implies, that robots.txt rules for this token are honored. - **may-ignore**: The operator states robots.txt may not apply, usually because a user initiated the fetch. - **control-token**: The token exists only in robots.txt; there is no separate crawler to block. - **not-stated**: The operator's documentation does not say. ## OpenAI - **GPTBot** (training; robots.txt: obeys; may feed AI training; high confidence). Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot signals the site's content should not be used for training. Docs: https://developers.openai.com/api/docs/bots · IPs: https://openai.com/gptbot.json Evidence (purpose): "It is used to crawl content that may be used in training our generative AI foundation models." Evidence (robots_txt): "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Archived source (2026-10-16): https://web.archive.org/web/20261016233530/https://developers.openai.com/api/docs/bots - **OAI-SearchBot** (search; robots.txt: obeys; training use not stated; high confidence). Surfaces sites in ChatGPT search features. Opted-out sites are not shown in ChatGPT search answers but can still appear as navigational links. Changes take about 24 hours. Docs: https://developers.openai.com/api/docs/bots · IPs: https://openai.com/searchbot.json Evidence (purpose): "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." Evidence (robots_txt): "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." Archived source (2026-10-16): https://web.archive.org/web/20261016233530/https://developers.openai.com/api/docs/bots - **ChatGPT-User** (user-fetch; robots.txt: may-ignore; training use not stated; high confidence). Visits a page when a ChatGPT or Custom GPT user asks a question, and for GPT Actions. Does not crawl automatically. OpenAI states robots.txt rules may not apply because the fetch is user-initiated. Docs: https://developers.openai.com/api/docs/bots · IPs: https://openai.com/chatgpt-user.json Evidence (purpose): "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." Evidence (robots_txt): "Because these actions are initiated by a user, robots.txt rules may not apply." Archived source (2026-10-16): https://web.archive.org/web/20261016233530/https://developers.openai.com/api/docs/bots - **OAI-AdsBot** (other; robots.txt: not-stated; operator says not used for AI training; high confidence). Checks landing pages submitted as ChatGPT ads for safety and relevance. Visits only submitted ad pages. OpenAI's page does not say whether robots.txt applies. Docs: https://developers.openai.com/api/docs/bots · IPs: https://openai.com/adsbot.json Evidence (purpose): "OAI-AdsBot is used to validate the safety of web pages submitted as ads on ChatGPT." Evidence (used_for_ai_training): "the data collected by OAI-AdsBot is not used to train generative AI foundation models." Archived source (2026-10-16): https://web.archive.org/web/20261016233530/https://developers.openai.com/api/docs/bots ## Anthropic - **ClaudeBot** (training; robots.txt: obeys; may feed AI training; high confidence). Collects web content that could contribute to training Anthropic's models. Also supports the non-standard Crawl-delay extension. Anthropic asks site owners to set rules on every subdomain. Docs: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler · IPs: https://claude.com/crawling/bots.json Evidence (purpose): "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." Evidence (robots_txt): "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt." Evidence (used_for_ai_training): "When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets." Archived source (2026-10-16): https://web.archive.org/web/20261016233536/https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler - **Claude-SearchBot** (search; robots.txt: obeys; training use not stated; high confidence). Navigates the web to improve search result quality for Claude users. Covered by Anthropic's general statement that its bots honor robots.txt. Blocking may reduce visibility in Claude search results. Docs: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler · IPs: https://claude.com/crawling/bots.json Evidence (purpose): "Claude-SearchBot navigates the web to improve search result quality for users." Evidence (robots_txt): "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt." Archived source (2026-10-16): https://web.archive.org/web/20261016233536/https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler - **Claude-User** (user-fetch; robots.txt: obeys; training use not stated; high confidence). Retrieves pages when Claude users ask questions. Covered by Anthropic's general statement that its bots honor robots.txt. Blocking prevents retrieval in response to user queries. Docs: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler · IPs: https://claude.com/crawling/bots.json Evidence (purpose): "When individuals ask questions to Claude, it may access websites using a Claude-User agent." Evidence (robots_txt): "Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query" Archived source (2026-10-16): https://web.archive.org/web/20261016233536/https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler ## Google - **Googlebot** (search; robots.txt: obeys; training use not stated; high confidence). Crawls for Google Search, including Discover, Images, Video, and News surfaces. Standard robots.txt rules apply. Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers · IPs: https://developers.google.com/static/search/apis/ipranges/googlebot.json Evidence (purpose): "Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features)" Evidence (robots_txt): "They always obey robots.txt rules when crawling automatically." Archived source (2026-10-20): https://web.archive.org/web/20261020085128/https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers - **Google-Extended** (control-token; robots.txt: control-token; may feed AI training; high confidence). Not a crawler. Controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and the Vertex AI API. Set rules for Google-Extended in robots.txt. It does not affect inclusion in Google Search. Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers Evidence (category): "the robots.txt user-agent token is used in a control capacity." Evidence (used_for_ai_training): "Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" Evidence (purpose): "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Archived source (2026-10-20): https://web.archive.org/web/20261020085128/https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers - **GoogleOther** (other; robots.txt: obeys; training use not stated; high confidence). Generic crawler used by Google product teams, for example for internal research. Standard robots.txt rules apply. Affects no specific product. Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers · IPs: https://developers.google.com/static/crawling/ipranges/common-crawlers.json Evidence (purpose): "GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites." Evidence (robots_txt): "They always obey robots.txt rules when crawling automatically." Archived source (2026-10-20): https://web.archive.org/web/20261020085128/https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers - **Google-CloudVertexBot** (owner-requested; robots.txt: obeys; training use not stated; high confidence). Crawls sites at the site owner's request to build Vertex AI Agents. No effect on Google Search. Docs: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers · IPs: https://developers.google.com/static/crawling/ipranges/common-crawlers.json Evidence (purpose): "Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents." Evidence (robots_txt): "They always obey robots.txt rules when crawling automatically." Archived source (2026-10-20): https://web.archive.org/web/20261020085128/https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers ## Microsoft - **bingbot** (search; robots.txt: obeys; training use not stated; medium confidence). Crawler for the Bing search index. Standard robots.txt rules apply. Microsoft's help page renders client-side; the IP list was verified directly. Docs: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 · IPs: https://www.bing.com/toolbox/bingbot.json Evidence: not-machine-verifiable: the documentation page renders in the browser, so its text cannot be fetched and checked automatically. Details rest on the token name and the verified IP list. Archived source (2026-10-19): https://web.archive.org/web/20261019101117/https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 ## Apple - **Applebot** (search; robots.txt: obeys; may feed AI training; high confidence). Powers search in Spotlight, Siri, and Safari. Apple says crawled data may also be used to train its foundation models. Apple says Applebot respects standard robots.txt directives in general search crawls targeted at Applebot, and falls back to Googlebot rules when no Applebot rules exist. Does not follow Crawl-delay. Docs: https://support.apple.com/en-us/119829 · IPs: https://search.developer.apple.com/applebot.json Evidence (purpose): "The data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari." Evidence (used_for_ai_training): "The data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products" Evidence (robots_txt): "Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot." Evidence (robots_txt): "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." Archived source (2026-10-16): https://web.archive.org/web/20261016065936/https://support.apple.com/en-us/119829 - **Applebot-Extended** (control-token; robots.txt: control-token; may feed AI training; high confidence). Not a crawler. Controls whether data Applebot already crawled may be used to train Apple's general-purpose foundation models. Blocking it does not remove pages from Apple search results. Docs: https://support.apple.com/en-us/119829 Evidence (used_for_ai_training): "Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file." Archived source (2026-10-16): https://web.archive.org/web/20261016065936/https://support.apple.com/en-us/119829 ## Perplexity - **PerplexityBot** (search; robots.txt: obeys; operator says not used for AI training; high confidence). Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. Perplexity recommends allowing it in robots.txt; its page does not state compliance in so many words. Docs: https://docs.perplexity.ai/guides/bots · IPs: https://www.perplexity.com/perplexitybot.json Evidence (purpose): "PerplexityBot is designed to surface and link websites in search results on Perplexity." Evidence (used_for_ai_training): "It is not used to crawl content for AI foundation models." Archived source (2026-10-20): https://web.archive.org/web/20261020085457/https://docs.perplexity.ai/guides/bots Conflict reported: Cloudflare, 2025-08-04, "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives" (https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/). Cloudflare reported that when Perplexity's declared crawler was blocked, it used "a generic browser intended to impersonate Google Chrome on macOS" from IPs outside its published list, "ignoring — or sometimes failing to even fetch — robots.txt files". Cloudflare "de-listed them as a verified bot". Operator response (2025-08-04, https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web): "Cloudflare fundamentally misattributed 3-6M daily requests from BrowserBase's automated browser service to Perplexity." - **Perplexity-User** (user-fetch; robots.txt: may-ignore; operator says not used for AI training; high confidence). Visits a page when a Perplexity user asks a question, and links it in the answer. Not used for crawling or training. Perplexity's page says both that Perplexity-User "controls which sites these user requests can access" and that, since a user requested the fetch, the fetcher "generally ignores robots.txt rules". Docs: https://docs.perplexity.ai/guides/bots · IPs: https://www.perplexity.com/perplexity-user.json Evidence (purpose): "When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response." Evidence (robots_txt): "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Evidence (robots_txt): "Perplexity-User controls which sites these user requests can access." Evidence (used_for_ai_training): "It is not used for web crawling or to collect content for training AI foundation models." Archived source (2026-10-20): https://web.archive.org/web/20261020085457/https://docs.perplexity.ai/guides/bots Conflict reported: Cloudflare, 2025-08-04, "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives" (https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/). Cloudflare reported that when Perplexity's declared crawler was blocked, it used "a generic browser intended to impersonate Google Chrome on macOS" from IPs outside its published list, "ignoring — or sometimes failing to even fetch — robots.txt files". Cloudflare "de-listed them as a verified bot". Operator response (2025-08-04, https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web): "Cloudflare fundamentally misattributed 3-6M daily requests from BrowserBase's automated browser service to Perplexity." ## Meta - **meta-externalagent** (training; robots.txt: obeys; may feed AI training; high confidence). Crawls for uses such as training foundation AI models or improving products by indexing content directly. Block with a disallow rule. Changes can take up to 24 hours. Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers Evidence: not-machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. Archived source (2026-10-20): https://web.archive.org/web/20261020091238/https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - **meta-webindexer** (search; robots.txt: not-stated; training use not stated; high confidence). Improves Meta AI search results so Meta AI can cite and link content. Meta's page does not describe its robots.txt behavior. Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers Evidence: not-machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. Archived source (2026-10-20): https://web.archive.org/web/20261020091238/https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - **meta-externalfetcher** (user-fetch; robots.txt: may-ignore; training use not stated; high confidence). Fetches individual links at a user's request for agentic AI features. Meta states this crawler may bypass robots.txt rules. Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers Evidence: not-machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. Archived source (2026-10-20): https://web.archive.org/web/20261020091238/https://developers.facebook.com/docs/sharing/webmasters/web-crawlers - **facebookexternalhit** (link-preview; robots.txt: may-ignore; training use not stated; high confidence). Fetches title, description, and thumbnail for links shared in Meta apps. Meta states it might bypass robots.txt for security or integrity checks. Docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers Evidence: not-machine-verifiable: the documentation page refuses automated requests (HTTP 400), so quotes cannot be checked automatically. Last read in a browser on the verified date. Archived source (2026-10-20): https://web.archive.org/web/20261020091238/https://developers.facebook.com/docs/sharing/webmasters/web-crawlers ## Common Crawl - **CCBot** (open-dataset; robots.txt: obeys; training use not stated; high confidence). Builds Common Crawl's open web archive, which anyone may analyze and which is widely used in AI training datasets. Common Crawl documents blocking with User-agent: CCBot / Disallow: /. Spoofed CCBot user agents exist; verify by reverse DNS. Docs: https://commoncrawl.org/ccbot · IPs: https://index.commoncrawl.org/ccbot.json Evidence (purpose): "producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone." Evidence (robots_txt): "To prevent Common Crawl from crawling your website, include the following in your robots.txt" Archived source (2026-10-05): https://web.archive.org/web/20261005215406/https://commoncrawl.org/ccbot ## Amazon - **Amazonbot** (training; robots.txt: obeys; may feed AI training; high confidence). Improves Amazon products and services. Amazon says content may be used to train Amazon AI models. Does not support Crawl-delay. May use a cached robots.txt up to 30 days old. Docs: https://developer.amazon.com/amazonbot · IPs: https://developer.amazon.com/amazonbot/ip-addresses/ Evidence (purpose): "Amazonbot is used to improve our products and services." Evidence (used_for_ai_training): "may be used to train Amazon AI models." Evidence (robots_txt): "Automated crawling from these listed user agents respects the Robots Exclusion Protocol" Archived source (2026-10-07): https://web.archive.org/web/20261007220743/https://developer.amazon.com/amazonbot - **Amzn-SearchBot** (search; robots.txt: obeys; operator says not used for AI training; high confidence). Improves search in Amazon products such as Alexa. Not used for generative AI training. If not named, follows the rules given to other search bots. Docs: https://developer.amazon.com/amazonbot · IPs: https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ Evidence (purpose): "Amzn-SearchBot is used to improve search experiences in Amazon products and services." Evidence (used_for_ai_training): "Amzn-SearchBot does not crawl content for generative AI model training." Evidence (robots_txt): "Automated crawling from these listed user agents respects the Robots Exclusion Protocol" Archived source (2026-10-07): https://web.archive.org/web/20261007220743/https://developer.amazon.com/amazonbot - **Amzn-User** (user-fetch; robots.txt: may-ignore; operator says not used for AI training; high confidence). Handles user-initiated requests, for example fetching live data for Alexa. Not used for generative AI training. Amazon states it may not always follow robots.txt because it acts on a user's behalf. Docs: https://developer.amazon.com/amazonbot · IPs: https://developer.amazon.com/amazonbot/live-ip-addresses/ Evidence (purpose): "Amzn-User supports user actions, such as responding to Alexa queries that require up-to-date information." Evidence (robots_txt): "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives." Evidence (used_for_ai_training): "Amzn-User does not crawl content for generative AI model training." Archived source (2026-10-07): https://web.archive.org/web/20261007220743/https://developer.amazon.com/amazonbot ## Mistral AI - **MistralAI-Training** (training; robots.txt: obeys; may feed AI training; high confidence). Collects web content to build training datasets for Mistral's generative models. Mistral says webmasters can disallow it in robots.txt. Docs: https://docs.mistral.ai/robots Evidence (purpose): "MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models." Evidence (robots_txt): "Webmasters can disallow this user agent in their robots.txt file." Archived source (2026-10-10): https://web.archive.org/web/20261010094213/https://docs.mistral.ai/robots - **MistralAI-Index** (search; robots.txt: not-stated; operator says not used for AI training; high confidence). Crawls for indexing only, feeding Mistral search. Not used for generative AI training. Mistral's page does not give bot-specific robots.txt guidance. Docs: https://docs.mistral.ai/robots · IPs: https://mistral.ai/mistralai-index-ips.json Evidence (purpose): "MistralAI-Index is for automated crawling of the web for indexing purposes only." Evidence (used_for_ai_training): "Content crawled by MistralAI-Index is not used for generative AI training of any kind." Archived source (2026-10-10): https://web.archive.org/web/20261010094213/https://docs.mistral.ai/robots - **MistralAI-User** (user-fetch; robots.txt: obeys; operator says not used for AI training; high confidence). Visits pages when a user asks Mistral's assistant a question. Not used for automatic crawling. Mistral says the MistralAI-User robots.txt tag governs which sites these user requests can be made to. Docs: https://docs.mistral.ai/robots · IPs: https://mistral.ai/mistralai-user-ips.json Evidence (purpose): "When users ask Vibe a question, it may visit a web page to help answer and include a link to the source in its response." Evidence (robots_txt): "MistralAI-User governs which sites these user requests can be made to." Evidence (used_for_ai_training): "It is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training." Archived source (2026-10-10): https://web.archive.org/web/20261010094213/https://docs.mistral.ai/robots ## DuckDuckGo - **DuckAssistBot** (user-fetch; robots.txt: obeys; operator says not used for AI training; high confidence). Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite sources. Not used to train AI models. Opt-outs take effect after 72 hours and do not affect organic search rankings. Docs: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot · IPs: https://duckduckgo.com/duckassistbot.json Evidence (purpose): "DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers, which prominently cite their sources." Evidence (used_for_ai_training): "This data is not used in any way to train AI models." Evidence (robots_txt): "Once you've updated the file to disallow the DuckAssistBot user agent, the change will take effect after 72 hours and DuckAssistBot will stop crawling your site." Archived source (2026-10-13): https://web.archive.org/web/20261013204641/https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot ## Diffbot - **Diffbot** (search; robots.txt: obeys; operator says not used for AI training; high confidence). General, proactive web crawling for Diffbot's Knowledge Graph and web search, so sites can be discovered and cited there. Diffbot says this user agent is not used for AI training. Honors disallow and crawl-delay by default. Diffbot says robots.txt can be overridden in specific cases, usually under a partnership or agreement with the site. Docs: https://www.diffbot.com/docs/crawl/faq/robots-txt Evidence (purpose): "General, proactive web crawling for building a general search engine." Evidence (used_for_ai_training): "It is not used for AI training." Evidence (robots_txt): "By default Diffbot's web crawls adhere to a site's robots.txt instructions, including the disallow and crawl-delay directives." Evidence (robots_txt): "the robots.txt instruction can be ignored/overridden." Archived source (2026-10-20): https://web.archive.org/web/20261020090915/https://www.diffbot.com/docs/crawl/faq/robots-txt - **Diffbot-User** (user-fetch; robots.txt: not-stated; training use not stated; high confidence). Requests made on behalf of a person browsing a URL with Diffbot software. Diffbot's robots.txt statement covers its crawls; the page does not say how robots.txt applies to user-initiated Diffbot-User requests. Docs: https://www.diffbot.com/docs/crawl/faq/robots-txt Evidence (purpose): "This is used by requests originating on behalf of a human user browsing a URL using Diffbot software, in response to their input." Archived source (2026-10-20): https://web.archive.org/web/20261020090915/https://www.diffbot.com/docs/crawl/faq/robots-txt ## You.com - **YouBot** (search; robots.txt: obeys; training use not stated; high confidence). The web crawler that powers the You.com search engine. You.com says YouBot respects robots.txt, including user-agent-specific rules and crawl-delay, and caches robots.txt for 30 minutes. Docs: https://you.com/docs/youbot Evidence (purpose): "YouBot is the web crawler that powers the You.com search engine." Evidence (robots_txt): "YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings." Evidence (robots_txt): "YouBot caches robots.txt for 30 minutes." Evidence (ip_ranges): "You.com supports three ways to verify that requests are genuinely from YouBot: cryptographic signatures (recommended), reverse DNS checks, and IP range validation." Archived source (2026-10-20): https://web.archive.org/web/20261020091006/https://you.com/docs/youbot ## Allen Institute for AI (Ai2) - **AI2Bot** (training; robots.txt: not-stated; may feed AI training; high confidence). Explores certain domains to find web content that Ai2 says is used to train open language models. The crawler page does not mention robots.txt. It says the user-agent string can be used to filter or reject the crawler's traffic. Docs: https://allenai.org/crawler Evidence (purpose): "The AI2 Bot explores certain domains to find web content." Evidence (used_for_ai_training): "This web content is used to train open language models." Evidence (robots_txt): "This user agent string can be used to filter or reject traffic from our crawler if desired." Archived source (2026-10-02): https://web.archive.org/web/20261002130200/https://allenai.org/crawler ## Often asked about, not documented by their operators - **Bytespider** (searched 2026-10-20): BOZOS found no ByteDance-published page documenting Bytespider's purpose, robots.txt behavior, or IP ranges. Third-party claims about it disagree, so they are not recorded here as fact. You can still name it in robots.txt (User-agent: Bytespider). Compliance cannot be confirmed from operator documentation, so enforce a block at your CDN or firewall if it matters. - **cohere-ai** (searched 2026-10-20): BOZOS found no Cohere-published crawler documentation for cohere-ai (Cohere's own pages use the name for its SDK packages). Third-party descriptions of its purpose conflict. You can name it in robots.txt (User-agent: cohere-ai). Third parties also list a separate cohere-training-data-crawler token; neither is documented by Cohere. ## Common robots.txt recipes Generate one for any policy: https://bozos.si/api/v1/robots?training=block&search=allow&user_fetch=allow (plain text). Guide: https://bozos.si/tools/robots-builder Corrections: https://bozos.si/contact (subject: [BOZOS agent] correction: ai-crawler-directory-data - )