# Bot Verification Methods: How to Tell the Real Crawler

Canonical: https://bozos.si/reference/bot-verification
Data: https://bozos.si/bot-verification.json
Verified: 2026-10-21 · License: CC0-1.0

> For each AI and search operator in the BOZOS crawler directory, which ways of confirming that a request really came from its bot the operator's own documentation describes: published IP ranges, forward-confirmed reverse DNS, and Web Bot Auth signatures. documented cells quote the operator and give the specifics a verifier needs. not-stated means the pages listed in searched do not mention the method; it is not evidence that the operator does not support it.

## Baseline

User-Agent string: Every operator in the crawler directory names a user-agent token, but a header is self-reported and any client can send it. Operators warn about this themselves.
  Operator says: "Since user agent strings can be spoofed, You.com supports three ways to verify that requests are genuinely from YouBot" (https://you.com/docs/youbot)
  Operator says: "we are aware of crawlers falsely identifying themselves as CCBot" (https://commoncrawl.org/ccbot)

## Stances

- **documented**: The operator's documentation describes the method, for the scope given.
- **not-stated**: The pages checked do not mention the method. The search terms and pages are listed so the absence can be re-checked.
- **unknown**: The operator's documentation cannot be read by a script, so neither a statement nor its absence could be verified.

## Grid

| Operator | published-ip-ranges | reverse-forward-dns | web-bot-auth |
| --- | --- | --- | --- |
| OpenAI | yes | not stated | yes |
| Anthropic | yes | not stated | not stated |
| Google | yes | yes | yes |
| Microsoft | unknown | unknown | unknown |
| Apple | yes | yes | not stated |
| Perplexity | yes | not stated | not stated |
| Meta | unknown | unknown | unknown |
| Common Crawl | yes | yes | not stated |
| Amazon | yes | not stated | not stated |
| Mistral AI | yes | not stated | not stated |
| DuckDuckGo | yes | not stated | not stated |
| Diffbot | not stated | not stated | not stated |
| You.com | yes | yes | yes |
| Allen Institute for AI (Ai2) | not stated | not stated | not stated |

## Methods

- **Published IP ranges** (operator-practice; Each operator, individually). How: Fetch the operator's list (usually JSON with a prefixes array of ipv4Prefix and ipv6Prefix entries) and test whether the connecting address falls inside a listed CIDR range. Limits: Only as current as the list: re-fetch on a schedule. An address proves the network, not which of the operator's bots sent the request, unless the operator publishes one list per bot.
  Source says: "Alternatively, you can identify Googlebot by IP address by matching the crawler's IP address to the lists of Google crawlers' and fetchers' IP ranges:" (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
- **Reverse then forward DNS (forward-confirmed reverse DNS)** (operator-practice; Each operator, individually). How: Resolve the connecting address to a hostname (PTR record), check the hostname is under the operator's domain, resolve that hostname back to an address, and require it to equal the original address. Limits: Needs two DNS lookups per new address (cache results). Operators say it is not available everywhere: Common Crawl states reverse DNS is not yet supported over IPv6.
  Source says: "Run a reverse DNS lookup on the accessing IP address from your logs, using the host command." (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
  Source says: "Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name." (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
- **Web Bot Auth (HTTP Message Signatures)** (ietf-draft-on-rfc; IETF webbotauth working group; signatures per RFC 9421). How: Read the Signature-Agent header, fetch the key directory it names (/.well-known/http-message-signatures-directory), and verify the Signature and Signature-Input headers over the request per RFC 9421. Operators say only some requests are signed, so fall back to IP or DNS checks. Limits: The profile is an Internet-Draft and may change. Operators sign a subset of traffic. Signature headers can be dropped by intermediaries. A valid signature proves control of the key, not good behavior.
  Source says: "The Web Bot Authentication (webbotauth) Working Group will standardize methods for cryptographically authenticating automated clients and providing additional information about their operators to Web sites." (https://datatracker.ietf.org/wg/webbotauth/about/)
  Source says: "This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message." (https://www.rfc-editor.org/rfc/rfc9421.html)

## Specifications

- HTTP Message Signatures (RFC 9421 (Proposed Standard)): https://www.rfc-editor.org/rfc/rfc9421.html. The signature mechanism Web Bot Auth builds on.
- HTTP Message Signatures for automated traffic Architecture (Internet-Draft (individual submission at last fetch)): https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/. How crawlers attach identity to requests with RFC 9421.
- HTTP Message Signatures Directory (Internet-Draft): https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/. The /.well-known/http-message-signatures-directory key list.

## What each operator says

- **OpenAI** (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot)
  published-ip-ranges: One list per bot: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. Lists: https://openai.com/gptbot.json, https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json, https://openai.com/adsbot.json. "Published IP addresses: https://openai.com/gptbot.json" (https://developers.openai.com/api/docs/bots)
  web-bot-auth: ChatGPT's Cloud browser (the article's current name for ChatGPT agent traffic) only. OpenAI's crawler documentation does not mention signatures. Signature-Agent: "https://chatgpt.com". Key directory: https://chatgpt.com/.well-known/http-message-signatures-directory. "Cloud browser uses Web Bot Auth to sign outbound HTTP requests, allowing website operators to verify that requests genuinely originate from ChatGPT." (https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting)
- **Anthropic** (ClaudeBot, Claude-SearchBot, Claude-User)
  published-ip-ranges: A single list for Anthropic's bots. Anthropic also says blocking by IP address may not reliably opt a site out, because it stops the crawler reading robots.txt; use the list to identify traffic, robots.txt to opt out. Lists: https://claude.com/crawling/bots.json. "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic." (https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- **Google** (Googlebot, Google-Extended, GoogleOther, Google-CloudVertexBot)
  published-ip-ranges: Separate lists for common crawlers, special-case crawlers, and user-triggered fetchers. Lists: https://developers.google.com/static/crawling/ipranges/common-crawlers.json, https://developers.google.com/static/search/apis/ipranges/googlebot.json. "Alternatively, you can identify Googlebot by IP address by matching the crawler's IP address to the lists of Google crawlers' and fetchers' IP ranges:" (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
  reverse-forward-dns: Hostnames end in googlebot.com, google.com or googleusercontent.com, depending on the crawler category. Hostnames: crawl-***-***-***-***.googlebot.com, geo-crawl-***-***-***-***.geo.googlebot.com, rate-limited-proxy-***-***-***-***.google.com, ***-***-***-***.gae.googleusercontent.com, google-proxy-***-***-***-***.google.com. "Verify that the domain name is either googlebot.com , google.com , or googleusercontent.com ." (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) "Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name." (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
  web-bot-auth: Experimental. A subset of requests from the Google-Agent user agent are signed; Google says it does not sign every request and recommends keeping IP and reverse-DNS checks. Signature-Agent: g="https://agent.bot.goog". Key directory: https://agent.bot.goog/.well-known/http-message-signatures-directory. "We don't sign every request of a particular agent." (https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) "Not all Google user agents are using Web Bot Auth." (https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)
- **Microsoft** (bingbot). Unreadable by script: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
- **Apple** (Applebot, Applebot-Extended)
  published-ip-ranges: One CIDR list for Applebot. Lists: https://search.developer.apple.com/applebot.json. "Another way is to match the IP address with a CIDR prefix contained in the following JSON file" (https://support.apple.com/en-us/119829)
  reverse-forward-dns: Hostnames are under applebot.apple.com. Hostnames: *.applebot.apple.com. "Traffic coming from Applebot is generally identified by using reverse DNS" (https://support.apple.com/en-us/119829) "The host command can also be used to verify that the DNS points to the same IP address" (https://support.apple.com/en-us/119829)
- **Perplexity** (PerplexityBot, Perplexity-User)
  published-ip-ranges: One list for PerplexityBot and one for Perplexity-User. The documentation recommends combining User-Agent matching with IP checks. Lists: https://www.perplexity.com/perplexitybot.json, https://www.perplexity.com/perplexity-user.json. "combine both User-Agent string matching and IP address verification for enhanced security" (https://docs.perplexity.ai/guides/bots)
- **Meta** (meta-externalagent, meta-webindexer, meta-externalfetcher, facebookexternalhit). Unreadable by script: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
- **Common Crawl** (CCBot)
  published-ip-ranges: One JSON list covering IPv4 and IPv6. Lists: https://index.commoncrawl.org/ccbot.json. "These IP ranges (v4 and v6) are also provided as JSON at https://index.commoncrawl.org/ccbot.json." (https://commoncrawl.org/ccbot)
  reverse-forward-dns: Hostnames under crawl.commoncrawl.org. Reverse DNS is not available over IPv6. Hostnames: *-*-*-*.crawl.commoncrawl.org. "CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)" (https://commoncrawl.org/ccbot)
- **Amazon** (Amazonbot, Amzn-SearchBot, Amzn-User)
  published-ip-ranges: Three HTML pages (Amazonbot, search bot, live/user fetch). They are web pages, not JSON, so they need scraping. Lists: https://developer.amazon.com/amazonbot/ip-addresses/, https://developer.amazon.com/amazonbot/searchbot-ip-addresses/, https://developer.amazon.com/amazonbot/live-ip-addresses/. "Published IP Addresses: https://developer.amazon.com/amazonbot/ip-addresses/" (https://developer.amazon.com/amazonbot)
- **Mistral AI** (MistralAI-Training, MistralAI-Index, MistralAI-User)
  published-ip-ranges: Lists for MistralAI-User and MistralAI-Index. The page shows no list under MistralAI-Training. Lists: https://mistral.ai/mistralai-user-ips.json, https://mistral.ai/mistralai-index-ips.json. "All published IP addresses:" (https://docs.mistral.ai/robots)
- **DuckDuckGo** (DuckAssistBot)
  published-ip-ranges: One list for DuckAssistBot. Lists: https://duckduckgo.com/duckassistbot.json. "originates from the following IP addresses (also available in JSON at https://duckduckgo.com/duckassistbot.json )" (https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)
- **Diffbot** (Diffbot, Diffbot-User)
- **You.com** (YouBot)
  published-ip-ranges: A single /24 range stated in prose, not a machine-readable list. "Legitimate YouBot requests originate from 68.67.112.0/24." (https://you.com/docs/youbot)
  reverse-forward-dns: Described generically; the hostname pattern is not stated, and the page's worked example uses a Google address. "a forward lookup on that hostname should resolve back to the same IP" (https://you.com/docs/youbot)
  web-bot-auth: YouBot, which the page calls the recommended method. Keys are Ed25519 JWKs. Key directory: https://you.com/.well-known/http-message-signatures-directory. "YouBot uses Cloudflare's Web Bot Auth standard for authentication." (https://you.com/docs/youbot)
- **Allen Institute for AI (Ai2)** (AI2Bot)

## Observed IP lists (a fetch on 2026-10-21, not operator documentation)

- https://openai.com/gptbot.json: HTTP 200, json, creationTime 2026-09-22T02:00:07.000000, 18 IPv4 and 0 IPv6 prefixes
- https://openai.com/searchbot.json: HTTP 200, json, creationTime 2026-01-02T11:00:00.000000, 39 IPv4 and 0 IPv6 prefixes
- https://openai.com/chatgpt-user.json: HTTP 200, json, creationTime 2026-10-20T17:07:39.673498, 234 IPv4 and 0 IPv6 prefixes
- https://openai.com/adsbot.json: HTTP 200, json, creationTime 2026-05-12T03:00:00.000000, 2 IPv4 and 0 IPv6 prefixes
- https://claude.com/crawling/bots.json: HTTP 200, json, creationTime 2026-10-20T00:00:00Z, 14 IPv4 and 0 IPv6 prefixes
- https://developers.google.com/static/crawling/ipranges/common-crawlers.json: HTTP 200, json, creationTime 2026-10-20T14:46:18.000000, 170 IPv4 and 147 IPv6 prefixes
- https://developers.google.com/static/search/apis/ipranges/googlebot.json: HTTP 200, json, creationTime 2026-10-20T14:46:18.000000, 170 IPv4 and 147 IPv6 prefixes
- https://search.developer.apple.com/applebot.json: HTTP 200, json, creationTime 2026-09-15T10:00:00.000000, 24 IPv4 and 0 IPv6 prefixes
- https://www.perplexity.com/perplexitybot.json: HTTP 200, json, creationTime 2025-02-07T16:56:00.000000, 8 IPv4 and 0 IPv6 prefixes
- https://www.perplexity.com/perplexity-user.json: HTTP 200, json, creationTime 2025-10-17T10:17:00.000000, 4 IPv4 and 0 IPv6 prefixes
- https://index.commoncrawl.org/ccbot.json: HTTP 200, json, creationTime 2026-08-11T13:40:00Z, 4 IPv4 and 1 IPv6 prefixes
- https://developer.amazon.com/amazonbot/ip-addresses/: HTTP 200, html-or-other
- https://developer.amazon.com/amazonbot/searchbot-ip-addresses/: HTTP 200, html-or-other
- https://developer.amazon.com/amazonbot/live-ip-addresses/: HTTP 200, html-or-other
- https://mistral.ai/mistralai-user-ips.json: HTTP 200, json, creationTime 2025-02-19T12:00:00.000000, 4 IPv4 and 0 IPv6 prefixes
- https://mistral.ai/mistralai-index-ips.json: HTTP 200, json, creationTime 2026-04-19T12:00:00.000000, 2 IPv4 and 0 IPv6 prefixes
- https://duckduckgo.com/duckassistbot.json: HTTP 200, json, creationTime 2026-09-01T12:44:58.000000, 486 IPv4 and 0 IPv6 prefixes

Build a robots.txt: https://bozos.si/tools/robots-builder. Corrections: https://bozos.si/contact