How do I tell a real AI crawler from a fake one?
For each AI and search operator in the BOZOS crawler directory, which ways of confirming that a request really came from its bot the operator's own documentation describes: published IP ranges, forward-confirmed reverse DNS, and Web Bot Auth signatures. documented cells quote the operator and give the specifics a verifier needs. not-stated means the pages listed in searched do not mention the method; it is not evidence that the operator does not support it.
Record what the operator states, scoped to the pages checked. A specification is quoted for what it is, never as proof that anyone implements it. observed_lists is an observation of a fetch on the date given, not operator documentation.
Quick answers
- Is the User-Agent header enough?
- No. Every operator in the crawler directory names a user-agent token, but a header is self-reported and any client can send it. Operators warn about this themselves.
Since user agent strings can be spoofed, You.com supports three ways to verify that requests are genuinely from YouBot
You.com ↗ - Which operators publish IP ranges?
- 10 of 14: OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl, Amazon, Mistral AI, DuckDuckGo, You.com. Most use one JSON format (a prefixes array of ipv4Prefix and ipv6Prefix entries); Amazon's lists are HTML pages.
- Which operators document reverse DNS?
- Google, Apple, Common Crawl, You.com. The check is two lookups: address to hostname, hostname back to the same address.
- Who signs requests with Web Bot Auth?
- OpenAI, Google, You.com, each for part of their traffic only. Google calls its implementation experimental and says it does not sign every request, so keep the IP or DNS check as a fallback.
Operator by method.
Yes: the operator's documentation describes the method (see the quote below). —: not mentioned on the pages checked. ?: the page cannot be read by a script, so neither a statement nor its absence could be verified.
| Operator | IP ranges | Reverse DNS | Web Bot Auth |
|---|---|---|---|
| OpenAI | Yes | — | Yes |
| Anthropic | Yes | — | — |
| Yes | Yes | Yes | |
| Microsoft | ? | ? | ? |
| Apple | Yes | Yes | — |
| Perplexity | Yes | — | — |
| Meta | ? | ? | ? |
| Common Crawl | Yes | Yes | — |
| Amazon | Yes | — | — |
| Mistral AI | Yes | — | — |
| DuckDuckGo | Yes | — | — |
| Diffbot | — | — | — |
| You.com | Yes | Yes | Yes |
| Allen Institute for AI (Ai2) | — | — | — |
Published IP ranges
Fetch the operator's list (usually JSON with a prefixes array of ipv4Prefix and ipv6Prefix entries) and test whether the connecting address falls inside a listed CIDR range.
Limits. Only as current as the list: re-fetch on a schedule. An address proves the network, not which of the operator's bots sent the request, unless the operator publishes one list per bot.
Alternatively, you can identify Googlebot by IP address by matching the crawler's IP address to the lists of Google crawlers' and fetchers' IP ranges:
Source ↗
Operators that document it: OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl, Amazon, Mistral AI, DuckDuckGo, You.com.
Reverse then forward DNS (forward-confirmed reverse DNS)
Resolve the connecting address to a hostname (PTR record), check the hostname is under the operator's domain, resolve that hostname back to an address, and require it to equal the original address.
Limits. Needs two DNS lookups per new address (cache results). Operators say it is not available everywhere: Common Crawl states reverse DNS is not yet supported over IPv6.
Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.
Source ↗
Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name.
Source ↗
Operators that document it: Google, Apple, Common Crawl, You.com.
Web Bot Auth (HTTP Message Signatures)
Read the Signature-Agent header, fetch the key directory it names (/.well-known/http-message-signatures-directory), and verify the Signature and Signature-Input headers over the request per RFC 9421. Operators say only some requests are signed, so fall back to IP or DNS checks.
Limits. The profile is an Internet-Draft and may change. Operators sign a subset of traffic. Signature headers can be dropped by intermediaries. A valid signature proves control of the key, not good behavior.
The Web Bot Authentication (webbotauth) Working Group will standardize methods for cryptographically authenticating automated clients and providing additional information about their operators to Web sites.
Source ↗
This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message.
Source ↗
Operators that document it: OpenAI, Google, You.com.
What Web Bot Auth is built on.
Specifications are quoted for what they define, not as proof that any operator implements them.
- HTTP Message Signatures ↗ (RFC 9421 (Proposed Standard)). The signature mechanism Web Bot Auth builds on.
- HTTP Message Signatures for automated traffic Architecture ↗ (Internet-Draft (individual submission at last fetch)). How crawlers attach identity to requests with RFC 9421.
- HTTP Message Signatures Directory ↗ (Internet-Draft). The /.well-known/http-message-signatures-directory key list.
What each operator says.
OpenAI
- IP ranges. One list per bot: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. Lists: gptbot.json ↗ searchbot.json ↗ chatgpt-user.json ↗ adsbot.json ↗
Published IP addresses: https://openai.com/gptbot.json
Source ↗ - Web Bot Auth. ChatGPT's Cloud browser (the article's current name for ChatGPT agent traffic) only. OpenAI's crawler documentation does not mention signatures. Signature-Agent:
"https://chatgpt.com". Keys: chatgpt.com ↗.Cloud browser uses Web Bot Auth to sign outbound HTTP requests, allowing website operators to verify that requests genuinely originate from ChatGPT.
Source ↗ - Not stated: Reverse DNS.
Anthropic
- IP ranges. A single list for Anthropic's bots. Anthropic also says blocking by IP address may not reliably opt a site out, because it stops the crawler reading robots.txt; use the list to identify traffic, robots.txt to opt out. Lists: bots.json ↗
If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.
Source ↗ - Not stated: Reverse DNS, Web Bot Auth.
- IP ranges. Separate lists for common crawlers, special-case crawlers, and user-triggered fetchers. Lists: common-crawlers.json ↗ googlebot.json ↗
Alternatively, you can identify Googlebot by IP address by matching the crawler's IP address to the lists of Google crawlers' and fetchers' IP ranges:
Source ↗ - Reverse DNS. Hostnames end in googlebot.com, google.com or googleusercontent.com, depending on the crawler category. Hostnames:
crawl-***-***-***-***.googlebot.com, geo-crawl-***-***-***-***.geo.googlebot.com, rate-limited-proxy-***-***-***-***.google.com, ***-***-***-***.gae.googleusercontent.com, google-proxy-***-***-***-***.google.com.Verify that the domain name is either googlebot.com , google.com , or googleusercontent.com .
Source ↗Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name.
Source ↗ - Web Bot Auth. Experimental. A subset of requests from the Google-Agent user agent are signed; Google says it does not sign every request and recommends keeping IP and reverse-DNS checks. Signature-Agent:
g="https://agent.bot.goog". Keys: agent.bot.goog ↗.We don't sign every request of a particular agent.
Source ↗Not all Google user agents are using Web Bot Auth.
Source ↗ - Not stated: none.
Microsoft
- Documentation that cannot be read by a script: www.bing.com ↗
- Not stated: none.
Apple
- IP ranges. One CIDR list for Applebot. Lists: applebot.json ↗
Another way is to match the IP address with a CIDR prefix contained in the following JSON file
Source ↗ - Reverse DNS. Hostnames are under applebot.apple.com. Hostnames:
*.applebot.apple.com.Traffic coming from Applebot is generally identified by using reverse DNS
Source ↗The host command can also be used to verify that the DNS points to the same IP address
Source ↗ - Not stated: Web Bot Auth.
Perplexity
- IP ranges. One list for PerplexityBot and one for Perplexity-User. The documentation recommends combining User-Agent matching with IP checks. Lists: perplexitybot.json ↗ perplexity-user.json ↗
combine both User-Agent string matching and IP address verification for enhanced security
Source ↗ - Not stated: Reverse DNS, Web Bot Auth.
Meta
- Documentation that cannot be read by a script: developers.facebook.com ↗
- Not stated: none.
Common Crawl
- IP ranges. One JSON list covering IPv4 and IPv6. Lists: ccbot.json ↗
These IP ranges (v4 and v6) are also provided as JSON at https://index.commoncrawl.org/ccbot.json.
Source ↗ - Reverse DNS. Hostnames under crawl.commoncrawl.org. Reverse DNS is not available over IPv6. Hostnames:
*-*-*-*.crawl.commoncrawl.org.CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)
Source ↗ - Not stated: Web Bot Auth.
Amazon
- IP ranges. Three HTML pages (Amazonbot, search bot, live/user fetch). They are web pages, not JSON, so they need scraping. Lists: ip-addresses ↗ searchbot-ip-addresses ↗ live-ip-addresses ↗
Published IP Addresses: https://developer.amazon.com/amazonbot/ip-addresses/
Source ↗ - Not stated: Reverse DNS, Web Bot Auth.
Mistral AI
- IP ranges. Lists for MistralAI-User and MistralAI-Index. The page shows no list under MistralAI-Training. Lists: mistralai-user-ips.json ↗ mistralai-index-ips.json ↗
All published IP addresses:
Source ↗ - Not stated: Reverse DNS, Web Bot Auth.
DuckDuckGo
- IP ranges. One list for DuckAssistBot. Lists: duckassistbot.json ↗
originates from the following IP addresses (also available in JSON at https://duckduckgo.com/duckassistbot.json )
Source ↗ - Not stated: Reverse DNS, Web Bot Auth.
Diffbot
- Not stated: IP ranges, Reverse DNS, Web Bot Auth.
You.com
- IP ranges. A single /24 range stated in prose, not a machine-readable list.
Legitimate YouBot requests originate from 68.67.112.0/24.
Source ↗ - Reverse DNS. Described generically; the hostname pattern is not stated, and the page's worked example uses a Google address.
a forward lookup on that hostname should resolve back to the same IP
Source ↗ - Web Bot Auth. YouBot, which the page calls the recommended method. Keys are Ed25519 JWKs. Keys: you.com ↗.
YouBot uses Cloudflare's Web Bot Auth standard for authentication.
Source ↗ - Not stated: none.
Allen Institute for AI (Ai2)
- Not stated: IP ranges, Reverse DNS, Web Bot Auth.
The published IP lists as fetched.
A fetch on 2026-10-21. The creation time is the list's own stamp, so an old one shows how often an operator updates it. Re-fetch before relying on a range.
Quotes and searches you can re-run.
Every Yes quotes the operator's own page, and a script checks that each quote still appears there. Every not-stated cell lists the pages and search terms used, and the same script fails if one of those pages starts mentioning the method, so a new statement is caught at the next review.
Microsoft (Bing) and Meta publish their crawler documentation in a form a script cannot read, so their cells are unknown rather than not stated. Bing does serve a bingbot.json list, but its documentation was not read, so it is not counted as documented here.
Found a statement we missed? Tell us. To build a robots.txt for these crawlers, use the robots.txt builder.