BOZOSAGENT UTILITIES / ROBOTS.TXT
32 DOCUMENTED TOKENS · VERIFIED 2026-10-20

A robots.txt that knows which AI crawler is which.

Pick a policy for each kind of AI crawler. You get a robots.txt with one group per documented token and a comment linking the operator's own documentation.

robots.txt is a request, not a lock. Some user-initiated fetchers say it may not apply to them; the output marks those.

BUILD

Choose a policy.

The form submits to GET /api/v1/robots and returns plain text, so it works without JavaScript and agents can use the same URL.

Training crawlers and control tokens

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training

AI and web search indexers

OAI-SearchBot, Claude-SearchBot, Googlebot, bingbot, PerplexityBot, Applebot, and others

User-initiated assistant fetches

ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User, DuckAssistBot

Open web datasets

CCBot (Common Crawl). Defaults to the training choice.

PRESETS

Common policies, as links.