{"schema_version":"1.0","requested_url":"https://bozos.si/reference/ai-crawlers.md","final_url":"https://bozos.si/reference/ai-crawlers.md","title":"AI Crawler Directory","author":null,"published_at":null,"declared_canonical_url":null,"description":null,"language":null,"extraction":"passthrough","markdown":"# AI Crawler Directory\n\nCanonical: https://bozos.si/reference/ai-crawlers\nData: https://bozos.si/ai-crawlers.json\nVerified: 2026-10-20 · License: CC0-1.0\n\n> Crawlers and user agents that AI and search companies document themselves. Every entry in crawlers is taken from the operator's own documentation, read on the verified date. Tokens people often ask about but whose operators publish no documentation are listed separately in undocumented_tokens, with what BOZOS searched for and found; they are not described as fact.\n\n## Categories\n\n- **training**: Collects content that may be used to train AI models.\n- **search**: Builds a search index that AI answers or search results draw on.\n- **user-fetch**: Fetches a page only when a person asks an assistant something. Often treated as outside robots.txt.\n- **control-token**: Not a crawler. A robots.txt token that controls how another crawler's data is used.\n- **open-dataset**: Builds a public archive that third parties, including AI developers,","headings":[],"links":[],"warnings":["Output was truncated to 1000 Unicode code points; use a larger max_characters to request more."],"retrieved_at":"2026-10-24T11:55:46.031Z","redirects":[],"http":{"status":200,"content_type":"text/markdown","charset":"utf-8","content_encoding":"identity","etag":"W/\"30810-1792841207\"","last_modified":null},"integrity":{"algorithm":"sha256","source_sha256":"44423498ff6b044b9cef794264f078d54fc1c98737e42384abe4185cb9e53a07","markdown_sha256":"2dbb4dbbb0813060cd8c026450cd040a3aa21ef8ec53adbc72da3d2b89be9ff5","source_bytes":30810,"source_representation":"Response body after HTTP decompression and before character decoding."},"output":{"characters":1000,"full_characters":30779,"truncated":true,"max_characters":1000},"citation":{"title":"AI Crawler Directory","url":"https://bozos.si/reference/ai-crawlers.md","author":null,"published_at":null,"accessed_at":"2026-10-24T11:55:46.031Z","metadata_note":"Publisher-supplied metadata is unverified. Cite the source URL. A hash identifies content; it does not establish truth."},"robots":{"user_agent":"BOZOSReader/1.0 (+https://bozos.si/tools/source-reader)","checked_origins":[{"origin":"https://bozos.si","status":200}],"note":"A robots.txt status of 404 or 410 means the site publishes no restrictions; 200 means its rules were read and allowed this path."},"safety":{"ai_used":false,"scripts_executed":false,"content_retained":false,"network_requests_made":2}}