{
  "answer": "7 results for \"GPTBot\". Best: AI crawler directory dataset — GPTBot (https://bozos.si/ai-crawlers.json). Open the page before citing it.",
  "query": "GPTBot",
  "terms": [
    "gptbot"
  ],
  "results": [
    {
      "resource_id": "ai-crawler-directory-data",
      "title": "AI crawler directory dataset",
      "section": "GPTBot",
      "url": "https://bozos.si/ai-crawlers.json",
      "type": "application/json",
      "kind": "reference-data",
      "updated": "2026-10-20",
      "score": 17,
      "snippet": "GPTBot (OpenAI): Crawls content that may be used to train OpenAI's generative AI foundation models. robots.txt: obeys. Used for AI training: yes. Source: BOZOS, AI Crawler Directory, verified 2026-10-20, high confidence. https://bozos.si/ai…",
      "record": {
        "id": "gptbot",
        "collection": "crawlers",
        "permalink": "https://bozos.si/ai-crawlers.json#gptbot",
        "cite": "GPTBot (OpenAI): Crawls content that may be used to train OpenAI's generative AI foundation models. robots.txt: obeys. Used for AI training: yes. Source: BOZOS, AI Crawler Directory, verified 2026-10-20, high confidence. https://bozos.si/ai-crawlers.json#gptbot",
        "fetch_id": "ai-crawler-directory-data#gptbot"
      }
    },
    {
      "resource_id": "ai-crawler-directory-guide",
      "title": "AI Crawler Directory",
      "section": "OpenAI",
      "url": "https://bozos.si/reference/ai-crawlers",
      "type": "text/html",
      "kind": "reference",
      "markdown_url": "https://bozos.si/reference/ai-crawlers.md",
      "data_url": "https://bozos.si/ai-crawlers.json",
      "updated": "2026-10-20",
      "score": 4,
      "snippet": "- GPTBot (training; robots.txt: obeys; may feed AI training; high confidence). Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot signals the site's content should not be used for training.…"
    },
    {
      "resource_id": "ai-optout-signals-data",
      "title": "AI opt-out signals dataset",
      "section": "openai",
      "url": "https://bozos.si/ai-optout-signals.json",
      "type": "application/json",
      "kind": "reference-data",
      "updated": "2026-10-20",
      "score": 4,
      "snippet": "OpenAI: robots-txt-training-optout honors (Disallow GPTBot in robots.txt); robots-meta-ai not-stated; noai-meta not-stated; content-signal not-stated; aipref-content-usage not-stated; tdmrep not-stated; rsl not-stated. Source: BOZOS, AI Opt…",
      "record": {
        "id": "openai",
        "collection": "operators",
        "permalink": "https://bozos.si/ai-optout-signals.json#openai",
        "cite": "OpenAI: robots-txt-training-optout honors (Disallow GPTBot in robots.txt); robots-meta-ai not-stated; noai-meta not-stated; content-signal not-stated; aipref-content-usage not-stated; tdmrep not-stated; rsl not-stated. Source: BOZOS, AI Opt-out Signals: Who Honors What, verified 2026-10-20. https://bozos.si/ai-optout-signals.json#openai",
        "fetch_id": "ai-optout-signals-data#openai"
      }
    },
    {
      "resource_id": "ai-crawler-directory-guide",
      "title": "AI Crawler Directory",
      "url": "https://bozos.si/reference/ai-crawlers",
      "type": "text/html",
      "kind": "reference",
      "markdown_url": "https://bozos.si/reference/ai-crawlers.md",
      "data_url": "https://bozos.si/ai-crawlers.json",
      "updated": "2026-10-20",
      "score": 3,
      "snippet": "Canonical: https://bozos.si/reference/ai-crawlers Data: https://bozos.si/ai-crawlers.json Verified: 2026-10-20 · License: CC0-1.0  Crawlers and user agents that AI and search companies document themselves. Every entry in crawlers is taken f…"
    },
    {
      "resource_id": "ai-optout-signals-guide",
      "title": "AI Opt-out Signals: Who Honors What",
      "section": "What each operator says",
      "url": "https://bozos.si/reference/ai-optout-signals",
      "type": "text/html",
      "kind": "reference",
      "markdown_url": "https://bozos.si/reference/ai-optout-signals.md",
      "data_url": "https://bozos.si/ai-optout-signals.json",
      "updated": "2026-10-20",
      "score": 3,
      "snippet": "- OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot) robots-txt-training-optout: honors. Disallow GPTBot in robots.txt. \"Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.\" (h…"
    },
    {
      "resource_id": "ai-optout-signals-guide",
      "title": "AI Opt-out Signals: Who Honors What",
      "section": "Signals",
      "url": "https://bozos.si/reference/ai-optout-signals",
      "type": "text/html",
      "kind": "reference",
      "markdown_url": "https://bozos.si/reference/ai-optout-signals.md",
      "data_url": "https://bozos.si/ai-optout-signals.json",
      "updated": "2026-10-20",
      "score": 1,
      "snippet": "…anges how already-crawled data is used. Syntax: User-agent: GPTBot / Disallow: /. Spec: https://www.rfc-editor.org/rfc/rfc9309.html Spec says: \"These rules are not a form of access authorization.\" - Robots meta tags as AI controls (nosnippe…"
    },
    {
      "resource_id": "ai-optout-signals-data",
      "title": "AI opt-out signals dataset",
      "section": "robots.txt opt-out from AI training",
      "url": "https://bozos.si/ai-optout-signals.json",
      "type": "application/json",
      "kind": "reference-data",
      "updated": "2026-10-20",
      "score": 1,
      "snippet": "robots.txt opt-out from AI training (ietf-standard, IETF (RFC 9309) plus each operator's own user-agent token): Whether a named crawler may fetch a path. Operators decide what blocking their token means; some offer a token only for AI train…",
      "record": {
        "id": "robots-txt-training-optout",
        "collection": "signals",
        "permalink": "https://bozos.si/ai-optout-signals.json#robots-txt-training-optout",
        "cite": "robots.txt opt-out from AI training (ietf-standard, IETF (RFC 9309) plus each operator's own user-agent token): Whether a named crawler may fetch a path. Operators decide what blocking their token means; some offer a token only for AI training, or a control token that changes how already-crawled data is used. Source: BOZOS, AI Opt-out Signals: Who Honors What, updated 2026-10-20. https://bozos.si/ai-optout-signals.json#robots-txt-training-optout",
        "fetch_id": "ai-optout-signals-data#robots-txt-training-optout"
      }
    }
  ],
  "deterministic": true,
  "docs": "https://bozos.si/tools/fetch#search"
}