For three decades, a single text file at the root of your domain has been the primary signal you send to automated crawlers: come in here, stay out of there. That contract held because the major crawlers — Googlebot, Bingbot — chose to honour it. OpenAI has now publicly acknowledged that ChatGPT’s page-fetching bot, the one that retrieves live URLs when a user pastes a link into a conversation, may not honour robots.txt chatgpt fetch bot disallow directives at all. The assumption that a disallow rule equals access control is no longer safe to make.
What the OpenAI Disclosure Actually Says
OpenAI distinguishes between two separate crawling contexts. The first is GPTBot, the training crawler that indexes content to build or update model weights. GPTBot does respect robots.txt, and blocking it via a disallow directive is a recognised and documented option. The second is the ChatGPT fetch bot, which operates in real time when a user explicitly asks ChatGPT to visit a URL or when a plugin or tool triggers a live page retrieval. OpenAI has stated that this second bot may not apply robots.txt rules, because the fetch is considered to be acting on behalf of the user rather than as an autonomous crawler. The practical consequence: any page that is publicly accessible over HTTP or HTTPS can be read by the fetch bot, regardless of what your robots.txt says about it.
Defining the Problem: Two Bots, Two Rule Sets
The distinction matters because most crawl governance strategies treat robots.txt as a single lever that controls all automated access. That model was already imperfect — robots.txt is advisory, not enforced — but the major search engines built their reputations partly on respecting it. The AI crawler landscape is different. Perplexity, Anthropic’s Claude retrieval systems, and Google’s AI Overviews infrastructure each have their own crawl agents with their own stated policies. Assuming uniform behaviour across all of them is the first governance error to correct.
The fetch bot scenario is specifically about on-demand retrieval: a human user triggers the fetch, so the bot argues it inherits the user’s right to read publicly available content. Whether that argument is legally or ethically sound is a separate debate. For a practitioner deciding what to do this week, the operational reality is that disallow directives do not reliably stop it.
Frequently Asked Questions on Robots.txt and AI Crawlers
Question: Does blocking GPTBot in robots.txt stop ChatGPT from reading my pages?
Answer: It stops the training crawler, GPTBot, from indexing your content for model updates. It does not stop the ChatGPT fetch bot from retrieving a page in real time when a user or plugin requests it. These are two separate systems with different stated policies.
Question: Which AI crawlers currently state that they respect robots.txt?
Answer: GPTBot (OpenAI’s training crawler), Google-Extended (Google’s AI training signal), and ClaudeBot (Anthropic’s training crawler) all publish documentation stating they honour robots.txt disallow directives. Real-time retrieval agents — including ChatGPT’s fetch bot and browser-based retrieval in Perplexity — operate under different rules and should be treated separately.
Question: What is llms.txt and does it solve this problem?
Answer: llms.txt is a community-proposed convention: a plain-text file placed at the root of your domain that signals to LLM systems which content you consider suitable for AI use, and which you prefer to exclude. It is not a standard ratified by any body, and compliance is voluntary. It does not enforce access control, but it provides a machine-readable statement of intent that some systems may choose to respect. Think of it as robots.txt for the AI era — with the same advisory limitation.
Question: Can HTTP headers provide stronger access control than robots.txt?
Answer: Yes, within limits. The X-Robots-Tag HTTP response header can carry noindex and nofollow directives and is served at the protocol level rather than as a separate file. Some AI crawlers read it; others do not. For content you genuinely need to protect, authentication — requiring a login or API key before serving the response — is the only technically enforceable barrier. A disallow directive, by contrast, is a request, not a lock.
Question: If I cannot reliably block AI crawlers, should I focus instead on what I want them to read?
Answer: That reframe is useful. Generative engines select a small number of sources — typically three to five — when composing a response. The more extractable, structured, and authoritative your content is, the more likely it is to be among those sources. Governance is not only about exclusion; it is equally about ensuring the right content is readable, well-structured, and associated with your brand entity.
Question: Does the ChatGPT fetch bot disclosure affect how Gemini or Perplexity behave?
Answer: Not directly — each system has its own crawler policy. However, the disclosure is a useful reminder that the AI crawler landscape is fragmented. Google’s AI Overviews infrastructure, Gemini’s retrieval layer, and Perplexity’s real-time search each operate independently. A governance strategy that assumes one rule set covers all of them will have gaps.
A Practical Crawl Governance Audit: Six Steps
Step 1 — Separate training crawlers from retrieval agents in your robots.txt
Review your current robots.txt and identify which user-agent strings you are targeting. Add explicit entries for GPTBot, Google-Extended, ClaudeBot, and PerplexityBot if you have not already. These training crawlers respect disallow directives and can be managed this way. Do not assume a wildcard disallow covers retrieval agents.
Step 2 — Audit which pages are publicly accessible over HTTP/HTTPS
Any page served without authentication is reachable by the ChatGPT fetch bot regardless of your robots.txt. Map your public surface area: product pages, blog posts, pricing pages, documentation. Decide which of these you are comfortable having retrieved and summarised in a live AI conversation.
Step 3 — Apply authentication to content that must not be publicly retrievable
If there is content that should not be accessible to any unauthenticated agent — internal documentation, draft content, sensitive commercial data — the only reliable control is a login gate or API key requirement. A disallow directive is not a substitute for authentication.
Step 4 — Implement X-Robots-Tag headers for finer-grained signals
For pages where you want to signal noindex or nofollow at the HTTP level rather than relying on a separate file, configure your server or CDN to return X-Robots-Tag headers. This is more robust than robots.txt for crawlers that read response headers, and it applies per-URL rather than per-path-pattern.
Step 5 — Consider publishing an llms.txt file
Draft a plain-text file that lists the sections of your site you consider appropriate for AI use, and those you prefer to exclude. Publish it at /llms.txt. It carries no enforcement weight, but it creates a documented, machine-readable statement of intent. As the convention matures, more systems may reference it. It also forces an internal conversation about which content you actively want AI systems to surface.
Step 6 — Shift governance effort toward structured, citable content
Because exclusion controls are unreliable for retrieval agents, the most durable governance strategy is to ensure that the content you want cited is maximally readable. That means clean HTML served in the initial response (not dependent on JavaScript execution), hierarchical headings, short autonomous paragraphs, FAQ blocks, and JSON-LD schema anchoring your brand entity. Generative engines extract chunks; make your chunks worth extracting.
Manual Governance vs. Continuous Auditing: What Each Covers
Consider the ongoing workload. If you monitor four AI systems — ChatGPT, Gemini, Perplexity, and Google AI Overviews — across, say, fifty representative queries, and you repeat that check monthly, that is two thousand four hundred individual data points per year before you account for any changes to crawler policies, new agent types, or shifts in how each system selects sources. Each policy update from OpenAI, Google, or Anthropic potentially invalidates part of your governance map and requires a fresh review cycle.
| Governance layer | Manual approach | Continuous automated auditing |
|---|---|---|
| Training crawler control | Edit robots.txt per user-agent; re-check after each crawler policy update | Automated diff alerts when crawler behaviour deviates from declared policy |
| Retrieval agent exposure | Manual HTTP fetch tests per URL; no systematic coverage | Systematic surface-area mapping across all public URLs |
| Citation monitoring | Ad hoc prompt testing across ChatGPT, Gemini, Perplexity; no baseline | Structured prompt monitoring with share-of-voice tracking over time |
| Schema and entity signals | One-off audit; gaps found manually by inspecting page source | Continuous scoring against entity and structured-data criteria |
| llms.txt and header signals | Manually drafted and rarely revisited | Validated against live crawl behaviour; flagged when out of sync |
A Realistic Scenario: When the Fetch Bot Surfaces the Wrong Page
A B2B software company had blocked its legacy pricing page in robots.txt after a rebrand, assuming it would no longer appear in any automated context. Several months later, a member of their sales team noticed that when prospects pasted the company’s domain into ChatGPT and asked about pricing, the fetch bot was retrieving and summarising the old page — the one with discontinued tiers and outdated figures. The robots.txt disallow had stopped GPTBot from including the page in training data, but it had no effect on the real-time fetch. The diagnosis took the better part of a week: the team had to work out which bot was responsible, confirm that the page was still publicly accessible despite the disallow, and trace why the old URL was still being shared in prospect conversations. The fix — redirecting the legacy URL and placing the correct content behind a clear canonical structure — took an afternoon. The ratio of diagnosis time to implementation time was roughly five to one. Without visibility into what the fetch bot was actually retrieving, the problem would have continued indefinitely.
This pattern repeats across organisations that treat crawl governance as a one-time configuration task. The expensive part is never the fix; it is finding the gap in the first place.
What Sound Crawl Governance Looks Like Now
The robots.txt chatgpt fetch bot disclosure is not a reason to abandon robots.txt — it remains the right tool for managing training crawlers that respect it. It is a reason to stop treating robots.txt as an access control mechanism and to build a layered governance model instead: authentication for content that must not be publicly retrievable, HTTP headers for finer-grained signals, llms.txt as a documented statement of intent, and structured content strategy for everything you actively want AI systems to surface and cite. The structural limitation of any manual version of this work is that crawler policies, agent behaviours, and citation patterns change continuously. A governance map that was accurate in January may have meaningful gaps by April. aisearchaudit.ai runs this audit continuously — tracking how your site is read, retrieved, and cited across ChatGPT, Gemini, Perplexity, and Google AI Overviews — so that gaps surface as signals rather than as surprises in a prospect conversation.
Run your first AI Search Audit report for free
If you want to see exactly where your site stands across the four major AI systems, aisearchaudit.ai runs a full citation audit and returns a structured report with the specific gaps to fix first. Check our plans or contact us for a walkthrough.
Featured image: photo by Tima Miroshnichenko on Pexels.
