Why robots.txt is the right lever
Unlike llms.txt, robots.txt is a genuinely respected standard. The major AI operators publish their crawler names and honor Disallow directives. That means robots.txt is where real control lives, and the key insight is this: the same company usually runs different bots for different jobs.
OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for live user fetches. Anthropic runs ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user fetches. These are controlled independently. Blocking GPTBot does not block ChatGPT search, and blocking ClaudeBot does not block Claude-SearchBot. [1] [2]
This separation is what makes a nuanced policy possible: stay visible in AI answers while opting out of model training.
The three job types
| Job | What it means | Example user-agents |
|---|---|---|
| Training | Content feeds long-term model training | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended |
| Search indexing | Content indexed to be retrieved and cited in AI answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Live user fetch | Bot fetches a page because a user asked | ChatGPT-User, Claude-User, Perplexity-User |
For most businesses that want AI visibility, the policy is: allow search and user fetches, decide separately about training.
Copy-paste: allow AI search, block AI training
This is the "keep me in AI answers, keep me out of training sets" configuration.
```txt
User-agent: OAI-SearchBot # ChatGPT search index — allow
Allow: /
User-agent: ChatGPT-User # live user fetch — allow
Allow: /
User-agent: GPTBot # training — block
Disallow: /
User-agent: Claude-SearchBot # Claude search index — allow
Allow: /
User-agent: Claude-User # live user fetch — allow
Allow: /
User-agent: ClaudeBot # training — block
Disallow: /
User-agent: Google-Extended # Gemini/Vertex training opt-out — block
Disallow: /
User-agent: PerplexityBot # search index — allow
Allow: /
User-agent: Perplexity-User # live user fetch — allow
Allow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Two facts that trip people up
Google-Extended does not stop AI Overviews. Google-Extended only opts your content out of Gemini and Vertex training. AI Overviews in Search draw from the live Googlebot index. To stay out of AI Overviews you would have to block Googlebot, which removes you from Google Search entirely. That is almost never worth it. [3]
Applebot-Extended does not crawl. It is a control token. The live crawler is Applebot; Applebot-Extended is purely the opt-out signal for Apple Intelligence training. Blocking it stops training use without affecting Siri or Spotlight results. [4]
The compliance caveat
robots.txt works because bots choose to honor it. Most declared AI crawlers do. Perplexity is the notable point of friction: Cloudflare documented cases where content was still accessed after PerplexityBot and Perplexity-User were disallowed, via undeclared user-agents and rotating IPs. Perplexity states it respects robots.txt. If you require a hard block rather than a request, enforce it at the CDN or WAF layer, not in robots.txt alone. [5]
Order and syntax notes
One User-agent block per bot is clearest. Grouping is allowed, but explicit blocks are easier to audit.
robots.txt matching is case-insensitive on the user-agent token, but use the canonical casing (OAI-SearchBot, GPTBot) so your file is readable.
Allow: / is the default; you only need it to override a broader Disallow. Keeping it explicit documents intent.
Put robots.txt at your root: https://example.com/robots.txt. Nowhere else works.
FAQ
Does robots.txt actually block AI crawlers?
For declared, well-behaved bots, yes. They read the file and honor Disallow. It is a request, not a firewall, so bad actors can ignore it.
Can I allow ChatGPT search but block training?
Yes. Allow OAI-SearchBot and ChatGPT-User, disallow GPTBot. OpenAI controls these independently.
Does blocking Google-Extended remove me from Google?
No. It only opts you out of Gemini training. Classic Search and AI Overviews are unaffected because they use Googlebot.
Will blocking training crawlers hurt my AI visibility?
Not directly. Citations come from search-indexing and live-fetch bots, which you keep allowed. You lose only training inclusion, whose visibility value is unproven.
Where does robots.txt go?
At the domain root, /robots.txt. Subdirectory copies are ignored.
What if a crawler ignores robots.txt?
Enforce at the CDN or WAF layer with user-agent and IP rules. That is the only hard block.
Get your robots.txt checked
A single wrong line can block the AI search bots you want and leak content to the ones you don't. SEO AEO Specialist audits your robots.txt against every current AI user-agent and flags conflicts. Free at €0 for 50 pages per project, €9 for a one-time report, €19/mo per domain for monitoring. Back to /llms-txt.
Sources: developers.openai.com, support.claude.com, muratulusoy.de, momenticmarketing.com, blog.cloudflare.com
Should I Block AI Crawlers? The Real Trade-Offs (2026)
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →