The mistake in the question
"Should I block AI crawlers" sounds like one decision. It is really two, and conflating them is how sites accidentally make themselves invisible to ChatGPT and Claude search.
The major operators split their crawlers by job:
Training bots collect content to train future models: GPTBot, ClaudeBot, Google-Extended, CCBot.
Search and user-fetch bots retrieve content to answer live questions and cite sources: OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User.
These are independent. Blocking GPTBot does not block ChatGPT search; blocking ClaudeBot does not block Claude-SearchBot. [1] [2] So the honest answer is: block some, keep others, based on what you're trying to protect.
The trade-off in one table
| If you block... | You gain | You lose |
|---|---|---|
| Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) | Content stays out of model training datasets | Little to no visibility, training's citation value is unproven |
| Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) | Content not indexed for AI answers | Citations and referral traffic in AI search |
| User-fetch bots (ChatGPT-User, Claude-User) | Pages not fetched when users ask | The exact live look-ups users trigger about you |
| Everything | Maximum content control | Effective invisibility to AI assistants |
When blocking makes sense
Block training bots if your content is your product and you don't want it feeding competitors' models: paywalled journalism, proprietary research, licensed data, large original datasets. The downside is minimal because training inclusion has no proven link to being cited today.
Block search and user-fetch bots if you have a genuine reason to be absent from AI answers: legal or compliance constraints, private/staging areas, or content you're contractually barred from redistributing. Understand the cost: AI assistants increasingly sit between users and your site, and blocking these bots removes you from that surface.
Block everything only if you're running private infrastructure that should never appear in any AI context. For a normal marketing site, this is self-sabotage.
When you should not block
Most businesses that want to be found should keep search and user-fetch bots open. If a user asks ChatGPT or Perplexity for "best X" and you've blocked OAI-SearchBot and PerplexityBot, you cannot be the answer. That is lost demand you never see in analytics.
A common accidental version: a blanket WAF rule or a copied robots.txt that blocks all bots with "bot" or "AI" in the name. It quietly kills your AI-search presence while you assume everything's fine.
The nuanced default most sites want
Allow search and user-fetch bots. Decide training separately. In robots.txt:
```txt
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This keeps you citable while opting out of training. If you'd rather also feed training, flip the Disallow lines to Allow.
Three facts to weigh
Google-Extended is training-only. Blocking it opts you out of Gemini training but does nothing to AI Overviews, which use the live Googlebot index. Blocking AI Overviews means blocking Googlebot, which means leaving Google Search. [3]
robots.txt is a request. Well-behaved bots honor it. Perplexity has been documented accessing content after being disallowed, via undeclared agents. A true block needs CDN/WAF enforcement. [4]
Training's visibility value is unproven. No provider has confirmed that being in training data helps you get cited. So blocking training is close to a free opt-out on the visibility ledger, whatever your view on data rights.
FAQ
Does blocking AI crawlers hurt SEO?
Blocking AI-specific bots (GPTBot, OAI-SearchBot) does not affect Google rankings, which use Googlebot. It only affects AI-assistant visibility.
Will I disappear from ChatGPT if I block GPTBot?
No. ChatGPT search uses OAI-SearchBot, which is separate. You'd only leave ChatGPT search by blocking OAI-SearchBot too.
Is blocking training crawlers a good idea?
It's a low-cost opt-out. You keep AI-search visibility and simply exclude your content from model datasets. Reasonable if you care about data rights.
Can I block AI crawlers but keep Google?
Yes. AI crawlers use their own user-agents. Google Search uses Googlebot, unaffected by blocking GPTBot, ClaudeBot, or Google-Extended.
What happens if I block everything?
You become effectively invisible to AI assistants: no citations, no referral traffic from AI search. Only do this for private infrastructure.
Do bots always respect my block?
Declared, reputable bots usually do. For guaranteed enforcement, add CDN or WAF rules on top of robots.txt.
Not sure what you're currently blocking?
Most sites don't actually know which AI crawlers they allow or block. SEO AEO Specialist maps your current policy against every AI user-agent and shows the visibility impact of each. Free at €0 for 50 pages per project, €9 one-time report, €19/mo per domain. See the /llms-txt hub.
Sources: developers.openai.com, support.claude.com, muratulusoy.de, blog.cloudflare.com
AI Crawler User Agents List: Master Reference Table (2026)
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →