Many sites accidentally block the AI search crawlers that create citations while trying to block AI training crawlers - or block everything with an overzealous WAF rule. The fix: allow OAI-SearchBot, PerplexityBot, and Claude-SearchBot (search/visibility) and decide separately about GPTBot, ClaudeBot, and Google-Extended (training). Each token is independent.
Blocking the wrong bot is the most common AEO mistake. Two categories:
Per OpenAI's docs, these settings are independent. You can allow ChatGPT Search while opting out of training - or vice versa.
You can look up any of these in the glossary if a token is unfamiliar.
Open https://yourdomain.com/robots.txt. Look for any User-agent: * with a broad Disallow: /, or explicit blocks of the tokens above. The robots.txt AI checker does this parse for you and flags stray blocks.
Most businesses that want AEO visibility should allow all search agents. Training opt-out is a separate, optional choice (e.g. for IP-sensitive content).
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlUser-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This keeps you citable in ChatGPT Search while excluding you from training.
Cloudflare "bot fight mode," AWS WAF, or security plugins often block AI agents by user-agent or IP before robots.txt is even consulted. Allowlist the AI bots at that layer too. OpenAI publishes IP ranges at openai.com/searchbot.json and openai.com/gptbot.json for allowlisting.
Disallow.Sitemap: line is present and valid.User-agent: * / Disallow: / left over from staging - blocks everyone.Once access is sorted, the next lever is being findable in answers - see how AI citations build. And to confirm the whole chain end to end, run a free audit of up to 50 pages: SEO AEO Specialist by MrChief reads your robots.txt, cross-checks your CDN/WAF, and tells you which AI engines can actually reach you.
No. GPTBot is training-only. ChatGPT Search uses OAI-SearchBot. Blocking GPTBot keeps you out of training but keeps you eligible for citations if OAI-SearchBot is allowed. These two tokens are fully independent, so one decision never dictates the other.
OpenAI, Anthropic, and Perplexity state they honor robots.txt for their crawlers. The exception is user-initiated fetches (ChatGPT-User, Claude-User), which may not apply robots.txt because a human triggered the request rather than an automated crawl.
Negligibly for most sites - these are standard crawl loads. There's no SEO penalty; if anything, being crawlable expands your AI visibility. If load is a concern, allowlist the bots at your CDN rather than blocking them outright.
Google-Extended controls Gemini training only. It does not affect Google Search or AI Overviews - those use Googlebot. Blocking Google-Extended won't remove you from AI Overviews, so keep the two tokens separate in your policy.
That's a valid "no training, yes citations" stance: Disallow ClaudeBot, Allow Claude-SearchBot. Anthropic honors both tokens independently, so you can opt out of training while remaining retrievable in Claude's search-driven answers.