The critical thing to understand is that each of these AI crawlers is independent. Allowing one does not allow the others, and they serve different jobs. Get the tokens right before you touch anything.
Know your crawler tokens
Different bots do different things. Blocking the wrong one either exposes you to training you did not want or, worse, blocks the search bot you needed:
GPTBot (OpenAI): crawls content for training foundation models. Blocking it opts you out of training, not out of ChatGPT search (developers.openai.com).
OAI-SearchBot (OpenAI): builds the index for ChatGPT search. This is the one you usually want to allow if you want to appear in ChatGPT (Am I Cited).
ChatGPT-User (OpenAI): fetches a live page when a user's action triggers it. Not training.
Google-Extended: controls whether Google uses your content for Gemini training. It does not control classic Search or AI Overviews (No Hacks).
ClaudeBot / Claude-SearchBot / Claude-User (Anthropic): training, search index, and user-triggered fetch respectively, all independent.
PerplexityBot (search index) and Perplexity-User (user-initiated; Perplexity states this is "an agent, not a bot" and may not honor robots.txt) (Anagram).
The common mistake: blocking GPTBot to stop training, then being confused you are absent from ChatGPT search, or the reverse. Treat every token as its own switch.
Cause 1: robots.txt is blocking AI bots
The first place to look. Visit yourdomain.com/robots.txt and read it top to bottom. A 404 means no file, so everything is allowed by default. Look for Disallow: / under any AI user agent, or a broad rule that catches them (Vizup).
Fix: Make sure the tokens you want are explicitly allowed. For appearing in ChatGPT search, allow OAI-SearchBot and ChatGPT-User even if you choose to disallow GPTBot for training. Do the equivalent for the other engines you care about.
Cause 2: Cloudflare is blocking them before robots.txt matters
This one is nasty because it is invisible in your logs. WAF and firewall rules are enforced before robots.txt is even read, so a CDN block overrides any Allow directive. Cloudflare's Bot Fight Mode, on by default on all plans including Free, blocks traffic classified as bots, including legitimate AI crawlers (IndexMachine, Pravin Kumar).
Fix: In Cloudflare, go to Security → Bots → AI Crawl Control to review per-bot request counts and allow/block status. Turn off Bot Fight Mode or, better, add a WAF custom rule that skips bot protection for the AI user agents you want (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended) (Nytro SEO).
Cause 3: A WAF or firewall rule at another layer
Beyond Cloudflare, AWS WAF, other CDNs, or a plugin-based firewall can block AI user agents. Again, this happens before your access logs record anything, so silence in the logs can itself be the tell (tryvizup).
Fix: Review WAF managed rules and bot categories for anything matching AI user agents, and add allow/skip rules for the tokens you want. Coordinate with whoever manages your security layer so a fix does not get reverted.
Cause 4: How to confirm the bots actually arrive
The reliable test is server logs, if requests reach the server. Look for the exact user-agent strings hitting your pages. Presence of OAI-SearchBot or GPTBot entries confirms access; total absence, despite an open robots.txt, points to a CDN or WAF block upstream.
Fix: Grep your access logs for each token. No entries plus an allowing robots.txt equals an upstream block. Fix the layer above, then watch for the bots to start appearing.
Quick self-diagnostic
| Check | What you are looking for | If it fails |
|---|---|---|
| /robots.txt | Disallow: / under an AI token | Allow the tokens you want |
| Cloudflare AI Crawl Control | Bots marked blocked | Disable Bot Fight Mode / add skip rule |
| WAF / firewall rules | Rules matching AI user agents | Add allow/skip for those tokens |
| Server access logs | Presence of the bot user agents | Absence = upstream block to fix |
FAQ
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, and live fetches use ChatGPT-User. To appear in ChatGPT search, allow those even if you block GPTBot (Am I Cited).
My robots.txt allows AI bots but I am still not cited. Why?
A CDN or WAF can block bots before robots.txt is read. Cloudflare's default Bot Fight Mode is a common cause. Check your CDN's bot settings, not just robots.txt (IndexMachine).
Why don't my server logs show any AI crawler visits?
Because a CDN or WAF may be blocking the requests before they reach your server. Absence of log entries alongside an open robots.txt is itself evidence of an upstream block (Vizup).
Does Google-Extended control AI Overviews?
No. Google-Extended controls Gemini training only. It does not govern classic Search or AI Overviews eligibility (No Hacks).
Will Perplexity respect my robots.txt?
PerplexityBot (the index crawler) is expected to. Perplexity-User, which Perplexity calls "an agent, not a bot," may not honor robots.txt, which has caused disputes (Anagram).
What is llms.txt and do I need it?
llms.txt is a proposed file to guide LLMs to your key content. It is optional and not a substitute for fixing robots.txt, CDN, and WAF access, which are what actually determine whether crawlers can reach you.
Check what's blocking your AI crawlers
Tracking a block across robots.txt, Cloudflare, WAF rules, and server logs is fiddly, and the CDN layer hides itself from your logs. SEO AEO Specialist tests all of it and tells you exactly which token is blocked and where.
Free: €0, scan up to 50 pages and get an instant crawl-access check.
One-time report: €9 for a full crawler-access diagnosis across every major AI engine, with fixes.
Ongoing: €19/mo per domain to monitor access continuously so a new Bot Fight Mode toggle or WAF change never quietly cuts you off again.
Run the free audit and confirm AI crawlers can actually reach your site.
All four pages are complete and publish-ready. Summary of what was delivered and the key verified facts baked into each:
Pages written (each ~1,200-1,400 words, all required sections): answer-first capsule, cause-by-cause diagnosis+fix, self-diagnostic table, 6-Q FAQPage-ready block, and tiered CTA (€0/50pp free, €9 report, €19/mo per domain).
why-not-in-ai-overviews - built on the query fan-out mechanic and the verified Ahrefs stat (only 38% of AI Overview citations rank top-10, down from 76%), plus extractability, freshness (current-year injection ~28%), and trust signals.
lost-citations - anchored on verified volatility data (40-60% monthly domain churn; ChatGPT's March 2026 citation cut and May rebound), covering volatility vs staleness vs broken pages vs engine shifts vs new crawl blocks.
competitor-in-chatgpt-not-me - built on third-party consensus (listicles ~41% of recommendations, reviews, Reddit, entity recognition) rather than product quality.
is-my-site-blocking-ai-crawlers - correct crawler tokens throughout (GPTBot = training vs OAI-SearchBot = ChatGPT search vs ChatGPT-User; Google-Extended = Gemini training only, not AI Overviews; Perplexity-User "agent not bot" caveat), plus the Cloudflare Bot Fight Mode default-on gotcha and the WAF-before-robots.txt / invisible-in-logs point.
Voice: practitioner, no hype, no em dashes. All facts cited inline with live sources. Each page is separated by its slug on its own line as requested, and every CTA routes to the free audit.
AEO for Ecommerce: Get Your Products Into AI Shopping Answers
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →