Allowing is a two-layer job
Most "how do I allow AI crawlers" problems aren't robots.txt problems. robots.txt might say Allow: / while a CDN bot-manager rule or WAF signature quietly returns 403 to the exact bots you want. To actually let AI crawlers in, you have to clear both layers.
Layer 1: robots.txt — the polite request layer. Reputable AI bots read it first.
Layer 2: CDN / WAF — the enforcement layer. This is where silent blocks happen, and it overrides robots.txt because it decides whether the request even gets served.
Layer 1: robots.txt allow rules
You mainly want to allow the bots that produce citations and referral traffic: search-index bots and live user-fetch bots.
```txt
User-agent: OAI-SearchBot # ChatGPT search
Allow: /
User-agent: ChatGPT-User # ChatGPT live fetch
Allow: /
User-agent: Claude-SearchBot # Claude search
Allow: /
User-agent: Claude-User # Claude live fetch
Allow: /
User-agent: PerplexityBot # Perplexity search
Allow: /
User-agent: Perplexity-User # Perplexity live fetch
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Notes:
- Allow: / only matters where a broader Disallow exists. If your file has no site-wide Disallow, these bots are already allowed by default. The value here is being explicit and auditable.
- Watch for a catch-all User-agent: * / Disallow: / earlier in the file. That blocks everything, and per-bot Allow blocks below it are the fix.
- Keep search and user-fetch bots allowed even if you block training bots — that's the standard visibility-preserving setup. [1] [2]
Layer 2: CDN / WAF allowlisting
This is where allow-attempts usually fail. Cloudflare, Akamai, Fastly, AWS WAF and similar often ship bot-management rules that challenge or block non-browser user-agents by default. Your robots.txt says yes; the edge says no.
What to check and fix:
Cloudflare. Bot Fight Mode and "Block AI Bots" / AI Crawl Control settings can block AI user-agents wholesale. If you want specific AI bots in, create an allow rule (WAF custom rule or the AI Crawl Control allowlist) for the user-agents and verified IP ranges you want, and ensure managed bot rules don't override it.
Generic WAF / firewall. Search your rules for anything matching bot, crawler, GPTBot, AI, or empty/non-browser user-agents. A single broad rule like "block user-agents containing bot" catches every AI crawler.
Rate limiting. Aggressive per-IP or per-user-agent rate limits can throttle a crawler into effectively giving up. Exempt the bots you want.
Verify by identity, not just string. User-agent strings can be spoofed, so operators publish verification methods: OpenAI, Anthropic, Google and Perplexity publish official IP ranges (and Google supports reverse-DNS verification) so you can allow the real bot without opening a hole for impersonators.
The verification-before-allowlisting caveat
Allowlisting a raw user-agent string is convenient but weak, because anything can send that string. Where security matters, allow the combination of user-agent plus verified IP range or reverse-DNS. This lets the genuine OAI-SearchBot or Claude-SearchBot in while keeping impostors out. The reverse case matters too: Perplexity has been documented accessing content via undeclared agents after being blocked, so identity-based rules are more reliable than string matching in both directions. [3]
How to confirm it worked
Server logs. Look for successful 200 responses to OAI-SearchBot, Claude-SearchBot, PerplexityBot. If you see 403s or challenges, the edge is still blocking.
Fetch as the bot. curl -A "OAI-SearchBot/1.0" https://example.com/ and check the status. A challenge page or 403 means Layer 2 needs work.
robots.txt tester. Confirm no earlier catch-all Disallow: / shadows your allows.
Live check. Ask ChatGPT, Claude or Perplexity about your page and see whether it can retrieve and cite it.
Don't forget the Google nuance
You don't need to "allow" Google-Extended to appear in Google's AI features — AI Overviews use Googlebot, and Googlebot access is your normal Search access. Allowing Google-Extended only permits Gemini/Vertex training use, which is a separate choice from visibility. [4]
FAQ
How do I allow AI crawlers in robots.txt?
Add a User-agent: block with Allow: / for each bot: OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User. Ensure no earlier catch-all Disallow: / overrides them.
My robots.txt allows them but they still can't reach me. Why?
Almost always a CDN or WAF rule blocking AI user-agents at the edge. That layer overrides robots.txt. Check bot-management settings and allowlist the bots.
Should I allow training bots too?
Optional. Search and user-fetch bots drive citations. Training bots (GPTBot, ClaudeBot) feed model datasets, with unproven visibility value. Allow them if you're comfortable with training use.
How do I safely allowlist without letting impostors in?
Allow user-agent plus verified IP range or reverse-DNS, using the operators' published IP lists, rather than trusting the user-agent string alone.
Does allowing AI crawlers hurt performance?
It adds bot traffic. If load is a concern, allow but rate-limit generously and cache aggressively rather than blocking outright.
How do I test that a bot can reach a page?
Use curl -A with the bot's user-agent, or check server logs for 200 responses. A 403 or challenge means the edge is still blocking.
Confirm the whole path is clear
Allowlisting fails silently: robots.txt looks right, but the CDN quietly blocks. SEO AEO Specialist tests your site end to end against real AI crawler identities and shows exactly where requests get dropped. Free at €0 for 50 pages per project, €9 one-time report, €19/mo per domain for continuous monitoring. Back to /llms-txt.
Sources: developers.openai.com, support.claude.com, blog.cloudflare.com, muratulusoy.de
All 5 cluster pages are complete and publish-ready. Summary of what was delivered:
Pages written (each with SEO title, meta description 150-160 chars, slug, target query, 40-60 word answer capsule, full body with tables/lists/code, 6-Q&A FAQPage, and soft CTA with all three prices):
/llms-txt/does-llms-txt-work — the honest evidence
/llms-txt/robots-txt-for-ai — configuration with copy-paste allow-search/block-training block
/llms-txt/should-i-block-ai-crawlers — trade-offs
/llms-txt/ai-crawler-list — master user-agent reference table
/llms-txt/allow-ai-crawlers — two-layer allowlist guide
Key verified facts baked in:
- Google's stance: Illyes (no support, Search Central Live 2025), Mueller's "meta keywords" comparison and blunt Bluesky "no", Google's June-2026 guidance that no AI/Markdown files are needed.
- Evidence: SE Ranking 300k-domain study (10.13% adoption), Ahrefs 137k domains (~97% of llms.txt files get zero requests), 500M+ bot visits with ~408 hitting llms.txt, OtterlyAI ~0.1%.
- Crawler tokens kept correct and distinct: GPTBot (training) vs OAI-SearchBot (search) vs ChatGPT-User (live); ClaudeBot (training) vs Claude-SearchBot (search) vs Claude-User (live); Google-Extended = Gemini/Vertex training opt-out only, does NOT stop AI Overviews (those use Googlebot); Applebot-Extended is a signal that doesn't crawl; CCBot = Common Crawl (corrected from a search-result error that mislabeled it Meta); Meta-ExternalAgent, Bytespider (ByteDance), Amazonbot, PerplexityBot/Perplexity-User.
- Perplexity compliance caveat (Cloudflare's undeclared-agent findings) included, plus the CDN/WAF enforcement point.
Voice constraints honored: no em dashes, no hype, inline citations, all pages link up to /llms-txt, and none duplicate the diagnostic "is my site blocking AI crawlers" page (these stay reference/how-to). Each runs ~1,000-1,400 words.
One correction I made against the raw search output: one result claimed CCBot crawls for Meta AI/Llama — that's inaccurate. CCBot is Common Crawl's crawler; its archive is used by many model builders. I labeled it accordingly.
What Are AI Citations? Definition and Why They Matter in 2026
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →