llms.txt & AI Crawler Access: The Complete Guide

What is llms.txt?

llms.txt is a file you place at your domain root (/llms.txt) containing a curated, markdown-formatted map of your key content, so that language models can find and parse your important pages without wading through navigation, ads, and boilerplate. The idea is reasonable on its face: give models a clean, structured index the way a sitemap gives search crawlers one. We cover the format in detail in what is llms.txt and the how-to in how to create llms.txt.

The problem is not the idea. It is that almost nothing consumes the file.

Does llms.txt actually work? The evidence

Be skeptical. Here is what the data shows as of 2026:

• No major engine supports it. In July 2025, Google's Gary Illyes confirmed Google does not support llms.txt and has no plans to, and John Mueller compared it to the discredited keywords meta tag (Search Engine Journal). No major provider, OpenAI, Anthropic, Google, Meta, or Mistral, has publicly committed to using it as a signal in production.

• No citation lift. An analysis of roughly 300,000 domains found llms.txt adoption is low and has no statistically significant correlation with AI citation frequency. When researchers removed llms.txt from their predictive model, accuracy actually improved (Search Engine Journal).

• Crawlers barely touch it. Monitoring of over 500M AI bot visits across a 90-day window found only 408 requests that targeted llms.txt directly, negligible against total AI crawler traffic (Search Engine Journal).

• Adoption is thin. Crawls put llms.txt on only around 10% of domains (Search Engine Journal).

Bottom line: do not expect llms.txt to move your AI visibility today. It is low-cost to add and harmless, so treat it as an optional bet on future adoption, not as a lever. Our full skeptical breakdown is in should I block AI crawlers and the spec review in what is llms.txt.

What actually governs AI crawler access

While llms.txt sits mostly unused, robots.txt genuinely controls whether AI bots can fetch your pages, and that has real consequences. AI crawlers generate more than 50 billion requests per day across Cloudflare's network alone (Cloudflare), and AI search visits grew 42.8% year over year, from 15.6 billion to 27.4 billion between Q1 2025 and Q1 2026 (Cloudflare via search results).

The key insight most sites miss: training bots and search bots are different, and you can allow one while blocking the other.

OpenAI's own documentation states operators can allow OAI-SearchBot while disallowing GPTBot (search results). That is the canonical move: stay visible in ChatGPT Search without feeding foundation-model training. The full roster is in the AI crawler list.

robots.txt for AI: the pattern that matters

If your goal is maximum AI-answer visibility, allow the search and user-action bots and only block the training crawlers you object to. A common allow-search, block-training pattern looks like this:

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: CCBot

Disallow: /

Two cautions. First, robots.txt is a request, not a wall: Cloudflare published evidence in August 2025 that Perplexity used undeclared crawlers circumventing robots.txt directives (search results), so enforcement varies by operator. Second, an over-eager block can quietly delete you from the exact AI answers you want to appear in. The full walkthrough is in robots.txt for AI.

Should you block AI crawlers at all?

There is no universal right answer, and the decision is strategic:

• Block training bots if your content is proprietary, paywalled, or you simply do not want it in training corpora. GPTBot, ClaudeBot, CCBot, Google-Extended.

• Allow search bots if you want to appear in AI answers, which for most marketing sites is the entire point. OAI-SearchBot, PerplexityBot, Claude-SearchBot.

• The common mistake is a blanket block that catches search bots too, silently removing you from ChatGPT Search, Perplexity, and Claude answers. In a world where 93% of AI Mode searches end without a click (Semrush via Pasquale Pillitteri), being absent from the answer is a real cost.

Work through the trade-offs in should I block AI crawlers.

Where SEO AEO Specialist fits

Crawler access is the kind of setting that fails silently. A single overbroad Disallow can cut you out of AI answers for months before anyone notices, because there is no error, just absence. SEO AEO Specialist audits your robots.txt and crawler configuration, flags whether you are accidentally blocking search bots, checks your llms.txt if you have one, and returns the exact directives to fix it.

Monitor-only tools like Profound, at roughly $99 to $499/month, report your AI visibility. They do not tell you that a stray robots.txt line is the cause, and they do not hand you the corrected file. SEO AEO Specialist diagnoses and fixes from €9.

Pricing

• Free (€0): audit up to 50 pages, including crawler-access checks, no card.

• One-time report (€9): full audit plus the exact robots.txt and access fixes.

• €19/month per domain: continuous monitoring so a bad deploy does not silently block AI bots.

FAQ