Part of our llms.txt guide.
robots.txt controls which crawlers may access which paths and is honored by major bots today, including AI crawlers. llms.txt is a proposed content index that suggests your key pages to models. robots.txt governs access and matters now; llms.txt is aspirational, with no confirmed consumers among major engines.
They look similar and do opposite jobs
Both are plain-text files at your site root. That is where the similarity ends. robots.txt is about permission: it tells bots where they may and may not go. llms.txt is about curation: it suggests what a model should read if it chooses to. One is a gate; the other is a recommended reading list.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Control crawler access | Suggest key content to LLMs |
| Age and status | Established since 1994, widely honored | Proposed Sept 2024, not a standard |
| Located at | /robots.txt | /llms.txt |
| Honored by major bots | Yes, including AI crawlers | No confirmation from major engines |
| Controls access | Yes | No |
| Affects citations | Indirectly (block = no crawl) | No measured effect |
| Matters today | Yes | Aspirational |
robots.txt: governs access, matters now
robots.txt is a decades-old convention that the major AI crawlers respect. This is the file that actually gives you control over AI systems today, and the control is binary: allow or disallow by user-agent and path.
The AI-relevant user-agents include:
GPTBot — OpenAI's training crawler
OAI-SearchBot — OpenAI's search/answers crawler
ChatGPT-User — fetches when a user's prompt triggers a live retrieval
ClaudeBot and Claude-Web — Anthropic
Google-Extended — Google's AI training opt-out token
PerplexityBot — Perplexity
CCBot — Common Crawl, which feeds many datasets
Example: allow normal indexing but block AI training crawlers.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Two cautions. First, blocking training crawlers can also reduce your chances of being cited in AI answers that draw on that data, so it is a genuine tradeoff, not a free privacy win. Second, robots.txt is a request, not a hard wall; well-behaved bots obey it, but it does not enforce anything at the network level. If you must guarantee blocking, use authentication or server-side rules.
llms.txt: aspirational, no confirmed consumers
llms.txt does the opposite job. It cannot allow or deny anything. It only proposes a curated Markdown index of your best pages, in the hope that a model reads it. As covered in what is llms.txt, the adoption picture on the consumption side is weak:
Google said in July 2025 it does not support llms.txt and compared it to the keywords meta tag.
No major provider has committed to using it as a signal.
Crawler logs show AI bots rarely request the file.
So if your goal is to influence AI systems today, robots.txt is the lever that works and llms.txt is a bet on a future that engines have not endorsed.
They are not alternatives
A common confusion is treating these as two options for the same job. They are not. You can and often should have both, and they never conflict:
Use robots.txt to decide who gets in. This is your real control surface for AI crawlers right now.
Use llms.txt, optionally, to hand a tidy index to any model that chooses to read it. Low cost, no confirmed payoff.
If you only do one, do robots.txt correctly, because it governs access and has real consequences. Treat llms.txt as cheap hygiene layered on top.
Quick decision guide
Want to block or allow AI training on your content? robots.txt.
Want to stop a crawler from hammering a section? robots.txt.
Want to offer a clean reading list to cooperative models? llms.txt.
Want a guaranteed citation or ranking lift? Neither file provides one.
FAQ
Is llms.txt a replacement for robots.txt?
No. They do different jobs. robots.txt controls access; llms.txt suggests content. Keep both.
Do AI crawlers respect robots.txt?
Major, well-behaved crawlers like GPTBot, ClaudeBot, and PerplexityBot respect it. It is a request, not enforcement, so bad actors can ignore it.
Can llms.txt block AI crawlers?
No. It has no access-control ability. Use robots.txt (or authentication) to block crawlers.
If I block GPTBot in robots.txt, will I still be cited?
Possibly less. Blocking training crawlers can reduce your presence in the data those systems draw on. It is a tradeoff.
Which file should I prioritize?
robots.txt. It governs access today and has real consequences. llms.txt is optional and unproven.
Do the two files ever conflict?
No. One controls access, the other suggests reading. They operate independently.
SEO AEO Specialist checks how AI crawlers see your site, from robots.txt access rules to how well your pages answer real queries. Start free at €0 for 50 pages per project, get a one-time report for €9, or monitor a domain for €19/mo.
llms-full.txt Explained: What It Is and When to Use It
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →