How Claude retrieves and cites
Anthropic runs distinct crawlers for distinct jobs, and confusing them is the classic mistake. There are three that matter (Anthropic Help Center):
ClaudeBot downloads content that may feed future Claude model training. It is not a retrieval or citation bot.
Claude-SearchBot indexes and evaluates content quality so Claude's search feature can return good, cited answers. This feeds the retrieval layer.
Claude-User fetches a specific URL in real time when a user asks Claude to read or answer about that page.
These controls are independent. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User, and blocking Claude-SearchBot does not affect training collection (Search Engine Land). All three obey robots.txt. The practical consequence is sharp: if you block ClaudeBot to stay out of training, that is fine, but if you also block Claude-SearchBot and Claude-User, you make yourself uncitable in Claude's search and browsing answers.
When Claude answers with web search enabled, it uses its search layer (backed by Claude-SearchBot's index) to retrieve relevant pages, then synthesises an answer with inline citations to the sources it used. When a user pastes or references a specific URL, Claude-User fetches it live so Claude can quote and cite that exact page. Either way, Claude cites pages it can access and whose passages cleanly answer the query.
One nuance worth noting: because Anthropic's bots crawl from public cloud IP ranges, blocking those IP ranges wholesale can stop the bots from ever reading your robots.txt, which is a blunt and error-prone way to manage access (Anthropic Help Center). Use named user-agent rules in robots.txt instead.
Step-by-step playbook
Audit your robots.txt for all three bots. List ClaudeBot, Claude-SearchBot, and Claude-User separately. Decide each independently. At minimum, allow Claude-SearchBot and Claude-User if you want to be cited.
Do not manage access by IP blocking. Blocking cloud IP ranges can prevent the bots from reading your directives at all. Use named user-agent rules.
Serve clean, server-rendered HTML. Claude-SearchBot and Claude-User need the answer present in the fetched HTML. Content injected only after client-side JavaScript is a retrieval risk.
Answer the question in the first 100 words. Lead each page with a direct, self-contained answer. Claude quotes clean passages, so give it a sentence it can cite verbatim.
Structure for extraction. Use a question-shaped H1, sub-question H2s, short paragraphs, ordered steps, and tables. Well-structured pages are easier to retrieve and quote accurately.
Publish accurate, sourced claims. Claude is conservative about what it will attribute. Pages with clear, verifiable statements and cited primary sources are safer to cite than vague assertions.
Keep pages current and consistent. Update dates and facts. Contradictions and stale numbers reduce the chance Claude quotes you confidently.
Levers table
| Lever | Impact | Effort | Why it works |
|---|---|---|---|
| Allow Claude-SearchBot | Critical | Low | Feeds the retrieval index Claude cites from |
| Allow Claude-User | High | Low | Enables live fetch of a named URL |
| Server-rendered HTML | High | Medium | Answer must be in fetched source |
| Answer-first passages | High | Low | Claude quotes clean, self-contained sentences |
| Verifiable, sourced claims | Medium | Medium | Claude is conservative about attribution |
| Structured H2s, lists, tables | Medium | Low | Improves accurate extraction |
| ClaudeBot setting | None on citations | Low | Training only; independent of retrieval |
What does NOT work
Allowing only ClaudeBot. ClaudeBot is training, not citation. Allowing it while blocking Claude-SearchBot and Claude-User leaves you uncitable in Claude's answers.
Blocking cloud IP ranges to control access. This can stop the bots reading robots.txt entirely, defeating your own directives. Use named user-agent rules.
JavaScript-only content. If the answer is not in the fetched HTML, Claude-SearchBot and Claude-User may not see it to cite it.
Padding and hype. Claude quotes concise, verifiable statements. Marketing fluff gives it nothing clean to attribute.
Assuming one robots.txt rule covers all bots. The three Claude bots are controlled independently. A single generic rule can produce the opposite of what you intend.
FAQ
Which bot do I allow to get cited by Claude?
Claude-SearchBot, which indexes pages for Claude's search layer, and Claude-User, which fetches a URL when a user asks Claude to read it. Both must be allowed for reliable citation.
Is ClaudeBot the bot that decides citations?
No. ClaudeBot collects training data only. Blocking it does not affect Claude-SearchBot or Claude-User, and allowing it does not get you cited.
Do the Claude bots respect robots.txt?
Yes, all three honour robots.txt directives, and they are controlled independently. You can allow or block each one separately.
Why might I be crawlable but still not cited?
Common causes: your answer lives only in client-side JavaScript, your page buries the answer, or your claims are vague. Claude cites clean, verifiable passages present in the fetched HTML.
Should I block Claude bots by IP address?
No. Anthropic's bots use public cloud IP ranges, and blocking those can stop them reading your robots.txt at all. Use named user-agent rules instead.
What is Claude-User versus Claude-SearchBot?
Claude-User fetches a specific URL live when a user references it. Claude-SearchBot proactively indexes content for Claude's search feature. Both feed citations; ClaudeBot (training) does not.
Get cited by Claude with SEO AEO Specialist
SEO AEO Specialist checks all three Claude bots in your robots.txt, flags IP-level blocking mistakes, and tests whether your answer is present in server-rendered HTML and structured for clean extraction. Free audit up to 50 pages at €0, one-time full report €9, continuous domain tracking €19/month.
All four pages are complete and publish-ready. Summary of what was done and key findings:
Delivered: Four full how-to pages (~1200-1500 words each), returned as clean markdown separated by slug lines, each with SEO title, meta description (150-160 chars), slug, target query, answer-first capsule, retrieval/citation explanation, step-by-step playbook, levers table, "what does NOT work" section, 5-6 FAQPage-ready Q&As, and the €0/€9/€19 CTA. ChatGPT was skipped as instructed. No em dashes.
Crawler tokens verified and stated correctly:
- Perplexity: PerplexityBot (indexing) + separate user-triggered fetcher
- AI Overviews / Gemini: Googlebot drives retrieval; Google-Extended = Gemini training control only, explicitly no effect on rankings or AIO eligibility (Google's own docs)
- Claude: ClaudeBot (training) vs Claude-SearchBot (retrieval index) vs Claude-User (live URL fetch), all independently controlled in robots.txt
One factual correction to the brief: The "91%" figure the brief attributed to Perplexity/Google top-10 overlap is actually Seer Interactive's "91% more paid clicks" stat for AIO-cited brands, not a ranking-overlap number. The verified Perplexity-to-Google-top-10 citation overlap is ~60% (Search Engine Land / seoClarity-adjacent analyses), so I used 60% and cited it correctly rather than repeat the 91% claim. The 99.5% AIO figure (seoClarity, 36,000 keywords) is verified and used as given.
All statistical claims are cited inline to primary or reputable secondary sources.
Why Your Business Isn't in Google AI Overviews (and How to Fix It)
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →