AI Crawler Checker: Are You Blocking GPTBot, PerplexityBot & ClaudeBot? | SEO AEO Specialist SEO & AEO Specialist

Find out in seconds if your robots.txt is hiding you from AI answer engines

Enter your domain. This tool reads your robots.txt and tells you exactly which AI crawlers you allow and which you block, bot by bot. It catches the most expensive mistake in AEO: quietly blocking the search crawler that would have let ChatGPT or Perplexity cite you, often without meaning to.

Most sites have never audited their robots.txt for AI user agents. Many are blocking crawlers they did not intend to, because they copied a rule from a blog post or their CMS added one. The result is invisibility in AI answers with no error message to warn them.

The critical distinction: search bots vs. training bots

Here is the thing almost everyone gets wrong. A single AI company runs several crawlers that do completely different jobs, and blocking one does not block the others. OpenAI is the clearest example:

• OAI-SearchBot indexes your pages so ChatGPT can retrieve and cite them in search answers. Block this and you vanish from ChatGPT search.

• GPTBot collects data used to train models. Block this and you opt out of training, but not from being cited in search.

• ChatGPT-User fetches a page when a user's prompt triggers a live visit.

These are controlled independently in robots.txt, and blocking GPTBot does nothing to OAI-SearchBot (OpenAI, Am I Cited). Many privacy-minded teams blocked GPTBot to keep their content out of training, which is a legitimate choice, then wrongly assumed they were still visible in ChatGPT search. They were, until someone added a broad rule.

Anthropic runs the same three-way split: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for direct user fetches, each needing its own robots.txt directive (Search Engine Journal). Google uses Google-Extended to let you opt out of Gemini training while staying in Google Search (Search Engine Journal).

Perplexity is the outlier. It runs PerplexityBot for indexing and Perplexity-User for retrieval, but Perplexity has said Perplexity-User is "an agent, not a bot" and is therefore not required to honor robots.txt, a stance Cloudflare has publicly disputed (Anagram). Worth knowing so you set realistic expectations.

How it works

1. Enter your domain. We fetch yourdomain.com/robots.txt.

2. We parse it against the full AI crawler list. Every major training bot, search bot, and user-fetch agent.

3. We show you allow/block status per bot, with a plain-language note on what each one does.

4. We flag the risky ones. Especially any rule blocking a search crawler you probably want to allow, and any overly broad User-agent: * Disallow: / that catches everyone.

What the result tells you

A clean table of every AI user agent and whether your site currently lets it in. More importantly, it separates the bots by intent, so you can make deliberate choices:

• Allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to be cited in AI answers. This is the default you almost certainly want.

• Decide on the training bots (GPTBot, ClaudeBot, Google-Extended) based on your view of your content being used for training. Blocking these does not hurt your AI-search citations.

• Fix the accidental blocks. The report highlights any broad rule silently excluding crawlers you meant to allow.

Search bots vs. training bots at a glance

The pattern: allow the search bots, choose deliberately on the training bots, and never let a broad rule decide for you by accident.

FAQ

What does a robots.txt AI checker do?

It reads your robots.txt file and reports which AI crawlers you currently allow or block, bot by bot, so you can confirm whether ChatGPT, Perplexity, and Claude can reach your site.

What is the difference between a search bot and a training bot?

A search bot (like OAI-SearchBot) indexes your pages so an AI can retrieve and cite them in answers. A training bot (like GPTBot) collects data to train models. They are separate crawlers with separate robots.txt rules, so blocking one does not affect the other.

If I blocked GPTBot, am I invisible in ChatGPT?

Not from GPTBot alone. GPTBot is the training crawler. ChatGPT search citations depend on OAI-SearchBot, which is controlled separately. Check both.

Should I block AI crawlers?

It depends on your goal. If you want to be cited in AI answers, allow the search bots. Blocking training bots is a reasonable privacy choice that does not cost you citations. The checker helps you make each call deliberately.

Does Perplexity respect robots.txt?

PerplexityBot, its indexing crawler, is expected to. But Perplexity has stated its Perplexity-User agent is not required to honor robots.txt, a position Cloudflare has publicly challenged. Set expectations accordingly.

Is this AI crawler checker free?

Yes. Checking your robots.txt against the full AI crawler list is free, no signup.

Check your robots.txt now, then see the full picture

Enter your domain and see exactly which AI crawlers you allow and block. Robots.txt is step one. Getting cited also depends on readability and answer-first structure, which the full audit covers.

Run a free AI visibility audit on up to 50 pages. Get the complete report with every crawler, readability, and structure fix ranked by impact for €9 once. Keep it monitored so a CMS update never silently blocks a bot again for €19/month per domain.

[Run my free AI visibility check →]

Free FAQ Schema Generator (FAQPage JSON-LD) for AEO