AI crawlers fall into two groups. Training crawlers, like GPTBot, ClaudeBot, Google-Extended, and CCBot, gather content to train models; blocking them keeps your material out of future training but does not stop you being cited. Search and answer crawlers, like OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Bingbot, fetch pages to answer live questions and cite sources in real time.
For most businesses that want AI visibility, the practical recommendation is to allow the search and answer crawlers so you stay eligible to be cited, and to decide on training crawlers separately based on your stance on model training. The two choices are independent.
To check, you need to test your robots.txt against the full list of known AI user-agents, since a single broad rule or a missing one can quietly block or expose more than you intend.
Check your site
Run SEO AEO Specialist's free AI crawler checker to see, in one scan, which AI bots your robots.txt allows or blocks. See the llms.txt & AI crawler guide for the full picture. SEO AEO Specialist diagnoses and fixes, not just monitors, so you know exactly what to change.
FAQ
Should I block AI crawlers?
It depends on your goal. Blocking training crawlers like GPTBot and CCBot keeps your content out of model training, but does not stop AI engines from citing you. Blocking search and answer crawlers like PerplexityBot removes you from AI answers entirely, costing you visibility most businesses want to keep.
How do I check every AI bot at once?
Use SEO AEO Specialist's free AI crawler checker to scan your robots.txt against the full list of known AI user-agents in a single pass. It flags which bots are allowed and which are blocked, so you can spot unintended blocks and fix them rather than testing each crawler by hand.
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →