This matters because it lets you control which crawlers can access your site and how. Search crawlers, AI training bots, and answer-engine fetchers each use their own user-agent name, so you can allow or disallow them individually in robots.txt. That control shapes whether AI systems are permitted to read and cite your content.
Understanding user-agents is essential for managing AI visibility, since blocking the wrong bot can quietly remove you from AI answers, while allowing the right ones opens the door. It works hand in hand with files like llms.txt that guide how models use your content. SEO AEO Specialist helps you get these signals right. Learn more in llms.txt. Back to the SEO & AEO glossary.
FAQ
How does a user-agent relate to robots.txt?
Robots.txt rules are written per user-agent, so you can allow or disallow each crawler by name. For example, you might permit a search bot while blocking an AI training bot. Matching rules to the correct user-agent string lets you control exactly which systems access your content.
What are some common AI bot user-agents?
Common AI-related user-agents include GPTBot, used by OpenAI, and PerplexityBot, used by Perplexity. Each answer engine and crawler declares its own name so site owners can identify and manage it. Checking your logs and robots.txt for these strings shows which AI systems are reaching your pages.
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →