Google-Extended is a robots.txt token, not a distinct crawler. It doesn't fetch pages itself; it's a control signal that tells Google whether your content may be used to train and ground its generative AI models such as Gemini and Vertex AI, separately from standard Search indexing.
Blocking Google-Extended has no effect on your normal Search ranking — Googlebot still indexes you — but it may limit how your content is used in AI-generated experiences.
How to check and control it
To check, look in your robots.txt for a rule targeting this user-agent. To allow or block it:
User-agent: Google-Extended
Disallow: / # blocks it — remove or set "Allow: /" to permitFor an instant check across your whole site, use SEO AEO Specialist's free AI crawler checker, and see the llms.txt & AI crawler guide.
FAQ
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only governs generative AI use of your content. Google Search indexing is handled by Googlebot, which is a separate rule. You can block Google-Extended and stay fully ranked in Search while opting out of AI training.
Is Google-Extended a real crawler?
No. It's a robots.txt token, not a crawler with its own user-agent traffic. Google reads the token to decide whether your content may train or ground Gemini and Vertex AI. SEO AEO Specialist flags whether your token is set and explains the trade-off, not just monitors it.
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →