Google-Extended: what it is & how to allow or block it
Google-Extended is not a separate crawler — it is a robots.txt user-agent token that controls whether your already-crawled content may be used to train and ground Google's generative AI (Gemini / Vertex AI). Google honors the Google-Extended token in robots.txt. It does not change how Googlebot crawls or how you rank in Search.
- User-agent
Google-Extended- Operator
- Type
- robots.txt control token (training)
- robots.txt
- Google honors the Google-Extended token in robots.txt. It does not change how Googlebot crawls or how you rank in Search.
- Official docs
- developers.google.com/search/docs/crawling-index
What allowing or blocking Google-Extended means
Disallowing Google-Extended opts your content out of Gemini training/grounding while keeping normal Google Search indexing intact. It does not remove you from Google Search.
Opt out of Google-Extended (robots.txt)
# Opt out of Google-Extended User-agent: Google-Extended Disallow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set Google-Extended to allowed or restricted and emits a compliant file.
FAQ
Does Google-Extended affect my Google Search ranking?
No. Google-Extended only controls generative-AI (Gemini) training and grounding use. Search crawling and ranking are governed by Googlebot and are unaffected.
Is Google-Extended a crawler I will see in my logs?
No. It is a control token, not a fetching user-agent. Google does not crawl with a 'Google-Extended' user-agent; it reads the token from your robots.txt.