Googlebot: what it is & how to allow or block it
Googlebot is Google's primary search crawler. Its index also powers Google's AI Overviews / AI Mode, so blocking it removes you from both Search and those AI answers. Googlebot strictly honors robots.txt. Blocking it removes your content from Google Search entirely.
- User-agent
Googlebot- Operator
- Type
- Search crawler (feeds AI Overviews)
- robots.txt
- Googlebot strictly honors robots.txt. Blocking it removes your content from Google Search entirely.
- Official docs
- developers.google.com/search/docs/crawling-index
What allowing or blocking Googlebot means
Allowing Googlebot is essential for Search visibility and AI Overview eligibility. To opt out of Gemini training while keeping Search, use the Google-Extended token instead of blocking Googlebot.
Block Googlebot (robots.txt)
# Block Googlebot User-agent: Googlebot Disallow: /
Allow Googlebot (robots.txt)
# Allow Googlebot User-agent: Googlebot Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set Googlebot to allowed or restricted and emits a compliant file.
FAQ
Can I appear in Google Search but not AI Overviews?
There is no separate opt-out for AI Overviews; they draw on the Search index. You can, however, use Google-Extended to opt out of Gemini model training while keeping Search and AI Overview eligibility.