ClaudeBot: what it is & how to allow or block it
ClaudeBot is Anthropic's crawler that collects publicly available web data used to help train and improve the Claude models. Anthropic documents that ClaudeBot respects robots.txt. Disallowing it opts your content out of Claude training crawls.
- User-agent
ClaudeBot· alsoanthropic-ai,Claude-Web- Operator
- Anthropic
- Type
- Model-training crawler
- robots.txt
- Anthropic documents that ClaudeBot respects robots.txt. Disallowing it opts your content out of Claude training crawls.
- Official docs
- support.anthropic.com/en/articles/8896518
What allowing or blocking ClaudeBot means
Allowing ClaudeBot helps Claude describe your domain accurately; blocking it opts out of training but does not affect on-demand fetches (Claude-User).
Block ClaudeBot (robots.txt)
# Block ClaudeBot User-agent: ClaudeBot Disallow: /
Allow ClaudeBot (robots.txt)
# Allow ClaudeBot User-agent: ClaudeBot Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set ClaudeBot to allowed or restricted and emits a compliant file.
FAQ
How do I block ClaudeBot?
Add a User-agent: ClaudeBot block with Disallow: / in robots.txt. Note that anthropic-ai and Claude-Web are older Anthropic user-agent tokens you may also choose to address.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot crawls for training. Claude-User fetches a page on demand when a Claude user asks it to read a specific URL.