CCBot: what it is & how to allow or block it
CCBot is Common Crawl's crawler. Its free, open dataset is one of the most widely used training corpora across the entire LLM industry. Common Crawl documents that CCBot honors robots.txt. Because so many models train on Common Crawl, blocking CCBot has broad downstream effect.
- User-agent
CCBot- Operator
- Common Crawl
- Type
- Open-dataset crawler
- robots.txt
- Common Crawl documents that CCBot honors robots.txt. Because so many models train on Common Crawl, blocking CCBot has broad downstream effect.
- Official docs
- commoncrawl.org/ccbot
What allowing or blocking CCBot means
Allowing CCBot puts your content into the dataset many models train on (high reach, no attribution). Blocking it is one of the broadest single opt-outs from AI training.
Block CCBot (robots.txt)
# Block CCBot User-agent: CCBot Disallow: /
Allow CCBot (robots.txt)
# Allow CCBot User-agent: CCBot Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set CCBot to allowed or restricted and emits a compliant file.
FAQ
Why does blocking CCBot matter so much?
Common Crawl's open dataset is reused by a large share of the AI industry, so a single CCBot block opts you out of many downstream training pipelines at once.