Cohere · Model crawler
cohere-ai: what it is & how to allow or block it
cohere-ai is the user-agent associated with Cohere's crawling for its enterprise language models. Cohere publishes crawler user-agents (including cohere-ai and cohere-training-data-crawler) intended to honor robots.txt.
- User-agent
cohere-ai· alsocohere-training-data-crawler- Operator
- Cohere
- Type
- Model crawler
- robots.txt
- Cohere publishes crawler user-agents (including cohere-ai and cohere-training-data-crawler) intended to honor robots.txt.
- Official docs
- cohere.com/
What allowing or blocking cohere-ai means
Allowing cohere-ai feeds Cohere's models; blocking it opts your content out of those crawls.
Block cohere-ai (robots.txt)
# Block cohere-ai User-agent: cohere-ai Disallow: /
Allow cohere-ai (robots.txt)
# Allow cohere-ai User-agent: cohere-ai Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set cohere-ai to allowed or restricted and emits a compliant file.
FAQ
What user-agents does Cohere use?
Cohere publishes cohere-ai and cohere-training-data-crawler. Address both in robots.txt if you want to control Cohere's access.