OpenAI · AI-search index crawler
OAI-SearchBot: what it is & how to allow or block it
OAI-SearchBot builds the search index that powers ChatGPT's search features, surfacing and linking sources inside answers. OpenAI documents that OAI-SearchBot honors robots.txt. It is distinct from the training crawler (GPTBot).
- User-agent
OAI-SearchBot- Operator
- OpenAI
- Type
- AI-search index crawler
- robots.txt
- OpenAI documents that OAI-SearchBot honors robots.txt. It is distinct from the training crawler (GPTBot).
- Official docs
- platform.openai.com/docs/bots
What allowing or blocking OAI-SearchBot means
Allowing OAI-SearchBot makes your pages eligible to appear and be linked as sources in ChatGPT search results — a growing referral channel.
Block OAI-SearchBot (robots.txt)
# Block OAI-SearchBot User-agent: OAI-SearchBot Disallow: /
Allow OAI-SearchBot (robots.txt)
# Allow OAI-SearchBot User-agent: OAI-SearchBot Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set OAI-SearchBot to allowed or restricted and emits a compliant file.
FAQ
Does blocking GPTBot also block ChatGPT search?
No. ChatGPT search indexing uses OAI-SearchBot. You can allow OAI-SearchBot (to appear in search) while disallowing GPTBot (to opt out of training).