GPTBot: what it is & how to allow or block it
GPTBot is OpenAI's crawler that gathers publicly available web content used to help train and improve OpenAI's foundation models (the GPT family). OpenAI documents that GPTBot honors robots.txt. Disallowing GPTBot removes your content from future model-training datasets.
- User-agent
GPTBot- Operator
- OpenAI
- Type
- Model-training crawler
- robots.txt
- OpenAI documents that GPTBot honors robots.txt. Disallowing GPTBot removes your content from future model-training datasets.
- Official docs
- platform.openai.com/docs/bots
What allowing or blocking GPTBot means
Allowing GPTBot can improve how accurately OpenAI models describe your domain; blocking it opts your content out of training but does not affect ChatGPT's live browsing (ChatGPT-User) or search (OAI-SearchBot).
Block GPTBot (robots.txt)
# Block GPTBot User-agent: GPTBot Disallow: /
Allow GPTBot (robots.txt)
# Allow GPTBot User-agent: GPTBot Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set GPTBot to allowed or restricted and emits a compliant file.
FAQ
How do I block GPTBot?
Add a User-agent: GPTBot block with Disallow: / to your robots.txt, and mirror the restriction in your llms.txt. OpenAI honors robots.txt for GPTBot.
Is GPTBot the same as ChatGPT browsing?
No. GPTBot crawls for model training. Live, on-demand fetches when a ChatGPT user asks it to open a page use the separate ChatGPT-User agent, and ChatGPT search uses OAI-SearchBot.