How to block AI crawlers
Last updated 2026-05-30
You can control AI crawlers with robots.txt — the major operators document that they honor it. Below is a copy-paste list to block every major AI crawler, plus the more useful middle ground: block training while keeping AI-search citations. First, the honest tradeoff.
Block ALL AI crawlers (robots.txt)
Paste into your /robots.txt. This covers the 18 major AI crawlers + control tokens in the txtfeed crawler reference:
User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: Claude-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: Googlebot Disallow: / User-agent: GoogleOther Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Applebot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: CCBot Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: /
Block training only — keep AI-search citations
The recommended middle ground: opt out of model training but stay eligible to be cited in ChatGPT search, Perplexity, and Google AI. This blocks the training crawlers + control tokens while leaving OAI-SearchBot, PerplexityBot, and Googlebot free to crawl:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GoogleOther Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-ai Disallow: /
(Left allowed for citations: ChatGPT-User, OAI-SearchBot, Claude-User, Googlebot, PerplexityBot, Perplexity-User, Applebot, DuckAssistBot.)
Verify the block is working
robots.txt is a politeness signal, not enforcement. Confirm your rule is served, and for crawlers where a hard block matters, add an edge/WAF rule and watch your logs:
# Confirm your robots.txt serves the rule curl -s https://yoursite.com/robots.txt | grep -A1 GPTBot # See whether a crawler is hitting you (check server/edge logs for the UA) # e.g. GPTBot, ClaudeBot, Bytespider
Block a specific crawler
Want to allow some and block others? Each crawler has its own reference page with a tailored snippet and what blocking it actually costs you:
FAQ
How do I block all AI crawlers?
Add a User-agent + Disallow: / block to your robots.txt for each major AI crawler (the full list is below). robots.txt is the standard mechanism and the major operators — OpenAI, Anthropic, Google, Apple, Perplexity, Amazon, Common Crawl — document that they honor it. Note that blocking them removes you from AI search results and citations.
Does blocking AI crawlers hurt my SEO?
Blocking AI-specific crawlers (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot) does NOT affect Google Search ranking — that uses Googlebot. But it removes you from AI answer engines (ChatGPT, Claude, Perplexity) and the referral traffic they drive. If you block Googlebot itself, you disappear from Google Search entirely — don't do that to manage AI.
Can I block AI training but keep AI search citations?
Yes — that's the common middle ground. Block the training crawlers (GPTBot, ClaudeBot, CCBot, and the Google-Extended / Applebot-Extended tokens) while ALLOWING the search/answer crawlers (OAI-SearchBot, PerplexityBot). You opt out of model training but stay eligible to be cited in AI search. Use the 'block training only' snippet below.
Do AI crawlers actually obey robots.txt?
The major operators document that they do, and most comply. Reports of non-compliance exist for a few (e.g. Bytespider). If a strict block matters, pair the robots.txt rule with an edge/WAF rule as a backstop and watch your server logs — robots.txt is a politeness signal, not an enforcement mechanism.
Should I block AI crawlers at all?
For most sites that want visibility, no — AI search is a growing referral channel and blocking removes you from it. Blocking makes sense if you have specific licensing, privacy, or bandwidth reasons. The honest default is to allow the search/answer crawlers and decide on training separately.