Skip to content
← txtfeed·Directory·llms.txt generator

AI Crawlers reference

The bots and control tokens behind today's AI assistants and answer engines — what each one does, its user-agent, whether it honors robots.txt, and how to allow or block it. Allowing the right crawlers makes your content eligible for AI citations and search; blocking the wrong ones can silently remove you from ChatGPT, Claude, Gemini, and Perplexity. Want to declare your policy in one machine-readable file? Use the llms.txt generator.

OpenAI

GPTBot
Model-training crawler
ChatGPT-User
User-triggered fetcher
OAI-SearchBot
AI-search index crawler

Anthropic

ClaudeBot
Model-training crawler
Claude-User
User-triggered fetcher

Google

Google-Extended
robots.txt control token (training)
Googlebot
Search crawler (feeds AI Overviews)
GoogleOther
General-purpose Google crawler

Perplexity

PerplexityBot
AI-search index crawler
Perplexity-User
User-triggered fetcher

Apple

Applebot
Search crawler (Siri / Spotlight)
Applebot-Extended
robots.txt control token (training)

Amazon

Amazonbot
AI / assistant crawler

ByteDance

Bytespider
Model-training crawler

Meta

Meta-ExternalAgent
Model-training crawler

Common Crawl

CCBot
Open-dataset crawler

DuckDuckGo

DuckAssistBot
AI-answer fetcher

Cohere

cohere-ai
Model crawler
Generate your llms.txt →·Validate a site →·AI SEO guide →·llms.txt directory →