Meta-ExternalAgent: what it is & how to allow or block it

Meta-ExternalAgent is Meta's crawler used to collect public web data for training and improving its AI products (the Llama family and Meta AI). Meta documents that Meta-ExternalAgent honors robots.txt. A related on-demand fetcher is Meta-ExternalFetcher.

What allowing or blocking Meta-ExternalAgent means

Allowing Meta-ExternalAgent feeds Meta AI / Llama training. Blocking it opts your content out of those crawls.

Block Meta-ExternalAgent (robots.txt)

# Block Meta-ExternalAgent
User-agent: Meta-ExternalAgent
Disallow: /

Allow Meta-ExternalAgent (robots.txt)

# Allow Meta-ExternalAgent
User-agent: Meta-ExternalAgent
Allow: /

Declare your policy in llms.txt

robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set Meta-ExternalAgent to allowed or restricted and emits a compliant file.

FAQ

Other AI crawlers