Bytespider: what it is & how to allow or block it
Bytespider is ByteDance's crawler, associated with training data for its AI products (including the Doubao models). ByteDance states Bytespider respects robots.txt, but independent operators have reported instances of non-compliance — if you must block it, monitor your logs and consider an edge-level (WAF) rule as a backstop.
- User-agent
Bytespider- Operator
- ByteDance
- Type
- Model-training crawler
- robots.txt
- ByteDance states Bytespider respects robots.txt, but independent operators have reported instances of non-compliance — if you must block it, monitor your logs and consider an edge-level (WAF) rule as a backstop.
- Official docs
- developers.cloudflare.com/bots/concepts/bot/veri
What allowing or blocking Bytespider means
Allowing Bytespider feeds ByteDance training data; many publishers choose to block it. If blocking, verify the block holds at the edge, not only in robots.txt.
Block Bytespider (robots.txt)
# Block Bytespider User-agent: Bytespider Disallow: /
Allow Bytespider (robots.txt)
# Allow Bytespider User-agent: Bytespider Allow: /
Declare your policy in llms.txt
robots.txt controls access; llms.txt states your intent in a machine-readable way LLMs prefer. Keep them consistent. The llms.txt generator lets you set Bytespider to allowed or restricted and emits a compliant file.
FAQ
Does Bytespider always obey robots.txt?
ByteDance states it does, but third parties have reported non-compliance. If blocking is important, pair the robots.txt rule with an edge/WAF rule and watch your logs.