What Bytespider does
Bytespider is operated by ByteDance, TikTok's parent company, and is widely understood to collect web data for its AI models. ByteDance publishes little official documentation about the bot's behavior or purpose.
Should I allow Bytespider to crawl my site?
ByteDance's assistant products have large user bases, particularly in Asia. That said, Bytespider is frequently reported to ignore robots.txt directives, so a robots.txt block may not be enforceable in practice.
How to control Bytespider
Target Bytespider in your robots.txt with the Bytespider token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Bytespider
User-agent: Bytespider
Allow: /Block Bytespider
User-agent: Bytespider
Disallow: /Multiple independent reports document Bytespider crawling despite robots.txt disallow rules. ByteDance has no public bot documentation, so compliance cannot be verified from primary sources.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
