What YouBot does
YouBot is You.com's crawler. It indexes web content to power You.com's AI search results and assistant answers.
Should I allow YouBot to crawl my site?
A smaller answer surface than ChatGPT or Perplexity, but the same logic applies: allowing YouBot keeps your pages retrievable and citable in You.com products.
How to control YouBot
Target YouBot in your robots.txt with the YouBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow YouBot
User-agent: YouBot
Allow: /Block YouBot
User-agent: YouBot
Disallow: /You.com documents that YouBot caches robots.txt for 30 minutes, so rule changes take up to half an hour to apply.
User-agent string
As published by You.com. Remember that user-agent strings can be spoofed, so verify suspicious traffic against the operator's published IP ranges.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod)YouBot FAQ
What is YouBot?
YouBot is the web crawler that powers the You.com search engine, automatically discovering and indexing web pages to provide search results for You.com's AI search and chat products.
How do I block YouBot?
Add a robots.txt group with "User-agent: YouBot" and "Disallow: /". YouBot also honors crawl-delay settings, and it caches robots.txt for 30 minutes, so changes can take up to half an hour to take effect. You can verify real YouBot traffic via reverse DNS (hostnames under search.you.com) or its published IP range 68.67.112.0/24.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
