What CCBot does
CCBot builds Common Crawl, a freely available archive of web crawl data maintained by a non-profit. The dataset has been used as training data by many AI labs, which makes CCBot's reach broader than any single company's crawler.
Should I allow CCBot to crawl my site?
Because many models train on Common Crawl, allowing CCBot effectively distributes your content into the training pipelines of numerous current and future models at once, and blocking it quietly removes you from all of them.
How to control CCBot
Target CCBot in your robots.txt with the CCBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow CCBot
User-agent: CCBot
Allow: /Block CCBot
User-agent: CCBot
Disallow: /User-agent string
As published by Common Crawl. Remember that user-agent strings can be spoofed, so verify suspicious traffic against the operator's published IP ranges.
CCBot/2.0 (https://commoncrawl.org/faq/)CCBot FAQ
What is CCBot?
CCBot is the web crawler operated by the Common Crawl Foundation to build its open repository of web crawl data, which is made freely available for research and analysis. It identifies itself with the user-agent string "CCBot/2.0 (https://commoncrawl.org/faq/)".
How do I block CCBot?
Add a robots.txt group with "User-agent: CCBot" followed by "Disallow: /". Common Crawl notes that some crawlers falsely identify themselves as CCBot, so genuine traffic can be verified by reverse DNS (hostnames resolve under crawl.commoncrawl.org) or against the IP ranges published at index.commoncrawl.org/ccbot.json.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
