Common Crawl bots

1 bot in the directory · Reviewed July 2026

Common Crawl is a non-profit whose CCBot builds a freely available web archive. Because many AI labs train on the Common Crawl dataset, one robots.txt decision here reaches numerous current and future models at once.

CCBot

Builds the open Common Crawl dataset, a widely used source of LLM training data.

AI TrainingRespects robots.txt
← Back to the AI Bot Directory