What CriteoBot does
The Criteo crawler analyzes web page content so Criteo can serve contextual advertisements, categorizing pages by topic (for example, classifying an article about running shoes under sport and running). It only accesses URLs on a domain when that website has asked Criteo to deliver ads there, and it revisits only when its compiled categories are missing or no longer up to date. Criteo states the crawler analyzes only publicly available published data and does not access data about users navigating the site. It identifies itself with the user agent CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/).
How to control CriteoBot
Target CriteoBot in your robots.txt with the CriteoBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow CriteoBot
User-agent: CriteoBot
Allow: /Block CriteoBot
User-agent: CriteoBot
Disallow: /Criteo publishes the ASNs (44788, 19750, 55569) and IPv4/IPv6 ranges its crawler uses, and the crawler respects Crawl-delay directives up to 30 seconds.
How to identify CriteoBot
Temso identifies CriteoBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
CriteoBotCriteoBot FAQ
What is CriteoBot?
CriteoBot is Criteo's crawler, which visits pages on sites where Criteo has been asked to deliver ads and categorizes their content so Criteo can serve contextual advertising. It identifies itself as CriteoBot/0.1 with a link to Criteo's crawler page.
How do I block CriteoBot?
Add a robots.txt group with User-agent: CriteoBot (Criteo's own examples use User-agent: CriteoBot/0.1) and Disallow: / to block the whole site, or Disallow: /path/ for specific paths. To slow it down instead of blocking, the crawler respects Crawl-delay values up to 30 seconds.
About Advertising Bots
Bots run by ad platforms to check landing pages, verify ad placements, and analyze content for brand safety and targeting.
If you run ads on a platform, its ad bots usually need access; blocking them can break ad approval and quality checks for your own campaigns.
