What Diffbot does
Diffbot operates two user agents: Diffbot, which performs general, proactive web crawling for building a general search engine, and Diffbot-User, which makes requests on behalf of a human user browsing a URL. By default its web crawls adhere to a site's robots.txt instructions, including the disallow and crawl-delay directives. Diffbot states that it does not crawl pages for training generative AI foundation models. Customers using its Extract or Crawlbot APIs control their own crawl parameters, with robots.txt adherence enabled by default.
How to control Diffbot
Target Diffbot in your robots.txt with the Diffbot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Diffbot
User-agent: Diffbot
Allow: /Block Diffbot
User-agent: Diffbot
Disallow: /Diffbot honors robots.txt disallow and crawl-delay directives by default, but its documentation notes that in specific cases, typically a partnership or agreement with the site being crawled, the robots.txt instruction can be ignored or overridden.
How to identify Diffbot
Temso identifies Diffbottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
DiffbotDiffbot FAQ
What is Diffbot?
Diffbot is a crawler that performs general, proactive web crawling for building a general search engine, alongside a separate Diffbot-User agent for requests made on behalf of a human user browsing a URL. Diffbot states that it does not crawl pages for training generative AI foundation models.
How do I block Diffbot?
Add a robots.txt group with User-agent: Diffbot followed by Disallow: /; by default Diffbot's crawls adhere to robots.txt, including the disallow and crawl-delay directives. Its documentation notes that robots.txt can be overridden in specific cases where the site has a partnership or agreement with Diffbot.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
