What AI2Bot does
AI2Bot is a crawler operated by the Allen Institute for AI (Ai2) that explores web domains to gather content for training open language models. It identifies itself in server logs with the user-agent string "Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)". Ai2's crawler page does not document crawl rates, IP ranges, or robots.txt handling, and it directs questions or concerns about the crawler to the Ai2 contact form.
How to control AI2Bot
Target AI2Bot in your robots.txt with the AI2Bot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow AI2Bot
User-agent: AI2Bot
Allow: /Block AI2Bot
User-agent: AI2Bot
Disallow: /How to identify AI2Bot
Temso identifies AI2Bottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
AI2Bot(?!-Dolma)AI2Bot FAQ
What is AI2Bot?
AI2Bot is a web crawler run by the Allen Institute for AI (Ai2) that gathers web content used to train Ai2's open language models. It appears in logs with the user-agent string "Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)".
How do I block AI2Bot?
Add a robots.txt group with "User-agent: AI2Bot" followed by "Disallow: /" (or a narrower path). Ai2's crawler page does not describe its robots.txt handling in detail, but it provides a contact form for questions or concerns about the crawler.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
