What SemanticScholarBot does
SemanticScholarBot is a crawler operated by the Allen Institute for AI (Ai2) that crawls academic domains to locate PDFs. The documents it finds are indexed and made available on semanticscholar.org for research discovery. It identifies itself with the user-agent string "Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)". The crawler page does not document robots.txt handling, crawl rates, or IP ranges, and it directs questions or concerns to the operator's contact channel.
How to control SemanticScholarBot
Target SemanticScholarBot in your robots.txt with the SemanticScholarBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow SemanticScholarBot
User-agent: SemanticScholarBot
Allow: /Block SemanticScholarBot
User-agent: SemanticScholarBot
Disallow: /How to identify SemanticScholarBot
Temso identifies SemanticScholarBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
SemanticScholarBotSemanticScholarBot FAQ
What is SemanticScholarBot?
SemanticScholarBot is an Ai2 crawler that searches academic domains for PDFs, which are then indexed and made discoverable on semanticscholar.org. It appears in logs with the user-agent string "Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)".
How do I block SemanticScholarBot?
Add a robots.txt group with "User-agent: SemanticScholarBot" followed by "Disallow: /". The crawler's documentation page does not describe its robots.txt handling, but it states that questions or concerns about the crawler can be sent to the operator.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
