What ShapBot does
ShapBot is Parallel's crawler, which discovers and indexes websites for Parallel's web APIs and supports visibility in Parallel's search results. It identifies itself with a user agent containing ShapBot/0.1.0. Parallel's documentation recommends allowing ShapBot in robots.txt and permitting connections from its designated IP ranges, and offers support@parallel.ai for webmaster questions.
How to control ShapBot
Target ShapBot in your robots.txt with the ShapBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow ShapBot
User-agent: ShapBot
Allow: /Block ShapBot
User-agent: ShapBot
Disallow: /Parallel publishes ShapBot's IP addresses in a JSON file at docs.parallel.ai/resources/shapbot.json, so traffic using this user agent can be checked against the published list.
How to identify ShapBot
Temso identifies ShapBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
ShapBotShapBot FAQ
What is ShapBot?
ShapBot is the crawler operated by Parallel that discovers and indexes websites for Parallel's web APIs and search results. It identifies itself with the ShapBot/0.1.0 token in its user agent string.
How do I block ShapBot?
Add a robots.txt group with "User-agent: ShapBot" and "Disallow: /" (or narrower rules for specific paths). You can additionally verify or filter its traffic against the IP list Parallel publishes at docs.parallel.ai/resources/shapbot.json.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
