What SemrushBot does
SemrushBot is the crawler Semrush uses to discover and collect new and updated web data. It starts from a list of known URLs, saves the hyperlinks it finds for further crawling, and revisits pages to detect content changes, new pages, and dead links. The data feeds Semrush products including Backlink Analytics, Site Audit, Backlink Audit, Link Building, and On Page SEO Checker. Semrush also operates separate user agents for specific tools, such as SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, and RyteBot, which can be allowed or blocked independently.
How to control SemrushBot
Target SemrushBot in your robots.txt with the SemrushBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow SemrushBot
User-agent: SemrushBot
Allow: /Block SemrushBot
User-agent: SemrushBot
Disallow: /Semrush states that robots.txt changes can take up to one hour or 100 requests for SemrushBot to discover, that Crawl-delay values above 10 seconds are capped at 10 seconds, and that blocking by IP is not reliable because the bot does not use consecutive IP blocks.
How to identify SemrushBot
Temso identifies SemrushBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
SemrushBot(?!-)SemrushBot FAQ
What is SemrushBot?
SemrushBot is Semrush's web crawler. It discovers and collects new and updated web data by following hyperlinks and revisiting known pages, and the resulting index powers Semrush tools such as Backlink Analytics and Site Audit.
How do I block SemrushBot?
Add a robots.txt group with 'User-agent: SemrushBot' and 'Disallow: /'. Note that it may take up to one hour or 100 requests for SemrushBot to discover the change, your robots.txt must return HTTP 200 to be honored, and other Semrush user agents (for example SiteAuditBot or SemrushBot-BA) need their own groups if you want to block them too.
About SEO & Analytics Crawlers
Crawlers behind SEO and marketing-intelligence platforms: backlink databases, site audits, rank tracking, and brand monitoring.
Blocking these has no effect on search or AI visibility. It mainly limits what third-party tools (including your competitors' tools) can see about your site.
