What SiteAuditBot does
SiteAuditBot is the crawler Semrush uses for its Site Audit tool, which analyzes websites for SEO and technical issues. It obeys robots.txt, and Semrush notes the file must sit in the site's top directory and return an HTTP 200 status. Semrush states it may take up to one hour or 100 requests for the bot to discover robots.txt changes, and offers bot@semrush.com as a contact for crawling problems.
How to control SiteAuditBot
Target SiteAuditBot in your robots.txt with the SiteAuditBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow SiteAuditBot
User-agent: SiteAuditBot
Allow: /Block SiteAuditBot
User-agent: SiteAuditBot
Disallow: /Semrush states its bots do not use any consecutive IP blocks (so IP-based blocking is not recommended), and robots.txt changes may take up to one hour or 100 requests to be discovered.
How to identify SiteAuditBot
Temso identifies SiteAuditBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
SiteAuditBotSiteAuditBot FAQ
What is SiteAuditBot?
It is Semrush's crawler for the Site Audit tool, used to analyze websites for SEO and technical issues. It identifies itself with the user agent token SiteAuditBot.
How do I block SiteAuditBot?
Add a robots.txt group with "User-agent: SiteAuditBot" followed by "Disallow: /"; the file must be in your site's top directory and return HTTP 200. Changes may take up to one hour or 100 requests to be discovered, and Semrush advises against IP-based blocking because its bots do not use consecutive IP blocks.
About SEO & Analytics Crawlers
Crawlers behind SEO and marketing-intelligence platforms: backlink databases, site audits, rank tracking, and brand monitoring.
Blocking these has no effect on search or AI visibility. It mainly limits what third-party tools (including your competitors' tools) can see about your site.
