How to control Claude-Web
Target Claude-Web in your robots.txt with the Claude-Web token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Claude-Web
User-agent: Claude-Web
Allow: /Block Claude-Web
User-agent: Claude-Web
Disallow: /How to identify Claude-Web
Temso identifies Claude-Webtraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
Claude-WebClaude-Web FAQ
What is Claude-Web?
Claude-Web is a user agent token associated with Anthropic, the company behind the Claude AI assistant. Anthropic's current crawler documentation describes three agents, ClaudeBot, Claude-User, and Claude-SearchBot, and no longer documents Claude-Web specifically.
How do I block Claude-Web?
Add a robots.txt group with User-agent: Claude-Web followed by Disallow: /, and repeat this on every subdomain you want to opt out. Anthropic states that its bots honor industry standard robots.txt directives and that blocking by IP address instead may not work reliably, since that prevents its crawlers from reading your robots.txt file.
About AI Assistant Fetchers
Agents that fetch a page in real time because a user asked an AI assistant about it. They don't build an index; each visit maps to a live conversation.
Blocking these removes your pages from live AI answers at the exact moment a user is asking about you or your category.
