What Claude-User does
Claude-User is Anthropic's user-initiated agent. When a Claude user asks a question that requires current web content, Claude-User retrieves the specific pages needed to answer. It acts on direct user requests rather than crawling autonomously.
Should I allow Claude-User to crawl my site?
Allowing Claude-User means Claude can read your pages directly when users ask about your product or category: your own content becomes the source, rather than a third party's description of it.
How to control Claude-User
Target Claude-User in your robots.txt with the Claude-User token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Claude-User
User-agent: Claude-User
Allow: /Block Claude-User
User-agent: Claude-User
Disallow: /Claude-User FAQ
What is Claude-User?
Claude-User is the agent Anthropic uses to access websites when individual users ask Claude questions that require live web content. It is separate from ClaudeBot, which collects content for AI model training, and Claude-SearchBot, which indexes content for search.
How do I block Claude-User?
Add a robots.txt group with "User-agent: Claude-User" and "Disallow: /". Anthropic states its bots honor industry standard robots.txt directives and respect Crawl-delay where appropriate. Anthropic notes that disabling Claude-User prevents its system from retrieving your content in response to user queries, which may reduce your site's visibility for user-directed web search.
About AI Assistant Fetchers
Agents that fetch a page in real time because a user asked an AI assistant about it. They don't build an index; each visit maps to a live conversation.
Blocking these removes your pages from live AI answers at the exact moment a user is asking about you or your category.
