Anthropic bots

5 bots in the directory · Reviewed July 2026

Anthropic, the maker of Claude, mirrors OpenAI's separation of concerns: ClaudeBot gathers training data, Claude-SearchBot indexes the web for Claude's search results, and Claude-User fetches pages when a Claude conversation needs live content.

Claude-User

Fetches pages on behalf of Claude users when a conversation needs live web content.

AI AssistantRespects robots.txt

Claude-SearchBot

Indexes the web to improve search results referenced inside Claude.

AI SearchRespects robots.txt

ClaudeBot

Anthropic's training crawler. It gathers public web data that may improve future Claude models.

AI TrainingRespects robots.txt

Claude-Web

Real-time browsing fetch used by Claude to compose answers.

AI Assistant

anthropic-ai

Anthropic's general-purpose data collection agent.

AI Training

Control every Anthropic bot at once

robots.txt lets one rule group target multiple user-agents. These snippets cover all 5 Anthropic tokens in this directory. For finer control, set rules per bot on each bot's page.

Allow all Anthropic bots

User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
Allow: /

Block all Anthropic bots

User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
Disallow: /

← Back to the AI Bot Directory