What ClaudeBot does
ClaudeBot crawls public web content that may be used to train and improve Anthropic's Claude models. Anthropic documents it separately from Claude-SearchBot and Claude-User, which serve search and live user requests.
Should I allow ClaudeBot to crawl my site?
Like GPTBot for OpenAI, ClaudeBot determines whether your own content is part of what future Claude models learn. Blocking it leaves your brand's representation in training data to whatever third parties publish about you.
How to control ClaudeBot
Target ClaudeBot in your robots.txt with the ClaudeBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow ClaudeBot
User-agent: ClaudeBot
Allow: /Block ClaudeBot
User-agent: ClaudeBot
Disallow: /ClaudeBot FAQ
What is ClaudeBot?
ClaudeBot is Anthropic's crawler that gathers public web content that may contribute to training future Claude models. Site owners can opt out by blocking it in robots.txt.
How do I block ClaudeBot?
Add a robots.txt group with "User-agent: ClaudeBot" and "Disallow: /". Anthropic also supports the non-standard Crawl-delay extension if you want to throttle requests instead of blocking. It publishes crawler IPs at claude.com/crawling/bots.json but warns that IP blocking may not persistently guarantee an opt-out, since it can prevent the crawler from reading your robots.txt.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
