What Claude-SearchBot does
Claude-SearchBot is Anthropic's search-indexing crawler. It navigates the web to improve the quality of search results that Claude references when answering with web content.
Should I allow Claude-SearchBot to crawl my site?
Allowing Claude-SearchBot keeps your pages eligible to appear and be cited when Claude searches the web; blocking it narrows what Claude's search layer can retrieve about you.
How to control Claude-SearchBot
Target Claude-SearchBot in your robots.txt with the Claude-SearchBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /Claude-SearchBot FAQ
What is Claude-SearchBot?
Claude-SearchBot is an Anthropic crawler that indexes web content to improve the quality of search results referenced in Claude. Blocking it prevents Anthropic from indexing your content for those search results.
How do I block Claude-SearchBot?
Add a robots.txt group with "User-agent: Claude-SearchBot" and "Disallow: /". Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json but warns that blocking by IP may not persistently guarantee an opt-out, since it can prevent its systems from reading your robots.txt file.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
