What Bravebot does
Bravebot is the crawler that builds the index for Brave Search. Brave states that its crawler does not advertise a differentiated user agent, a choice made to avoid discrimination from websites that allow only Google to crawl them, and that if a domain or page is not crawlable by Googlebot, Brave Search's bot will not crawl it either. The index is also partially powered by data from users who opt in to Brave's open-source Web Discovery Project. To remove a page from the index, Brave requires a robots noindex directive rather than robots.txt rules.
How to control Bravebot
Target Bravebot in your robots.txt with the Bravebot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Bravebot
User-agent: Bravebot
Allow: /Block Bravebot
User-agent: Bravebot
Disallow: /Brave documents that robots.txt is not used to prevent a page from being indexed; delisting requires a robots noindex directive, and Brave must re-fetch the page (which you can request via its URL submission tool) before the change takes effect.
How to identify Bravebot
Temso identifies Bravebottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
BravebotBravebot FAQ
What is Bravebot?
Bravebot is the crawler behind the Brave Search index, which is partially powered by opt-in data from Brave's Web Discovery Project. Brave states the crawler does not advertise a differentiated user agent and will not crawl anything that is not crawlable by Googlebot.
How do I block Bravebot?
A robots.txt group with "User-agent: Bravebot" and "Disallow: /" targets the token, but Brave's documentation says its crawler does not advertise a differentiated user agent and that robots.txt is not used to prevent indexing. To delist a page, add a robots noindex directive and have Brave re-fetch the page, for example through its URL submission tool.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
