What OAI-SearchBot does
OAI-SearchBot crawls the web to build and maintain the index used by ChatGPT search. OpenAI documents that it is used to surface and link to websites in search results, and is not used to train OpenAI's models.
Should I allow OAI-SearchBot to crawl my site?
This is the crawler that decides whether ChatGPT search can find and cite you at all. Blocking OAI-SearchBot removes your pages from ChatGPT's search index, independent of any GPTBot training decision.
How to control OAI-SearchBot
Target OAI-SearchBot in your robots.txt with the OAI-SearchBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow OAI-SearchBot
User-agent: OAI-SearchBot
Allow: /Block OAI-SearchBot
User-agent: OAI-SearchBot
Disallow: /OpenAI publishes OAI-SearchBot's IP ranges so crawl traffic can be verified against the official list.
User-agent string
As published by OpenAI. Remember that user-agent strings can be spoofed, so verify suspicious traffic against the operator's published IP ranges.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbotOAI-SearchBot FAQ
What is OAI-SearchBot?
OAI-SearchBot is an OpenAI crawler that builds the search index behind ChatGPT search by surfacing websites in search results. It is separate from GPTBot, the crawler OpenAI uses to gather content for training its models, and is not used for model training.
How do I block OAI-SearchBot?
Add a robots.txt group with "User-agent: OAI-SearchBot" and "Disallow: /". OpenAI says it takes about 24 hours for its systems to adjust after a robots.txt change. Sites that block the bot will not appear in ChatGPT search answers, though they may still show as navigational links.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
