What Googlebot does
Googlebot crawls the web to build Google's search index. That same index now feeds Google's AI surfaces: AI Overviews and AI Mode compose answers from content Googlebot has crawled and indexed.
Should I allow Googlebot to crawl my site?
There is no separate crawler for AI Overviews; eligibility flows from the standard Google index. Blocking Googlebot removes you from Google Search and every AI feature built on it, making it the most consequential block on this list.
How to control Googlebot
Target Googlebot in your robots.txt with the Googlebot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Googlebot
User-agent: Googlebot
Allow: /Block Googlebot
User-agent: Googlebot
Disallow: /Google publishes Googlebot's IP ranges and a reverse-DNS verification method.
User-agent string
As published by Google. Remember that user-agent strings can be spoofed, so verify suspicious traffic against the operator's published IP ranges.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Googlebot FAQ
What is Googlebot?
Googlebot is one of Google's common crawlers, used to find information for building Google's search indexes. Crawl preferences addressed to the Googlebot user agent affect Google Search, including Discover and all Google Search features, as well as products such as Google Images, Google Video, and Google News.
How do I block Googlebot?
Add a robots.txt group with "User-agent: Googlebot" followed by "Disallow: /"; Google's common crawlers always respect robots.txt rules for automatic crawls, but blocking Googlebot affects your site's presence in Google Search and related products. Because the user-agent string can be spoofed, verify real Googlebot traffic by reverse DNS (hostnames match crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com) or against the IP ranges Google publishes in the common-crawlers.json object.
About Search Engine Crawlers
Classic search indexers that now also feed AI features: Google's AI Overviews and AI Mode build on Googlebot's index, and Microsoft Copilot builds on Bingbot's.
Blocking these removes you from both traditional rankings and the AI answers built on top of those indexes, usually the most costly block of all.
