Google bots
25 bots in the directory · Reviewed July 2026
Google's AI surfaces build on its search infrastructure: Googlebot's index feeds Search, AI Overviews, and AI Mode, while GoogleOther handles non-Search product crawls. Gemini training is controlled separately via the Google-Extended robots.txt token, without touching Search visibility.
Google-Extended
Not a crawler: a robots.txt token that controls whether Google uses your content for Gemini training and grounding.
Googlebot
Google's main indexer: the crawl behind Search, AI Overviews, and AI Mode.
GoogleOther
Google's generic crawler for product teams: research and development fetches outside Search.
Google-Agent
Google agent that fetches pages on behalf of generative experiences.
Google-NotebookLM
Fetches sources a user adds to NotebookLM so it can summarize and cite them.
Google-CloudVertexBot
Crawls site-owner-requested pages to build Vertex AI Search agents.
Googlebot-Image
Indexes images for Google Image Search.
Googlebot-Video
Indexes video content for Google Video Search.
Googlebot-News
Indexes articles for Google News.
Storebot-Google
Crawls shopping content for all Google Shopping surfaces.
Chrome Lighthouse
Lighthouse page-experience and performance auditing.
AdsBot-Google-Mobile
Checks mobile ad landing-page quality for Google Ads.
AdsBot-Google
Checks ad landing-page quality for Google Ads.
Google-Display-Ads-Bot
Verifies site eligibility during the AdSense approval process.
Mediapartners-Google
Crawls participating sites to serve relevant AdSense/AdMob ads.
Google Stackdriver
Google Cloud uptime checks and availability monitoring.
Google-InspectionTool
Search Console URL Inspection and Rich Results Test.
Google-Safety
Abuse-specific crawling such as malware discovery.
Google Site Verifier
Fetches Search Console verification tokens.
Google Image Proxy
Image caching proxy used by Gmail and other Google services.
Chrome Prefetch Proxy
Fetches traffic-advice to enable privacy-preserving prefetch hints.
Google Feedfetcher
Crawls RSS/Atom feeds for Google News and PubSubHubbub.
Google Publisher Center
Fetches feeds publishers supply for Google News landing pages.
Google Read Aloud
Fetches and reads out web pages using text-to-speech on request.
APIs-Google
Delivers push-notification messages for Google APIs.
Control every Google bot at once
robots.txt lets one rule group target multiple user-agents. These snippets cover all 24 Google tokens in this directory. For finer control, set rules per bot on each bot's page.
Allow all Google bots
User-agent: Google-Extended
User-agent: Googlebot
User-agent: GoogleOther
User-agent: Google-Agent
User-agent: Google-NotebookLM
User-agent: Google-CloudVertexBot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
User-agent: Storebot-Google
User-agent: Chrome-Lighthouse
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google
User-agent: Google-Display-Ads-Bot
User-agent: Mediapartners-Google
User-agent: GoogleStackdriverMonitoring-UptimeChecks
User-agent: Google-InspectionTool
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleImageProxy
User-agent: Feedfetcher-Google
User-agent: GoogleProducer
User-agent: Google-Read-Aloud
User-agent: APIs-Google
Allow: /Block all Google bots
User-agent: Google-Extended
User-agent: Googlebot
User-agent: GoogleOther
User-agent: Google-Agent
User-agent: Google-NotebookLM
User-agent: Google-CloudVertexBot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
User-agent: Storebot-Google
User-agent: Chrome-Lighthouse
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google
User-agent: Google-Display-Ads-Bot
User-agent: Mediapartners-Google
User-agent: GoogleStackdriverMonitoring-UptimeChecks
User-agent: Google-InspectionTool
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleImageProxy
User-agent: Feedfetcher-Google
User-agent: GoogleProducer
User-agent: Google-Read-Aloud
User-agent: APIs-Google
Disallow: /Careful: this group includes a search engine crawler. Blocking it removes you from traditional search rankings and the AI answers built on that index, not just AI features.