What Google-CloudVertexBot does
Google-CloudVertexBot is one of Google's common crawlers and crawls sites for building Vertex AI Agents. It fetches pages only when site owners specifically request crawls, and its crawls have no effect on Google Search or other Google products. It always obeys robots.txt rules when crawling automatically and can match either its own Google-CloudVertexBot token or the fallback Googlebot token in robots.txt.
How to control Google-CloudVertexBot
Target Google-CloudVertexBot in your robots.txt with the Google-CloudVertexBot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Google-CloudVertexBot
User-agent: Google-CloudVertexBot
Allow: /Block Google-CloudVertexBot
User-agent: Google-CloudVertexBot
Disallow: /Google's common crawlers can be verified against the IP ranges published in common-crawlers.json and by reverse DNS hostnames matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com, since the user agent string alone can be spoofed.
How to identify Google-CloudVertexBot
Temso identifies Google-CloudVertexBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
Google-CloudVertexBotGoogle-CloudVertexBot FAQ
What is Google-CloudVertexBot?
Google-CloudVertexBot is a Google crawler that fetches site content for building Vertex AI Agents, and it only crawls when a site owner requests it. Its activity has no effect on Google Search or other Google products.
How do I block Google-CloudVertexBot?
Add a robots.txt group with "User-agent: Google-CloudVertexBot" and "Disallow: /". If no group names it specifically, the crawler also matches rules addressed to the fallback token Googlebot.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
