What Google Feedfetcher does
Feedfetcher is Google's fetcher for RSS and Atom feeds, used by Google News and WebSub. It stores and periodically refreshes feeds, retrieving a feed only after users have explicitly started a service or app that requests data from it. Google states it should not retrieve feeds from most sites more than once every hour on average, though some frequently updated sites may be refreshed more often.
How to control Google Feedfetcher
Target Google Feedfetcher in your robots.txt with the Feedfetcher-Google token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Google Feedfetcher
User-agent: Feedfetcher-Google
Allow: /Block Google Feedfetcher
User-agent: Feedfetcher-Google
Disallow: /Feedfetcher ignores robots.txt entries because it behaves as a direct agent of a human user rather than a robot, and its requests can be verified against Google's published user-triggered-fetchers-google.json IP address list.
How to identify Google Feedfetcher
Temso identifies Google Feedfetchertraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
Feedfetcher-GoogleGoogle Feedfetcher FAQ
What is Google Feedfetcher?
Feedfetcher is the Google service that crawls RSS and Atom feeds for Google News and WebSub, storing and periodically refreshing feeds that users have explicitly requested through a Google service or app.
How do I block Google Feedfetcher?
A robots.txt group with "User-agent: Feedfetcher-Google" will not work: Google states Feedfetcher ignores robots.txt entries because it acts as a direct agent of the human user. To block it, configure your server to return a 404, 410, or other error status to requests from the Feedfetcher-Google user agent.
About Operational Bots
Monitoring, security scanning, link previews, webhooks, and feed fetchers: the utility traffic that keeps integrations and shared links working.
Most of this traffic is service-critical or user-triggered. Webhooks and previews don't follow robots.txt, and blocking them tends to break things you rely on.
