What facebookexternalhit does
facebookexternalhit is a Meta crawler that fetches the content of an app or website when someone shares its link on one of Meta's apps, such as Facebook, Instagram, or Messenger. It collects metadata like titles, descriptions, and thumbnail images to build the link preview. Meta's documentation says servers should support gzip and deflate encodings, serve content within a few seconds, and place Open Graph properties within the first 1 MB of the page so the crawler can read them. It identifies itself with user-agent strings such as facebookexternalhit/1.1.
How to control facebookexternalhit
Target facebookexternalhit in your robots.txt with the facebookexternalhit token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow facebookexternalhit
User-agent: facebookexternalhit
Allow: /Block facebookexternalhit
User-agent: facebookexternalhit
Disallow: /Meta states that robots.txt changes can take up to 24 hours to take effect and that the crawler may bypass robots.txt when performing security or integrity checks, such as scanning for malware or malicious content.
How to identify facebookexternalhit
Temso identifies facebookexternalhittraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
facebookexternalhitfacebookexternalhit FAQ
What is facebookexternalhit?
facebookexternalhit is Meta's crawler that fetches a page when its link is shared on Facebook, Instagram, or Messenger. It collects the title, description, and thumbnail image used to render the link preview.
How do I block facebookexternalhit?
Add a robots.txt group with "User-agent: facebookexternalhit" and a Disallow rule for the paths you want excluded. Meta says changes can take up to 24 hours to take effect. Note that blocking it prevents link previews from being generated, and the crawler may still bypass robots.txt for security or integrity checks such as malware scanning.
About Operational Bots
Monitoring, security scanning, link previews, webhooks, and feed fetchers: the utility traffic that keeps integrations and shared links working.
Most of this traffic is service-critical or user-triggered. Webhooks and previews don't follow robots.txt, and blocking them tends to break things you rely on.
