What Meta-ExternalFetcher does
Meta-ExternalFetcher performs user-initiated fetches for Meta's AI products: when a Meta AI user shares or asks about a specific link, this agent retrieves it to complete the request.
Should I allow Meta-ExternalFetcher to crawl my site?
Meta AI reaches users across Facebook, Instagram, and WhatsApp. Allowing this fetcher lets those assistants read your pages when users ask; because fetches are user-initiated, Meta documents that it may bypass robots.txt rules.
How to control Meta-ExternalFetcher
Target Meta-ExternalFetcher in your robots.txt with the Meta-ExternalFetcher token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher
Allow: /Block Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher
Disallow: /Meta documents that user-initiated fetches may bypass robots.txt because they serve a direct user request rather than automated crawling.
Meta-ExternalFetcher FAQ
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is a Meta crawler that fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.
How do I block Meta-ExternalFetcher?
You can add a robots.txt group with "User-agent: Meta-ExternalFetcher" and "Disallow: /", but Meta documents that this crawler may bypass robots.txt rules because it performs fetches that were requested by the user.
About AI Assistant Fetchers
Agents that fetch a page in real time because a user asked an AI assistant about it. They don't build an index; each visit maps to a live conversation.
Blocking these removes your pages from live AI answers at the exact moment a user is asking about you or your category.
