Meta bots

6 bots in the directory · Reviewed July 2026

Meta crawls for its Llama models and Meta AI assistant: Meta-ExternalAgent collects training data, while Meta-ExternalFetcher performs user-initiated fetches for Meta AI across Facebook, Instagram, and WhatsApp.

Meta-ExternalFetcher

Fetches individual links on behalf of Meta AI when a user's request requires them.

AI AssistantUser-triggered; robots.txt may not apply

Meta-ExternalAgent

Meta's crawler for AI training data and content indexing across its products.

AI TrainingRespects robots.txt

FacebookBot

Meta's general-purpose crawler used across Facebook surfaces.

Operational

Meta-WebIndexer

Crawls web content to provide search results for Meta AI users.

AI Search

meta-externalads

Crawls the web to improve Meta's advertising and business products.

Advertising

facebookexternalhit

Fetches shared links on Meta platforms to generate rich previews.

Operational

Control every Meta bot at once

robots.txt lets one rule group target multiple user-agents. These snippets cover all 6 Meta tokens in this directory. For finer control, set rules per bot on each bot's page.

Allow all Meta bots

User-agent: Meta-ExternalFetcher
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: meta-webindexer
User-agent: meta-externalads
User-agent: facebookexternalhit
Allow: /

Block all Meta bots

User-agent: Meta-ExternalFetcher
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: meta-webindexer
User-agent: meta-externalads
User-agent: facebookexternalhit
Disallow: /

← Back to the AI Bot Directory