What Meta-WebIndexer does
Meta-WebIndexer is a Meta crawler that navigates the web to improve Meta AI search result quality for users, analyzing online content to enhance the relevance and accuracy of Meta AI responses. It identifies itself with the user agent meta-webindexer/1.1. It is separate from Meta's other crawlers such as Meta-ExternalAgent, used for AI training and indexing, and Meta-ExternalFetcher, which fetches links requested by users.
How to control Meta-WebIndexer
Target Meta-WebIndexer in your robots.txt with the meta-webindexer token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Meta-WebIndexer
User-agent: meta-webindexer
Allow: /Block Meta-WebIndexer
User-agent: meta-webindexer
Disallow: /Meta's documentation says that allowing meta-webindexer in your robots.txt file helps Meta cite and link to your content in Meta AI's responses; it separately notes that Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt in certain cases.
How to identify Meta-WebIndexer
Temso identifies Meta-WebIndexertraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
meta-webindexerMeta-WebIndexer FAQ
What is Meta-WebIndexer?
Meta-WebIndexer is a Meta crawler that navigates the web to improve Meta AI search result quality, and Meta states that allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses. It identifies itself with the user agent meta-webindexer/1.1.
How do I block Meta-WebIndexer?
Add a robots.txt group with "User-agent: meta-webindexer" followed by "Disallow: /". Per Meta's documentation, allowing the crawler helps Meta cite and link to your content in Meta AI's responses, so blocking it means Meta is less able to do so, and it does not affect Meta's other crawlers, which use different tokens.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
