Meta-ExternalFetcher: what it is, what it does with your content, and what blocking it costs
What Meta-ExternalFetcher is
Meta-ExternalFetcher fetches a page because a person asked for it. They pasted your URL, or followed a link, and asked the assistant to read it. There is no crawl schedule and no training corpus involved. Blocking it means failing a visitor who explicitly asked for your page.
Operator: Meta
robots.txt token: Meta-ExternalFetcher
User-agent on the wire: meta-externalfetcher/1.1
Category: User-triggered fetch
What blocking it actually costs
Meta AI cannot fetch your page when a user asks about a specific link.
How to block Meta-ExternalFetcher
Add this to your robots.txt:
User-agent: Meta-ExternalFetcher
Disallow: /
The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.
How to allow Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher
Allow: /
An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.
robots.txt is not the only thing that can block it
A permissive robots.txt does not mean Meta-ExternalFetcher can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes Meta-ExternalFetcher and whose edge returns 403 to it is blocked in every way that matters, and nothing in robots.txt will tell you so.
This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.
Official documentation
Meta documents Meta-ExternalFetcher at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers.
Common questions
- What user-agent does Meta-ExternalFetcher send?
Meta does not publish an exact user-agent string for Meta-ExternalFetcher. Match on the
Meta-ExternalFetchertoken in robots.txt rather than on a full string you found elsewhere.- How do I block Meta-ExternalFetcher?
Add a group to robots.txt naming the token exactly:
User-agent: Meta-ExternalFetcher Disallow: /The group must name
Meta-ExternalFetcheron its own line. AUser-agent: *group does not combine with a named one. Under RFC 9309, the most specific matching group replaces the wildcard entirely, it does not inherit from it.- What does blocking Meta-ExternalFetcher cost me?
Meta AI cannot fetch your page when a user asks about a specific link.
- Can I tell a real Meta-ExternalFetcher request from a fake one?
Not reliably. Meta does not publish verified IP ranges for Meta-ExternalFetcher, so the user-agent string is the only signal, and anyone can send it. Treat robots.txt as a statement of policy rather than as enforcement.
Is Meta-ExternalFetcher blocked on your site right now?
Reading your own robots.txt only answers half of it, because a firewall rule can block Meta-ExternalFetcher while robots.txt says it is welcome. The checker tests both.
Check your site free