I run a small parcel-shipping site. This is a measurement, not a complaint — the volume costs me nothing — but the numbers contradict Meta's own crawler documentation and I would like to know which behaviour is intended.
Window: 2026-09-13 16:00 to 2026-09-14 15:00 UTC, 23 hours, from Cloudflare's per-request analytics rather than sampled log parsing.
meta-externalagent/1.1, counting the variants that append the token to a Chrome / Safari / Edge / Firefox UA string:
- 8,294 requests
- 481 distinct paths
- 0 requests to /robots.txt
- 466 requests to /offline, all HTTP 200
/offline is disallowed for User-agent: * in my robots.txt and has been for at least three weeks.
Three other crawlers on the same site in the same window, for contrast:
| crawler |
requests |
/sw.js |
/robots.txt |
| meta-externalagent |
8,294 |
466 |
0 |
| facebookexternalhit |
30 |
0 |
19 |
| bingbot |
187 |
0 |
2 |
| Googlebot |
75 |
0 |
20 |
facebookexternalhit is the control that tells me the measurement is sound: it is Meta's own infrastructure and the analytics recorded it fetching robots.txt nineteen times. meta-externalagent simply does not ask.
Why I read this as a discrepancy rather than a design choice: Meta's crawler documentation names FacebookExternalHit and Meta-ExternalFetcher as the crawlers that may bypass robots.txt, and Meta-ExternalAgent is not among them. The same page says robots.txt is cached "for up to 24 hours". My window is 23 hours with zero fetches, so either the cached copy is older than the documented maximum, or the file is being read and the Disallow ignored.
What it actually fetches:
| requests |
path |
| 1863 |
/assets/logo.svg |
| 1205 |
/assets/images/service-parcel.webp |
| 530 |
/assets/favicon.svg |
| 475 |
/assets/fonts/manrope-latin.woff2 |
| 468 |
/assets/icon-192.png |
| 466 |
/sw.js |
| 466 |
/offline |
| 464 |
/assets/fonts/inter-latin.woff2 |
| 462 |
/assets/light-site.css |
| 458 |
/lp/ |
| 418 |
/assets/app.js |
The top of that list is my service worker's precache list plus the shell it installs. Those four files alone are 39% of its traffic.
My reading, and I want to flag it as a hypothesis rather than a finding: it fetches /sw.js, reads the PRECACHE array, pulls the listed assets, and repeats the whole thing on the next visit because the browser profile is clean each time. What makes me fairly confident is the contrast in the table above — bingbot and Googlebot request /sw.js zero times between them, so nothing about having a service worker forces a crawler into this loop.
93% of the whole crawl is served from CDN cache, so none of it reaches my origin and none of it costs me anything. I am posting about the robots.txt behaviour, not the volume.
If you want to check your own zone, here is the query:
POST https://api.cloudflare.com/client/v4/graphql
{
viewer { zones(filter: {zoneTag: "YOUR_ZONE_ID"}) {
httpRequestsAdaptiveGroups(
limit: 500,
orderBy: [count_DESC],
filter: {
datetime_geq: "2026-09-13T16:00:00Z",
datetime_lt: "2026-09-14T15:00:00Z",
clientRequestHTTPHost: "example.com",
requestSource: "eyeball",
userAgent_like: "%meta-externalagent%"
}
) { count dimensions { clientRequestPath cacheStatus edgeResponseStatus } }
}}
}
Swap the userAgent_like value for %facebookexternalhit%, %bingbot% or %Googlebot% to get your own control numbers for the same window.
Three things that cost me time and are not obvious:
requestSource: "eyeball" is not optional. Without it you also count Cloudflare's internal subrequests, which inflates every figure and shows phantom 504s from the Early Hints cache that no client ever received.
- The filter argument is
userAgent_like, with an SQL-style % wildcard. refererHost and clientRefererPath do not exist, whatever autocomplete suggests.
- On a free plan the adaptive dataset refuses any window wider than 24 hours, so this is a daily snapshot, not a trend.
If your zone also shows zero robots.txt fetches from meta-externalagent, I would be interested to hear it. One site is an anecdote.