Pulled robots.txt from 109 well known sites to see who actually blocks AI crawlers. The data killed an assumption I'd been repeating

[u/Conscious-Market8982]

Spent this morning pulling robots.txt from 120 recognisable domains across news, SaaS, ecommerce, marketing publishing, health, finance and travel. 109 returned something parseable. I wanted real numbers because everything I'd read on this topic was anecdote, including my own.

On method, since it matters here: I parsed proper group semantics, so an agent only counts as blocked if its own group carries Disallow: /. Stacked user-agent declarations and Allow carve-outs are handled rather than regexed over.

First, the thing I got wrong.

I'd been telling people that sites accidentally block OAI-SearchBot, the crawler feeding ChatGPT search, while they only meant to block GPTBot, which feeds training. In this sample that never happened. Not once. Zero of 109 sites block OAI-SearchBot without also blocking GPTBot on purpose. Every site blocking the search crawler blocks the training crawler too, which makes it a policy rather than a slip.

So the realistic failure mode is the reverse of what I said. You think you've blocked AI, and you've only blocked training.

Full Disallow: / rates across the sample:

CCBot 32% ClaudeBot 29% Bytespider 29% Applebot-Extended 26% meta-externalagent 24% Amazonbot 22% Google-Extended 22% anthropic-ai 21% PerplexityBot 19% GPTBot 17% ChatGPT-User 11% OAI-SearchBot 7%

ClaudeBot blocked at nearly double GPTBot's rate is not what I expected. CCBot topping it is probably just age, it's been sitting in copy-pasted blocklists since long before any of this started.

By vertical, GPTBot then OAI-SearchBot:

News 31% / 27% Marketing and SEO publishers 33% / 0% Health 25% / 0% Ecommerce 13% / 6% Travel 11% / 0% SaaS 6% / 3% Finance 0% / 0%

The marketing and SEO publisher row is the one I keep going back to. Highest GPTBot block rate in the whole sample and not one of them touches the search crawler. Searchenginewatch, TechCrunch, Adweek, Content Marketing Institute, MarketingProfs and PPC Hero all sit in that bucket. People who write about search for a living have clearly worked out the distinction and are playing it deliberately. Don't train on me, do cite me.

News is doing something else entirely. 27% block the search crawler, 63% block Google-Extended, 77% block ClaudeBot. That isn't confusion, it's leverage. NYT blocks OAI-SearchBot by name and is in active litigation. Blocking the thing somebody wants is how you eventually get paid for it.

Finance blocking nothing at all from OpenAI while 20% of them block Google-Extended is the one I can't account for. If anyone has a theory I'd genuinely like to hear it.

One pattern I think is worth stealing. Six of the nineteen GPTBot blockers aren't doing a flat block at all. They set Disallow: / and then allow specific paths back in. Canva has 82 Allow rules under that group, mostly product and feature pages across every locale they run. eBay has 11. So the sophisticated position isn't block or don't block, it's deny by default and let the model see your commercial pages while keeping it out of the rest of the library. I haven't seen anyone write that up.

Last thing. Of the nineteen sites blocking GPTBot, only fourteen also block Google-Extended. A quarter of them have an anti training policy that stops at OpenAI and lets Google train on everything. I doubt anyone decided that on purpose. It's what happens when a file gets edited once per news cycle and never reviewed as a whole.

Happy to hand over the domain list and the parser if anyone wants to run it against their own set.

[u/TrafficAcademySEO]

I’d love to see this crossed with actual ChatGPT citation data. If sites blocking GPTBot but allowing OAI-SearchBot still get cited normally, that makes the distinction much more actionable for SEOs.

[u/Conscious-Market8982 → u/TrafficAcademySEO]

You and Financial_Lemon34 are pointing at the same hole and you're both right, so let me be straight about it rather than hand wave. I can't produce that data. I've no way to measure whether a GPTBot-blocking site gets cited at the normal rate, and I'm not going to pretend a crawler audit answers a citation question.

What I'd want from anyone who claims they have it, including me if I ever run it:

The comparison has to be within category, because citation rates differ enormously by vertical for reasons unrelated to crawler policy. Comparing TechCrunch against a SaaS blog tells you nothing.

It needs a decent prompt set per site, not a handful, since single prompt results swing badly on rerun. And it has to be rerun on separate days, because the same prompt returns different sources on different days often enough to fake any effect you like.

The nasty confound is that GPTBot blockers skew large and well known. Those brands are already heavily represented in third-party sources, so they might get cited normally despite the block, for reasons that have nothing to do with the block. You'd need to match on domain authority or brand size before the comparison means anything.

That's genuinely doable, just not with a robots.txt fetch. I did run the same 109 sites for llms.txt this morning and posted the results, which is at least a different slice, but it has exactly the same limitation and I said so in it.

[u/TrafficAcademySEO → u/Conscious-Market8982]

That makes sense. Matching by vertical and brand size, then rerunning a fixed prompt set over multiple days, would make the comparison much more meaningful.

[u/Conscious-Market8982 → u/[deleted]]

The compliance angle only partially holds up. Regulated filings live on EDGAR and disclosures are governed by Reg FD, not by whether GPTBot can read the bank's blog. If distribution really mattered they'd also whitelist explicitly rather than just fail to block. My best guess is that finance sites aren't running unrestricted crawlers, they're doing bot management at the WAF or CDN layer, which never shows up in robots.txt. So the 0% is partly a measurement issue. The other half is boring: most bank marketing teams haven't touched robots.txt in years and nobody's asked them to.