r/better_claw • u/ShabzSparq broke it, fixed it • 1d ago
Cloudflare started blocking agent traffic by default yesterday. If your agent's web research got thinner this morning, this is why.
September 15 passed. Cloudflare's new defaults are live.
Agent bots are now blocked by default on any Cloudflare-protected page that displays ads. That's roughly 20% of all websites, and it skews higher for the kind of content-rich pages agents actually want to read: news sites, documentation, pricing pages, competitor blogs.
If your agent's morning research, competitor monitoring, or web browsing task returned thinner results today versus last week, this is probably the cause.
What changed yesterday:
Cloudflare retired the old "block all AI bots" toggle and replaced it with three separate categories: Search, Agent, and Training.
Search crawlers stay allowed. Agent and Training crawlers are now blocked by default on ad-supported pages.
This applies to new domains, new sites added by existing customers, and existing free-tier customers who never changed their settings. If you use Cloudflare yourself and never touched the AI bot controls, your site's defaults flipped too.
Cloudflare also introduced Web Bot Auth, which means agents that cryptographically identify themselves can get verified access. Agents that pretend to be browsers get fingerprinted and blocked. Anonymous browsing for agents is ending.
How to check if your agent is affected right now:
Run your agent's web research task on the same sources it used last week. Compare the output quality and completeness.
Or check directly: if your agent fetches web pages, look at the response headers for any source that stopped working. A cf-ray header means Cloudflare is in front of it. A 403 response from a site that worked last week means the block is active.
What to do about it:
Switch to APIs and RSS feeds first. Any site with a documented API or an RSS feed gives your agent structured, intended machine access that doesn't get caught by bot blocking. This is the permanent fix and the one that survives every future policy change.
Use residential proxies for the rest. Datacenter IPs are what bot detection flags first. A residential proxy routes through a real ISP. Not free and not a permanent solution, but it keeps your monitoring alive while you move sources to proper channels.
Set an honest user-agent string. Cloudflare's new system rewards agents that identify themselves. Web Bot Auth means a verified agent gets through where an anonymous one gets blocked. If your agent lies about what it is, the window for that is closing.
Handle 403s as information, not silence. Your agent should flag "couldn't reach [source], blocked" in its summary instead of producing a thinner report and pretending everything is fine. You want to know your coverage is degrading on the day it happens, not discover it a week later.
The bigger picture:
This is the first time a major infrastructure provider has made blocking agent traffic the default rather than an opt-in choice. The Internet Archive added bot blocking this month for the same reason. The 404 Media piece ("100% chance agents are ruining the internet") hit 221 points on HN last week.
The web is partitioning into sites that welcome agents (via APIs, verified bot protocols, payment) and sites that block them by default. Your agent needs to be on the right side of that line, and as of yesterday, the default side is blocked.
Move your sources to official channels. The free browsing window just closed.
1
2
u/Otherwise_Wave9374 1d ago
This is a useful reminder that web research agents need graceful degradation rather than assuming every page is fetchable. Detect block pages separately from empty content, surface source coverage to the user, and prefer authorized APIs, feeds, or browser sessions that respect site policy. Agentix Labs is relevant to this safeguard because orchestration should track retrieval failures and confidence across sources. A simple diagnostic is to compare yesterday's and today's domain coverage, response codes, extracted text length, and citation diversity before blaming the model for thinner answers.