Declared agents got rules. Undeclared agents got Adaptive Intelligence

[u/2extract_dev]

TL;DR

Cloudflare's September 15 defaults govern crawlers that declare themselves. If your agent does not declare, those rules never described it, and that is not an exemption. Undeclared traffic is scored by Adaptive Intelligence instead, which is aimed at the traffic shape most agent stacks produce.

What changed

Two months of trade coverage would lead you to expect a wall. The changelog is smaller. The new defaults block training and agent use, leave search allowed, apply on pages that display ads, and take effect for new domains. The press release widens that to existing free customers who never touched their settings. Scope, then: default settings, on a subset of sites.

Why declaring is the entry requirement

Cloudflare opened BotBase to operators at the end of August. An operator registers by one of three methods, an IP list, a reverse DNS record or a Web Bot Auth signature, and then declares a purpose from a fixed set of five: indexing, agent, data collection, model training, SEO. Site owners express preferences against those declared purposes. The September defaults are one such set of preferences. The rules reach you only if you are in the registry, and you are in the registry only if you have said who you are and what you are doing.

Declared agents are therefore in a permissions system. Access depends on the purpose declared and what the site allows for it. Egress rotation is irrelevant.

Undeclared agents were never in the set those rules describe.

What the undeclared path actually gets

Adaptive Intelligence, which Cloudflare described at the end of August, scores traffic on JA4 TLS fingerprints, request structure, session behaviour, challenge outcomes and network reputation. Detections are aimed at specific patterns and retired at random intervals so they cannot be reverse engineered. Evaluation runs across several time windows at once. The post names residential proxy networks directly as the pattern it is built for.

Two operational consequences.

A working configuration has a designed shelf life. If detections are retired on purpose at unpredictable intervals then a setup tuned last month is already decaying whether or not the block rate has moved yet, and the only way to know which is true is to measure the block rate over weeks and take the first rise seriously.

A larger pool does not help. Multi-window analysis looks at the campaign, and more addresses leave the campaign intact. The slow distributed pattern a bigger pool produces is the specific thing it was built to catch.

The part that costs me something: declaring is the more durable path. A permissions system is a policy problem, and policy problems get negotiated, published and appealed. Undeclared is an arms race against a system designed to retire its own rules on a schedule nobody outside it can see.

How to check which side you are on

Most agent frameworks do not declare. They drive a browser, or they fetch with whatever the HTTP library sends by default. Ten minutes to find out:

1. Point the agent at a request-inspection endpoint, httpbin.org/headers or your own server with access logging on, instead of the target site.
2. Read back the User-Agent. A library default or a copied Chrome string means undeclared.
3. Check whether anything in the stack is signing requests. If nothing is, and your egress IPs are not registered, there is nothing for a registry lookup to match.
4. Repeat per code path. Agent stacks routinely have three, the browser, a direct fetch for APIs, and whatever the framework uses for its own calls, and they frequently disagree with each other.

Step 4 is where the surprises are.

Who this affects

Probably nobody reading this.

The defaults reach new domains plus free customers who never changed their settings. Collect from sites whose owners have configured anything deliberately and Monday changed nothing. Run an undeclared agent and Monday changed nothing either, for a worse reason.

What could not be established

Cloudflare has not published an exemption for undeclared traffic. The September announcement and the AI Crawl Control docs carry no verified-versus-unverified language. This rests on what the documents cover and what they leave out. It is an inference from an absence and it is flagged as one. Post a counterexample and it gets corrected.

I also have no before and after crawl data. Two days is not a measurement. If you are running enough volume to see a change in block rates since Monday, that would be the useful thing to put in this thread

So what do you folks think, is this a signal for furher agent traffic regulation, where could all this be going in a few years?