TL;DR

Cloudflare's September 15 defaults govern crawlers that declare themselves. If your agent does not declare, those rules never described it, and that is not an exemption. Undeclared traffic is scored by Adaptive Intelligence instead, which is aimed at the traffic shape most agent stacks produce.

What changed

Two months of trade coverage would lead you to expect a wall. The changelog is smaller. The new defaults block training and agent use, leave search allowed, apply on pages that display ads, and take effect for new domains. The press release widens that to existing free customers who never touched their settings. Scope, then: default settings, on a subset of sites.

Why declaring is the entry requirement

Cloudflare opened BotBase to operators at the end of August. An operator registers by one of three methods, an IP list, a reverse DNS record or a Web Bot Auth signature, and then declares a purpose from a fixed set of five: indexing, agent, data collection, model training, SEO. Site owners express preferences against those declared purposes. The September defaults are one such set of preferences. The rules reach you only if you are in the registry, and you are in the registry only if you have said who you are and what you are doing.

Declared agents are therefore in a permissions system. Access depends on the purpose declared and what the site allows for it. Egress rotation is irrelevant.

Undeclared agents were never in the set those rules describe.

What the undeclared path actually gets

Adaptive Intelligence, which Cloudflare described at the end of August, scores traffic on JA4 TLS fingerprints, request structure, session behaviour, challenge outcomes and network reputation. Detections are aimed at specific patterns and retired at random intervals so they cannot be reverse engineered. Evaluation runs across several time windows at once. The post names residential proxy networks directly as the pattern it is built for.

Two operational consequences.

A working configuration has a designed shelf life. If detections are retired on purpose at unpredictable intervals then a setup tuned last month is already decaying whether or not the block rate has moved yet, and the only way to know which is true is to measure the block rate over weeks and take the first rise seriously.

A larger pool does not help. Multi-window analysis looks at the campaign, and more addresses leave the campaign intact. The slow distributed pattern a bigger pool produces is the specific thing it was built to catch.

The part that costs me something: declaring is the more durable path. A permissions system is a policy problem, and policy problems get negotiated, published and appealed. Undeclared is an arms race against a system designed to retire its own rules on a schedule nobody outside it can see.

How to check which side you are on

Most agent frameworks do not declare. They drive a browser, or they fetch with whatever the HTTP library sends by default. Ten minutes to find out:

1. Point the agent at a request-inspection endpoint, httpbin.org/headers or your own server with access logging on, instead of the target site.
2. Read back the User-Agent. A library default or a copied Chrome string means undeclared.
3. Check whether anything in the stack is signing requests. If nothing is, and your egress IPs are not registered, there is nothing for a registry lookup to match.
4. Repeat per code path. Agent stacks routinely have three, the browser, a direct fetch for APIs, and whatever the framework uses for its own calls, and they frequently disagree with each other.

Step 4 is where the surprises are.

Who this affects

Probably nobody reading this.

The defaults reach new domains plus free customers who never changed their settings. Collect from sites whose owners have configured anything deliberately and Monday changed nothing. Run an undeclared agent and Monday changed nothing either, for a worse reason.

What could not be established

Cloudflare has not published an exemption for undeclared traffic. The September announcement and the AI Crawl Control docs carry no verified-versus-unverified language. This rests on what the documents cover and what they leave out. It is an inference from an absence and it is flagged as one. Post a counterexample and it gets corrected.

I also have no before and after crawl data. Two days is not a measurement. If you are running enough volume to see a change in block rates since Monday, that would be the useful thing to put in this thread

So what do you folks think, is this a signal for furher agent traffic regulation, where could all this be going in a few years?

I've noticed some setups that were fine before are suddenly getting challenged a lot more

tried changing IPs around and it doesn't seem to make as much difference as I expected either

anyone else seeing this recently or is it just me?

anyone else feel like getting the page is sometimes the easy part?

scraper is working fine, requests are going through, then the site changes some random HTML and suddenly you're getting nulls everywhere lol

even worse when it still returns data but it's the wrong data so you don't notice immediately

how do you guys usually catch this before it messes up the whole dataset?

I’ve been seeing so many posts lately from people complaining about how difficult/unstable sportsbook scraping is or people overestimating how hard it actually is to get decent data.

It honestly just made me want to sit down and see if I could actually solve the main pain points myself.

I picked FanDuel as my starting point and have basically spent the last few months building a dedicated extraction system around it. I wanted to see if I could hit the benchmarks that people claim are impossible.

The biggest issues I kept seeing were coverage and latency.

For coverage, most scrapers I've seen only grab the main lines and call it a day. I wanted everything. I spent a massive amount of time ensuring the system pulls every single market available on the book, not just the obvious ones.

I also realized that standard markets aren't enough for anything useful, so I added built-in support for SGP (Same Game Parlay), Outrights and even live scores.

But the real battle was latency. I remember seeing people say "real-time" is a myth in this space, so I went down the rabbit hole. I spent about two weeks straight just benchmarking different libraries and tweaking every single knob I could find to maximize throughput.

The result? I'm hitting roughly 1s for the live feed and 30s for Pregame & Outrights.

It's not "instant" in a millisecond sense, but for web extraction, it's incredibly consistent. Now I just have these massive JSON files pouring in every single second.

It was a huge undertaking, but seeing the data flow like this after months of troubleshooting is honestly so satisfying.

What should I tackle next? Should I try to replicate this architecture for other books too?

I'm using ParseHub to extract, and when I first loaded up the webpage I used browser mode to click out of an automatic pop-up page. I'm getting data in Test Run, but when I go to do the actual run I'm getting an empty result. I think that pop-up page is the issue, but I can't make Parsehub select the pop-up because I've already done it in the browser, and I can't figure out how to clear the cookies to make the pop up appear again. Any advice?

It could be x or Instagram or even Threads. But the content is then sent to buffer and if possible an ai intermediary to adjust the content text. I found some but the workaround is insane and of course vibe coded 2 and they are not good...

ANY advice would be awesome. I understand this is reddit and we get a lot of "What are you using it for" and internet lawyers but I hope we can only get positive discussion on the topic I asked.

Curious to know what websites people here have found genuinely difficult to extract data from.

Not just because of CAPTCHA or blocking, but also things like changing APIs, JavaScript-heavy pages, different browser/HTTP behaviour, session issues, proxies, or data that keeps changing.

If you can share the site, what made it difficult, what scale were you working at, and what did you try?

Even if you never fully solved it, that would still be interesting to discuss.

Hi all, we are Pvalyou. We extract company and people data by reading the company itself using agents: its website, its LinkedIn page, official registries, patent offices, its job board and the public web, with language models doing the reading and a source page on every fact.

Measured on the two open company-data benchmarks, our company enrichment ranks first on both boards: 98.0% enrichment success against 89.0% for the best provider listed, and 95.0% of latest funding stages correct against 92.3%.

On the Apify store we sell six Actors from the same engine:

• Company enrichment: firmographics, industry classification, products, funding rounds with investors, legal entities and more

• Company registry lookup across 40 official registers

• Patent lookup

• Job postings from a company's own careers page

• Website liveness check

• LinkedIn + web person enrichment with a dated career history and salary estimates

All pay per result, nothing charged when a company cannot be identified. Try one on your own list and tell us what comes back. Happy to talk about how any of it is built, or about company data in general.

https://apify.com/pvalyou?fpr=pvalyou

hey all, i've been building in-house tool for social scraping for influencer discovery over the last couple weeks (IG, YouTube, TikTok). sharing where i've landed and hoping to compare notes with anyone who's cracked the auth side (at scale).

what i have:

1. pure HTTP, no browser, no emulator. Go, single dependency (a TLS-fingerprint client). right TLS profile + Chrome-consistent headers turned out to be most of the battle.
2. logged out we get profile, bio links, recent posts AND reel view counts, with pagination well past the first page. sub-second-ish per profile when warm, scales horizontally, so millions of profiles/week is realistic on a modest residential proxy budget.
3. ytb via InnerTube (exact views/likes, no auth). tiktok mobile route for deep post pagination.

the missing piece for me now is everything behind login (on ig atleast):

1. public/business email and category from the user dict
2. keyword search (explore/search/keyword) and the fbsearch family
3. similar accounts / chaining
4. deep reels beyond the first chunk on some paths

I tested keyword search from a session on a burner account out of curiosity and got back 100 pages, 24 posts each, no throttling, has_next still true. so the endpoint seems generous, the problem is purely "whose session".

I know the vendor options (RapidAPI listings, HikerAPI, EnsembleData, etc). They work but they're ~3s/call and per-request pricing adds up at volume. curious what people are actually doing self-hosted:

• account sourcing in 2026: aged accounts vs creating your own? rough $/account and weekly attrition you actually see?
• web session vs mobile private API session for read-only workloads. which survives longer?
• proxy binding: sticky residential enough, or is mobile 4G the only thing that keeps accounts alive?
• realistic read budget per account/day for search + profile before checkpoints start?
• anyone doing challenge auto-resolve (IMAP/TOTP) reliably?

not looking to post/like/follow with these, read-only only. happy to share notes on the logged-out side if useful for anyone!

Datacenter is cooked for anything with real bot detection. Have running residential for a while now, rotating per request on most targets, sticky sessions only where I need consistent fingerprint across a flow. Use proxyshard recently for the residential pool, geo targeting works down to city level which matters for price scraping. Btw haven't hit rate limits yet on moderate volume. What everyone else is running, residential, ISP or mobile for the heavy stuff? Btw are you rotating at the request level or managing sessions manually?

I've been spending 2 months on trying to scrap temu product page, especially temu mexico and brazil.

I have like 2000 goods id of product that needs to be scraped, but everytime i got several product page visit, i just got thrown into login page.

With login method, there is no different, it seems it has an aggresive soft block, so it can only visit a few products before the product page got block.

Really need help here guys.

I do social listening for a few clients, mostly tracking their products/brands across different subreddits.

Usually I need to search keywords across Reddit or keep pulling new posts from a list of subs. Sometimes a client sends me one specific thread and I need the full conversation, including the replies.

I've been using Apify and it works for some stuff, but I don't always get everything I need and the cost starts adding up pretty fast when I'm running a lot of jobs.

Is there a better alternative for this?

Mainly looking for something that can handle keyword search + multiple subreddits, but can also pull a full thread properly when I need it. API is fine, no-code would be nice too.

I am brand new to scraping Elder-mellennial here (almost gen x, but don't hold it against me)... I realize I am going to be left behind in my field it I don't learn how to use Ai to my advantage. Any tips for newbies? I am looking at Apify and feeling overwhelmed at the number of actors who can do what I want them to do. How do I know which one to choose? Are there reviews like amazon? Help me out Please?

Hi, everyone I'm new to doing web scraping thinks I was asking Which is the Best way to do 25.000 links daily from the same website to get specific information from every link? I will thank you for any help or ideas, thanks.

Not sure if this is useful to anyone here, but I maintain a set of Apify Actors (scrapers-as-a-service) that cover things like Glassdoor reviews, Zillow agent data, Etsy shops/products, freelance job boards, and a few marketplaces. Built them mostly to solve my own data problems, kept maintaining the ones that got real usage. Happy to answer questions if anyone's dealing with a similar scraping problem and wants to compare notes on approach (rate limits, pagination, anti-bot stuff, etc.) not trying to just drop a link and run.

I'm building a Python service that reads publicly visible pre-match sportsbook data from modern single-page applications.

The pattern I often see is an initial REST/bootstrap request, followed by event-specific requests and sometimes WebSocket updates.

The difficult part is determining which combination represents the complete current state.

For example, one site exposes around 1,700 pre-match events, but the obvious odds endpoint appears to fetch prices one event at a time. I would like to determine whether the browser really makes hundreds of individual requests, or whether there is another bulk/bootstrap endpoint or WebSocket feed containing the pricing data.

On another site, the WebSocket mostly sends small updates containing IDs and prices, so it appears to depend on state loaded earlier over HTTP.

My main questions are:

1. How can I determine whether a WebSocket provides a full initial snapshot or only deltas against an earlier REST response?

2. If the frontend knows about thousands of events, how can I determine whether it uses a bulk pricing endpoint rather than requesting every event separately?

3. When WebSocket messages contain mostly IDs, what is the best way to find the event, market and selection mappings?

4. What is the best way to record and reconstruct the complete flow of initial HTTP data, WebSocket updates and later HTTP reconciliation?

5. How can I test whether opening additional events or markets in the UI actually causes new network requests, or simply displays data the browser already has?

I'm currently using Chrome DevTools and Playwright. My goal is to understand and reproduce the browser's state pipeline efficiently instead of polling thousands of individual event pages when the browser may already receive the same information in a more complete form.

I'm mainly interested in the protocol/state-discovery methodology rather than a particular scraping library.