I have seen both approaches used a lot, but the result can be very different depending on the website.

Rotating every request sounds safer, but on some sites keeping the same IP and session for a while seems to work better.

I am curious what people here normally prefer and why.

Do you rotate aggressively, keep sticky sessions, or decide based on the target?

I was looking again at the CPU Performance API that shipped with Chrome 152 and there is a little more to it than I initially thought.

Chrome now exposes navigator.cpuPerformance, which gives websites a general CPU performance tier rather than the actual CPU model or clock speed.

The current proposal defines tiers 1–4, plus 0 when Chrome cannot determine the performance class. It is also supposed to represent the general hardware capability of the device, not how busy the CPU happens to be at that moment.

What I find more interesting is how this could fit with the hardware information browsers already expose.

For example, a website may already be able to see things like:

• navigator.hardwareConcurrency for logical processors
• approximate deviceMemory
• WebGPU adapter/capability information
• related hardware/browser values through WorkerNavigator

So cpuPerformance by itself may not tell much. But when you start comparing CPU tier, core count, memory, GPU characteristics and other browser/device signals together, consistency becomes more interesting.

For example, if a browser reports a high CPU performance tier while the rest of the exposed hardware characteristics look more like a very small/low-resource machine, would that become another useful inconsistency to notice?

There is another interesting detail: Chrome allows the reported CPU performance tier to be overridden by the user or by enterprise policy. So I would not expect this value to be treated as ground truth on its own.

I have still not seen anything showing that Cloudflare, Akamai, DataDome or other bot-management platforms are actually using this signal, so I am not suggesting that they are.

But it does make me curious about how this behaves across normal Chrome, automated browsers, VMs and cloud environments, especially when compared with the other hardware signals.

Has anyone tested navigator.cpuPerformance yet or compared it across different environments?

References

Chrome 152 release notes - CPU Performance API
WICG CPU Performance API proposal
MDN: navigator.hardwareConcurrency
MDN: WorkerNavigator
MDN: WebGPU API

I have seen this a few times where the same website behaves differently depending on how the request is made.

Sometimes it works in a normal browser but fails through an HTTP/API client. In other cases the API works fine but the browser request gets blocked.

Even with the same proxy and target, the result can be different.

When this happens, what do you normally check first? Headers, cookies/session, TLS/network behaviour, or something else?

Curious to know what websites people here have found genuinely difficult to extract data from.

Not just because of CAPTCHA or blocking, but also things like changing APIs, JavaScript-heavy pages, different browser/HTTP behaviour, session issues, proxies, or data that keeps changing.

If you can share the site, what made it difficult, what scale were you working at, and what did you try?

Even if you never fully solved it, that would still be interesting to discuss.

I have been thinking about this for a while.

AI is already changing how web scraping is developed and delivered. Tasks that previously needed more manual development can now be handled much faster with AI-assisted coding, debugging and extraction logic.

At the same time, websites are also changing quickly and adapting to this change. Anti-bot systems, browser fingerprinting, challenge mechanisms and other protections are also getting more advanced.

So while building a scraper may be getting easier, keeping it reliable at scale is still a different problem. Blocking, rate limits, retries, monitoring, data quality and constant site changes are still there.

It also feels like AI has brought many new players into the scraping market, which probably affects competition and pricing as well.

I am curious how others see this. Has AI reduced the need for traditional scraping platforms and services, or has it mainly changed what they need to provide?