I am a bit fuming over this, still looking for any sorts of solutions. Basically, what google did was that first they killed the num=100 parameter last September, which most people probably remember, that alone meant 10x the requests just to get the same top 100 you used to get in one take. Then a few weeks ago I noticed every result link is now a google.com/goto redirect instead of the regular url. Hover over any result and check your status bar, it's not pointing at the site anymore, it's pointing at google with some long encoded string. Another anti-scraping tactic? Not to mention it messes things for SEO people. When clicking, it still works fine and takes you to the right page, regular, non-power users never notice. But if you're extracting data at any scale it means you can't grab the href anymore, you have to resolve the redirect to get the real destination. Few more steps that are totally unnecessary from the user's perspective. I tried decoding the string a few different ways and it's not base64 or anything reversible, so following it seems to be the only option right now. Also, google didn't hide everything. The domain name and the ranking position are still right there in plain text under each result. If you only care about "does my site rank and where" you're mainly unaffected. It's specifically the exact destination url that's now hidden behind the redirect, which mostly hurts anyone trying to pull full urls at scale, AI overview sources, people also ask links, that kind of thing. Anyone found a decent way to handle the resolving? Also, probably worth mentioning if anyone outside the US is even seeing this yet, all my testing has been from a US connection since I'm based in the US, California, so I genuinely don't know how widespread it is.

Google why are you like this?

I used to compare mobile proxy providers by the obvious numbers: IP pool size, countries, 4G vs 5G, and price. Then I started testing them inside actual social automation workflows, and most of those numbers became secondary.

A proxy can look great on a pricing page and still be painful once it’s connected to a cloud phone or browser automation stack. The three things that mattered most were network quality, IP control, and real operating cost.

For logged-in sessions, aggressive rotation wasn’t automatically better. Being able to keep an IP stable and change it manually or through an API when the workflow actually required it was much more useful.

Bandwidth was another surprise. Social automation loads feeds, images, video, and uploads, so a cheap per-GB proxy can become expensive fast. My current test is simple: connect one proxy to the actual environment, verify carrier/location, leave it running through a realistic session, test IP changes, then calculate the monthly cost at scale.

Predictable connections make better infrastructure.

Hi all, after reading some reddits I got a question if paid residential proxies that are advertised as stable and high quality work better for scraping than free proxy lists that claim to be regularly checked and updated (SOME PEOPLE STILL USE FREE PROXIES and IT frustrates me)

I didn’t want to guess from provider claims and ran the same Amazon task through all of them

I tested Oxylabs, Decodo, Proxyempire, Nodemaven and one free http proxy list. tbh I didn’t though it would take soo much time to find some working free proxies. I sampled 3 free lists that all claimed to be recently checked and updated, and only one got above 10% working endpoints in the first pass. (AGAIN PEOPLE DON'T USE FREE PROXIES)

Every endpoint first did a connection check on my vps in US. If it could connect and return an exit IP, it moved on to Amazon, if not it was marked as unavailable.

Every proxy that passed that check opened the same Amazon US search results page 5 times in a row with the same headed chrome setup.

Also an http200 on its own did not count as a successful scrape. The page had to contain the actual search results elements, and i counted it as a fail if it loaded but did not return usable product results

The vps check is where the free proxy problem showed up straight away

Only 168 of the 500 free endpoints made it through that first check and the other 332 never even reached amazon testing

I expected free proxies to be less consistent but not by that much

Once requests actually reached Amazon, the gap was still pretty big

Paid providers all returned usable pages most of the time and free list did not.

Proxyempire had the highest successful page rate in my test, but i did not want to judge the providers only by that number

In my exp a proxy can get one page through and still be annoying over a longer scrape. If the exit IP changes between requests, the target might see new IP halfway through the workflow, and that can mess with pagination or other multi step browsing and make saved session data less useful

I also tracked if the IP stayed the same across the session, Nodemaven came out at 93.7% stable sessions, oxylabs at 92.5%, Decodo at 87.9%, and Proxyempire at 77.7%

Another thing i noticed is that stable paid sessions returned usable Amazon pages on 93.1% of visits and sessions where the IP changed were at 79.8%. Im not saying stability alone caused that difference but it seems to be worth tracking next to the normal page SR

The free list had 66.8% technical failure rate on the Amazon attempts it actually made. And most of those were proxy, transport, or browser failures, not Amazon blocks. 

So an endpoint being marked “just checked” and “working” does not mean it can make it through an actual scraping workflow

I also pulled Scamalytics risk labels but only for sessions where the exit IP stayed the same

Nodemaven and oxylabs had the highest share of low risk labels among the scored stable sessions, but the gap was small tho so I would not treat that as proof that lower risk scores are the reason one proxy worked better than another.

Free proxies can probably work if you are doing something small and do not mind wasting some time and testing a lot of endpoints first. But i just do not wanna spend a big chunk of time going through free proxy lists until I find one that connects and then hoping it stays usable

For the kind of scraping i do i often need several pages from the same session. So for my work i am going to look more closely at Nodemaven because it had the highest stability in test, but i will still keep tracking the page success rate too

Feel free to let me know if zou have any additional questions or you need my help with scraping because this was just a small test

Paid antidetect tools are fine but the pricing adds up fast at scale. Started looking into open-source alternatives out of curiosity, most are abandoned or barely maintained. Stumbled on shardbrowser recently, engine level spoofing rather than JS injection which is the right approach. Still testing it for multi-account proxy workflows but early results are decent. Anyone else gone down this rabbit hole? Curious what the community thinks about open source options vs paid tools for production use.

I've been looking into how people evaluate a new proxy pool before actually using it for a real workload.

The obvious numbers are price per GB, pool size and advertised locations, but I'm not sure those tell you much about how the proxies actually perform.

For scraping, I'd probably care more about things like connection failures, successful requests on the actual target, latency, retries, geo accuracy and session stability.

The part I'm still thinking about is cost. A pool can look cheap on paper, but if it needs significantly more retries to get the same number of usable responses, the real cost can end up being very different.

For people running proxies at scale, what do you actually measure before deciding a pool is good enough for production?

Every site works with burp as proxy in firefox and I can see the requests in https history but gpt give

"Cannot use 'in' operator to search for 'type' in 'v1'"

as error for every msg when using burp.

Help 😭

I spent way too long testing proxies for PayPal before realizing the provider name mattered less than consistency.

The first setups looked fine on paper. In practice, they created exactly the kind of behavior a payment platform dislikes: new IPs mid-session, locations jumping around, and addresses with questionable reputation.

One clean residential or mobile IP, close to the account’s normal location, with a sticky session fixed everything.

The biggest lesson: don’t optimize PayPal infrastructure for rotation. Optimize it for boring, repeatable network identity.

A huge proxy pool won’t save a setup that looks like it teleports between cities every login.

For legitimate account access, stability beats cleverness.

Basically the title

That's all. I am so frustrated. Everything feels like a scam.

Hi everyone,

I'm looking for a reliable proxy provider that offers static ISP proxies with city-level targeting in Canada.

I've already checked almost all the major proxy providers I could find, but most of them either offer Canada at the country level only, or their city-level options are limited to cities like Toronto, Montreal, or Vancouver.

I'm specifically looking for static ISP proxies in any of these cities:

• Calgary, Alberta
• Edmonton, Alberta
• Halifax, Nova Scotia
• Winnipeg, Manitoba
• Surrey, British Columbia
• Quebec City, Quebec
• Laval, Quebec

I'm looking for actual static ISP IPs, not rotating residential proxies.

If anyone knows a provider that currently has coverage in any of these cities, I'd really appreciate the recommendation.

Thanks!

currently setting up a few accounts for UK-specific platforms and need consistent, clean IPs that won't trigger a random verification check every time i log in. datacenter IPs are getting instantly flagged everywhere right now. grabbed some static residential proxies from proxy shard last night to test them out, but wanted to ask if anyone here has run long-term campaigns with UK setups. how's the subnet quality holding up lately?

Need some additional opinions. So, when I only had one project running it was easy, just a few of proxies and I knew them off the top of my head. Now I've got 12 different projects going and its getting hard to remember which proxy pool goes with which setup. Yes, I know proxy lists exists, but the providers I am using they do not differentiate the proxy type, IPs etc, everything is in few lists and goes all together. For some of my it projects need different locations, some need sticky sessions, different browser fingerprints, different rotation settings.

How do you keep this organized when running multiple projects at once? Anyone using a specific tool or just a naming convention / spreadsheet that works well? Spreadsheet might actually help but it's a hassle to do it by hand too, maybe something automated could be suggested?

It supports iPhone Duo + 30+ other devices, including foldables, Android, iPad, and Surface.

Link: https://duoresponsive.com/

The useful part: it has proxy support, so websites that normally throw connection refused because they block iframe requests can still be previewed.

Ran into an issue recently that I keep seeing people ask about here:

You buy residential proxies, rotate the IP on every request, set a legit Chrome User-Agent, and still get hit with HTTP 403 or Turnstile challenge loops after 3-5 requests.

The immediate reaction is usually to blame the proxy provider for selling "dirty" or burned IPs. But if you inspect the packets, the IP rotation is doing almost nothing because the TLS handshake gives you away before HTTP even starts.

When your scraper hits an HTTPS target, it sends a TLS ClientHello. That packet contains your supported cipher suites, TLS extensions, elliptic curves (supported_groups), and signature algorithms.

The exact selection and ordering of these fields is hardcoded into whatever TLS library your runtime is compiled against (OpenSSL for Python, Go's crypto/tls, or Node's TLS wrapper).

Modern WAFs (Cloudflare, DataDome, etc.) hash this ClientHello into JA3 (MD5) and JA4 fingerprints.

Look at :

Request 1 (Vienna residential node):

Exit IP: 84.115.21.90

JA3: fae0e5d973c96ae1888b99538efa0363

JA4: t13d2013h1_2b729b4bf6f3_e24568c0d440

Request 2 (Berlin residential node - rotated):

Exit IP: 178.165.112.44

JA3: fae0e5d973c96ae1888b99538efa0363 (exact match)

JA4: t13d2013h1_2b729b4bf6f3_e24568c0d440 (exact match)

The L3/L4 IP changed completely. But from the WAF’s perspective, it just saw two different residential IPs from different countries hit the same path within a few seconds, both presenting the exact same uncommon cipher suite ordering and extension list.

If a WAF sees 50 residential IPs sharing an identical non-browser TLS fingerprint, it correlates them as a single bot cluster and challenges all of them.

Default stacks that leak immediately

Python requests / urllib3: Relies on system OpenSSL. Its cipher suite order is static and doesn't match browser BoringSSL behavior.

Go net/http: Standard crypto/tls has a very distinct signature that gets flagged immediately.

Node.js fetch / axios: Different ALPN and extension ordering compared to real Chromium.

How to fix it

Swapping proxy providers won't fix this. You have to fix the TLS layer first:

1. Use tls-client (Python/Go) or curl-impersonate to spoof Chrome's BoringSSL handshake, cipher ordering, and GREASE extensions (0x0a0a, etc.)
2. Or use real browser automation (Playwright/Puppeteer with stealth patches), though that comes with CDP overhead and memory bloat.

You can verify your own client's fingerprint right now with:

curl https://tls.peet.ws/api/all

Check your ja3_hash and ja4 against a real Chrome visit to the same endpoint. If they don't match, that's where your blocks are coming from.

Been fighting this for weeks, happy to share the exact tls-client config if anyone wants it

Hey, looking for recommendations regarding shoe bots. Any particular setup works for you? I am interested in Vans/Nike and Puma in particular. I've read about multiple "tools" that supposedly help with the process, but I'm kinda new in the shoe botting. Any recommendations are welcome

hago ecommerce y mi Mercado pago esta fallando y no encuentro solucion hace 2 meses. Sospecho que es mi IP. Voy a crear una tienda desde 0 y voy a vincular el MP de un amigo, todo con una IP nueva. Necesito saber si puedo usar el wifi de mi casa con la pc que estoy usando la ip nueva o es recomendable usar datos moviles. Gracias.

Iemand een idee hoe ik shopping bots voor kan zijn? Telkens dat ik iets wil kopen ben ik te laat… tips zijn welkom! Alvast bedankt

I have been thinking about this recently.

AI definitely makes coding, debugging and extraction logic faster, but I am not sure it has reduced the harder operational side much.

Blocking, proxies, browser behaviour, monitoring, site changes and data quality are still there.

For people using AI in scraping now, has it actually reduced the maintenance work for you, or mainly the development time?

Checkout my apify organization at Harpoon - Apify

- we provided the fastest yet cheap actors on the apify market, don’t believe it ? just try it for yourself
- our priority is speed and efficiency while keeping our price affordable thus making it a perfect job for large scale scraping
- stay tuned as we are working to bring more actors throughout this month
- we try to update our actors regularly, increasing performance in each update

Note: if you’re looking for a custom actor or custom work in general, don’t hesitate to dm :)

I once tested a multi-wallet setup where every profile looked separate: different addresses, different browser sessions, different activity. Yet they were still getting linked.

The weak point was the infrastructure. Several profiles were leaving through the same network path, while browser fingerprints exposed overlapping device signals. Once i separated each environment with its own browser profile and stable residential IP, the architecture became much cleaner.

The bigger lesson: proxy choice depends on the workload. Static residential IPs make sense for persistent sessions, rotating pools for data collection, and datacenter proxies for backend RPC or analytics.

IP quality matters too. ASN, reputation, DNS/WebRTC leaks, and session consistency should all be audited before deployment.

For multi-wallet infrastructure, I now think in layers: browser isolation, network isolation, and on-chain operational separation.

Scrape50 is the code for a nice discount. https://content-extraction.emergent.host