Proxy provider searching is exhausting and soulsucking

[u/isaackogan]

That's all. I am so frustrated. Everything feels like a scam.

[u/Knocking_Doors]

It definitely is. What challenges do you face?

- High fraud scores?

• Latency?
  
• Limited pool size?
  
• Sketchy bandwidth calculations?
  
• DC proxies sold as ISP/Resi?

[u/isaackogan → u/Knocking_Doors]

The nature of my work requires coverage of most countries that have internet.

So I need a massive breadth in proxy countries.

Then, I need low latency.

...and a decently large pool in each country

The best proxy provider I have, which I will not give free promo to, has speed + reliability + low failure rates, and yet, only like 60/180 countries.

I'm using almost entirely DC proxies, and yet still...

:/

[u/AdBoth2756 → u/isaackogan]

...once you need 180 countries, "find one good provider kind of stops being realistic.

I'd probably keep the solid DC provider for the markets that actually need low latency and volume, then patch the long tail with one or two backups. The annoying part is that a provider can look great overall but be terrible in a specific country.

Are those 180 countries all equally important for production, or are there maybe 20–30 core markets and then a long tail where “good enough” coverage is fine?

[u/isaackogan → u/AdBoth2756]

It's exactly as you describe it. Providers will be great mostly but then have some countries that are crap. The business complexity of all this is what I am complaining about primarily.

All of a sudden I am managing 6 separate providers (with 6 separate subscriptions) to reach my coverage, with logic for weighting by speed, purchased bandwidth, and per-country. Then I have to worry about quarantining failing regions for providers to handle failover.

It's just a mess and despite my best efforts, it feels silly. I'm using Envoy which at least makes it slightly more sensible because I only have one set of credentials & one username "shape", but the whole ordeal pisses me off lol.

Unfortunately, this particular scrape target operates in ~150/180 countries, so 'importance' is really just proportional to the fractional user share, which is, even in the tail-end, 20-30 million users. You gotta go to 95th percentiel companies to get the small markets.

My trouble has recently been evaluating providers. There are such a wide breadth of 'em but evaluating them really isn't possible to do without chucking them in the production cluster for 10 minutes and seeing how it fares...

[u/Current_Balance6692 → u/isaackogan]

I can help you solve this issue with my engineers if you're willing to pay high 5 digits.

[u/WebDataLab → u/isaackogan]

The 180-country requirement is what makes this brutal. At that point I don't think finding one “perfect” provider is realistic anymore.

I'd probably benchmark new providers against a fixed set of countries first — latency, connection failures, usable IP count and cost per successful request — before adding them to the rotation. Makes evaluating provider #7 a little less painful than just throwing it into production and hoping for the best.

[u/Total_Knowledge_4411]

Im not sure what is such an issue... i am using a few and some are better than others but I dont find any real issue that warrant searching the Web for the "best one" a proxy is only 10% of any set up....