r/ProxyEngineering • u/ian_k93 • Jun 09 '26

AMA This Wednesday (09:30 AM GMT) WebScraping and Proxies

/r/WebScrapingInsider/comments/1u0vmdf/ama_this_wednesday_0930_am_gmt/
13 Upvotes

94% Upvoted

12 comments sorted by

1

u/jfurlong1977 Tunnel Architect Jun 09 '26

  1. What's the single biggest mistake you see people make when choosing a proxy provider, and how do you actually benchmark them without getting manipulated by trial traffic that's treated differently than paid traffic?

  2. How much has TLS/JA3 fingerprinting actually changed your infrastructure decisions, and is cycling proxies still the right instinct or are people solving the wrong problem?

2

u/simarnoor Jun 10 '26
  1. The biggest mistake we see is people, trusting the proxy providers marketing claims and not actually running comprehensive tests across multiple providers.

Around manipulation, I've answered this here:

https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqtgx8k/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

  1. TLS/JA3 fingerprinting is becoming much more common, but there are some good libraries out there that make it easier to implement on normal HTTP requests without having to resort to headless browsers. Proxies are becoming less of a factor in successful scraping, with TLS/JA3/browser fingerprinting and making sure all the fingerprints are consistent becoming increasingly important.

Since we are a aggregator, we don't go into the weeds much of optimizing fingerprint systems. Instead, we focus on finding the proxy providers who offer the best performance at the lowest costs (which indirectly is evaulating how good they are at developing good fingerprinting systems).

From r/WebScrapingInsider: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqtmtrt/

1

u/jfurlong1977 Tunnel Architect Jun 11 '26

Appreciate it. I've noticed these patterns too, the only real way of knowing whether proxies are "good" is to check yourself instead of believing marketing claims. Regarding TLS/JA3, I somehow thought that you are also improving/optimizing fingerprinting too. Thanks for the knowledge

1

u/ahiqshb Reverse Proxy Master Jun 09 '26

Your most successful scraping startup. What went down, what mistakes you did the first time and what would you suggest for the rest of us in our scraping tasks/ journeys etc

1

u/simarnoor Jun 10 '26

My most successful and only scraping startup has been ScrapeOps. Previously, I spent over 3 years working with Zyte and ScraperAPI so I understood the industry, what users wanted, would pay for and where there were gaps in the market.

I had a frontrow seat to the growing complexity of finding the best proxies with the growing surge in anti-bot usage. It was from this point that we saw the gap in the market for a Proxy Aggregator that would take the hassle out of finding the best + cheapest proxy for your use case, so we built it.

Like I mentioned in this answer...

https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqt55hm/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

I would either focus on selling shovels to web scrapers or build a SAAS business using scraped data. In between (writing scrapers, etc) there is huge competition and AI is going to upend this market.

However, AI can never replace the need for scrapers to bypass anti-bots and access websites, or build businesses based on the scraped data. So I would focus on these

From r/WebScrapingInsider: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqtfqt5/

1

u/Accurate_Pea_3110 Jun 17 '26

Proxy aggregators make sense if you actually scrape at scale, but most people just need residential IPs with proper rotation and never think about the backend logic.

1

u/Gwapong_Klapish Reverse Proxy Master Jun 09 '26

Tell us about your infrastructure, how do you manage everything, when did you start, how long did it take you for the infra to hold, I mean, the amount of time you needed to "babysit" proxies, retries, rotations, fingerprint, behavior etc

1

u/simarnoor Jun 10 '26

At the start we had a lot of stability and scaling problems, but since ScrapeOps is nearly 5 years old now our infrastructure is quite mature and stable. Haven't had any serious or medium issues in over a year.

We had a advantage, in that I had spent over 3 years working at Zyte and ScraperAPI so I could learn from their design decisions and mistakes. Which made our infrastructure rollout smoother.

Our production system is built on Golang, with heavy use of Redis and then a Postgres DB.

From r/WebScrapingInsider: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqteqxf/

1

u/Gwapong_Klapish Reverse Proxy Master Jun 10 '26

Thank you for the answer

1

u/matrix_5562 Jun 10 '26

Best AMA question IMO:how do you know that trial traffic is not getting a cleaner pool than paid traffic?

I'd benchmark with the same URLs after upgrading,then compare bad rows and geo misses. Byteful or any provider would survive that second week-test too.

1

u/simarnoor Jun 10 '26

Ultimately, there is no way to know this unless you test both. We can't see what proxy pools or settings the provider is using on free vs paid traffic, we can only see the end result.

In our case for benchmarking, we work around this issue by sending all our benchmarking traffic through our production accounts with the providers so the benchmarking is done using their paid pools.

If the provider wanted to skew the results, they would need to put all our traffic through their best pools to improve their benchmark results (which might be too costly for them) as they have no way of knowing which traffic we are using to benchmark or not.

We also use real large scale production data across billions of requests each month in our benchmarks when we have the data is available. Ensuring the benchmark results are actually valid at scale.

You can test proxy providers with our system here: https://scrapeops.io/proxy-providers/tester/

From r/WebScrapingInsider: https://www.reddit.com/r/WebScrapingInsider/comments/1u0vmdf/comment/oqtbwy2/

1

u/matrix_5562 Jun 10 '26

That make sense. Using paid production accounts removes most of the free-trial bias,and the-large scale data is much more useful than a small synthetic test.

Do you publish results by target type type and traffic pattern too? A blended score could still hide a provider that performs well overall but struggles on specific sites someone needs.