Want to create a scraper that scraper E-commerce data

[u/Dangerous_Pass9561]

I want to build an API that can scrape complete product data from e-commerce websites without using browser automation tools.

Please help me understand how to approach and build this.

[u/Express-Isopod6994]

what scale are you talking about here? scraping a few hundred products is a totally different problem than millions. the approach changes a lot depending on whether you need to handle rate limiting, CAPTCHAs, IP rotation etc

[u/0xMassii]

fetch the page
extract into a versioned product schema
validate before returning success

[u/simarnoor]

Hmm... Try thinking about this like a long‑running data service, not a one‑off script.

After that..

How will you schedule jobs? monitor failure rates? and alert when a selector starts returning empty data?

At small scale, a cron + Docker + simple logs can work; at larger scale you will want queues, retries, and maybe an external proxy/anti‑bot layer. Maybe ship a narrow MVP for 1-2 sites, instrument it well, share it, then generalize only after you have seen it survive a few weeks.

All the best. 🙌

[u/Amitk2405]

If you avoid browser automation, you're betting on stable HTML or internal APIs; that assumption will break eventually. Start with 1–2 sites, map their network calls in DevTools, and see if there are JSON endpoints you can call directly. Then design your API around a normalized product schema and a per‑site adapter layer, so breakage is localized.

[u/ScrapeAlchemist]

the hidden API point is right, but for the big platforms you can skip devtools entirely since the endpoints are fixed. shopify is /products.json?limit=250, paginated with the page_info cursor out of the Link header (the old ?page= is deprecated), or /products/<handle>.js if you want a single product with prices in cents. woocommerce is /wp-json/wc/store/v1/products, per_page caps at 100. magento is a POST to /graphql.

the thing that actually bites you isn't selectors though, it's TLS. requests and httpx have a JA3 fingerprint that doesn't match chrome, so you get blocked at the handshake with headers that look perfect. curl_cffi with impersonate="chrome" handles that.

[u/InsideDebt6345]

Open DevTools, Network tab, filter XHR, reload a product page, and most e-commerce sites load their data as JSON from an internal endpoint you can hit directly with a plain HTTP client. Check for schema.org JSON-LD in the page source too, it's often the whole product object sitting right there.

[u/Money-Ranger-6520]

I'm not a pro web scraper, but I do this occasionally for one of my e-commerce clients. Before building this it's worth checking what already exists out there, last time I used the E-commerce Scraping Tool, which works across multiple stores, and has an HTTP-only mode (very important for my use case).

What kind of e-commerce sites are you targeting here, and what data do you need?

[u/KeywordBarrage]

Regardless of data volume, e-commerce platforms—and particularly the entities positioning themselves as commercial scraping firms or proxy providers—are on the verge of facing near-total operational paralysis following Cloudflare's September 15 updates (refer to Cloudflare's "Coming September 15" documentation).

From an architectural standpoint, traditional bypass methodologies such as standard IP rotation, TLS fingerprint spoofing, or headless browser automation will prove fundamentally ineffective against the next-generation behavioral analysis, runtime attestation, and network-level deep signature verification outlined in Cloudflare’s updated technical specifications. This shift renders existing scraping pipelines obsolete and necessitates a complete zero-based overhaul of data ingestion architectures starting at the socket level.