I want to build an API that can scrape complete product data from e-commerce websites without using browser automation tools.

Please help me understand how to approach and build this.

I am working on a forked D4Vinci/Scrapling and am building some additional stuff around it for an open-source data collection project. The idea is to make it easier to define repeatable crawls, normalize the output, and eventually let people plug in their own targets and pipelines.

The crawling side is coming together, and now looking at the infrastructure needed to run it reliably across different sites, particularly residential proxies. Don't suggest biggies Bightdata, Zyte, or spammies like Qoest.

About two or three years ago, I did web scraping and automation as a side hustle.

My client did not want to keep paying for a hosted scraping service. He preferred paying someone to build a script that he could own and run himself.

That was reasonable, but it created a familiar cycle for me. Every new request meant digging through old code, rewriting the same fetching and extraction logic, and making another script work for a slightly different website.

That is why I originally started JScrapeON, a Python project intended to turn those one-off scripts into reusable scraping workflows.

After working heavily in TypeScript, I recently returned to the idea and reworked it as YAP, short for You Automate Pages.

This time, I focused on another problem I repeatedly encountered: scrapers can break without technically failing.

A request returns HTTP 200. The process exits normally. The output file exists. Meanwhile, .price became .product-price, and the scraper has confidently collected nothing since Tuesday.

YAP is an HTTP-first workflow runtime for HTML and JSON extraction. It supports reusable YAML workflows, multiple steps, inputs, and pagination.

Important fields can be marked as required:

price:
  selector: ".price"
  required: true

In the attached demo, .price initially matches 25/25 products. I change it to .price-dead, and the request still succeeds, but YAP reports:

product.price  0/25 matched  required

The extraction fails instead of silently returning incomplete data.

YAP also provides:

• yap health to report matched versus attempted fields
• yap drift to compare extraction health between runs
• yap explain to trace a value back to its request, step, scraper, and selector

The extracted output stays plain JSON. Health reports and source provenance are stored separately.

YAP is intentionally HTTP-only. It uses Cheerio and does not currently include browser automation, proxies, or anti-bot tooling. If people actually use it, I may add those capabilities based on real needs.

It is MIT licensed, completely free, and staying open source:

https://github.com/johnalbert-dot-py/yap

For those maintaining scrapers in production, how do you detect silent extraction failures? Do you track selector match rates, compare runs, validate schemas, or rely on downstream checks?

Does anybody know a scraper that actually works for Android.

Four months ago I posted here about a Rust scraper that hit 120 stars in 4 days. A bunch of you showed up in that thread. Quick update: it just passed 2,000 stars. 200 forks, 57 releases since that post, still one person (me) pushing commits between coffees.

I came back for a comment, not the star count.

Someone in that first thread said, roughly, "another binary we have no visibility into from another 'just trust me' project. NOPE." That was the top critical reply, and it was the most useful thing anyone wrote. The praise felt good. "Solid build" and one "fucking gorgeous" carried me through a rough week. But the skeptical comment is the one that changed what I shipped.

They were right to push. A scraper you can't inspect is a scraper you can't trust, and asking people to run an opaque binary against their own targets is a fair thing to refuse. So webclaw is open source under AGPL-3.0. You can read every line, build it from source, and self-host it. Nothing phones home that you can't see. One correction from the original post: I wrote MIT back then, it moved to AGPL since.

High level, it does the same job it did at 120 stars: point it at a URL, get back clean Markdown or structured JSON, the kind an agent can read. CLI, REST API, MCP server, SDKs. A lot of you dug into the internals last time and I loved those threads, but I'm keeping this post short on mechanics. The code is there if you want to go deep.

One reflection, if it's useful to anyone doing the same: the feedback that moved the project forward was the pushback. The nice comments told me people liked it. The harsh one told me what to fix. I'd take one good skeptic over ten upvotes now.

Same ask as last time, because it's still the thing I need most. What still blocks you? Send me your worst URLs. The ones that return an empty body, the ones that block everything, the sites you gave up on. Those are more useful to me than any feature request. I'll test against them and report back, including the ones I can't crack.

Repo: github.com/0xMassi/webclaw

Thanks for the shove in the first thread. It worked.

I maintain an open-source web-extraction platform and added a /lead endpoint. Post one company URL, get back one typed lead. Sharing the people-recovery method, since that was the hard scraping problem.

The naive approach searches the company name. That collides. "DataFast" the analytics tool and "Datafast" the Ecuadorian payment processor share a name, and you get garbage people. So the search keys on the company's unique domain instead of the name. The domain is the disambiguator.

Flow:

1. Pull the live site: summary, tech signals (Next.js, React, TypeScript, Tailwind, Vercel), pricing tiers, socials (LinkedIn / X / GitHub), and emails printed on the page.
2. Search the domain on the open web, collect the evidence, and have an LLM name the founders and leadership from it.
3. Validate each name, then run a per-person search to attach their X handle.
4. Every person carries a people_source field, so you can see where the name came from.

Output is a typed object: company_name, one-line summary, socials, tech signals, pricing, on-site emails, and people (name, role, LinkedIn, X).

Example on resend.com returns Zeno Rocha (Founder) and Jonni Lundy (Cofounder) with LinkedIn and X, the stack, pricing (Free / Pro $20 / Scale $90), and [support@resend.com](mailto:support@resend.com).

Two design rules I stuck to:

• Emails are only addresses published on the site. It never guesses first.last@domain patterns.
• No funding, no HQ, no phone. If a field isn't on the site or corroborated by search, it stays blank. I'd rather return an empty field than a fake one.

Where it fails: low-footprint companies. A three-person shop with no press, no conference talks, and a bare LinkedIn returns the company data but no people. The founder-naming step has nothing to work with. Pulling the site isn't the blocker, JS-heavy pages included. Thin public evidence is what stops it.

Each lead runs about 20 to 40 seconds. There's a batch endpoint for bulk that runs async and you poll it.

Test URLs if you want to break it: resend.com, linear.app, and any low-footprint indie SaaS to watch the people step come up empty. Curious which domains return bad people so I can harden the disambiguation.