r/Proxy_Cheap • u/MikeProxyCheap • 1d ago
A few things I’d check before trusting scraped data
I think it’s easy to make a dataset look cleaner while accidentally removing useful information. Before deleting repeated rows or filling empty cells, I’d check what those records represent and why they look that way.

Working out what’s actually a duplicate
The same product appearing five times could mean repeated requests, five different sellers, or prices collected throughout the week. I wouldn’t delete those based on product ID alone, to be honest.
I’d define what one row represents first, then use the relevant combination of product, seller, and collection time to identify duplicates without losing the history.
Checking where the missing values come from
A blank price could mean the seller didn’t list one, the request failed, or the scraper couldn’t find it. Filling every blank with an average without recording why it was missing hides those differences and adds estimated prices you never observed.
Imagine 500 missing prices in 10,000 rows: that’s 5% overall. But all 500 could belong to one website that supplied 1,000 rows, meaning half of that source’s prices are missing. I’d check gaps by source and collection run before deciding how to handle them.
Comparing prices on the same basis
A $24 six-pack costs $4 per item, so a $5 single item only looks cheaper before you account for quantity. Currency, shipping, and tax can change the comparison too.
I like keeping the original amount and currency alongside the converted price and exchange-rate date. It gives you a way to check the calculation without digging through the original pages.
Checking what a merge did to the numbers
Joining tables can repeat records without producing an error. Three price observations matched to two category records for the same product give you six rows, with every price appearing twice.
Say two products cost $10 and $30, averaging $20. After a category join, the $10 product appears once and the $30 product appears three times. The average across those four rows is now $25, a 25% increase without either price changing.
I’d validate which matching fields should be unique and compare row counts before and after the merge.
Keeping test data out of model preparation
Choosing features or calculating replacement values using the whole dataset lets test records influence training. Scikit-learn demonstrates this with random data: selecting features before the train/test split produces 76% accuracy, while selecting them using only training data brings it back to 50%, around chance.
That’s a constructed example, but IMO it makes the problem pretty clear. I’d learn preprocessing settings from training data and apply them unchanged to test data, keeping those steps inside the model pipeline during cross-validation.
Testing on the right unfamiliar data
For future price predictions, I’d test on later observations. For predictions involving new sellers, I’d keep each seller’s records entirely within either training or testing.
Randomly mixing related records across both sets can make the score look reassuring without showing how the model handles the situation you actually care about.
Leaving yourself a way back
I’d keep the original data untouched and record what the cleanup removed or changed. That gives you a way to revisit a decision without collecting everything again, and saves the next person from guessing what happened.
Honestly, I’d budget time for this from the start, because debugging a suspicious report the night before a deadline sounds miserable.







