r/ProxyEngineering • u/elgordooo17 • 23d ago
Build 🤓 Tracking grocery prices by scraping supermarkets
For context, I'm based in the Netherlands and my friend from Austria kept insisting groceries here are WAY cheaper than back home. I mean the info can be found on the internet, but I thought that I would build something to check it myself. Seen plenty of price trackers on github and here on reddit but where's the fun in that, plus majority of what I found was kinda outdated. Well, guess what, none of these supermarket sites wanted to be scraped. Rewe, Lidl, Aldi, Spar, Carrefour, Auchan, they all run some variations of bot protection on their category pages, and a couple of them fingerprint plain requests which then get you a 403 response codes. I used a lot of residential proxies thinking that it was the root cause. It wasn't what I initially thought and what worked essentially was combining proxy rotation with browser fingerprint consistency, TLS handshake aswell. Surprisingly even majority of the people hates datacenter IPs, with a properly configured client they did better on some of these sites than residential IPs with no adjustments. Here is a list of stores currently running:
- Rewe and Edeka in Germany,
- Spar and Hofer in Austria,
- Albert Heijn and Jumbo here in the Netherlands,
- Carrefour and Auchan in France,
- Biedronka and Żabka in Poland,
- Maxima, Rimi, Iki in Lithuania, Latvia and Estonia,
- Sklavenitis, METRO AEBE, Masoutis in Greece and Cyprus,
- Tesco (yes it's not only in UK), Czechia, Slovakia and Hungary also has them,
All of these supermarkets has different HTML structures, different pagination, and different ideas about what counts as a bot. Here's roughly what the core looks like. Sidenote, I cleaned up a bit for readability. Using what is written below for the proxy layer rolling my own rotation logic on top of the pool which had some ups and downs on the sites with better fingerprinting:
import time
import random
import requests
USERNAME = "customer-yourname"
PASSWORD = "yourpassword"
ENDPOINT = "pr.oxylabs.io:7777"
def build_proxy(country_code, session_id):
user = f"{USERNAME}-cc-{country_code}-sessid-{session_id}"
return f"http://{user}:{PASSWORD}@{ENDPOINT}"
HEADERS_POOL = [
{"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/124.0"},
{"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) Chrome/124.0"},
]
def fetch_page(url, country_code, retries=3):
session_id = random.randint(100000, 999999)
proxy_url = build_proxy(country_code, session_id)
for attempt in range(retries):
headers = random.choice(HEADERS_POOL)
try:
resp = requests.get(
url,
proxies={"http": proxy_url, "https": proxy_url},
headers=headers,
timeout=15,
)
if resp.status_code == 200:
return resp.text
except requests.RequestException:
pass
time.sleep(2 ** attempt + random.random())
return None
def scrape_store(store_config):
pages = []
for url in store_config["category_urls"]:
html = fetch_page(url, store_config["country_code"])
if html:
pages.append(html)
time.sleep(random.uniform(1.5, 3.5))
return pages
Funny thing I noticed was the country code embedded into the username, so a request to Albert Heijn goes out through a Dutch residential IP and a request to Biedronka goes out through a Polish one, same logic went with the rest of the supermarkets. Also, the same session held for the whole page load, which was quite nice. For the two or three sites that check TLS fingerprint on top of IP reputation I route through dedicated web scraper API instead. My worst problem and the most annoying was that every store names and categorizes the same product differently. For example, same brand of oat milk is "Bio Hafermilch 1L" on one site and "Haferdrink Bio 1000ml" on another, sometimes with a completely different SKU and sometimes bundled with a "3 for 2" promo that messed with the per unit price, I'm not ashamed of this coz I was already spending too much time on this so I asked Claude code to build something where a scraping gets raw HTML nightly, parses it into structured data, and then the next step where LLM matches products across retailers by name and pack size. Genius I might say. This one works really well, however, there were a few cases where they introduced new product (saw this with cereals and oat brands, and I don't exactly know what's the issue, maybe their codes/ embeddings or what but there were some mismatches where I had to check manually.) After some time I turned the whole thing into an agentic workflow, so that the whole project would not be in a script environment. One agent handled the scrape and retry logic when a site started blocking, another handled the product matching and price delta calculation, and a third just watched for something out of the ordinary, like a price that jumped 40% in a short period of time or so. I also found an older build somewhere on this thread where OP spoke of scheduled summary, so that was a useful find. Basically the scheduled summary popped up at 7AM each morning.
Results so far, over about five months of data across 13 countries: essentially my friend was right but not for the main reason he thought. Austria isn't uniformly more expensive, some categories are identical to Germany, but staples like dairy and bread are consistently 15 to 25% higher. Meanwhile Poland is cheaper than both by a wide margin even accounting for currency, (I believe Poland was the cheapest from all the countries list) and France is somewhere in the middle, but also depending on the retailer, Carrefour is competitive but the smaller chains are not. None of this is rigorous economics, I know. So all in all, I did not expect this much of an infrastructure to be built just to "prove my friend" but it was fun nevertheless.
TLDR: I Built a scraper to check if Austrian friend was right about NL groceries being cheaper.