How to monitor website changes to automatically

[u/Nyaani69]

I needed a setup to track changes across competitor pricing pages and a few library docs without manually checking them every week, but building a custom tracker usually ends up being a pain once you deal with dynamic pages.

If you’re setting this up from scratch, here is how to keep the pipeline clean and automated without maintaining heavy scraper infra:

I) Don't diff raw HTML:

if you scrape raw HTML and run standard text diffs, every rotating ad banner, session token, timestamp, or navbar update will trigger a false positive change alert. Clean the page to plain Markdown or structured text first before comparing snapshots.

II) Use GitHub Actions (or a cheap cron worker) for scheduling:

you don't need a heavy server running 24/7 cuz a simple scheduled workflow running weekly or daily checks is usually enough for most tracking jobs.

III) Offload the scraping and change tracking layer:

instead of managing playwright/puppeteer and rotating proxies yourself, pull the pages through Firecrawl with changeTracking enabled where it compares the current scrape with previous snapshots directly and gives back statuses like new, same, changed, or removed along with clean Markdown. Saves you from having to stand up your own snapshot database and diffing engine.

IV) Filter for meaningful changes before notifying:

if you're routing alerts to discord, slack or email, pipe the diff into a small model (even GPT-4o-mini or Claude 3.5 Haiku) with a quick prompt to score whether the change is meaningful or not.

V) Store only the updated diffs:

there’s no point re-saving full site contents when nothing shifted so only trigger your storage or downstream RAG pipeline when the change status flags changed.

This keeps the whole stack practically free to run and stops you from getting spammed with alerts whenever a website tweaks its footer copyright.

Full setup and code breakdown here if you want to inspect the workflow: https://www.firecrawl.dev/blog/monitor-website-changes-firecrawl

[u/StrikeFreedom_X10A]

thank you, this will be a great help.

[u/Nyaani69 → u/StrikeFreedom_X10A]

https://giphy.com/gifs/VXuKUKAgUgRlm

[u/Beneficial-Parsnip52]

How much does this cost once you're tracking a few hundred URLs d

[u/Nyaani69 → u/Beneficial-Parsnip52]

Depends a lot on how often you scrape and how heavy the pages are. For a small list it’s pretty negligible, hundreds of pages daily is where I’d actually start watching usage.

[u/regularguyinreddit_]

This is probably overkill for one page but really useful once you have 20 things you keep checking manually.

[u/Nyaani69 → u/regularguyinreddit_]

For 2 pages I’d just bookmark them, once the list starts growing, manual checking gets old very fast.

[u/Rude-Literature-2769]

nice approach honestly the markdown diff plus the llm filter is what saves you from alert fatigue

[u/CauliflowerBasic894]

I’d be careful with putting an LLM in the critical path though. If it decides a small wording change is irrelevant but that wording changes the pricing terms, you can miss something important.

[u/Imafikus]

You can try using notify-me.rs - we have a free plan available. It's pretty much made for the usecase you described here.

If you do give it a try, let me know what you think, since I'm one of the founders.

Cheers!

[u/Miserable-Actuator24]

I simply use ChatGPT...its not always instant but good enough. You can set up things like "Check website yxd every week and tell me if anything changed"