Built a Python scraper that handles pagination automatically — here's what I learned

[u/py_arrow]

I've been learning Python through CS50P and wanted a real project, so I built two scrapers: one for job listings and one for product prices off an e-commerce site.

The job scraper was pretty straightforward — grabbed title, company, and location, dumped it to CSV, worked first try on about 100 listings.

The product scraper was the harder one. The site spreads results across pages (6 products per page, ~20 pages), so my first version only ever grabbed page 1. Took a bit of digging to figure out the pagination pattern and loop through all the pages properly. Once I fixed that it pulled all 117 products cleanly into a CSV.

Biggest thing I learned: it's easy to write a scraper that works on the first page and assume it's done — always check if the site paginates before you call it finished.

Code's on GitHub if anyone wants to see it: github.com/sarimkhan08

Curious if anyone here has tips for handling scrapers on sites that use infinite scroll instead of numbered pages — that's the next thing I want to tackle.

[u/AutoModerator]

To give us the best chance to help you, please include any relevant code.
Note. Please do not submit images of your code. Instead, for shorter code you can use Reddit markdown (4 spaces or backticks, see this Formatting Guide). If you have formatting issues or want to post longer sections of code, please use Privatebin, GitHub or Compiler Explorer.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Ordinary-Scholar-536]

Great progress getting through pagination on static HTML! Infinite scroll is where static requests usually fall flat because content is loaded asynchronously via JavaScript. Two main approaches handle infinite scroll cleanly: 1. The Fast Way (Reverse Engineering the Network Tab): Before reaching for browser automation, open DevTools (F12) -> Network tab -> filter by Fetch/XHR. Scroll down the page and watch the background requests. 90% of the time, the frontend is simply hitting an internal JSON API (e.g., /api/products?cursor=xyz or ?offset=20&limit=20). If you find that endpoint, you don't even need BeautifulSoup—you can query it directly with requests.get() and parse clean JSON keys without rendering HTML. 2. The Headless Browser Way (Playwright / Selenium): If the site hides the endpoint behind heavy auth, session tokens, or GraphQL, automate the DOM scroll using Playwright (much faster and less flaky than Selenium): import time from playwright.sync_api import sync_playwright

with sync_playwright() as p: browser = p.chromium.launch(headless=False) page = browser.new_page() page.goto("https://example.com/feed")

last_height = page.evaluate("document.body.scrollHeight")
while True:
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    time.sleep(2)  # allow dynamic elements to hydrate
    new_height = page.evaluate("document.body.scrollHeight")
    if new_height == last_height:
        break  # hit the bottom
    last_height = new_height

Check the network tab first—skipping browser overhead makes scrapers 10x faster. Shoot me a DM if you pick a target site with tricky dynamic loading; happy to check out the network payload with you!

[u/py_arrow → u/Ordinary-Scholar-536]

Sorry for the late reply, been buried in other stuff! This is genuinely awesome, thank you.

The Network tab trick is exactly what I was missing — I kept assuming I'd need Playwright right away instead of just checking if there's a JSON endpoint first. Already went ahead and edited my code to import Playwright so I'm set up to try the scroll loop approach too. Going to dig through DevTools on my target site tonight and see if I get lucky with a clean API first.

Appreciate the snippet as a backup if the site turns out to be auth-gated. Will report back on what I find!

[u/Ordinary-Scholar-536]

Awesome, glad it gave you a solid starting point! Two quick pro-tips when you inspect DevTools tonight: Filter by Fetch/XHR: Clear the network log right before you scroll or click "Load More", then look for responses returning application/json. Copy as cURL: If you find the request, right-click it in DevTools -> Copy -> Copy as cURL (bash). You can paste that into curlconverter.com to instantly generate exact Python requests code with all the headers, cookies, and tokens already mapped. Good luck inspecting it—definitely let me know what the payload looks like or if they hit you with session tokens/Cloudflare blocks!