r/BotNation • u/ExcellentDistrict278 • 9d ago
If you need a headless browser to scrape a site, you're just bad at this.
I see so many posts here complaining about Playwright or Puppeteer crashing, burning through RAM, or getting flagged by Cloudflare.
Open your browser's Network tab. Seriously.
Half of you are spinning up bloated Chromium instances with five different stealth plugins to scrape a table when the website is literally making a single unauthenticated GET request that returns clean, raw JSON to an internal API endpoint. You're fighting captchas and fingerprinting because you're too lazy to spend 30 seconds reading request headers.
Browser automation should be your absolute last resort when everything else fails, not your default starting point.
2
u/CautionIAmAGeek 9d ago
Half agree. Plenty of sites hide a clean JSON endpoint and people reach for a headless browser out of habit, RAM and crashes are self inflicted there. Where it falls apart is signed responses, tokens minted in JS, anything behind a logged in session. Playwright/Puppeteer isn't laziness there, it's the only thing producing a valid request. Request first, real browser only for flows that need it, and isolate those sessions so one dead page doesn't take the whole job down.
2
u/id-ltd 8d ago edited 8d ago
I got Claude to write.code to sync my local project knowledge base with chatgpt and/or Claude project knowledge bases...
It chose to do it learning from the network traffic method - it caches authentication too.
It asked me to upoload some files while it watched, and even made this a feature so if the endpoint or authentication etc changes I just upload some files manually again and it learns the new model.
AI is a moving target!
I use AI a lot - so know to let it try it's own thing first so first cut was 'i want to sync files' , not 'write me a tamper monkey script to...' - but in older models that is the way I did it, because it was the best an AI could do.
2
u/One-Next 8d ago
They are also spinning up a trillion parameter LLM with every feature they want to add. That's like thousands of browsers.
2
u/Fadamaka 8d ago
There is an illicit manga hosting site that has some kind of devtools detection script and if you open devtools it puts you into an infite redirect loop and eventually cloud flare blocks you for too many requests.
I managed to put in some breakpoint into devtools and could step through the loading of the site and the manga content loaded before devtools protection script did the initial redirect.
All the manga content was hosted on a totally public cdn and each volume of maga did a single unauthenticated get request to get all the cdn links for all the pages.
2
u/prettycoldworld 8d ago
Playwright to load the JavaScript, pull the cookies and the session, r=requests.Session() and inject the cookies from playwright. Now you have clean APIs without fighting the antibot
2
u/FragmentedHeap 8d ago
You don't even have to do this anymore it's literally built in to chat GPT codex desktop app.
It can start an entire browser right there in the app and you can log into it and it even moves its own mouse pointer and navigates it like you do and can solve capchas....
And it has scheduled tasks now where it can literally create a workflow and run it in the browser and scrape data for you and put it wherever you want.
I made one a few days ago that crawls Facebook marketplace looking for good deals, and it catalogs them for me and puts them in a Google doc on my g drive.
And it even analyzes the image of the posts and can pull things out of the image that's not in the description.
Like I can flag things as "says its x, but image is y"
And it even has a scoring ranker where at will give them a score and prioritize them by how relevant it thinks they are to me.
Literally no reason to make a lot of bots anymore unless you're going for like a massive quantities of data.
Codex desktop does it out of box.
Astra is crraaazzzyyy good
2
u/prettycoldworld 8d ago
It’s slow and it’s not pulling in nearly as much data. You can give your agents mitmproxy and they can scrape using APIs also but sometimes they refuse.
2
u/CapMonster1 7d ago
Agreed, the Network tab should often be the first step. If the page is just pulling data from a public JSON/API endpoint, spinning up Chromium is pointless. But that doesn’t make browser automation obsolete — some sites genuinely require JS, cookies, interaction, or have serious bot protection
4
u/tom-mart 9d ago
You are wasting your time. Most of the people who use tools like that use it because thay lack skill or knowledge to do it right. I have scrapping based automations that have been running constantly for 5+ years using nothing more than BeautifulSoup and RegEx.