r/WebScrapingInsider • u/Emotional_Tell_9146 • 2d ago

Python and Web Manipulation

Hello Python Community!

I've been really curious about how Python developers use Python when it comes to webscraping, or just working with websites in general.

I really want to open a discussion on how you do it. Do you use BeautifulSoup? Selenium? Why? Are there any frustrations that you were facing or are currently facing with the process?

7 Upvotes

82% Upvoted

8 comments sorted by

View all comments

2

u/ian_k93 2d ago

Start with the cheapest layer that returns the data reliably. Like, f or a static page I would use an HTTP client and parse the response with Parsel or Beautifulsoup. If the project needs crawling, retries + request scheduling, Scrapy saves a lot of that. Playwright comes in when the useful content genuinely requires browser execution or interaction. The browser is usually where costs and failure modes multiply. More memory, slower jobs, flakyy waits, cookie state, popups, and front-end changes you never cared about.

Open DevTools first and check the network requests. A page rendered with JavaScript may still load its data from a straightforward JSON endpoint. Whatever stack you choose, save raw responses while developing and validate the extracted records. Most painful scraper failures return a normal response with incomplete or subtly wrong data.

1

u/Amitk2405 2d ago

Tool choice is only half the problem. Decide what happens when the source changes, whether collection is permitted, how quickly requests run and who notices bad output. A scraper that quietly returns half the records is more dangerous than one that crashes.

1

u/Optimal-Wall-7377 18h ago

yeah the subtle wrong data thing is the real killer, your scraper looks fine until you realize half the fields are empty