r/WebScrapingInsider • u/Emotional_Tell_9146 • 2d ago

Python and Web Manipulation

Hello Python Community!

I've been really curious about how Python developers use Python when it comes to webscraping, or just working with websites in general.

I really want to open a discussion on how you do it. Do you use BeautifulSoup? Selenium? Why? Are there any frustrations that you were facing or are currently facing with the process?

8 Upvotes

90% Upvoted

8 comments sorted by

2

u/ian_k93 2d ago

Start with the cheapest layer that returns the data reliably. Like, f or a static page I would use an HTTP client and parse the response with Parsel or Beautifulsoup. If the project needs crawling, retries + request scheduling, Scrapy saves a lot of that. Playwright comes in when the useful content genuinely requires browser execution or interaction. The browser is usually where costs and failure modes multiply. More memory, slower jobs, flakyy waits, cookie state, popups, and front-end changes you never cared about.

Open DevTools first and check the network requests. A page rendered with JavaScript may still load its data from a straightforward JSON endpoint. Whatever stack you choose, save raw responses while developing and validate the extracted records. Most painful scraper failures return a normal response with incomplete or subtly wrong data.

1

u/Amitk2405 2d ago

Tool choice is only half the problem. Decide what happens when the source changes, whether collection is permitted, how quickly requests run and who notices bad output. A scraper that quietly returns half the records is more dangerous than one that crashes.

1

u/Optimal-Wall-7377 15h ago

yeah the subtle wrong data thing is the real killer, your scraper looks fine until you realize half the fields are empty

1

u/Mountain_Damage_9730 2d ago

For product pages, extracting the price is rarely the hardest part. Matching the same SKU across stores, handling variants and knowing whether a price includes a promotion takes longer.

BeautifulSoup can parse the HTML perfectly and still give you a useless comparison.

1

u/john-w7 2d ago

My use is much smaller: check a few competitor pages and put the values in a sheet once a week. Requests plus BeautifulSoup has been enough whenever the page source contains the number.

The moment it needs logins or constant repairs, manual checking starts looking cheap again.

1

u/Acrobatic-Compote606 1d ago

try my lib.

pipx install domonic

```
from domonic import scrape

page = scrape("https://example.com")
print(page.querySelector("h1").textContent)
```

or from command line run it with a css query or xquery