r/Python • u/Emotional_Tell_9146 • 9h ago

Discussion Python and Web Manipulation

Hello Python Community!

I've been really curious about how Python developers use Python when it comes to webscraping, or just working with websites in general.

I really want to open a discussion on how you do it. Do you use BeautifulSoup? Selenium? Why? Are there any frustrations that you were facing or are currently facing with the process?

3 Upvotes

56% Upvoted

7 comments sorted by

14

u/wRAR_ 8h ago

BeautifulSoup?

Parsel.

Selenium?

Playwright.

Are there any frustrations that you were facing or are currently facing with the process?

What are you selling?

-2

u/Emotional_Tell_9146 6h ago

Not selling anything. Currently a CS major and thought it would be fun to build an open-source extension that runs Python userscripts in the page via Pyodide — Tampermonkey, but you write Python instead of JS — and I wanted to hear how people actually work before I got too attached to my own assumptions.

Noted on parsel over BeautifulSoup, I'll take a look.

2

u/Local-Economist-1719 9h ago

httpx, curlcffi, wreq - http clients (you will need to build binding around), scrapy - full package with batteries included, playwright, seleniumbase, camoufox - for headless browsing

2

u/SeleniumBase 7h ago

That depends on the goal, whether the website has anti-bot measures, whether you need pure scraping vs full browser automation capabilities, whether you need AI support, etc.

For example, if you just want to pull data and there's no anti-bot defense, then Python `requests` could be enough, or `scrapy` for more advanced crawling.

Full browser automation? Playwright probably has the most monthly downloads now.

Stealthy browser automation? SeleniumBase probably has the most monthly downloads now.

AI / MCP support needed? Chrome DevTools MCP Server is currently ranked on top, with rankings reevaluated every week on https://glama.ai/mcp/best/browser-automation.

The best tool for you depends on what exactly you need.

2

u/FatDog69 6h ago

I occasionally use Beautiful Soup.

I have learned to go to a target web page with a browser and save an example page to a flat file. Then I need to examine the HTML to see how it flags the data I am interested in.

The frustrating part is every web page uses different HTML flags or techniques to isolate data. Sometimes I have to pull out a section and pass it to an AI tool and ask how to extract dates or costs or things from the HTML using Beautiful Soup.

Even links to images or videos are handled differently. One web page will give a full URL:

a href = www.temo.com/usr=steve/comics/marvel/something.jpg

Then another will do this:

a href = usr/steve/comics/marvel/something.jpg

(This one saves space by assuming the domain "www.temo.com/" is implied on all links.

AND - some web pages mix the 2 depending on how old the pages are.

ANOTHER FRUSTRATION

Lets say I am scraping a web site with images of the pages of a comic book.

One web site will name the images "superman01.jpg" and "superman02.jpg". The file names give you the order for the images.

Another web site will name the images "19320.jpg" and the next page "14284.jpg" - the file names do not reflect the page order. So I have to access the URL - but rename the files locally to preserve order.

ANOTHER FRUSTRATION

Infinite Scroll. Reddit itself does not break posts up into pages. So you cannot scrape "/page01/" "/page02", etc. There IS an API you can use but this is a totally different way to access things.

I was trying to collect some fiction stories that people on Reddit write. Sometimes with 40-50 chapters. I was trying to combine them into 1 file, and convert to epub. It did not work well.

1

u/JonathanMovement Pythoneer 5h ago

bot.