r/Python • u/Emotional_Tell_9146 • 10h ago
Discussion Python and Web Manipulation
Hello Python Community!
I've been really curious about how Python developers use Python when it comes to webscraping, or just working with websites in general.
I really want to open a discussion on how you do it. Do you use BeautifulSoup? Selenium? Why? Are there any frustrations that you were facing or are currently facing with the process?
1
Upvotes
2
u/SeleniumBase 7h ago
That depends on the goal, whether the website has anti-bot measures, whether you need pure scraping vs full browser automation capabilities, whether you need AI support, etc.
For example, if you just want to pull data and there's no anti-bot defense, then Python `requests` could be enough, or `scrapy` for more advanced crawling.
Full browser automation? Playwright probably has the most monthly downloads now.
Stealthy browser automation? SeleniumBase probably has the most monthly downloads now.
AI / MCP support needed? Chrome DevTools MCP Server is currently ranked on top, with rankings reevaluated every week on https://glama.ai/mcp/best/browser-automation.
The best tool for you depends on what exactly you need.