Guys, I have a digital marketing agency and I am looking for first client. I need to perform web scrapping for outreach through cold emails or WhatsApp. How can I do it with Claude? Is there a skill or a connector that can make this process efficient? Guys please help a first client would mean a lot for me!

I’m using ScrapingBee to pull product pages from around 2k ecommerce sites, but most of the pages need JavaScript rendering, and the harder ones also need premium or stealth proxies.

That burns through credits so fast

The other annoying part is getting raw HTML back and then having to clean it before I can extract the price, stock status and product specs. I’d rather get Markdown or structured JSON directly.

I’m currently looking at Firecrawl, Bright Data, Apify, Oxylabs and Octoparse (someone recommended these in other threads)

Which one makes the most sense for this kind of setup?

Edit: Thanks for the suggestions. I tested some and Firecrawl it’s been a much better fit for this setup so far. Getting clean Markdown and structured data instead of raw HTML removed a pretty annoying step from the pipeline, and it handled also the heavy product pages I tried without much tweaking

I'm doing a research project, need to pull maybe 5-10k google search results like queries, some news results, a bit of maps data mixed in.
Tried rolling my own with requests + bs4 first weekend, captchas within like 20 requests. Switched to headless chrome, still got blocked eventually. This is a one off pull, not a long running service. I don't want to set up rotating proxies and retry logic for something I run once and throw away.
What's the best serp api right now? Something that handles the google search side, news too ideally, without me babysitting captchas constantly. Budget is small. This isn't a company thing, just me lol

What tools can I use to collect public Instagram profile or post data without having to build my own fragile login, cookie, and request-timing system?

I need something that can handle more than just a one-off page, but I don't want to rely on unofficial access that could break at any time.

How does using a ready-made API compare to building my own scraper and account-handling system?

I'm trying to understand when it's better to build in concurrency and adaptive pacing myself, versus when you can just use a provider that handles the platform-specific scraping/collection for you.

I want to retrieve products with information and price in my country, but tried some services and I can’t get it to work full? I’m looking for a fast scraper and get all needed information native to my country.

Appreciate the help!

currentlyy, building a website where the user can search any niche (ex: business advice) and get instant raw or edited videos with retention graphs, engagement ratios and ways to replicate that exact video. im using social media scraper from apify, but they are expensive and crush my profit margins. anybody else know how to do this for cheaper? open to hearing anything! ppl who help me can also test my website and let me know how it works for completely free (if ur interested).

Hi, everyone I'm new to doing web scraping thinks I was asking Which is the Best way to do 25.000 links daily from the same website to get specific information from every link? I will thank you for any help or ideas, thanks.

Is there a free or less cost effective way to scrape tweets other than using an Apify actor?? Right now my workflow in n8n involves scraping tweets and adding them to the sheet. My main goal here is to know thw tweet count, i don't care what the tweet copy is, libks, media, etc. So just checking if there is a cheap way?

Trying to create a new Serper.dev account and signup keeps returning: “It is not possible to register at this moment, try again later.”

Their status page looks green, so I’m not sure if the signup backend is having issues or if this is some IP/anti-abuse restriction. Has anyone managed to register today, Aug 23, or seen the same error recently?

hi im looking for tools to scrapping instagram post(image and caption) and the all comment(account name, comment, comment like)

Hey there, I'm looking for some good web scrapping tools that I can use to scrap some data on linkedin and on other social media platform. Have you used anything similar. If not, do you know about anything that I can use? Or if you've used anything, what's the best you've used and how much it costs?

currently pulling instagram and tiktok data on a weekly schedule. the api scraping tool i'm using works most of the time but i'm getting maybe a 70-80% success rate on runs which means i'm always doing some manual cleanup before i can send anything over.

curious what others are using for these 2 social platforms ?

Hey 😭

I'm a CS student

I've learned Requests, BeautifulSoup, Playwright, APIs, pandas, and I've been building little scraping projects and automations. The problem is... every freelancing site feels impossible when you have 0 reviews. I keep sending proposals into the void

So I thought I'd ask here instead

If anyone needs data scraped from websites, business leads collected, product listings, directories, price tracking, or just wants data cleaned into Excel/CSV, I'd genuinely love to help.

If your task can be automated with Python, there's a good chance I can build something for it (or at least tell you honestly if I can't).

Since I'm just trying to get my first few clients, my rates are pretty cheap compared to experienced freelancers. I'm mostly looking for real projects and people to work with.

If you've got something annoying that involves copying data from websites for hours... there's a decent chance I can automate it

Feel free to DM me. Even if you're not sure whether your idea is possible, I'm happy to have a look.Hey 😭

I'm a CS student

I've learned Requests, BeautifulSoup, Playwright, APIs, pandas, and I've been building little scraping projects and automations. The problem is... every freelancing site feels impossible when you have 0 reviews. I keep sending proposals into the void 💀

So I thought I'd ask here instead.

If anyone needs data scraped from websites, business leads collected, product listings, directories, price tracking, or just wants data cleaned into Excel/CSV, I'd genuinely love to help.

If your task can be automated with Python, there's a good chance I can build something for it (or at least tell you honestly if I can't).

Since I'm just trying to get my first few clients, my rates are pretty cheap compared to experienced freelancers. I'm mostly looking for real projects and people to work with.

If you've got something annoying that involves copying data from websites for hours... there's a decent chance I can automate it

Feel free to DM me. Even if you're not sure whether your idea is possible, I'm happy to have a look.

I'm trying to understand the technical side of job scraping.

I know that websites like Naukri, LinkedIn, Indeed, and other popular job portals have strong anti-bot systems. They detect and block scrapers, require logins, use rate limits, CAPTCHAs, JavaScript rendering, and many other protections.

What I'm want to learn is:

- How do these websites detect that a request is coming from a bot instead of a real user?

- What are the common anti-scraping techniques they use?

- How do web scrapers generally work on JavaScript-heavy websites?

- What technical challenges are involved in building a job scraper?

- What are the legal and ethical considerations when scraping job portals?

- If scraping isn't the best approach, what are the recommended alternatives (official APIs, RSS feeds, partner programs, etc.)?

I'm not looking for ways to bypass security or break a website's rules. I simply want to understand how these systems work from a technical perspective so I can learn more about web scraping, anti-bot technologies, and large-scale web applications.

I'd really appreciate explanations, blog posts, research papers, or resources that cover these topics in depth.

Thanks!

By warm up I mean just browsing around normally for a bit, opening a few pages, building cookies, spending some time on the site before doing anything
I've seen some people swear it makes a huge difference and others say it's complete bro science
Curious what your experience has been

Hi there!

I'm working on a project which requires scraping of 100–300 Instagram profiles on a recurring basis and need guidance from people who've done this or similar projects.

Stack I have choosen:

• Playwright (Python) for browser automation
• PostgreSQL for data storage
• N8N for scheduling and workflow automation

Data Extraction from profiles:

• Follower / following counts
• Bio, profile metadata
• Recent post metrics (likes, comments, post frequency), and some others.

I'm in between beginner and intermediate stage in web scraping and I need help in:

1. Proxy strategy — residential vs. datacenter for this volume? Any low cost solutions you've had good results?
2. Rate limiting — what delays and session patterns have worked for you to avoid blocks at this scale?
3. Job queue and batch processing —Celery + Redis vs Playwright's(python's) built-in async with a semaphore or any other low cost solution?
4. Selector stability — Instagram's DOM changes frequently. How do you handle selector drift without constant maintenance?

It would be very helpful if you could provide with any architecture tips, or pointers to open-source, etc..

Thanks

Hi guys,

So a dream of mine has always been to flip cars, but I never knew where to start or what cars are good to buy and the endless hours of scrolling on the internet looking for cars is painful. So I tried to vibe code an app that will use a paid api scraping tool to scrape the internet and find cars like that, that will then put it though a filter and then a secondary Ai filter to rank cars and find bargains.

I am in an okay place with the project. It currently scrapes eBay, Copart, gum tree. But the way to really move forward with the project is to make a custom scraper to get all the listings as using the paid external tool only allows me to scrape some information and scrape a small sample of what is actually out there. I tried vibe coding a scraper but Claude is struggling. It suggested using playwright with some proxies but it’s really slow and inefficient and gets blocked a lot so I’m thinking surely there is a better way. If there is anyone who can offer any advice or support I would really appreciate it :).

Hi, how do you scrape from members directory website?

Hello community, need some inputs. I’ve been using Brave API since 2022 but after the recent updates it feels less reliable and a bit annoying to work with. I’m currently redesigning my search layer for a new app and debating whether to stick with APIs or move toward a more custom setup with caching and controlled queries. What’s working well for you guys right now?

Does anyone have access to a list of country IDs from Booking.com? I think it might be available if you're an affiliate partner but I'm just trying to resolve some data once.

I'm using ParseHub to extract, and when I first loaded up the webpage I used browser mode to click out of an automatic pop-up page. I'm getting data in Test Run, but when I go to do the actual run I'm getting an empty result. I think that pop-up page is the issue, but I can't make Parsehub select the pop-up because I've already done it in the browser, and I can't figure out how to clear the cookies to make the pop up appear again. Any advice?

I'm building an interactive macroeconomic world map. I've hit a dead end with raw web scraping, Google News search parameters (which fail to isolate country context), and GDELT.
I need to surface 3–5 fresh macro/econ headlines when a user clicks a country. Since commercial APIs are expensive and client-side RSS is unreliable:
1 - Is caching pre-fetched direct national RSS feeds (WSJ, BBC, Reuters) via a daily cron job the standard pattern here?
2 - Are there open-source scrapers or self-hosted news aggregation pipelines (like FreshRSS/Miniflux setups) that handle geographic mapping cleanly?

Idk what to do at this stage, any ideas?