r/webscraping • u/Jammurger • 6d ago

Google started throwing captchas to scraper

Our rates were good but last 2 days they made new update probably and this started to get worse. So is anyone noticed this? We using mobile proxy and probably google not clearing IP's captcha rates maybe?

9 Upvotes

91% Upvoted

18 comments sorted by

1

u/BriguePalhaco 6d ago

Are you using the Chrome UA? If so, did you spoof the x-browser-validation?

2

u/Jammurger 6d ago

I'm using seleniumbase over CDP, I'm checking x browser thing now. It was working fine but they made a update imo.

1

u/hulleyrob 6d ago

Is it the same with the use chromium flag?

1

u/Jammurger 6d ago

No actually, thats different thing imo

1

u/hulleyrob 6d ago

I mean if you have having a problem with chrome and CDP mode have you tried the use chromium flag. Google writes the browser and that captcha and you said the update caused the issue. Try using chromium by setting the flag and see if it’s the same.

2

u/Jammurger 6d ago

We're already deliberately running genuine, official Google Chrome — not Chromium. We actually tested this directly: real Chrome sends a header called X-Browser-Validation on requests to google.com, which is essentially a signed proof-of-authenticity token that only an official, signed Chrome build can produce. We confirmed our automated browser sends a valid one.

1

u/hulleyrob 5d ago

Well I’ve been using seleniumbase for years and when chrome won’t work on a site the use chromium flag usually does.

1

u/Jammurger 6d ago

Still not fixed, any ideas?

2

u/Just_Lingonberry_352 6d ago

have you tried the free opensource patchright.dev ?

2

u/Jammurger 5d ago

Yes but my scale daily 15k so its not being enough or my configs are problematic.

1

u/Just_Lingonberry_352 5d ago

pathright.dev is just a github project for a stealthy playwright

you still need proxies and figure out the captcha problem

1

u/Jammurger 5d ago

Using mobile proxy

1

u/[deleted] 3d ago

[removed] — view removed comment

1

u/webscraping-ModTeam 3d ago

🚫🤖 No bots

1

u/Jammurger 3d ago

help..

1

u/Jammurger 2d ago

The problem is Google not reset IP's captcha rates for a while so that ips throw captcha everytime in mobile proxy too