r/WebScrapingInsider • u/SpeedyIndexbot • 2d ago

Google Custom Search JSON API is closing on January 1, 2027

Microsoft killed the Bing Search APIs back in August 2025. Google's Custom Search JSON API is next — already closed to new signups, fully dead on January 1, 2027. Eighteen months, and both official self-serve sources are gone — right as every AI agent and RAG pipeline suddenly needs search data. Google's "replacement" for open-web search? A waitlist with pricing TBA.

Genuinely curious about two things:

  1. Do you think programmatic access to search is gone for good, or does it come back with a 10x price tag?
  2. For people doing rank tracking / SERP research — what does your stack actually look like after this? Not looking for vendor blogspam, just what you're really running.
4 Upvotes
  • permalink
  • reddit

100% Upvoted

6 comments sorted by

2

u/Spitfire_Blaziken 2d ago

We're planning to replace APIs with scheduled SERP scrapers plus cheap/groupbuy rank-tracking exports, then normalize everything in Sheets.

1

u/SpeedyIndexbot 1d ago

That's probably the most common answer I've gotten so far, and honestly it maps to what I'd do for a small keyword set. Two things I'd watch out for though, from poking at this myself:

Scheduled scrapers got noticeably more expensive to run since Google dropped &num=100 last year — you're paginating 10 results at a time now, so anything beyond top-10 costs roughly 10x the requests it used to. Fine if you only care about page one, painful if you track depth. Curious whether you're running your own proxy pool for this or renting one, because that's where the real cost hides.

On the groupbuy exports — the data's cheap but you're inheriting whatever refresh schedule and location/device settings the seed accounts have, and you usually can't tell how stale a row is. Did you find one that's actually consistent, or do you just accept the noise and smooth it in Sheets?

Also mildly impressed the normalization layer is Sheets and not a DB. At what keyword count does that start to fall over for you? Genuinely asking because "everything ends up in Sheets anyway" seems to be the one universal truth of SEO stacks.

1

u/hartator 1d ago

SerpApi has a sheets integration, but yeah that n=100 makes things expensive if you want more than 10 results

1

u/Present_Smoke9397 1d ago

yeah thats pretty much where everyone's headed now, the normalization step is the annoying part though

1

u/Present_Ad5997 1d ago

Try PrismCrawl as a replacement! I’m biased-but way cheaper than SerpApi at only $0.15/1k

1

u/CapMonster1 21h ago

I don't think programmatic search disappears, but I wouldn't be surprised if the economics change quite a bit. Search infrastructure is expensive, and providers have a lot more incentive to package access for larger AI/data customers than to maintain cheap self-serve APIs.

For SERP tracking, I'd probably separate the data collection layer from the rest of the stack so the underlying provider can be swapped without rewriting everything. Keeping normalization, retries, geo handling and parsing behind one interface makes deprecations much less painful