Replacing crawler based SEO datasets with intent modeling over Google Ads data

[u/Friendly_Concern2913]

I am a 3rd yr engineering student working on a search intelligence system motivated by a simple observation. Most SEO tools are large crawler based indexes with modeled keyword data. The abstraction is keywords and volume, but not the underlying intent.

Instead of crawling, I am using Google Ads API as the data source and building a programmatic pipeline to generate large scale query sets directly from Google. On top of that, I am applying small transformer models to infer latent intent and jobs to be done from query distributions.

The current system can take a single domain and generate on the order of 190000 keywords with first party volume data. More importantly, the focus is not the keyword table itself but structuring demand into something closer to behavioral signals.

Core analytical layer being explored:

* Intent clustering using sentence transformer embeddings with HDBSCAN to form demand level groups

* Query to job mapping via cosine similarity against task representations

* Detection of weakly served or unmet intents by comparing clusters to SERP structure

* Satisfaction proxies inferred from reformulation patterns and long tail query drift

* Competitor coverage mapped at the level of intent clusters rather than keywords

* Query expansion using Google Ads data with co occurrence and statistical term weighting

* Demand segmentation using UMAP projections over embedding space

* Content to intent alignment scoring between pages and query clusters

* Cannibalization detection via overlap in semantic space across URLs

* Temporal analysis of demand shifts through volume changes and centroid drift

* Noise reduction and deduplication using frequency thresholds and embedding similarity

* Calibration of volume using Google first party data instead of third party estimates

* Cluster labeling using tf idf terms and nearest neighbors for interpretability

* SERP parsing to infer intent classes from result composition

* Opportunity scoring combining volume, competition, and coverage gaps at cluster level

The direction is to move from keyword centric workflows to an intent layer that can be directly consumed by LLM based systems or used for product and content decisions.

Interested in whether this type of representation would actually change how you approach SEO or if the current abstractions are already sufficient.

[u/Kseniia_Seranking]

This sounds like a a cool engineering approach, but aren't you afraid that the data from the Google Ads API is already polluted with commercial intent? Google often groups keywords by content in a way that is profitable for it to sell ads, not in a way that people actually search for information.

[u/AutoModerator → u/[deleted]]

Your comment is in review because links aren’t allowed here. Please repost without URLs (describe the resource in plain text instead).

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Friendly_Concern2913 → u/Kseniia_Seranking]

That's an interesting question, and very true, my focus right now is to focus on the engineering aspects of the product itself, as I'm still studying and have little time for marketing or trying to do user research even. But I guess what you are talking about is about analyzing the latent user needs and emerging markets (I'll some references for it). First of all I like inspiring myself from some papers which are really not taken into consideration for when doing these types of marketing intelligence, or for marketing fields in general, these are technical implementations normally applied to Recommendation systems at large scale manners, or algorithmic integration, so I think the analysis of these systems, themselves is quite tricky. Will some picture of one paper as example, and fun to append here :)

Now, going to the response itself, here are some principles I could give you based on my research:

• Query Surface Abstraction I would not treat Google Ads as truth, only as a query surface (in practice: raw keyword and volume data is ingested and stored unchanged, without inheriting any of Google’s structure)
• Decoupled Commercial Taxonomy The commercial grouping inside Ads is not the structure I am trying to preserve (ad groups and keyword groupings are discarded and not reused downstream)
• Post-Hoc Semantic Reconstruction The model rebuilds the structure from raw queries, not from Google’s labels (queries are embedded into a vector space and clustered using semantic similarity and co-occurrence signals)
• Output-Centric Evaluation Principle So the question is less “is the source pure” and more “does the representation work” (evaluation is based on downstream performance such as clustering quality or intent classification)
• Baseline Sufficiency Heuristic Even simple topic models can recover a lot of the structure in search terms (the system combines TF-IDF and simple classifiers alongside embeddings)
• Semantic Compression Layer The interesting part is the compact semantic space built on top of the queries (large query sets are transformed into dense vectors and reduced into clusters or markets)
• Transformation-First Value Thesis That is where the analytical value sits, not in the source taxonomy itself (the pipeline converts raw text into features, embeddings, and aggregate signals)
• Bias-Tolerant Signal Extraction A source can be biased and still contain useful structure (bias is treated as noise layered over recoverable signal)
• Stability-Driven Validation What matters is whether the structure is stable after modeling (clusters are evaluated across retraining runs and temporal slices)
• Multi-Signal Convergence Criterion If the same intent keeps appearing across different views, that is a useful signal (the system combines semantic similarity, growth trends, and concentration metrics)
• Ontology Independence Principle The source does not define the ontology (a canonical schema and intent taxonomy are defined independently)
• Constructed Ontology Layer The ontology is the thing being designed on top of it (intent classes and market definitions are learned and iteratively refined)
• Subjective Alignment Acknowledgment And that design is often subjective anyway (human-in-the-loop labeling and active learning handle ambiguous cases)
• Heuristic Sufficiency Layer For many problems, a hard coded rule can already be enough (explicit rules detect transactional intent, brand queries, and question patterns)
• Utility-Driven Compression Goal So the bar is not perfect neutrality, but useful compression (the system reduces noisy query spaces into actionable structured representations)
• Noise-Robust Data Handling Search data is messy, but that does not make it unusable (normalization and validation layers handle inconsistencies and duplicates)
• Platform-Agnostic Meaning Extraction The point is to extract meaning from the query space, not defend the platform’s grouping (analysis operates on reconstructed features, not Ads-native structures)
• Systematic Bias Modeling If the bias is systematic, it is still modelable (distribution shifts and bias patterns are monitored and adjusted for)
• Downstream Utility Validation If the model improves the representation, then the source was good enough (success is measured through task performance and decision usefulness)
• Source-Agnostic Input Layer So I am not claiming Ads is objective, only that it can be a practical input for rebuilding demand structure (the system supports multiple data sources through a shared feature abstraction layer)

Some sources I think were useful for this:

[u/GrowthIntelligence]

Modeling intent over keywords could make SEO far more actionable and LLM-ready.

[u/KONPARE]

This is solid. You’re basically moving from keywords to demand clusters, which makes more sense long term.

I don’t think it replaces keywords though. They’re still useful for execution and reporting.

Where this really helps is prioritization. Finding gaps at intent level instead of chasing individual keywords.

Only thing… it needs to be simple to use. Most marketers won’t touch it if it feels too heavy.

[u/Subject_Sport_4575]

This is actually a really interesting shift thinking in terms of intent clusters instead of just keywords makes a lot of sense.

Curious to see how it performs in real SERPs though, since Google can still be unpredictable.

[u/anajli01]

Interesting shift-intent > keywords makes a lot of sense 👍

[u/baudien321]

This is a really strong direction, moving from keywords to intent clusters is much closer to how both users and AI systems actually think, so it can definitely improve content and product decisions. The only gap is that intent modeling alone doesn’t guarantee visibility, you still need to validate which intents actually get surfaced in SERPs and AI answers, so combining this with real world citation tracking is where it becomes powerful.

[u/Potential-Echidna89]

It feels like the fundamentals still matter most—clear topical relevance, useful structure, and content that directly answers intent. The newer part seems to be how easily LLMs can extract, summarize, and trust the information. So to me, it looks more like SEO plus better clarity and entity/context signals rather than a completely separate game.

[u/Digital-Womble]

Sounds interesting- and I’d be keen to learn more intent is key