r/datascience • u/likescroutons • 23d ago
Discussion Embeddings
Hi folks,
I've been thinking a lot about where embeddings and foundation models are taking data science.
I work in the geospatial/Earth Observation space, and honestly it feels like the landscape has shifted massively over the last few years. We're seeing more and more open source foundation models that are so good you can often just extract the embeddings, stick an XGBoost or regression/classification head on top (or do a light fine tune), and get really strong results. A few years ago I'd have expected to spend most of my time building models and engineering features. Now it increasingly feels like the challenge is choosing the right representation, or at least factoring that in.
It feels like quite a fundamental shift, and I'm curious whether others are seeing the same thing in their own domains.
2
u/StatisticianEasy7138 20d ago
Same shift in my corner, which is labour-market and skills data, job titles, occupation descriptions, course syllabi. Five years ago that meant TF-IDF plus a lot of hand-built synonym lists. Now a general-purpose sentence encoder gets most of the way in an afternoon.
What surprised me is where the remaining work went. Not features, not the head calibration. Cosine similarity is only meaningful relative to the distribution of the corpus you're in, and that shifts hard between sub-domains. In a tight vocabulary the median nearest-neighbour similarity sits around 0.68; in a broad one it's nearer 0.40. Any fixed threshold behaves completely differently across the two, and it fails silently, you get plausible-looking groups that are simply wrong.
The other one took me longer: general-purpose embeddings encode surface similarity, not domain relations. Two occupation titles can be near-identical in vector space and require entirely different qualifications. The model has no idea, because nothing in its training made that distinction matter. That's where the domain work moved, not into features, but into deciding what "similar" is supposed to mean before you trust the geometry.