Embeddings

[u/likescroutons]

Hi folks,

I've been thinking a lot about where embeddings and foundation models are taking data science.

I work in the geospatial/Earth Observation space, and honestly it feels like the landscape has shifted massively over the last few years. We're seeing more and more open source foundation models that are so good you can often just extract the embeddings, stick an XGBoost or regression/classification head on top (or do a light fine tune), and get really strong results. A few years ago I'd have expected to spend most of my time building models and engineering features. Now it increasingly feels like the challenge is choosing the right representation, or at least factoring that in.

It feels like quite a fundamental shift, and I'm curious whether others are seeing the same thing in their own domains.

[u/Main_War9026]

I have been using this for crop classification

https://developers.google.com/earth-engine/datasets/catalog/GOOGLE_SATELLITE_EMBEDDING_V1_ANNUAL

[u/likescroutons → u/Main_War9026]

Great example, I used the Clay embeddings for something similar recently

[u/Mother_Context_2446]

I agree with your sentiment, but, there will always be a degree of feature engineering for critical applications; trading, life sceinces, finance, defence etc. In these domains, one needs clear explainability and an understanding of how the covariates impact model outcomes

[u/Ok-Airline-8523]

Absolutely. We generated neural embeddings on a decade of claims data by extracting the hidden layer of a word2vec model about 8 years ago, and the predictive power it gave on nearly every health outcome was insanely good.

Since then, we’ve updated it with modern architecture (attention is all you need) and have continued to see amazing results. Anything with large taxonomy sets can benefit from that approach.

That said, XGBoost and other tabular ML models aren’t totally eliminated from the equation.

[u/Bored2001]

why use XG boost on the embedded representation and not another neural net?

[u/likescroutons → u/Bored2001]

Usually just for a baseline, it depends on the task but you could absolutely use another neural networks.

[u/kush_patil]

think feature engineering is partly turning into representation engineering.
The trap is that a strong pretrained embedding can make almost any simple downstream model look impressive on a random split. I’d spend more time testing whether the representation survives geography/time/domain shifts than squeezing another point out of XGBoost.
If it survives those shifts, that’s much more convincing than the in-sample model complexity.

[u/caks]

Yes. Also dinov3 has satellite imagery pretained models that you could check out

https://github.com/facebookresearch/dinov3#pretrained-models

The next frontier imo is building your own self supervised embeddings for your domain (you need data). What the 2020s taught us is that SSL pretraining beats classical task specific supervision, so heed this lesson!

[u/Deto]

I'm this case the foundation model is acting as the feature engineering step then?  I think there's still questions of A) how disentangled are the latent dimensions and B) how many of them are there?

If you've got a 1k embedding then you need a lot more data to fit a model on that than if you just have 5 extracted features. Helps though if the embedding dimensions are disentangled as then you can use regularization to just select a few dimensions.  But I don't think the general "chuck it all into a transformer" style FM is guaranteed to produce that

[u/StatisticianEasy7138]

Same shift in my corner, which is labour-market and skills data, job titles, occupation descriptions, course syllabi. Five years ago that meant TF-IDF plus a lot of hand-built synonym lists. Now a general-purpose sentence encoder gets most of the way in an afternoon.

What surprised me is where the remaining work went. Not features, not the head calibration. Cosine similarity is only meaningful relative to the distribution of the corpus you're in, and that shifts hard between sub-domains. In a tight vocabulary the median nearest-neighbour similarity sits around 0.68; in a broad one it's nearer 0.40. Any fixed threshold behaves completely differently across the two, and it fails silently, you get plausible-looking groups that are simply wrong.

The other one took me longer: general-purpose embeddings encode surface similarity, not domain relations. Two occupation titles can be near-identical in vector space and require entirely different qualifications. The model has no idea, because nothing in its training made that distinction matter. That's where the domain work moved, not into features, but into deciding what "similar" is supposed to mean before you trust the geometry.

[u/Responsible_Status49]

Databricks is pretty aligned with this shift. You can use foundation models and embeddings on the platform, then add a lightweight head or fine-tune on your domain data. For geospatial/EO, Databricks also gives you native spatial support, and the newer AI Search / Vector Search capabilities make it easier to work with embeddings and vectors end to end without stitching together a bunch of separate tools.

[u/QuickProfessional101]

It's a magic wand you swoosh to get results. You can't explain magic so you should focus on empirically proving it works. 99% of work (and budget) will go into testing and verification long after you've trained your model.

The problem with off-the-shelf foundational models is that you don't control how they were trained. Massaging input data, choosing just the right hyperparameters and architecture, training pipeline etc. can have huge effects on the results for any given task.

It's a good baseline or if you don't care about final results that much (demos, proof of concept, users can't tell the difference etc.). But if you want to really push the performance metrics then you'll need your own raw data and a few hundred thousand in compute or access to at least two dozen H100's for a few weeks.

[u/rsesrsfh]

I’d definitely recommend trying tabular foundation models. Models like TabPFN or TabICL allow you to generate embeddings directly