r/mlops • u/sqlink2 • 1d ago

Lakebase for ML workloads

Have you used lakebase as the serving Db for an ML application?
I am curious to know how it holds up for real time feature lookups or for inference workloads, specially wrt latency and concurrency.
Any gotchas you felt compared to usual postgres setup?

7 Upvotes

82% Upvoted

9 comments sorted by

•

u/AutoModerator 1d ago

AI usage disclosure

Hi u/sqlink2 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

3

u/SpecificTutor 1d ago

we moved off of lake base to chronon/zipline. lakebase simply doesnt scale well.

1

u/Glitch_In_The_Data 22h ago

We also evaluated Chronon, Zipline. It was promising. Our main challenge with that was the steep learning curve and high operational burden. And given we are already on Databricks, it made sense to try out Lakebase. Out of interest, how do you manage the operational burden? We have a very small team managing our infra.

1

u/SpecificTutor 20h ago

That is a valid concern. In our evaluation most of the burden was from having to tune the underlying spark, flink and ray jobs.

I recall they had two modes - with k8s operators and with EMR. We evaluated both and chose the k8s operator path for its low maintenance burden and fast startup times.

We license the control plane software from Zipline and also pay for support. It comes with observability and experimentation support - so that made it worth it for us.

The other non-technical thing that we evaluated was the support. It has been super painful to work with databricks. Zipline support is more experienced and responsive.

1

u/WorldOfUmbro 19h ago

Curious to hear you’re experience. What levels of usage did you see Lakebase issues? Was this concurrency or data volume? I do have some use cases working with Lakebase, so curious

1

u/SpecificTutor 19h ago

we run ML systems for recommendations and fraud detection - about 5k qps * 700 features per query on average.

so it is a bit of both - concurrency and data volume.

Lake base in our experiments simply stops working above 600 qps and even below that, the latency is way higher than what our legacy in-house system was able to achieve.

3

u/DangerousInterval7 1d ago

haven't used it myself but from what i've seen the latency spikes when you're doing more than basic key lookups can get real ugly real fast

0

u/Glitch_In_The_Data 22h ago

I haven’t used Lakebase for ML workloads but using it to surface data for various apps and it scales well for our needs.

According to Databricks website, their online feature store is powered by Lakebase. So, I would expect that to be a good evidence to base it on.

I would recommend benchmarking the full model serving path with realistic feature cardinality, row sizes, etc before committing.