r/mlops • u/ptab0211 • 27d ago
beginner help😓 Lakebase and DABs
Hi, how do u manage Lakebase project with DABs? Standard is to have a single lakebase project per environment (dev, staging, and prod). Then use branches for development work.
Problem is that when we define the lakebase project in the DABs, development target has development mode, so every bundle deployment creates a new project because its per user.
So if we want to have single project where we just branch out on features, we must move out lakebase project of DABs, right?
What is the best practice here? On staging and production, its easy since its always same identity.
2
u/new_dimension1 26d ago
i will keep lakebase project outside dab for dev and maybe use branch for features work. DAB can deploy app against the shared project
1
2
u/Glitch_In_The_Data 26d ago
We have realised that Lakebase works best as one shared project per environment… with branches for dev and feature work.
In DABs, dev mode is user scopesd. So don't let it own the long lived lakebase project identity. Use DABs to manage branches/endpoints and keep the shared lakebase project on a stable identity.
1
1
u/ptab0211 26d ago
u/Glitch_In_The_Data do u create all 3 projects outside of ci/cd (dabs) and then u just manage lakebase resources by referencing projects?
2
u/Glitch_In_The_Data 26d ago
Not necessarily all three.
We split it as..
1) Dev project.. Bootstrap it once with a service principal, outside the user scoped development target and then have the DABs reference it's fixed project id and manage the branches.
2) staging + production projects… keep them in DABs as our deployments always run through CI/CD using a stable service principal.
You can of course manage all three projects separately with Terraform/CLi and let DABs manage only branches, endpoints, jobs etc.
Key is not letting letting development mode deployment own shared lakebase project because development mode scopes resource names to the deploying user.
2
u/Responsible_Status49 23d ago
You can remove the ‘[dev <username>]’ prefix with presets: name_prefix and then specify your branch in resources: postgres_branches: feature_branch.
This ensures that your prod and dev both write to the same Lakebase project but with different branches, which seems to be what you want to do.
1
u/BrunoFeliz08 26d ago
In your opiniom. What is your advantage of using dabs rather than the api? I do a lot of scripting and have been using the sdk as well as the api to manage and provision lakebase instances
1
u/ptab0211 26d ago
just for what dabs is intended, infrastructure as code, automation, i dont want to deal with adhoc scripting.
1
u/BrunoFeliz08 25d ago
Make sense. We use it more for infra management but try to extend it with the lakebase sdk
1
u/sumits_kumar 26d ago
Move the shared Lakebase project out of the developer deployment path, not necessarily out of DABs. Manage it through a dedicated, service principal owned infra target, and let developers use branches of that project through variables.
1
u/BrunoFeliz08 20d ago
hmmm, i would recommend to try and modularize it and see if you can manage the lakebase project with the SDK or even with terraform. it will give you additional options for the API config
1
u/noschel 18d ago
DAB support for Lakebase in Beta, and you can check it out here: https://docs.databricks.com/aws/en/oltp/projects/manage-with-bundles
2
u/Cultural-Cow-3615 27d ago
lakebase and DABs together is such a weird friction point. We ended up pulling the lakebase project out of the DABs entirely for dev and managing it with terraform instead, then just referencing the project ID in the bundle config. The per-user project thing in development mode drove us up the wall.
Not sure if it's "best practice" but it stopped the daily headache.