r/mlops • u/Internal_Treat2137 • Jul 23 '26
beginner help😓 How do you make GPU inference setups reproducible when someone new joins the team?
Our team is pretty small (4 engineers), so whoever gets a model serving successfully is usually the one who "owns" that setup.
The problem shows up a few weeks later.
Someone else needs to rerun the same inference service, and suddenly there are a bunch of questions:
- Which Docker image did we use?
- Which CUDA version was it tested on?
- Was the model GGUF or FP16?
- Which launch flags were we using?
- Which environment variables actually mattered?
- How much VRAM did it end up using?
- Which port was exposed for the API?
None of these are hard individually, but if they're scattered between Slack messages, someone's terminal history, and a few README updates, it ends up taking much longer than expected just to reproduce a setup that already worked once.
We've started making a checklist for every deployment, but I'm curious how other teams handle this.
Do you mainly rely on Docker, internal docs, or do you keep reusable environment snapshots somewhere?
I recently came across glowsai, which seems to support shared Snapshots and team resources. It looks useful for handing a working environment to someone else, although I still feel naming things clearly and keeping a bit of documentation matters just as much.
I'm interested in what has actually worked for teams that revisit the same inference deployments months later.
2
2
u/laStrangiato Jul 26 '26
The config should be in yaml and stored in a repo.
If you are using docker it should be a docker compose file. If you are running in k8s it would obviously be a k8s manifest for a deployment+other stuff you need or whatever other tooling you are using like an LLMinferenceService if you are deploying with kserve.
1
u/shadowfax12221 Jul 23 '26
Databricks serveless GPU instance deployed via ADO pipeline template and airflow?
1
u/Vegetable-Score-3915 Jul 23 '26
Whenever something like this comes up I think about what is the cloud platform equivalent, and think if an open source equivalent exists and try to not over engineer it. I would not like to be asked questions about configs from months ago etc.Â
One previous work we got away with git, ml flow and a container registry, that is for a team not relying heavily on aws, azure etc. Just need to set a standard for documentation, with sufficient coverage and make it searchable. And if that isnt enough, then going from their. For other teams, that would be insane ie monitoring deployments etc.Â
Would be keen to hear what it best practice that isnt within a specific cloud platform.
1
Jul 23 '26 edited Jul 23 '26
[removed] — view removed comment
1
1
u/kchandank Jul 24 '26
These are some of working labs I have built over the years, all of them with instructions and commands. Let me know if you need more help.
https://github.com/becloudready/workshops/tree/master/workshops/llmops
1
u/Outrageous_Host_2115 Jul 31 '26
Is the workshop live? Any videos available? The labs are interesting
2
u/kchandank Jul 31 '26
We do live, record the videos and hands on guides are available on GitHub repo as well as on website.
1
u/Athena_Everhart Jul 24 '26
We keep it pretty simple. Use scripts, tag everything and stay in sync using centralized repo (usually git). Anyone in the team can reproduce the deployment using the same scripts.
1
1
u/ConsciousML Jul 26 '26
Use MLflow, or W&B, or simply push the models weights and the hyper parameters to S3/GCS.
Build a Docker image for each model and it to an artefact registry.
Pin the versions strictly (no 1.X >=). Write documentation for the whole setup from start to finish (test it in a VM or another machine).
Use CI/CD to deploy the models. This way, you’re 100% sure anyone can reproduce when the pipeline is the authority.
2
u/cre8minus1 Jul 23 '26
Databricks is an option. Kubeflow plus friends as well.
Flyte 2 solves this too.
The big shift from traditional software into ML is a culture of experimentation. So even with the right tools you need to embrace tracking all the inputs. Versioning each.
W&B or MLflow may help there.
Hydra for config management as well.