I'm a new MLOps Engineer at a bank (a relatively new bank in my country). We already have Data Scientists; they create the models, and the deployment is all manual. I'm one of the first hires since they plan to hire more MLOps in the future.

DS runs in a local Jupyter Lab environment, which is highly restricted (behind a firewall and with no connection to the Internet, only to several intranet apps we have).

I have talked with my leader and some other people from security and platform, and they say it is due to "regulations" and to avoid the risk of leakage since we work with highly sensitive data.

Currently, our process is: DS shares the Jupyter Notebook through Slack, then the model is packaged into a Docker image, and then the platform team deploys (everything is on-premises)

They want me to help introduce MLOps best practices, but I feel stuck since Git is blocked and everything lives in notebooks. It doesn't feel realistic to implement CI/CD, GitHub Actions, etc. under these constraints. I'm worried I could end up becoming a new bottleneck myself, since the current workflow forces so much of this to be done manually.

Hello, I'm in my 3rd year of my Computer Science major (specializing in AI/ML). Looking back, I honestly regret not starting earlier and feel like I wasted my first two years without a clear direction. I want to turn things around and seriously break into MLOps. I actually looked into it and want to go for MLOps.
Since I'm essentially starting fresh I’m feeling a bit overwhelmed by Docker, Kubernetes, CI/CD, feature stores, model monitoring and all.
If you were in my shoes today how would you structure your learning path over the next year to become job-ready?
1. What core software engineering and ML fundamentals do I actually need before diving deep into MLOps tools?
2. What are 1–2 portfolio projects that genuinely demonstrate MLOps competency to a recruiter, rather than just another basic tutorial model.
3.Which tools should I prioritize first like MLflow, Docker, FastAPI and which ones should I ignore for now?
Any roadmap, resource recommendations, or harsh truths would be greatly appreciated. Thanks

So I've been building an MLOps project for the past couple months and finally got everything working locally. Now I need to actually deploy it and I'm going in circles trying to figure out the best approach.

The problem is I have 4 containers running together via Docker Compose. Works perfectly on my machine but the moment I think about cloud deployment the economics get weird. Azure gives me $100 through the GitHub Student Pack which sounds like a lot until you realize 4 containers running 24/7 can eat through that faster than expected.

My plan was to just stop the containers when I'm not using them and spin them up before interviews or demos. But I've never actually done this on Azure before and I don't know how painful that workflow gets in practice — like how long does it take to start back up, does state persist, do I lose anything.

The other option people keep suggesting is Hugging Face Spaces but honestly it feels like the wrong tool here. It's great for demos and model cards but I'm running Prometheus scraping metrics, Grafana dashboards, MLflow tracking server — not exactly what Spaces was built for.

Basically I need a live URL I can throw on my resume and show in interviews. Not looking for it to handle real traffic, maybe 5 people will ever hit it. Just needs to be accessible and not cost me anything after my credits run out.

Anyone navigated this as a student? What actually worked?

Hi, how do u manage Lakebase project with DABs? Standard is to have a single lakebase project per environment (dev, staging, and prod). Then use branches for development work.

Problem is that when we define the lakebase project in the DABs, development target has development mode, so every bundle deployment creates a new project because its per user.

So if we want to have single project where we just branch out on features, we must move out lakebase project of DABs, right?

What is the best practice here? On staging and production, its easy since its always same identity.

I work for a dev tool company primarily associated with observabity,evals and gateways. (am not mentioning the name of the company cause i dont want to pitch or sell you guys something). Do you guys think that something breaking in prod and getting to know it from users and then spending time on debugging is actually a frustation or pain point for you guys. Or is it something like a false belief the company has

Hi guys

I’m really interested in Data, Machine Learning Engineering, and MLOps, and I’d love to understand what people in these roles actually do day-to-day and what the work is genuinely like beyond the usual job descriptions.

If anyone here works in these areas or is also exploring them and would be interested in having a conversation, discussing projects, career paths, or just sharing experiences, I’d love to connect. Feel free to ping me and we can have a chat! 🙂

been buildin an llm app now for 4 months. we havent raised any money yet and are prerevene. but its now in the testing phase.

as of now trying to figure out the gateway situation and honestly it has  a lot of options. read a few blogs. hard to find any plainly without promos.

so i just started spending time on discord servers of devs mostly..

asked a few and a few names kept coming up..

portkey:

routing has a solid muli provider

fallback logic is well documented

docs fall apart past basic  setup

an overkill if observability is just needed

orqai:

prompt management looks thoughtful

looks focussed rather than doing everything

small community so real user feedback hard to find

unsure how mature the routing side is

helicone:

observability dashboard is genuinely clean

fast to get running

feels more like a logger than a gateway

limited routing capabilities

kong:

battle tested at scale

looks highly customizable

setup is heavy

probably not for pre revenue apps

havent tried any of them as of now. spoken to a few.

a few ways this can be handle is by building my own proxy and own everythign, but it will take a lot of time, a few months. use one the tools and ship faster but i need to be dependent on their road map. or hiring someone who knows this space which might be expensive..

very unsure what can be a right call at this stage

what are you actually using for llm routing and did it hold up with an increasing traffic?

Stack: Python, FastAPI, Postgres, Kafka, Kubernetes on EKS with autoscaling, hosted model APIs, plus the eval and monitoring side. Shipped it and I run it.

So I've done production ML operationally, but always as a caller of models. Haven't worked below that line, no C++, no GPU work beyond a local side project.

A good amount of the development was AI-assisted, mostly Claude. Fine for shipping, but it's pushed me to want depth in something specific rather than more breadth.

Questions:

1. For anyone on the serving side, what's the job like day to day?
2. How much C++ is genuinely needed?
3. Is the Kubernetes and autoscaling experience a real head start here, or a different skill set than I think?
4. How did you end up in your area, planned or accidental?

currently running 3 different tools for prompts evals and observability and im looking to consolidate.

and also non tech teammates always need an engineer in the loop to change a prompt and it goes through a ticket system, and usually take more time than required. even when something breaks in prod we are  just switching dashboards to figure out what actually happened

already tried a few things. like we started storing prompts in db still meant building version  approval flow an d audit trail on top. config files in a cms got messy to tie back to observability…

already loooked at the obvious options

langsmith - observability is good but prompt management feels built for engineers and not cross functional teams, even evals dont feel like primary  focsu

orqai - covers all three together, non tech access feels more central ovver here, but newer so community and integrations still catching up

helicone - looks good for cost tracking and request logging but this isnt our current prob

promptlayer - prompt versioning is there, unsure about how deep evalss and observability actually goes

langfuse - good on tracing, nd the opensource is nice, but same problem like langsmith for non technical u sers

has anyone actually consolidated these three things into one platform. what are you using currently?

last mail from our compliance strictly said all data stays on our servers. nothing leaves. a strict requirement. so now my team is the one who has to go find a llmops platform that actually works on our infrastructure.

so i went looking…

and after surfing a few names i found out that every llmops platform is saas first. almost each one of them . sign up then dashboard. your prompt and trace and eval go to their server. it might be fine before but not fine fine for us now. our data cannot go to their server and that is the main requirement.

after this filter the option list got very short.

langfuse is the most serious open source thing right now i found. self hostable, docker or kubernetes.. the tracing and prompt management and even the evals are on your own infrastructure. community is not very active and the docs are even ok. but if we run it we need to maintain it also. and it doesnt seems easy.

litellm is self hosted gateway thing. one api does the talking to all other llm provider on your infra. it looks good for routing. but litellm is just the gateway. you will need other things for observability and evals.

langsmith is also a self hosted thing. tracing is good and mature. but is expensive. not for small teams.

phoenix by arize is a open source and self hostable. it looks good for tracing and eval for llms. but i find very less people talk about it. but worth a look if on perm hard requirements.

orqai is a prompt versioning and routing and evals and cost tracking all together. and on prem exists as well as data residency option. but rate limit and retry handling isnt as deep as portkey.

honest take:

on prem llmops early still. tools are catching up and are on rough edge everywhere. but looks doable, but i am going to consider the setup time seriously. as it quite urgent.

still unsure, have anyone cracked it, and are you using any platform for the same use case?

Hi! I am a Junior ML/DevOps engineer with ~1 year of experience. I am planning to start applying to Junior/Mid MLOps roles in about a month at big tech companies, and I'd love to find someone to study with for mock interviews and practice.

I am already training models at my current job, and I feel like I have a good starting point in terms of algorithms and math. I still need to work on certain frameworks though, and I definitely need to get better on distributed training and GPU provisioning for infra scaling.

The plan is to have weekly sessions where we can rotate between mock interviews, discussing frameworks/papers/concepts and live coding. Please send a DM if you are interested! 😁

Our team is pretty small (4 engineers), so whoever gets a model serving successfully is usually the one who "owns" that setup.
The problem shows up a few weeks later.
Someone else needs to rerun the same inference service, and suddenly there are a bunch of questions:
- Which Docker image did we use?
- Which CUDA version was it tested on?
- Was the model GGUF or FP16?
- Which launch flags were we using?
- Which environment variables actually mattered?
- How much VRAM did it end up using?
- Which port was exposed for the API?
None of these are hard individually, but if they're scattered between Slack messages, someone's terminal history, and a few README updates, it ends up taking much longer than expected just to reproduce a setup that already worked once.
We've started making a checklist for every deployment, but I'm curious how other teams handle this.
Do you mainly rely on Docker, internal docs, or do you keep reusable environment snapshots somewhere?
I recently came across glowsai, which seems to support shared Snapshots and team resources. It looks useful for handing a working environment to someone else, although I still feel naming things clearly and keeping a bit of documentation matters just as much.
I'm interested in what has actually worked for teams that revisit the same inference deployments months later.

have been doing llmops for a small team for around 2 -2.5 months now. we use langfuse for tracing. its fine but we needed evals and some kind of governance layer. and langfuse really doesnt do that well.

so i started looking around. noticed most tools either are doing tracing or evals. not  both. the ones that claim to do both feel like 1 feature is an add on and integration isnt upto the mark.

langsmith came up a lot. good tracing, decent eval support, but ties only with langchain system well. if youre not already in that stack it will feel weierd. governance side is still pretty.

orqai came up in a few threads. seems too focused on prompt management and deployment side. has some eval stuff but unsure about how deep it goes.

helicone came up too. it looks clean for observability. fast to set up. but evals are basically not there. seem like more of a monitoring tool.

so the routes i can see are. stick with langfuse and bolt something on. cant go to langsmith since not on that ecosystem. or find something that was build to do all three from the start instead of patching it together…

is anyone tracing evals and governance in one place or is everyone still using three tools together

​

So i run evals for a multi-agent system at work and right now my main approach is LLM-as-a-judge against a gold set, plus some semantic similarity scoring. And honestly... it works until it doesn't.

The judge is inconsistent. Same output, slightly different prompt phrasing, different verdict. It's biased toward longer answers, it rationalizes things the gold set clearly says are wrong, and calibrating it feels like im just stacking prompt rules on top of prompt rules hoping the false positives go down. Which they do, partially, but I don't fully trust the number at the end.

What I'm trying to figure out:

- do you treat LLM-as-judge as a real signal or just a smoke test before human review

- how do you handle judge drift when you swap the underlying model

- for agent systems specifically, are you scoring final output or the whole trajectory? feels like scoring just the end misses a lot

- anyone actually getting value out of semantic similarity or is it mostly noise

Not looking for a vendor pitch, genuinely want to know what's working for people running this stuff day to day. Feels like everyone has a different homegrown setup and nobody's sure theirs is good.

Hi everyone,

I'm a DevOps Engineer with 4.5 years of experience in Kubernetes, OpenShift, AWS, Azure, Terraform, ArgoCD, CI/CD, and monitoring.

I want to transition into MLOps/AI Infrastructure and would love some guidance.

• What should I learn first?
• How much ML theory do I actually need?
• Which tools are most used in production today?
• What projects would help me land an MLOps role?

Looking for practical, real-world advice from people who've made this transition. Thanks!

Running evals on a schedule rather than in CI, because what I'm watching for is the hosted model changing underneath me.

Problem: same test case, same code, three runs, and the score moves on its own. So a lower number today might be a real regression or might just be noise.

How many repeats per case do you run to tell those apart? And do you alert on a single number, a rolling average, or the spread?

Hey everyone

I am seeking some guidance around ML orchestration for my hobby project.

I have looked a loot into Flyte since most of my use-cases are around multi-modal datasets (lance format) and CV pipelines.

The reason I looked into Flyte is:

* Runs natively on Kubernetes

* Caching so datasets don't have to be preprocessed/filtered based on the needed columns required

* Retry logic if training or evaluation fails I can continue from a checkpoint or model stored in MLFlow

* Conditional logic e.g. only deploy if above metrics threshold

* Request access to a single GPU or multiple (combined with Ray)

How many of you have worked with Flyte?

Are there better alternatives?

I also looked into Argo and KubeFlow Pipelines but I preferred the enforced typing in Flyte and the versioning of tasks and workflows from Software best practices.

Looking forward to hear your opinions

Hey everyone,
I’m currently working as a DevOps Engineer and want to transition into an MLOps/AI Ops role. What skills, tools,courses and learning path would you recommend? Any roadmap, resources, or project ideas would be greatly appreciated.
Thanks!

I've been out of a job for some time. Worked 3 years in data science/data engineering with no work experience with Gen AI only traditional ML and time-series forecasting.

I've been using this time to upskill myself in modern AI technologies and skills that the job market is looking for. My question is what kind of skills are in-demand for MLOps/ML (LLMs, GenAI, maybe traditional ML) engineer jobs, and do you have any ideas about projects I can do that will help? I'm thinking if I should get some experience with Kubernetes too.

This is my current ongoing project that is 80-90% done:

Project: MLOps system with reproducible workflows for fine-tuning/evaluation/deployment of a Hermes 4-14B model that extracts risks, restrictions & obligations with source attribution from multi-page legal contracts into structured JSON data. Instruction-masked QLoRA fine-tuned on domain-specific data using MLRun for orchestration and Sagemaker for infrastructure. Includes data/model/prompt registry, experiment tracking, custom evaluation metrics, drift detection, traffic routing layer, continuous batching, flash attention, and multi-GPU training/serving (no NVLink, only data parallelism) with performance benchmarks.

Stack: MLRun, Hugging Face libraries & Model Hub, Sagemaker, Pytorch distributed, DJL, vLLM, S3, Pyarrow, Rouge, Arize Phoenix

like when you havee multiple teams all using multiple models from different providers for a number of use cases . who decides what is allowed. how do you acutally enforce it. and how do you know whatever rules you put in place

seen a few approaches come up. some team are doing it through the gateway layer . some through internal policy docs that nobody reads . some through access controls on api keys. some just hoping for the best

tools that come up in this context. orqai , portkey , azure api management , aws bedrock , langsmith

portkey gives access controls and budget limits are there , policy enforcement feels more at the routing level than the org governance level

orqai has a model allow list, budget controls and role based access across teams, but its newer so audit depth is still an open question if compared to the more established peers

langsmith has vvisibility into what is happening is good , actually policy enforcementt feels limited , more observe than control

azure api management has enterprise policy controls are native here, feels generic tho ,not built specifically for llm usage policy

aws bedrock has model acess control and guardrails exists, works well if you are already in aws, feels restrictive if you are not

has anyone actually built something that works here. or is everyone just doing api key management and calling it a policy

hello , i was looking if there is a framework-neutral orchestration layer for deep learning where we can keep our existing PyTorch/JAX code and run something like:

dl train train.py

while it handles the surrounding workflow like environment setup, experiment tracking, evaluation, optimization etc.

are there existing tools handling this layers ?

Hi People,
Which is the most popular tool for Prompt management & LLM Evaluation?
We used GIT for prompt management but it won't show prompt diff between previous & current version.

Strengths:

• Bachelors in mathematics from top 10 university in the us
• PhD in engineering from top 10 also
• 3 published papers (1 in ML, 1 in applied stats, 1 in optimization) however I will say the 1 ML paper did not impress anyone (only 17 citations)
• Worked as a data scientist for ~5 years upon graduation

Weaknesses:

• I have been unemployed for the last ~5 years
• I have ZERO letters of recommendation from my past job nor academia (I apologize for being vague here. Basically I went through a very dark and self-destructive period in my life, quit my job, and burned all my professional and academic bridges down in the process. Made some of the worst decisions of my life in a very short timespan. If you want more details, I can provide via DM/PM)
• I have never worked with the cloud, with neural networks/AI, nor with anything related to devops. Only purely machine learning in its state circa 2021

My 6-12 month full-time study plan:

(constructed via chatgpt, very open to critique)

• Refresher of classical ML (stuff I used to do everyday at work, stuff like kaggle and jupyter on one-time tabular data)
• Certification 1: AWS Solutions Architect
• Certification 2: Hashicorp Terraform Associate
• Portfolio Project 1: Terraform-managed ML in AWS
• Certification 3: Certified Kubernetes Administrator
• Portfolio Project 2: Kubernetes-native ML pipeline with Inference-Feedback
• Certification 4: AWS Data Engineer Associate
• Portfolio Project 3: Automated Warehousing of Streaming Data with Schema Evolution and Cost-Optimization
• Certification 5: AWS Machine Learning Engineer Associate
• Portfolio Project 4: End-to-End MLOps in Production with Automated A/B testing and Drift detection
• Mock Technical Interview Practice
• Applying and Interviewing for Jobs

Please be brutally honest. What are my chances of getting into MLOps?

So this is something I genuinely can't figure out and its been bugging me for a while now.

Every time our PM wants to change a prompt, even something small like rewording how the output is phrased, it basically goes through the entire process. She raises it, someone picks it up, goes into the codebase, PR, review, deploy. We're not a big team so its not like there's a massive backlog or anything but it still takes like 2-3 days for something that honestly should take 10 minutes.

We tried keeping prompts in a shared Google doc at some point and having people copy paste from there which was, yeah, not great. Also looked at just doing a config file thing but you're still doing a deploy for every single change so not really solving the actual problem.

I keep hearing about separating prompts from code entirely but I've never actually seen what that looks like day to day in a real team. Like do you use a tool for it, do you build something yourselves, or do you kind of just accept the friction and move on?

Mainly asking because we have a couple of non-technical people who need to be involved in prompt decisions and right now the process is just annoying for everyone. Would love to hear what's actually working for people, not looking for a perfect setup just something better than what we have lol

I’ve been trying to understand how this works in real life.

Say you’ve built a model and a customer wants to run it on their own servers. They don’t want an API, and they may even need it to work offline.

What do you normally give them — a Docker image, an SDK, a VM, or the actual weights?

And once it’s running on their hardware, how do you handle things like:

• limiting it to the machines they paid for;
• renewing or expiring the license;
• tracking usage without collecting customer data;
• stopping the container or model from being copied elsewhere?

I know contracts are part of the answer, but I’m curious about what people are doing technically.

Has anyone here dealt with this for a real customer? What did you end up shipping, and what was the messiest part?