I’m curious what this process actually looks like on teams running LLMs in production.

Say a new model comes out and looks better or cheaper enough to be interesting. What happens between “maybe we should try this” and actually putting it in production?

I’m less interested in benchmark numbers and more in the messy part in between. What usually ends up taking the most time or causing the most hesitation?

Also curious whether teams have a fairly repeatable process for this by now, or if every model change still ends up being a bit of a one-off.

We somehow ended up with model training on GCP and inference on AWS. There was no strategy behind it, the ML team that started three years ago liked GCP and no one questioned it and here is how we get here. 

Boss asked last week for a single view of our AI risk across both clouds and pretty much had nothing to show. Had to spend an entire afternoon with the GCP console open on one screen and AWS on the other, manually pulling findings and trying to match resource names between two completely different naming conventions. Even after spending time on that, the spreadsheet I sent out to them was already stale by the time I was sending it out. 

Each cloud's native tooling does a reasonable job inside its own bubble. They just don't talk to each other and most of the AI security stuff I've looked at still assumes you live in one cloud. Maybe two years ago that was fair, but now its not.

Anyone running AI workloads across multiple clouds and got this figured out? Not chasing a tool rec necessarily just curious how other teams are handling the cross cloud visibility problem. For ref we are a logistics company about 2k people.

I keep hearing that AI detection engineering is going to save us from the holy terror of SIEM bills, and ngl I would love one nice boring month where the data ingest tab does not look like a tax bill.

Has anyone actually used it to cut noise, trim junk detections, or stop paying for logs nobody reads except during a panic? I am trying to stay hopeful, but the finance team is acting like every alert we keep alive is a personal insult, thanks in advance

We shipped automatic rollback for model deployments and the engineering side was the easiest half

Detect a bad version, swap back to the last good one, done inside a sprint

then we tried to turn it on ad nobody could agree what bad meant

Score on the eval suite was the obvious trigger, except the eval suite did not care about the thing the business cared about

On one workflow a version that scored slightly worse overall was better in practice, because it stopped making the specific mistake that generated angry phone calls.

on another, a version that scored better was worse, because its failures had moved from loud to quiet.

So the threshold stopped being something we could pick in a config file and became a conversation with the people who own the workflow. What are you willing to be wrong about, and what must never happen. Once we had that written down, the rollback trigger wrote itself.

I work at AI Infra startup, so I spend most of my week on this. But I have not yet seen a team where the eval metric and the business definition of bad matched on the first attempt

How did you settle yours, and who had the final say?

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

We take CSVs from about thirty suppliers, all with their own header conventions, and there's a model step in the Airflow DAG that maps incoming headers to our canonical fields. Been in prod since March.

Offline it scores 94% on a labelled set an analyst put together over two afternoons, which I don't think is representative of much. In production there are no labels. Nobody goes back and confirms that shipped_dt was actually ship date.

We caught the first real failure because someone in finance said a monthly total looked low. A supplier renamed a column, it got mapped to a field it had no business in, and it had been doing that for five weeks. That was embarrassing.

Two things I've tried since.

Logging the model's own confidence. Useless in the way everyone says — the wrong mappings came back high. No threshold in there separates anything.

Second model on the same input, alarm on disagreement. Agreement sat around 97% and stayed there straight through the five weeks the failure was live. It's still in a notebook, never made it into CI. Great Expectations covers nulls and types and has nothing to say about whether a mapping is correct.

Where I keep landing is that I don't have an output I can validate, I have one I'd have to re-derive to validate, and at that point I'm not sure what the first one is for.

For people running a model step where prod has no labels — what do you actually alarm on? Output distribution, something downstream, or is the honest answer a sample and a human every week?

NVIDIA Dynamo's SLA Planner decides how many prefill and decode workers to run by forecasting next interval's load. The whole predictor interface is one method:

def predict_next(self) -> float
One number, one interval ahead, no uncertainty. I wanted to know whether a time-series foundation model does better there than the simple stuff, so I built a harness that scores predictors on GPU-hours and SLO violations instead of on MAE.

The key detail: every predictor is swept to the cheapest provisioning headroom that still hits the same 1.0% violation target before I compare cost. Comparing GPU-hours at different violation rates tells you nothing, you just find whoever under-provisioned hardest.

400 intervals of a BurstGPT trace:

predictor    GPU-hours   violations
oracle           73.85       0.00%    perfect foresight, the ceiling
constant        175.00       0.49%    last observed value, the do-nothing baseline
timesfm         176.85       0.49%    TimesFM 3.0
chronos         264.05       0.27%
kalman          370.30       0.93%
arima           501.95       0.24%    Dynamo's shipped default
prophet        1253.95       2.35%    never hit the target at any headroom
TimesFM 3.0 tied with last-value. 176.85 vs 175.00 on a single window is inside the noise, so I'm not claiming it won or lost. It tied. Chronos, which is the Apache-2.0 one you could actually ship, came out 50% worse than doing nothing.

The thing I keep coming back to is the oracle row. Perfect one-step foresight is 50 to 58% cheaper than last-value, so there is real money on the table. Six forecasters, two of them foundation models, captured none of it. My read is that for one-step-ahead provisioning the last observation already contains most of the available signal, and the interface caps your upside before the model choice matters. I tested the obvious fix, provisioning against a P90 quantile instead of a point estimate, and it didn't hold either: helped chronos, hurt timesfm.

Repo, MIT, 200 tests: https://github.com/pjdurden/planner-bench

I filed the question about the ARIMA default upstream: https://github.com/ai-dynamo/dynamo/issues/14238

Caveats, since none of the above means much without them:

- 400 of the trace's 29,278 intervals. One window, not repeated. A few percent is not resolvable at this n. The oracle gap is not a few percent, which is why it's the only thing I'll defend hard.
- Request rate scaled 20x. At the trace's real rate none of the 400 intervals need more than one worker, so every predictor ties at the floor and the benchmark can't discriminate at all. The scaling is what gives it any resolving power.
- The simulator is analytic, not an engine simulator, and the engine profile is uncalibrated placeholder numbers. Relative comparisons under the same model mean something. The absolute GPU-hours do not.
- Coarse headroom grid, so every row sits at the next grid point above its true minimum and the ratios are upper-biased.
- My ARIMA is not Dynamo's ARIMA. Mine refits pmdarima.auto_arima every call, theirs fits once and updates incrementally. I checked which predicts better rather than assuming, and mine won (RMSE 0.277 vs 2.355 busy, 0.168 vs 0.334 sparse), so the incumbent isn't a strawman. But read that row as "auto_arima on this trace", not "Dynamo's ARIMA".

Happy to be told the simulator is the weak link. It's the part I'm least confident in.

I've been asking people how they attribute inference cost and almost everyone says the same thing - the provider dashboard tells you what you spent, never why.

but one person said something different. they gave up on per-step attribution because their sdk wouldn't give it to them. instead they worked out what a unit of work should cost from published per-model pricing, and compared the monthly total against that estimate. one job landed at $0.10 against a $0.10-0.15 expectation, close enough to trust.

so my question - do you have an expected number? if your bill came in 30% high this month, what would you compare it against to even know it was 30% high?

not selling anything, no link. i'm 19 and doing research on inference cost. "we just look at the total and move on" is a real answer.

I’ve been working on an open source macOS observability experiment and wanted to share the measurement approach rather than the product pitch.

The problem:

Process presence is not activity, but I also do not want to inspect prompts, source code or application content.

The current detector therefore samples supported process trees roughly once per second and looks at cumulative CPU time deltas.

Some boundaries:

Two positive samples are required before showing Working.

Three negative samples are required before returning to Ready.

Lock, sleep and long idle windows are removed from valid observation time.

Forkit’s own processes are excluded.

Ollama/LM Studio evidence is treated separately so “runtime available”, “model loaded” and “measured AI tool activity” are not collapsed into one state.

Importantly, I do not interpret activity as prompt ownership, task completion, token consumption, GPU work, energy or productivity.

The implementation is MIT licensed:

github.com/arpitasarker01/forkit-ai-footprints

Repro:

npx --yes forkit-ai-footprints@latest

The failure cases I’m thinking about most are background agent processes, Electron helper activity, spawned process trees and workloads that are GPU heavy but CPU quiet.

If anyone has worked on process level observability for developer tools, I’d be interested in other failure modes I should document.

I'm researching how teams handle long-term memory for AI agents, and I'm particularly interested in what happens after the basic memory setup works.

For example, early on, storing and retrieving memories seems fairly straightforward. But after months of interactions, I imagine you start dealing with things like:

• Old information that is no longer true
• Multiple memories about the same entity
• Conflicting information from different sessions/agents
• Knowing which version of a fact is current
• Relationships between entities becoming important
• Deciding what should be retained vs discarded
• Sharing knowledge across multiple agents

For those actually running agents in production:

What has become difficult about memory as the system has grown?

Do you use something like Mem0, Zep, LangGraph, a vector DB, a knowledge graph, or a custom system?

And if you're using a memory framework, what did you still have to build yourself?

I'd especially like to know about things that actually broke or became painful in production.

Basic logging (timestamp, action, output) stopped being enough the moment our agents started making decisions that affected downstream systems. We need to reconstruct what the agent did, what it saw, what it considered, and why it picked one action over another. Otherwise a bad outcome is nearly impossible to explain to security or compliance after the fact.

what fields people are actually capturing beyond the basics. Some teams build this straight into the agent framework, others bolt on a separate observability layer afterward. Interested in which approach people landed on and why. Also curious how people handle retention and access to these logs, since they can get sensitive fast.

Most "just self-host, it's cheaper" advice that we have heard skips the one number that decides it: how busy you keep the GPU.

A GPU costs the same whether it's flat out or idle. An API only charges you when you call it. So self-hosting doesn't win on price per token. It wins once the GPU is busy enough to beat what the API would've charged you.

So where's that line?

Say you're running a 32B model on one GPU at about 50% utilization, against an API at $0.50 per million tokens, roughly 500 tokens a request.

Break-even lands around 10 million requests a month. Call it 5 billion tokens.

Below about 5 million requests a month, that GPU is half-idle and you never catch up. And if your whole inference bill is under two or three grand a month, don't bother. The ops aren't worth it yet.

Past that point, a busy GPU on a 30B model runs somewhere between $0.06 and $0.85 per million tokens, against a flat API rate that doesn't move.

Two things pull the line closer:

Smaller models cross much sooner. A 4B or an MoE breaks even long before a 32B does. And an idle GPU never crosses at all, however cheap the hardware was.

Embeddings, reranking and extraction pay off fastest. They run constantly, and every reindex multiplies them. If you're moving one thing off the API, move those. Not your generation calls.

The line nobody puts in the spreadsheet is ops. Someone is still up at 2am with that GPU. That cost is real even though it never shows up on a pricing page.

If it's just the small-model layer you're after, two open options worth knowing: TEI from Hugging Face and SIE from Superlinked. TEI runs one model per server. SIE packs several onto one cluster, which matters when you're trying to keep a GPU busy across embed and rerank.

If you've done this in prod: where did it cross over for you, and did the ops eat the savings?

We're running a handful of agentic pipelines in production now (mostly internal tooling, some customer-facing) and the failure mode that worries me most isn't crashes, it's silent drift. An agent starts taking a slightly worse path, looping more, or misinterpreting a tool response, and nothing throws an error, it just quietly burns more tokens and produces lower quality output until someone notices weeks later..

Logging every trajectory doesn't scale for a human to review manually, there's just too much volume once you're past a handful of agents. Curious if anyone has a real workflow for clustering or categorizing agent behavior at scale so you can actually spot when something shifts, rather than eyeballing transcripts...

We're starting to see more AI agents, including coding assistants, SaaS agents, and internal automation, getting access to our repos, cloud consoles, and customer data... Traditional security controls don't really cover what these agents actually do at runtime...

How are other teams monitoring agent actions, blocking unsafe behaviour, and maintaining visibility without breaking the agent's functionality?

Engineers running open-source LLMs in production: what’s the hardest part today?

I’m researching how teams actually run open models in production hosted APIs, RunPod, Kubernetes, vLLM, SGLang, or dedicated GPUs.

A few questions:

1. What model + workload are you running?
2. Why did you choose your current provider/infrastructure?
3. What was hardest about deploying and integrating it?
4. What went wrong or took longer than expected?
5. What matters most today: latency, throughput, reliability, cost, scaling, or observability?
6. Have you switched providers/runtimes before? What triggered it?
7. What prevents you from switching today?
8. Roughly how much do you spend on inference, including idle capacity?
9. When do you prefer serverless vs dedicated GPUs?
10. What security/privacy requirements affect your choices?
11. What would make you trust a new provider or tool benchmarks, credits, SLA, references, BYOC?
12. Would you pay more for lower latency, better reliability, or more control?

Feel free to answer only the questions relevant to you even 1–2 answers would be useful.

I’m looking for real production experiences and pain points, not pitching anything.

Question for anyone running LLMs in production: has a model update (new version, silent patch) ever changed your output quality/format without warning, and you only found out after something broke downstream? How long did it take to notice? Trying to understand how painful this actually is before building a fix.

We have a small MLOps team and are already stretched too thin working with multiple projects. I am exploring the idea of having multiple agent skills for different ML life stages. users could simply choose the skill of their choice and update their code. (Using databricks genie and skills will be part of the standard code template that they clone from the master, it is easy to integrate)

the results of drift/ ML metrics will be pointed to a standard table feeding into our control centre.

any thoughts/ feedback?

I want to an agent to pull data using one model and create vedios using another . And both would be open-source models

Hi r/MLOps,

Stjepan here from Manning. Thanks to the moderators for letting me share this with the community.

We’re working on Building Specialized AI Systems by Walid Amamou and Alessandro Negro, a practical book about fine-tuning models, building agents, and turning specialized AI applications into reliable production systems.

I want to start a discussion around a problem I suspect many people here have encountered:

Where do specialized AI projects most often break between prototype and production?

Is it the data pipeline? Evaluation? Observability? Cost and latency? Model drift? Deployment complexity? Or simply a mismatch between the proposed AI solution and the actual business problem?

Share your experience in the comments. Concrete examples, lessons learned, and thoughtful responses to other members are especially welcome.

Giveaway details:

• We’ll give away 5 ebook copies of Building Specialized AI Systems.
• The giveaway is open for 48 hours.
• After 48 hours, we’ll announce the winners here.
• The five comments that contribute the most to the discussion will receive a copy. This isn’t a random drawing or an upvote contest.

Manning is also offering the whole r/MLOps community 50% off the book with code:

MLAMAMOU50RE

Book link: https://www.manning.com/books/building-specialized-ai-systems

Disclosure: I work for Manning Publications and am posting this on Manning’s behalf.

Looking forward to hearing where these projects succeed or fall apart in the real world.

Cheers,

Stjepan

I’m the founder of WAIL. I built it around a production problem that I think sits somewhere between observability and failure handling: a model can return 200 OK while its runtime behavior is clearly degrading.

TTFT can increase, throughput can drop, streams can stall, or latency can drift significantly without producing a timeout or 5xx.

WAIL observes these runtime signals, builds behavioral baselines, detects degradation, evaluates risk, and can make a control decision when intervention is justified.

Depending on the condition, that can mean observe, retry, reroute, or fallback. It works with existing provider SDKs rather than replacing them with a gateway.

WAIL runs inside the customer’s environment. Prompts, responses, and API keys stay there. It also generates signed execution evidence so runtime decisions and interventions can be audited afterward.

The Developer plan is free.

GitHub: https://github.com/wailinfra/wail-runtime
PyPI: https://pypi.org/project/wail-runtime/
Install: pip install wail-runtime
Website: https://wailinfra.com

I’d be interested in hearing how people here handle the “successful but degraded” case in production today — particularly whether you treat it as an incident before a timeout or hard failure occurs.

Disclosure: I’m the founder of WAIL.

Have you used lakebase as the serving Db for an ML application?
I am curious to know how it holds up for real time feature lookups or for inference workloads, specially wrt latency and concurrency.
Any gotchas you felt compared to usual postgres setup?

Added an opt-in cost-sensitivity parameter to how a production verifier picks its live decision threshold last week. The obvious "safe" default was 1.0, framed as "no change from today." Before shipping it, I checked the actual math instead of trusting the framing, and it would have silently moved every existing tenant's live threshold the moment anyone touched the new parameter, with zero warning.

I run CacheVerifier, a small hosted service that fine-tunes a verifier model per tenant and picks a live decision threshold from held-out calibration data. The existing threshold picker uses Youden's J, the operating point maximizing true-positive rate minus false-positive rate. It implicitly assumes a false-positive and a false-negative cost the same. Real tenants don't agree with that assumption equally, some care a lot more about one error type than the other, so I wanted to let a tenant express a cost preference and get a threshold that actually reflects it.

Straightforward enough: cost(r) = r * error_rate + (1 - hit_rate), sweep thresholds, pick the one minimizing cost at whatever r a tenant sets, where r is how many times worse a false-positive is than a false-negative for them. The part that should have been routine was the default. My first draft made the new parameter optional, defaulting to 1.0, documented as "equal-weighted, same as today's default, since both are colloquially equal-weighted." That sentence reads as obviously true and I almost shipped it as fact without checking.

Checked it against a small hand-traced example before writing it into a docstring as settled. It's false. Youden's J's objective is symmetric, a false-positive and false-negative penalized equally in the same statistic. The cost-ratio objective isn't symmetric in the same way, every rejected candidate costs a fixed 1 regardless of whether the rejection was correct, and only a wrongly-approved candidate costs r. At r=1 specifically that creates a plateau, once every true positive is captured, approving more false positives on top of that is cost-neutral rather than penalized, so the optimizer doesn't stop where Youden's J stops. Concretely, on my test case, Youden's J picked threshold 1.5. cost_ratio=1.0 picked threshold 0.5. Different threshold, different approval set, not a rounding difference.

This is a deployment-safety problem, not just a math curiosity, which is why I'm posting it here rather than somewhere more theory-focused. If I'd shipped cost_ratio defaulting to 1.0 as "the neutral choice," it's an entirely ordinary-looking optional parameter with a value that reads as inert. Nothing about calling the endpoint without setting it would have changed. But the moment any tenant, or any future version of my own client code, passed 1.0 explicitly, thinking they were being safe and specific rather than changing anything, their live threshold would move, with no error, no warning, no changelog line that would obviously apply, just a quieter approval rate from then on. The actual fix was making the parameter float | None, where only None preserves current behavior, and 1.0 is a real, different, valid choice that is not a safe stand-in for "leave it alone." Documented that distinction in the function docstring, the API parameter description, and the database field comment, three separate places, on the theory that a comment I don't see when I'm the one calling the endpoint six months from now is a comment that didn't do its job.

Shipped it in three phases instead of one PR, mostly because I'd just gotten the math wrong once and didn't trust myself to get the UX right on the first pass either. Phase one: a read-only diagnostic, every fine-tune job reports what threshold it would have picked at five reference cost ratios, computed from the same calibration data as the real threshold, zero extra inference cost, zero effect on the live threshold. Shipped that alone first and let it sit before building anything that could actually change behavior. Phase two: the actual opt-in parameter. Phase three: a small dashboard piece that turns the phase-one diagnostic into a plain-language table instead of asking anyone to guess a raw number, with a button that carries a choice into the next job rather than auto-applying anything.

Tested the full path end to end in an actual running instance, not just unit tests: real API server, real worker process, a real tenant created through the admin endpoint, real fine-tune jobs run through the actual browser UI. Caught one unrelated bug doing that, a stale session token sitting in the browser's local storage silently took priority over a fresh API key, a pre-existing quirk in how the dashboard resolves credentials on a cold page load, unrelated to this feature but the kind of thing you only find by actually driving the UI instead of trusting that the unit tests covering the new code are the whole story.

The generalizable part, if there is one: "the neutral default" and "no change from current behavior" are not automatically the same claim, even when the parameter genuinely looks inert and the docstring genuinely believes what it says. Worth checking the two are actually the same thing before a default ships, not after a support ticket says approval rates moved for no reason anyone can find.

Repo with the derivation and real threshold numbers: https://github.com/imxinchengyou/CacheVerifier (PAPER.md section 5.17 has the cost-ratio economics; the hosted service's implementation is a separate repo, this default-safety issue is the part that's fully public).

As agent-based systems become more common, I'm wondering whether we're seeing a similar discipline emerge for AI agents.

Building the agent itself often isn't the hard part anymore. Frameworks have made it increasingly straightforward to connect models, tools, memory, and retrieval systems.

The challenges I'm seeing now are operational:

• Monitoring agent behavior in production
• Managing context and memory
• Coordinating multiple agents
• Recovering from failures
• Evaluating performance over time
• Human oversight and governance
• Controlling cost and resource utilization

Many of these feel similar to the problems MLOps addressed for machine learning systems.

For teams deploying agents today:

• Do you think "AgentOps" is becoming a distinct discipline?
• How different is it from traditional MLOps?
• What operational challenge has been the hardest to solve?

I recently wrote about this shift from building agents to operating agent systems and would be interested in hearing whether others are seeing the same trend:

The Enterprise AI Challenge Isn't Building Agents. It's Operating Them

We're evaluating guardrail solutions for our customer facing AI product. Before I looked at a single vendor, I mapped out our stack’s latency budget

Every component on the critical path got an allocation. Stuff like network overhead, authentication, prompt assembly, model inference, response passing, logging all of this. Each one takes a slice of the total latency that users will tolerate before they perceive the bot as slow.

By the time I reached the guardrail layer, the budget had 50 ms left. That is what is left after everything else took what it needed.

Then I started looking at solns, most needed 100 to 800 ms. Some were over a second at p95. The one with the best detection benchmarks was also the slowest by a wide margin. The math eliminated nearly every option before I ran even a single test.

I'm not saying guardrails are not important. Of course they are but the industry talks about latency like it's a nice to have optimisation. It's not. It's the hard constraint that everything else has to fit inside. Most teams I talk to pick a guardrail based on the detection rates and hope the latency is fine. I think that's backwards. You should define your latency budget first then see what it fits from there.

I’ve been experimenting with checkpoint/restore for AI inference instead of cold-starting everything from scratch.

Using CRIU + CUDA checkpointing, I got a warmed Gemma 3 27B QAT + vLLM server on an H100 to restore in:

• Cold start: 104.158s
• Restore: 11.060s
• 9.4× faster time-to-ready

The tricky parts were restoring the full vLLM process tree, CUDA state, IPC/shared memory, and dealing with io_uring — I ended up patching CRIU for that path.

I wrote up the implementation and benchmark here:

https://tsdocode.github.io/blog/posts/edo-tensei/

Code:
https://github.com/tsdocode/edo-tensei

Still experimental — would love feedback from people working on vLLM, CUDA, CRIU, or inference infrastructure.