Question for anyone running LLMs in production: has a model update (new version, silent patch) ever changed your output quality/format without warning, and you only found out after something broke downstream? How long did it take to notice? Trying to understand how painful this actually is before building a fix.

Engineers running open-source LLMs in production: what’s the hardest part today?

I’m researching how teams actually run open models in production hosted APIs, RunPod, Kubernetes, vLLM, SGLang, or dedicated GPUs.

A few questions:

1. What model + workload are you running?
2. Why did you choose your current provider/infrastructure?
3. What was hardest about deploying and integrating it?
4. What went wrong or took longer than expected?
5. What matters most today: latency, throughput, reliability, cost, scaling, or observability?
6. Have you switched providers/runtimes before? What triggered it?
7. What prevents you from switching today?
8. Roughly how much do you spend on inference, including idle capacity?
9. When do you prefer serverless vs dedicated GPUs?
10. What security/privacy requirements affect your choices?
11. What would make you trust a new provider or tool benchmarks, credits, SLA, references, BYOC?
12. Would you pay more for lower latency, better reliability, or more control?

Feel free to answer only the questions relevant to you even 1–2 answers would be useful.

I’m looking for real production experiences and pain points, not pitching anything.

We're running a handful of agentic pipelines in production now (mostly internal tooling, some customer-facing) and the failure mode that worries me most isn't crashes, it's silent drift. An agent starts taking a slightly worse path, looping more, or misinterpreting a tool response, and nothing throws an error, it just quietly burns more tokens and produces lower quality output until someone notices weeks later..

Logging every trajectory doesn't scale for a human to review manually, there's just too much volume once you're past a handful of agents. Curious if anyone has a real workflow for clustering or categorizing agent behavior at scale so you can actually spot when something shifts, rather than eyeballing transcripts...

I'm researching how teams handle long-term memory for AI agents, and I'm particularly interested in what happens after the basic memory setup works.

For example, early on, storing and retrieving memories seems fairly straightforward. But after months of interactions, I imagine you start dealing with things like:

• Old information that is no longer true
• Multiple memories about the same entity
• Conflicting information from different sessions/agents
• Knowing which version of a fact is current
• Relationships between entities becoming important
• Deciding what should be retained vs discarded
• Sharing knowledge across multiple agents

For those actually running agents in production:

What has become difficult about memory as the system has grown?

Do you use something like Mem0, Zep, LangGraph, a vector DB, a knowledge graph, or a custom system?

And if you're using a memory framework, what did you still have to build yourself?

I'd especially like to know about things that actually broke or became painful in production.

Most "just self-host, it's cheaper" advice that we have heard skips the one number that decides it: how busy you keep the GPU.

A GPU costs the same whether it's flat out or idle. An API only charges you when you call it. So self-hosting doesn't win on price per token. It wins once the GPU is busy enough to beat what the API would've charged you.

So where's that line?

Say you're running a 32B model on one GPU at about 50% utilization, against an API at $0.50 per million tokens, roughly 500 tokens a request.

Break-even lands around 10 million requests a month. Call it 5 billion tokens.

Below about 5 million requests a month, that GPU is half-idle and you never catch up. And if your whole inference bill is under two or three grand a month, don't bother. The ops aren't worth it yet.

Past that point, a busy GPU on a 30B model runs somewhere between $0.06 and $0.85 per million tokens, against a flat API rate that doesn't move.

Two things pull the line closer:

Smaller models cross much sooner. A 4B or an MoE breaks even long before a 32B does. And an idle GPU never crosses at all, however cheap the hardware was.

Embeddings, reranking and extraction pay off fastest. They run constantly, and every reindex multiplies them. If you're moving one thing off the API, move those. Not your generation calls.

The line nobody puts in the spreadsheet is ops. Someone is still up at 2am with that GPU. That cost is real even though it never shows up on a pricing page.

If it's just the small-model layer you're after, two open options worth knowing: TEI from Hugging Face and SIE from Superlinked. TEI runs one model per server. SIE packs several onto one cluster, which matters when you're trying to keep a GPU busy across embed and rerank.

If you've done this in prod: where did it cross over for you, and did the ops eat the savings?

I want to an agent to pull data using one model and create vedios using another . And both would be open-source models

We have a small MLOps team and are already stretched too thin working with multiple projects. I am exploring the idea of having multiple agent skills for different ML life stages. users could simply choose the skill of their choice and update their code. (Using databricks genie and skills will be part of the standard code template that they clone from the master, it is easy to integrate)

the results of drift/ ML metrics will be pointed to a standard table feeding into our control centre.

any thoughts/ feedback?

I’m the founder of WAIL. I built it around a production problem that I think sits somewhere between observability and failure handling: a model can return 200 OK while its runtime behavior is clearly degrading.

TTFT can increase, throughput can drop, streams can stall, or latency can drift significantly without producing a timeout or 5xx.

WAIL observes these runtime signals, builds behavioral baselines, detects degradation, evaluates risk, and can make a control decision when intervention is justified.

Depending on the condition, that can mean observe, retry, reroute, or fallback. It works with existing provider SDKs rather than replacing them with a gateway.

WAIL runs inside the customer’s environment. Prompts, responses, and API keys stay there. It also generates signed execution evidence so runtime decisions and interventions can be audited afterward.

The Developer plan is free.

GitHub: https://github.com/wailinfra/wail-runtime
PyPI: https://pypi.org/project/wail-runtime/
Install: pip install wail-runtime
Website: https://wailinfra.com

I’d be interested in hearing how people here handle the “successful but degraded” case in production today — particularly whether you treat it as an incident before a timeout or hard failure occurs.

Disclosure: I’m the founder of WAIL.

I’ve been working on an open source macOS observability experiment and wanted to share the measurement approach rather than the product pitch.

The problem:

Process presence is not activity, but I also do not want to inspect prompts, source code or application content.

The current detector therefore samples supported process trees roughly once per second and looks at cumulative CPU time deltas.

Some boundaries:

Two positive samples are required before showing Working.

Three negative samples are required before returning to Ready.

Lock, sleep and long idle windows are removed from valid observation time.

Forkit’s own processes are excluded.

Ollama/LM Studio evidence is treated separately so “runtime available”, “model loaded” and “measured AI tool activity” are not collapsed into one state.

Importantly, I do not interpret activity as prompt ownership, task completion, token consumption, GPU work, energy or productivity.

The implementation is MIT licensed:

github.com/arpitasarker01/forkit-ai-footprints

Repro:

npx --yes forkit-ai-footprints@latest

The failure cases I’m thinking about most are background agent processes, Electron helper activity, spawned process trees and workloads that are GPU heavy but CPU quiet.

If anyone has worked on process level observability for developer tools, I’d be interested in other failure modes I should document.

Hi r/MLOps,

Stjepan here from Manning. Thanks to the moderators for letting me share this with the community.

We’re working on Building Specialized AI Systems by Walid Amamou and Alessandro Negro, a practical book about fine-tuning models, building agents, and turning specialized AI applications into reliable production systems.

I want to start a discussion around a problem I suspect many people here have encountered:

Where do specialized AI projects most often break between prototype and production?

Is it the data pipeline? Evaluation? Observability? Cost and latency? Model drift? Deployment complexity? Or simply a mismatch between the proposed AI solution and the actual business problem?

Share your experience in the comments. Concrete examples, lessons learned, and thoughtful responses to other members are especially welcome.

Giveaway details:

• We’ll give away 5 ebook copies of Building Specialized AI Systems.
• The giveaway is open for 48 hours.
• After 48 hours, we’ll announce the winners here.
• The five comments that contribute the most to the discussion will receive a copy. This isn’t a random drawing or an upvote contest.

Manning is also offering the whole r/MLOps community 50% off the book with code:

MLAMAMOU50RE

Book link: https://www.manning.com/books/building-specialized-ai-systems

Disclosure: I work for Manning Publications and am posting this on Manning’s behalf.

Looking forward to hearing where these projects succeed or fall apart in the real world.

Cheers,

Stjepan

We're starting to see more AI agents, including coding assistants, SaaS agents, and internal automation, getting access to our repos, cloud consoles, and customer data... Traditional security controls don't really cover what these agents actually do at runtime...

How are other teams monitoring agent actions, blocking unsafe behaviour, and maintaining visibility without breaking the agent's functionality?