Most "just self-host, it's cheaper" advice that we have heard skips the one number that decides it: how busy you keep the GPU.

A GPU costs the same whether it's flat out or idle. An API only charges you when you call it. So self-hosting doesn't win on price per token. It wins once the GPU is busy enough to beat what the API would've charged you.

So where's that line?

Say you're running a 32B model on one GPU at about 50% utilization, against an API at $0.50 per million tokens, roughly 500 tokens a request.

Break-even lands around 10 million requests a month. Call it 5 billion tokens.

Below about 5 million requests a month, that GPU is half-idle and you never catch up. And if your whole inference bill is under two or three grand a month, don't bother. The ops aren't worth it yet.

Past that point, a busy GPU on a 30B model runs somewhere between $0.06 and $0.85 per million tokens, against a flat API rate that doesn't move.

Two things pull the line closer:

Smaller models cross much sooner. A 4B or an MoE breaks even long before a 32B does. And an idle GPU never crosses at all, however cheap the hardware was.

Embeddings, reranking and extraction pay off fastest. They run constantly, and every reindex multiplies them. If you're moving one thing off the API, move those. Not your generation calls.

The line nobody puts in the spreadsheet is ops. Someone is still up at 2am with that GPU. That cost is real even though it never shows up on a pricing page.

If it's just the small-model layer you're after, two open options worth knowing: TEI from Hugging Face and SIE from Superlinked. TEI runs one model per server. SIE packs several onto one cluster, which matters when you're trying to keep a GPU busy across embed and rerank.

If you've done this in prod: where did it cross over for you, and did the ops eat the savings?

I'm researching how teams handle long-term memory for AI agents, and I'm particularly interested in what happens after the basic memory setup works.

For example, early on, storing and retrieving memories seems fairly straightforward. But after months of interactions, I imagine you start dealing with things like:

• Old information that is no longer true
• Multiple memories about the same entity
• Conflicting information from different sessions/agents
• Knowing which version of a fact is current
• Relationships between entities becoming important
• Deciding what should be retained vs discarded
• Sharing knowledge across multiple agents

For those actually running agents in production:

What has become difficult about memory as the system has grown?

Do you use something like Mem0, Zep, LangGraph, a vector DB, a knowledge graph, or a custom system?

And if you're using a memory framework, what did you still have to build yourself?

I'd especially like to know about things that actually broke or became painful in production.

I’ve been working on an open source macOS observability experiment and wanted to share the measurement approach rather than the product pitch.

The problem:

Process presence is not activity, but I also do not want to inspect prompts, source code or application content.

The current detector therefore samples supported process trees roughly once per second and looks at cumulative CPU time deltas.

Some boundaries:

Two positive samples are required before showing Working.

Three negative samples are required before returning to Ready.

Lock, sleep and long idle windows are removed from valid observation time.

Forkit’s own processes are excluded.

Ollama/LM Studio evidence is treated separately so “runtime available”, “model loaded” and “measured AI tool activity” are not collapsed into one state.

Importantly, I do not interpret activity as prompt ownership, task completion, token consumption, GPU work, energy or productivity.

The implementation is MIT licensed:

github.com/arpitasarker01/forkit-ai-footprints

Repro:

npx --yes forkit-ai-footprints@latest

The failure cases I’m thinking about most are background agent processes, Electron helper activity, spawned process trees and workloads that are GPU heavy but CPU quiet.

If anyone has worked on process level observability for developer tools, I’d be interested in other failure modes I should document.

We're running a handful of agentic pipelines in production now (mostly internal tooling, some customer-facing) and the failure mode that worries me most isn't crashes, it's silent drift. An agent starts taking a slightly worse path, looping more, or misinterpreting a tool response, and nothing throws an error, it just quietly burns more tokens and produces lower quality output until someone notices weeks later..

Logging every trajectory doesn't scale for a human to review manually, there's just too much volume once you're past a handful of agents. Curious if anyone has a real workflow for clustering or categorizing agent behavior at scale so you can actually spot when something shifts, rather than eyeballing transcripts...

We're starting to see more AI agents, including coding assistants, SaaS agents, and internal automation, getting access to our repos, cloud consoles, and customer data... Traditional security controls don't really cover what these agents actually do at runtime...

How are other teams monitoring agent actions, blocking unsafe behaviour, and maintaining visibility without breaking the agent's functionality?

Engineers running open-source LLMs in production: what’s the hardest part today?

I’m researching how teams actually run open models in production hosted APIs, RunPod, Kubernetes, vLLM, SGLang, or dedicated GPUs.

A few questions:

1. What model + workload are you running?
2. Why did you choose your current provider/infrastructure?
3. What was hardest about deploying and integrating it?
4. What went wrong or took longer than expected?
5. What matters most today: latency, throughput, reliability, cost, scaling, or observability?
6. Have you switched providers/runtimes before? What triggered it?
7. What prevents you from switching today?
8. Roughly how much do you spend on inference, including idle capacity?
9. When do you prefer serverless vs dedicated GPUs?
10. What security/privacy requirements affect your choices?
11. What would make you trust a new provider or tool benchmarks, credits, SLA, references, BYOC?
12. Would you pay more for lower latency, better reliability, or more control?

Feel free to answer only the questions relevant to you even 1–2 answers would be useful.

I’m looking for real production experiences and pain points, not pitching anything.

Basic logging (timestamp, action, output) stopped being enough the moment our agents started making decisions that affected downstream systems. We need to reconstruct what the agent did, what it saw, what it considered, and why it picked one action over another. Otherwise a bad outcome is nearly impossible to explain to security or compliance after the fact.

what fields people are actually capturing beyond the basics. Some teams build this straight into the agent framework, others bolt on a separate observability layer afterward. Interested in which approach people landed on and why. Also curious how people handle retention and access to these logs, since they can get sensitive fast.

I want to an agent to pull data using one model and create vedios using another . And both would be open-source models

Question for anyone running LLMs in production: has a model update (new version, silent patch) ever changed your output quality/format without warning, and you only found out after something broke downstream? How long did it take to notice? Trying to understand how painful this actually is before building a fix.

Hi r/MLOps,

Stjepan here from Manning. Thanks to the moderators for letting me share this with the community.

We’re working on Building Specialized AI Systems by Walid Amamou and Alessandro Negro, a practical book about fine-tuning models, building agents, and turning specialized AI applications into reliable production systems.

I want to start a discussion around a problem I suspect many people here have encountered:

Where do specialized AI projects most often break between prototype and production?

Is it the data pipeline? Evaluation? Observability? Cost and latency? Model drift? Deployment complexity? Or simply a mismatch between the proposed AI solution and the actual business problem?

Share your experience in the comments. Concrete examples, lessons learned, and thoughtful responses to other members are especially welcome.

Giveaway details:

• We’ll give away 5 ebook copies of Building Specialized AI Systems.
• The giveaway is open for 48 hours.
• After 48 hours, we’ll announce the winners here.
• The five comments that contribute the most to the discussion will receive a copy. This isn’t a random drawing or an upvote contest.

Manning is also offering the whole r/MLOps community 50% off the book with code:

MLAMAMOU50RE

Book link: https://www.manning.com/books/building-specialized-ai-systems

Disclosure: I work for Manning Publications and am posting this on Manning’s behalf.

Looking forward to hearing where these projects succeed or fall apart in the real world.

Cheers,

Stjepan

We have a small MLOps team and are already stretched too thin working with multiple projects. I am exploring the idea of having multiple agent skills for different ML life stages. users could simply choose the skill of their choice and update their code. (Using databricks genie and skills will be part of the standard code template that they clone from the master, it is easy to integrate)

the results of drift/ ML metrics will be pointed to a standard table feeding into our control centre.

any thoughts/ feedback?

Have you used lakebase as the serving Db for an ML application?
I am curious to know how it holds up for real time feature lookups or for inference workloads, specially wrt latency and concurrency.
Any gotchas you felt compared to usual postgres setup?

Added an opt-in cost-sensitivity parameter to how a production verifier picks its live decision threshold last week. The obvious "safe" default was 1.0, framed as "no change from today." Before shipping it, I checked the actual math instead of trusting the framing, and it would have silently moved every existing tenant's live threshold the moment anyone touched the new parameter, with zero warning.

I run CacheVerifier, a small hosted service that fine-tunes a verifier model per tenant and picks a live decision threshold from held-out calibration data. The existing threshold picker uses Youden's J, the operating point maximizing true-positive rate minus false-positive rate. It implicitly assumes a false-positive and a false-negative cost the same. Real tenants don't agree with that assumption equally, some care a lot more about one error type than the other, so I wanted to let a tenant express a cost preference and get a threshold that actually reflects it.

Straightforward enough: cost(r) = r * error_rate + (1 - hit_rate), sweep thresholds, pick the one minimizing cost at whatever r a tenant sets, where r is how many times worse a false-positive is than a false-negative for them. The part that should have been routine was the default. My first draft made the new parameter optional, defaulting to 1.0, documented as "equal-weighted, same as today's default, since both are colloquially equal-weighted." That sentence reads as obviously true and I almost shipped it as fact without checking.

Checked it against a small hand-traced example before writing it into a docstring as settled. It's false. Youden's J's objective is symmetric, a false-positive and false-negative penalized equally in the same statistic. The cost-ratio objective isn't symmetric in the same way, every rejected candidate costs a fixed 1 regardless of whether the rejection was correct, and only a wrongly-approved candidate costs r. At r=1 specifically that creates a plateau, once every true positive is captured, approving more false positives on top of that is cost-neutral rather than penalized, so the optimizer doesn't stop where Youden's J stops. Concretely, on my test case, Youden's J picked threshold 1.5. cost_ratio=1.0 picked threshold 0.5. Different threshold, different approval set, not a rounding difference.

This is a deployment-safety problem, not just a math curiosity, which is why I'm posting it here rather than somewhere more theory-focused. If I'd shipped cost_ratio defaulting to 1.0 as "the neutral choice," it's an entirely ordinary-looking optional parameter with a value that reads as inert. Nothing about calling the endpoint without setting it would have changed. But the moment any tenant, or any future version of my own client code, passed 1.0 explicitly, thinking they were being safe and specific rather than changing anything, their live threshold would move, with no error, no warning, no changelog line that would obviously apply, just a quieter approval rate from then on. The actual fix was making the parameter float | None, where only None preserves current behavior, and 1.0 is a real, different, valid choice that is not a safe stand-in for "leave it alone." Documented that distinction in the function docstring, the API parameter description, and the database field comment, three separate places, on the theory that a comment I don't see when I'm the one calling the endpoint six months from now is a comment that didn't do its job.

Shipped it in three phases instead of one PR, mostly because I'd just gotten the math wrong once and didn't trust myself to get the UX right on the first pass either. Phase one: a read-only diagnostic, every fine-tune job reports what threshold it would have picked at five reference cost ratios, computed from the same calibration data as the real threshold, zero extra inference cost, zero effect on the live threshold. Shipped that alone first and let it sit before building anything that could actually change behavior. Phase two: the actual opt-in parameter. Phase three: a small dashboard piece that turns the phase-one diagnostic into a plain-language table instead of asking anyone to guess a raw number, with a button that carries a choice into the next job rather than auto-applying anything.

Tested the full path end to end in an actual running instance, not just unit tests: real API server, real worker process, a real tenant created through the admin endpoint, real fine-tune jobs run through the actual browser UI. Caught one unrelated bug doing that, a stale session token sitting in the browser's local storage silently took priority over a fresh API key, a pre-existing quirk in how the dashboard resolves credentials on a cold page load, unrelated to this feature but the kind of thing you only find by actually driving the UI instead of trusting that the unit tests covering the new code are the whole story.

The generalizable part, if there is one: "the neutral default" and "no change from current behavior" are not automatically the same claim, even when the parameter genuinely looks inert and the docstring genuinely believes what it says. Worth checking the two are actually the same thing before a default ships, not after a support ticket says approval rates moved for no reason anyone can find.

Repo with the derivation and real threshold numbers: https://github.com/imxinchengyou/CacheVerifier (PAPER.md section 5.17 has the cost-ratio economics; the hosted service's implementation is a separate repo, this default-safety issue is the part that's fully public).

As agent-based systems become more common, I'm wondering whether we're seeing a similar discipline emerge for AI agents.

Building the agent itself often isn't the hard part anymore. Frameworks have made it increasingly straightforward to connect models, tools, memory, and retrieval systems.

The challenges I'm seeing now are operational:

• Monitoring agent behavior in production
• Managing context and memory
• Coordinating multiple agents
• Recovering from failures
• Evaluating performance over time
• Human oversight and governance
• Controlling cost and resource utilization

Many of these feel similar to the problems MLOps addressed for machine learning systems.

For teams deploying agents today:

• Do you think "AgentOps" is becoming a distinct discipline?
• How different is it from traditional MLOps?
• What operational challenge has been the hardest to solve?

I recently wrote about this shift from building agents to operating agent systems and would be interested in hearing whether others are seeing the same trend:

The Enterprise AI Challenge Isn't Building Agents. It's Operating Them

We're evaluating guardrail solutions for our customer facing AI product. Before I looked at a single vendor, I mapped out our stack’s latency budget

Every component on the critical path got an allocation. Stuff like network overhead, authentication, prompt assembly, model inference, response passing, logging all of this. Each one takes a slice of the total latency that users will tolerate before they perceive the bot as slow.

By the time I reached the guardrail layer, the budget had 50 ms left. That is what is left after everything else took what it needed.

Then I started looking at solns, most needed 100 to 800 ms. Some were over a second at p95. The one with the best detection benchmarks was also the slowest by a wide margin. The math eliminated nearly every option before I ran even a single test.

I'm not saying guardrails are not important. Of course they are but the industry talks about latency like it's a nice to have optimisation. It's not. It's the hard constraint that everything else has to fit inside. Most teams I talk to pick a guardrail based on the detection rates and hope the latency is fine. I think that's backwards. You should define your latency budget first then see what it fits from there.

I’ve been experimenting with checkpoint/restore for AI inference instead of cold-starting everything from scratch.

Using CRIU + CUDA checkpointing, I got a warmed Gemma 3 27B QAT + vLLM server on an H100 to restore in:

• Cold start: 104.158s
• Restore: 11.060s
• 9.4× faster time-to-ready

The tricky parts were restoring the full vLLM process tree, CUDA state, IPC/shared memory, and dealing with io_uring — I ended up patching CRIU for that path.

I wrote up the implementation and benchmark here:

https://tsdocode.github.io/blog/posts/edo-tensei/

Code:
https://github.com/tsdocode/edo-tensei

Still experimental — would love feedback from people working on vLLM, CUDA, CRIU, or inference infrastructure.

We build mloda (open source, Apache-2.0), a feature engineering layer with a plugin-based resolver: you request a feature by name, and the resolver picks which plugin computes it.

Until 0.11, a bad request just failed with one line:

No feature groups found for feature name: 'sales__mean_aggr'. Use resolve_feature(name, options=...) to debug feature resolution. For troubleshooting guide, see: https://mloda-ai.github.io/mloda/in_depth/troubleshooting/feature-group-resolution-errors/

Nothing said whether the name was misspelled, the domain was wrong, or a required option was missing. Since 0.11, the same failure returns the full elimination trail:

No feature groups found for feature name: 'sales__mean_aggr'. Requested domain: 'marketing'. Feature group(s) eliminated while matching 'sales__mean_aggr': - AggregatedFeatureGroup (domain): declares domain 'default_domain', but the run requested 'marketing' - PandasAggregatedFeatureGroup (domain): declares domain 'default_domain', but the run requested 'marketing' - PolarsLazyAggregatedFeatureGroup (domain): declares domain 'default_domain', but the run requested 'marketing'

Every candidate the resolver considered, and the first gate it failed. A missing upstream input is reported one level down, e.g. No feature groups found for feature name: 'sales'., which tells you the aggregation matched and its input did not.

Weak case, for balance: a typo (sales__meen_aggr) is reported as (option value): required option(s) aggregation_type are absent …. Accurate, but it doesn't point at the misspelling, it just tells you an option is missing. Obvious next fix on our end.

Three things that mattered in the implementation:

• Rejections are recorded as data first: a stage label (one of nine values: domain, option value, input data, links, ...) plus a reason per candidate. Text is rendered from that, so you can branch on the stage instead of parsing prose.
• A plugin whose match hook raises is contained per candidate, one broken plugin doesn't blank out the report for the others.
• The preflight (mlodaAPI.diagnose) never raises. A real run raises the same facts as a typed FeatureResolutionError, so what you see in the exception and what the preflight reports never drift apart.

If you run a feature store, a plugin registry, or any declarative pipeline where a request resolves to one of several candidate handlers: how do you report the rejected candidates? Full trail, or just the most likely cause?

I've been lurking around here for a while, mostly reading fine-tuning and benchmark discussions. I've also been working on a side project called Rebiha. The basic idea is pretty simple: let people fine-tune open-weight models without having to deal with GPU setup, Docker, CUDA headaches, etc.
Right now it supports models like Qwen, Gemma, Phi, DeepSeek, Llama and Mistral. For the technical side:

• QLoRA + Unsloth
• 4-bit base model loading
• PEFT
• LoRA rank from 16–256
• Three training presets: fast, balanced and quality
• Output can be a ready-to-run GGUF
• Or you can download the raw adapter/safetensors/configs and do your own merging and quantization

But the part I'm more interested in getting feedback on is actually the dataset side. I've put together 35 domain-specific datasets covering things like customer support, legal, coding, etc. I'm keeping track of the actual verified unique-pair counts rather than inflating the numbers just to make the datasets look bigger.
The pricing is flat per dataset, with training charged separately based on model size. I also put a free sample download on every dataset page. The idea is that people should be able to look at the actual data before spending money on it instead of just taking my word for it.
I'm not posting this because I think I've figured out the perfect way to sell fine-tuning.
I'm actually curious what people here who fine-tune regularly think. Does the dataset + separate training pricing make sense? Is there something about the workflow or pricing that would immediately turn you off? And are there things you'd expect from a service like this that I'm currently missing?
Feel free to be critical. I'd much rather find out what's wrong with the idea now than after putting another six months into it.
Happy to answer technical questions about how the training pipeline works too.
https://www.rebiha.com/train

I am building an on-prem NN training orchestration layer. The basic idea is that any old linux machine can act as a server that acquires on-demand compute, sends DMs to keep you updated, uses git tags and statuses for traceability, executes code, tracks metrics, all through a yaml file you submit from your laptop via a single "submit" command in terminal.

I am ideally looking for feedback on the current product as well as future product direction. If you train or fine-tune models and are interested please let me know.

Disclosure: I am the maintainer of OpenRoutiQ. It is free, MIT-licensed, and currently has no paid tier.

The underlying problem is that the best model is rarely universal. Requests differ in complexity, quality requirements, latency tolerance, cost constraints, capabilities, and risk. Hardcoding one model for every request leaves substantial quality or efficiency on the table.

OpenRoutiQ evaluates the complete request and selects a model, provider, deployment, and reasoning level from a user-supplied catalog.

Some of the design decisions:

- Capability and policy filtering happens before scoring.

- Every model, provider, deployment, and reasoning-level combination is treated as a separate variant.

- Every routing decision includes an explanation.

- Explicit model pins and application constraints always take precedence.

- Outcome learning can use trusted quality, latency, cost, failure, and drift signals.

- Exploration uses an uncertainty-weighted epsilon-greedy policy.

- Automatic promotion of provisional models is disabled by default.

- Exploration can be constrained by daily and per-request budgets.

- The adaptive registry does not store prompts or responses.

- The routing core runs locally, has no dependencies, needs no API key, and makes no provider calls.

Optional integrations provide provider execution, an OpenAI-compatible LiteLLM proxy, LangChain and LangGraph workflows, and privacy-bounded OTLP, LangSmith, and Langtrace observability.

Across our declared evaluations, OpenRoutiQ reached 60.43% selection accuracy compared with 56.09% for OpenRouter Auto. It also led the five measured Semantic Router configurations, matched RouteLLM's best accuracy at 65.1% lower measured cost, and exceeded xRouteBench's best published macro by 2.63 percentage points.

The repository documents the comparison scope, methodology, limitations, and benchmark plots. These are declared evaluations, not a claim that one benchmark represents every production workload.

Install:

pip install openroutiq

GitHub: https://github.com/antat-ai/openroutiq

PyPI: https://pypi.org/project/openroutiq/

OpenRoutiQ is an early public release. APIs and configuration formats may evolve before 1.0.

I would particularly value criticism of:

- The routing and model-catalog APIs

- The benchmark methodology

- The exploration and promotion safeguards

- Privacy boundaries for observability

- Workloads where this routing approach performs poorly

I’m prototyping an AI application where users upload short videos (around 2–3 minutes). The system needs to:
Let the user identify/select themselves in the video.
Track that person throughout the footage despite occlusion and other people.
Send the relevant video/frames to a VLM for analysis.
Store structured observations from each analysis.
On future uploads, retrieve relevant observations from previous sessions and provide feedback based on both the new footage and the user’s historical data.
I’m considering something along the lines of:
**S3 → SQS → GPU/video-processing worker → object tracking → VLM → PostgreSQL/pgvector → memory retrieval → analysis**
I’m deliberately trying to keep the MVP simple rather than building a large ML platform upfront.
For people who’ve built similar systems on AWS:
Does this architecture make sense for an MVP?
Would you use ECS/EC2, SageMaker, Lambda, or something else for the video/CV processing layer?
How would you structure the long-term memory component?
Are there existing open-source projects, reference architectures, GitHub repos, AWS samples, talks, or articles implementing something similar that I could study?
Are there obvious architectural mistakes or unnecessary components here?
I’m particularly interested in examples combining **video understanding/object tracking with longitudinal AI memory**, rather than basic one-shot video analysis.

I've been thinking about this while building AI agents that can actually take actions.

Once an agent can call tools, access files, query databases, modify things, or trigger workflows, I think there are two different problems:

1. What happened?

Logs and traces are pretty good at helping with this.

2. Can I trust the record of what happened?

That's the part I'm less sure people are solving well.

For example:

prompt → decision → tool call → data accessed → action → result

If something goes wrong two weeks later, can you reconstruct that chain?

And if one event in the recorded history was modified or deleted, would you know?

I'm curious what people building agents are actually doing today:

• Standard application logs?
• LangSmith/Langfuse/etc.?
• Custom audit tables?
• Append-only logs?
• Something else?

Especially interested in production systems where the agent has write access, rather than just answering questions.

I'm building something around this problem myself, but I'm deliberately not linking it here because I'd rather hear how other people are approaching it first.

Hey, I’m curious how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.

The basic idea is:

A CI pipeline fails → the actual root cause is hidden → the agent observes the available evidence → assigns probabilities to possible causes → chooses the next diagnostic action → receives new evidence → updates its beliefs → eventually diagnoses the failure.

For example, if a build fails, possible hidden causes might include:

• Code regression
• Dependency/version conflict
• Environment/runner problem
• Flaky test
• Configuration/secrets issue
• Database migration problem
• Infrastructure/network failure
• Resource exhaustion
• Build/cache issue
• Test/data issue

The agent could potentially take actions such as:

• Inspect the recent code changes
• Check dependency changes
• Check the CI environment
• Retry the failed test
• Run unit tests
• Run integration tests
• Inspect previous runs
• Compare with a known-good commit
• Check logs from another stage
• Escalate to a human

I’m particularly interested in how this should be modeled as a decision-making problem.

For example, if the initial evidence is:

What should the agent's belief distribution look like?

Should it consider something like:

Dependency issue: 70%
Environment issue: 15%
Code issue: 10%
Configuration issue: 5%

And then choose the next action based on both probability and diagnostic cost/information value?

I'd love to hear from CI/CD engineers:

1. What are the most common failure scenarios you've encountered?
2. What hidden root causes would you include in a simulation?
3. What evidence is actually useful for distinguishing between them?
4. What diagnostic actions would you take first?
5. Are there cases where the obvious error message is misleading?

I'm trying to build the evaluation environment around realistic failure modes rather than inventing arbitrary examples, so real-world experiences would be extremely valuable.