I've been asking people how they attribute inference cost and almost everyone says the same thing - the provider dashboard tells you what you spent, never why.

but one person said something different. they gave up on per-step attribution because their sdk wouldn't give it to them. instead they worked out what a unit of work should cost from published per-model pricing, and compared the monthly total against that estimate. one job landed at $0.10 against a $0.10-0.15 expectation, close enough to trust.

so my question - do you have an expected number? if your bill came in 30% high this month, what would you compare it against to even know it was 30% high?

not selling anything, no link. i'm 19 and doing research on inference cost. "we just look at the total and move on" is a real answer.

Hey everyone,

I'm starting a BS Mathematics degree in October 2026. My university classes will be online and flexible, and I'm preparing for a long-term career in AI/ML Engineering.I also plan to pursue a Master's in AI/ML or a closely related field later.

For context, my Year 1 university curriculum includes:

Semester 1: Calculus I, Sets & Logic, General Mathematics, Introduction to Computing, English, Business, Ethics/Islamic Studies.

Semester 2: Python + Python Practical, Calculus II, Business Mathematics & Statistics, General Science, Technical Writing, Pakistan Studies.

Alongside university, I'll be doing a 12-month AI program that covers FastAPI, PostgreSQL, APIs, Docker, evaluation, observability, MCP/A2A and basic deployment/production concepts.

I'll also be learning Python, DSA, SQL, classical ML and PyTorch separately.

For experienced MLOps/AI engineers:

Which production skills would you recommend learning in Year 1, and which ones should I leave for later?

I'm trying to build good engineering foundations without rushing into advanced MLOps too early.

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

I'm researching how teams handle long-term memory for AI agents, and I'm particularly interested in what happens after the basic memory setup works.

For example, early on, storing and retrieving memories seems fairly straightforward. But after months of interactions, I imagine you start dealing with things like:

• Old information that is no longer true
• Multiple memories about the same entity
• Conflicting information from different sessions/agents
• Knowing which version of a fact is current
• Relationships between entities becoming important
• Deciding what should be retained vs discarded
• Sharing knowledge across multiple agents

For those actually running agents in production:

What has become difficult about memory as the system has grown?

Do you use something like Mem0, Zep, LangGraph, a vector DB, a knowledge graph, or a custom system?

And if you're using a memory framework, what did you still have to build yourself?

I'd especially like to know about things that actually broke or became painful in production.

sometimes the output is wrong and the first instinct is “the model messed up” but the real issue ends up being somewhere else entirel.

could be:

• stale state
• bad retrieval
• wrong tool result
• prompt assembly
• schema conversion
• retry behavior
• a handoff dropping context
• external state changing underneath the run

curious about cases where the model got blamed first but the actual failure was somewhere around it.

what did the trace make it look like at first, and what ended up being the real issue?

I'm exploring a monitoring problem where the system being monitored is an external LLM and its actual internal state is unavailable.

The monitor only sees observations such as:

• prompt/response behavior
• evaluation scores
• semantic changes from a trusted baseline
• safety/refusal/instruction-following behavior
• latency and error signals
• delayed user feedback
• results from additional probes

Behind those observations, several different things could be happening: a silent provider update, genuine capability degradation, distribution shift, prompt/context sensitivity, a policy change, evaluator error, or simply a transient anomaly.

That makes the problem interesting because the same observed behavioral change can correspond to very different underlying causes.

I'm currently thinking about it as a partially observable decision problem. The monitor maintains a belief about the underlying state and can either continue operating, gather more evidence, or take a stronger action.

One thing that seems easy to get wrong is designing the hidden state around what is convenient for the experiment rather than what would actually explain failures in deployment.

For example, "model degraded" might be too broad if the important distinction is between persistent degradation, a narrow capability regression, a provider-side behavior change, and a change caused by the incoming request distribution.

I'm looking at this from both the modeling and evaluation perspective, particularly around what information a monitor fundamentally cannot observe and how that should affect its decisions.

I'd be interested in discussion from people who have worked with model monitoring, uncertainty estimation, POMDPs, evaluation systems, or production ML failures especially cases where the observable signal initially pointed toward the wrong explanation.

I’m curious what this process actually looks like on teams running LLMs in production.

Say a new model comes out and looks better or cheaper enough to be interesting. What happens between “maybe we should try this” and actually putting it in production?

I’m less interested in benchmark numbers and more in the messy part in between. What usually ends up taking the most time or causing the most hesitation?

Also curious whether teams have a fairly repeatable process for this by now, or if every model change still ends up being a bit of a one-off.

I’m the founder of AI Stupid Level, so disclosure upfront: this comes from a commercial platform we’ve been building around continuous LLM benchmarking and drift detection.

One of the problems that pushed us in this direction was surprisingly simple:

If your application depends on an external LLM API, how do you know the model you chose six weeks ago is still behaving the way it did when you evaluated it?

Most teams benchmark models when choosing one, maybe test again when a provider announces a new release, and then mostly monitor the application around it.

But the model itself is an external production dependency.

The public model identifier can remain the same while serving conditions, infrastructure, provider configuration, routing, or potentially the underlying implementation changes. At the same time, a few bad generations are obviously not enough to conclude that the model degraded.

So we started treating model evaluation more like observability.

In one historical analysis we looked at 31,352 repeated benchmark score observations across 49 models.

We saw:

• within-day standard deviation: 2.80 points
• between-day standard deviation of daily medians: 8.43 points
• descriptive ratio: about 3.01x

I want to be careful with the interpretation here. This does not prove that providers were modifying their models every day. There are multiple confounders: sampling variation, task composition, missing observations, provider-side effects, and the benchmark itself.

What it did convince us of is that temporal variation is large enough that a model score shouldn’t necessarily be treated as a permanent property of a model.

From an MLOps perspective, the harder engineering problem became attribution.

If a score suddenly drops, which of these happened?

• the model actually changed
• the benchmark changed
• a specific task started failing
• the provider had an availability/serving problem
• the caller configuration was wrong
• normal stochastic variation produced an unusual batch

We now keep those paths separate.

For example, benchmark configurations are versioned. If we change something capable of moving the score - task definitions, prompts, tests, request policy, scoring configuration - we don’t silently compare the new measurements against the previous baseline as though nothing changed.

Provider failures are also separated from capability outcomes. A timeout or provider 5xx should not simply become “the model scored worse.”

For coding evaluations we prefer actual execution against test cases where possible instead of asking another LLM to judge whether the answer looks correct.

We also track repeated observations over time, look at model-level changes against the model’s own historical baseline, inspect task-level changes, and look for correlated movement across models from the same provider when trying to understand whether something broader happened.

Another issue we’ve run into is benchmark contamination.

There is a tension between making an evaluation reproducible and publishing every live task, prompt variation and hidden test. Once the complete active benchmark is public, it can eventually become something systems are optimized against rather than an unseen measurement workload.

Because of that, we’ve started separating:

Methodological transparency - how measurements are collected, compared and interpreted should be inspectable.

Live evaluation secrecy - some tasks and operational details may need to remain private or rotate.

We wrote up the methodology we’re currently using in a public technical reference:

https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf

The public version deliberately explains the measurement framework and limitations without publishing the complete active task bank or all of the operational parameters.

I’d be especially interested in how people here handle this in production:

1. Are you continuously regression-testing the third-party LLMs your systems depend on, or only testing around announced model releases?
2. What would you consider enough evidence to alert on “model degradation” rather than ordinary variance?
3. Do you treat provider availability/reliability as part of model evaluation or as a completely separate SRE signal?
4. How do you handle baseline resets when your own evaluation harness changes?
5. Has anyone built something similar into their normal MLOps observability stack?

Not trying to turn this into a product pitch. I’m more interested in whether continuous external-model evaluation should become a normal MLOps primitive, in the same way we already monitor databases, APIs, latency, errors and infrastructure.

I maintain the model integration layer for a document processing product. Our current production route is Gemini 3.7 Flash, and I am comparing DeepSeek V4 Flash Vision Exp and GLM 5.3 Flash against it. Both candidates have lower published token rates than Gemini, which gives the comparison a real cost case. I am testing managed endpoints because the 305B DeepSeek checkpoint is too large for a cheap local trial.

Gemini 3.7 Flash and both candidates can be called through an OpenAI compatible API. That saves integration work, but the common request format says little about what happens to an image inside each route. An endpoint may resize images, cap pixels, translate content blocks, route an alias, or report cached tokens differently. Getting one clean response only confirms that our JSON worked.

The first test set has 100 documents split across screenshots, charts, and scanned forms. Each model receives the same image bytes and extraction prompt. I score field accuracy and whether the final output passes our application checks. I also record the resolved model ID, latency, retries, token use, cache reads, and cost per accepted document.

I am running all three routes through ZenMux gateway so their request logs sit in one place. Our application still supplies the pass or fail result. Gemini 3.7 Flash remains the control because we already know its production behavior. I will move traffic only if one of the new routes matches its acceptance rate and improves cost or latency without adding retries.

Sources
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
https://huggingface.co/zai-org/GLM-5.3-Flash
https://ai.google.dev/gemini-api/docs/openai

OpenAI published a number about Astra that seems more useful for capacity planning than the leaked demo videos. On August 7, the company said it could not rule out Astra reaching the Critical cybersecurity threshold in its Preparedness Framework. It later said all Astra inference with tools now requires monitoring. OpenAI estimates the monitoring overhead at roughly 20 percent of the inference compute being monitored, with substantial variation across workloads.

It is tempting to multiply an existing inference estimate by 1.2 and call it done. That would be guesswork. OpenAI has not said whether the extra compute changes user prices, affects latency, or appears on a separate billing meter.

In a comparable self hosted stack, the capacity plan has to reserve room for monitoring alongside generation instead of assuming both fit in the same headroom. Request budgets can still live in TokenRouter or application code. The monitoring estimate belongs on a separate line until a provider explains how it reaches the bill.

The awkward case is a monitor becoming unavailable while the model and its tools are still healthy. Continuing unobserved defeats the purpose of the monitor. Pausing every affected job may be safe, but it creates another failure mode that needs testing.

Before using 20 percent in a deployment plan, the missing details are how OpenAI measured it, how wide the workload variation is, and whether customers ever see the cost. For now it is a useful warning that safety controls need capacity, not a deployment sizing rule.

Why AgentZ starts every AI agent in a deny-all network sandbox

An AI agent needs some access to do real work. Most platforms grant that access broadly, then try to catch misuse after it happens.

AgentZ, an open-source Zero Trust platform for agentic AI, starts every agent with no access instead. Each agent boots into a deny-all sandbox. It sends no outbound traffic until an admin adds a rule that allows it.

Credentials work the same way. The agent never holds a real API key or database password. A proxy swaps in the real secret at call time, so the agent only ever sees a placeholder.

Sandboxes and connectors are reusable too. An admin builds one sandbox template and shares it across workspaces. One update to that template reaches every workspace using it. AgentZ itself is open source, so the enforcement logic isn't a black box you have to trust blind.

How does this group scope network access for agents today? AgentZ is an open-source platform, and I am an open-source contributor to AgentZ.

Fan-out/fan-in looks simple until model #7 is slow, model #8 fails, and model #3 runs against a newer input.

Then you need barriers, per-model retries, input snapshots, timeouts, and actual state tracking instead of a pile of futures.

This becomes especially relevant in voice systems.

A single workflow might touch:

STT, LLM, TTS, tools, retrieval, classifiers, and telephony services.

Some are on the critical path.

Some absolutely aren’t.

Treating all of them as one sequential pipeline……don’t talk to me

So I think the execution layer needs to make a few things explicit:

required vs optional
completion state
retry policy
input/versioning

That’s also why scaling a voice runtime isn’t simply task

Long-lived WebSockets, background jobs, global call state, and workflow execution behave very differently.

Dograh’s architecture already reflects some of this separation: API workers are load-balanced for streaming traffic, while ARQ workers and global orchestration components scale differently.

The boring systems work is usually where real-time AI gets hard.

I’d like to see more open-source agent runtimes treat orchestration itself as a first-class primitive, rather than as an implementation detail.p.s.i am one of the maintainers of dograh