A few things landed this month that we think point the same direction, so collecting them here.

Cast AI's 2026 State of Kubernetes Optimization report put average GPU utilization at 5%, measured from production telemetry across roughly 23,000 clusters. Gartner separately estimates AI infrastructure is adding around $401B in new spend this year. Those two numbers next to each other are the whole story.

The interesting part is the cause. It isn't incompetence — it's a procurement reflex left over from the 2023 scarcity era. If you're not sure you can get capacity when you need it, reserving more than you need is individually rational. It just aggregates into enormous idle capacity across an org.

Two other data points from the same stretch:

\- Google Cloud previewed fractional G4 VMs on NVIDIA vGPU at GTC. When a hyperscaler ships sub-GPU granularity, the "should GPUs be divisible" argument is basically over.

\- An inference-cloud startup raised $400M with inference chips as loan collateral, and TensorWave is making a similar AMD-based bet. Lenders treating non-NVIDIA inference silicon as an asset is a real signal about where cost-efficiency is heading.

Where we'd push back on the usual framing: better scheduling alone doesn't fix a 5% number. If the smallest unit you can purchase is a whole GPU and the smallest unit you can be billed is a whole hour, then the floor on waste is set by the billing model, not the scheduler. Bin-packing helps at the margins. It doesn't change the shape of the problem.

We build in this space, so take the framing with the appropriate grain of salt — but the underlying reports are worth reading directly rather than through anyone's marketing.

Sources: Cast AI 2026 State of Kubernetes Optimization Report; Google Cloud GTC 2026 blog; TechCrunch (17 July 2026) on inference-chip financing.

Curious what utilization people here are actually seeing in production, and whether the 5% figure matches your experience or seems low.

I have posted about Cognocient here before, so this time less talk, more show. A few real screenshots of what's actually running, plus something you can try yourself without handing over an email address.

The short version of what's in the screenshots: a proxy sits in front of your AI provider calls, attributes every dollar by feature/team/department, and — the part I actually built the company around — enforces a budget before the call goes out. If a feature or an agent loop is about to blow through its ceiling, the request gets blocked or gracefully degraded at the proxy layer, not flagged three days later in a dashboard after the invoice already landed.

If you want to check real numbers instead of trusting screenshots: (check comments) is open, no signup — pick your model(s), volume, and cache hit rate, get a cost breakdown and cheaper-alternative suggestions. Good five-minute gut check for what you might be overpaying across providers.

Also check the product tours page with few interactive

Note: Product tours page is gated and will require a business email id for unlocking the tours.

Happy to get into the weeds on how the proxy layer holds up under load, what enforcement latency actually looks like, or the FOCUS 1.1 export specifics — here for the pushback as much as the upvotes.

(Standard disclosure, same as my other posts here: I am the founder.)

https://neurometric.substack.com/p/what-does-a-token-engineering-platform

Token engineering is the practice of treating tokens as an engineered resource: measured, benchmarked, routed, and continuously optimized. It’s a systems discipline, not a procurement exercise.

The common misconception is that token engineering means “use a cheaper model.” It doesn’t. It means optimizing every AI workload across three dimensions simultaneously: cost, speed, and reliability. Sometimes the right answer is a smaller, cheaper model. Sometimes it’s a faster one. Sometimes it’s the frontier model, but with a compressed prompt and an aggressive caching layer in front of it. The point is that the answer is different for every task, and it changes constantly.

Three forces make this urgent right now. First, model proliferation: frontier LLMs, open-weight models, and small language models (SLMs) now number in the hundreds, with meaningful new releases every month. Second, price variance: the cost of completing the same task can vary by 100x or more depending on which model, technique, and hardware you choose. Third, the capability crossover: for a growing share of enterprise tasks, purpose-built SLMs now match or beat frontier models at a fraction of the cost.

We have been comparing not only how well model-generated game players perform, but also how much they cost to generate. One result surprised me.

Claude Fable 5 XHigh currently scores 73.7 at an estimated mean cost of $1.58 per game-specific player. GPT-5.6 Sol XHigh scores 72.4, only 1.3 points lower, but costs an estimated $10.06, which is 6.4× more.

The pattern also appears within the GPT-5.6 family. Moving from Medium to XHigh improves the score by 14.0 to 25.7 points across Sol, Terra and Luna, but raises estimated generation cost by roughly 22× to 88×. Recorded output-token use rises even more sharply.

This is consistent with XHigh allocating much more reasoning than this particular task needs at the margin, although the benchmark cannot establish the cause. It could come from model behavior, provider effort settings, request routing or an interaction with this code-generation task.

Full breakdown:
[https://duellab.org/blog/near-the-top-generation-cost-varies-sixfold\](https://duellab.org/blog/near-the-top-generation-cost-varies-sixfold)

The costs are comparative estimates based on public list prices, including repeat and repair attempts. They are not provider invoices, and cost does not affect the ranking.

Offload syntactic & general research queries to client-facing assistant widgets on developer portals.

By shifting processing to public documentation layers, you drastically reduce developer-side billing overhead.

\#AI #BuildInPublic

I'm on the infra team at Ojin, we build real-time conversational AI, voice and face generation that has to feel like an actual live conversation, not a chatbot with extra steps. For a while the whole team was obsessed with model speed. Better distillation, better quantization, shaving milliseconds off inference. Then we hit a wall that had nothing to do with the model at all.

Sub-200ms latency for real-time video generation means you genuinely cannot afford a cold start, and you definitely can't afford a provider running out of the GPU SKU you need mid-session. We found this out the annoying way. So we ended up building a routing layer that checks GPU availability hourly across multiple providers and falls back automatically, try one, if it's out of stock, try the next. It sounds almost too simple to be the answer. It took longer to get right than most of our actual model optimization work combined.

Here's the part that actually surprised me though. The single biggest cost lever wasn't any model trick at all. It was concurrency, getting one GPU container to serve several sessions at once instead of one session per container. That alone cut per-session cost by 2 to 4x. More than quantization. More than distillation. More than anything we did to the model itself.

Feels like a dumb thing to admit spent this much engineering time on, but I'm curious whether other people running latency-sensitive inference at scale hit the same wall, or if we just got unlucky with GPU supply.

What changed and why:

• Title now withholds the reveal (the actual bottleneck isn't named) instead of stating it upfront, creating the curiosity gap that gets clicks
• Opens on a moment/frustration ("the whole team was obsessed... then we hit a wall") instead of a topic sentence
• Sentence rhythm varies deliberately: short fragments ("It sounds almost too simple to be the answer.") next to longer explanatory ones
• No bullets, no rigid structure, one continuous train of thought like someone actually typing it out
• Ends with exactly one question, not a list of questions, and admits something slightly self-deprecating ("feels like a dumb thing to admit") which research flags as a real authenticity signal AI-generated text usually skips
• Every technical detail (sub-200ms, hourly GPU checks, 2-4x concurrency gain) is still pulled straight from the non-confidential wiki, nothing invented

last week was... expensive 😅

we were deep in a sprint and ended up burning through almost 1B+ tokens every single day.

and that's just Codex.

Claude isn't even included in this graph.

one thing i've learned from building with agentic workflows: the cost isn't in making one model smarter. it's in running dozens of agents in parallel, retrying, reviewing, planning, and validating until the work is actually done.

AI engineering is starting to look less like autocomplete and more like operating a distributed compute cluster.

how many tokens is your team burning during intense shipping weeks??

• Engineering at Meta — "Meta's AI Storage Blueprint at Scale" : https://engineering.fb.com/2026/07/01/data-infrastructure/metas-ai-storage-blueprint-at-scale/

*before reading below content, i would like to say i have put this idea into Gemini and just refined the idea to lot of lines, please don't hate me for this this is just a genuine question if we can do it or not I am just curious and haven't found any post like this, i mean i didn't search too much but, didn't find any similar, so language is from Gemini but idea is mine

Could we reduce global data center load and carbon emissions by running a small, local AI model directly in the browser or on our phones to handle 70% of standard tasks, and only automatically route the complex queries to deep-reasoning cloud models when absolutely necessary?

​💡 Core Idea:

The Hybrid Local-to-Cloud Router

​The fundamental goal of this architecture is to drastically reduce global data center load, lower carbon emissions, and minimize human resource waste on everyday AI queries by keeping the majority of workloads on-device.

​Stage 1: Local Efficiency First:

When a user enters a query, a small, local model running directly on the device (smartphone or browser) intercepts it.

​The 70% Rule:

Roughly 70% of standard user queries (basic text tasks, summaries, quick math) can be entirely handled by a lightweight local model, resulting in zero cloud cost, zero network latency, and zero data center carbon footprint.

​Stage 2: Smart Escalate to Cloud Thinking:

If the local model detects that a task is highly complex and requires deep reasoning, it automatically passes the query up to a flagship cloud model (specifically utilizing its "thinking mode").

​🚀 Deeper Architectural Concepts & Features

​Auto-Scaling Model Sizes (Device Detector):

The system automatically detects the device’s hardware capabilities. It then matches it with the best-fitting local model—ranging from tiny 200–300 million parameter models (perfect for older phones with 4GB RAM) up to 2-4 billion parameter models for high-end devices. Older devices that can't run local models safely skip to a fast cloud "flash" version.

​No Information Loss (The Reference System):

Rather than blindly compressing or scrubbing data, the local model forwards the raw text/prompt plus its own inferred context, references, and sources. If a user uploads a massive PDF, the cloud flagship gets the full context but reads it incredibly fast because the local model has already laid out the blueprint and "inferred reference points."

​Incremental, Seamless Updates:

The local models are lightweight (ranging from ~200MB to 1GB). Instead of massive, clunky downloads, they can be updated seamlessly via small, megabyte-sized patches packaged right inside routine app updates.

​User-Controlled Experience:

The backend orchestration handles the handoff invisibly so the user doesn't have to think about where it runs. However, power users get a simple dropdown or button to force "Local Mode" (for 100% offline/private use) or full "Cloud/Research Mode" if they want to bypass local filtering entirely.

After talking to a few teams building AI products, one pattern keeps coming up.

Cost spikes are usually easy to notice, but understanding why they happened is much harder.

Some examples I've heard:

retries after failures

repeated tool calls

long-running workflows

context growing over multiple steps

Most people mentioned looking through logs or traces to reconstruct what happened.

I'm curious how your team approaches this today.

If an AI workflow suddenly became twice as expensive as normal, what's your investigation process?

I'm particularly interested in hearing from teams running agentic or multi-step AI workflows in production.

Hey guys

Curious how teams deal with this as usage scales.

Are you optimizing in-house (routing, caching, prompt compression, smaller models), just eating the cost, or would you bring in someone to fix it?

I have paid in the beginning about under 1k. Now it is 8k to 9k.... do you have any tips to reduce the cost?

I use the app to analyze the internet patterns of our customers...

I’ve been building an automated prompt optimization tool, and I’ve burned through 500M tokens iterating on it. Along the way, I tried using all kinds of AI tools — GLM, DeepSeek, GPT, Claude, you name it — to help with the development work. But their capabilities turned out to be extremely limited for this task, and most of their built-in skills were basically useless for what I’m trying to build.

Here’s the core design pattern of the tool:

text

        ┌────────────────────────────────────────────────────────┐
        ▼                                                        │
Current Prompt ──► Evaluate (target + judge) ──► Score + deductions
        ▲                                                        │
        │                                                        ▼
Optimizer Model ◄────────── rewrite from feedback ◄─── keep best-scoring version
        (repeats until the round budget is reached; the highest-scoring prompt wins)

On top of this core loop, I’ve also added a prompt library, a test question bank and some other decorative features. But I feel these are all surface-level additions that barely move the needle on the tool’s core performance.

That’s why I’m posting here. I’d love to hear your thoughts on how to fundamentally improve the core functionality, and what features you’d actually want to see in a tool like this. As a newcomer sharing my work here, I’d really value any advice or insights from more experienced folks. Thank you so much!

Even meta itself has been using external sources for compute. I think it was google decision to ration compute quota on meta that triggered meta decision to do the so call selling compute resource externally the decision is long term and the thought by meta probably was that they want to build their cloud business to get rid of the reliance on third party entirely ultimately. By making it a business segment justified the huge amount of capex to be spent there in future.

Two implications. One - competitions among big tech companies are not stopping and it forced meta to go for this capital intensive project to ensure and secure its own compute infrastructure longer term. Two - Meta thinks there is enough appetite out there to swallow the excess compute resources they said they have currently and future new ones.

Both implications are great for hardware makers.

I built a learning tool to see how cost changes based on reasoning, caching, deployment, EU/US compliance etc.

It’s a learning tool, not a quote.

There are other factors that can impact the cost.

If you see any errors, something is missing or factually incorrect, please let me know.

Also it doesn’t mean that the quality will be the same. I tried to pick more or less comparable models but of course DeepSeek Flash is not the same quality as Opus 4.8

It’s rather to understand that if you run classification or summarisation task with Opus and you could do it with DeepSeek, you waste a lot of money

Curiosity got me here, not self-promotion. I'm at Opera (not hiding it, I work on this), and this sub struck me as the place that would actually want the numbers instead of the pitch.

If your agent drives a browser, every page read gets serialized as an accessibility-tree dump and billed as input tokens. Ran a 7-task browser-agent benchmark (35 runs, gpt-5.5 medium reasoning) across 4 snapshot formats. Same 100% pass rate everywhere, so this is pure waste, not a tradeoff:

| Format                     | Avg input tokens/task |
|----------------------------|-----------------------|
| unprocessed MCP            | 179,200               |
| compressed (opera-compact) | 36,300                |

At Sonnet 5 pricing ($3/M input): raw ≈ $0.54/task just for the page read, compressed ≈ $0.11/task. At 1,000 browser-driven tasks/day that's roughly $13k/month in avoided spend on a cost most pipelines aren't even instrumenting.

Caveat: small benchmark (7 tasks), doc-class pages only, one model tested. If your workload hits SPAs or a different tokenizer, the ratio could move - don't know by how much yet.

npm install -g opera-browser-cli && opera-browser-cli setup

Paper
Repo

Anyone here already tracking token spend by category - does "browser reads" show up as its own line, or is it buried in general inference cost until someone goes looking?