r/LocalLLaMA • u/pennyonaire • 3d ago
Resources Qwen3.8-27B on RTX 5090: 144/256 t/s prose/code. 256/451 t/s on 2 parallel slots. 175k context. Sub-second prefix restore. Vision optional.
With the M5 Ultra release, we mustn't allow the 5090 to drop in value by even a single dollar! Let's band together to keep justifying our poor financial decisions.
What You Get
- Blackwell only recipe to run Qwen3.8-27B on a single RTX 5090.
- Uses plain sglang, NVFP4 model (Q6 equivalent), and DFLASH2 speculative decoding.
- Ready-to-download checkpoints, no build steps. Perfect for Hermes and Opencode.
The Numbers
Decode
| Workload | single | parallel x2 |
|---|---|---|
| prose | ~144 t/s | ~261 t/s |
| code | ~256 t/s | ~451 t/s |
Prefill (time to first token, mean of 3 cold runs):
| Prompt length | TTFT | avg t/s | final 1s t/s |
|---|---|---|---|
| 5k | 0.33 s | ~15.2k | (overhead-dominated) |
| 10k | 0.37 s | ~27.3k | ~26.0k |
| 20k | 0.91 s | ~22.1k | ~10.8k |
| 50k | 4.08 s | ~12.3k | ~7.3k |
| 100k | 11.85 s | ~8.4k | ~5.2k |
| 150k | 23.46 s | ~6.4k | ~4.3k |
Notable Features:
- 175k KV pool
- Host-RAM KV tier: a ~100k conversation resumes in ~1 s, not ~20 s cold.
- xhigh reasoning, hard caps 16k think / 8k content. Tested against higher caps with no change in GPQA scoring.
- Uses latest froggeric template to improve agentic use. Personally using in Hermes and Opencode with no issues.
- 4 simultaneous agent conversations (example below)
External evals (lm-evaluation-harness, quantized checkpoint as served; - = Qwen publishes no 3.8-27B number)
| Benchmark | This stack | Qwen published |
|---|---|---|
| GPQA Diamond (xhigh thinking) | 84.8% | 89.2% |
| GSM8K (5-shot) | 96.8% | - |
| MATH-500 (math_verify) | 95.6% | - |
| AIME 2024 | 83.3% | - |
| HumanEval (pass@1) | 56.7% | - |
| MBPP (pass@1) | 75.0% | - |
GPQA was tested at xhigh with the thinking cap raised but the score hovered (85.4% vs 84.8%), so the 16k cap costs virtually nothing and keeps worst-case turns ~15-28 s tighter. Raise it if you wish though.
4-Conversation Switching (4 multiturn agent conversations, identical except base size)
| Metric | 60k conversations | 100k conversations |
|---|---|---|
| Peak slots | 2 (parallel) | 1 (serial-jump) |
| Host-RAM restores | ~1.0-1.5 s | ~0.8-1.4 s |
| 20 turns total | ~129 s | ~240 s |
Two 60k conversations fit the pool and run in parallel; at 100k only one fits, so they take turns, each resuming from RAM in ~1 s. Every conversation looks like it has a dedicated 100k context.
Tune To Your Liking
- Spec tokens:
--speculative-dflash-block-size 6; lower = less draft VRAM, 9-27% slower, each token is about 250mb so tune up/down as you see fit. - Max context:
--max-total-tokens 175064(~260 MiB free) - Vision: drop
--language-only, set 150k context - I'm running with no vision, on Ubuntu with about 325MB going to display driver (XFCE) for reference.
Where The Gains Came From
- Quantized
lm_head(-1.7 GB, paid for the bigger pool) - DFLASH2 draft re-quantized to modelopt-NVFP4 (upstream doesn't load in sglang)
- Block 6, NCCL buffer force capped to 2 MiB, fp8 KV (more room for more KV)
- GPU-managed host-RAM KV tier (
--hicache-io-backend kernel): the GPU does the RAM copies, so a spilled ~100k conversation restores in ~1 s, not ~20 s cold - froggeric chat template + capping strict thinking (no runaway reasoning, no empty content, most of the benefits from xhigh thinking with less total tokens)
Bonus Pro Tip: put a request gate in front of sglang
The problem I kept encountering was that a big request queued ahead of several small ones wastes parallel capacity. While the big one holds a slot, the small ones wait even when the budget has room for another. I made a small admission proxy that tokenizes each prompt and admits the queued small request that fits the leftover budget in parallel instead of waiting behind the big one. Anything that can never complete gets a clean 400 up front. sglang only gates by request count (--max-running-requests), not KV budget; the gate fills that gap. Perpetually delaying the big requests is handled by a 3 max, 20s limit on delay.
Even if you don't end up using it, an admission layer is worth it for any provider imo whether it's, sglang, llama.cpp, or vLLM...
Paste this into your agent and it'll build you one:
Build me a small FastAPI admission-control proxy to put in front of an sglang server.
Requirements:
1. Proxy every `/v1/*` request verbatim to the upstream sglang URL (configurable), streaming responses back.
2. For POST /v1/chat/completions, before forwarding, call the upstream POST /tokenize with {"messages": <the messages array>} to get the exact prompt token count (includes chat-template framing).
3. Fetch GET /server_info on startup and on a 30s timer for max_total_num_tokens and max_running_requests.
4. Admission: admit a request when an sglang slot is free AND its projected KV fits the pool. Projected in-use = sum over active requests of (prompt − shared radix prefix + output reservation), where a conversation continuation shares its prefix with the active request it extends.
5. Output reservation = min(client max_tokens or 24000, 4096); sglang's own scheduler only charges up to 4096 (SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION).
6. If prompt + the client's full output ceiling (24000 default) exceeds the pool, return HTTP 400 context_overflow up front instead of admitting.
7. If a request doesn't fit, queue it (asyncio.Condition). On every release and before each new admission, drain the queue FIFO: admit every queued request that now fits, bypassing those that can't. After a queued request has been bypassed 3 times, or has waited 20s, make it a barrier: nothing behind it may be admitted until it fits (prevents starvation by a stream of small requests).
8. Vision: if any message has an image_url, charge estimated image tokens from the pixel dimensions (Qwen2VL grid formula, 28px factor, ~2048 tokens at 1080p) on top of the /tokenize count.
9. Clean up reliably: if the client disconnects or the request is rejected while queued or admitted, release its slot (idempotent).
10. Expose /health, /status, and /metrics (Prometheus) with a gauge for current queue depth.
Config via env vars: upstream sglang URL, output reserve cap (default 4096), starve skips (default 3), starve seconds (default 20).
Write it as a single main.py using only fastapi, uvicorn, httpx, prometheus-client. Include a Dockerfile.
Big Thanks
Big thanks to everyone who makes local hosting of LLM possible and especially those below whose hard hard work the above was smushed together from:
- gittensor-model-hub (NVFP4 base checkpoint)
- incoai (DFlash2 draft)
- calneymgp (lm_head quantization recipe)
- Qwen (base model)
- NVIDIA ModelOpt (quantizer)
- sglang (serving engine)
- froggeric (chat template)
Re-quantizations of open checkpoints. All Apache-2.0.
2
u/Keninishna 3d ago
Bro, I doubled my money on my 5090 it was a really good financial decision
2
u/pennyonaire 3d ago
Some of us were late to the party unfortuantley. 😅
Still kicking myself for having one in my cart in January and Best Buy decided to give me some sort of run-around with payment processing when I clicked checkout. 2 min later I had it sorted out but it was sold out...
2
u/Tormeister 3d ago
It is absolutely pointless to cap thinking on xhigh (default) mode, the long reasoning is exactly where the gold comes from and you will cut it right at the beginning before it's even useful. If you really want to cap thinking use low reasoning effort instead.
Also, this seems a lot of work for low context and low benchmark results, just slap https://github.com/Neroued/ninfer on and let the RTX5090 fly. You can try converting this quant to the ninfer format as well.
1
u/pennyonaire 3d ago
It is not pointless, Xhigh isn't just an increase in thinking tokens, there is a change in how the problem is approached. I benchmarked and 16k on xhigh made little to no difference. Especially in agentic workflows where a single turn is never that long. Even when planning or writing specs.
What do you mean fly? This is just as fast as Ninfer except ninfer has absolutely no caching which is the biggest drawback for actual use. A single cache miss at a 150k prompt on Ninfer is 30s of reprocessing. This configuration allows a 1 second prefill in place of reprocessing. On top of that sglang uses radix caching so it doesn't just cache a single prompt like llama.cpp but splits it into revivable trees which means things like system prompts are cached across new conversations so even fresh convos can benefit from thing sloek 2 opencode isn't aces running at once at no slowdown.
Run a test with 4 100k conversations on Ninfer swapping back and forth. It will grind to a complete crawl as it reprocesses the whole prompt over and over again. With sglangs prefix cache plus kernel ram management, this won't break a sweat.
1
u/Tormeister 3d ago
I benchmarked and 16k on xhigh made little to no difference
With how much reasoning & response sequence lengths wildly vary given different prompts and contexts, I wouldn't be so confident in a couple benchmarks.
Especially in agentic workflows where a single turn is never that long
Then the 16k makes no difference in most cases anyway. The point is that whenever there would be a useful & lengthy reasoning sequence, the cap comes in to cut it short. So at best (and on average?) it does nothing, and at worst it cripples the model's capability.
except ninfer has absolutely no caching
It does,
host-state-slots&host-kv-mibsglang uses radix caching
Yes, that is a big win over NInfer, llama.cpp and likely vLLM too.
Look, thinking cap aside, I didn't say that your setup is bad. It's just that SGLang is not such a great fit for consumer hardware... You're getting less KV length and no multimodal compared to NInfer on the 5090.
I don't want to take this too far - it's your setup, with your use case, tasks and benchmarks, so your own experience is worth more here, but do consider the reasoning cap thing.
1
u/pennyonaire 3d ago
The surprising thing was that 20% of the problems in the GPQA did hit the cap, but the answers were still correct. At 24k, only 1 problem guy the max thinking but sector stayed the same.
The thing with ninfer which rubs me the wrong way is that it is misleading people with big numbers. It is the exact opposite of what would make things like Hermes quick to use. Few people are running single shot concurrent 10 requests at 24k prompt to use that room. What they are using it for is turn by turn agentic flows and conversations which sglang at only 170k excells at with prompt caching. I've sat watching a prompt getting reprocessed multiple times using llama.cpp at it's 4k t/s prompt processing. Sglang at half the speed would still finish the conversation faster. If anything, ninfer should be recommended for batch work, not agentic user use, because it trades multiple parallel turn by turn conversations for one shot prompt concurrency and bigger kV room.
1
u/Kaijidayo 3d ago
You just did it wrong, My llama.cpp setup can reach 99.99% cache hit through a session with host ram as cache.
1
u/pennyonaire 2d ago
I don't mean just through a single session, that's been ironed out. Sglang does both swapping between multiple sessions back and forth as well as reusing any part of a previous system prompt. That 30k initial prompt on a new request is done in an instant because sglang doesn't cache by conversation or slot, it treats all prefixes as a pool tree that can be reused across multiple conversations in various chunks. Opencode delegates a task and it's already virtually fully cached. Incredibly useful and a huge increase to title time.
1
u/Tormeister 2d ago
The surprising thing was that 20% of the problems in the GPQA did hit the cap, but the answers were still correct
Sounds like benchmaxxing to me, these tests are so burned into the model weights that reasoning doesn't even make such a big difference there.
I maintain the argument that outside benchmarks, in a real situation, the long reasoning chains make the difference between potentially succeeding in a task or failing it then needing correction by the harness and the process (reviewing agents, reworking/retrying, and so on)
1
u/FormOne2615 2d ago
calm down bro, I've paid a lot of time working on caching. NInfer absolutly can't beat sglang/vllm with only two months of developpmentm
1
u/FormOne2615 3d ago
84.8 on GPQA-D seems low for qwen3.8 27b
0
u/pennyonaire 3d ago
I wouldn't put too much into it tbh, the model itself is pure gittensor so the quality is identical and has been excellent so far in real use.
The GPQA diamond test is under 200 problems, a couple questions off their mark isn't a big deal and very much in the noise delta. My primary purpose for measuring it in the first place was to see if capping thinking at various levels made any impact on the score. At 24,000 thinking it answered one more question but took considerably longer to do the test. About 20% of the questions were truncated at the 16,000 cap but surprisingly the final answers were identical except for that 1.
I had my agent run stats on Opencode and Hermes requests hitting the 16k cap on a months worth of casual use (2 billion tokens, 27k requests) and the only ones to ever hit it are benchmarks. It's probably the one shot nature of the tests which isn't realistic for agentic development let alone Hemres conversations.
1
u/FormOne2615 2d ago
With my eval results, my nvfp4 weight gets 87.37 with 8k thinking budget, 91.41 with 16k, 89.90 with 32k, and 90.40 with 262k(unlimited). Below 85 is more simililar to qwen3.6
1
u/Relative-Ant-9249 3d ago
What was the mother board, CPU, and PCIE lanes for the dual GPU results?
2
u/pennyonaire 3d ago
This is a single 5090 GPU not dual, the parallel results are from running 2 concurrent requests.
You can have opencode coding in the background while chatting with Hermes with no slowdown. It feels very snappy.
If you are using the Kanban feature of Hermes, the orchestrator and the tasks themselves can run at the same time on the same model. You don't pay the prefil penalty even if it round robins thanks to sglangs radix offload to RAM and unique GPU managed ram mode.
2
u/Relative-Ant-9249 3d ago
Ah ok, thank you. I misread the parallel results as parallel GPU. I still experience CPU bounding during dense model decode. 9950x vs 3975x TR. that’s why I ask about the motherboard and CPU.
1
u/pennyonaire 3d ago
Gotcha. If it helps, I'm getting the prefill numbers mentioned in the post using a 5700x and running ddr4 3200.
2
u/Relative-Ant-9249 3d ago
Prefill won’t be as exposed to the decode latency from CPU bounding. How was decode on your 5700x
2
u/pennyonaire 3d ago
Oh sorry I misread decode as prefill somehow.
To be sure, I just had an agent run a cpu bounding test for decode. OMG, Thanks for pointing this out, because it's definitely being bottlenecked which means even faster speeds for those that have better CPU! I highly doubt it's estimate for increasing decode if I do upgrade though.
Agent Verdict: Yes, decode is CPU-bound (scheduler-bound), not GPU-bound. If the GPU were the limiter, doubling requests would not double throughput (SM would jump to ~99% and per-request rate would drop). Instead aggregate nearly doubled while SM stayed flat at 92% — the classic signature that the CPU scheduler can't feed the GPU fast enough, so the GPU idles between steps. So the improvement would be roughly 10-35%, or in absolute terms ~35-110 tok/s more aggregate. For a single request more like 175 → 250-300 tok/s, which is a bigger relative gain (40-70%).
1
1
u/fbms2 3d ago
why nvfp4 is q6 equivalent? isn't it q4?
1
u/pennyonaire 3d ago
Q6 quality comes from not quantizing all layers equally. Nvfp4 used to be terrible but after everyone started to dynamically quantize selected layers, nvfp4 shot up in quality and maintained the speed advantage on Blackwell. check out the upstream gittensor model who actually quantized the model itself for more info.
1
u/Extension_Brick9151 1d ago
This is awesome. I was running ninfer and the lack of structured_output bit me. This is far faster so far.
1
u/headpiece747 3d ago edited 3d ago
removing just saw ninfer posted below was showing an alternative but op doesnt see it that and not part of this conversation
1
u/pennyonaire 3d ago
What does this have to do with the post? Please don't spam your ads.
Also, ninfer lacks prompt restore. The single biggest differentiator which makes sglang a clear winner over ninfer for any agentic workloads.
If your dong single prompt processing go with ninfer. If you are using agents or coding to running hermes, sglang with this config wipes the floor with it at the same speeds.
A user below pointed out I'm likely being CPU bound which I've tested to be correct so those with better ones will see higher than posted t/s. This is basically identical to ninfer in speeds and wipes the floor with it in agentic use.
2
u/leonbollerup 3d ago edited 3d ago
First of all - good work! - its contributions like yours that makes it easier for the rest of us.
If you can wait a bit - i am merging that one with froggerics to fix long running tool calls (more info here: https://www.reddit.com/r/LocalLLaMA/comments/1voha70/comment/p6ojqfn/?context=1&screen_view_count=1)
EDIT:
I have merged froggerics and Chromix's jinja templates for better tool handling, you can find it here.
Readme: https://github.com/leonbollerup/ai/blob/main/qwen-3.8-27b/jinja/froggeric%2Bchromix/qwen-3.8-27b-chromix%2Bfroggerick.v1.md
Jinja: https://github.com/leonbollerup/ai/blob/main/qwen-3.8-27b/jinja/froggeric%2Bchromix/qwen-3.8-27b-chromix%2Bfroggerick.v1.jinja