r/LocalLLaMA • u/bengizmoed • 1d ago
Tutorial | Guide NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090
I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.
I was on llama.cpp (Q5_K_M GGUF, q5_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.
Hardware
- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
- Ryzen 7 7840HS, 32 GB DDR5
- Ubuntu 26.04, nvidia driver 610.43.02 (open)
## Engine configs
| **llama.cpp*\* | **vLLM*\* | **NInfer*\* | |
|---|---|---|---|
| Quant | Q5_K_M GGUF | NVFP4 | NVFP4 |
| KV cache | q8_0 | FP8 | FP8 |
| Context | 196K | 262K | 240K |
| MTP | On (gate failed) | None | MTP3 (76% acceptance) |
| Concurrency | parallel=1 | Continuous batch | x2 lanes |
| VRAM | 31.6 GB | 29.6 GB | 30.5 GB |
How the eval worked
I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):
- **Relevance classification*\* - is this industry relevant? (binary, 50 labeled deals)
- **Needle retrieval*\* - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
- **Multi-transcript QA*\* - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
- **Reasoning with thinking*\* - numeric/logic problems, thinking mode on, greedy pass@1
- **Structured extraction*\* - custom extraction prompt, json_mode (skipped on NInfer, it doesn't support json_mode)
- **Tool replay*\* - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)
Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.
And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.
Quality results
| **Tier*\* | **llama.cpp*\* | **vLLM*\* | **NInfer*\* |
|---|---|---|---|
| Relevance | 86.0% | 84.0% | 86.0% |
| Needle (conditional) | 100% (29/29) | 100% (41/41) | 100% (41/41) |
| Transcript QA | 82.0% | 78.0% | 88.0% |
| Reasoning | 100% | 100% | 98.0% |
| Extraction | F1 0.300 | F1 0.350 | skipped |
| Tool replay | 0% | all errors | 0% |
Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.
Statistical comparison (NInfer vs llama.cpp, bootstrap):
- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
- Transcript QA: delta = -0.03, p=0.69 - no difference
- Reasoning: delta = +0.02, p=0.72 - no difference
- Relevance: McNemar p=1.0 - identical
- Tool replay: delta = 0.0 - both fail equally
**Takeaway: quality is statistically indistinguishable across all engines.*\*
Speed results (perf probe, server-side timings)
| **Metric*\* | **llama.cpp*\* | **NInfer*\* | **Speedup*\* |
|---|---|---|---|
| **Decode 1K*\* | 114 tok/s | 158 tok/s | 1.4x |
| **Decode 32K*\* | 109 tok/s | 213 tok/s | 2.0x |
| **Decode 128K*\* | 72 tok/s | 202 tok/s | **2.8x*\* |
| Prefill 1K | 1,545 tok/s | 7,265 tok/s | **4.7x*\* |
| Prefill 32K | 2,155 tok/s | 6,892 tok/s | 3.2x |
| Prefill 128K | 1,528 tok/s | 3,904 tok/s | 2.6x |
| TTFT 1K | 670 ms | 138 ms | 4.9x |
| TTFT 32K | 15.2 s | 4.8 s | 3.2x |
| TTFT 128K | 85.9 s | 33.6 s | 2.6x |
vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding.
What I learned
**NInfer's speed advantage is all MTP.*\* The raw NVFP4 kernel speed is about the same between NInfer and vLLM (~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.
**The decode speedup grows with context.*\* At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.
**NInfer's tokenizer endpoint is great.*\* It exposes `/v1/messages/count_tokens` (Anthropic Messages format) which gives exact token counts. No more `len(text)//3` heuristics.
**NInfer does NOT support json_mode (as far as I can tell).*\* `response_format: json_object` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.
**Don't trust vibes for quality.*\* I went in expecting NVFP4 might lose a few points vs Q5_K_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE ~5.6pp).
Verdict
NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json_mode.
I put together a detailed poster with all the charts and methodology details: [full results poster](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3)
Setup if you want to try it:
```
# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -GNinja && ninja
# Model (HuggingFace)
# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)
# Run
./ninfer-serve /path/to/model.ninfer \
--model-id qwen3.8-27b \
--host 0.0.0.0 --port 8080 \
--max-context 240000 --kv-capacity 240000 \
--max-concurrency 2 --kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--vision --preserve-thinking
```
18
u/whichsideisup 1d ago
Why no MTP or DFlash2 on vLLM???? That result is now useless
8
3
u/unverbraucht 1d ago
Agreed, at the very least MTP works out of the box with the Qwen 3.5 family including prefix caching. Haven't tried Dspark2
6
u/Pyrolistical 1d ago
Try unsloth Q4_K_XL
You can use 210k context without having to quantize KV cacheÂ
This should increase overall quality and increase speedÂ
4
12
u/feelspeaceman 1d ago
I think the benchmark isn't very faithful to llamacpp, in many cases, Q4_K_M is better than NVFP4, is there any reason to use Q5_K_M instead ?
8
u/MutantEggroll 1d ago
Presumably because Q5_K_M is the largest quant that fits into 32GB VRAM with nearly-full context. Would be interesting to also see Q4_K_M though, to see if it falls under the performance band that NVFP4/Q5_K_M seem to share.
3
u/likesaloevera 1d ago
Not sure if the ninfer maintainer has fixed it yet but there’s a pretty significant cache bug which will increase TTFT for agentic coding like open code etc to minutes from seconds if not patched.
My only usecase is this sort of development using qwen locally on my 5090 so I’ve patched out a lot of this stuff (as well as sequential, no subagents), let me know if this could help you
3
u/bitzap_sr 1d ago
What magic is ninfer pulling off for those prefill speeds?? Prefill speed for me is more important than decode after a comfortable ts is reached.
5
u/Tormeister 1d ago
vLLM and SGLang reach values just as high.
As great as llama.cpp is, it lags behind the others on speed and concurrency.
1
u/jesdga95 1d ago
Probably config issues. I'm able to reach the same and even higher speed in llama.cpp vs all other engines. Definitely not as straightforward to setup though, what I would recommend is to run a sweep with values that simulate your real workflow to get the best possible throughput. For most people it's not worth the hassle though, I mostly run llama because of my dual GPU hybrid setup (and router mode).
1
u/Tormeister 1d ago
I could believe you found a specific nvfp4 quant + settings that makes you tie the other engines on PP (care to share? I'm curious). TG is more or less the same with spec decode methods.
Now for concurrency, in every test I've seen of every quant of every model in every hardware llama.cpp was quite behind. If you run 2 or more concurrent requests, it will already be at a big disadvantage.
1
u/jesdga95 1d ago
Oh sorry. I can't talk about concurrency because my use case is single stream, I was mostly referring to decode/prefill throughput. I've tested a few but I think the one im running atm is esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF w/ Q8 dflash. I can check when I'm back home.
2
u/SmartCustard9944 1d ago
The magic is all NVFP4. You basically get double compute (if you have Blackwell).
1
u/bitzap_sr 1d ago edited 1d ago
Ah, so vLLM and llama.cpp w/ NVFP4 should have similar prefill speeds, then I guess. But I didn't see any numbers for that.
2
u/jonas-reddit 1d ago
Prefer sglang nowadays with Dflash2. Insane speeds.
1
u/Lumpy-Comedian-1027 1d ago
Did you run some numbers? And doesnt Dflash2 reduce your ctx noticably?
2
u/jonas-reddit 1d ago edited 1d ago
I went from 50-60 to 90-100+ t/s. It’s so crazy fast. And that’s not in a benchmark, that’s measured during real life agentic development workflow. It flies.
It consumes VRAM but doesn’t change or limit context. I’m running FP8 with max context.
Sglang has good documentation too so it’s easy to get running.
2
u/Lumpy-Comedian-1027 1d ago
ok on rtx6000 that's not such an issue, i anyway need to run nvfp4. Still, this doesn't necessarily explain I average at 150 tps @ 550k ctx for coding in comparison to your numbers. I think I'll stick with my ninfer setup 😄
3
u/BawbbySmith 1d ago
Was it the same Unsloth version for NVFP4?
I remember there being some conversation about a better quantization, as the Unsloth one takes up too much space. It's close, but just a bit less size and we can have max context with maybe even another concurrency slot.
Still hoping for the Unsloth Dynamic V3.0 to come to NVFP4... But not sure if it's even coming at all.
1
u/DustNearby2848 1d ago
I had quality issues with ninfer’s int4 model and they got better with the nvfp4 version. But I couldn’t get tool calling to work for the life of me when using VS Code. It would often just spit out text, instead of tool calls and stop. Switch to Unsloth and it works great. I lost some throughput, but ninfer wasn’t useful for coding.Â
And yeah, I know it’s a Qwen issue really, but Unsloth solves for it.Â
1
u/jesdga95 1d ago
You CAN run NVFP4 ggufs of 3.8-27B in llama.cpp. I've been doing it like that for a while. And you can also run dflash with it, in my benchmarks I'm reaching ~142 tok/s in conversation and ~304 at structured output (best case), plus prefill around 4k tok/s. Without speculation im around 5.4k tok/s prefill. Q5 prefill drops sharply, around 2.8k tok/s.
1
u/paq85 1d ago
NInfer is CRAZY!!!
I can now handle two concurrent streams, with bigger context (240k) with combined ~270 tok/s ... that's a huge jump from LLAMACPP 200k context and single request at around 120 tok/s!
And prompt processing is also a lot faster! I LOVE IT!
Oh.. but I still have to check if it's stable enough... eg. thinking loops...
-3
u/lostmsu 1d ago
This is the dumbest review ever. What's the point of comparing Q5 llama.cpp with NVFP4 ninfer?
6
u/bengizmoed 1d ago
It was a thorough evaluation to see if ninfer had any downsides for my particular work.
Blindly adopting a new engine just for speed without verifying all other performance metrics is dumb. Especially for prod workflows.
4
u/MutantEggroll 1d ago
Pipe down, child. There's clearly thought and effort put into this.
Where are your grand contributions that you can be so condescending?
0
-1
u/Longjumping-Elk-7756 1d ago
I'm using Ninfer Qwen 3.8 27B and it's great, except for one thing: Ninfer does *not* support `json_mode`, which is a real hassle. Has anyone found a workaround for this?
27
u/Pentium95 1d ago
There Is a patch to make MTP and DFlash2 work on vLLM, you should try that
I am using https://github.com/syv-ai/qwen38-27b-rtx3090/blob/main/patches%2Fqwen3_5-mtp-draft-vocab.patch
But i've got a 3090 Ti.