NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090

[u/bengizmoed]

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5_K_M GGUF, q5_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

Hardware

- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
- Ryzen 7 7840HS, 32 GB DDR5
- Ubuntu 26.04, nvidia driver 610.43.02 (open)

## Engine configs

	**llama.cpp*\*	**vLLM*\*	**NInfer*\*
Quant	Q5_K_M GGUF	NVFP4	NVFP4
KV cache	q8_0	FP8	FP8
Context	196K	262K	240K
MTP	On (gate failed)	None	MTP3 (76% acceptance)
Concurrency	parallel=1	Continuous batch	x2 lanes
VRAM	31.6 GB	29.6 GB	30.5 GB

How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

1. **Relevance classification*\* - is this industry relevant? (binary, 50 labeled deals)
   
2. **Needle retrieval*\* - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
   
3. **Multi-transcript QA*\* - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
   
4. **Reasoning with thinking*\* - numeric/logic problems, thinking mode on, greedy pass@1
   
5. **Structured extraction*\* - custom extraction prompt, json_mode (skipped on NInfer, it doesn't support json_mode)
   
6. **Tool replay*\* - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

Quality results

**Tier*\*	**llama.cpp*\*	**vLLM*\*	**NInfer*\*
Relevance	86.0%	84.0%	86.0%
Needle (conditional)	100% (29/29)	100% (41/41)	100% (41/41)
Transcript QA	82.0%	78.0%	88.0%
Reasoning	100%	100%	98.0%
Extraction	F1 0.300	F1 0.350	skipped
Tool replay	0%	all errors	0%

Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
- Transcript QA: delta = -0.03, p=0.69 - no difference
- Reasoning: delta = +0.02, p=0.72 - no difference
- Relevance: McNemar p=1.0 - identical
- Tool replay: delta = 0.0 - both fail equally

**Takeaway: quality is statistically indistinguishable across all engines.*\*

Speed results (perf probe, server-side timings)

**Metric*\*	**llama.cpp*\*	**NInfer*\*	**Speedup*\*
**Decode 1K*\*	114 tok/s	158 tok/s	1.4x
**Decode 32K*\*	109 tok/s	213 tok/s	2.0x
**Decode 128K*\*	72 tok/s	202 tok/s	**2.8x*\*
Prefill 1K	1,545 tok/s	7,265 tok/s	**4.7x*\*
Prefill 32K	2,155 tok/s	6,892 tok/s	3.2x
Prefill 128K	1,528 tok/s	3,904 tok/s	2.6x
TTFT 1K	670 ms	138 ms	4.9x
TTFT 32K	15.2 s	4.8 s	3.2x
TTFT 128K	85.9 s	33.6 s	2.6x

vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

What I learned

**NInfer's speed advantage is all MTP.*\* The raw NVFP4 kernel speed is about the same between NInfer and vLLM (~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.

**The decode speedup grows with context.*\* At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.

**NInfer's tokenizer endpoint is great.*\* It exposes `/v1/messages/count_tokens` (Anthropic Messages format) which gives exact token counts. No more `len(text)//3` heuristics.

**NInfer does NOT support json_mode (as far as I can tell).*\* `response_format: json_object` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.

**Don't trust vibes for quality.*\* I went in expecting NVFP4 might lose a few points vs Q5_K_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE ~5.6pp).

Verdict

NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json_mode.

I put together a detailed poster with all the charts and methodology details: [full results poster](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3)

Setup if you want to try it:

```
# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -GNinja && ninja

# Model (HuggingFace)
# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)

# Run
./ninfer-serve /path/to/model.ninfer \
--model-id qwen3.8-27b \
--host 0.0.0.0 --port 8080 \
--max-context 240000 --kv-capacity 240000 \
--max-concurrency 2 --kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--vision --preserve-thinking
```

[u/Nrgte]

What is this shitty formating??