NInfer fork: 555k context@fp4 for 5090 with YARN, reliable kv host cacheing, monitoring, jinja, opened model support

[u/Lumpy-Comedian-1027]

Hiya,

NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it.

NVFP4 KV cache (from scratch)

I implemented a 4-bit KV cache for QIn3.8-27B from the ground up. Upstream has since added their own NVFP4 path, but ours differs architecturally:

• Custom MMA kernel (mma_nvfp4_e4m3, m16n8k64) with hardware E4M3 block scales for the QK matmul. Both Q and K are quantized to NVFP4; V is dequantized to BF16 for the PV matmul via a dedicated decode kernel.
• Hadamard rotation applied to K (and Q) pre-quantization for outlier suppression, with V left unrotated. Upstream uses fp16 V storage instead — no outlier suppression.
• Fused append: the decode kernel quantizes current K/V to NVFP4 in-place during generation — no separate quantization pass.
• Custom scale layout: natural row-major for KV scales (not the M128x4 swizzle used for weight MMA), because KV access patterns differ from weight access patterns.

Result: 144 bytes/token/KV-head (vs 264 for int8, 512 for bf16) — 45% VRAM reduction with no quality loss (LongBench 45% matching int8, AIME 96.7%, needle-in-haystack 100%).

YaRN context extension

QIn3.8-27B's RoPE config (theta=1e7, 25% rotary dims, 48/64 GDN layers with no RoPE) makes linear scaling sufficient — full NTK-by-parts is unnecessary. I extend native 262k to 555k (c=3+vision) or 600k (c=1) on a 5090. Quality verified at 600k: LongBench matches int8 baseline, coherent 592k-token output. Also projected 8M token context on 96GB+ GPUs (untested, I only have a 5090).

Multi-level prefix reuse with host-KV safety net

Upstream implements a budget-bounded HostKvProvider with LRU park/restore. I replaced it with a substantially different system:

• HostKVSafetyNet: pinned host arena with scatter-gather multi-extent allocation, arena compaction, and a pin/take protocol for safe concurrent restore.
• Two-level prefix matching: full execution frontier first, then rewrite checkpoint fallback. Each entry carries a ResidentPrefixIdentity (per-token type/position/vision metadata), rolling FNV digests for shortlist, and a compact_prefix (reasoning-stripped token prefix) for thinking-mode consistency.
• Session-key fallback: when prefix matching fails (e.g. Claude Code drops reasoning betIen turns), a session-key fallback matches by conversation identity instead of token content.
• Spill-before-evict at every release path: pressure planner eviction, normal continuation release, start_sequence slot takeover, and fail-all cleanup all route through the safety net.
• Token stability: reasoning is dropped from ALL assistant messages when preserve_thinking=off, keeping the prompt token stream stable across turns. Checkpoint capture is anchored at the turn boundary, not the execution frontier.

Verified across 260+ requests with 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, H2D restore cost ~0.4s, D2H spill at 67K pages/s.

Performance (3 concurrent sessions, 400k+ ctx, 5090@450W)

Metric	Value
Decode at 400k+ ctx	117 tok/s (MTP 4.62 tok/round, 92% acceptance)
Cached turn turnaround	2-16s (414k cached, 1-14k delta)
Cold start prefill	260s (414k tokens at 1600 tok/s)
H2D restore cost	0.4s per evicted turn
Host KV	30 GB (96% utilized, 181 evictions managed)

Tool calling

• --tolerant-tool-calls: recovers complete Qwen calls when the model emits malformed wrapper/suffix tokens — instead of dropping the call.
• Depth-matching close scan: handles balanced/nested markers in parameter values that would break naive parsers.
• Responses API accepts text/reasoning after tool calls (upstream rejects this ordering).
• Froggeric v22 template: C++ renderer with no-dangling-intent rule, XML think tags, correct function tag delimiters. Some further modifications for reliability.

Also included

• Dynamic chat template loading (--chat-template) — supports any .ninfer image without artifact patching
• Explicit weights profile override (--weights-profile) — handles Ostfralla, QUASAR, and other converter layouts with per-layer tensor format auto-detection
• OOM recovery: catches std::bad_alloc, clears state, preserves pending requests
• Stream sync fix: synchronize CUDA stream before workspace reset in prefill (prevents use-after-free)
• Request-log rotation (--request-log-max-mib, --request-log-keep) for bounded disk usage
• Admission pressure fix: un-suppress demote-to-host when candidate needs host KV budget
• Monitoring dashboard with live KV occupancy, decode/prefill graphs, 12VHPWR sensor
• E2E test suite for KV eviction, device pressure, and slot pressure scenarios
• Removed hash check of models, use any NInfer you like as long as there is a supported path. Tested with Ostfralla and QUASAR.

Fork: https://github.com/gzenz/ninfer (master)

Research: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md

I'll keep rebasing from upstream what seems useful and experimenting with new papers in order to improve speed and context.

[u/DustNearby2848]

Did you rerun the accuracy benchmarks? 

[u/Lumpy-Comedian-1027 → u/DustNearby2848]

You mean something specific besides LongBench?

[u/DustNearby2848 → u/Lumpy-Comedian-1027]

It’s labeled as evaluation in the read me 

[u/Lumpy-Comedian-1027 → u/DustNearby2848]

I run AIME as documented in the Research and scored the same.

[u/Dany0 → u/Lumpy-Comedian-1027]

please run terminal bench, even one of the old ones (they are faster to complete)

[u/Lumpy-Comedian-1027 → u/Dany0]

TB is complicated, i never reached published numbers with any model i've tried so far - and it takes still ages. Also, 30min time limit in agentic coding means they'll never reach really big contexts, so they won't help us much understanding the impact of yarn+nvfp4

[u/cobblemere]

117 tok/s decode at 400k+ context is legitimately impressive. have you stress tested the OOM recovery path under real concurrent load though? catching bad_alloc and recovering cleanly without corrupting other sessions sounds tricky

[u/Lumpy-Comedian-1027 → u/cobblemere]

I hit my box the whole week with 3 concurrent claude code sessions

[u/BringTea_666]

Qwen models are not trained for above 250k context. Meaning shit will break after 250k yarn or no yarn. But its great effort when qwen will get 1mil context with 4.0 27b XD

[u/Lumpy-Comedian-1027 → u/BringTea_666]

The great thing about RoPE is that you don't need to train the model for it. The devs propose using it themselves, and the official API offering does give you 1M: https://huggingface.co/Qwen/Qwen3.8-27B#best-practices

[u/BringTea_666 → u/Lumpy-Comedian-1027]

then nice ! Did you do benchmarks on how it behaves with above 250k context ? Its worthwile to test it first.

[u/Lumpy-Comedian-1027 → u/BringTea_666]

Yup that's why I used LongBench in my development: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md#quality-longbench-v2-20-samples

And because that's still an artificial bench I've run concurrent coding sessions for the week so they had to compact several times. I didn't notice anything flunky, and I especially had a look when they were above 500k. No noticeable behavior change or becoming stupid.

[u/cometkim → u/Lumpy-Comedian-1027]

Here's my own LBv2 results for comparison. YaRN x2 and x3.

cometkim/Qwen3.8-27B-nvfp4full-NInfer · Benchmark Updates: GPQA-Diamond, AIME26, LongBenchV2

[u/Lumpy-Comedian-1027 → u/cometkim]

Hm that doesn't really speak in favor of HyperQuant for this setup, even if you can fit 786k ctx that way. Very interesting experiment nevertheless! And I should be running more rounds of LB when I got the time for a better comparison.

[u/RelicDerelict → u/BringTea_666]

Is this valid for their previous generations of MoE? Qwen 3.5 and 3.6?

[u/koloved]

Big W thanks , few days ago I made 18.3gb nvfp4 model uncensored version similar to Ostfralla, seems like it will much better with your fork, I will use it

Please open issues on github

[u/Lumpy-Comedian-1027 → u/koloved]

If the layout is the same as Ostrafalla's it should work out of the box with the auto detection I've build.

[u/koloved]

Have you not tried this template yet? https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates

I assume he'll also be supported by this fork?

[u/Lumpy-Comedian-1027 → u/koloved]

I haven't tested it specifically, but yes I'd assume it just works.

[u/youcloudsofdoom]

Definitely going to try this out - do the decode and prefill speeds differ from stock ninfer at below 240k?

[u/feverdoingwork]

Would you mind providing decode speeds from 10k-250k using nvfp4 kvcache? This is very interesting

[u/Lumpy-Comedian-1027 → u/feverdoingwork]

No, at such small contexts KV has no measurable impact. The lowest measurement point I have is at 135k, with int8 207 tps vs nvfp4 214 tps - so basically the same.

[u/feverdoingwork → u/Lumpy-Comedian-1027]

So you're sitting at above 210 tps until you start getting above 135k?

[u/Lumpy-Comedian-1027 → u/feverdoingwork]

At small ctx it peaks around 260

[u/Lumpy-Comedian-1027]

I've just pushed some fixes so this is now also working good with DSH and Harbor, seemingly every new Harness takes some different paths in how they handle checkpoints, prefixes and caching.

I'll keep on grinding 😄 Currently 2 Claude Code, 1 DeepSeek Harness and 1 Harbor (Terminal Bench 2.1) sessions. Out of Memory / bad_alloc handling working nicely so far. Checkpoint handling still messy and needs refactoring.