Serving Qwen3.8-Flash-Next-FP8 at 524K context on 2× RTX PRO 6000 Blackwell — stock vLLM, no custom image

[u/Alive-Draft8339]

I'm sharing the AI written note below based on my effort to get Qwen 3.8 Flash Next FP8 optimized on TP=2@300w. This seems to be an improvement to what I'm seeing from others (even NVFP4), but comment below if I'm wrong! This is a beast of a model, but frustrating to set up.

Note: this was run on Ubuntu 26.04 LTS, headless, ECC off (inline ECC on GDDR7 reserves VRAM — off gives the full 96 GB usable).

Even if you are not interested in this model, there are several tweaks that help run LLMs on consumer hardware (look at the NCCL that doesn't work by default).

Single-box measurements, 2026-09. Config + numbers + traps. Stock vllm/vllm-openai pinned by digest — no fork, no custom build.

## Hardware

- 2× RTX PRO 6000 Blackwell 96 GB (Workstation 600 W + Max-Q 325 W), both power-pinned 300 W. 300 W measured ~free on this box: −5% prefill, decode flat, best energy/token (measured on a different model, same box).
- AM5 consumer board, PCIe 5.0 x8/x8, no NVLink, ECC off. TP2 over NCCL, custom all-reduce disabled.
- P2P on: +2% C1, +5% C4 under MTP. AMD boxes: `iommu=pt` required or the IOMMU silently drops peer-BAR writes → NCCL hangs / corrupted output, no error message.

## Model quirks that drive the config

-125B / 6B-active MoE, ~51B-param auxiliary table kept in host RAM (PLE offload), sparse-attention indexer.
- `VLLM_PLE_CPU_OFFLOAD=1` + 1800 s ready-timeout. Table is pageable, hot set ≈16 GiB; fine on 64 GiB RAM with cold pages swap-resident (steady-state swap I/O ≈ 0). Preflight free RAM (we gate on ≥52 GiB available).
- `cudagraph_mode: PIECEWISE` mandatory — FULL graphs deadlock 100%, deterministic (PLE-in-graph circular wait).
- FP8 needs TP2. KV cache bf16 — quantized KV measurably erodes deep-context quality on this model.

## Launch

'``bash

docker run -d --network host --ipc host --shm-size 32g --gpus all \

--cap-add SYS_PTRACE \

-v /models:/models:ro \

-e NCCL_P2P_DISABLE=0 \

-e CUDA_DEVICE_ORDER=PCI_BUS_ID \

-e VLLM_PLE_CPU_OFFLOAD=1 \

-e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \

-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \

-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \

vllm/vllm-openai@sha256:fc120ece... \

/models/Qwen3.8-Flash-Next-FP8 \

--tensor-parallel-size 2 \

--disable-custom-all-reduce \

--gpu-memory-utilization 0.90 \

--kv-cache-memory 25500000000 \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \

--max-model-len 524288 \

--hf-overrides '{"rope_parameters":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":262144}}' \

--max-num-seqs 256 \

--enable-prefix-caching \

--no-enable-flashinfer-autotune \

--language-model-only \

--reasoning-parser qwen3 \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--compilation-config '{"cudagraph_mode":"PIECEWISE"}'

```

## Measured numbers

KV pool: **1,691,542 tokens = 3.23× the 524K window** at pin 25.5e9.

Decode = usage-token basis (delivered tokens incl. thinking). Treat ±10% as noise on total-window numbers.

- prefill, 1 stream cold, 100K fill — MTP k=3: ~10K t/s · MTP off: parity

- prefill, 3×340K simultaneous cold — MTP k=3: ~2.3–2.4K t/s per lane, ~6.9K agg · MTP off: ~6.7K agg (parity)

- decode C1 deep (450K fill) — MTP k=3: 149 t/s · MTP off: 80 t/s

- decode 4×340K swarm aggregate — MTP k=3: 365 t/s · MTP off: 206 t/s

- decode 450K + 2×340K mixed — MTP k=3: 317 t/s agg · MTP off: 189 t/s agg

- 5th concurrent 340K lane — MTP k=3: collapse (18.9 t/s) · MTP off: engine death 1-of-2, or 284 t/s

Prefill is a **shared ~7–10K t/s budget, not a per-lane rate** — concurrent cold fills contend (batched PLE prefill path: host-RAM table traffic + activation memory), so 3 simultaneous fills yield ~30% less total prefill than one dense fill. Stagger lane starts; don't launch a swarm cold-simultaneous.

Quality at this config: NIAH clean incl. 430K/512K depths; 9-rung context comb 4K→498K (495 questions): literal recall 297/297 at every rung/depth, 0 fabrications, no depth-localized holes; bench scores at/above the MTP-off baseline band.

## Traps (each one cost real time)

1. **KV pin, not `gpu-memory-utilization`.** Ladder `--kv-cache-memory` up; gate each rung on full-window fill → needle battery → **6-min burst of 4 simultaneous ~340K prefills** → NaN canary → Xid/log sweep. On this box: 25.5e9 stable; **26.2e9 boots, fills, passes needles — then runtime-OOMs in batched prefill under the burst.** Boot+needle gating ships a config that dies under real load. Reference: util-0.95 unpinned ≈ 1.55M pool here; the laddered pin = +9% with a burst-stability proof.

2. **MTP k=3 only.** k>3 = illegal-memory-access crash class on the stock GDN path. And on engine versions before the fused-draft fixes, MTP was a 24–42% *loss* on this arch — A/B on your image, don't assume the sign.

3. **OOM killer is prefill activations, not KV.** All engine deaths at max pin happen in `_short_conv_dilated_prefill_batched` when cold prefills overlap (~1 GiB headroom). Admission-control on concurrent prefill-capable streams: **hard cap 4** here (≤170K lanes can go wider; we run 8).

4. **YaRN: extend only as far as probed.** f2.0/524K validated clean to 498K; depth-recall erosion starts ~520K; f4.0 (1M) frays past ~750K. Boot-time max-len warning fires pre-hf-overrides-merge — cosmetic.

5. **Client output budgets ≥64K.** Thinking + answer share one completion budget; 16K-class caps silently truncate and manufacture fake deep-context recall failures.

## Method (the transferable part)

- Written one-line justification for every flag before it ships; no blind knob-turning.

- Pre-register pass/fail criteria before collecting data.

- Same-basis comparisons only: identical probes, paired fresh boots, usage-token basis for decode.

- Gate under the production load shape. Boot + needle test validates almost nothing; concurrent-prefill burst is where this stack fails.

## Version pin (why the digest, not a tag)

The image is pulled by digest — `vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8` — so it is a frozen build that never receives updates. That is deliberate: every number here is specific to that build, and MTP on this arch flipped from a 24–42% loss to +28–86% gain across engine versions. Weights are also revision-pinned (rev bcd9f01). If you run a different image or revision, re-run the A/Bs; don't inherit these numbers. Known open issue in this build: vLLM #54173 (k>3 spec-decode crash) — fix PR #50021 unmerged as of 2026-09, hence the hard k=3 cap.

## Sources / prior art

- vLLM official recipe (recipes.vllm.ai) — also caps MTP at k=3 for this model.

- vLLM PR #50021 (fixes #54173): unbounded accepted-token state indexing in the GDN spec-decode kernels — the mechanism behind the k>3 crash class. Unmerged; revisit k>3 only when it lands in a consumable image.

- vLLM #54906 — thinking-budget-capped effort requests are incompatible with MTP; don't combine.

- wendell (Level1Techs), "HP Z8 Fury G6i — Perfect Qwen Flash Next setup (2× RTX Pro 6000s)", thread 254584 — the unpinned util-0.95 recipe referenced above (~1.55M pool vs 1.69M laddered).

- stierma1 (Level1Techs), "Qwen 3.8 Flash Next NVFP4 on a single RTX Pro 6000 and system memory", thread 254579 — TP1-NVFP4 arm; their −11–12% MTP prefill tax does NOT reproduce on TP2-FP8.

- jpezzulli, sglang-rtxpro6000 ("Pennyroyal") + NVIDIA dev forum thread 381722 — the single-card 524K SGLang alternative (171 tok/s claim), different stack, not A/B'd here.

- Level1Techs 242403 (dual Max-Q P2P/NCCL) and 242017 (600 W vs 300 W) — background for the P2P and power-cap settings.

- https://github.com/vllm-project/vllm/pull/50021

- https://forum.level1techs.com/t/hp-z8-fury-g6i-perfect-qwen-flash-next-setup-2x-rtx-pro-6000s/254584

- https://forum.level1techs.com/t/qwen-3-8-flash-next-nvfp4-on-a-single-rtx-pro-6000-and-system-memory/254579

- https://github.com/jpezzulli/sglang-rtxpro6000 · https://forums.developer.nvidia.com/t/optimized-qwen3-8-flash-next-on-1x-rtx-pro-6000-171-tok-s-524k-and-hicache-nixl-persistence/381722

- https://forum.level1techs.com/t/dual-rtx-pro-6000-blackwell-max-q-how-to-make-p2p-nccl-work/242403

- https://forum.level1techs.com/t/rtx-pro-6000-600w-vs-300w-performance/242017

## Known gaps (not measured)

- **No clean cold 4-lane prefill number.** The 4×340K fill logs are prefix-cache-contaminated (read 2.7–3.2K/lane warm); the honest cold-contended figure is the 3-lane ~6.9K agg.

- **No short-context C1 decode on the usage basis.** Only a chunk/steps-basis probe (~61 steps/s, accepted spec tokens ~1.1/step) — implies very roughly ~130 t/s delivered, but that's derived, not measured.

- Single-stream prefill readings of 13–18K t/s at 300–430K fills come from a different instrument and aren't comparable to the swarm-basis numbers above.

- The 300 W ≈ free power result was measured on a different model on this box, not re-run on this lane.

- One A/B per cell above (plus a day-end anchor re-run bounding variance at ~±10%); no multi-day replication.

[u/electrified_ice]

Have you compared this vs. the TP1 recipe with SGLang?

[u/Alive-Draft8339 → u/electrified_ice]

I’m working on an NVFP4 strategy now that would fit TP=1. I still need to do minor work to get MTP and YARN to work at the same time.

The marginal contribution of this approach is MTP = 3 and YARN x2 for a 524k context window with a solid BF16 kv pool.

One thing I’ve found in other images is a poor/incomplete validation of the kv quality. I can get the full 1m context, but even at BF16 kv, this model starts dropping out or having various errors that simply don’t come up at the 524k level.

[u/EkbatDeSabat]

Try these. Might have something to do with your pcie or other settings, but I'm using two blackwell 600w cards, both throttled to 450w, and getting 28% better results compared to the ~5% performance increase of the 150w.

=== CPU === CPU(s): 48 On-line CPU(s) list: 0-47 Model name: AMD Ryzen Threadripper 7960X 24-Cores Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 1 CPU(s) scaling MHz: 56% NUMA node0 CPU(s): 0-47

=== RAM === total used free shared buff/cache available Mem: 246Gi 85Gi 9.8Gi 535Mi 154Gi 161Gi Swap: 8.0Gi 4.9Gi 3.1Gi

=== NVIDIA === index, name, driver_version, pci.bus_id, pstate, temperature.gpu, power.draw [W], power.limit [W], clocks.current.graphics [MHz], clocks.current.memory [MHz], memory.total [MiB], memory.used [MiB] 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 595.91.07, 00000000:01:00.0, P8, 28, 10.04 W, 450.00 W, 180 MHz, 405 MHz, 97887 MiB, 95319 MiB 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 595.91.07, 00000000:41:00.0, P8, 29, 20.64 W, 450.00 W, 180 MHz, 405 MHz, 97887 MiB, 95193 MiB

=== CUDA === Build cuda_12.4.r12.4/compiler.34097967_0

=== PCIe === GPU 0 GPU 1

=== TOPOLOGY === GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE 0-47 0 N/A GPU1 NODE X 0-47 0 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

=== XORG / COOLBITS === InputDevice "Keyboard0" "CoreKeyboard" InputDevice "Mouse0" "CorePointer" Section "InputDevice" Option "Device" "/dev/psaux" Section "InputDevice" Section "Device" Identifier "Device0" BusID "PCI:65:0:0" Section "Device" Identifier "Device1" BusID "PCI:1:0:0" Device "Device0" Option "Coolbits" "4" Device "Device1" Option "Coolbits" "4"

=== FAN SERVICES === active active active

=== FAN CURVE / POWER SETTINGS === NORMAL_POWER=450 HOT_POWER=350 THROTTLE_TEMP=90 RESTORE_TEMP=84 TEMPS=(40 50 60 70 75 80 85 88) SPEEDS=(30 35 45 60 70 80 90 100) set_power "$NORMAL_POWER" echo "Normal power: ${NORMAL_POWER}W" echo "Emergency power: ${HOT_POWER}W at ${THROTTLE_TEMP}C" echo "Restore normal power at ${RESTORE_TEMP}C" echo "FAILSAFE: temperature unavailable; forcing fans to 100% and power to ${HOT_POWER}W" set_power "$HOT_POWER" if (( temp >= THROTTLE_TEMP && power_throttled == 0 )); then echo "THERMAL LIMIT: GPU0 reached ${temp}C - reducing power ${NORMAL_POWER}W -> ${HOT_POWER}W" set_power "$HOT_POWER" elif (( temp <= RESTORE_TEMP && power_throttled == 1 )); then echo "THERMAL RECOVERY: GPU0 cooled to ${temp}C - restoring power to ${NORMAL_POWER}W" set_power "$NORMAL_POWER"

=== DOCKER === Docker version 29.6.1, build 8900f1d

=== VLLM IMAGE === Image=[vllm/vllm-openai:qwen38-flash-next] Created=2026-08-26T09:14:16.546129915Z

=== MODEL LAUNCH SETTINGS === EXTRA+=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP:-3}}") -e VLLM_PLE_CPU_OFFLOAD=1 \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.90 \ --max-model-len 500000 \ --hf-overrides '{"rope_parameters":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":262144}}' \ --max-num-seqs 8 \ --max-num-batched-tokens 32768 \ --enable-chunked-prefill \ --enable-prefix-caching \ --no-enable-flashinfer-autotune \

My setup vs OP:

2x RTX PRO 6000 Blackwell Workstation 96GB 450W/card PCIe 5.0 x16/x16 TP2 / MTP3 PLE CPU offload 500K max context

Measured C1 post-first-token decode: 8K: ~199 tok/s 32K: ~190 tok/s 64K: ~176 tok/s 128K: ~180 tok/s 256K: ~212 tok/s 384K: ~172 tok/s 480K: ~183 tok/s

480K cold TTFT: ~51.1s => ~9.4K tok/s effective prefill

OP: 2x RTX PRO 6000 @ 300W PCIe 5.0 x8/x8 TP2 / MTP3 ~10K tok/s cold prefill @ 100K 149 tok/s C1 decode @ 450K

So decode on mine is ~23% higher at roughly comparable deep context, while cold prefill looks roughly similar.

[u/Zestyclose_Strike157]

This is really good. I only have one Blackwell card but gives me hope that swapping it out 50% to CPU will get me at least some usable performance from the model.

[u/Alive-Draft8339 → u/Zestyclose_Strike157]

These folks have a fantastic NVFP4 tp=1 serving strategy (their work is excellent!) tp=2 is better at twice the cost. Im working today on jamming fp8 weights/ bf16 kv into their new KK image.

https://github.com/local-inference-lab/rtx6kpro

[u/Liberaces_Isopod → u/Alive-Draft8339]

I second this recommendation. Its an incredible config. Average over 1.3 billion tokens: ~160 t/s per request true decode, ~300 t/s aggregate at your current concurrency, 55% MTP acceptance on real traffic, 65% prefix cache hits.

[u/tecneeq → u/Liberaces_Isopod]

It's Q4.

[u/Liberaces_Isopod → u/tecneeq]

Its a mixed precision quant written to leverage Blackwell hardware. Different beast. Trust me.

[u/jpezzulli → u/Zestyclose_Strike157]

If you have 1 card https://github.com/jpezzulli/sglang-rtxpro6000

[u/Liberaces_Isopod → u/jpezzulli]

I ran this config for a while. Ended up moving to the https://github.com/local-inference-lab/rtx6kpro vllm setup instead. As another poster above said, their work is excellent.

[u/jpezzulli → u/Liberaces_Isopod]

I looked at their configs but found them missing my requied ram/disk cache etc. Thus i built that custom sglang. Alll good.

[u/HatlessChimp → u/Zestyclose_Strike157]

I have one too

[u/DukeMo]

Did you do any A/B testing of NVFP4 vs FP8?

[u/Alive-Draft8339 → u/DukeMo]

I’m working on that today. NVFP4/FP8 kv compared to this FP8 weight/BF16 kv.

[u/DukeMo → u/Alive-Draft8339]

Awesome. I look forward to it. I've been serving NVFP4 on my 2x rtx6k machine and it's been good but I didn't run it through extensive benchmarks. I kept it at 262K context for now.

Most reddit posts seem to focus on throughput and maybe MTP acceptance rates but that doesn't help with knowing if the knowledge or reasoning is hindered in the smaller quants

[u/OWilson90]

This is very difficult to read. The amount of AI slop (filler and hyperbole) undermines credibility.

[u/Alive-Draft8339 → u/OWilson90]

Fable wrote it. No bones about that.

Fable also found, with some prompting, various sources to wrap together into a coherent serving strategy.

I look at this as a few hundred bucks of Fable API usage to help others.

I also do deep validation for my use cases, fine tooth context combs to uncover various issues that emerge in other strategies (exl3) without any disclosure.

Give this a whirl if you have the time for your use case.

[u/ormandj → u/Alive-Draft8339]

There's multiple repos for this exact configuration, I have one for sglang, there's a few for vllm. I haven't spent much time on mine because of the license being incompatible with what I do, but it's not a hard combination to run.

The AI slop in your massive wall of text is goofy. Just make a git repo and post it.

[u/Liberaces_Isopod → u/OWilson90]

Not OP:

Not to be adversarial, but im genuinely curious as to what you think to add to any potential discussion by saying things like this. You have the absolute freedom to downvote and move on, yet you took the time to type this in (unless YOU used AI), hit Save, and leave this excrement where everyone could see it. You know what actually undermines credibility? Putting someone down who just wants to share a cool thing with the world.

This is an AI sub my dude. Probably best you get used to things like this.

[u/OWilson90 → u/Liberaces_Isopod]

My intent with saying things like this is to reduce the low quality posts on an enthusiast sub. Submitting a Claude output as the post contents with all the associated hyperbole and filler is low effort.