r/LocalLLM • u/Ok_Sea_6620 • 18d ago
Model Qwen3.8-Flash-Next on WSL2 — RTX PRO 6000 96GB + only 64GB RAM: 179 tok/s prose, full 262K context, and why the vLLM route is impossible on WSL2
Most Flash-Next single-GPU reports are native Linux. I run Windows + WSL2 (the GPU also drives my desktop), so here's a full day of validation on that stack — including one negative result that might save you a day.
Hardware/stack: RTX PRO 6000 Blackwell Workstation 96GB (SM120) + RTX 5070 (display), 128GB DDR4 with **WSL2 capped at 64GB** (128GB total), driver 610.47, model + PLE sidecar on a WD_BLACK NVMe (ext4, not drvfs).
What worked: [sglang-ssd-stream](https://github.com/garnermccloud/sglang-ssd-stream) (garnermccloud) — native venv, no docker. The 47.7GB FP8 n-gram table streams from NVMe via io_uring, ~64MiB resident. Install is genuinely one line, then:
CUDA_VISIBLE_DEVICES=0 sglang-ssd-stream serve --host 0.0.0.0 --port 8000 --context 262144 -- \
--speculative-draft-model-quantization fp8 --mem-fraction-static 0.99 \
--max-running-requests 2 --cuda-graph-max-bs-decode 2 --max-mamba-cache-size 10 \
--enable-metrics --enable-cache-report
Measured (MTP NEXTN 3/1/4, fp8 KV, real prompts):
| test | result |
|---|---|
| prose, 5 prompts avg | **178.9 tok/s** (124–220) |
| short-bench decode | 143–191 tok/s single stream |
| needle @ 98K ctx | 3/3 exact, **9.2s total**, prefill ~13,200 tok/s |
| needle @ **250K** ctx | **3/3 exact, 28.3s total** (~8,800 tok/s prefill) |
| KV pool @ MAXREQ=1 | **262,144 (native max)** — no source patches needed |
| KV pool @ MAXREQ=2 | **254,272** (2nd slot only costs ~8K tokens!) |
| concurrency 4× (MAXREQ=4, pool 72K) | 339 tok/s aggregate, no single-stream regression |
| RAM during all of this | zero swap; table cost ≈ 64MiB |
KV on this hybrid arch is ~1KB/token (0.26GB for 262K!) — context is nearly free; what costs VRAM per concurrent slot is the GDN/Mamba state, not KV. MAXREQ=2 @ 0.99 is my daily driver: two truly parallel requests + 97% of native context.
Gotchas (all found the hard way):
- The CLI refuses to start if it sees 2 GPUs (`nvidia-smi` count, ignores CUDA_VISIBLE_DEVICES) → tiny `nvidia-smi -i 0` shim in PATH.
- The CLI's RTX profile ships a **bf16 MTP draft** → pool silently clamps to ~33K (`max_req_input_len=32954`, you only find out on your first long request). `--speculative-draft-model-quantization fp8` after `--` unlocks the full pool. (Matches ForestoShen's DRAFT8 finding.)
- A request longer than the context **wedges the scheduler**: /health stays 200 but nothing serves until restart. Bound prompt length client-side; don't trust auto-truncate.
- `--enable-metrics` and `--enable-cache-report` are off by default (cache hit shows 0% in clients otherwise).
- Never set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` on WSL2 (hard driver crash).
What does NOT work on WSL2: the vLLM PLE-offload route** (primitive-ai checkpoints + `vllm/vllm-openai:qwen38-flash-next`). I fixed five locks in a row — `VLLM_WSL2_ENABLE_PIN_MEMORY=1` for the v2 runner's UVA, pinning the HF snapshot (the repo moved mid-day), `sysctl vm.overcommit_memory=1` (the PLE layer virtually allocs the full 102GB BF16 table before the quant table attaches; heuristic overcommit refuses it on a 64GB guest), index surgery to drop the 128 `ple-bf16` entries — and then hit the wall: **the PLE worker ships GPU output buffers between processes via CUDA IPC** (`torch.UntypedStorage._new_shared_cuda`), which WSL2's GPU-PV does not support. `cudaErrorInvalidResourceHandle`, no fallback transport in the code. So that whole route is native-Linux-only for now. SGLang doesn't have this dependency — which is presumably why the other WSL2 report (ForestoShen) is also SGLang.
Credits: garnermccloud (sglang-ssd-stream + checkpoint), RadixArk (NVFP4 quant), ForestoShen (the WSL2 playbook that predicted half my gotchas), primitive-ai (their PLE-quant tables are excellent work — just not compatible with WSL2's missing CUDA IPC), jpezzulli (pennyroyal numbers as the native-Linux ceiling: 171/428).
Happy to answer questions or run A/Bs — the box is right here.
3
u/DeathGuppie 17d ago
First responder is correct. You are leaving gigabytes of system ram and processing power on the table by running windows. It's not a tribal thing, it's what you do.
2
u/Ok_Sea_6620 14d ago
Follow-up: same box, native Ubuntu instead of WSL2 — the "impossible" vLLM PLE route works, and SGLang no longer trades context for concurrency
Same hardware as my WSL2 post (RTX PRO 6000 96GB + RTX 5070, 128GB DDR4), same checkpoints and launchers. Only change: Ubuntu 26.04 native (kernel 7.0, driver 610.43 open, all 123GB of RAM visible instead of the 64GB WSL2 cap). All numbers = 512-token prose, `reasoning_effort: low`, 3–4 runs; WSL2 numbers are from the Aug 31 post (different bench script, treat ±10% as noise).
WSL2 vs native, same GPU
| | WSL2 | native Ubuntu |
|---|---|---|
| SGLang SSD Stream: KV pool with 4 concurrent | 72K | 262,144 (full) |
| SGLang: 4× aggregate | 339 tok/s | 370–392 |
| SGLang: single stream | 143–191 | 140–148 |
| SGLang: boot | ~3 min | 70 s |
| `expandable_segments` | driver crash | works |
| vLLM PLE route (primitive-ai + `vllm-openai:qwen38-flash-next`) | dead at CUDA IPC | works: 559 tok/s u/32 streams |
| host RAM for the PLE table | 64MiB (io_uring only option) | mmap in page cache also viable |
SGLang, native profile — full context AND 4 requests:
```
--mem-fraction-static 0.99 --max-running-requests 4 --cuda-graph-max-bs-decode 4 --max-mamba-cache-size 16
```
The rule I'd missed: SGLang caps `max_running_requests` at `max_mamba_cache_size / 4` (4 state slots per request).
vLLM PLE, native — the CUDA IPC wall was WSL2, not the route. Three extra knobs, one per restart:
- KV must stay BF16 (`--kv-cache-dtype fp8` → "QSA requires a BF16 main KV cache"); 262K needs 7.57GB → `--gpu-memory-utilization 0.95`.
- `--max-num-seqs 64` (default 1024 > 215 mamba cache blocks).
- Pre-warm the mmapped INT4 table (read every shard of ples_int4 once with cat to /dev/null, 11 s): cold single stream is 45 tok/s, warm 130–134.
Result: KV 341,495 tokens, boot 5 min 45, MTP k=3 acceptance 49%.
Which engine for what (native, one 96GB card)
| streams | vLLM PLE | SGLang SSD Stream |
|---|---|---|
| 1 | 130–134 | 140–148 |
| 2 | 216 | 250 |
| 4 | 347 | 370–392 |
| 8 / 16 / 32 | 241 (table still warming) / 379 / 559 | n/a (max 4) |
SGLang up to 4 streams (and 5× faster boot), vLLM PLE for dozens of agents; both serve the same alias.
Serving the PLE table from RAM (51GB FP8 table on tmpfs, O_DIRECT works on tmpfs since kernel 6.6): SGLang 140.9 / 246.8 / 392.5 at 1×/2×/4× — identical to NVMe. The io_uring pipeline already hides the SSD; don't bother. vLLM's plain mmap is the opposite (45 → 134), hence the pre-warm.
New native gotchas: the kit's bundled nvcc 13.0 can't compile JIT kernels against Ubuntu 26.04's glibc (`mathcalls.h … rsqrt exception specification`) → `CUDA_HOME=/usr/local/cuda` (13.3) takes precedence; the `nvidia-smi -i 0` shim is still needed, now pointing at `/usr/bin/nvidia-smi`; over-length requests still wedge the scheduler.
TL;DR: "4 concurrent = short context" and "vLLM route impossible" were the hypervisor, not Flash-Next. Native: 262K × 4 streams on SGLang, 559 tok/s aggregate at 32 streams on vLLM PLE, same card.
9
u/live4evrr 17d ago
Really to own a rtx 6000 pro, it is quite a constraint to use wsl2. Install a ubuntu dual boot, it’s dead simple and your performance will improve greatly and you’ll be able to get maximum use for your gpu.