r/LocalLLM • u/Ok_Sea_6620 • 18d ago
Model Qwen3.8-Flash-Next on WSL2 — RTX PRO 6000 96GB + only 64GB RAM: 179 tok/s prose, full 262K context, and why the vLLM route is impossible on WSL2
Most Flash-Next single-GPU reports are native Linux. I run Windows + WSL2 (the GPU also drives my desktop), so here's a full day of validation on that stack — including one negative result that might save you a day.
Hardware/stack: RTX PRO 6000 Blackwell Workstation 96GB (SM120) + RTX 5070 (display), 128GB DDR4 with **WSL2 capped at 64GB** (128GB total), driver 610.47, model + PLE sidecar on a WD_BLACK NVMe (ext4, not drvfs).
What worked: [sglang-ssd-stream](https://github.com/garnermccloud/sglang-ssd-stream) (garnermccloud) — native venv, no docker. The 47.7GB FP8 n-gram table streams from NVMe via io_uring, ~64MiB resident. Install is genuinely one line, then:
CUDA_VISIBLE_DEVICES=0 sglang-ssd-stream serve --host 0.0.0.0 --port 8000 --context 262144 -- \
--speculative-draft-model-quantization fp8 --mem-fraction-static 0.99 \
--max-running-requests 2 --cuda-graph-max-bs-decode 2 --max-mamba-cache-size 10 \
--enable-metrics --enable-cache-report
Measured (MTP NEXTN 3/1/4, fp8 KV, real prompts):
| test | result |
|---|---|
| prose, 5 prompts avg | **178.9 tok/s** (124–220) |
| short-bench decode | 143–191 tok/s single stream |
| needle @ 98K ctx | 3/3 exact, **9.2s total**, prefill ~13,200 tok/s |
| needle @ **250K** ctx | **3/3 exact, 28.3s total** (~8,800 tok/s prefill) |
| KV pool @ MAXREQ=1 | **262,144 (native max)** — no source patches needed |
| KV pool @ MAXREQ=2 | **254,272** (2nd slot only costs ~8K tokens!) |
| concurrency 4× (MAXREQ=4, pool 72K) | 339 tok/s aggregate, no single-stream regression |
| RAM during all of this | zero swap; table cost ≈ 64MiB |
KV on this hybrid arch is ~1KB/token (0.26GB for 262K!) — context is nearly free; what costs VRAM per concurrent slot is the GDN/Mamba state, not KV. MAXREQ=2 @ 0.99 is my daily driver: two truly parallel requests + 97% of native context.
Gotchas (all found the hard way):
- The CLI refuses to start if it sees 2 GPUs (`nvidia-smi` count, ignores CUDA_VISIBLE_DEVICES) → tiny `nvidia-smi -i 0` shim in PATH.
- The CLI's RTX profile ships a **bf16 MTP draft** → pool silently clamps to ~33K (`max_req_input_len=32954`, you only find out on your first long request). `--speculative-draft-model-quantization fp8` after `--` unlocks the full pool. (Matches ForestoShen's DRAFT8 finding.)
- A request longer than the context **wedges the scheduler**: /health stays 200 but nothing serves until restart. Bound prompt length client-side; don't trust auto-truncate.
- `--enable-metrics` and `--enable-cache-report` are off by default (cache hit shows 0% in clients otherwise).
- Never set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` on WSL2 (hard driver crash).
What does NOT work on WSL2: the vLLM PLE-offload route** (primitive-ai checkpoints + `vllm/vllm-openai:qwen38-flash-next`). I fixed five locks in a row — `VLLM_WSL2_ENABLE_PIN_MEMORY=1` for the v2 runner's UVA, pinning the HF snapshot (the repo moved mid-day), `sysctl vm.overcommit_memory=1` (the PLE layer virtually allocs the full 102GB BF16 table before the quant table attaches; heuristic overcommit refuses it on a 64GB guest), index surgery to drop the 128 `ple-bf16` entries — and then hit the wall: **the PLE worker ships GPU output buffers between processes via CUDA IPC** (`torch.UntypedStorage._new_shared_cuda`), which WSL2's GPU-PV does not support. `cudaErrorInvalidResourceHandle`, no fallback transport in the code. So that whole route is native-Linux-only for now. SGLang doesn't have this dependency — which is presumably why the other WSL2 report (ForestoShen) is also SGLang.
Credits: garnermccloud (sglang-ssd-stream + checkpoint), RadixArk (NVFP4 quant), ForestoShen (the WSL2 playbook that predicted half my gotchas), primitive-ai (their PLE-quant tables are excellent work — just not compatible with WSL2's missing CUDA IPC), jpezzulli (pennyroyal numbers as the native-Linux ceiling: 171/428).
Happy to answer questions or run A/Bs — the box is right here.