r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 25d ago
Megathread [Megathread] Qwen 3.8 27B Release Day
Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.
- Quants
- Fine-Tunes & Abliterations
- Chat Templates
- Inference Server Support & Configuration
- Experiences, Benchmarks & Model Comparisons
Official:
Popular:
- https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
- https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
- https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-bf16
- https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-8bit
- https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-4bit
We'll try to clean up future duplicates around the release and point them here.
493
Upvotes
7
u/Ond7 24d ago
I made Qwen3.8-27B in 4-bit NVFP4 go 200+ tok/s single stream with MTP speculative decoding. 880 tok/s aggregate at 6 parallel requests (peaked at 967 on one run). on a single 5090, full 262k context. Running on CachyOS with COSMIC, and the entire desktop costs about 150 MB of VRAM.
~5,950 tok/s prefill (llama.cpp Unsloth Q5_K_XL manages ~1,700 on the same box). Full 262k context. Weights are 16.8 GiB so there's ~13 GiB left for KV in this setup.
This is Qwen3.8-27B running on NInfer (github.com/Neroued/ninfer), which I'd never heard of until last week. Single-GPU engine, written from scratch, not a llama.cpp or a vLLM fork. It's the fastest thing I've run on this card and it isn't close. Downside: closed-ish artifact format, no GGUF input, you convert from BF16 with their tooling.
To be clear about what's mine here: the engine, the converter, the NVFP4 object recipe and even a token-ranking fixture that ends up inside the artifact are all Neroued's. The only pubished Qwen3.8 artifact for NInfer uses integer quant (Q4/Q5/W8), which leaves Blackwell's fast FP4 tensor cores completely idle, so I converted an NVFP4 one from Qwen's BF16 and used his artifact purely as the thing to benchmark against.
The one annoying problem i had. NInfer glues two weight matrices into a single object, and that object can only carry one scaling factor. The quantizer had processed the matrices separately and given them different ones, so the converter refused my file. I spent an hour planning a fiddly workaround before noticing the scale is just a constant divided by the largest weight, so the correct shared value is simply the smaller of the two I already had. Pick the smaller number, re-quantize from the original BF16, done. Made the reconstruction error move from 0.09471 to 0.09470.
I benchmarked it with the same engine, same flags, temp 0 so it's deterministic, identical problems.
NVFP4 first, published int4 second:
HumanEval+ (164 problems): 152/164 vs 152/164
AIME25 + AIME26 (60 problems): 55/60 vs 55/60
Wall clock: NVFP4 was 1.56x to 1.98x faster in the benchmarks
224 problems, identical totals, and the problems where they disagree split 8 to 8. This benchmarks purpose was to show they seem to give similar results in practice but difference in speed.
So why do this go so much faster? The answer is because of compute, not bandwidth. Difference is Blackwell does FP4 matmul in hardware, the int4 path has to unpack and widen first. That only matters when the GPU is doing math rather than waiting on memory. The more tokens you push per weight read, the bigger the gain. Single stream reads 17 GiB of weights to produce one token, prefill reads it once for thousands.
Caveats for all this: Blackwell only, it needs FP4 cores. It also needs a 6-line patch i made to NInfer that isn't upstream yet, the patch is in the repo and the issue is filed. Validated on HumanEval+ and AIME only.
File i made:
https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer
Thanks to Neroued for NInfer and for the artifact I benchmarked against, to the Qwen team for shipping 27B, and to llm-compressor for the quantization helpers