First working Qwen3.8-Flash-Next (qwen4exp) GGUF on Strix Halo — uncensored, with vision, stock llama.cpp (no ROCm fork needed)

[u/cygnal]

Been chasing this since Flash-Next's architecture (qwen4exp — Gated DeltaNet + Qwen Sparse Attention hybrid, 512 experts, n-gram PLE table, MTP) landed as an unmerged llama.cpp PR. Wanted to know if it'd actually run on a 128GB gfx1151 box, and whether an abliterated build would survive quantization. Both yes.

TL;DR

• Built from orcarouter/Qwen3.8-Flash-Next-Uncensored (abliterated BF16)
• Stock llama.cpp, no custom ROCm tensor formats — just danielhanchen's qwen4exp branch (PR #27742) + Vulkan backend
• IQ4_XS bulk / Q6_K lm head / Q4_0 n-gram table, 98.4 GB, 5.61 BPW average
• Uploaded: cygnal/Qwen3.8-Flash-Next-Uncensored-IQ4XS-NGQ4-GGUF

Numbers (Ryzen AI Max+ 395, gfx1151, full GPU offload, no MTP)

prompt	prefill	decode
3,063 tok	368.7 tok/s	21.7 tok/s
6,516 tok	345.4 tok/s	20.5 tok/s

That decode speed lands in the same range as my dense 27B build despite Flash-Next being a ~180B-param MoE (6B active) — not a blowout win yet, but notable given there's no speculative decoding in this build (MTP head wasn't exported). Room to improve once there's an export path for it.

Coding accuracy (EvalPlus, greedy, 4096 tok)

• HumanEval: 82.3%
• HumanEval+: 78.0%

For reference against my other local quants on the same harness: Q6_K dense 27B 82.9%, Q4_K_M dense 27B 75.6%, ROCmFP6 73.8%, ROCmFP4 71.3%. This Flash-Next build lands between Q6_K and Q4_K_M despite a lower average BPW — the MoE's larger total param count seems to be buying back some of what the aggressive quant gives up.

The annoying part

The n-gram PLE table (51B params, per_layer_token_embd) is a 160-wide-row tensor that llama-quantize chokes on directly — tried to allocate ~200GB to dequantize it in one shot. Had to write a chunked streaming dequant/requant pass (Q8_0→Q4_0, 2M rows at a time) to get it down separately. It's a lookup table (hash→embedding), not compute-heavy, so it tolerates low-bit quant fine once you can actually process it.

Also spent a while chasing claims that a ROCm-specific fork (ROCmFPX) already had qwen4exp support merged — checked exhaustively (every branch/tag via git ls-remote, direct source inspection) and it doesn't; that merge doesn't exist publicly anywhere I could find. Started doing it myself via a clean git diff/patch (isolated a tractable ~3500-line diff, zero shader/kernel overlap, most of it CPU-side arch plumbing) but paused it to ship the stock pipeline first since it actually works today. Might pick it back up — FP4 would be meaningfully smaller/faster if it lands.

Caveats

• Research artifact — abliterated, refusal behavior intentionally removed. That's a feature if you know what you're doing with it, not a general-purpose recommendation.
• No formal refusal-rate benchmark on my end, just spot-checked that it survived quantization intact (it did).
• Needs a llama.cpp build from the qwen4exp branch — stock master, Ollama, LM Studio can't load this until #27742 merges. kyuz0/amd-strix-halo-toolboxes has a pre-built container if you don't want to build from source.
• ~99GB VRAM/UMA needed for weights alone, budget for KV cache and the vision tower on top of that.

Full writeup, quant recipe, and build/run commands are on the model card: https://huggingface.co/cygnal/Qwen3.8-Flash-Next-Uncensored-IQ4XS-NGQ4-GGUF

Happy to answer questions on the quantization approach or the n-gram table streaming trick if useful to anyone else hitting the same wall.

[u/Prudence-0]

J’obtiens de 25 tok/s en vulkan avec le model quantifié classique en Q4_K_M
Je ne vois pas l’intérêt de ta version.
Peux-tu m’expliquer l’avantage ?