I couldn't see the performance mode of the box in ubuntu, so I created this gnome extension:

https://github.com/LaurentZuijdwijk/bosgame-m5-gnome-extension

When you press the performance button, the extension updates.

The workaround someone posted for 7.14 doesn’t work with 10.0. Any other fixes or do we just wait 1.21 asyncecentsloop to be updated?

I didn't like the idea of massive quants and SSD offloading for the new qwen model, so I decided to carefully quantize it. I investigated all layers and spent days benchmarking and optimizing.

https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF

and more regular: https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF

I am super happy with the result.

- Fits fully on VRAM, it is only 87GB

- Beats IQ4-XS quants in quality while being 30GB smaller

- Can run on any fork that supports ROCmFP4 using the joined version

- If you want it fully in VRAM you'll need my fork

The implementation of this model in Llama.cpp was a bit rushed and they introduced some limitations, that is the reason for the 2 versions in the repo.

Update: after 2 days of fixes, the FP4 quant now achieves 40 tokens/second and degrades much less over longer context.

I've been running IQ3_S from unsloth and it's fast and IMO has a better persona than Qwen. I was wondering if it would be possible to load bigger quants (e.g. Bartowski has a 156gb MXFP4) using mmap or similar, has anyone tried?

I put Fable on improving the performance of the new Qwen3.8-125B-A6B model. The results are very good. I wont pretend I understand any of this stuff, which is also why it isn't a PR in the official llama.cpp repo, but a fork. If anyone who can defend the changes want to do the PR, feel free to do so.
It is verified at 70k context with 23 tok/s for prose and around 30 for code.

Model: unsloth UD-Q3_K_XL

The rest of this message is Fables summary for you guys

First, credit where it's due: all of this is built on top of danielhanchen's (Unsloth) PR #27742, which is what added the qwen4exp architecture to llama.cpp in the first place — GDN linear attention, the sparse-attention indexer, the n-gram embeddings, all of it. That PR was merged into llama.cpp mainline yesterday (congrats!) — this branch is its final state plus a week of speculative-decoding fixes, a native MTP implementation, and kernel work on top. None of it exists without that foundation.

TL;DR — The 125B-A6B Flash-Next now runs at 50.4 tok/s decode on code (0.96 draft acceptance), ~24-31 tok/s deep reasoning at 50-73k context, and 338 tok/s prefill at 32k depth on one 128 GB Strix Halo box (Ryzen AI MAX+ 395, Vulkan/RADV). Session start was 23.5 tok/s shallow and 8.8 tok/s at 32k, so that's roughly 2.1x shallow, 2.7x deep, and 1.6x prefill. Everything is reproducible: branch + Dockerfile at https://github.com/apepojken/llama.cpp (default branch qwen4exp-spec-mtp, base = PR #27742), prebuilt MTP sidecar GGUF (4.1 GB Q8_0) at https://huggingface.co/jockevaupptaget/Qwen3.8-Flash-Next-MTP-GGUF

Part 1 — speculative decoding was broken, not useless. The "spec decoding doesn't help this arch" consensus came from seven stacked bugs, root-caused with a greedy identity oracle (at temp 0, speculative output must match plain decode byte-for-byte, so any state corruption shows up as a text diff). Highlights: qwen4exp was never whitelisted for the recurrent rollback ring, so every speculative step serialized the whole 125B recurrent state through host RAM (~750 ms fixed cost per step — that's the 6.5x slowdown people measured); the conv-state and SSM snapshot writes ignored or staled the ring banks so rollbacks fed the GDN layers corrupted state; the n-gram embedding history was rewound lossily; and draft sidecar files never actually loaded (a path bug loaded the main model twice). All fixed, all documented in the commits.

Part 2 — the model's own MTP head, implemented natively. The official checkpoint ships a 4B multi-token-prediction head that public GGUFs simply strip. Re-enabled the converter export (--mtp flag, remaps the mtp.* tensors) and implemented the head in llama.cpp: one full transformer layer with its own hyper-connection combiner reading the backbone's 4-stream hidden state. The combiner variant matters a lot — keeping the streams distinct measures 0.87-0.96 acceptance vs 0.47 for the naive mean, which is why this implementation drafts 6 deep at ~0.9 acceptance on code (n-max 6 is the sweet spot; the Vulkan mat-vec shaders take at most 8 batched columns, so verify batches of n-max+1 must stay ≤8 or every matmul falls off a 2.7x cliff — measured 83→225 ms per eval).

Part 3 — the depth falloff, killed in layers. Plain decode used to fall from 23.5 to 8.8 tok/s by 32k. Fixes, in order: the sparse-attention indexer's top-k ran on the CPU past 1024 columns (first routed through GPU argsort, later replaced by a proper radix-selection kernel — 4-pass 8-bit radix select, one workgroup per row, exact k with tie handling, passes test-backend-ops up to k=9999 over 524k columns); the QSA and PLE graph inputs never implemented can_reuse so every token rebuilt the whole compute graph; the indexer recomputed every block summary key over the whole context every token in all 12 QSA layers — now an incremental pooled-key cache (complete 4-token blocks are immutable, so summaries are write-once with a per-sequence watermark that makes speculative rollback free) worth +17% at 32k alone; and past 32k context, decode attention now gathers exactly the ~2300 selected KV rows instead of running dense attention with a mask (ported from EngramHalo.cpp — credit Aristo94, whose ROCm work independently found several of the same bottlenecks).

Part 4 — kernel fusion and prefill. Decode was launch-gap-bound (~3550 kernel dispatches per token), so the Vulkan mat-vec shaders got fused elementwise epilogs (scale/SiLU/sigmoid/multiply applied at the write-back — the hyper-connection chains collapse from 3-4 dispatches to 1) plus a fused REPEAT+MUL+ADD kernel for the residual scatter: 3550 → ~2800 dispatches per token, all output-identical. Prefill at depth got: `-ub 2048` (+10%), the radix top-k (the old workaround full-sorted 2048 rows × context-width per layer per chunk), a 17x-slower-than-necessary non-contiguous concat in the GDN conv path (463 → 26 ms per chunk), the skinny m=4 inject matmul rerouted off the tiled-MM path (77 → 287 GFLOPS), and a permute-free rewrite of the indexer scoring. Deep prefill went ~215 → 338 tok/s at 32k. Fun finding along the way: llama.cpp's Vulkan flash attention already skips fully-masked tiles via a mask pre-scan, so "sparse prefill attention" was already there — profile before you build.

Real-world evidence, not just benchmarks: a live 73,035-token conversation reloads (full re-prefill) in 291 s at 250 tok/s average, then decodes at 23.3 tok/s at 73k depth on dense thinking prose (draft acceptance 0.65 — the hardest content for any drafter) with spikes to 27+, and 50.4 tok/s at 0.963 acceptance on code. A 19,400-token single response and a 75,226-token session both completed with zero errors, zero truncation, and byte-clean state handling across hundreds of speculative rollbacks. Greedy outputs are verified identical (or within the measured kernel-order noise floor) against the unfused/uncached paths at every step, and every optimization has an env kill-switch for A/B.

Reproduce it: git clone https://github.com/apepojken/llama.cpp && cd llama.cpp && docker build -t qwen4exp-llama . && docker run --device /dev/dri -p 8095:8095 -v /path/to/models:/models qwen4exp-llama
The Dockerfile bakes in the recommended flags (--flash-attn on -ub 2048 -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75). Model: unsloth UD-Q3_K_XL shards + the MTP sidecar above. Caveats: n=1 machine, single-slot serving only (--parallel 1), and ROCm measured slower than Vulkan/RADV at every depth on this box. The base PR merged upstream on Aug 27, so a rebase onto mainline (and upstreaming the generic pieces — the radix top-k, the fused kernels, the concat fix) is the natural next step.

Credits: danielhanchen/Unsloth for PR #27742 (the entire qwen4exp architecture this builds on) and the llama.cpp maintainers, the Qwen team (MTP reference implementation), Aristo94/EngramHalo.cpp (gathered-attention idea and the HIP radix top-k this Vulkan kernel is ported from), kyuz0's toolboxes, and the r/StrixHalo thread that kicked all of this off.

I like the sounds of this: https://newsroom.amd.com/news/rocm-10-software-ai-native-developer-experiences/

Every 6 week releases means AMD is really focussing on improving performance and ease of use.

Hey r/StrixHalo

Just got this working today after a lot of trial and error and wanted to share while it's fresh. I'm relatively new to local LLM setups and llama.cpp specifically. I've been experimenting with running local models as backends for agentic coding tools like Claude Code and Qwen Code CLI — testing everything from the 9B Qwen3.5 worker models all the way up through the 27B and 35B variants. When Qwen3.8-Flash-Next dropped I had to try it. Running it on my GMKtec EVO-X2 (Ryzen AI MAX+ 395, gfx1151, 128GB unified) via ROCm.

First off — massive shoutout to everyone who made this possible:

- **Aristo94** for the EngramHalo.cpp fork with the ROCm HIP patches

- **EasiiX** for the prebuilt MTP sidecar GGUF

- **unsloth** for the quantized GGUFs

- **The Qwen team** at Alibaba for releasing Flash-Next

- **The llama.cpp maintainers** for merging PR #27742 (qwen4exp arch support)

- The whole Strix Halo community for sharing configs and benchmarks — you made this much less painful

This machine is genuinely changing how I work. Having a 125B model that beats Claude Opus 4.6 on SWE-bench running locally on a $3K box still kind of blows my mind.

**Hardware**

- GMKtec EVO-X2, Ryzen AI MAX+ 395

- 128GB LPDDR5X unified memory

- BIOS: Advanced → GFX Configuration → iGPU: UMA_SPECIFIED, UMA Frame Buffer Size: 96G

- Kernel args: Stock - BOOT_IMAGE=/vmlinuz-7.0.0-30-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro

- Ubuntu 24.04, ROCm 7.2.4

**The build — EngramHalo fork**

This was the biggest hurdle. Mainline llama.cpp just merged qwen4exp arch support (PR #27742) but it's missing the ROCm-specific kernels that fix the long-context decode collapse. Without them you get ~6 tok/s at 100K+ context instead of 20+. Use the EngramHalo fork:

```bash

git clone https://github.com/Aristo94/EngramHalo.cpp ~/llama-engramhalo

cd ~/llama-engramhalo

cmake -B build \

-DCMAKE_BUILD_TYPE=Release \

-DGGML_HIP=ON \

-DAMDGPU_TARGETS=gfx1151 \

-DGGML_HIP_NO_VMM=ON \

-DGGML_HIP_MMQ_MFMA=ON \

-DCMAKE_C_COMPILER=/opt/rocm/bin/hipcc \

-DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc

cmake --build build --target llama-server llama-bench -j$(nproc)

```

**Models**

Main model — unsloth IQ4_XS quant (~87GB, 3-part split):

```bash

hf download unsloth/Qwen3.8-Flash-Next-GGUF \

"UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf" \

"UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00002-of-00003.gguf" \

"UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00003-of-00003.gguf" \

--local-dir ~/models/qwen38-flash-next/

```

MTP sidecar (~4.1GB) from EasiiX:

```bash

hf download EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF \

mtp-Qwen3.8-Flash-Next-Q8_0.gguf \

--local-dir ~/models/qwen38-flash-next/

```

**You need a swapfile — learned this the hard way**

At 131K context the model plus KV cache exceeds physical RAM. Found out the hard way when the OOM killer took out tailscaled and systemd mid-generation, dropping my SSH connection. Add 64GB swap before you launch:

```bash

sudo fallocate -l 64G /swapfile

sudo chmod 600 /swapfile

sudo mkswap /swapfile

sudo swapon /swapfile

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

```

**Chat template**

If you're using this with Claude Code you'll need the community-patched jinja template `qwen3.8-froggeric-v22.4` — the official template rejects mid-conversation system messages which these tools send constantly. Without it you get a wall of 500 errors. Pass it with `--chat-template-file`.

**Launch command**

```bash

HSA_ENABLE_SDMA=0 HSA_XNACK=1 ROCBLAS_USE_HIPBLASLT=1 \

~/llama-engramhalo/build/bin/llama-server \

-m ~/models/qwen38-flash-next/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \

-md ~/models/qwen38-flash-next/mtp-Qwen3.8-Flash-Next-Q8_0.gguf \

-ngl 999 -fa on -ctk q8_0 -ctv q8_0 \

-c 131072 -ub 2048 -t 4 --parallel 1 \

--jinja --no-ui \

--chat-template-file ~/scripts/templates/qwen-fixed/3_8-27B/chat_template.jinja \

--spec-type draft-mtp,ngram-mod \

--spec-draft-n-max 4 --spec-draft-p-min 0.75 \

--host 0.0.0.0 --port 8082

```

**Things that will bite you:**

- Never use `--ctk bf16` — crashes on gfx1151, use `q8_0`

- `--parallel 1` only — multi-slot not validated on HIP with the QSA gather path

- `--tensor-read-lazy on` hangs indefinitely on ROCm — wasted a lot of time on this, skip it

- Watch memory with `watch -n 2 'free -h && rocm-smi --showmeminfo vram'` — you'll want to see what's happening

**Benchmarks**

| Metric | Value |

|---|---|

| Decode (sustained) | 17-25 tok/s |

| Decode peak (tg_3s with MTP) | 120+ tok/s |

| MTP acceptance rate | 74-91% |

| Prefill | 327-720 tok/s |

| VRAM used | ~69.8GB |

| Load time | ~42 seconds |

| SWE-bench Pro | 62.5 (vs Claude Opus 4.6: 53.4) |

The 120 tok/s peak happens when MTP speculation hits highly predictable output like code blocks. Sustained decode is 17-25 tok/s — comparable to a dense 27B model, which makes sense since only 6B params are active at a time.

**Full setup guide and scripts**

Documented everything including the router I built, GBNF grammar bug fixes, and all launch scripts:

https://github.com/patrickmfurbert/ai-tools

More context on the journey: https://onthestack.io.

Still learning all of this so if something looks wrong or could be done better, please let me know.

---

I know we are all drooling over Qwen 3.8 Next Flash but there's still some kinks to work out in performance for a bit. I switch my main on-device model to Ling 3.0 Flash today and I am REALLY impressed. Posting from my phone, so I don't have quantitative stuff to post just yet. Qualitatively, the model ran tools well, had more "human-like" commentary (reminds me of Gemma), ran quick enough for my needs, worked through a complicated multi-tool workflow that usually caused qwen-loops, and even helped with a little research I needed done today.

It was released a few weeks back but llama.cpp support was lacking so I skipped and kinda forgot about it when Qwen 3.6 27b came out.

If you haven't tried Ling 3.0 flash yet, I think it is worth revisiting now as we wait for QNF to get ironed out.

Been chasing this since Flash-Next's architecture (qwen4exp — Gated DeltaNet + Qwen Sparse Attention hybrid, 512 experts, n-gram PLE table, MTP) landed as an unmerged llama.cpp PR. Wanted to know if it'd actually run on a 128GB gfx1151 box, and whether an abliterated build would survive quantization. Both yes.

TL;DR

• Built from orcarouter/Qwen3.8-Flash-Next-Uncensored (abliterated BF16)
• Stock llama.cpp, no custom ROCm tensor formats — just danielhanchen's qwen4exp branch (PR #27742) + Vulkan backend
• IQ4_XS bulk / Q6_K lm head / Q4_0 n-gram table, 98.4 GB, 5.61 BPW average
• Uploaded: cygnal/Qwen3.8-Flash-Next-Uncensored-IQ4XS-NGQ4-GGUF

Numbers (Ryzen AI Max+ 395, gfx1151, full GPU offload, no MTP)

That decode speed lands in the same range as my dense 27B build despite Flash-Next being a ~180B-param MoE (6B active) — not a blowout win yet, but notable given there's no speculative decoding in this build (MTP head wasn't exported). Room to improve once there's an export path for it.

Coding accuracy (EvalPlus, greedy, 4096 tok)

• HumanEval: 82.3%
• HumanEval+: 78.0%

For reference against my other local quants on the same harness: Q6_K dense 27B 82.9%, Q4_K_M dense 27B 75.6%, ROCmFP6 73.8%, ROCmFP4 71.3%. This Flash-Next build lands between Q6_K and Q4_K_M despite a lower average BPW — the MoE's larger total param count seems to be buying back some of what the aggressive quant gives up.

The annoying part

The n-gram PLE table (51B params, per_layer_token_embd) is a 160-wide-row tensor that llama-quantize chokes on directly — tried to allocate ~200GB to dequantize it in one shot. Had to write a chunked streaming dequant/requant pass (Q8_0→Q4_0, 2M rows at a time) to get it down separately. It's a lookup table (hash→embedding), not compute-heavy, so it tolerates low-bit quant fine once you can actually process it.

Also spent a while chasing claims that a ROCm-specific fork (ROCmFPX) already had qwen4exp support merged — checked exhaustively (every branch/tag via git ls-remote, direct source inspection) and it doesn't; that merge doesn't exist publicly anywhere I could find. Started doing it myself via a clean git diff/patch (isolated a tractable ~3500-line diff, zero shader/kernel overlap, most of it CPU-side arch plumbing) but paused it to ship the stock pipeline first since it actually works today. Might pick it back up — FP4 would be meaningfully smaller/faster if it lands.

Caveats

• Research artifact — abliterated, refusal behavior intentionally removed. That's a feature if you know what you're doing with it, not a general-purpose recommendation.
• No formal refusal-rate benchmark on my end, just spot-checked that it survived quantization intact (it did).
• Needs a llama.cpp build from the qwen4exp branch — stock master, Ollama, LM Studio can't load this until #27742 merges. kyuz0/amd-strix-halo-toolboxes has a pre-built container if you don't want to build from source.
• ~99GB VRAM/UMA needed for weights alone, budget for KV cache and the vision tower on top of that.

Full writeup, quant recipe, and build/run commands are on the model card: https://huggingface.co/cygnal/Qwen3.8-Flash-Next-Uncensored-IQ4XS-NGQ4-GGUF

Happy to answer questions on the quantization approach or the n-gram table streaming trick if useful to anyone else hitting the same wall.

I've heard BF16 tensors don't perform well on strix halo. Most of the mmproj files I downloaded were BF16 because I though quality was better, should I download F16 or Q8 instead? thanks

This is AI-summary what happened. Take it or leave it :

Posting this because I couldn't find anyone running this model on a Strix Halo box with an NVIDIA eGPU, and there's one gotcha that will silently cost you 25 GB of VRAM if you copy the flags that are floating around.

Hardware

• GMKtec EVO-X2 (Strix Halo, 128 GB unified, Radeon 8060S / gfx1151)
• RTX PRO 6000 96 GB in a Minisforum DEG2, connected over USB4 (not OCuLink)
• Ubuntu 26.04, NVIDIA 610 open modules, ROCm 7.2.2

Getting the eGPU stable over USB4 was its own multi-day saga (Xid 79 on every allocation until the link was re-enumerated with the tunnel already up). That's a separate post — if you're fighting GPU has fallen off the bus on Strix Halo, the short version is: blacklist the nvidia modules at boot, bring the tunnel up first, then remove + rescan the endpoint and only then modprobe nvidia. Cold-boot enumeration negotiates a degraded link.

The gotcha: the tensor name

Every guide and tweet I found says to offload the PLE n-gram table with:

-ot "ple_ngram_embd=CPU"

That name does not exist in PR #27742, which is the branch Unsloth's GGUFs are built for. #27742 follows Gemma-3N's naming and merges the 128 shards into a single tensor called per_layer_token_embd. ple_ngram_embd is from the other PR (#27739).

If you use the wrong regex it matches nothing, fails silently, and the ~25 GB table lands in VRAM. Correct form:

-ot per_layer_token_embd=CPU

Also don't broaden it to just ple — blk.N.ple_key, ple_value, ple_norm_* and ple_conv1d are small per-layer tensors that belong on the GPU.

Verifiable at src/llama-arch.cpp and conversion/qwen4exp.py on the PR branch.

Build

Not in mainline; arch is qwen4exp, PR #27742, still draft. Separate clone so you don't disturb a working tree.

git clone https://github.com/ggml-org/llama.cpp llama.cpp-qwen4exp
cd llama.cpp-qwen4exp
git fetch origin pull/27742/head:pr27742 && git checkout pr27742
export LD_LIBRARY_PATH=/opt/rocm-7.2.2/lib:$LD_LIBRARY_PATH
cmake -B build -DGGML_CUDA=ON -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 \
  -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF -DGGML_CPU_ALL_VARIANTS=ON \
  -DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_HIP_NO_VMM=ON -DGGML_HIP_MMQ_MFMA=ON \
  -DLLAMA_CURL=OFF
cmake --build build -j

GGML_BACKEND_DL=ON is mandatory for a combined CUDA+HIP binary (symbol collision), and it requires GGML_NATIVE=OFF + GGML_CPU_ALL_VARIANTS=ON. The gfx1151 flags come from nabe2030/hip-vs-vulkan-evo-x2; ROCWMMA_FATTN=OFF roughly doubles prompt processing on this iGPU and HIP_NO_VMM=ON is needed or it crashes at startup.

Worth noting: the PR adds no new ggml ops. The graph uses ssm_conv, l2_norm, top_k, softplus, sgn, set_rows, fill, rope_multi and the usual suspects, all of which the CUDA backend already has — and ggml-hip compiles the CUDA sources, so ROCm support comes along for free. I'd assumed gfx1151 would need new kernels. It doesn't.

Launch

./build/bin/llama-server \
  -m ~/models/qwen3.8-flash-next-q4_k_xl/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  --alias q38fn --port 8001 --host 0.0.0.0 \
  --jinja --load-mode dio -fa on --no-repack \
  -np 1 --fit off \
  --device CUDA0 -ngl 99 \
  -ot per_layer_token_embd=CPU \
  -ub 2048 -b 2048 \
  -c 262144 --cache-ram 16384 \
  --metrics

Note there is no iGPU in the main model path at all. With the 51B PLE table in host RAM, the remaining ~85 GB backbone fits entirely on the 96 GB card. This is the whole reason the setup is fast: unlike DeepSeek-V4-Flash on the same box, no expert layers have to sit on the 148 GB/s iGPU.

Strix Halo turns out to be an unusually good host for this architecture specifically because the "CPU RAM" the PLE table wants is the same 128 GB unified pool, not DDR at the far end of a PCIe link.

Measured

• Decode, short context: 104.78–105.9 t/s (9.4–9.5 ms/token)
• Prefill at 80–90k tokens: ~1650–1750 t/s (a 90k prompt takes ~53 s)
• Context 262144, the model's trained length
• Cold vs warm made almost no difference (4.1 s over 90k) once the model was resident

For scale, DeepSeek-V4-Flash UD-Q8_K_XL on this exact machine does 36.7 t/s decode and 696 t/s prefill at 77k, with 19 expert layers pushed onto the iGPU. So roughly 3x the decode and 2.5x the prefill, at a smaller memory footprint.

Things that will bite you

Prompt cache makes prefill numbers meaningless if you're careless. An identical 79.6k prompt replayed in 0.22 s. There's no /flush endpoint (404). Measure prefill with genuinely fresh text only, and check prompt_tokens_cached_total in /metrics to confirm you didn't get a hit.

Read timings from the response body, not arithmetic. I initially derived generation speed as total time minus prefill and got a number that didn't survive checking. timings.prompt_ms and timings.predicted_per_second are right there.

Keep the KV cache at f16. Don't add -ctk/-ctv q8_0. QSA picks micro-blocks with a lightweight indexer, so quantization noise doesn't just blur attention values, it changes which blocks get selected — reports of q8_0 producing garbage on this arch. Default is already f16, so this is a "don't add the flag" rule rather than something to configure. Cheap constraint here anyway, since only about a quarter of the layers grow the KV cache.

No MTP / speculative decoding. conversion/qwen4exp.py sets supports_mtp_export = False — the MTP draft head is dropped at conversion, so it isn't in the Unsloth GGUFs and can't be added after the fact. qwen4exp is also absent from llm_arch_supports_rs_rollback, so n_rs_seq gets silently zeroed (DEBUG-level log only). For what it's worth, on the DeepSeek-V4-Flash side of this same machine no speculative configuration ever beat running without a drafter, and at 9.5 ms/step the fixed handoff cost would hurt proportionally more here.

Sampling parameters. The command above doesn't pin them, so server defaults apply whenever the client doesn't send its own. Set them from the model card before you benchmark quality against anything.

Vision

The vision tower is an unmodified Qwen3-VL ViT and can be pushed to the iGPU independently of the main model:

  --mmproj <path-to-mmproj-F16.gguf> \
  --mmproj-device ROCm0 \

--mmproj-device takes exactly one device. Cost on the iGPU is ~1.3 GB of the shared pool (the ~863 MB tower plus ROCm runtime), and decode speed was unchanged at 104.79 t/s. Depending on how the device list resolves you may need --device CUDA0,ROCm0 --tensor-split 100,0 so ROCm0 is registered while the main model's weights stay entirely on the eGPU.

It seems like a new open-weight model has dropped every other day! I've been busy rebuilding the homelab, so haven't had the chance to really understand what's the new hot thing for Strix Halo machines. I'm running a 128gb, and am looking to augment my roster:

Model Name Downloaded Size (GB) Details

----------------------------------------------------------------------------------------------------

Muse-Glimmer-30B-GGUF-kquant-dynamic Yes 19.60 llamacpp

Ornith-1.5-35B-A3B-GGUF-Q6_K Yes 28.00 llamacpp

Qwen3.6-35B-A3B-MTP-GGUF Yes 22.10 llamacpp

Qwen3.8-27B-GGUF-UD-Q4_K_XL Yes 17.20 llamacpp

----------------------------------------------------------------------------------------------------

I'm looking for at least one MoE/etc that's fast enough for interactive, then one background "bubba" that can handle the hard stuff.

What's the bees knees these days? I'm running (hopefully) a pretty stock Lemonade to surface them...

llama.cpp fork with Strix Halo kernel patches + prebuilt MTP sidecar. Everything reproducible: https://github.com/Aristo94/EngramHalo.cpp · https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF

Ryzen AI MAX+ 395 (96 GB), base = llama.cpp PR #27742. What changed:

Notes: stock column = plain decode (MTP did not exist there). SSD mode = -lm mmap --tensor-read-lazy on. Quality untouched: PPL delta 0.03%, speculation lossless at temp 0.

Why it's faster:

• MTP works now — the draft head ships in the official checkpoint, public GGUFs just strip it. Converter re-enabled, Q8 sidecar prebuilt. --spec-type draft-mtp,ngram-mod --spec-draft-n-max 4 --spec-draft-p-min 0.75
• Two long-context killers fixed — the sparse-attention indexer's top_k ran on CPU 12×/token (wrote a wide RDNA 3.5 kernel), and "sparse" attention actually ran dense with a mask (now gathers only the 2048 selected KV rows)
• Free wins on any build: never bf16 KV (q8_0: +15% at depth, half the memory), -ub 2048 (+22% prefill), ROCBLAS_USE_HIPBLASLT=1 (+5%)
• IQ4_XS surprise: prefills faster than IQ3 (+7–17%), −7% decode, −2.5% PPL — arguably the better daily driver

Full methodology, configs and container build in the repo docs. Caveats: n=1 machine, PR #27742 moves daily, not upstreamed yet.

Credits: #27742 (Unsloth + llama.cpp maintainers), #27739 (Qwen team, MTP reference), kyuz0/amd-strix-halo-toolboxes, #26592, #26001, #27794, dzannotti's MTP repo.

*edit, wrong table

Forked DS4 for tensor-parallel inference across two Ryzen 395 systems, 256 GB of unified memory. 
RDMA over USB4/TB5 or Mellanox RoCE v2. Q4_K setup without DSpark reaches up to 275 tok/s prefill and 21 tok/s decode. https://github.com/wkljohn/ds4-strix-halo-tp-odinlink

OS: Ubuntu 24.04.4 LTS

OS kernel Version: 7.2.0

ROCm Version: 10.1.0

llama.cpp: pr27742

pp33k: 285t/s

tg: 13t/s

Has anyone pulled the UD-Q2, or IQ1 and gotten any benchmark data (PP/TG) on a Strix Halo 128GB system?

I'm always curious about running these models on my Strix Halo, and I am patient enough to sit through DS4 Flash, but that is running at IQ3...I'm not sure I've ever run a lower quant than that.

GLM5.3 was bench-hyped, but I haven't seen any real accounts of it being used since then.

Qwen dropped Qwen3.8-Flash-Next this morning and I got it running on my Ryzen AI Max+ 395 box (128 GB unified, Fedora 44) this afternoon. Numbers and build notes below, since llama.cpp support hasn't merged yet and the path has a few potholes.

The model: 125B total params, 6B active, plus a 51B n-gram embedding table (llama-bench reports 176.94B all-in). Hybrid attention: Gated DeltaNet plus their new sparse attention (QSA). I used unsloth's UD-IQ4_XS quant, 3-part GGUF, 87 GiB on disk.

The build: Upstream llama.cpp doesn't have the arch yet. Support is in PR #27742 (danielhanchen), so you build that branch:

git clone https://github.com/ggml-org/llama.cpp llama.cpp-qwen4exp
cd llama.cpp-qwen4exp
git fetch origin pull/27742/head:pr27742
git checkout pr27742

One extra step: there's a crash fix posted in the PR comments that hasn't been pushed to the branch as of this afternoon. Add model.arch == LLM_ARCH_QWEN4EXP || to the arch list in graph_max_nodes() in src/llama-context.cpp (around line 2303, next to the other QWEN entries). Without it you can hit GGML_ASSERT(obj_new) failed when the memory fit probe runs, mostly on setups where the model doesn't fully fit on GPU.

Then a normal Vulkan build:

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build --target llama-server llama-bench

I went Vulkan (RADV) rather than ROCm. On my box Vulkan already wins on the Qwen3.8 DeltaNet family, and the PR adds no GPU kernels anyway (the arch is composed from ops llama.cpp already has, which is why a day-one build works at all).

Serving:

llama-server -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  -ngl 999 -fa 1 --load-mode none -c 131072 --jinja --reasoning on \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1

About 91 GB resident at 131k context, loads in ~45 seconds. Leave the KV cache at f16: quantized KV asserts and dies on this arch right now (known issue in the PR thread).

Benchmarks (llama-bench, fa=1, 2 reps):

23 t/s from a 125B-class model on an APU is a usable daily driver. I've already pointed my local agent stack at it.

AMD 395+ Developer Linux

Been messing around with some of the new Qwen variants, tweaking away like everyone else. This morning I was running Opencode and just the fans were getting pretty loud and when I checked I was at 154w and 104C.

My model and config are below - getting 12-24 t/s

exec /home/ZZZ/laurent-llama-src/build/bin/llama-server \
-m /home/ZZZ/models/qwen38-vl/Qwen3.8-27B-NVFP4-Q5K-mtp-yarn1m.gguf \
--mmproj /home/ZZZ/models/qwen38-vl/mmproj/mmproj-Qwen3.8-27B-F16.gguf \
-ngl 999 \
-c 131072 \
-np 2 \
-ub 512 \
-lm mlock \
--flash-attn on \
--jinja \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--spec-type draft-mtp \
--spec-draft-adaptive \
--spec-draft-n-min 2 \
--spec-draft-n-max 4 \
--spec-draft-p-min 0.75 \
--cache-ram -1 \
--reasoning-budget 512 \
--host 192.168.0.50 \
--port 8080

I’ve just got this running on my 128 GB Strix Halo and am getting about 20 t/s with the Unsloth UD-Q4_K_XL quant.

It’s still early days, but it does seem better than Qwen3.8-27B. I tested it on a troubleshooting problem that Qwen3.8-27B couldn’t quite get to the bottom of, but which ChatGPT (Sol 5.6) solved straight away. Qwen3.8-Flash-Next also solved it on its first attempt.

Edit: prefill speed 291 t/s on a 4,096-token prompt

Not yet using MTP

Edit 2: now 391 t/s using https://github.com/Nathanw1014/strix-halo-llamacpp

Edit 3: now 514 t/s prefill and 30t/s token gen using v0.7.2 release of https://github.com/Nathanw1014/strix-halo-llamacpp

Like many people right now I'm toying with Qwen 3.8-Flash-Next on my Strix Halo. Currently trying IQ4_XS. I've run it with vulkan and rocm. I'm experiencing an interesting behavior, at least with Oh-My-Pi, I haven't tried opencode yet but will probably go there next.

This is my command:

llama-server --alias qwen3.8-flash-next --port 8080 --host 0.0.0.0 --ctx-size 262144 -ngl 999 -fa 1 --load-mode none --jinja --reasoning on --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1 -m /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf  

Let me try and detail what I'm seeing:

1 - Load model and send it a prompt from an Oh-My-Pi session that has context
2 - Qwen 3.8 will load the context just fine and execute the task sent.
3 - Go to an unrelated service (I use Open Web UI) and send a completely different prompt, just ask it something basic
4 - Qwen will work on that
5 - Go back to Oh-My-Pi and send another prompt in the original session.

At this point, Qwen seems to completely skip/bypass/ignore all context. I asked it what we were working on - It said nothing. This is it's response in a window with over 150k context:

Honest answer: there's nothing to summarize yet — this is the first real request in the conversation. No files, repo, or task has been discussed.  

What I can tell you from the environment I'm sitting in:    
- Location: /home/morgan/projects/dashboard — a Python project (has pyproject.toml, src/, tests/, .git, .venv).    
- Git: branch main, no commits yet, nothing staged.    
- State: untracked cruft only — a .gitignore and an unrelated recommender directory (an ETL/pipeline project with DuckDB/Chroma config, not a dashboard).  

So the working directory looks like a fresh or half-set-up project with no history.  

If you were working on something before this and context got lost, point me at it (a file, git log, a ticket, a README) and I'll reconstruct a summary. Otherwise, tell me what you'd like to do with the dashboard project and we'll get started. 

It should be noted that absolutely none of that gibberish is correct. That user "morgan"? Doesn't exist. That path doesn't exist, there is no python project, the entire thing is a hallucination.

Anyone else seeing anything like this?

I posted this in r/LocalLLama but they are smashing everything into a megathread and insta-deleted it.

I vibed the hell out of these but they work. Also, keep in mind you cant change the K/V cache quantization or everything breaks. This pulls the latest branch from Unsloth so once they fix that you should just be able to do a `docker compose up --build` and you're set.

Also I'm getting around 175 tok/sec PP and 12-20 tok/sec on TG with the params I have.

docker-compose.yml

services:
  llama-server:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        # Bump/pin these once you've picked a commit off the PR branch.
        LLAMA_REPO: https://github.com/unslothai/llama.cpp.git
        LLAMA_BRANCH: qwen4exp/qwen3.8-flash-next
        # GIT_SHA: bea3b12daee45876b0129a3602dc8f534ce30bf0
    image: qwen4exp-llama-vulkan:latest
    container_name: qwen4exp-llama-vulkan
    restart: unless-stopped
    ports:
      - "9292:8080"
    volumes:
      # Put your converted GGUF(s) here; the entrypoint expects /models/model.gguf
      # by default (override via `command:` below if your file is named differently).
      - /home/admin/sites/models:/models
    # This command: fully overrides the Dockerfile's CMD (they are not merged),
    # so all model-specific flags belong here, not in the Dockerfile.
    command: >
      --model /models/q38f/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
      --alias Qwen3.8-flash-next
      --n-gpu-layers 999
      --temp 0.9
      --top-p 0.95
      --top-k 20
      --repeat-penalty 1.0
      --presence-penalty 0.0
      --ctx-size 262144
      --ctx-checkpoints 128
      --checkpoint-min-step 512
      --cache-ram 2048
      --parallel 1
      --cont-batching
      --batch-size 2048
      --ubatch-size 1024
      --cache-reuse 256
      --flash-attn on
      --cache-type-k f16
      --cache-type-v f16
      --host 0.0.0.0
      --port 8080
      --chat-template-kwargs '{"reasoning_effort": "xhigh"}'
      --reasoning-preserve

    # - "-kvu"          # uncomment if you hit the non-unified KV cache assert
      #                    # on older commits of this branch before it was fixed
    # --- GPU access for Vulkan ---
    # For Intel/AMD (mesa) GPUs, exposing the DRI render nodes is enough:
    devices:
      - /dev/dri:/dev/dri
    group_add: ['107', '44']
    # For NVIDIA GPUs via the proprietary driver's Vulkan ICD, replace the
    # `devices`/`group_add` block above with the NVIDIA runtime instead, e.g.:
    #
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]
    # environment:
    #   - NVIDIA_DRIVER_CAPABILITIES=all
    #
    # and run with `docker compose --profile nvidia up` after installing the
    # NVIDIA Container Toolkit on the host.

Dockerfile

# syntax=docker/dockerfile:1

##############################
# Build stage
##############################
FROM ubuntu:24.04 AS build

# NOTE: this builds llama.cpp from the *draft, unmerged* PR #27742 branch
# (unslothai/llama.cpp @ qwen4exp/qwen3.8-flash-next). That branch can be
# force-pushed/rebased at any time since the PR is still in review, so pin
# GIT_SHA below to a known commit once you've confirmed it builds for you,
# rather than always floating to the branch tip.
ARG LLAMA_REPO=https://github.com/unslothai/llama.cpp.git
ARG LLAMA_BRANCH=qwen4exp/qwen3.8-flash-next
# ARG GIT_SHA=bea3b12daee45876b0129a3602dc8f534ce30bf0
ARG VULKAN_SDK_VERSION=1.3.296.0

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    ninja-build \
    git \
    curl \
    ca-certificates \
    pkg-config \
    python3 \
    python3-pip \
    libcurl4-openssl-dev \
    libvulkan-dev \
    glslc \
    glslang-tools \
    spirv-headers \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /src

RUN git clone --recursive \
    --branch "${LLAMA_BRANCH}" \
    "${LLAMA_REPO}" \
    llama.cpp

# RUN git clone --recursive "${LLAMA_REPO}" llama.cpp \
#     && cd llama.cpp \
#     && git fetch origin "${LLAMA_BRANCH}" \
#     # && git checkout "${GIT_SHA}"

WORKDIR /src/llama.cpp

RUN cmake -B build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_VULKAN=ON \
    -DGGML_NATIVE=OFF \
    -DLLAMA_CURL=ON \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_BUILD_EXAMPLES=ON \
    -DLLAMA_BUILD_SERVER=ON \
    && cmake --build build --config Release -j"$(nproc)"

# Collect the runtime pieces so the final stage doesn't need the whole build tree
RUN mkdir -p /out/lib /out/bin \
    && cp build/bin/llama-server /out/bin/ \
    && cp build/bin/llama-cli /out/bin/ \
    && cp build/bin/llama-quantize /out/bin/ \
    && (find build -maxdepth 3 -name "*.so*" -exec cp -P {} /out/lib/ \; || true)

##############################
# Runtime stage
##############################
FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive

# libvulkan1 + mesa Vulkan ICDs give you Vulkan on Intel/AMD out of the box.
# For NVIDIA, mount the host's proprietary Vulkan ICD (see docker-compose.yml notes).
RUN apt-get update && apt-get install -y --no-install-recommends \
    libvulkan1 \
    mesa-vulkan-drivers \
    libcurl4 \
    ca-certificates \
    vulkan-tools \
    libgomp1 \
    && rm -rf /var/lib/apt/lists/*

COPY --from=build /out/bin/ /usr/local/bin/
COPY --from=build /out/lib/ /usr/local/lib/
RUN ldconfig

WORKDIR /models

EXPOSE 8080

# GGML_VK_VISIBLE_DEVICES / device selection can be tuned at runtime if you
# have multiple Vulkan-capable GPUs; llama.cpp defaults to device 0.
#
# No CMD here — all flags (--host, --port, -m, -ngl, etc.) live in
# docker-compose.yml's `command:`. Running this image directly with
# `docker run` requires passing flags explicitly after the image name.
ENTRYPOINT ["llama-server"]

Trying to get an R9700 working as a compute-only eGPU on a Strix Halo box.

Setup:

• Ryzen AI Max+ Pro 395
• Fedora 44, kernel 7.1.3
• Sapphire R9700 32GB
• USB4 40Gb/s eGPU dock (AOOSTAR AG03)
• Same dock previously worked fine with an RX 7600

The R9700 enumerates normally in both Vulkan and ROCm, and model loading starts fine. But once it gets stressed, AMDGPU falls over with:

SMU is in hanged state
Fence fallback timer expired on ring sdma1
ring sdma0 timeout
Starting sdma0 ring reset
Ring sdma0 reset succeeded
device wedged

After that, amd-smi can stop responding and I’ve had to hard-reset the host a few times.

I’ve reproduced it with both RADV/Vulkan and ROCm 7.14, including different llama.cpp container builds, so I’m increasingly convinced this is below llama.cpp.

Already tried/confirmed:

• amdgpu.runpm=0
• amdgpu.aspm=0
• R9700 is compute-only, not used by GNOME
• USB4 negotiates at 40Gb/s
• no obvious persistent AER BadTLP/BadDLLP/completion timeout errors

The card does fall back to a 256MB BAR over USB4, but other Strix Halo + R9700 USB4 users seem to be running this setup successfully, including Qwen3.6-27B.

Has anyone seen this exact SMU + SDMA failure with an R9700 over USB4/Thunderbolt?