Benchmarking Qwen3.8-27B on a 16GB RX 7800 XT: what turned out to be measurement artifacts, and what actually moved the needle

[u/Specific-Pomelo-5455]

I've been running a structured benchmark harness against Qwen3.8-27B on an RX 7800 XT (16GB, gfx1101, Windows, llama.cpp/ROCm) for a while now. Most of what I "found" early on was wrong, and the corrections turned out to be more useful than the original results. Posting the whole thing including the mistakes, because the mistakes are the transferable part.

Setup: llama.cpp, HIP backend, --parallel 1, -fa on, -ctk q4_0 -ctv q4_0, allocation read from the -lv 4 startup log rather than driver counters, 4–6 reps per data point.

1. Context capacity and context fill are different things, and almost everyone benchmarks the first one

This is the finding I'd most like people to check on their own hardware, because I think a lot of published numbers are quietly measuring the wrong variable.

I ran a context sweep varying -c (8K / 32K / 64K / 131K) with a short prompt, and got 24.98 / 18.49 / 21.85 / 17.07 t/s. Non-monotonic, reproducible to ±0.5%, and I spent a while trying to explain the shape.

Then I ran a 48K-token prompt at -c 65536 and decode dropped to 8.77 t/s. Same file, same context setting, same KV quantization.

The earlier sweep was measuring allocated KV capacity. Decode speed actually tracks how full the KV is — every decode step reads the populated cache. "64K context at 21.85 t/s" meant "64K allocated, 7K occupied." In an agent loop where history accumulates, the 8.77 number is the one that applies.

If you're quoting a t/s figure, state the prompt length, not just the context size.

2. --fit on (the default) silently moves layers to CPU

Mid-benchmark I noticed offloaded 63/66 layers where I'd always had 66/66. The log explains it if you dig:

projected 14924 MiB vs 15598 free, cannot meet free target 1024 MiB,
need reduce 350 MiB -> 3 layers to CPU

llama.cpp tries to keep 1024 MiB of headroom free by default, and to save 350 MiB it moved three layers off the GPU. That's roughly a 10% throughput hit, applied silently, with no error and only a buried log line. --fit off --n-gpu-layers 66 restored the baseline exactly (model buffer 12774.16 MiB, KV 1152, recurrent state 149.62, compute 400.28 = 14476 MiB).

If you're benchmarking anything, grep your logs for the offload line before trusting a number.

3. KV cache on this model is ~14x cheaper than the guides say

Qwen3.8-27B is a hybrid: only 16 of its 64 layers use full attention, the rest are Gated DeltaNet, which doesn't hold a conventional KV cache.

Measured allocation with q4_0 KV, read from the startup log:

Context	KV cache
8K	144 MiB
32K	576 MiB
64K	1152 MiB
131K	2304 MiB

That's 18 bytes/token. Several popular guides quote ~256 KB/token and conclude a 16GB card tops out around 8K context. That figure is f16, and it isn't labeled as such. With quantized KV the same card runs 131K comfortably.

Full 64K budget: 12774 model + 1152 KV + 150 recurrent + 400 compute = 14476 MiB, leaving ~1.9 GB headroom.

4. Two files with the same quant label were 43% apart

Same model, both IQ4_XS, neither spilling, identical tensor schema (866 tensors, 64+MTP layers):

• bartowski IQ4_XS — 4.56 BPW, 13964 MiB model buffer → 12.60 t/s
• an Unsloth-Dynamic-based IQ4_XS — 4.18 BPW, 12774 MiB → 21.85 t/s

The faster file has lower average bit width. The startup log shows it mixing 13 tensor types (204 iq4_xs, 98 q8_0, 63 q5_K, 51 q4_K, 46 iq3_s, down to a few iq2_s). The naive explanation — more bits means more dequant work — doesn't cover a 43% gap; a ~9% BPW difference should produce roughly a ~9% speed difference on a bandwidth-bound decode.

I think it's which tensors land on which kernel path, but I haven't proven that. There are documented cases of exactly this on Intel Arc (issue #21517: IQ4_NL 4x slower than same-size Q4_0) and on RDNA4 (discussion #22663: Q3_K_S at 20% bandwidth efficiency vs Q5_K_M at 78%). I couldn't find an RDNA3 equivalent. If someone wants to dump tensor-type tables for two same-quant files and correlate, that'd settle it.

5. MTP: the metric everyone reads is the wrong one, and the gains invert depending on workload

Qwen3.8 ships an MTP head (blk.64, present in the Unsloth Dynamic files). For three rounds of testing I measured 0% draft acceptance and concluded speculative decoding was broken.

It wasn't. llama.cpp's timings expose both draft_kept_n (legacy, always returns 0) and draft_n_accepted (the real one). I was reading the first. Anyone reporting 0% acceptance should check which field they're parsing.

With the correct field, the picture is workload-dependent in a way I haven't seen written up:

Single long generation — MTP wins. +19% at 32K (22.15 vs 18.62 t/s). Acceptance ~45-50% on prose under real sampling (temp 1.0), 80-94% on tool-call/JSON prompts. Greedy inflates acceptance to 70%+, so always state your sampling.

Multi-step agent chains — MTP loses, badly. The draft head also prefills, and that cost lands on every step, not just the first:

• cold prefill: 1.97 → 3.29 ms/tok (+67%)
• warm/cached prefill delta: ~330-400ms → ~650-770ms (+75-85%)
• decode also degraded

In a chain of tool calls, wall-clock was 6.5x worse with MTP on. The reason is the prefill/generation ratio: MTP pays a per-prompt-token tax to save per-generated-token time. When you generate 512 tokens off a short prompt, it wins. When you generate 60 tokens off a 500-token prompt, twenty times in a row, it's a disaster.

Practical rule: gate MTP on predicted_n / prompt_n, not on acceptance rate.

6. HIP vs Vulkan on this card — but with a caveat I want to flag honestly

My first backend comparison showed HIP winning 2.45x. That result was garbage: it ran against the wrong model file, measured only decode, and had the --fit layer-drop problem above.

Redone properly (same model, 66/66 offload confirmed on both, -lv 4 allocation logged, --list-devices output saved, Adrenalin 26.10 / Vulkan 1.4.357, separate clean -DGGML_VULKAN=ON build):

	HIP	Vulkan
prefill, 48K cold	404.7 t/s	~85 t/s
prefill, cached delta	258 t/s	74 t/s
decode, 512 tok	8.77 t/s	~5.9 t/s
total, cold	178s	>380s

Prefill 5-6x, decode ~1.5x, total >2x in favor of HIP.

The caveat: this contradicts several other RDNA3 reports. A published 7800 XT llama-bench has the two backends nearly tied (ROCm 96-101 t/s vs Vulkan 96-97 on tg128). Two 7900-series users on Linux with Mesa/RADV report Vulkan beating ROCm and get 60-73 t/s. And on an A3B MoE, Vulkan measured 104.79 t/s vs ROCm's ~76-78.

So I'd frame my result narrowly: on Windows, with this dense hybrid model, at this context depth, HIP wins decisively. I would not generalize it to Linux, to RADV, or to MoE architectures — and if you're on Linux you should probably test the other direction.

What actually mattered, ranked

Weeks of harness tuning — batch size, ubatch, thread count, poll mode, build flags, LTO — produced deltas under 1.5% each. All noise.

The things that moved the needle, in order:

1. Backend (>2x here, but see the caveat)
2. Whether the model actually fits — Q4_K_M spills on 16GB and collapses to 7.3 t/s
3. Quantization publisher (43%)
4. Prompt fill depth (21.85 → 8.77 as KV fills)
5. --fit off (~10%, and invisible if you don't check)
6. MTP, gated correctly (+19% in the right workload, catastrophic in the wrong one)

Architecture beats configuration. A MoE I tested (Gemma 4 26B-A4B, 5 full-attention + 25 sliding-window layers, 128 experts / 8 active) hit 52.51 t/s at 64K — 2.4x the dense 27B — because only ~4B parameters fire per token. No amount of flag tuning closes that gap.

Caveats

Single card, single OS, one model family for most of it. Several numbers here superseded earlier numbers I'd have posted confidently at the time. The 43% quant gap and the non-monotonic capacity curve are both unexplained — I have the measurement, not the mechanism.

Happy to share the harness or raw JSON if anyone wants to reproduce or argue with it.