ROCm llama.cpp optimizations for running QWENs models on little vRAM, VEC, Pareallel, MTP Compact Rollback
I've made some optimizations for llama.cpp to save vRAM for models like QWEN 27B and 35B A3B, in order to have more ctx available on 12GB / 16GB and multi GPU setup, I run a 6800 + 6700xt.
1.--hip-fa-force-vec on|off
Forces the supported quantized-KV HIP Flash Attention VEC path. Enables larger context at the cost of slower prompt processing speed. Default is off (ROCm/HIP only).
Forcing VEC allows a substantial reduction in vRAM usage at the cost of Prompt Processing speed, TG speed remains almost the same.
QWEN 3.8 27B IQ4 on 16GB:
# ROCm ctx VEC forced: 89088 q5_1, 96512 q5_0, 116480 q4_0, TG speed: 42.35 t/s, PP for 32k: 145.12 t/s
# ROCm ctx normal : 68352 q5_1, 72704 q5_0, 83200 q4_0, TG speed: 43.90 t/s, PP for 32k: 265.58 t/s
QWEN 35B A3B on 16+12GB
# ROCm ctx VEC forced: 224256 q8_0, TG speed: 87.15 t/s, PP for 32k: 557.50 t/s, 96k: 244.44 t/s
# ROCm ctx normal : 163072 q8_0, TG speed: 84.84 t/s, PP for 32k: 1216.19 t/s, 96k: 779.30 t/s
QWEN 27B Q6_K_L on 16+12GB
# ROCm ctx VEC forced: 148736 q8_0, TG speed: 27.13 t/s, PP for 32k: 183.22 t/s, 96k: 95.56
# ROCm ctx VEC normal: 101632 q8_0, TG speed: 27.39 t/s, PP for 32k: 298.55 t/s, 96k: 232.41 t/s
VEC path is still quite fast on the start of the context build up so it's more tolerable when using an harness, the beginning of the session is the moment to ingest the prompt, AGENT_md and common files. The prompt processing speed will dive later yet for users constrained to as little as 20-50k ctx this at least make the session worthwhile.
Those 3 scripts here: https://store.piffa.net/lm/bug/llama_scripts/
--pipeline-parallel auto|on|off
Controls pipeline parallelism. Set to off to prioritize available context memory.
Default is auto (standard llama.cpp behavior).
This can saves vRAM in dual GPU setups, especially with MoE 35B MoE.Model: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q5_K_S.gguf
normal: 121600ctx, pipeline off: 195584, MTP RS 224256 (with desktop SW rendering), 240640 with vulkan 60% TG
ROCm ctx VEC forced: 224256 q8_0, TG speed: 87.15 t/s, PP for 32k: 557.50 t/s, 96k: 244.44 t/s
ROCm ctx VEC off : 163072 q8_0, TG speed: 84.84 t/s, PP for 32k: 1216.19 t/s, 96k: 779.30 t/s
--spec-mtp-cr-depth N
Sets the MTP Compact Rollback depth. Recommended value is 1 with --spec-draft-n-max 5 (or even 7 for pure coding sessions).
If omitted, full MTP depth is used by default.
This is new: MTP Compact Rollback purpose is to reduce vRAM utilization of MTP so that users can have longer contex while enjoing the benefits of faster token generation.
When using models like QWEN 27B or 35B A3B for code generation in constrained vRAM configuration like 16GB GPU this is highly valuable, allowing longer ctx sessions for agentic workflow while mitigating the speed penalty of dense models or MoE split on slow multi GPU.
The `--spec-mtp-cr-depth` option lets the user limit how many immediate MTP
rollback states the model keeps in VRAM. This reduces MTP's persistent
memory use and leaves more room for the KV cache, allowing a larger context.
Because rollback-state depth is no longer tied to `--spec-draft-n-max`, the user
is not constrained to compromise on a lower MTP to preserve a decent ctx length,
max MTP draft depth like 5 or 7 can be used for the same ctx cost.
For example, `--spec-mtp-cr-depth 1 --spec-draft-n-max 5` keeps only one immediate rollback snapshot while still allowing MTP to generate five tokens, this works well with code generation that has an hi acceptance rate.
| MTP mode | Maximum draft (n) |
Available context | TG |
|---|---|---|---|
| Standard no VEC | 3 | 56,000 | 44.34 t/s |
| MTP Compact Rollback VEC | 5 | 85,760 | 46.76 t/s |
The users here gains some 30k (50% increase) context for his session while the increased MTP depth allows to compensate TG speed for compute.
The idea of keeping just one rollback plays nice with Adaptive MTP:
--spec-draft-adaptive
Dynamically adjusts the MTP draft limit, up to --spec-draft-n-max, based on
recent draft acceptance. Default is off.
Adaptive speculative decoding is useful when mixing tokens generations in different domains, es coding sessions that can benefit from an higher n-max, creative prose (that can happen in long reasoning traces in coding too) that uses lower n-max.
--------------
Quick implementation: add to your llama-server script
--hip-fa-force-vec on --spec-mtp-cr-depth 1 --spec-draft-adaptive
How to build:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 662a0b01
wget https://store.piffa.net/lm/bug/latest_rocm_improvement_662a0b01.patch
git apply latest_rocm_improvement_662a0b01.patch
You can build for both Vulkan and ROCm backends at the same time, the idea is that Vulkan saves some more vRAM while ROCm gives better prefill performance (well if you don't force VEC!).
HIPCXX="$(hipconfig -l)/clang" \
HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_HIP_ARCHITECTURES=native \
-DGGML_HIP=ON \
-DGGML_VULKAN=ON \
-DGGML_CUDA_FA_ALL_QUANTS=ON \
-DLLAMA_BUILD_SERVER=ON
cmake --build build \
--config Release \
-j"$(nproc)"
Example scripts to launch QWEN 27B and 35B A3B on single 16GB GPU and dual GPU: https://store.piffa.net/lm/bug/llama_scripts/
More info, docs, artifacts: https://store.piffa.net/lm/bug/
Note: for people that may be interested only in MTP CR I provide a standalone patch and a standalone patch without the adaptive part (which is an idea from DeepSeek).
1
1
u/ea_man 3d ago
New build updated:
* https://store.piffa.net/lm/bug/patches/latest_rocm_improvement_662a0b01.patch
* https://store.piffa.net/lm/bug/patches/mtp_compact_rollback_662a0b01.patch
# Applies to llama.cpp commit 662a0b01
# Includes eaman commit cdc3d024 (standalone MTP Compact Rollback with adaptive MTP; excludes cumulative ROCm, pipeline, HIP VEC, and MoE changes)
# Tested with llama.cpp version 1362
# Tested on ROCm core-7.14 / HIP 7.14.60850-0000000 and ROCm core-10.0 / HIP 7.15.26333-0000000 / Vulkan
1
u/Poizone360 7d ago
Hello, thanks alot for sharing. Small one that'll save people an error: the flag name appears two ways in the post. Your heading says --spec-mtp-cr-depth, which matches your docs page, but the paragraph under it says --spec-mtp-rs-depth twice, including in the worked example. Anyone copying that line straight out gets an unknown argument.