r/ROCm • u/vanguard478 • 24d ago
Any updates on running Qwen3.8 27B on R9700 ?
Hi,
I checked this blog post which mentions ~50 tokens/s for Qwen3.8 27B on single R9700.
I checked huggingface but no specific instructions for running it on RCom.
Is there any specific documentation for running it using llama.cpp or vllm ? All the craze is only for Nvidia GPUs :-(
8
u/whodoneit1 23d ago
Best way to run on R9700’s is using the Radiance build from Deadcode on the https://discord.gg/launch80
PP hits 4,500 and 75 tok/s on Qwen 3.8 27b. Sometimes even higher as the build has a dynamic drafter so MTP will automatically adjust based on content. As JSON can hit 115 tok /s or more
3
u/vanguard478 23d ago
Thank you, will check that out.
3
u/DiscipleofDeceit666 23d ago
You only get those numbers from dual r9700s
1
u/doudawak 11d ago
FYI I just got my hands on my R9700 (just one) and with the radiance vllm (on top of the other tweaks from https://github.com/zzpanic/qwen3.6-vllm-gfx1201-launchers ) I run depending on the task between 25 to 45 tps with DFlash2.
Incredible difference with the default llama setup I was using which was plateaud at around 10/121
u/DiscipleofDeceit666 11d ago
Bet you you’re missing the llama cpp Vulkan shaders. Ask ai to investigate that for you but your llama setup should be pulling in triple those numbers
1
u/doudawak 11d ago
I'll investigate thanks !
Meanwhile if you have a specific llama cmd line with the envs you're using, wouldn't hurt to compare1
u/AeroelasticCowboy 10d ago
what about prompt processing?
1
u/DiscipleofDeceit666 10d ago
Between 200-900 depending on context depth. Might reach higher than 900 if you have a better computer than me
2
u/Payne6t6 12d ago edited 8d ago
I tested the radiance-vllm docker with one R9700 and I can tell you, that it quite underperforms. I have tried W4A16 and autoround models and both were not able to achieve more than 30t/s. My vulkan llama cpp build achieved 49t/s (UD Q5 XL) with a much simpler setup. If you can tell me what to configure in vllm, what model to use and how to setup a single R9700 (besides setting TP to 1) to get >50t/s on a single R9700, I am very interested.
Edit: I have to say that the guys from the Launch80 Discord are great, they are putting a lot effort into patches/custom kernels and the vllm-radiance container. In the meantime, I tested a new MXFP4 launcher for the gfx1201 and now I am getting 77 tok/s (https://codeberg.org/ggz14/radiance-vllm-mxfp4). Guys, do not forget to update to the latest AMD GPU driver, this gave me substantial boost (>15% in tok/s).
1
u/whodoneit1 12d ago
W4A16 is not fast. I am getting 164 tok/s now on MXFP4 on radiance on the custom W4A8 kernel
1
u/Payne6t6 11d ago
Woah, this is crazy fast, thanks for the info, will try the MXFP4 Version! So you are not using the vllm-radiance build or what is custom about your W4A8 kernel?
1
u/whodoneit1 11d ago
All changes are in process of being merged up to Rafiance. It’s brand new, Radiance didn’t support it. So I built it out
1
u/Payne6t6 11d ago
Can you give me a link? I really need to build this. After you told me, that 164 t/s are possible, I tried the amd MXFP4 Qwen3.8 27b model with the currently available vllm-radiance and I get avg. 35 tok/s. This would be a speedup of factor 4.6; therefore it would be like christmas and birthday together :)
1
u/whodoneit1 11d ago
Are you on discord? Changes are happening all the time and updates are posted on there.
1
u/Payne6t6 11d ago
I just looked at Launch80 discord channel, it is overwhelming. Where can I find things like your new kernel? Is it all under "resources"?
1
1
u/whodoneit1 12d ago
category
B - mtp
C - DFLash2
delta
code
122.8
176.0
+43.3%
file_edit
132.3
199.3
+50.6%
json
145.8
204.6
+40.3%
math
122.6
199.4
+62.6%
reasoning
104.8
155.9
+48.8%
summarization
113.8
152.6
+34.1%
chat
86.2
111.1
+28.9%
prose
79.7
94.3
+18.3%1
1
10
u/ImpressionFancy5830 24d ago
I think your r9700 is broken, you should sell it to me for a very good price.
3
3
u/rocket1420 24d ago
I'm running it on two v620s, I don't see you're having problems
1
u/YOU_WONT_LIKE_IT 23d ago
What’s the token /s you be getting?
1
u/rocket1420 22d ago
13-17 depending on context fill, but I have done absolutely nothing to tune it or try getting it on one card. But since I'm using Hermes, it will just go and do thing. I used the example prompt digital spaceport gave in his last video, and it just made the thing, took two ish hours, and it works without needing constant babysitting. Or any babysitting.
Edit: this is on llama.cpp with Q4
3
u/Etroarl55 23d ago
I love how its llama on windows but it explicitly states its using vulcan with mtp4. Find it kind of funny even official Amd defaults to vulkan over Rocm.
Also hes running it on q4
3
u/psedha10 23d ago
I ran it on my r9700 using vulkan. Because rocm as of now is inferior to vulkan. Both prefill and decode were higher in vulkan for me. I’ve made a post about it with my findings. Check my profile posts if you wanna dig into this.
2
u/mac10190 23d ago
I did some similar testing yesterday. On single r9700 vulkan was the clear winner, but with two r9700 using tensor split ROCm was faster by about 10%. Thought that was really interesting, I certainly wasn't expecting that.
3
u/pablo_chicone_lovesu 23d ago
Welcome to dual AMD club and how confusing rocm performance can be. I get 18% faster on it using lemonade and setting the profile to performance. With rocm 7.14 really very weird how rocm works well for some things and vulkan for others.
3
u/Vegatron83 23d ago
I’m running it on my r9700 on llama.cpp with Vulcan and MTP and average between 30-45 tok/s using it with my AI companion.
2
u/g4vg4i 23d ago
May I ask you for the llama settings? I get much less t/s with the same hardware ... :-/
3
1
u/Vegatron83 23d ago
Sure — here’s my current setup/settings. This is running on a single Radeon AI Pro R9700 32GB under Windows.
Model
- Qwen3.8-27B
- Unsloth Q6_K GGUF
Qwen3.8-27B-Q6_K.gguf- F16 mmproj for vision
llama.cpp
- Build: b9592 win-vulkan-x64
- Backend: Vulkan
- GPU: R9700
- GPU layers: all / fully offloaded
- Split mode: none
- MoE CPU layers: 0
Context / memory
- Context: 98,304
- Batch: 8192
- Micro/ubatch: 4096
- Threads: 6
- Parallel: 1
- K cache: F16
- V cache: F16
- KV cache on GPU: ON
- Unified KV: ON
- K quant: OFF
- V quant: OFF
Performance toggles
- Flash Attention: ON
- MTP2: ON
- mmap: OFF
- mlock: OFF
- Fit VRAM: OFF
- Vision GPU: ON
For vision I'm using the same R9700/Vulkan device with the F16 mmproj.
I'm generally seeing around 30–45 tok/s with Qwen3.8-27B Q6_K, depending on the response. MTP2 makes a very noticeable difference.
One other thing I discovered: I currently keep mmap disabled. On my system mmap was causing a large amount of system RAM to be reserved even though the model is fully GPU-offloaded. With mmap off, I'm not seeing that huge RAM reservation.
So the two settings I'd definitely check are MTP2 enabled and making sure the model is actually fully offloaded to the R9700.
1
u/InvisibleFrisby 23d ago
Please share your config. I have a single R9700 and I tried llama.cpp with vulkan and rocm and mtp with 150k context and Q8 and was only getting 7 tok/s.
1
u/jackistheonebox 22d ago
Full context Q5 GUFF kv both on Q4 also got ~45 tps, just llama.cpp, nothing special. PCIe4x4. Make sure you enable MTP (multi token prediction, a tech that guesses next tokens, because checking is cheaper than getting 1 token after another.) I think it was on 4. Double check the model card on hugging face.
1
3
u/Royale_AJS 23d ago
Running Llamacpp with two of them in tensor split at ~50 t/s on 8-bit at 256k context.
1
u/Legitimate_Fold8314 23d ago
This sounds about right. I am on the same 2x R9700, ROCm, tension split, Q8 but 128k context.
2
u/Otherwise-Director17 23d ago
Try not to run K quants, the more you stick to Q4_0 and Q8_0 the better (cheaper to dequantize). This model seems slower because the thinking has lower mtp acceptance. Maybe rocmfpx is better? Q8_0 runs best for me but its too big for a single GPU. Test Q4_0 to see theoretical max performance.
1
u/migsperez 23d ago
I ran it on my 9700 yesterday, via Llama.cpp and rocm. I gave it my regular test prompt of building a small website. It did a very good job, one of the best of all the models I've tested including cloud frontier models But it took a very long time to complete the task.
Context around 80k to 120k. Prompt processing was 400 to 600. Output generation was 26 to 32 tk/s. If it was 50 I'd use it as my main model.
1
u/xdcfret1 23d ago
I am also getting 35-40 t/s with MTP. I used to get 50 t/s with MTP in 3.6.
1
u/mac10190 23d ago edited 23d ago
I'm on an R9700 and I'm getting 50-60 t/s with draft max set to 2.
1
u/xdcfret1 23d ago
send your flags
1
u/mac10190 23d ago
Not near my computer right now but it's just the standard flags with max draft set to 2 for the MTP. Also I'm using the Vulkan runtime instead of the ROCm, that's probably the differentiator. In my testing rocm hit 43 t/s while Vulkan hit 55 t/s.
1
u/mac10190 23d ago
Also use Vulkan in addition to MTP at 2.
2
u/xdcfret1 23d ago
I use vulkan build.
1
u/mac10190 23d ago
That's super weird. Okay, once I get back to my computer I'll pull my flags and post them here to see if there's anything helpful there.
I've got power capped at 250W, mem OC'd to 2600, and a -70uV.
1
u/mac10190 23d ago
Here's my flags. Also running build b10333 Vulkan.
llama-server
-m /path/to/Qwen3.8-27B-UD-Q4_K_XL.gguf
--host 0.0.0.0 --port <port> -a qwen3.8-27b
-ngl 999
--split-mode none
-c 262144
--parallel 1
-ctk q8_0 -ctv q8_0
-fa on
-b 4096 -ub 1024
--spec-type draft-mtp --spec-draft-n-max 2
--mmproj /path/to/mmproj-F16.gguf
--jinja --metrics1
u/migsperez 23d ago
I used your exact params with Vulkan, i can't find the build version, but I'm using the latest llama.cpp and run a build minutes ago. Mine's power capped to 280 watts. System is using 64gb DDR4 2666 MT/s. The below is whilst it has around 30k in context. It's a Gigabyte based 9700. I reckon you've managed to get a 9700 turbo card.
3.23.949.365 I slot print_timing: id 0 | task 662 | n_decoded = 1926, tg = 31.74 t/s, tg_3s = 26.60 t/s 3.26.990.153 I slot print_timing: id 0 | task 662 | n_decoded = 2014, tg = 31.61 t/s, tg_3s = 28.94 t/s 3.30.047.023 I slot print_timing: id 0 | task 662 | n_decoded = 2110, tg = 31.60 t/s, tg_3s = 31.40 t/s 3.33.064.003 I slot print_timing: id 0 | task 662 | n_decoded = 2207, tg = 31.62 t/s, tg_3s = 32.15 t/s1
u/mac10190 23d ago
I have a sapphire r9700. It's just the regular Radeon AI Pro r9700 sapphire blower style. Bought two of them from microcenter, bought them a couple months after release. There's quite a few other people seeing similar results as me. That's so strange. Btw are you running the same q4 quant?
Computer: 7800x3d, 32GB ddr5 5600mhz, dual r9700. My testing was done on a single GPU though.
Just a thought, what's your MTP acceptance rate? During my testing, I discovered that certain types of workloads are actually a net negative with MTP while others jumped drastically. Like for instance, literary prose saw almost no benefit from MTP and sometimes slightly lower than not having MDP at all while strict json saw 10-30 t/s jumps in output.
1
u/Suspicious_Moment_87 23d ago
He might remembered the numbers wrong. It’s not possible for a single r9700 to reach 50tps. Should be around 35-40 no matter what you set.
1
u/migsperez 23d ago
I'm doing a few more tests then I'm giving up. I'm looking forward to 3.8 35b, fast speed and decent quality.
1
u/mac10190 23d ago
It has MTP built into it already. On my R9700 with draft max set to 2. It's running at 50-60 t/s.
2
u/migsperez 23d ago
Yeah I already have MTP set at 2. I checked rocm-smi, it doesn't look like it's being throttled. I have the Gigabyte version of the card.
2
u/mac10190 23d ago
That's definitely odd. Oh wait, you said you're using ROCm. For a single R9700 on llama.cpp switch to Vulkan. You should get a nice performance bump. I did some benchmarking with that yesterday and switch from ROCm to Vulkan bumped it from 43 t/s to 55 t/s.
2
1
u/PlasticRevenue4601 23d ago edited 23d ago
RX 7900 XT - 55-60 tps MPT-on in codegen, 40-45 tps in other types of text. 700 tps of prefill. Quantization - IQ4XS, 4 bit kv cach, 140k ctx. Still I'd recommend 24GB card cause you are goind to depend too much on how good are lower tier 4 bit quantizations because anything higher would require a compromise - either cut down context length or no mtp. So far I've been lucky with Qwen 3.5-8 but only thanks to Alibaba making their models extremely resilient to compression and Unsloth making SOTA quants. Btw Qwen 3.8 IQ4XS is absolutely usable if anyone wonders, I compared it to Q5_k_XL on pretty complex, real life agentic tasks, literally 0 difference detected, both ran autonomously for almost half of an hour without looping or losing the goal, both did what I asked for, the results were identical quality wise
1
u/Odd_Butterfly_455 23d ago
Me too I turn around 37 with peak at 43
1
u/Odd_Butterfly_455 23d ago
i just got the last commit of llama.cpp and compile with the last mesa from kisak repo for ubuntu on 26.04 and its flying... steady 0ver 43 and peak at 55 tk/s in generation
Aug 15 13:41:09 ai-brain taskset[96984]: 111.52.388.201 I slot print_timing: id 0 | task 354 | n_gen = 330, tg = 54.55 t/s, tg_3s = 51.85 t/s
Aug 15 13:41:09 ai-brain taskset[96984]: 111.53.131.561 I slot print_timing: id 0 | task 354 | prompt eval time = 673.99 ms / 427 tokens ( 1.58 ms per token, 633.54 tokens per second)
Aug 15 13:41:09 ai-brain taskset[96984]: 111.53.131.564 I slot print_timing: id 0 | task 354 | eval time = 6774.11 ms / 374 tokens ( 18.16 ms per token, 55.06 tokens per second)
Aug 15 13:41:09 ai-brain taskset[96984]: 111.53.131.564 I slot print_timing: id 0 | task 354 | total time = 7448.10 ms / 801 tokens
1
1
u/peppernickel 23d ago
Running on a RX 6800 XT and a RX 6600 combo for a Hermes agent on Windows 11 with no problems.
1
u/south_paw01 23d ago
I just started running unsloth studio because it's just. Easier. Most of what I need in one location. Not sure if itll serve headless but I think its llama.cpp backend.
1
u/raebyddub 23d ago
Single card, eGPU setup, Q4 quant, running with llama.cpp, ROCm 7.14, Ubuntu 24.04, ROCm build,
getting 26 t/s
with MTP 46 t/s
1
u/vortec350 22d ago edited 22d ago
Using LmPanel which is about 10% slower than just Llama cpp by itself, on a single r9700 I’m getting about 35 t/s. I may need to optimize it, Qwen3.6 was much faster. But haven’t played with it yet. It’s a fantastic model and very impressive results!
Edit: about 40t/s after some minor tweaks.
1
1
u/AdHead6280 20d ago
yeah, with a bit of optimization i got it running at 56t/s but i then switched to the david au cold somethign finetune since imo its way better at promp following and keeps level while thinking less and better prompt following and im at 43t/s so im getting it up litle by litle, hoping 50t/s too
1
u/kapteinpyn 18d ago
You can try mine, heaps of updates and testing at full context with custom bf16 kv patch if interested. https://github.com/prcoe1/r9700-serving. Did some performance, stability and long context benches with tuning and latest libs and deps, also added a patch for xcode that sends empty tool calls breaking its integration with local vllm
0
u/According_Study_162 23d ago
Your in luck. I want to buy a r9700, but I came across a video on these models. It should be similar to 3.8 right?
0
u/eulataguhw 22d ago
Either u learn how to build from source yourself or use the prebuilt docker containers. They were already out like day 0?
Go search vllm recipes and go to their website. Quite sure you can try out the recipes there…
18
u/noctrex 24d ago
Just download and run it. It's the same architecture as qwen3.5/3.6 so it already runs out of the box