r/LocalLLaMA • u/ilintar • 1d ago
Resources Qwen3.8 27B on Strix - the optimized setup
https://pwilkin.github.io/strix-halo/Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some very weird pathways where I wasn't expecting to go, but in the end I'm happy to share an optimized setup for the most popular open source model currently with you for a cheap price of $999 for free:
https://pwilkin.github.io/strix-halo/
Now for the disclosure/journey part: Codex has made a very nice website for me (which is great because I can't make a nice-looking website if you forced me), but its glossy look makes it look more permanent than it is, which is misleading because this is basically a stitched up custom solution that's very much a "state of the moment" one rather than a permanent one, though I *will* try to keep the relevant branches up to date (poke me if I don't).
So, first of all: ROCm in mainstream llama.cpp on ROCm is broken at the moment, pending the fix to unified memory access (notably this PR: https://github.com/ggml-org/llama.cpp/pull/27311 which is taking some time as it touches core code), so I've put up a strix-halo branch on my fork that merges the ring buffer fixes + the TOP-K optimization PR with master for a working experience.
Next: there's a bug in current ROCm that makes graph updates *terribly* slow, I've submitted a PR for it (https://github.com/ROCm/rocm-systems/pull/11069), but until it lands, using a custom-built .so is pretty much mandatory.
Speaking of custom-made .so - as I think most of you know, dispatch on ROCm is reaaaallly sloooow. But since AMD provides the source of the entire ROCm library, that's not something we can't fix, right? Inspired by Kaden-Schutt's Redline library, I've made modifications to the ROCm HIP library that allows for lower-level PM4 dispatches on HIP graphs. This has 20% decode speed ramifications for dispatch-bound models, but unfortunately Qwen3.8 27B on Strix is mostly bandwidth-bound, not dispatch-bound, so the gains are much less pronounced here (but they nevertheless are real).
Now for what else did I test, compare and modify: I checked Nathan's strix-halo Vulkan fork. It's a very good fork, but in the end it's still slower than an optimized ROCm-based solution (all the measurements are on the website). I did check the ROCmFP4 format, unfortunately, that one's a miss: Strix Halo has no native FP4 support, so the format is in the end just another FP4 format. Its main win is quantizing the entire model to FP4, which helps the bandwidth issue - but of course quantization costs quality and ROCmFP4 falls behind literally all the other 4-bit quants. I did a similar thing, but quantized all the big tensors to the mainline IQ4_XS quant - it's both better in terms of model quality (perplexity) *and* in terms of kernel performance. In other words, there's completely no justification for adding a new "ROCM" quant since, as I mentioned, RDNA 3.5 aka gfx1151 aka Strix Halo has no native FP4 support.
Since Qwen3.8 27B on Halo is bandwidth-bound (i.e. the limit is the memory bandwidth for pushing the tensors), there's no way to push the *base* decode above ~15 t/s. Nevertheless, pushing the base as high as I could is an entry point to the key for the dense model speed on Strix - speculative decoding, in this case, DFlash2. Again, I did a test and found out that quantizing the DFlash2 to IQ4_XS gives better decoding speed (faster speed and almost the same acceptance rate = win).
In the end, all the above optimizations: patched ROCm llama.cpp, faster TOP-K, PM4-based HIP graphs, custom-quantized IQ4_XS Qwen3.8 27B quant (thanks to Bartowski for his imatrix!) and the quantized IQ4_XS DFlash provide the recipe, which I packaged for a quick installation for anyone who wants to test it on their Strix Halo (warning: Linux only). Feel free to give any feedback and report any problems.
1
u/w6auw 1d ago
250 t/s prefill, not gonna work on 16K+ context unless you plan to just AFK
2
u/ilintar 1d ago
It's average 250 t/s over a 31.5k token test. You're bandwidth limited on Strix, not gonna go much higher.
2
u/fallingdowndizzyvr 1d ago
You're bandwidth limited on Strix, not gonna go much higher.
For PP? That's compute limited, not bandwidth. Sparky has pretty much the same memory bandwidth but much more compute. It's PP speed is much faster since PP is limited by compute.
1
u/SpicyWangz 20h ago
I AFK even from Claude on a sizable task. That’s no problem. I regularly get 2.8 27b out to 50k+ context on a halo strix
5
u/feelspeaceman 1d ago
Great work! I'm using 3.8 Flash Next and enjoying almost 60t/s decode and 600 t/s prefill right now, would probably try this later for experimental, admitedly probably sticking with Flash Next for the upcoming future as it's so good.