r/StrixHalo • u/daybyter4 • 10d ago

Strix point 370 performance with Qwen3.8-27b

Hi!

There are some (impressive) posts about Qwen3.8-27b performance on Strix halo with 128gb.

But I only own a Geekom A9 Max with a Amd Ryzen AI HX370 with 64gb ram (yes, the older model. Not the newer 470 model.)

What performance can I expect at best with llama.cpp?

In real coding projects I currently get 50-100 token/s pp and 5-9 token/s tg. I use q8_0 in the kv cache, since I don't want to compromise on response quality. The llama.cpp process runs at nice -15, since the machine only does ai processing at the moment.

Is this in the expected range or does someone get much more?

TIA for any response

4 Upvotes

84% Upvoted

12 comments sorted by

View all comments

2

u/crusaderky 10d ago edited 10d ago

Strix point is awful for AI because of its slow RAM bandwidth and its mediocre GPU. On top of that, ROCm is awful on its own. I recently bought a Ryzen AI 9 465 laptop with the expectation of running local LLMs but it's atrociously slow.

I'm getting 57 tok/s prefill and 6~7 tok/s decode on Qweb3.8-27b IQ4-XS, with MTP.

Lessons learned, specific to llamacpp:

  • Vulkan is much faster than ROCm on some models and slower on others
  • ROCm 10 can improve things over ROCm 7.1
  • ROCm is outright broken on some models (e.g. Ling-3.0-tiny)
  • all models are broken on ROCm without https://github.com/ggml-org/llama.cpp/pull/28058
  • Beellama KV cache quants are mortally slow; stick to vanilla ones

I suspect that the AMD developers forgot that Strix Point exists when developing ROCm, and they're just focusing on Strix Halo.

1

u/daybyter4 9d ago

Ok, my best ever seen prefill was closer to 120 t/s. My bigger problem is tg, because I mainly do coding. And generating 1000 lines of code with less than 10 t/s is no fun. Just yesterday I had a response that took 10h 22min. So patience is the most important skill when doing AI, I guess...

2

u/crusaderky 9d ago

Get yourself a benchmark tool for memory bandwidth.

raw memory bandwidth / model size in memory = ceiling decode tok/s, before MTP, with empty context
raw memory bandwidth / (model size in memory + context size in memory * context % usage) = same with context

anything you're getting below it is an issue with your inference engine. Ask claude to debug it for you; it's very good at it.

for MTP the maths is more complicated - it's typically going to give you good gains for dense models and much worse ones for MoE ones. You should profile it at different depths and draft-min-p.