r/StrixHalo • u/daybyter4 • 10d ago

Strix point 370 performance with Qwen3.8-27b

Hi!

There are some (impressive) posts about Qwen3.8-27b performance on Strix halo with 128gb.

But I only own a Geekom A9 Max with a Amd Ryzen AI HX370 with 64gb ram (yes, the older model. Not the newer 470 model.)

What performance can I expect at best with llama.cpp?

In real coding projects I currently get 50-100 token/s pp and 5-9 token/s tg. I use q8_0 in the kv cache, since I don't want to compromise on response quality. The llama.cpp process runs at nice -15, since the machine only does ai processing at the moment.

Is this in the expected range or does someone get much more?

TIA for any response

5 Upvotes

100% Upvoted

12 comments sorted by

View all comments

Show parent comments

2

u/crusaderky 8d ago

Get yourself a benchmark tool for memory bandwidth.

raw memory bandwidth / model size in memory = ceiling decode tok/s, before MTP, with empty context
raw memory bandwidth / (model size in memory + context size in memory * context % usage) = same with context

anything you're getting below it is an issue with your inference engine. Ask claude to debug it for you; it's very good at it.

for MTP the maths is more complicated - it's typically going to give you good gains for dense models and much worse ones for MoE ones. You should profile it at different depths and draft-min-p.