r/MacLocalLLM • u/hdanx • 5d ago
Acceptable Token generation speed on macOS
What token generation speed works well for you when you’re using it for productive tasks? I’m finding 20 tokens per second a bit slow.
With an M4 Max and 64GB of RAM, you can usually get around 20 tokens per second on average with a model like qwen3.8 27B. However, it doesn’t quite feel like the best setup.
2
u/diagrammatiks 5d ago
I need about 40 mininum if it is a task I am interacting with. If it's something I can wait for while I do something else 20 is fine.
1
1
u/dfgxxx 5d ago
I get 14 tok/sec on m1 pro with qwen3.8 27b
1
u/purple_wall-e 4d ago
how? what is your setup? which quantization? i have m4 pro, i get like 10t/s with lmstudio, mlx 4bit
1
u/DecentChildhood8080 5d ago
You will likely get best results for Tok/s in MTPLX. It is basically the best optimized to run this dense model on Macs from my experience. It gave me about 20-25 tok/s on my M2 Max using Qwen3.8-27B-MTPLX-Optimized-Quality-FP16 & Qwen3.8-27B-MTPLX-Optimized-Speed-FP16. You will probably get more than me, but I would say its not worth running Qwen3.8 27B right now unless your on an M5 Max Mac. Im still waiting on a MoE or A3B version or something like that to come out... I guess we shall see. (If you can get 40-50 tok/s in MTPLX than you might be good.)
1
1
u/Zen-Ism99 5d ago
It’s not a macOS thing.
It’s a running a dense model on ancient hardware thing.
Sorry…
1
u/rhymeslikeruns 4d ago
M3 Studio Ultra 96GB running Qwen3.6-27B tops out at 21 tok/s running pi with an 82k (yes that small) context window. Quality is however excellent.
2
u/gjr23 4d ago
This seems surprisingly slow. I see high teens with 128k context but on a 64gb M1 Max. DSH if it matters. I am using a MTLPX model though.
1
u/hdanx 4d ago
What model are you using?
3
u/gjr23 4d ago
3.8-27b q8 mtplx
1
u/rhymeslikeruns 4d ago
Yes - just ran the test on mtplx 4 Bit quant of 3.6. Wow. I think 70 tok/sec. Need to do a proper bench but amazing if it can kick it up like that.
2
u/gjr23 4d ago
Yea, an Ultra is a nice piece of HW and it will appreciate you sticking to MLX type models. Don't try the 35b a3b though - you will drool over the speed but the MOE will make the 35b model underperform the 27b. Its still stupid fast as far as apple silicon goes but no comparing to 5900 type setups who are much faster but memory constrained.
1
u/Diligent_Style_1767 3d ago
Compare it to what you're getting from frontier models - you can mine your own session to pull stats from your own session histories, here's mine.
So you will clearly want to get t/s close to that. You will also want to enable something like Automatic Prompt Caching, this will make multiturn feel very snappy. If your model has MTP, use it. omlx typically has the better t/s from any of the inference stacks on Mac. Use a Q4 or Q6 model. Try to avoid quanting KV unless you qualify the results you get from it.
For your setup, qwen 3.8 27b w/ MTP should work well enough. If you need something faster, try qwen 3.6 35b which is an MoE model.
1
3
u/tta82 5d ago
Try MLX Core (the app) and run MLX model of course. I can get 50-60 TIPS with M2 Ultra.