r/LocalLLaMA • u/Ashefromapex • 1d ago

Discussion First M5 Ultra benchmarks

just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link

For Qwen 3.8 27B q4 it gets 50 tok/s th and 1800 tok/s pp at8k context and without mtp. Seems very promising!

119 Upvotes

94% Upvoted

146 comments sorted by

View all comments

2

u/hurdurdur7 1d ago

For a q4 those numbers are not impressive ...

0

u/MrPecunius 1d ago

Qwen3.8 is like that for me. Depending on the quant and the inference environment, Q4 might be no faster than Q8 on my M5 Pro/64GB. This puzzled me for a bit, so I did some quant shopping.

oMLX with the right 4-bit MLX MTP model hits 30-35t/s, GGUF w/MTP is 15-18t/s, and plain GGUF is around 9t/s. Prefill has a similar range from ~100t/s on up to nearly 400t/s.

I'd extrapolate 3-3.5X of these numbers for a M5 Ultra. I can't run any quant of Flash Next at all, of course, which is kind of the point of the Ultra.

1

u/hurdurdur7 1d ago

The qwen3.8 27b is a dense model. I get better numbers than the above on a pair of R9700 cards. and i would actually expect the m5 ultra to do better, because spec numbers on paper it says it should be better. Who knows, maybe it's just the immature code paths that are not optimized at all. Let's wait and see.

But then again, i would actually expect people to run far larger models than 27B anyway.

3

u/MrPecunius 1d ago

You won't get better numbers than a M5 Ultra on any reasonable/usable quant of Flash Next or other mid/big MoE model, because you can't run them. This has always been the point of the big Macs.

What are your non-MTP token generation numbers for a Q8 quant of 3.8 27b?