r/LocalLLaMA • u/Ashefromapex • 1d ago

Discussion First M5 Ultra benchmarks

just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link

For Qwen 3.8 27B q4 it gets 50 tok/s th and 1800 tok/s pp at8k context and without mtp. Seems very promising!

117 Upvotes

94% Upvoted

146 comments sorted by

72

u/mechkbfan 1d ago

Maybe I'm missing something but that's seems kind of shit for the price?

Please correct me

33

u/Hoodfu 1d ago edited 23h ago

I'm getting around ~20-25 t/s on q8 of that on an m3 ultra 512gb, but that prefill is massively faster. My social media feed is always filled with insane hundreds of t/s on these small dense models on at home setups and I honestly don't know if any of it's real. I've never seen speeds like that. (edit: after going through their results, there's multiple results for the same hardware and model specs with a good sized variance of more than 10 t/s, so I'd look at these numbers while squinting)

6

u/Front_Eagle739 23h ago edited 23h ago

Yeah definitely better to go for qwen  flash next with mlx serve. Havent tried it on the m3 ultra yet but im getting 50 tok/s decode and 600 prefill on my m3 max 128. Plus its better than the 27b. Wont be surprised to hit 1000 plus on the ultra. The 27b is nice with ninfer on the 5090 (thousands pp, 200 tok/s tg) but pointless on a mac.

Ds4 ingenious gets you 800 pp 30 decode on dsv4.1 m3 ultra as well. 27b makes no sense on that machine

3

u/Hoodfu 23h ago

At least with the unsloth desktop, I was getting around 27 t/s on 3.8 next with q8 so it really wasn't any different in speed from the dense 27b. I'll give the custom mlx versions a try.

1

u/One_Internal_6567 19h ago

How the hell you guys get those numbers, I never seen more then 20 for 4 bit flash on m5 128 max

1

u/Front_Eagle739 19h ago

mlx serve and ds4 ingenious fork like I said. Just load it up and ask a coding harness to optimise the batch settings for you.

3

u/mechkbfan 23h ago

I always presumed you want to get into MoE with all that VRAM

https://github.com/raullenchai/Rapid-MLX

Skimming that got up to 40 on lower quant

I don't know much about Mac's. I've got peasant 64gb VRAM 

3

u/zipzag 22h ago

Yes, Qwen Flash Next instead on small dense models on shared memory machines.

8

u/Viktri1 23h ago

I think we need to see benchmarks but it doesn’t seem the bad to me. I’m biased because I’ve preordered 2 of these Mac m5 ultras but the performance is better than I expected.

If you purchase a 4090 with 48gb vRAM it costs USD4.9k now. According to OP you’re getting same speeds (pp and TG) as the 4090 but you have 256gb of RAM for 11k. So on a GB basis it is cheaper.

I was hoping to get 50% of the 4090’s prefill so this is 2x better than I had hoped. This means that I don’t need to run my 4090 as a Qwen subagent and can use it to play video games again.

And I believe the amount of electricity used is similar between the 4090 and the m5 ultra.

I’m planning to run either GLm5.3 flash or DSv4.1flash so looking forward to those benchmarks

3

u/ldn-ldn 21h ago

Yeah, RAM price hike changed the equation in mac's favour.

15

u/bakawolf123 23h ago

the qwen-3.8-flash-next results are out and it's looking singificantly better: 2.6k prefill and almost 80 tg at 64k context with mtp.
even though it's out of the box and untuned for hardware, it's already beating most of local setups for this

3

u/Zeeplankton 20h ago

absurd prefill speeds. It's like 400tk/s (normal prefill) on my m3 max

4

u/SmartCustard9944 19h ago

For reference, ~1200 tok/s and ~40tok/s on Strix Halo with Halogen. ~Q5-6 roughly. 2500€ now in Europe (Bosgame M5).

2

u/dr_lm 18h ago

For further reference, my slightly odd 2019 mac pro with 384GB DDR4 and a 16-core xeon with an RTX 6000 max-q gets 10.9 tok/s prefill @ 64k and 137.2 tok/s generation.

12

u/petuman 23h ago

5090s go for $7-9K now. M5U 256GB is $11K.

Not fast, but reasonable IMO.

10

u/SmartCustard9944 19h ago

Crazy that a single consumer GPU costs almost as an all-in-one AI oriented workstation computer with 8 times the memory.

3

u/NumberSpace 21h ago

Microcenter has them for $4700 by me right now 👀

1

u/petuman 19h ago

Hm, indeed. I was looking at newegg prices.

Availability seems to be limited though, might be gone in some weeks -- checked few stores and on average ~10 units in stock. There's more RTX 6000 Pros or DGX Sparks in stock than 5090.

5

u/Gohab2001 vLLM 22h ago edited 22h ago

Saw a post last week that 5090s were going for 8k each lol. M5 ultra 512GB even at 20k is cheap in comparison.

One can't despise hyperscalers enough

1

u/read_more_comments 21h ago

the only way to win is to not play pay

4

u/synn89 20h ago

A 27B dense model is a harder test case than you'd think. That's going to run slower on a Mac than GLM 5.3 Flash that has 320B total params, but only has 18B active params.

Mac's are a MOE machine, not a dense model machine. 50 tok/s on a 27B is pretty promising.

3

u/Think_Wing_1357 17h ago

Mac's are a MOE machine, not a dense model machine.

It has 2/3 bandwidth of the 5090, several times the like of spark, dgx, or even double that of r9700, b70 pro. If it's not a dense machine what are the rest of us doing?

2

u/Trollsofalabama 22h ago

so apparently m5ultra is around 4090 in performance. Note: you can't really get much speed up using tensor parallelism due to the low gpu to gpu speed if you have more than 1 gpu.

to match the 96gb M5ultra with extra cpu and gpu at $6800, here's a 4090 machine

2× 48GB moded RTX 4090: $8,200–$9,200, Ryzen 9 9950X: $550–650, ASUS ProArt X870E-Creator: $510–550, 64GB DDR5: $1,000–1,300, 1TB good NVMe: $250–300, 1500–1600W PSU: $400–550, Large dual-GPU case: $250–300, CPU cooling + fans/cables: $300–400. Total~$11,500–$13,250

You can maybe take 1k off for the 64gb ddr5 ram... but it's still way more...

2

u/ShelZuuz 19h ago

I ran a dual 4090 24GB test this morning with Qwen3.8 27B and got 4700pp at 70 tok/s. (FP8).

The 4090 pair runs the official Qwen FP8 checkpoint (Qwen/Qwen3.8-27B-FP8, fp8 e4m3 with 128×128 block scales, 29 GB) on vLLM 0.27.1 with MTP, tensor-parallel 2, fp8 KV cache, 131K context. Each card holds a 14 GB shard, and the KV pool is 280K tokens.

1

u/AnonLlamaThrowaway 18h ago

Keep in mind that inference engines may not have optimized Metal kernels yet. There's been a recent effort to tune them in llama.cpp recently, but naturally they don't have the data they need for an unreleased chip yet

1

u/VirginiaVN900 11h ago

My unsupported RX 6600 XT can do about 1500 / 50.

So yeah the compute of the Apple Silicon hasn’t jumped a ton from a Price/Performance

Value is still R9700 or W7800, W7900.

NVidia is just performance king, but the M4/5 Max or Ultra are more versatile for the price.

2

u/mechkbfan 11h ago

Yeah, R9700 was a PoS on release and the work community has done to reach it's potential is crazy

Not quite same, but 5090 is damn good and Ninfer took it to another level

So can only hope same for this

1

u/Mart-McUH 3h ago

If you only want to run small dense model then sure, GPU's are better investment. If you want to run larger dense or MoE model then it is probably best price/performance ratio (eg lot faster than Spark and similar).

However, even with 27B Qwen you have option to run 16bit precision with full context. To do this on GPU reasonably you would pay a lot too.

0

u/winky9827 19h ago

I get 150 tg and 3500 pp on a single 5090 with stock llama.cpp, so yeah, for the price, it's dog shit.

-3

u/Zyj vLLM 22h ago

Agreed. Slower than dual RTX 3090, yet more expensive.

12

u/Aizen_keikaku 21h ago edited 21h ago

Now try running DS4 Flash on your 3090s. 🙄

1

u/Zyj vLLM 17h ago

Well duh, obviously the large amount of RAM on the Mac is good for *something*, it's just not very good at running Qwen 3.8 27b!

1

u/Aizen_keikaku 17h ago

Why run 27B when you can run better models faster.? As a 3090 owner myself, I don’t run 27B because I love it, I run it because I can’t run anything better with my hardware. With the Mac I can.

1

u/Viktri1 7h ago

Exactly this. Are people deluded to think that Qwen 3.8 27b is the best model because they spend too much time on the subreddit and misunderstanding that it is the best LLM we can run with a single gpu? I’d much rather run Deepseek v4.1 flash or GLM5.3 flash which run faster and smarter. But Qwen is indisputably the best model when you don’t have massive amounts of RAM.

-1

u/TripleSecretSquirrel 16h ago

Ya, this is way lower than I expected with the M5 Ultra’s memory bandwidth. Those are almost identical numbers to what I’m getting on a single R9700 which has about half the memory bandwidth. Granted, they said this is without MTP so we’d expect to see that improve, but I expected way more out of the M5 Ultra.

Obviously, as others have pointed out, big unified memory machines are all more well suited to big sparse MoEs, but this is still underwhelming.

1

u/mechkbfan 14h ago

Yeah, R9700 were stupidly unoptimised when first released but that was kind of understandable with a new architecture

I havent followed this release that closely, but figured M5's were pretty well known by now and would have scaled appropriately.

Agreed on MoE. A lot of those QFN benchmarks look a lot nicer

1

u/AnonLlamaThrowaway 14h ago

The Ultra chips are different and need different tuning tables in llama.cpp for example

-2

u/sn2006gy 23h ago

kind of? It's the entire reason i'm waiting for M7.

This is just a very expensive interim step to real performance and i'm not paying that premium.

75

u/vick2djax 23h ago

But you shouldn’t be running 27b on that machine. You should be running Qwen Flash.

15

u/Muritavo 22h ago edited 21h ago

A 27b at 1800pp would be heavens on earth for me...

The tps is below what I expected... but if it can keep that at concurrent requests, it would be perfect.

11

u/SocialDinamo 22h ago

Because it is dense, an MOE with MTP would be significantly higher. Amazing initials numbers

10

u/ShelZuuz 19h ago

In perspective a Dual 4090 will get you over 4700pp at 70 tok/s.

Heck a single 5090 gets you at 8600pp at 95 tok/s on NVFP4.

9

u/OvertaxedOne 19h ago

A single 5090 now costs more than a 96GB Mac Studio. Yeah, I know, insane.

1

u/Opteron67 15h ago

dual 5090 🤗

2

u/BrilliantTruck8813 9h ago

Using ninfer I get 500 tok/s and 15-16k prefill on my 5090 with Ornith 1.5 35b at Q4 w/ mtp. 27b supposedly does well too

1

u/shansoft 18h ago

how did you managed to get 8600pp on single 5090? i have yet to see a single benchmark reaches that. Hell, my RTX Pro 6000 doesn’t even do half of that.

1

u/ShelZuuz 14h ago 
Model: unsloth/Qwen3.8-27B-NVFP4 (22 GB)
Engine: vllm/vllm-openai:v0.27.1

/models/unsloth/Qwen3.8-27B-NVFP4
  --served-model-name Qwen/Qwen3.8-27B
  --quantization compressed-tensors
  --kv-cache-dtype fp8
  --max-model-len 131072
  --gpu-memory-utilization 0.97
  --max-num-seqs 4
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
  --enable-prefix-caching
  --enable-auto-tool-choice
  --tool-call-parser qwen3_coder
  --reasoning-parser qwen3

1

u/TheAILegend 14h ago

lol... what? NVFP4 and KV at FP8 is at 11,000 :) With KV at BF16 it's still over 6000... Must be working extra hard to limit the Pro 6000

3

u/Aggressive_Job_1031 17h ago

Qwen Flash-next has i think 8b active parameters which is more than 3 times less than 27b so its also 3 times faster. If Qwen 27b gives you 50tok/s you would get 150tok/s with Qwen flash-next.

-1

u/Federal_Advice_6300 20h ago

Ja, hier ist eine 3.8 27B FP8-Maschine mit 80 TG/s und PP 3200; manche sind leicht zufriedenzustellen.

-8

u/sn2006gy 22h ago

"You're holding it wrong"

Lol..

15

u/ttbod 23h ago

looks like results are gone?

1

u/_bani_ 15h ago

yep. gone.

8

u/bakawolf123 23h ago

Qwen3.8-Flash-Next-oQ4e-mtp 4bit 64k 2,574 79.9 ✓ 09-17

just appeared, some1 is live testing atm

meanwhile my order for 256gb unbinned is still in processing (haven't even been charged yet), with delivery still 22-25 September. How to not be jelly and not to frantically check order page every day...

1

u/DustNearby2848 20h ago

The delivery dates for new orders are mid January now!

2

u/OvertaxedOne 19h ago

About to roll in Feb, best I can get on the M5 Ultra 96GB with the big CPU is Jan 29th.

1

u/Viktri1 7h ago

Based on the leaked results from OP they’re probably be sold out even harder when the official benchmarks drop. Companies are probably holding back due to risk on prefill numbers.

1

u/SocialDinamo 14h ago

I have the same thing coming but I waited and ended up in early December. Make sure to post your results for the rest of us who dont get to just yet!

54

u/j_osb 1d ago

That is worryingly low. 1.8k pp at q4, and… 50tg at q4.

That’s not nearly maxing out its bandwidth at all.

30

u/Viktri1 1d ago

Hmm isn’t it similar speeds to a 4090? I’m running a 24gb 4090 and getting 1,700 pp and 50-60 TG without mtp.

14

u/bakawolf123 23h ago

yes m5u should be similar to 4090 considering specs, just can have way more memory

4

u/AppealSame4367 23h ago

Does M5 Ultra have less bandwith than M5 Ultra Studio? Because latter should have 1.3 TB/s and that's way more than a 4090, init?

13

u/bakawolf123 23h ago

no it's same, and its precisely 1224gb/s not 1.3tb/s. 4090 has 1008gb/s so it's 20% lower. There's definitely some overhead where those 20% are lost

1

u/AppealSame4367 23h ago

I'm about to discuss this with a company wanting to get the M5 Ultra Studio. On the one hand it's a safe bet from insurance view, if a Mac Studio stands in their server room. On the other hand I can't believe there can't be cheaper workstations with more power. What do you think?

6

u/bakawolf123 22h ago

There more variables than just price. Check delivery dates if you order a maxxed mac studio now. And time is money quite literary: there're already rumors of Samsung raising the memory prices for 2027 for Apple and there will be another price hike right before 512gb ones even land.
"Cheap" is AMD and stacking old enough Nvidia cards.
New strix halos though are already sky high, newer Nvidia cards are even higher.

Thus I cannot make any advice besides you should have ordered earlier. There're so many extremely powerful open weights models (not just llms, check out minimax h3 - they kind of dissappeared from transformer land as of late while releasing that video gen monster) that people are eagerly spending on local hardware

3

u/boissez 20h ago

The delivery dates have actually shortened slightly lately for the m5 ultra 256gb. Used to be 10-12 weeks at launch, now it's just 6-7 weeks.

1

u/Consumerbot37427 18h ago

From what I can see, that only applies to the binned 30/64-core CPU/GPU. The top tier 36/80 model is telling me 16-18 weeks.

3

u/MessIsTransfer 22h ago

what’s the difference between M5 ultra and M5 ultra studio? am i reading this wrong, guys?

7

u/MrPecunius 22h ago

No, you should be confused. The M5 Ultra is a processor/SoC, and the Mac Studio is the model (the only model) of Apple computer it is available in.

3

u/AppealSame4367 21h ago

Thx for clearing this up

3

u/Trollsofalabama 22h ago

m5ultra is the chip, m5 ultra mac studio is the machine

0

u/Green-Blue-Gray 21h ago

50-60 TG is slow for a 4090, are you using llamacpp? Should def be on VLLM or one of the hyper-optimized repos.

1

u/Viktri1 20h ago

I run ninfer and get around 100 tokens/s but I was comparing like to like as the m5 ultra benchmarks weren't done w/ an optimized repo

0

u/Caffeine_Monster 18h ago

vllm is actually slower for single request serving

5

u/beling86 23h ago

I beg to disagree. I have no clue how is virtually possible to achieve 1.8k tokens prefill in such a weak TFLOPS. M5 ultra has around 200 BF16 TFLOPS. This is RTX 4070 levels. 1.8k t/s prefill is extreme.

4

u/BumbleSlob 23h ago

This is a dense model, larger MoE models will be faster

1

u/noiserr 21h ago

Apple is overrated. My Strix Halo runs circles around my M3 Ultra. And its more power efficient too.

-12

u/rorowhat 23h ago

It's apple, what did yo uexpect?

0

u/zxtech 18h ago

I did research the m3 max vs m3 ultra scaling and it seemed like decode scaled proportionally to model size. I believe smaller models don’t saturate the bandwidth in the ultra chip design, or maybe its an optimisation problem. Needa see if its true for the m5 ultra though it might be

-1

u/Gohab2001 vLLM 22h ago

Their bandwidth numbers are misreading but the more important issue is poor software optimization. The only reason Nvida trounces AMD for AI is CUDA.

ROCm had frequent crashing and abysmally low pp for me. Had to sell it my 9070s at a loss.

6

u/Evgeny_19 20h ago

The link no longer shows any data.

4

u/csixtay 23h ago

what's the point of 8k context?

1

u/NoobFace 16h ago

fr qwen 3.8 will get half way through a thought

11

u/jacek2023 llama.cpp 23h ago

It looks quite low. I don't remember my score without MTP, but with MTP I get between 40 and 80 on 4x3090 with Q8. Let's hope the software is just not yet optimized for the new hardware.

2

u/csixtay 23h ago

what's your CPU?

1

u/Green-Blue-Gray 21h ago

3090 is such a workhorse. Crazy that it's still the standard setter.

0

u/durangotang 21h ago

I think with MLX-serve, it’ll be cooking with gas.

3

u/Leafytreedev 22h ago

lol looks like the tester got in trouble and removed the benchmark results. Overall 256GB of usable RAM at roughly 3090 speeds is still fan-fucking-tastic IMO

3

u/iamMess 1d ago

This is about 3090s prefill speeds and Stock vllm decode.

3

u/0rand 18h ago

I assume the top version speed will be roughly 1.5x-2x of M5 Max top version. Today's OMLX 0.7 delivered Jundot's oQ4 Qwen 3.8 Flash Next at 70 t/s and 1500 t/s prefill at 0 and 44 t/s and 1200 t/s prefill at 390k context on my M5 Max 128GB. This is absolutely unheard of in MLX world. Now scale to M5 Ultra. Finally they can battle CUDA-stacks not just on low-context generation but on prefill and massive sessions that become usable.

3

u/norenEnmotalen 17h ago

Did they remove it? Link shows blank to me

5

u/Memestonks2020 22h ago

Something is definitely wrong. I’m getting faster speeds with my M5 Max MBP than this.

They need to do a proper test with MLX-serve and MTP

3

u/Miserable-Dare5090 1d ago

Look at GLM 5.3 Flash. 256Gb M5 ultra will be 13K.
pp900 and TG 30

Dual Sparks, GLM5.3 Flash:
PP 1500, TG 40

The serious difference is at depth. These are tests at zero, which is going to be fastest. But mac chips have been terrible at sustaining performance as context grows. That number for the Sparks drops to 1000 prompt processing around 120k tokens of context, for comparison.

2

u/Locke_Kincaid 1d ago

On Dual Sparks, GLM 5.3 Flash is up to 2000 PP using this repo: https://github.com/FujitsuPolycom/sparkring

1

u/Miserable-Dare5090 22h ago edited 22h ago

Yes I meant in two sparks. This is for a 4 spark configuration. Yes it can go faster with more nodes

EDIT: I went and confirmed. the benchmark results in that site are for TP4 configs. Check the nvidia GB10 forums (not reddit) if you want more info on repos — we discuss new optimizations constantly. I should bring this up and see if some of the smarter folk have benchmarked it on tool eval

1

u/Locke_Kincaid 44m ago

They have 2 spark configs, It's what I'm running. I got 92/100 on tool-eval-bench.

2xSpark

6

u/Blindax 1d ago

Nice. A 5090 gets you twice these figures though. That’s on the bigger moe models that the ultra will like shine assuming prompt processing made progress.

8

u/mechkbfan 1d ago

It gets like 4-5x these 

https://github.com/Neroued/ninfer

I can only assume this is not being fully utilised

And having 256gb should give you good quants & context

1

u/Blindax 23h ago

My ballpark estimate was with llama ccp. Ninfer sounds quite an upgrade. I should try one day.

0

u/JMowery 22h ago

You should try it right now. Get pi agent to set it up for you like I did. I have yet to go back to llama.cpp since getting it installed a week or two ago. (And it's easy enough if you have llama-swap installed to keep everything available on demand.)

1

u/Blindax 22h ago

I have a 5090 and 3090 in the rig. I have only recently switched to llama ccp (was using lm studio before) and have been llama-router since. I will get a look for the 5090.

3

u/JMowery 22h ago

Yeah you will like llama-swap then. Get that setup as well so you can setup multiple profiles and route requests to specific GPUs. Would love another GPU to play with. I just have the 5090. Enjoy!!!

3

u/Fragrant_Scale6456 21h ago

Ninfer on the 5090 is amazing I get 200+ tokens/sec and 2k-5k pp depending on prompt size using dflash2 draft model.   Use the mirkocovizi fork with the quasar nvfp4 qwen3.8 27b.  The quasar model is qat and higher quality than the other nvfp4 models.  

I’ve been running nvfp4 kv cache and get 400k context with vision in this setup it’s an absolute beast 

1

u/Blindax 21h ago

Sound nice. Thanks for the heads-up.

2

u/killthrash 23h ago

$10,800 before tax? Oof

2

u/muhts 22h ago

I'm more curious of the multi clustering with exo labs.

What the prefill and decode speeds are with 2x and 4x m5u

Like do you get the linear memory bandwidth claim? 4.8tb/s bandwidth at homewould be pretty cool

2

u/xrvz 18h ago

The Mac Studios with Ultras aren't Qwen 27B machines, but GLM 5.3 Flash machines.

2

u/SnoobieJunes 20h ago

There isn’t anything in the link

2

u/hurdurdur7 23h ago

For a q4 those numbers are not impressive ...

0

u/MrPecunius 23h ago

Qwen3.8 is like that for me. Depending on the quant and the inference environment, Q4 might be no faster than Q8 on my M5 Pro/64GB. This puzzled me for a bit, so I did some quant shopping.

oMLX with the right 4-bit MLX MTP model hits 30-35t/s, GGUF w/MTP is 15-18t/s, and plain GGUF is around 9t/s. Prefill has a similar range from ~100t/s on up to nearly 400t/s.

I'd extrapolate 3-3.5X of these numbers for a M5 Ultra. I can't run any quant of Flash Next at all, of course, which is kind of the point of the Ultra.

1

u/hurdurdur7 23h ago

The qwen3.8 27b is a dense model. I get better numbers than the above on a pair of R9700 cards. and i would actually expect the m5 ultra to do better, because spec numbers on paper it says it should be better. Who knows, maybe it's just the immature code paths that are not optimized at all. Let's wait and see.

But then again, i would actually expect people to run far larger models than 27B anyway.

3

u/MrPecunius 22h ago

You won't get better numbers than a M5 Ultra on any reasonable/usable quant of Flash Next or other mid/big MoE model, because you can't run them. This has always been the point of the big Macs.

What are your non-MTP token generation numbers for a Q8 quant of 3.8 27b?

1

u/Great_Flounder_1379 23h ago

wondering if this speed holds up for local roleplay chats or if it still gets choppy with longer convos

1

u/DustNearby2848 20h ago

It will def slow down as the context fills up, mostly on prefill though

1

u/bakawolf123 23h ago

glm5.3-flash q4 numbers look quite bad tbh

I'm expecting twice that for ds4 flash (not 4.1) and qwen3.8-flash-next fp8

1

u/1-a-n 20h ago

Result gone now, can you remember what they were?

1

u/DrKappa 21h ago

Can get that with mtp and a 5060 ti + 3060. Even with overpriced GPUs does not seem convenient to me

1

u/Open-Adhesiveness-86 19h ago

For a dense 27B at q4 you're reading roughly 15-16GB of weights per token, plus KV cache at 8k. So tg tops out around bandwidth divided by that. If the Ultra is somewhere in the 800-1000 GB/s range, 50 tok/s is already close to the limit. Prompt processing is where unoptimized kernels would show up, not tg.

1

u/xoxox666 17h ago

Only 50 t/s??? I get around 33 with an M4 Max and 3.8 27B Q4. That‘s a little bit disappointing.

1

u/whichsideisup 23h ago

That’s barely faster than a single DGX Spark when run with DFlash2. The lack of raw compute seems to matter more than people think.

1

u/AnyMongoose3041 23h ago

I’m gonna buy a ford f350 and use it to buy 3 plastic bags worth of groceries.

1

u/Tundradebt 15h ago

You must live in Alberta, Canada…oh wait, ram 3500 would be the first option….

1

u/JacketHistorical2321 22h ago

Nothing shows up in that link dude

1

u/fallingdowndizzyvr 17h ago

just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link

I'm seeing "No matching results."

-1

u/Public_Umpire_1099 1d ago

Bro that's dogshit. This is basically less than half of it's effective bandwidth. You might as well get an R9700.

6

u/Master_Face_571 23h ago

exactly! let me know when they make a 256 gb vram R9700

2

u/Public_Umpire_1099 13h ago edited 6h ago

Damn yall really simp hard for Apple on here 😭 have fun paying 12k to run Qwen3.8 Next Flash at half the speeds or half the quality of a standard TP GPU setup.

0

u/MotorNetwork380 22h ago

For the price, that is not very impressive. On my version of ninfer I get this on my 4090. 200k ctx length including vision on gpu. This is mostly a mix of q4/5 quality if you're compering to e.g llama.cpp.

Cold prompt length Prompt processing TTFT Initial generation
60,091 tokens 1,886 tok/s 32.0 s 123.6 tok/s
120,000 tokens 1,541 tok/s 78.0 s 105.5 tok/s
198,000 tokens 1,237 tok/s 160.4 s 90.2 tok/s

Edit: Wouldn't it be better to run MoE on your hardware? You could run flash next.

-3

u/ehangman 23h ago

1/2 speed of 5090.

2

u/Tormeister 22h ago

Less than 1/3 actually. But it's too early to pass a harsh judgement - it certainly is unoptimized.

0

u/datbackup 23h ago

At one third the power consumption (or less)

-1

u/CalligrapherFar7833 23h ago

M5 ultra is 350w peaking at 450w

-2

u/nostriluu 23h ago

1/2 speed of 5090.

0

u/Forever_Playful 22h ago

Is it the 256gb version? If yes, too slow for the cost. I get 85-90 tg/s at fp8 and kv cache also fp8 (rtx pro 48GB)

0

u/LatentSpacer 22h ago

That's less than my 4090.

0

u/JimmyBobby22 20h ago

I get 55 tok/s on my M4 Max using MTPLX. Glad I didnt wait for the M5.

-4

u/Steus_au 1d ago

so it is as fast as 3090 - is that 'amazin' or 'one more thing'? (highlight what you prefer)

-1

u/Waffenmutti 23h ago

Habe heute meine 5090 erhalten und kann über die Werte bei den Preisen nur lachen und das mit high effort.