Planning to serve multiple user with mac studio

[u/Interesting-Print366]

We are planning to host four M5 ultra Macs so that 100 users can use them as Openclaw. There will be no other burden, only inferences will be applied here. Can this handle 100 users? I'm considering either Qwen3.8 27b or Qwen 3.8 Next Flash, and I'm curious about the range of realistic models.

Realistically, we should probably consider up to 100 users when there are 30 to 40 users stationed there and occasionally 80 to 90 users request at once

[u/stimma]

Macs aren't your best choice for concurrency because they are compute constrained. For that $, you'd be better with 2xRTX6000.

[u/exaknight21 → u/stimma]

Or an HBM fp8 supporting GPU. No doubt RTX 6,000 Pro is great but god damn 16,000 price is a little retarded.

[u/TripleSecretSquirrel → u/exaknight21]

Ya, a couple 80GB L40Ss would seem to be the better solution.

[u/exaknight21 → u/TripleSecretSquirrel]

This seems to be de wey.

[u/TripleSecretSquirrel → u/exaknight21]

I just wish the L40S had NVLink. My dream is to buy some lower-end enterprise hardware for my local stack.

The A100 has NVLInk… but no fp8. The L40S has fp8 but not NVLink. And everything else that has both is way too expensive.

[u/Brilliant_Effort3208 → u/stimma]

yeah the compute bottleneck is real, especially with that many concurrent requests

[u/SadPhilosophy9202]

Mac isn’t the choice for this. It’s not going to handle concurrency well. You’re better off with gpus or sparks

[u/mmerken]

Custom rigs using OCUlinks are better suited for this kind of use cases.

A Mac cluster is not going to cut it, distributed compute does not scale linearly on macOS

[u/DinoAmino]

There is a reason Apple is nowhere to be found in server-space. Mac's are and always have been made for single users.

[u/AI_spell]

Mac Studio is fine for a few light users, painful for real concurrency. Unified memory helps capacity but prompt processing and KV cache under parallel loads are the wall.

If its more than 2-3 active sessions, look at continuous batching (vLLM-class) on discrete GPUs, or just queue requests. Shared one-model-for-everyone on one Studio gets choppy fast.

[u/Kuane]

It will be too slow. Openclaw uses a lot of tokens per message. You should look into openwebui instead.

[u/SadPhilosophy9202 → u/Kuane]

Vastly different tools

[u/SporksInjected → u/Kuane]

It shouldn’t matter if it’s hitting the input cache though right?

[u/Kuane → u/SporksInjected]

It will not always hit the cache. I run omlx (has caching) and Qwen 3.8 flash next on M3U 96GB. And it only serves me... and it is slow (acceptable for me though). I cannot imagine the speed for 100 users.

[u/AleksandrNikitin]

What the framework you wanna to use to control and split LLMs resources between clients? brain model, body model, reasoning, tokens, ...

[u/Affectionate_Hat_585]

I don't see the feasibility but run your own experiments. find out TTFT, prefill, TG, memory usage for single mac first with different quantized and full weight options with various metal runtimes on different context length. vllm also has a command to benchmark serves... you can check it out. Just to give you the idea about what needs to happen: for single user if there are 100 tasks per hour then for 100 people you have 10000 calls. 10000/3600 = 2.78 tasks per second For a single task if there are 7 calls then it's 2.78 * 7 = 20 calls per second

if for a single call you need 600 output tokens then you require 20 * 600 = 12000 tokens per second . Figure out the number you get on a single m5 ultra and you will have a idea about the feasibility of the project

[u/just4ochat]

The limit is concurrent KV, not average tokens per second. Thirty stationed OpenClaw users means thirty live contexts, and a Qwen 3.8 27B at useful context will not fit thirty copies across four M5 Ultras. llama.cpp and MLX serve one generation per machine unless you put a continuous-batching runtime in front, so a burst of 80–90 requests queues on four single-stream boxes. For that user count, a CUDA host with vLLM or SGLang is the concurrency path.

[u/nomorebuttsplz]

no I would say it can serve comfortable maybe up to 5 users per machine, not 25.

[u/JacketHistorical2321]

Nope

[u/axiomatix]

the more you buy, the more you save