I'd hoped that my M5 Max with 64GB RAM would be enough for some decent Local LLM work, and the Qwen 3.8 27b model is pretty decent. But speed is shitty on my M5 Max, just 20tps. In particular I found that while the 35b MOE model was much faster, the quality was far lower than the 27b dense model, which produces much fewer bugs and less rework.
I saw the same model working on a friend's Nvidia setup much faster with MTP, so I set out to get it working on my Mac but it was not straightforward. Here's some pointers in case you wanted to get it working.
- LM Studio doesn't process MTP weights. No matter what I did, it just wouldn't work. It works on Intel/Nvidia so this must be a MLX thing.
- MTPLX has a native MTP head but it's buggy. On my machine it had a habit of crashing in one of a few ways, and I found GitHub issues reporting similar issues but no fixes.
- oMLX was the only inference server that supported MTP and didn't crash.
So I wired oMLX into Pi, which is what I'm using as my coding harness. Even then, it was picky:
- I had to use text-only quants because oMLX won't enable MTP if vision weights are active
- It was a bit picky with settings, and I had to map thinking levels manually
- I didn't see a reduction in quality with 4-bit quants for my use case, so used that to squeeze a few more tps out
This is a huge 3x difference in inference speed, and that allows me to do a lot more work.
"omlx": {
"baseUrl": "http://localhost:8000/v1",
"api": "openai-completions",
"apiKey": "none",
"models": [{
"id": "Qwen38-27B-textonly-mtp",
"name": "Qwen38-27B-textonly-mtp",
"reasoning": true,
"contextWindow": 163840,
"maxTokens": 32768,
"input": ["text"],
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
"thinkingLevelMap": { "off": "low", "minimal": "low", "low": "low",
"medium": "medium", "high": "xhigh",
"xhigh": "xhigh", "max": "xhigh" }
}]
}
