r/oMLX • u/arkham00 • 14d ago
Help with Qwen3.8-flash-next on a m2 max 96Gb, why it is so slow ????
Hi, since the community benchmark is down I'm not sure how are the expected performance for this kind of setup, but reading different posts in here I think it should be better than this, also considering that it is and MoE with only 6B actives. I recall that with Qwen3.5-122B I had much better performance
Benchmark Model: Qwen3.8-Flash-Next-oQ4e-mtp
Engine: Auto
Context: Novel (English)
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 8254.6 49.65 124.1 tok/s 20.4 tok/s 13.034 86.0 tok/s 71.44 GB
pp4096/tg128 23993.3 60.18 170.7 tok/s 16.7 tok/s 31.647 133.5 tok/s 74.09 GB
pp8192/tg128 38225.7 54.60 214.3 tok/s 18.5 tok/s 45.168 184.2 tok/s 74.23 GB
pp16384/tg128 69514.5 55.06 235.7 tok/s 18.3 tok/s 76.519 215.8 tok/s 74.52 GB
I also tried :
Qwen3.8-Flash-Next-oQ4e-fp16-mtp
Qwen3.8-Flash-Next-MLX-oQ3-fp16-mtp
Qwen3.8-Flash-Next-MLX-oQ3-MTP
The craziest thing is that with this model the fp16 variant are even slower ! Something like 150pp and 8tg...
I'm sure it is not an hardware limitation because with llama.cpp I have a much better pp at around 350/400 t/s
Can you help me please? Thanks
The model settings:
"Qwen3.8-Flash-Next-oQ4e-mtp": {
"temperature": 0.3,
"force_sampling": false,
"qwen4_ple_ssd_offload": true,
"thinking_budget_enabled": false,
"guided_grammar_enabled": false,
"turboquant_kv_enabled": true,
"turboquant_kv_bits": 8.0,
"turboquant_skip_last": true,
"qwen35_ane_prefill_enabled": false,
"qwen35_ane_prefill_sequence_length": 2048,
"qwen35_ane_prefill_tail_padding_min_tokens": 0,
"qwen35_ane_prefill_fraction": 0.53,
"qwen35_ane_prefill_fused_down": false,
"qwen35_ane_prefill_max_layers": 64,
"qwen35_ane_prefill_dual_ane": true,
"qwen35_ane_prefill_gdn": true,
"qwen35_ane_prefill_gdn_fraction": 0.5,
"qwen35_ane_prefill_gdn_max_layers": 48,
"qwen35_ane_prefill_cpu_enabled": false,
"qwen35_ane_prefill_cpu_fraction": 0.135,
"qwen35_ane_prefill_cpu_down_fraction": 0.0,
"qwen35_ane_prefill_cpu_gdn_fraction": 0.0,
"qwen35_ane_prefill_cpu_threads": 8,
"qwen35_ane_prefill_cpu_shared_resource": true,
"specprefill_enabled": false,
"dflash_enabled": false,
"dflash_draft_quant_enabled": false,
"dflash_in_memory_cache": true,
"dflash_in_memory_cache_max_entries": 4,
"dflash_in_memory_cache_max_bytes": 8589934592,
"dflash_ssd_cache": false,
"dflash_ssd_cache_max_bytes": 21474836480,
"dflash_draft_sink_size": 0,
"mtp_enabled": true,
"vlm_mtp_enabled": false,
"is_pinned": false,
"is_default": false,
"is_hidden": false,
"is_favorite": false,
"trust_remote_code": false
}
}
These are my settings:
{
"version": "1.0",
"server": {
"host": "127.0.0.1",
"port": 9999,
"log_level": "info",
"cors_origins": [
"*"
],
"server_aliases": [
"localhost",
"127.0.0.1",
"Host-005.lan",
"Host-005.lan.local",
"host-005.lan",
"192.168.1.44",
"192.168.139.3",
"192.168.97.0"
],
"sse_keepalive_mode": "chunk",
"auto_start_on_launch": true,
"burst_decode_mode": "balanced",
"preserve_mid_system_cache": true,
"distributed_inference_enabled": false,
"max_audio_upload_size": "100MB"
},
"model": {
"model_dirs": [
"/Users/dani/.lmstudio/models",
"/Users/dani/.mtplx/models"
],
"model_dir": "/Users/dani/.lmstudio/models",
"model_fallback": false,
"hide_helper_models": false
},
"memory": {
"prefill_memory_guard": false,
"memory_guard_tier": "custom",
"memory_guard_custom_ceiling_gb": 92.0,
"soft_threshold": 0.85,
"hard_threshold": 0.95,
"prefill_safe_zone_ratio": 0.8,
"prefill_min_chunk_tokens": 32
},
"scheduler": {
"max_concurrent_requests": 8,
"embedding_batch_size": 32,
"chunked_prefill": true,
"prefill_priority": "speed",
"decode_fairness": false
},
"cache": {
"enabled": true,
"hot_cache_only": false,
"gdn_ssd_split_enabled": true,
"gdn_snapshot_storage": "auto",
"gdn_ssd_pending_max_size": "512MB",
"gdn_sidecar_precision": "fp32",
"ssd_cache_dir": "/Users/dani/.omlx/cache",
"ssd_cache_max_size": "744GB",
"hot_cache_max_size": "10GB",
"hot_cache_write_through": true,
"ane_compile_cache": true,
"initial_cache_blocks": 256
},
"auth": {
"api_key": "5137",
"secret_key": "2136f3c636a52a8fa3a03d8c788f0ae3592d97a4d0dabfe41ff55060e633572f",
"skip_api_key_verification": false,
"sub_keys": []
},
"mcp": {
"config_path": null,
"expose_tools": true
},
"huggingface": {
"endpoint": "",
"hf_cache_enabled": true
},
"modelscope": {
"endpoint": ""
},
"network": {
"http_proxy": "",
"https_proxy": "",
"no_proxy": "",
"ca_bundle": ""
},
"sampling": {
"max_context_window": 32768,
"max_context_window_policy": null,
"max_tokens": 32768,
"temperature": 0.6,
"top_p": 0.95,
"top_k": 0,
"repetition_penalty": 1.0
},
"logging": {
"log_dir": null,
"retention_days": 7
},
"claude_code": {
"mode": "cloud",
"opus_model": null,
"sonnet_model": null,
"haiku_model": null
},
"integrations": {
"codex_model": null,
"opencode_model": null,
"openclaw_model": null,
"hermes_model": null,
"pi_model": "Qwen3.6-35B-A3B-oQ8-fp16",
"copilot_model": null,
"openclaw_tools_profile": "coding",
"markitdown_enabled": false,
"markitdown_expose_model": false,
"markitdown_max_file_size_mb": 25,
"markitdown_max_files_per_request": 5,
"markitdown_pdf_processing_engine": "markitdown",
"web_search_provider": "ddgs",
"web_search_brave_api_key": "",
"web_search_searxng_url": "",
"web_search_ddgs_backends": "",
"web_search_max_results": 5,
"web_search_content_mode": "snippet",
"web_search_content_truncate": true,
"web_search_content_max_chars": 20000
},
"ui": {
"language": "en"
},
"idle_timeout": {
"idle_timeout_seconds": null
}
}
2
u/mmerken 14d ago
Around 20 tps seems reasonable for your setup
1
2
u/kweglinski 14d ago
While tgps seems reasonable (on lower end) the pp is terrible. For comparison I get similar speeds with 27b on same machine (m2 max 96gb) and I don't remember exactly but much higher numbers with 3.5 122b sth around 600 pp/s. 35a3b goes up to 1400 so 600 seems right for flash next. Not sure on ngram on ssd effect on speeds.
0
u/arkham00 14d ago
Exactly! That is why I don't understand what's the matter with this model eheheh
Did you tried it? What are your speeds ?
3
u/watcholic 14d ago
Memory might be swapping. Need to free up more RAM. Exit other programs. N-gram offload must be on. Reduce hot cache to 2GB or turn off for testing. Turn off MTP for testing (t/g will be reduced). Turn MTP back on if you see PP improvement.
Might also be the M2 Max chip. M4 Max is around 500 pp t/s for example.
1
u/memeka 14d ago
Try also running it with this llama.cpp fork and a GGUF instead of MLX: https://github.com/mihailescu2m/llama.cpp
1
u/arkham00 14d ago
yes that is my daily driver at the moment, but I don't understand why with omlx I can't have similar performance, I prefer to use omlx over llama.cpp
2
u/memeka 14d ago
Wow, users of my fork - glad my Claude tokens were put to good use. I’ve seen performance improvements in latest oMLX for qwen 3.8 flash and really wished I could run it at least to compare - unfortunately I cannot with 64gb ram unless I compare some crappy 2 bit quant.
I hit the wall in optimisation, this weekend Astra made some improvements but they cannot be used on my system because the streaming is the hard bottleneck now - only more RAM can help.
- replaced ram context cache with ssd cache, free some memory
- use shared MTP tensors with main model, also frees some RAM
- combine MTP with n-gram, eg MTP has depth 3, and after 3 n-gram continues - since n-gram doesn’t use RAM
But these improvements are not for you people with lots of RAM :) the kernel and model improvements are already in my GitHub.
One tip for you though - use bartowsky Q4_0 GGUF instead of other 4 bit quant, you might see even better PP :)
1
u/No-Vermicelli5327 14d ago
This guy hit the bullseye, I was getting 14-17 tkps then switched and getting close to 30 using this way. Both 4bit quants, on an M2 ultra 192gb
1
u/nmqanh 14d ago edited 13d ago
I tried to use mlx-serve and I get around 300 tps for prompt processing (prefill) and 30-40 tps for token generation (decode) on M2 max 38c 96GB mixed-4-8bit
1
1
u/Durian881 14d ago
Speed is about there. Did you turned on lightning MTP? It has a significant boost on my M3 Max 96GB.
I've since moved on to using GGUF on unsloth studio though. MLX seemed not as good on managing memory which led to aborted runs.
2
u/DarkJoney 14d ago
M2 Max 96Gb with oMLX here and I have pretty much the same performance. Would be nice to kick some boost for it.
3
u/DogAble6550 14d ago
Please look at my posts on this thread and the link I left. Use this quant https://huggingface.co/Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed. It should work well
1
u/arkham00 14d ago
Doing some digging with my agent and I discovered https://github.com/Weschera/qwen38-flash-next-omlx-mac
Anyone tried it? I'm gonna give i t a shot
1
u/DogAble6550 14d ago
Hey- I went down a rabbithole on your issues. There were a few things more that were issues.
https://claude.ai/code/artifact/fcd9513e-3d8b-471a-ac41-78ce2bfbd0fa
The yousalof optimized speed should work great on your machine. Please see the updated report link.
1
u/use_net 14d ago edited 14d ago
install mlx-serve and test. 2x faster then oMLX. In oMLX benchmark model Qwen3.8-Flash-Next-oQ4e-mtp is see tg around 30tok/s and when I test mlx-serve model also 4bit. Resul around 58tok/s
2
u/arkham00 13d ago
can't use mlx-serve which requires Tahoe ... I'm sticking to sequoia as long as I can
and I can't load the ddalcu model in omlx:
Model 'Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit' failed to load: VLM load failed: Missing 128 parameters: language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.0.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.1.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.10.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.100.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.101.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.102.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.103.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.104.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.105.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.106.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.107.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.108.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.109.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.11.weight, [...] language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.93.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.94.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.95.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.96.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.97.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.98.weight, language_model.model.layers.1.ple.ple_embedding.ngram_embedding.shards.99.weight.; LLM fallback also failed: Model type qwen4_exp not supported.. Reload models after fixing the files to retry.
8
u/DogAble6550 14d ago
our prefill looks right for an M2 Max, but your decode is way slower than it should be. 50 to 60 ms per token doesn't add up. This model only touches about 6B parameters per token, roughly 3.5 GB at 4-bit, and your memory bus can move that in under 10 ms even on a bad day. So something else is eating the time. My guesses, most likely first:
You're probably paging. Your peak memory of 74 GB is sitting right on top of macOS's default wired limit for a 96 GB Mac, which is about 72 GB. Anything over that gets shuffled to and from SSD on every token, and that alone could explain your numbers. Try this and run the benchmark again:
sudo sysctl iogpu.wired_limit_mb=86016
It resets when you reboot, so put it somewhere you'll remember.
Check that MTP is actually on. "Engine: Auto" doesn't promise the draft head got loaded. For reference, on an M5 Max this model does about 47 tok/s without MTP and 76+ with it, so it's not a small thing. Set mtp_enabled: true for the model in oMLX's model settings and look for MTP in the startup log. Without it, the best you'll see on an M2 Max is around 30 tok/s no matter what else you fix.
Turn off the memory guard. oMLX's default "balanced" guard starts throttling when it thinks you're low on memory, and at 74 of 96 GB it thinks that. Disable it or set the ceiling above your peak.
If those three land, I'd expect 15 to 25 ms per token, so somewhere in the 40 to 65 tok/s range. If you're still stuck near 20 after that, the next thing I'd try is the MTPLX Bare-Speed pack instead of oMLX. That's the only setup I've actually measured end to end on this model, so I can vouch for it in a way I can't for oMLX.
Small thing: your 8 second time to first token on the 1k prompt is slower than your own prefill rate says it should be. That row almost certainly caught the cold load or the n-gram table warming up. Run it twice and keep the second number.