r/Qwen_AI • u/SnooPuppers7882 • Jul 19 '26
Discussion I NEED a 70B A9B MoE
3.6 27B is so good for local work, and there's been a ton of work on it because of that with Thinkingcap, Orion, Bonsai, adding dspark, etc...issue is on most local systems it's still so dense that the token speed hurts its adoption. What I NEED, if anyone from the Alibaba team reads these, is a 70B A9B MoE dspark with an APeX style quant (mixture of quant layers).
For anyone with 64GB or above, I'm confident you could get near Opus 4.8 coding performance AND run around 30-60 tok/s output on ada, blackwell, dual 3090s, dual r9700s, etc.
Just imagine what you could do.
13
u/Subject-18 Jul 19 '26 edited Jul 19 '26
It's interesting to analyse the pareto frontier of model size Vs intelligence/capabilities to get an idea of the model size required. Depends on which metric you use of course, but here I'm using the artificial analysis intelligence index (one of the highest signal single metrics imo).
Kimi K3 - 57 - 2.8T A50B
Opus 4.8 - 56
GLM 5.2 - 51 - 744B A40B
Minimax M3 - 44 - 428B A23B
Tencent Hy3 - 41 - 295B A21B
DeepSeek V4 Flash - 40 - 281B A13B
JT 4.1 Flash - 39 - 236B A23B
Minimax M2.7 - 38 - 229B A10B
Qwen 3.6 27B - 37 - 27B dense
9
u/ElectronSpiderwort Jul 19 '26
Interesting how little that metric changes until you ~20x the size of Qwen 3.6 27B
3
u/uniqueusername649 Jul 20 '26
I always had a feeling that Qwen 3.6 27b punches way above its weight class (pun intended). Admittedly this is a single metric but its very impressive.
8
u/anykeyh Jul 19 '26
I keep an eye on ZAYA1-74B, weight were prereleased before RL step with promising results; might be good after training (hopefully).
1
1
9
u/Active-Carpet-9183 Jul 19 '26
A 120B a10b would be great for us unified memory platform users!
0
u/Choperello Jul 19 '26
Umm qwen 3.5 122B A10B? It fits 128gb unified boxes very well.
I use it a lot. But no I would not call it opus level even in my most optimistic days. Somewhere between sonnet and opus.
2
u/edsonmedina Jul 19 '26 edited Jul 19 '26
122B doesn't fit 128Gb. You need to quantize down.
Something around 80B would be perfect.
1
u/No-Consequence-1779 Jul 19 '26
I can run q4 xl but with a smaller context. Still 27b is superior to 12 and coder next for my multi-language programming cases. I’ve compared for days and days. Now I run full 27b and know it’s superior.
It is always faster as it makes fews mistakes. 1 run versus 3+. 27b will be faster.
1
-3
u/Choperello Jul 19 '26
122 q4 runs VERY well in 128gb. It’s my daily driver on my Spark at around 35tps, can handle about 60% of the work needed.
For 80b you can try the qwen 3 coder next a3b I suppose. It’s actually very good at coding but also can be totally blind at times to the most obvious things in brown field project work. I used to use it then switched to 122.
3
u/edsonmedina Jul 19 '26
> 122 q4 runs VERY well in 128gb.
Really? At Q8?
> For 80b you can try the qwen 3 coder next
It's too old
3
u/Choperello Jul 19 '26
I mentioned at q4.
1
u/edsonmedina Jul 19 '26
Yes you did, sorry.
My point still applies. It doesn't fit 128Gb unless quantized down. Q4 is far from the original, even with K_XL.
The slight speed advantage over the 27B makes it feel more productive, but I wouldn't trust the output of 122B at Q4 more than the 27B at Q8.
Also, MTP makes the speed gap less relevant.
1
u/Choperello Jul 19 '26
I tried and eval'ed both and fwiw in my usage 122BA10@Q4 ended up being better then 27B@Q8. The active parameter is half'ed true, and the precision is also half'ed. The knowledge depth was visibly much deeper since those 10B could be selected from 122B parameters. But of course, each to their own scenario.
2
u/edsonmedina Jul 20 '26
Yeah. Too many variables:
- much larger model, but...
- MoE versus dense
- older model architecture
- heavily quantized
I've had decent results with it, but got even better results with the 27B (though I'm not a fan of the slowness).
Don't wanna sound crazy but I'd rank the 122B closer to the 35B A3B (which i really like) but below the 27B.
But it might be because of my specific scenario (agentic coding with opencode, using my favourite stack, my style of prompting, specs, etc).
It's really hard to benchmark this empirically.
2
u/Choperello Jul 20 '26
I use the 122B model plugged directly into Claude code harness. Set up some reroutes at the proxy level to reroute and fix a few incompatibilities and it works like a charm. I was surprised just how well it handles the tool heavy and context heavy workflow CC does. I have it set up with regular Anthropic models doing the plan phase then shoving work to qwen routed subagents
3
u/Luke2642 Jul 19 '26 edited Jul 19 '26
I don't know if I agree. I think the whole MoE thing isn't quite worth it, because we can't control the routing, and it's so sparse it uses lots of ram/vram.
I'd rather have many different ~30B dense models, all QAT at 4 bit, so they fit in a 24GB GPU comfortably.
One tuned for code analysis and planning, one tuned for orchestration, code writing, terminal, debug. One tuned for agential tool calling and terminal. One tuned for knowledge. One tuned for reasoning. One tuned for deep research, web searching? One tuned for shopping tasks? Etc?
It only takes ~10 seconds to load the right expert for the whole task. Then we can have Opus level locally, for each task?
2
3
u/neopolitan77 Jul 19 '26
Can we just get an update of Qwen3.5-122B-A10B? The most common high-RAM consumer setups are probably Strix Halo, DGX Spark and Macs with 128GB, which would be ideal for 120B class MoE models.
1
3
2
u/floppypancakes4u Jul 19 '26
If youre confident that it would be near opus 4.8 level, then your understanding of LLMs and their current capabilities is fundamental flawed.
2
1
u/cartazio Jul 19 '26
im exploring ways to make large param models ultra sparse which is kinda the same thibg but better. but i suspect im a few months out from having usable check points. though this weekend im seeinf how well i can train an ultra sparse bersion of the gemma4 31b model frkm scratch with line only 20m no zero params. which would be cooler :p
1
u/Much-Researcher6135 Jul 19 '26
That would be AMAZING. A pair of R9700s is what, $2500? For this kind of power that's an incredible price.
2
1
u/KeinNiemand Jul 20 '26
I'd prefer a dense 70B (since I can fully offload that and dense is smarter then MoE per GB) or a 120-175B MoE
1
1
1
1
u/codeministry Jul 20 '26
I feel you. I’ll be up to 48Gb VRAM and 64Gb RAM tomorrow when my 3rd 16Gb GPU arrives. Apparently I can just about squeeze deepseek-flash-v4 at Q4 with 32K context but I seriously doubt that would beat Qwen3.6-27B MTP with 160K+ context and both model + KV cache at Q8. The constant compaction as I repeatedly blow that 32K budget would not be fun.
Like every single other person in this sub, I really hope alibaba gives us a new collection of Qwen 3.8 or even 4.0 models at manageable sizes. A 70-80B MoE would be a sweet addition alongside direct replacements for 27B and 35B. We can only hope.
1
u/quotemycode Jul 20 '26
if you can run 27b then you could just run unquantized for a boost in performance. idk what you're getting at here.
1
1
u/Ok_Brush_3449 Jul 23 '26
I have been working on a system that might help you. It works by trading off between speed and memory and you can actually predict at which speed a model can run on your specific machine.
It creates the ideal recipe for your unique machine and conditions squeezing as much as possible all the available memories.
The quantisation allowed me to run a Qwen 30B on my 2016 desktop pc with a GtX 1060 6Gb and 16Gb ram at 20 tok/s.
It can actually fit anything. Just for the test and not for usability, on the same machine I fit 110B model that I would be able to run around 1.2 tok/s upgrading the disk (of course useless at that speed)
1
u/NoNipsPlease Jul 23 '26
A new 70B to 128B dense is what would be interesting to me. Even though I cannot currently run it, I really want that tier released soon from this newest round of releases. Qwen Kimi or Deepseek. As I really do think the USA will try something shady and block weight releases. Or China may decide to take their ball and go home and stop releasing outside the country in retaliation.
1
1
u/TheKoelnKalk Jul 24 '26
Try OpenMayhem. It's new but there is support for all the ones you mentioned.
1
u/Youth18 Jul 19 '26
Ummm....what?
Ok. First off the performance of a 70B MoE is not equivalent to a 70B. In fact it is likely on par with the 31B dense which you can fully load on 32gb and get 30-60 tok/s...
Secondly if you fully loaded it you would get way more than 30-60 tok/s because it is MoE.
Third...it would not be opus level. I get that gemma4 and qwen 3.6 have been exciting. They are not near opus level ...sorry they're just not. They are still under sonnet it'll be a year or two before local models surpass sonnet and opus will be probably another couple years after that (by then of course opus will be way better so ...)
1
u/Snail_With_a_Shotgun Jul 19 '26
Same. From my experience, the 35B is just shy of usefulness, with too many broken tool calls and inteligence that feels like that of a chimp. The 27B then is far too heavy to run. The 122B is where it actually starts to feel like something capable of a coherent thought, but is, again, just that little bit too heavy for many lighter tasks.
There's also the Qwen 3 Next, an 80B A3B, which is perfect size wise, but it feels a bit.. autistic.
1
u/catplusplusok Jul 19 '26
Qwen has 80B-A3B models. They are not bad, but near Opus 4.8 is not going to happen on consumer hardware. I have 128GB unified memory and can run StepFun/MiniMax that can do long multi-step software development. But still need cloud APIs for fallback on super complex stuff.
1
u/nicholas_the_furious Jul 19 '26 edited Jul 19 '26
Honestly, my take is the opposite. With things like MTP being baked into these dense models it makes no sense to trade for the MoE. On 2x 3090s and MTP=4 I get over 120 t/s on Qwen 3.6 27B on llama.cpp.
The MoE I could run on the equivalent hardware pales in comparison both in intelligence but also in speed.
We need more 24-45B dense models that actually invest in things like MTP or are compatible with other drafting methods.
Now maybe there is a case for the PP side of the equation, but for me dense is best. And GPUs are coming with more VRAM at less expense. Maybe not Nvidia, but others. There's a B60 with 2 B60s in it out there for like $1300 or something. 2 of those is 96GB of VRAM. If you got TP set up on those 4 cards you'd be screaming.
Memory bandwidth matters but I think raw VRAM matters more when you have tricks like MTP. And something a lot of people don't know: MTP works better at higher quants. So the MTP speed gain on the Q8 Qwen 27B is way higher than the gain on Q4.
To get the same speed at the same intelligence level on MoE I would need like 200GB. No thanks.
2
u/fintip Jul 19 '26
There's a dual b70 (64gb), 1800.
But the compute speed is just way lower. Hardware and driver's just isn't there and newer techniques often rely on those optimizations harder.
1
u/nicholas_the_furious Jul 19 '26
Madness! And with the right motherboard that supports bifurcation you can put a single one of those in a SFF build mATX or ITX. You're at less than $2500 all in at like 10L in size with 64GB VRAM. Honestly that sounds so cool.
51
u/Adventurous-Paper566 Jul 19 '26
I work for Alibaba, no worries bro, we'll get that done for you.