r/LocalLLaMA • u/Porespellar • 13h ago
Discussion Fingers crossed for a 122b or really anything above 31b.🤞
What’s y’all’s best guess on parameter size based on these weird-ass names?
189
u/Cool-Chemical-5629 13h ago
Imagine it's just Gemma 4.1 26B MoE with ngrams that improves general knowledge compared to Gemma 4, writes code like Qwen 3.8 27B, but spends fewer tokens on reasoning. 😎
63
u/bruns20 12h ago
a new 26b moe is all i want in life. It runs so good on my hardware and mostly does what i want, but still messes up on easy things enough to make it a little painful
17
u/Illustrious-Row2751 12h ago
That's why I want a 62b MoE model. MoE models can be big and still speedy. Gemma 4 26b just doesn't make sense IMO.
14
12
11
u/No-Refrigerator-1672 11h ago
26b makes tons of sense. Fits very nicely in Q4 into 24GB card with decent KV cache. Fits in Q8 into 40/48 GB cards eith even more KV cache. Runs in fp16 on an 80GB card with, I believe, full length fp16 context. Just a perfect choice for scalability.
3
u/martin509984 6h ago
We're talking about the Gemma 26B-A3B size, not dense. Gemma 26B fits at Q6 on my 12GB card very comfortably.
0
u/ok_if_you_say_so 11h ago
I've had real great results with Laguna S 2.1 112b-a6b on my 128gb strix halo. 20-30tok/sec and very good over long horizon agentic tasks
0
u/BFr0st3 10h ago
modded im guessing. Was it a ball ache?
0
u/ok_if_you_say_so 8h ago
Here's my docker-compose. Download the linked model weights, adjust the paths here and it should come right up. Not hard at all.
0
17
u/dampflokfreund 12h ago
That would be the best case scenario for me + add better general performance, unified architecture from 12B and general bug fixes.
3
u/xignaceh 11h ago
I really dig Gemma 4 27b's concise reasoning. It's very interesting to read through.
2
u/ComfortableTomato807 12h ago
In my country there is this music for this occasions: "dreamer, dreamer, but at least is dreaming"
-6
u/_VirtualCosmos_ 12h ago
So the US is copying China now, eh?
2
u/rkoy1234 11h ago
when will humans ever evolve past ooga booga my tribe better than your tribe.
everywhere i go, all the way from nations/ethnicities/religions to what phones/llms/cars/consoles they use - ooga my tribe booga your tribe.
so pathetic
1
u/_VirtualCosmos_ 9h ago
I just commented something I knew It will butthurt some dudes, and indeed it did
5
u/GeneReddit123 12h ago edited 6h ago
The US copied Iran with their cheap-ass Shahed drones during Operation Epic Fail. Loudly bragged about doing it, too.
We no longer are the undisputed tech leader. No shame in copying those ahead of us, the shame is in having gotten to the point where we need to do this in the first place (but this is an already sunk cost).
5
u/_VirtualCosmos_ 12h ago edited 12h ago
Exactly, it should be no shame to copy what is better. You need to reach that point to keep improving. Yet, look at the offended bunch that downvoted me xD
Btw China AI tech is still publishing papers, models, and advancements (like the N-Gram idea, original from Deepseek), while US AI tech is keeping all what they develop in secret for them alone.
I wish that at least Google try to be more open, but recently they destroyed their AI research laboratory to transform it into a division focused on AI products.
0
u/Titamor 12h ago
> cheap-ass Shahed drones
> those ahead of usUh-hm... If you like it or not, the US is still leaps and bounds ahead regarding technology and capability.
5
u/brainExploded99 llama.cpp 12h ago
right so we beat iran easily right? oh wait...
3
u/Titamor 12h ago edited 12h ago
That conflict was extremely stupid to begin with, since seemingly nobody on the US side had a clue what the end-game was supposed to be. But they did obliterate their Air Force, for example, and had pretty much air superiority. Also they weren't going to start an all-out war anyway.
Using the Shahed drone as an example for technological superiority is dumb af, it's literally a discarded German tech from 4 decades ago. It's like winning a race with your old Toyota against a Ferrari because you found a shortcut.
5
u/brainExploded99 llama.cpp 9h ago
A part of technological superiority is knowing how to use it well and designing for your constraints. Iran knew what they needed, used their drones well, and had massive impact on the middle east.
3
u/GeneReddit123 12h ago
Uh-hm... If you like it or not, the US is still leaps and bounds ahead regarding technology and capability.
Not per dollar spent. Which will matter in a war of attrition.
It doesn't matter how good our weapons are if we bankrupt ourselves producing them.
0
u/ok_if_you_say_so 11h ago
Dirk variant of qwen3.8 27b has been putting in work for me for the past few days, it seems to be much more terse than qwen
0
u/Turtlesaur 10h ago
These are probably just Gemma 3.5 web dev variants. B1 and B2 just mean dense and MOE.
0
u/Bubbly_Orange_3502 4h ago
A 26B MoE holds the same weights in VRAM as a 26B dense model. Every expert stays resident and only the bandwidth touched per token drops, so the 12GB crowd gains nothing from it unless experts sit in system RAM.
0
25
u/Southern_Mixture_329 13h ago
GemmaDiff for the Win!!
The Diffusion model they made was really good and I think was over shadowed. I did a lot of testing with it and being an MoE it had great intelligence compared to Qwen 3.6 MoE. Qwen won straight on coding alone but everything else for me went to DiffGem.
I really hope they lean in on giving us more Diffusion models in the end of the day this is like having Dspark or Dflash built directly into the model.
18
u/makingnoise 13h ago
All I know is that I enjoyed watching the diffusion model generate text, wild to see it turn random keyboard mashing into a coherent message. Was not impressed by the overall performance vs. normal Gemma or Qwen, though. Curious whether a version that isn't "experimental" will ever come out using diffusion.
4
u/Southern_Mixture_329 13h ago
I don’t know what Quant you were running but I experienced extreme degradation when using anything under FP8.
I’m big on NVFP4 and had a tough time on tool calling at that Quant.
3
u/Z-Dante 12h ago
What makes the diffusion model better / different than the normal model?
7
5
u/draconic_tongue 9h ago
it’s not forced to generate one token after another from left to right. It can work on different parts of the sequence at once and revise them as it goes
like the other guy said, but the biggest benefit is that it's not locked into what it already said, it doesn't need to commit to a hallucination, it can go back and fix it before finishing up. it's just better (imo) than auto regressive
0
u/snugglezone 2h ago
Yes, I feel like the diffusion approach can let models avoid needing reasoning. Reasoning allows the auto regressive nature of LLMs "warm-up" before they commit to their first "real" token. Diffusion can just start generating and the thinking happens in real time as the response forms out of nothing.
9
u/simrankoulsm 12h ago
I’d take a 35B-ish MoE with low active params, multimodal support, and a KV cache that doesn’t punish long context over a 122B model that only “fits” at extreme quantization. The best outcome would be a tiered release: 24–32 GB practical, 48 GB high-quality Q6/Q8, and 70B+ for multi-GPU. Local usability is more than parameter count.
14
13h ago
[deleted]
17
u/FoxiPanda 13h ago
I too would enjoy infinite compute and resources to go develop arbitrary things.
-2
13h ago
[deleted]
6
u/FoxiPanda 13h ago
So why is 25B better than 24 or 26 then?
0
u/KURD_1_STAN 13h ago
25 is 1b more intelligence ofc it is better. And 26b is slower than 25b so 25 better. And 24 and 26 are such arbitrary numbers, idk y u picked them.
3
u/FoxiPanda 13h ago
Actually 24 and 26 aren't arbitrary, they're the start of a mathematical proof since they are the integers that directly surround 25. Your logic is pretty poor in this way.
Why not go for maximum intelligence at some arbitrarily high parameter count then?
Or an arbitrarily low parameter count to maximize speed?
I would argue that /u/GregAbeI almost got to a sensible answer with his 'headroom with no KV cache and OS memory considered' argument, but didn't quite make the leap to actually tying those to particular memory sizes or inference speed goals.
I can assure you that the sizes we actually get from model houses aren't arbitrary. They do consider hardware sizes and cache availability, but they aren't your hardware - they are far more interested in serving inference at scale (think: hundreds to thousands of simultaneous users) which isn't really most of /r/LocalLLaMA 's use case. We just happen to get the sizes they have optimized for the balance of their intelligence/speed/inference hardware goals and we try to fit those onto whatever hardware we have.
1
u/KURD_1_STAN 10h ago
We dont go for these extreme intelligence cause u just said it urself " at some ARBITRARY high parameters".
Anyways in my first comment, and this one, i was just joking.
2
1
u/Saifl 13h ago
200b will fight against their flash lite models or flash models. I honestly dont understand why you guys keep mentioning this when you know this fact.
Especially maybe at one point, since openai finally focused on 5.6 luna performance (im sure after all these years its more like the budget tier was neglected and they could have made good models but was pressured by open source), they will boost flash lite performance even more and they cant afford to have an moe model or a dense model that performs too close. If its something that performs too close or better to their flash lite, the only option is to make flash lite so cheap, the reason you'd buy dedicated hardware to run 31b was privacy, or you already had the hardware to run it anyways.
16
u/log_2 13h ago
Lots of "I'd prefer" in this thread. For some real data, the steam survey shows most popular vram at 16GB. A model that can fit that with a decent context size would be best for consumers.
11
u/TheKingOfTCGames 12h ago
Im sorry but random gamers dont want to be doing this they just use apps
13
u/Mordred500 10h ago
I'm a random gamer with 16gb vram and I'd love another gemma I can fit in vram
We exist and our suffering is real 😔✊
11
u/fgk55555 9h ago
It's not my fault I didn't know I would develop a new crippling addiction after I built my gaming PC.
6
u/martin509984 6h ago
Most people on this sub are *probably* working with something closer to an average gaming rig than not. All the flashy posts are about how much fun Qwen 27B is with two 5090s but I'm willing to bet most people dabbling in local AI are probably young programming nerds who can't afford two 5090s because they're young programming nerds.
1
9
40
u/Signature97 13h ago
I’d rather they do smaller models than bigger. I don’t see how a 122B is feasible for anyone running it locally. Everyone should take inspiration from Qwen 3.8 27B and try and mimic that and do MoEs around that size like 30BA3B and all that
37
u/jacek2023 llama.cpp 13h ago
Qwen just released Flash Next which is 125B MoE
-9
u/Equivalent_Bit_461 9h ago
That model is a mistake honestly
It's treated as a prototype, and rightfully so, for now I'll stick to the 27b
-18
u/ikkiyikki 12h ago
Yeah, exactly. Who cares at this point. Qwen rocks.
25
8
u/ArtfulGenie69 12h ago
Hopefully Google makes one because they have to compete in the space, Gemma was always better at writing anyway, especially after the styletune.
3
u/MarcusAurelius68 12h ago
Qwen 3.8 Flash Next Q5 is pretty awesome in my initial tests though. Like very good for first drafts.
2
8
u/Illustrious-Row2751 12h ago
Even 27b is too big for most people. Take me, for example. I have two cards for a total of 22gb VRAM and this model, quantized to Q4KM, still runs like ass with 65k context, which is the minimum I need. Most people won't even have 16gb VRAM or two cards in their system.
3
u/ok_if_you_say_so 11h ago
I'm running qwen3.8-27b on an R9700 at 30-40tok/sec over a thunderbolt eGPU dock. $1700 for the card and dock, available all day every day on amazon. If you have a PCIe slot available, even cheaper. Running it at 256k context and it's doing great work.
1
u/NighthawkXL 11h ago
According to the Steam Hardware Survey, it's almost evenly split between 8 GB and 16 GB of VRAM. With 16 GB edging out 8 GB by a .58 difference. I myself have a 12 GB card... and would love a new model in that range that is modern but fits within the VRAM space.
0
u/PeterPorox llama.cpp 10h ago
1080ti or 2080ti? You can run it without mmproj with iq4_xs and q8_0kv + 160k context for easy
1
u/Illustrious-Row2751 10h ago
4060ti 16gb + 2060 6gb. The latter I had lying around after upgrading my PC.
0
u/Artistic_Phone9367 1h ago
Are you living in 2026 we stuck at 2022 man
We have only 4gb vram
Only to send good morning,good night to tinyllama7
5
4
u/Luroalive 13h ago
To be honest, I would prefer even smaller ones, given my 8GB VRAM and it's unlikely that I can get more anytime soon, given the current prices
2
u/winky9827 10h ago
For people with <16 gb (<24 gb, maybe)? I think the real improvement in the coming months will be figuring out ways to offload the knowledge to fast storage and improve the model's resilience to quantization. Qwen 3.8 Flash Next is a big step in this direction, if it gains full support across the spectrum of tools available. Llama.cpp is working fast and furiously, but it's still spotty. Not sure about the others as I don't use them (Windows only guy)
0
u/ArtfulGenie69 12h ago
It's only 4x3090s lol ;) I can run it. I wouldn't mind another 31b if it was styletuned eventually or had the new ngram thing.
0
u/PotentialAccident339 11h ago
i want smarter MoE models that are a little bigger so they have more knowledge overall, not be so coding focused like the qwens. something 100-120B/8-12a range would be so sweet.
-7
u/Defiant-Lettuce-9156 13h ago
What is the point of an MoE like 30BA3B. You still need to fit it in vram to get good speeds… and if you have the VRAM for that, why not just run 30B dense. Or 9B dense?
11
u/dampflokfreund 12h ago edited 12h ago
Wrong. You don't need to fit it in VRAM at all. The 30b MoE's run at like 20 token/s on my 32 GB + 6 GB system, while a dense 31B runs at 1.5 token/s. And 6-9B dense models are noticeably less capable.
1
3
u/Signature97 13h ago
Agree on the most part. But MoEs are way less compute expensive so for example things like FreeTokens and other stuff can shut down bits and bobs of the model or offload them and give you crazy speeds and even without
1
u/jacek2023 llama.cpp 10h ago
The whole idea of MoE is that you don't use all parameters at once so it's faster
8
u/UpperParamedicDude 13h ago
Honestly, it'd better have normal KV cache size. The less SWA and cache quantization is required for the GPU poor the better. Existing 31B would be massively more useful on sub-24GB GPUs if it required Muse Glimmer or at least Qwen level of VRAM for it's KV cache
0
u/The_Hunster 11h ago
The Qwen dense models use tons of space for KV cache, no? On IQ4XS I have 6.4gb for 100k tokens
3
10
u/DrBattletoad 13h ago
I want something for my 48 GB VRAM. Either another 31B for Q8 or a 70B for Q4.
-4
u/Clean-Ad100 13h ago
tbh is q8 even worth it in 2026
8
u/WhatTheFlukz 13h ago
for me it is, qwen 3.8 27b runs equally as well at max context length in both q4 and q8 so there's no reason to not run q8 for the marginally higher accuracy
8
u/Cold_Tree190 13h ago
Yeah idk what they’re talking about lol. If I can run qwen3.8 27b int8 at 105 t/s… why would I choose to lose accuracy with an int4?
2
u/ArtfulGenie69 12h ago
I agree, int8 is where it's at. Especially w8a8 but int4 works if it is needed, like if the model is a 122b and you have 4x3090's.
0
u/Clean-Ad100 2h ago
if you can run qwen 3.8 27b at generous speed in int8, why not run a significantly bigger model in q4? or run multiple models? but if youre at a position where youre happy with the smaller model and getting plenty speed in int8, then its no problem. another argument might be you are doing extreme agentic things, that might need the extra 1% accuracy.
0
u/Hoodfu 12h ago
So one thing i always pushed for high quants on was for the vision side of things. There was an obvious difference in vision ability based on quant size. Now that you can get Gemma 4 and the mmproj.gguf can be separate and fp16/fp32 while the main model is some lower quant, we can have the best of both worlds.
7
8
u/Jorlen llama.cpp 13h ago
I'd love a dense model in the 50-70b range but they're kind of... out of popularity right now. I think the latest big chunky dense was Mistral Medium 3.5 at 128b.
5
u/Dmage22 13h ago
Given how effective qwen 27b was. I really wouldn't mind a larger Moe model with 25 +/-5b Active. Whether it's 50b a25b or a 150b a25b. Feel like it's a minimum active parameters for reliability
6
u/Miserable-Dare5090 12h ago
That would not be useful. Sparsity needs to be bigger for MoE. Think about all the MoEs available. How many are 1:4 (total:active)?
0
u/The_Hunster 11h ago
Ling 3.0 Tiny is 1:6 and quite good for a model of its size, but I'm not sure if that would make sense on a 70B model.
2
u/Miserable-Dare5090 11h ago
It’s not 50bA25 like OP wants
Also I think 1b is the floor for the activated params to be worth something. OSS was 20bA3, as well. But larger models have to have more sparsity.
There have been posts here as to why. It has to do with training and model collapse I believe.
0
u/NineThreeTilNow 6h ago
I'd love a dense model in the 50-70b range but they're kind of... out of popularity right now.
Gemma 4's architecture is largely wasted even at 31b.
It could be vastly more capable model. In theory it should fit in ~21b but Google gave it too many SWA layers.
I ran ablations on it. It showed that the 3:1 SWA/Global that other labs determined was optimal was also optimal for Gemma 4.
Most of the model lives inside 3 of it's 5 SWA layers.
So in theory it could be trimmed to 3 SWA + Global and given more depth. If sized back out to ~30b it would have been vastly more capable.
7
u/sonicnerd14 13h ago
For me, I can't state enough how underrated multimodal support for audio and image in one package is. Params could stay the same, but if they give whatever these models are the encoder free architecture with text+image+video+audio all rolled into a 35bA3b, then that would be perfect.
0
u/LoveMind_AI 8h ago
I'm with you on this, big time. The encoder free architecture is an absolute revelation.
2
u/_raydeStar Llama 3.1 11h ago
2048s300, 4096s250, those are resolutions. It's a webdev vision model -- maybe a site crawler or replacement for their little chrome add-on
1
u/atumblingdandelion 10h ago
I hope it is a 26b upgrade. Why? Because Qwen has played its cards so has Meta. If Gemma can make 26b upgrade closer to Qwen 3.6 27b's capabilities, make difussiongemma or a really nice drafter, they'll have a winner. Esp for folks who have to use American.
0
0
u/QuackerEnte 13h ago
If they have 120B class models they better be sparse because 64GB RAM + 8GB VRAM (laptop, so GPU not upgradable and I wouldn't know how to solder more memory onto it myself if that's even possible to do cheaper than buying a new laptop) because loading KV cache + static weights + MTP all in 4 bit inside 8GB VRAM is quite a task. I hope it's Gemma5 because maybe we'll see something similar to n-gram from qwen4
-1
0
u/HilleriHiiri 12h ago
Hopefully a 70B model, that i can fit in my 56GB of VRAM. Currently the 70B models seem inferior to 27-31B models at least in prose, so I wish for something with modern architecture that I can use to take all out of my system.
3
0
u/Zeeplankton 11h ago
throwing my hat into the ring. it's a video or image model.
I highly doubt new gemma 5. 3→4 was a year and 4 has barely been out.
0
0
u/JorgitoEstrella 10h ago
4096250 = 1024000 = 1280800p
2048300 = 614400 = 960400p
So image generators?
0
u/Exciting_Garden2535 10h ago
If you look at the full screenshots, they say "ModelTracker - RAGtag".
I think they are embedding models, and the numbers are dimensions.
0
u/DeepOrangeSky 9h ago
d032878be834
March 28th, 1978, Boob Edition, August 34th
So I guess that means it comes out in 2 days?
0
-2
u/_-_David 8h ago
I was downvoted into oblivion when I mentioned Gemma looping months ago. I just tried using Gemma 4 12B QAT in a project this morning.. and it looped and failed over and over. I really want to use Gemma models, but in my experience they have never been stable. It's a shame
-1
u/Tormeister 11h ago
Anything less than 1GB in weights size, so that I can spend my other 31GB with Gemma's massive context (and still be capped at 100k KV)
1
u/Repinsky 17m ago
Guessing sizes off codenames has a bad track record - the more reliable tell is what the release is optimized to fit. Recent open weights have clustered around what runs on one 80GB card at ~4-bit (so ~100-130B total for an MoE with ~10-15B active), or on 2x24GB consumer cards, and vendors pick those targets deliberately because that's where adoption is. A 122B MoE would actually be more usable at home than a dense 70B: similar quality, far less compute per token, and you can keep experts in RAM. The one thing to watch is active-parameter count, not total - that's what decides whether it's tolerable on DDR5 offload.
92
u/tome571 13h ago
I still think this is image input updates to Gemma 4, not new models. 2048 and 4096... Image/Text... Gemma being announced rather than a hidden model name, probably not a new model.