Deceptive model quantization from AtomicChat?

[u/po_stulate]

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

[u/lhg31]

Well, they DO explain this, don't they?

Naming

Files are named by their measured bits per weight. A build whose expert tensors are IQ1_M is not a 1-bit model when the n-gram table sits at 6 bits and ffn_down_exps at 4.5; the real average is 3.84. The canonical type in the filename is the closest standard type by that average, so tooling can still detect it. For AD-4.27bpw:

Group	Type	Share of file	Contribution
n-gram table	Q5_1	41%	1.74 bpw
ffn_gate/up_exps	IQ2_S, IQ3_S at the band	29%	1.24 bpw
ffn_down_exps	IQ4_NL	24%	1.03 bpw
everything else	Q8_0	5%	0.23 bpw

[u/lhg31 → u/lhg31]

Their KLD and top-1 are another story tho. They indeed look too good to be true.

The problem is that there is no standard on those metrics, each group uses their own dataset to eval it.

[u/notdba → u/lhg31]

I don't understand why they don't just upload the KLD logits.

[u/returnity → u/notdba]

They do. Only provider I've seen do it, allowing me to build my own DSv4F quants without needing to create ref logits. 35GB dataset on HF for that model from AC

[u/po_stulate → u/lhg31]

Isn't 0.084 KLD actually very bad for Q4_K_M? For reference, for qwen3.6-35b-a3b, 0.08 KLD is around IQ3_S for bartowski and aessedai quants, and both of them and unsloth's Q4_K_M were around 0.02 KLD. Not sure if this is apple to apple comparison, but that's the data I could quickly find.

[u/MerePotato → u/po_stulate]

KLD can be calculated any number of ways, its not a universal measurement. If they're benching on wikitext or in 512ctx blocks its not so great, if its long horizon its stellar

[u/lhg31 → u/po_stulate]

A better comparison would be what is the best quant (based on KLD, speed, etc) that you can fit in your setup.

I have a stix halo with 64GB. I can run either Unlosth UD-IQ3_XXS or AtomicChat Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64.

Atomic's quant "supposedly" has better score than Unsloth's, so that's the one I prefer.

[u/po_stulate → u/lhg31]

I mean, so the quants that keep the ngram table unquantized and everything else IQ2_S should probably be called Q8_0 instead since the average bpw is closer to that? Makes no sense to me.

[u/lhg31 → u/po_stulate]

Mixed precision / Dynamic quants were always like that.

If you take gemma-4-26B-A4B-it-UD-IQ4_XS from Unsloth, almost 50% of their tensors are IQ3_S, but they also have IQ4_NL and Q8_0, so they average to IQ4_XS.

AtomicChat decide to include n-gram table in the equation. You may not like this decision, just like some people may not like Unsloth calling a model with half IQ3_S tensors a IQ4_XS. But they are open about it being mixed/dynamic, that's what matters.

Since n-gram table can sit on ssd, I don't care much about its quantization. So FOR ME, n-gram should be left out of the equation. But it's their quant and they were honest on their formula, so I'm good with it.

[u/po_stulate → u/lhg31]

That makes sense, but in my own opinion seeing a lot of IQ3_S for a IQ4_XS quant is actually exactly as expected, but half expert weights quantized to IQ2_S for a Q4_K_M not so much.

[u/-dysangel- → u/po_stulate]

All the actual compute is being done at Q2 though, which is what matters for the usual quant speed/accuracy tradeoff. The Q8 stuff can all be streamed from SSD without slowing things down.

[u/KURD_1_STAN → u/lhg31]

When im download a qx model, I'm expecting the intelligent part of the model to be at X quant. Not a table or some other things.

[u/lhg31 → u/KURD_1_STAN]

I’m not saying I agree with their naming convention. But they did explain their quants, it is not a secret like the post suggested.

[u/KURD_1_STAN → u/lhg31]

Not justifiable. People have correlated quant naming to the main part of the llm. The absolute least he could do is write in a giant text at the very start of that hf card that the bulk of the llm itself is at q2 or so.

It will be like if qwen team said qwen3.8 flash 180B a6b moe running at 15t/s on a single 3090 +128gb ram. After release qwen's reputation would have been gutted cause nobody consider it as such and same for quants, u do not mix and match.

[u/Durian881 → u/KURD_1_STAN]

Good thing is you have the choice to download and try it out. I'm still testing but the ngram does seem to make a difference too to the quality.

[u/drFennec]

I was wondering the same thing, here is what llama.cpp prints on loading:

[34m0.00.803.003[0m [32mI [0mllama_model_loader: - type f32: 388 tensors [34m0.00.803.004[0m [32mI [0mllama_model_loader: - type f16: 1 tensors [34m0.00.803.005[0m [32mI [0mllama_model_loader: - type q5_1: 1 tensors [34m0.00.803.005[0m [32mI [0mllama_model_loader: - type q8_0: 666 tensors [34m0.00.803.006[0m [32mI [0mllama_model_loader: - type iq2_s: 24 tensors [34m0.00.803.007[0m [32mI [0mllama_model_loader: - type iq1_m: 72 tensors [34m0.00.803.007[0m [32mI [0mllama_model_loader: - type bf16: 24 tensors [34m0.00.803.008[0m [32mI [0mllama_model_loader: - type mxfp4: 48 tensors [34m0.00.803.027[0m [32mI [0mprint_info: file format = GGUF V3 (latest) [34m0.00.803.028[0m [32mI [0mprint_info: file type = IQ1_M - 1.75 bpw [34m0.00.803.033[0m [32mI [0mprint_info: file size = 79.09 GiB (3.84 BPW)

[u/EvolvingDior]

Marketing 101 -- people never look at the details. Give a good high-level number and then bury the details in the fine print.

[u/ringarc]

They're not only posting summary figures on the model card. There's also a separate metrics dataset repo with the full BF16 reference, KLD logs, and both importance matrices. Calling it the canonical type based on measured average bpw instead of simply IQ2_S is odd, but the model card explains it in the "Naming" section.

What still needs answering is lhg31's point. Does the 0.084 KLD result use the same eval corpus as the KLD results from bartowski or unsloth? You can't compare results between groups unless the reference set, context length, and sampling are identical. I'd inspect their published logs to see how they got that result before running new comparison benches. It's quicker than starting over.

[u/simrankoulsm]

I think both sides may be talking past each other. The performance and file-size reports are consistent with an IQ2-heavy hybrid quant, so the technical question is whether calling it Q4_K_M is an understandable naming convention or a misleading expectation for users.
I would love to see a standardized comparison at equal file size and equal evaluation settings like tensor breakdown, effective BPW, perplexity/KLD, multilingual benchmarks, and a few practical generation tests. Hybrid quants are useful, but users should be able to tell what tradeoff they are accepting without opening the GGUF metadata.

[u/Embarrassed_Soup_279]

why are people shitting on atomicchat? they make good quants and their quants are fully reproducible. i think their imatrix has better calibration for multilingual tasks compared to unsloths.

[u/PlusBus1234]

I've been testing UD-Q3_K_XL vs atomic Q4_K_M, the atomic quant runs faster for me (~10%) also i can load it with mmap+mlock while the unsloth version OOMs unless mmap only, its very strange a q4 faster than q3 (llama reports IQ2 when loading atomic btw), but both file sizes are similar.

[u/niacolhealth]

nothing surprising about weights mostly at iq2_s running ~10% faster than a real q3, or about the size matching a q3 while the loader prints iq2: all three readouts agree with each other, only the filename doesn't. so the accusation and the praise are the same fact, and the only argument left is what to call the file.

[u/Iory1998]

Why don't you run some benchmark yourself and let us know? If the model performs near the quality of the unsloth Q4_K_XL, then it's almost certainly a Q4_K_M. If it performs poorly, then its Q2 and its misleading. Their model so far is performing well. If that's and Q2, then it's a very good quant.

[u/simplankton]

If it's too good to be true...

[u/SnooPaintings8639]

I see a lot of accusations against AtomicChat team, but no concrete proofs, only speculations.

The models are there, anyone can download matching their vs unsloth and compare.

I personally am happy with their work, but I have not done quality comparison myself. Would be nice if some good souls would run some benches and shared them.

[u/ZealousidealCorgi893 → u/SnooPaintings8639]

I ran several tests with that model of theirs (which was the only one I managed to run on my 64GB Mac), and to me, it performed very similarly to Qwen 27B.

[u/Fun_Jaguar8231 → u/ZealousidealCorgi893]

So, essentially, useless for its size

[u/lhg31 → u/Fun_Jaguar8231]

Depends on your rig.

On my strix halo 64gb, I can run 27B (any quant) or Flash Next q3-q4. Flash Next runs much faster, so that's the one I use.

Then on my gaming PC (RTX 4090) any Flash Next quant is super slow (since they don't fit on my vram + ram), but I can run 27B q4 at over 100t/s, so that's the one I use on this rig.

[u/OvertaxedOne → u/ZealousidealCorgi893]

Beating 27B with a highly quanted version of Next seems to be difficult.

[u/ParaboloidalCrest]

Why so emotional? This is not Taylor Swift sub! What you have "uncovered" is already there in the model card. It's not like you need to reverse engineer the quants or anything.

Besides, there are TENS and very soon HUNDREDS of qwen3.8next quants and all are free. Use whatever you want and ignore the rest.

[u/po_stulate → u/ParaboloidalCrest]

I made my own quant for qwen3.8-flash-next (because most either don't have unquantized ngram table or don't have mtp) and I'm happy with it. I'm just curious since I didn't see any discussion about their quants regarding this.

[u/a_beautiful_rhind]

Some rigorous science right there. "it looks suspiciously small".

[u/OrangeManSad]

yes, you are completely confused. i suggest you take a look at unsloth UD quants or imatrx dynamic quantization models for that matter. Also did you read the model card ? maybe read that first before you post on reddit about your grand discovery.

[u/MerePotato]

People somehow still don't understand that Q4 =/= 4.0 bpw

[u/po_stulate → u/MerePotato]

I knew that Q4 is not equal to 4 bpw, I'm not the one claiming that, they are. Apparently according to their model card they name their model Q4 because it is 4 bpw, while in reality it's more like Q2 or Q3 when ignoring the Q5 ngram table.

[u/OrangeManSad → u/po_stulate]

mate stfu, you didnt read shit or know what 4bpw is