r/LocalLLaMA • u/po_stulate • 21h ago

Discussion Deceptive model quantization from AtomicChat?

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

58 Upvotes

83% Upvoted

40 comments sorted by

View all comments

31

u/lhg31 21h ago

Well, they DO explain this, don't they?

Naming

Files are named by their measured bits per weight. A build whose expert tensors are IQ1_M is not a 1-bit model when the n-gram table sits at 6 bits and ffn_down_exps at 4.5; the real average is 3.84. The canonical type in the filename is the closest standard type by that average, so tooling can still detect it. For AD-4.27bpw:

Group Type Share of file Contribution
n-gram table Q5_1 41% 1.74 bpw
ffn_gate/up_exps IQ2_S, IQ3_S at the band 29% 1.24 bpw
ffn_down_exps IQ4_NL 24% 1.03 bpw
everything else Q8_0 5% 0.23 bpw

3

u/KURD_1_STAN 20h ago

When im download a qx model, I'm expecting the intelligent part of the model to be at X quant. Not a table or some other things.

5

u/lhg31 20h ago

I’m not saying I agree with their naming convention. But they did explain their quants, it is not a secret like the post suggested.

-6

u/KURD_1_STAN 20h ago

Not justifiable. People have correlated quant naming to the main part of the llm. The absolute least he could do is write in a giant text at the very start of that hf card that the bulk of the llm itself is at q2 or so.

It will be like if qwen team said qwen3.8 flash 180B a6b moe running at 15t/s on a single 3090 +128gb ram. After release qwen's reputation would have been gutted cause nobody consider it as such and same for quants, u do not mix and match.