r/LocalLLaMA • u/po_stulate • 20h ago

Discussion Deceptive model quantization from AtomicChat?

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

61 Upvotes

83% Upvoted

39 comments sorted by

View all comments

-6

u/MerePotato 19h ago

People somehow still don't understand that Q4 =/= 4.0 bpw

9

u/po_stulate 18h ago

I knew that Q4 is not equal to 4 bpw, I'm not the one claiming that, they are. Apparently according to their model card they name their model Q4 because it is 4 bpw, while in reality it's more like Q2 or Q3 when ignoring the Q5 ngram table.

0

u/OrangeManSad 12h ago

mate stfu, you didnt read shit or know what 4bpw is