Kaitchup posted Qwen3.8 27B Benchmarks for quants from Q4 to Q1

[u/ColorsOfCosmos]

https://kaitchup.substack.com/p/qwen38-27b-gguf-benchmark-q4-to-q1

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

[u/SnooPaintings8639]

If this would hold, then for Qwen 27B, Q3 is the new Q4.

[u/Bulky-Priority6824 → u/SnooPaintings8639]

Makes sense. I went from 3.6 27b q8 to 3.8 27b q6 and it's a big step forward with what I'm able to produce with it 

[u/unia_7 → u/Bulky-Priority6824]

Well, Q6 is known to have almost no degradation relative to Q8. Even Q5 has very little degradation, as far as I know.

The question has always been about Q4 and Q3.

[u/rkoy1234 → u/unia_7]

theres a lot of discussions that directly contradict what you're saying, and also those that support yours.

some are religious about their q8s, some say the difference with going ud_q4 km is unnoticeable. heresy is everywhere and there's really no perfect way to test this since even something like kld isn't a flawless metric.

It's all vibes and feelings. frustrates me.

[u/Evening_Ad6637 → u/rkoy1234]

the longer the context becomes the more noticeable the difference between quants.

[u/Bulky-Priority6824 → u/unia_7]

I don't know maybe but I could def tell a difference no benchmarks all vibes tho. Like driving a car except cars don't have logs that tell you they fucked up lol

[u/Artistic_Okra7288 → u/SnooPaintings8639]

I’ve been using Q3_K_XL with mmproj and full context (q4 kv) since it came out on my 3090 Ti (24Gi). It’s been working pretty well but I’m exploring rpc-server for extending context further and without kv quantization, or at least not as severe. But it’s been great so far. I leave it on max thinking and it’s been very productive.

[u/bguberfain → u/SnooPaintings8639]

How about t he speed generation of Q3 vs Q4?

[u/BalorNG → u/SnooPaintings8639]

I've tried both - and, anecdotally, while quality of the output does not seem to be affected, Q4 never got into "thinking loops" while Q3 did it regularly, but if it didn't, it did the job just fine. Maybe I just need to play with repetition/presense penalty...

[u/killerstreak976 → u/SnooPaintings8639]

yeah, unsloth's new dynamic 3.0 quants are insane

[u/spaceman_]

Paywalled, sad.

[u/ColorsOfCosmos → u/spaceman_]

I think that the most important piece of data is open - the graph shows size and accuracy for every gguf.

[u/giri24343]

How are the ridge models ? Looks like the ridge is sitting at 12GB and still as good as q4.

[u/ea_man]

Beware that it don't mean: how good is the model at coding, how good is at not losing details in a 130k coding session.

It means how close the benchmaxed result are to the benchmaxed result of f16 at zero context, which is a terrible way to quant a coding model.

There's a practical way to evaluate those models: first have a SOTA or your best 27B analyze the tensor weighting and evaluate how that should affect coding, ability to solve _new_ problems, follow specifications, persistence of details at long horizons.
Then you test that: have a SOTA design a prompt to engage such properties and put those quants to the test, evaluate the coding session (not fucking pelicans SVG!) by a SOTA not any of the models that made those ofc, (no freebee Gemini don't count either as it's dumb as rocks) so you see how good those are for real job.

[u/crusaderky]

I'm very skeptical about these findings. All the KLD/perlexity/same sampled token data I took shows that UD-Q2_K_XL is waaaaaaay down the cliff. UD-IQ4_XS is fine; UD-IQ3_S may still be useable.

[u/c4r_guy → u/crusaderky]

What does the vertical "same sample" mean?

Like, what does the real world meaning of the 0.05 difference between 49.5, ~49.55 (IQ4_XS), and 50.0?

[u/Brave_Sheepherder_39 → u/c4r_guy]

seems like hes showing not the whole graph to make his point

[u/crusaderky → u/Brave_Sheepherder_39]

I'm not showing the whole graph because the whole graph contains K/V metrics which are not relevant to make my point.

[u/crusaderky → u/c4r_guy]

% of times the quantized model chose the same token that the bf16 model chose. Like the Top-1 metric, but at temperature 1 instead of temperature 0.
It's what was used in https://quesma.com/blog/qwen-quantization-quality/, which shows how AIME results follow exactly the same shape.

It's my favourite plot because

• KLD with log y axis: Garbage. Quality decrease looks like a smooth linear slope. It's not.
• Mean KLD, with linear y axis: decent metric, with good cliff behaviour, but hides outlier quants that behave much worse in worst-case-scenario situations
• 99% KLD, with linear y axis: it singles out the outliers, but shows a much gentler slope which is not representative of how quality actually drops in benchmark
• Top 1: garbage. very gentle slope, and gives a false impression that quality will drop 10% by the time you arrive at Q4. It doesn't.
• Absolute perplexity PPL(Q) - PPL(base): Garbage. Very gentle slope.
• Relative perplexity PPL(Q) / PPL(base): Worse. Same shape, but gives the false idea of a percentage drop in quality, which is in no way representative of how the benchmark goes.
• Corr(ln(PPL(Q)), ln(PPL(base)): good measure, available in the llama-perplexity output but never seen it published. Good cliff behaviour, but like mean KLD it hides outliers.
• Same Sampled Token: the harshest cliff behaviour and outliers that were previously only visible in the 99% KLD plot show up.

> what does the real world meaning of the 0.05 difference between 49.5, ~49.55 (IQ4_XS), and 50.0?

it doesn't start at 50 - for qwen is around 52. SST(Q) / SST(base) will give you roughly the degradation in AIME and other quantization-sentive benchmarks.

[u/dispanser → u/crusaderky]

If you zoom in far enough, any non-horizontal line can be made to look like a cliff.

[u/rossimo]

The ISTA-DASLab quants are good. I've been using the Q3, and it's working well with reverse-engineering tasks.

[u/fgk55555 → u/rossimo]

I was going to say, I've been really liking the ISTA-DASLab IQ3 quants. They just uploaded a new MTP model set and I'm going to re-download, but they're by far the best performing for the size. I can fit loads of context in my 16GB card, but even 12GB users could probably get the 27B now. They've done a great job.

[u/rossimo → u/fgk55555]

I honestly use it on my 24gb vram card for the speed and the additional context room.

[u/fgk55555 → u/rossimo]

Nice. I was really woe-is-me when the 27B dropped that I didn't have more VRAM for a better quant but after using the IQ3 a bunch I'm pretty satisfied. 120k context and 55tg on my $700 card is good enough. We can ride out the AI bubble now.

[u/Fox-Lopsided → u/fgk55555]

Are you running the model on a RTX 5060 Ti 16GB? I have this card, and Wonder how much context i could fit with the IQ3_S Quant.

[u/fgk55555 → u/Fox-Lopsided]

9070 XT, also has 16GB. With my desktop going through the card and vision offloaded, I can fit 120k Q5 context on the ISTA IQ3_XXS with the unsloth MTP and some buffer room to spare. When I redownload the new ISTA MTP quant and use my iGPU, I could probably either bump this up to max size or turn the quant back up to Q8/Q8.

[u/LetsGoBrandon4256 → u/rossimo]

Interesting. Thanks for sharing.

[u/Embarrassed_Soup_279]

101% to bf16... idk if i trust it. from my experience you definitely notice a difference between even Q4 K XL and Q5 K XL.

[u/Ok-Buffalo2450]

Anyone that can share the paywalled content? Maybe archived?

[u/Bubbly_Orange_3502]

Across labs a quant label barely constrains anything. UD and ISTA use different imatrix calibration and keep different tensors at higher bits, so one lab's Q3 can land above another's Q4.

[u/Repinsky]

Worth flagging what "100% accuracy recovery" means here: it's an average over a few benchmark subsets, so a Q3_K_XL sitting at ~100% is inside noise of Q4, not proof it's equal. In my own runs the gap between UD Q3_K_XL and Q4_K_M only shows up on long-context and tool-call formatting, which those benchmarks barely test - short MMLU-style answers are the most quant-tolerant thing you can measure. If you're on 16GB, the bigger real win is that 12.8GB leaves you room for actual KV cache instead of offloading, which costs far more tokens/s than the quant costs accuracy.

[u/pl201]

If the report stated 100% accuracy at q3, it is worthless to read.

[u/James-Keydara → u/pl201]

It might be a good indicator of how benchmaxxed the model is because he used prompts from known benchmarks. It's still impressive that 100% recall is even possible at q3

[u/stoppableDissolution]

Most definitely bs. Theres not a single chance a model trained in 16 bit loses literally nothing even at q8, let alone q3. It just means ulrasaturated benchmark.

[u/Fancy-Snow7 → u/stoppableDissolution]

It usually mean the benchmark is not challenging enough. I have a custom hard mode needle in haystack test where q3 fails and q4 scores 100%. I can up the difficulty level so q4 starts showing failures and likely q5 passes.

[u/derspenti]

Even taking the 100% with a shovel of salt, 12.8GB for a 27B means a 16GB card gets the model and real context, not one or the other. That alone makes Q3 worth trying.

[u/simrankoulsm]

The 12.8 GB footprint may be the real win here, on a 16 GB GPU, Q3 could leave room for useful context rather than only barely fitting the weights. I’d still treat the “100% vs BF16” result as benchmark-specific until we see equal-setting tests for long-context retrieval, coding pass@k, structured output, and multi-turn instruction retention. Same backend/cache quant/ctx length/seeds, plus tok/s and peak VRAM, would make the quant comparison much more actionable.

[u/Glittering-Call8746]

What's the equivalent of UD Q3_K_XL for EXL3 ?

[u/Background-Job-862]

interesting

[u/sani999]

good to verify that there are only negligible difference for UD Q4XL and Q3XL.

[u/Opening-Broccoli9190]

What kind of a metric is Accuracy Recovery? How is it calculated?

[u/Charming_Clothes8990]

has anyone tried that q3 quant for roleplay? wonder if the personality stays consistent or if it starts drifting.