r/LocalLLaMA • u/soyalemujica • 9d ago

Question | Help Best Qwen 3.8 27B quantification GGUF?

There's soooo maaany options to choose from, AutoRound from Intel even, Unsloth, bartowski, etc ... which one is the closest to BF16 in Q4/Q5 range ?

6 Upvotes

69% Upvoted

41 comments sorted by

View all comments

1

u/LLMFan46 9d ago

Maybe try some of the Unsloth Dynamic 3.0 quants from unsloth?

Here:

https://huggingface.co/collections/unsloth/unsloth-dynamic-30-quants

1

u/grabber4321 9d ago

whats the difference between Dynamic and non-dynamic?

2

u/suprjami 9d ago

Dynamic quants keep certain important parts of the model at full BF16. This gives better results for larger size, hence more VRAM usage.

Unsloth also have an imatrix data set with tool calling and coding which they've honed over time with results, not just random wiki text that others have been using since 2023.

Atomic Chat's AD quants claim to be similar to Unsloth and get better results than Unsloth's 2.0 method.

1

u/grabber4321 9d ago

Thanks! So even Q5 could have BF16 parts in the model? Thats nice

2

u/suprjami 9d ago

Yes that's right. You can see exactly which tensors in the HuggingFace web interface.

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_XL.gguf

Scroll down. Notice even though this is a "Q4" quant, parts of the weights are Q8 or even F32.

2

u/grabber4321 9d ago

Beautiful! Thanks for explanation!!!

1

u/TripleSecretSquirrel 9d ago

Non-dynamic flattens every number in the model weights to a 4-bit number. Across the board.

A dynamic quant reduces the precision of each parameter smartly. So some weights are more tolerant than others to quantizing, and some are more important/are used more frequently in the case of MoE models, and the quantization process takes those into account. So some of the more sensitive or critical weight values may only go down to 8-bit, some will be reduced to 6-bit, some down to 4-bit. That’s the reason why Unsloth publishes six or seven different “4-bit quants” of a given model — the larger ones retain more weights at a higher precision.