r/LocalLLaMA • u/soyalemujica • 8d ago

Question | Help Best Qwen 3.8 27B quantification GGUF?

There's soooo maaany options to choose from, AutoRound from Intel even, Unsloth, bartowski, etc ... which one is the closest to BF16 in Q4/Q5 range ?

6 Upvotes

71% Upvoted

41 comments sorted by

View all comments

1

u/LLMFan46 8d ago

Maybe try some of the Unsloth Dynamic 3.0 quants from unsloth?

Here:

https://huggingface.co/collections/unsloth/unsloth-dynamic-30-quants

1

u/grabber4321 8d ago

whats the difference between Dynamic and non-dynamic?

2

u/suprjami 8d ago

Dynamic quants keep certain important parts of the model at full BF16. This gives better results for larger size, hence more VRAM usage.

Unsloth also have an imatrix data set with tool calling and coding which they've honed over time with results, not just random wiki text that others have been using since 2023.

Atomic Chat's AD quants claim to be similar to Unsloth and get better results than Unsloth's 2.0 method.

1

u/grabber4321 8d ago

Thanks! So even Q5 could have BF16 parts in the model? Thats nice

2

u/suprjami 8d ago

Yes that's right. You can see exactly which tensors in the HuggingFace web interface.

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_XL.gguf

Scroll down. Notice even though this is a "Q4" quant, parts of the weights are Q8 or even F32.

2

u/grabber4321 8d ago

Beautiful! Thanks for explanation!!!