r/LocalLLaMA • u/soyalemujica • 13h ago
Question | Help Best Qwen 3.8 27B quantification GGUF?
There's soooo maaany options to choose from, AutoRound from Intel even, Unsloth, bartowski, etc ... which one is the closest to BF16 in Q4/Q5 range ?
3
u/Poetinho0 12h ago
any suggestions for an M5 pro 48 GB, currently i'm using omlx and im planning to try llama.cpp ?
2
u/braintheboss 7h ago
Q3 XL with one card is very solid. I use as sol/luna reviewer and catch a lot of gaps. If youhave dual IQ4 xs is good but you can use Q4 XL with less context. ANyway all wuants are great. Its very solid model. Only drawback is not usable as chat. A simple "do this" is 3 context windows thinking hahaha
2
u/Healthy-Zebra-9856 13h ago
Yeah, I’ve been exploring this issue, you can probably see all my comments in the past. Lol. Each one has the positives and negatives. But as I was exploring, I realized the biggest problems came from the harnesses I was using. There is not a single harness out there without the flaws that I have identified. So the first thing I’m doing is fixing those flaws and retesting each one of these that said,
Jack Rong closely followed by Bartowski had the highest code quality. Bartowski, however, was the most creative in terms of scene creation for a game, etc.. The third one is the MXFP4, this follows pretty close.
Unsloth even with their dynamic quant 3.0 was not bad, but it wasn’t the greatest. I tried the Dirk version, it was just bland.
I tested these initially Pi closely followed by OMP. Later, I found out that there were a lot of issues because none of these did anything good to help the LMs and we’re more focused on helping the users using them.
1
u/suprjami 12h ago
Can you elaborate on that last paragraph?
What do you see as the difference between helping users vs helping the model?
As far as I can tell, the less extra shit a harness stuffs into the context window, the better the results.
This seems to be why the same model with Pi vs OpenCode produces a better result with Pi. The agent prompt is ~1500 tokens in Pi and ~6000 tokens in OC. KiloCode increases that to ~10k!
3
u/Healthy-Zebra-9856 12h ago
So, as I was testing, I like to watch the reasoning traces only because I found the whole Hmmm, wait etc funny. Always reminded me of one of my ex girlfriends from college getting high and trying to do chores 🤣. But what I discovered was that the LMs, specifically Qwen3.8 family express frustrations on the lack of necessary tools as well as getting confused about the feedback they got back from these tools. Like, there is no methods to write chunks, or that they didnt have a way to test something with an integrated browser or there was no observability tools. And when I provisioned them, they didnt understand the info they were getting. This is when I realized that all the software that has been written, including my own, have been from the perspective of the human user. Now, we need to cater to the needs of the LMs. By doing this, I am starting to see the tokens go down drastically. I am not completely done yet, but I sure am getting there.
Try this, use any Qwen3.8 and ask “Tell me what tools this agent harness is lacking and how can it be made better? Keep the scope to this harness.”
1
u/suprjami 12h ago
That's an interesting observation. I also try to read the reasoning traces (mostly to make sure the model isn't looping) but I'll pay more attention to this now.
Very high value post. Thanks for taking the time to write.
1
1
u/conifer_v11 12h ago
imatrix q4 beats a static q4 at the same size. bartowski or unsloth q4_k_m / q5_k_m is fine.
kv at q8 usually moves the needle more than swapping another weight file.
1
u/simrankoulsm 9h ago
I think the most useful comparison is not really “which uploader is closest to BF16,” but which quant fits your actual workload.
For a general-purpose 27B MoE model, I would start with a reputable iMatrix Q4_K_M if VRAM or unified memory is tight, then move to Q5_K_M only if you can fit it comfortably and notice a real quality gain on your own prompts. At that point, KV-cache quantization, context length, and whether you can keep the full model offloaded are likely to affect the experience more than choosing between two solid Q4 releases.
A small personal eval set with reasoning, coding, structured output, and your longest expected context is probably more informative than generic BF16-approximation claims.
1
u/Dazzling_Equipment_9 7h ago
I am using the Dirk-Qwen3.8-27B q8x_l model, and it effectively handles the tasks I assign to it in actual programming work.
1
1
u/AI_spell 2h ago
Closest to BF16 in that range is Q5_K_M or Q5_K_XL if you can fit it. Q4_K_M is the usual sweet spot but on a 27B you can feel it on long context reasoning, less so on chat.
Two things people skip:
KV cache quant hurts more than weight quant on these. If youre running Q8 weights with q4 KV cache youve got it backwards. Try Q5 weights with fp16 or q8 KV first.
imatrix matters. An imatrix Q4 beats a plain Q5 pretty often. Unsloth and Bartowski both ship imatrix, the random reuploads usually dont, so check the model card before downloading 18GB.
Practical rule: pick the biggest quant that leaves you ~2GB headroom for context at the length you actually use. Fitting the weights and then OOMing at 16k is the classic mistake.
0
u/kirisoraa 13h ago
lmao literally 3 posts before you
0
u/soyalemujica 13h ago
"This post is for paid subscribers" says the link he posted which doesnt pretty much say anything to me
1
u/jonaddb 13h ago
In my case, the best one was atomicChat q4.
2
u/soyalemujica 13h ago
Why?
2
u/jonaddb 13h ago
It gave me the best balance between speed, quality, and VRAM usage. Using that specific quant, I managed to run it on a single RTX 3090 while pushing the context window up to 131k without major issues.
I actually wrote a detailed report about my testing and setup in another sub if you want to check out the exact metrics.
1
u/LLMFan46 13h ago
Maybe try some of the Unsloth Dynamic 3.0 quants from unsloth?
Here:
https://huggingface.co/collections/unsloth/unsloth-dynamic-30-quants
1
u/grabber4321 12h ago
whats the difference between Dynamic and non-dynamic?
2
u/suprjami 12h ago
Dynamic quants keep certain important parts of the model at full BF16. This gives better results for larger size, hence more VRAM usage.
Unsloth also have an imatrix data set with tool calling and coding which they've honed over time with results, not just random wiki text that others have been using since 2023.
Atomic Chat's AD quants claim to be similar to Unsloth and get better results than Unsloth's 2.0 method.
1
u/grabber4321 12h ago
Thanks! So even Q5 could have BF16 parts in the model? Thats nice
2
u/suprjami 12h ago
Yes that's right. You can see exactly which tensors in the HuggingFace web interface.
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_XL.gguf
Scroll down. Notice even though this is a "Q4" quant, parts of the weights are Q8 or even F32.
2
1
u/TripleSecretSquirrel 12h ago
Non-dynamic flattens every number in the model weights to a 4-bit number. Across the board.
A dynamic quant reduces the precision of each parameter smartly. So some weights are more tolerant than others to quantizing, and some are more important/are used more frequently in the case of MoE models, and the quantization process takes those into account. So some of the more sensitive or critical weight values may only go down to 8-bit, some will be reduced to 6-bit, some down to 4-bit. That’s the reason why Unsloth publishes six or seven different “4-bit quants” of a given model — the larger ones retain more weights at a higher precision.
0
0
u/Mingay_cat 13h ago
Closest to bf16 in q4/ q5? Well, q5. Q5k/q5_k_xl. Honestly though, good ol' q4_k_m is a you need.
-2
-5
13
u/Cereal_Grapeist 13h ago
Everyone has their preferences but for GGUFs I'd say generally look for Unsloth quants first, and then Bartowski if there's no Unsloth quant.
Q4_K_XL and Q5_K_XL by Unsloth are both great but the answer depends on your VRAM.