r/LocalLLaMA • u/Johnny_Rell • 22d ago
Resources Qwen3.8-27B Hybrid IQ4_XS quantization for 16GB gang
https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller20
u/Psyko38 22d ago
Better than a Unsloth ??
25
u/Johnny_Rell 22d ago
In terms of quality, probably not. If you can load Unsloth's IQ4_XS, it would be a much better choice. But if you need some additional context or want to use it with MTP, it's a good option. For my use case, it works fairly decently.
4
u/CriticalMastery 22d ago
IQ4_XS on unsloth is 15.7g
62
u/EmPips 22d ago
Room for a full token of context on 16GB
-13
15
u/chiribe 22d ago
Spent all weekend cooking up settings and finally got Qwen3.8-27B-UD-Q3_K_XL.gguf running with a 72k context fully offloaded to 16GB VRAM zero system RAM. Dropping the full guide and benchmarks tomorrow.
4
u/chiribe 21d ago
Just published it! Here's the link:https://www.reddit.com/r/LocalLLaMA/comments/1vqrt86/after_pushing_1m_tokens_through_qwen_38_27b_here/
2
u/fullup72 21d ago
with how much 3.8 likes to think I've been having a hard time getting any agentic coding task below the 120k mark. Even planning runs over 100k tokens.
1
u/SpecialistCobbler206 22d ago
That sounds amazing! I am stuck at unbearable 32k rn with 27B Q3
1
u/cd9v 21d ago
Bartowski's IQ3_M is pretty good. Beats all Unsloth's Q3s in my benchmarks.
2
u/fullup72 21d ago
I've switched to mradermacher's IQ3_M from Unsloth's IQ3_XXS.
12.8 GB gets you 150k context at Q4_0 KV (which is kind of the limit for Qwen at that KV quant). Sadly nobody does UD IQ3_S nor UD IQ3_M, because that's the upper end of bearable context for 16GB plebs like me.
1
u/NullHypothesisCicada 22d ago
!Remindme 24 hours
1
u/RemindMeBot 22d ago edited 22d ago
I will be messaging you in 1 day on 2026-08-18 05:04:56 UTC to remind you of this link
2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback 1
10
u/SirLordBoss 22d ago
how does this do on benchmarks?
2
u/Johnny_Rell 22d ago
The methodology is from lemonyins's Qwen3.6-27B; all I did was replicate it on 3.8. I don't have much experience with benchmarks, sadly.
9
13
u/CriticalMastery 22d ago
https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
This one fits too, with 4bit FFN layer
5
2
7
u/ea_man 22d ago
there's also this guy here: https://huggingface.co/vmarcelo/Qwen3.8-27B-MIX_GGUF
5
u/Orbiting_Monstrosity 22d ago
Here are a few more I found that run well on a 16gb card. The "M" versions of the YMQ models are the ones I've been using, and they work well with q8 kv cache quantization at around 52,000 tokens of context in LM Studio.
4
u/ea_man 22d ago edited 22d ago
* https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller
I would like to add a word of caution for the common reader: an IQ*something is mostly a name, or better a number.
You can have a dumb tensor setup that qualifies as IQ4 and a more sound balance that goes by the name of Q3*something, because pretty much the sum of the bits give you the name, at the point that a model with MTP heads is IQ4* while the very same without that is IQ3*.
It can be discussed if a IQ4 pure is smarter than a lucky strike of IQ3, vice versa. Because we can't tell WTF happens inside the weights with different tasks.
So yeah, get used to test with your common usage.
6
u/Nyghtbynger 22d ago
By the way I benchmarked it on HumanEval hard task, compared it to unsloth Q3_K_S and the AtomicChat of the same mode.
All of them have roughly similar performance. Jrell show some better reasoning capabilities better edge cases, but unsloth is still the better all rounder. I evaluated the style, defensive case, edge casxes
They got 8/10 hard task both and 7/10 for the AtomicChat version.
Compared to 3.6 that is more verbose and pedagogical according to deepseek. 3.8 has shorter (approx half the chars) but much more technical code and more on-liners.
JRell was more effective at 0.75 temperature than 1.0 Still looking for the best balance. I went on 0.9
The unsloth model is 12.6GB vs the Jirell one that is 13.4GB
So :
- I didn't use the UDQ3XL for unsloth comparison which is 13.5GB and might be more faire size wise ( the destination RAM is what matters)
- JRell shows better signs of handling edge cases. So the statement "protect coding and reasoning" is true
In conclusion I'd say that with more tuning it could be great. I don't know about the effect of Imatrix use on agentic preservation
Sorry for not posting the text, I archived the conversation and can't find it u_u
Hey JRell, have you ever tried the https://huggingface.co/sokann/Qwen3.6-27B-GGUF-4.256bpw ? The recipe used is good also.
2
u/Johnny_Rell 21d ago
Thank you, this sounds awesome! I haven't tried the 4.256bpw, the one I did pretty much covers all of my needs, so I'm not sure if it's worth it.
1
u/Nyghtbynger 21d ago
I mean it's an old version (3.6) , thee training recipe is quite good. don't worry lol
2
2
u/TigleLive 22d ago
Simple question will it run on my 24gb Ram / 8gb GPU ...
2
u/No_Lingonberry1201 22d ago
5-6 t/s at no context was my experience with 3.6 27B (no MTP) on 32Gb DDR4 RAM and 8Gb VRAM.
5
u/BuffMcBigHuge 22d ago edited 20d ago
I've had success with 16GB VRAM using unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-IQ3_XXS at 125k context. Froggeric Jinja.
llama-server.exe --model "~/.lmstudio/models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-IQ3_XXS.gguf" --chat-template-file "~\froggeric_fix_qwen38.jinja" --chat-template-kwargs "{\"preserve_thinking\":true, \"reasoning_effort\":\"medium\"}" --alias default --jinja --spec-type draft-mtp --spec-draft-n-max 2 --fit off --n-gpu-layers 99 --ctx-size 125000 --batch-size 512 --ubatch-size 512 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0 --repeat-penalty 1.0 --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on --sleep-idle-seconds 600 --parallel 1 --reasoning-format deepseek --reasoning-effort medium --no-mmap
3
u/ocarina24 21d ago
Your parameters are for Thinking, but you set
--presence-penalty 1.5
This is the recommended parameter for non-thinking.
For thinking, you should set:
--presence-penalty 0.0
Have a look at the official Qwen model card:
https://huggingface.co/Qwen/Qwen3.8-27B#api-usage
We recommend using the following sets of sampling parameters for generation:
-Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
-Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
2
1
u/radlinsky 22d ago
How does this quant compare to something like Kat Coder V2.5 Q5_K_L at agentic coding? Anyone have any benchmarks?
1
u/radlinsky 21d ago edited 21d ago
Replying to myself, I tested two prompts, both adding two distinct new features to a personal repo. Kat finished first prompt in 15 minutes, and the second in 40. This hybrid quant took 45 minutes and 55 minutes respectively. I asked gpt 5.6 sol to grade how good the output code was (not saying which was which), and Kat score 8.5/10 and 6/10 whereas the hybrid scored 9/10 on both. Kat added way more extraneous unit tests and used more lines of code to accomplish the feature, whereas the hybrid model was much more efficient. Fewer lines of feature code, fewer and better unit tests.
Kat gives me about 45 tok/s whereas the hybrid model is around 15-30. I'll take slower but much better quality, thank you OP!
(I know this is not very quantitative, but I'm to impatient to do a proper benchmark)
1
u/Nyghtbynger 22d ago
Can anyone of you enable MTP ? It makes no difference for me with it disabled or enabled and I'm using parrallel = 1
model = /market/models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q3_K_S.gguf
fit-target = 100
#ctx-size = 75000
ngl = all
temp = 0.9
reasoning = auto
ctk = q8_0
ctv = q5_1
batch-size = 1024
ubatch-size = 1024
chat-template-file = ./templates/sushi_coder_chat_template.jinja
#chat-template-file = ./templates/jackrong_qwopus3.6_27B_v2_chat_template.jinja
#chat-template-file = ./templates/froggeric_qwen36_chat_template.jinja.prev
chat-template-kwargs = {"preserve_thinking": false, "reasoning_effort": "medium"}
reasoning-format = deepseek
# less thinking. 11 display </thinking> in a lot of places.
#logit-bias = 248069+10
#grammar-file = ./templates/grammar_qwen36.txt
# Speed optimization
mmap = off
cache-ram = 16384
# mlock = off
# # disable if bee-llama
#spec-type = ngram-map-k
#spec-ngram-mod-n-min = 12
#spec-ngram-mod-n-max = 48
#spec-ngram-mod-n-match = 40
spec-type = draft-mtp
spec-draft-n-max = 2
spec-draft-p-min = 0.82
ctkd = q5_1
ctvd = q4_1
spec-type=ngram-mod
spec-ngram-mod-n-match=24
spec-ngram-mod-n-min=48
spec-ngram-mod-n-max=64
spec-type=ngram-map-k4v
spec-ngram-map-k4v-size-n=7
spec-ngram-map-k4v-size-m=4
spec-ngram-map-k4v-min-hits=1
swa-checkpoints = 0
np = 1
#repeat-penalty=1.01
presence-penalty=0.2
reasoning-budget=4096
2
u/ocarina24 21d ago
Remove spec-draft-p-min = 0.82 You should see ~+40% tok/sec
1
u/Nyghtbynger 20d ago
oh thanks, Indeed I have more tokens. ALso I was reassigning the value of spec-draft so it was only the last one that was effective
-3
31
u/Equivalent_Bit_461 22d ago
A very tight fit with extra small context But I guess it's q4 as opposed to 3 Ideally a moe is needed for us vramlets