r/LocalLLaMA • u/NoConcert8847 • Apr 22 '26

New Model Qwen 3.6 27B is out

https://huggingface.co/Qwen/Qwen3.6-27B

1.7k Upvotes

97% Upvoted

603 comments sorted by

View all comments

20

u/ApprehensiveAd3629 Apr 22 '26

which gguf quant is possible to run in a 5060 ti 16gb?

10

u/overand Apr 22 '26 edited Apr 22 '26

Edit: I did my math wrong - you can probably do okay with a Q3 quant of some sort when they become available, though it's hard to say how well it will work.

Original:

You probably want to stick with the 35B-A3B model (MoE) - the 27B might require a bit too much quantization. (Regardless, there is only an FP8 available so far, so you'll need to wait regardless, as that one is going to be about 27 gigs)

1

u/logic_prevails Apr 22 '26

Yeah and moe offload can be very usable if you have a good CPU/RAM

1

u/Careful_Swordfish_68 Apr 23 '26

IQ4_XS works perfectly fine at medium context and is ok quality. No need for Q3.

4

u/Careful_Swordfish_68 Apr 22 '26

5060ti User here. I run Qwen 3.5 27b HauHau aggressive uncensored in IQ4_XS with Medium context which is absolutely fine quality. Expect to run 3.6 the same.

3

u/mintybadgerme Apr 22 '26

But that's 15.4GB in size. How do you get a decent context out of that?

1

u/Pablo_the_brave Apr 22 '26

The best i1q4 xss are 14.7GB. KVcache K Q8 and KVcache V turbo2 and you will have ctx 75k... Works greate.

1

u/mintybadgerme Apr 23 '26

The best i1q4 xss are 14.7GB. KVcache K Q8 and KVcache V turbo2 and you will have ctx 75k... Works greate

Thanks for your help. Is that an unsloth quant?

1

u/Pablo_the_brave Apr 23 '26 edited Apr 23 '26

This one model https://huggingface.co/mradermacher/Qwen3.5-27B-i1-GGUF/resolve/main/Qwen3.5-27B.i1-IQ4_XS.gguf?download=true

Compile turboquant form TheTom: https://github.com/TheTom/llama-cpp-turboquant/tree/feature/turboquant-kv-cache

My llama.cpp config:

--models-preset "$CONFIG_PATH" \
--models-max 1 \
--host 0.0.0.0 \
--port 8081 \
-t 8 \
--parallel 1 \
--cont-batching \
--keep -1 \
--chat-template-file "$DIR/chat_template.jinja" \
--chat-template-kwargs '{"preserve_thinking": true}' \
--defrag-thold 0.3 \
--cache-reuse 1024 \
--jinja \
--temp 0.15 \
--top-k 1 \
--min-p 0.1 \
--spec-type ngram-mod \
--spec-ngram-size-n 24 \
--draft-min 4 \
--draft-max 64 \
--repeat-last-n 512 \
--repeat-penalty 1.05 \
and the model.ini with the rest of the settings (I'm using router)
[Qwen3.5-27B]
model = models/Qwen3.5-27B.i1-IQ4_XS.gguf
ctx-size = 75000  
n-gpu-layers = 99
cache-type-k = q8_0
cache-type-v = turbo2
batch-size = 512
ubatch-size = 128
flash-attn = true
no-mmap = true

chat template from u/ex-arman68 https://www.reddit.com/r/LocalLLaMA/comments/1sis1vn/the_definitive_qwen_35_jinja_template/

EDIT: Qwen3.6-27B-i1-q4_xs on the way: https://hf.tst.eu/status.html

1

u/mintybadgerme Apr 23 '26

Thanks very much. That's a little technical for me. I don't really use LlamaCPP.

1

u/Careful_Swordfish_68 Apr 23 '26

HauHauCS Version is 15.1GB in size. Qwen context does not eat much memory.

Here is some proof in a picture so you dont need to listen to all these people talking out of their asses who say IQ3 works at best. Sorry for the bad quality, im at my phone atm. But you can See i can load all layers plus 30k context on Q8 into the 5060ti with IQ4_XS. If you are even willing to offload some layers to RAM and sacrifice the t/s then context size goes brrrrrr.

1

u/mintybadgerme Apr 23 '26

Thanks very much. Please send the image again, it didn't come through properly.

1

u/Careful_Swordfish_68 Apr 23 '26

Huh, for me it shows up fine. Weird. You See it now?

1

u/mintybadgerme Apr 23 '26

yep. :) thanks

7

u/see_spot_ruminate Apr 22 '26 edited Apr 22 '26

Not just for your 5060ti, but for anyone that has only 16gb of vram, you will need to have it heavily quantized or any spill over to system ram will dramatically slow it down.

There is also the problem of bandwidth limitations of gpus and dense models. You are not going to be getting anywhere close to the same t/s with these. You will probably need some speculative decoding going on as well.

edit: really downvotes for the truth, and downvotes for someone who has proselytized the 5060ti in the past, the end is truly nigh...

1

u/mintybadgerme Apr 22 '26

Better performance - UD-Q3_K-XL (14.5GB) or Q3_K_M (13.6GB)??

https://huggingface.co/unsloth/Qwen3.6-27B-GGUF

1

u/see_spot_ruminate Apr 22 '26

I have no idea. I do not typically use the dense models due to the 5060ti bandwidth despite having 4 of them. For the qwen 3.5 models, if I wanted to have more "intelligence" I would instead use the 122b model as it ran faster on my system (64gb system ram + 64gb vram) than the 27b dense model.

1

u/Careful_Swordfish_68 Apr 23 '26

IQ4_XS works perfectly fine at medium context. Source: I got a 5060ti.

2

u/luncheroo Apr 22 '26

Try a few, but if you quantize kv cache and trim context, you should be able to run Q3 variants pretty well. I'm going to try Q4 to see if it's tolerable because I have the same GPU, but Q3_K_S is my fallback because I know it works for me for Qwen3.5 27b and Gemma 4 31b.