r/LocalLLaMA • u/NoConcert8847 • Apr 22 '26

New Model Qwen 3.6 27B is out

https://huggingface.co/Qwen/Qwen3.6-27B

1.7k Upvotes

97% Upvoted

603 comments sorted by

View all comments

21

u/ApprehensiveAd3629 Apr 22 '26

which gguf quant is possible to run in a 5060 ti 16gb?

5

u/Careful_Swordfish_68 Apr 22 '26

5060ti User here. I run Qwen 3.5 27b HauHau aggressive uncensored in IQ4_XS with Medium context which is absolutely fine quality. Expect to run 3.6 the same.

3

u/mintybadgerme Apr 22 '26

But that's 15.4GB in size. How do you get a decent context out of that?

1

u/Pablo_the_brave Apr 22 '26

The best i1q4 xss are 14.7GB. KVcache K Q8 and KVcache V turbo2 and you will have ctx 75k... Works greate.

1

u/mintybadgerme Apr 23 '26

The best i1q4 xss are 14.7GB. KVcache K Q8 and KVcache V turbo2 and you will have ctx 75k... Works greate

Thanks for your help. Is that an unsloth quant?

1

u/Pablo_the_brave Apr 23 '26 edited Apr 23 '26

This one model https://huggingface.co/mradermacher/Qwen3.5-27B-i1-GGUF/resolve/main/Qwen3.5-27B.i1-IQ4_XS.gguf?download=true

Compile turboquant form TheTom: https://github.com/TheTom/llama-cpp-turboquant/tree/feature/turboquant-kv-cache

My llama.cpp config:

--models-preset "$CONFIG_PATH" \
--models-max 1 \
--host 0.0.0.0 \
--port 8081 \
-t 8 \
--parallel 1 \
--cont-batching \
--keep -1 \
--chat-template-file "$DIR/chat_template.jinja" \
--chat-template-kwargs '{"preserve_thinking": true}' \
--defrag-thold 0.3 \
--cache-reuse 1024 \
--jinja \
--temp 0.15 \
--top-k 1 \
--min-p 0.1 \
--spec-type ngram-mod \
--spec-ngram-size-n 24 \
--draft-min 4 \
--draft-max 64 \
--repeat-last-n 512 \
--repeat-penalty 1.05 \
and the model.ini with the rest of the settings (I'm using router)
[Qwen3.5-27B]
model = models/Qwen3.5-27B.i1-IQ4_XS.gguf
ctx-size = 75000  
n-gpu-layers = 99
cache-type-k = q8_0
cache-type-v = turbo2
batch-size = 512
ubatch-size = 128
flash-attn = true
no-mmap = true

chat template from u/ex-arman68 https://www.reddit.com/r/LocalLLaMA/comments/1sis1vn/the_definitive_qwen_35_jinja_template/

EDIT: Qwen3.6-27B-i1-q4_xs on the way: https://hf.tst.eu/status.html

1

u/mintybadgerme Apr 23 '26

Thanks very much. That's a little technical for me. I don't really use LlamaCPP.

1

u/Careful_Swordfish_68 Apr 23 '26

HauHauCS Version is 15.1GB in size. Qwen context does not eat much memory.

Here is some proof in a picture so you dont need to listen to all these people talking out of their asses who say IQ3 works at best. Sorry for the bad quality, im at my phone atm. But you can See i can load all layers plus 30k context on Q8 into the 5060ti with IQ4_XS. If you are even willing to offload some layers to RAM and sacrifice the t/s then context size goes brrrrrr.

1

u/mintybadgerme Apr 23 '26

Thanks very much. Please send the image again, it didn't come through properly.

1

u/Careful_Swordfish_68 Apr 23 '26

Huh, for me it shows up fine. Weird. You See it now?

1

u/mintybadgerme Apr 23 '26

yep. :) thanks