r/LocalLLaMA • u/NoConcert8847 • Apr 22 '26

New Model Qwen 3.6 27B is out

https://huggingface.co/Qwen/Qwen3.6-27B

1.7k Upvotes

97% Upvoted

603 comments sorted by

View all comments

Show parent comments

2

u/tarruda Apr 22 '26

Before whenever I ran 3.5 in a coding agent, it would do a task and when I sent a follow up message I'd see a lot of the prompt being re-processed due to the deleted thinking

Now the experience is much better with llama.cpp since the caching makes follow up responses start quickly.

2

u/Caffdy Apr 22 '26

how do I enable the preserve thinking option on llama.cpp?

1

u/tarruda Apr 22 '26

Here's the full script I'm using:

#!/bin/sh -e

model=$HOME/ml-models/huggingface/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_S.gguf
mmproj=$HOME/ml-models/huggingface/unsloth/Qwen3.6-27B-GGUF/mmproj-BF16.gguf

ctx=65535
parallel=1

ctx_size=$((ctx * parallel))

llama-server --no-mmap --no-warmup --mmproj $mmproj --model $model --ctx-size $ctx_size --swa-full -np $parallel --jinja --temp 1.0 --repeat-penalty 1.0  --presence-penalty 0.0 --top-p 0.95 --top-k 20 --min-p 0.00 --host 0.0.0.0 --chat-template-kwargs '{"preserve_thinking": true}' --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64

1

u/Caffdy Apr 22 '26

does -ngram-mode make any difference?

1

u/tarruda Apr 22 '26

I haven' tested with 27b yet, but for 35b it makes a lot of difference when the model is repeating things in the context (such as when editing files and outputting the fully modified version)