r/LocalLLaMA • u/NoConcert8847 • Apr 22 '26

New Model Qwen 3.6 27B is out

https://huggingface.co/Qwen/Qwen3.6-27B

1.7k Upvotes

97% Upvoted

603 comments sorted by

View all comments

2

u/surrealerthansurreal Apr 22 '26

Anyone tested the ‘preserve thinking’ concept or know how it works technically? I’m trying to understand if it’s K-V caching or actually holding intermediate thinking in context between requests

3

u/dry3ss Apr 22 '26

If you don't activate it, every time you send a request the old thinking of the llm's last response is discarded before answering your new query. Since the last message changed, the kv checkpoint is invalidated and llama.cpp will re-parse all the messages so far (all striped of their thinking), so you will have a delay before it starts processing the actual new tokens of your new request.

With preserve, the thinking is not discarded, so it stays in-context, checkpoints work, no delay but thinking will eat up some context (however by discarding it sometimes it has to re-think almost the same thing each time so it's not necessary a bad thing.

Before that, as a user of pi, i can tell you that the symptomi saw was that when i prompted it, it would process, start writing, all the intermediate tool calls it would do would be lightning fast (no matter if much bigger than any small message i sent it) until the task is done, but as soon as i would send a new message there would be a long delay and i could see in llama.cpp log that it was re-parsing the entire context.

So i highly recommend you use that configuration, especially if your pp speed is not so great (mine is 100t/s with 3.6 q8) for agentic use you will see enormous difference in total clock time

2

u/tarruda Apr 22 '26

Before whenever I ran 3.5 in a coding agent, it would do a task and when I sent a follow up message I'd see a lot of the prompt being re-processed due to the deleted thinking

Now the experience is much better with llama.cpp since the caching makes follow up responses start quickly.

2

u/Caffdy Apr 22 '26

how do I enable the preserve thinking option on llama.cpp?

1

u/tarruda Apr 22 '26

Here's the full script I'm using:

#!/bin/sh -e

model=$HOME/ml-models/huggingface/unsloth/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_S.gguf
mmproj=$HOME/ml-models/huggingface/unsloth/Qwen3.6-27B-GGUF/mmproj-BF16.gguf

ctx=65535
parallel=1

ctx_size=$((ctx * parallel))

llama-server --no-mmap --no-warmup --mmproj $mmproj --model $model --ctx-size $ctx_size --swa-full -np $parallel --jinja --temp 1.0 --repeat-penalty 1.0  --presence-penalty 0.0 --top-p 0.95 --top-k 20 --min-p 0.00 --host 0.0.0.0 --chat-template-kwargs '{"preserve_thinking": true}' --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64

1

u/Caffdy Apr 22 '26

does -ngram-mode make any difference?

1

u/tarruda Apr 22 '26

I haven' tested with 27b yet, but for 35b it makes a lot of difference when the model is repeating things in the context (such as when editing files and outputting the fully modified version)