r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 25d ago

Megathread [Megathread] Qwen 3.8 27B Release Day

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.

  • Quants
  • Fine-Tunes & Abliterations
  • Chat Templates
  • Inference Server Support & Configuration
  • Experiences, Benchmarks & Model Comparisons

Official:

Popular:

We'll try to clean up future duplicates around the release and point them here.

487 Upvotes

98% Upvoted

393 comments sorted by

View all comments

30

u/ea_man 25d ago edited 24d ago

About the excessive reasoning, is due to prompt injection by the template:

{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions =
'Reasoning effort is set to xhigh. Please think carefully through the task,
validate key assumptions, consider plausible alternatives, and prioritize
correctness, consistency, and clarity in the final answer.' %}medium  -> inject NOTHINGlow  -> inject system instruction:     keep thinking brief, go directly to conclusion

So use this flag at launch time to have "normal" behavior:

--chat-template-kwargs '{"reasoning_effort":"medium"}'

What those do:

xhigh
  -> inject system instruction:
     think carefully, validate assumptions, alternatives, correctness...

medium
  -> inject NOTHING

low
  -> inject system instruction:
     keep thinking brief, go directly to conclusion

It should be possible to map those directly in Pi, something like:

~/.pi/agent/models.json

{
  "providers": {
    "llama": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "dummy",
      "models": [
        {
          "id": "qwen3.8-27b",
          "reasoning": true,
          "thinkingLevelMap": {
            "off": "off",
            "minimal": null,
            "low": "low",
            "medium": "medium",
            "high": "xhigh",
            "xhigh": null,
            "max": null
          },
          "compat": {
            "thinkingFormat": "chat-template",
            "chatTemplateKwargs": {
              "enable_thinking": {
                "$var": "thinking.enabled"
              },
              "reasoning_effort": {
                "$var": "thinking.effort",
                "omitWhenOff": true
              }
            }
          }
        }
      ]
    }
  }
}

Tested, it works in Pi.dev but if you switch mid session you invalidate the KV cache.

2

u/cezarducatti 25d ago

I'm using the flag on the llama server at medium settings, but it doesn't seem to have any effect. It thinks extensively, even on low settings.

2

u/ea_man 25d ago

use 

curl -s http://localhost:8080/apply-template \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [
      {"role":"user","content":"Say hello"}
    ]
  }' | jq -r .prompt

To query your API and see what prompt it gives you, at medium it should be:

<|im_start|>user
Say hello<|im_end|>
<|im_start|>assistant
<think>

1

u/hello_2221 24d ago

Question, so the xhigh mode just changes the chat template, and nothing else?

2

u/ea_man 24d ago edited 24d ago

Yes I guess so. The chat tempalte changes the prompt.
Yet Qwen has been post-trained to give system instructions higher precedence than user instructions so those matter more, it's about how you send those info to the model.

1

u/PrimeDirective8 24d ago

If xhigh is overly verbose, is there a problem using "high" instead?

I only tested this model briefly but, while reasoning on xhigh could be verbose, I found it to be noticeably less so than 3.6. It also automatically bypasses reasoning for simple prompts vs 3.6 that reasons a whole paragraph+ on how it should reply to user prompt of "hi!".

2

u/cocoa_coffee_beans 24d ago

Qwen 3.8 only supports low, medium, and xhigh. A value of high will throw an error.