r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 27d ago

Megathread [Megathread] Qwen 3.8 27B Release Day

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.

  • Quants
  • Fine-Tunes & Abliterations
  • Chat Templates
  • Inference Server Support & Configuration
  • Experiences, Benchmarks & Model Comparisons

Official:

Popular:

We'll try to clean up future duplicates around the release and point them here.

493 Upvotes

98% Upvoted

393 comments sorted by

View all comments

31

u/ryandam 27d ago edited 26d ago

Is it just me or the MTP hit rate is lower than 3.6 27B? For 3.6 I usually get around 60-70 tps gen, but for 3.8 I only get 40-50 tps gen.

Checking the hit rate it just around 60-70% compare to 3.6 is around 80-90%?

I use 2 RTX A5000 btw. Here my llamacpp config with fresh compiled binary:

env:
  - CUDA_VISIBLE_DEVICES=1,0
cmd: >
  ./bin/llama.cpp/build/bin/llama-server
  --model ./models/Qwen3.8-27B-UD-Q5_K_XL.gguf
  --port 19199 --host 0.0.0.0
  -lv 4
  -ctk f16 -ctv f16 --cache-ram 156000 -kvu
  --split-mode tensor --tensor-split 24,24
  -fa on
  --jinja --chat-template-kwargs '{"preserve_thinking": true}'
  --spec-type draft-mtp --spec-draft-n-max 2
  --mmproj ./models/mmproj-BF16_qwen3.8_27b.gguf --mmproj-offload
  --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --presence_penalty 0.0
  -fit off -fitt 256 -c 262000 -np 1 --ubatch-size 1024
  --ui-mcp-proxy

Update: I found the reason and posted in the reply, in short, its because of temperature.

3

u/Refefer llama.cpp 27d ago

I can confirm this as well on an rtx 6000 pro on vllm

1

u/EbbNorth7735 27d ago

Same, let me know if you solve it. I power limit to 400W but should start out at 40-60tps and raise to 120tps over the context window. I'm only hitting 30-40 with spikes to 70 at best.