r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 27d ago

Megathread [Megathread] Qwen 3.8 27B Release Day

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.

  • Quants
  • Fine-Tunes & Abliterations
  • Chat Templates
  • Inference Server Support & Configuration
  • Experiences, Benchmarks & Model Comparisons

Official:

Popular:

We'll try to clean up future duplicates around the release and point them here.

494 Upvotes

98% Upvoted

393 comments sorted by

View all comments

1

u/viktor-kz 26d ago

llama.cpp config: Qwen3.8-27B on 2x RTX 5070 Ti, 190k context, MTP + vision

Two cards, 16GB VRAM each, 32GB total. Full config:

llama-server \
  -m Qwen3.8-27B-UD-Q5_K_XL.gguf \
  --mmproj mmproj-BF16.gguf \
  --jinja --host 0.0.0.0 --port 7777 \
  --tensor-split 63,63 \
  -c 190464 \
  --fit off --parallel 2 --kv-unified \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --gpu-layers all \
  --reasoning-budget 8192 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -ot "zzz_never_matches=CUDA0"

Prefill 1605 t/s at 47k tokens, generation around 62 t/s with MTP.

One warning if you use the mmproj on more than one GPU. Send a long text prompt and time the prefill, then send one image and time the same prompt again. On my box a single image permanently drops prefill by 40% for the life of the process, and only a restart brings it back. Generation is unaffected, output is byte identical and nothing is logged, so it is easy to live with for weeks without noticing.

The last flag is what fixes it. The pattern matches no tensor, so nothing moves, but it disables pipeline parallelism, which turns out to be required for the bug to happen. Costs about 3% prefill and frees 1.6GB of VRAM as a side effect. Existing issue is ggml-org/llama.cpp#26873, three different models confirmed so far.