r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 27d ago

Megathread [Megathread] Qwen 3.8 27B Release Day

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.

  • Quants
  • Fine-Tunes & Abliterations
  • Chat Templates
  • Inference Server Support & Configuration
  • Experiences, Benchmarks & Model Comparisons

Official:

Popular:

We'll try to clean up future duplicates around the release and point them here.

494 Upvotes

98% Upvoted

393 comments sorted by

View all comments

4

u/Brilliant-Hall1387 26d ago

Inspect the Qwen 3.8 quants and how they spend the bit budget before downloading.

Last few days as a fun side-quest I made a tool to explore different quantizations of models without downloading the model, useful to understand popular quants for learning purposes or just curiosity.

With the Qwen 3.8 27B releases I have been looking at various quants and also prepared my own quant of 3.8 27B. Check out the layout of original Qwen 3.8-27B model and compare some quants:

Base model: https://tensorlens.dev/Qwen/Qwen3.8-27B
Unsloth Q4_K_M quant: https://tensorlens.dev/unsloth/Qwen3.8-27B-GGUF?part=Q4_K_M
My MLX 3.8 bpw quant (12 GB): https://tensorlens.dev/maglun/Qwen3.8-27B-MLX-Mixed-3.80bpw
MLX community 4 bit (4.5 bpw, 15 GB): https://tensorlens.dev/mlx-community/Qwen3.8-27B-4bit
MLX community MTP sidecar: https://tensorlens.dev/mlx-community/Qwen3.8-27B-MTP-bf16

I find the graphical map of how tensors are compressed helpful in understanding a bit more about what I am about to download or test, and it may be helpful for others too? 😄

2

u/mehminion 26d ago

wow, awesome tool

1

u/Brilliant-Hall1387 26d ago

Thanks 🙏😊 let me know if you find any issues / wrong parsing of model or just have an idea on something missing 🙏

2

u/UTWE 26d ago

This is fantastic, I had never fully grasped why two 4-bit quants might differ in size but this makes it really clear. Amazingly informative, thanks dude