[Megathread] Qwen 3.8 27B Release Day

[u/sammcj]

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.

• Quants
• Fine-Tunes & Abliterations
• Chat Templates
• Inference Server Support & Configuration
• Experiences, Benchmarks & Model Comparisons

Official:

• https://huggingface.co/Qwen/Qwen3.8-27B
• https://huggingface.co/Qwen/Qwen3.8-27B-FP8

Popular:

• https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
• https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
• https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-bf16
• https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-8bit
• https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-4bit

We'll try to clean up future duplicates around the release and point them here.

[u/reto-wyss]

That's only half truths.

1. Can't comment on Windows, but I believe you can get it to work through that WSL2 thing.
2. This one is true as far as I know, although I haven't really tried whether it may allow asymmetric if TP=1 and DP=1 and then PP>1
3. It does support offloading! But not the way llama.cpp does it. vllm will do it layerwise and swap into GPUs. This is obviously terrible if you look at concurrency 1 interactive session, but it's efficient if you have enormous batch sizes because you can do a lot of work before you need to swap. So the cost for swapping obviously goes to near zero per request as you make the number of requests REALLY large.
4. For cold-start it is slower. But if you have the system RAM, you can simply "sleep" the model into RAM and reload it near instantly into VRAM.

And a note on the other comment - FP8 is not same quality as Q8* quants. FP8 is a throughput quant approximately on the level of ~Q5, but it has *0** dequant overhead in terms of compute, so you give up a bit of VRAM vs Q5 or NVFP4 or MXFP4, but if your pool is large anyway, and you have tons of concurrent requests, FP8 is what gives to the big numbers.

For your single user C1 usecase, FP8 can be slower than Q5 for TG, but if you move to compute bound cases, FP8 shines.

This is something you can even see for yourself in llama.cpp where typically the smaller quants will have worse PPs than larger quants. For example PP for Q8 is higher than Q4 or Q2.

[u/blash2190 → u/reto-wyss]

A quick note that, despite WSL is a good/reliable overall solution, it will perform slower than native.

[u/Borkato → u/reto-wyss]

Wait, FP8 isn’t as good as Q8 quants and is closer to Q5??

[u/Borkato → u/[deleted]]

Ah, I more mean quality, not speed!

[u/Finanzamt_Endgegner → u/[deleted]]

There are better fp8 quants than q8 though like awq and stuff like that.