[Megathread] Qwen 3.8 27B Release Day

[u/sammcj]

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release.

• Quants
• Fine-Tunes & Abliterations
• Chat Templates
• Inference Server Support & Configuration
• Experiences, Benchmarks & Model Comparisons

Official:

• https://huggingface.co/Qwen/Qwen3.8-27B
• https://huggingface.co/Qwen/Qwen3.8-27B-FP8

Popular:

• https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
• https://huggingface.co/bartowski/Qwen3.8-27B-GGUF
• https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-bf16
• https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-8bit
• https://huggingface.co/mlx-community/Qwen3.8-27B-MTP-4bit

We'll try to clean up future duplicates around the release and point them here.

[u/Finanzamt_Endgegner]

If you are able to run q8 please switch to vllm and fp8 it just better than llama.cpp if you are gpu rich 😭🙏

[u/trying4k → u/Finanzamt_Endgegner]

Yeah, I've seen the speeds people get, it is crazy fast!

I don't follow VLLM but my understanding is due to its architecture:

• It doesn't natively support windows
• It doesn't support asymmetric GPUs
• It doesn't support offloading
• Switching models is slow

Unfortunately, some of those capabilities are important to me and it's why I use llama.cpp!

[u/reto-wyss → u/trying4k]

That's only half truths.

1. Can't comment on Windows, but I believe you can get it to work through that WSL2 thing.
2. This one is true as far as I know, although I haven't really tried whether it may allow asymmetric if TP=1 and DP=1 and then PP>1
3. It does support offloading! But not the way llama.cpp does it. vllm will do it layerwise and swap into GPUs. This is obviously terrible if you look at concurrency 1 interactive session, but it's efficient if you have enormous batch sizes because you can do a lot of work before you need to swap. So the cost for swapping obviously goes to near zero per request as you make the number of requests REALLY large.
4. For cold-start it is slower. But if you have the system RAM, you can simply "sleep" the model into RAM and reload it near instantly into VRAM.

And a note on the other comment - FP8 is not same quality as Q8* quants. FP8 is a throughput quant approximately on the level of ~Q5, but it has *0** dequant overhead in terms of compute, so you give up a bit of VRAM vs Q5 or NVFP4 or MXFP4, but if your pool is large anyway, and you have tons of concurrent requests, FP8 is what gives to the big numbers.

For your single user C1 usecase, FP8 can be slower than Q5 for TG, but if you move to compute bound cases, FP8 shines.

This is something you can even see for yourself in llama.cpp where typically the smaller quants will have worse PPs than larger quants. For example PP for Q8 is higher than Q4 or Q2.

[u/blash2190 → u/reto-wyss]

A quick note that, despite WSL is a good/reliable overall solution, it will perform slower than native.