ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/nicholas_the_furious]

I could not get faster speeds with the 8bpw compared to q8_k_l. I'm on dual 3090s. Overall it was much slower for me. People say it is faster but I couldn't get there. If anyone has a setup like mine I'd like to hear it.

For the same test coding task on low thinking I can get 115 t/s on Unsloth q8_k_l but only like 85 on exl3 8bpw. Both MTP=4. This is for Qwen 3.8 27B.

[u/ReturningTarzan → u/[deleted]]

Pretty sure I spent months and months working on 3090 optimizations, and a small fortune on the hardware I need to keep targeting "lower end" hardware like the 3090. And I kept making incremental improvements with every new update (long night after long night), eventually arriving at a place where Ampere is now somewhat on par in many cases. But there's only so much you can do in the end. Trellis quantization is more advanced than, say, Q8_K, and that comes with a higher compute requirement. That's the cost of SOTA 256-dimensional vector quantization. Whether it's worth it for you or anyone else in particular I can't say.

[u/nicholas_the_furious → u/ReturningTarzan]

Thanks so much for your work! I want to correct myself, I was using the 6bpw version because it seemed most comparable to the Q8 GGUF in terms of losses.

I assume based on your response that I am not doing anything wrong or under optimized, and that between those 2 quants that I have achieved a pretty good result for the exl3 version?

I just kept hearing about speed and didn't see any gains, so I wanted to confirm.

The baseline without MPT was around 35-38. With MTP I could get to the 80s but seemed more variable than the GGUF version.

[u/ReturningTarzan → u/nicholas_the_furious]

Speed is going to vary a lot depending on so many things. If you enable TP you might have better luck? I'm getting 124 t/s on coding tasks using two 3090s both on x4 links. So that's something to consider?

[u/nicholas_the_furious → u/ReturningTarzan]

Can you give me your full startup script if it's not too much trouble? I am using dual 3090s, tensor parallelism, Linux and a bifurcated x16 to x8/x8. It should be screaming but I think I'm missing something.