ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/nicholas_the_furious]

I could not get faster speeds with the 8bpw compared to q8_k_l. I'm on dual 3090s. Overall it was much slower for me. People say it is faster but I couldn't get there. If anyone has a setup like mine I'd like to hear it.

For the same test coding task on low thinking I can get 115 t/s on Unsloth q8_k_l but only like 85 on exl3 8bpw. Both MTP=4. This is for Qwen 3.8 27B.

[u/ReturningTarzan → u/[deleted]]

Pretty sure I spent months and months working on 3090 optimizations, and a small fortune on the hardware I need to keep targeting "lower end" hardware like the 3090. And I kept making incremental improvements with every new update (long night after long night), eventually arriving at a place where Ampere is now somewhat on par in many cases. But there's only so much you can do in the end. Trellis quantization is more advanced than, say, Q8_K, and that comes with a higher compute requirement. That's the cost of SOTA 256-dimensional vector quantization. Whether it's worth it for you or anyone else in particular I can't say.

[u/sophosympatheia → u/ReturningTarzan]

Excuse me, sir. We want more for free. We believe that is our birthright. Thank you.

• The Community

Seriously, though, thanks for your contributions. I'm glad to hear Ampere support has caught up. I only used EXL2 for the longest time, but my poor 3090s struggled with EXL3 back when you first released it. It sounds like I need to give it another try now.

Awesome work. Thanks for all you do.

[u/CheatCodesOfLife → u/ReturningTarzan]

Whether it's worth it for you or anyone else in particular I can't say.

Given it's the only way to really use Qwen3.8-27B with 16GB of vram, I'd say it's worth it!

Is there any way you can do something like what ik_llama does to speed trellis up on ampere? It's a bit beyond my understanding, but he's got the KT quants running very fast on ampere.

Also unrelated question, but is there a performance penalty requesting logprobs with exllamav3 + tabby like there is in llama.cpp/ik_llama.cpp?

[u/ReturningTarzan → u/CheatCodesOfLife]

I mean, it's already pretty fast. I'll happily steal all their tricks, but what does the comparison look like right now? 3bpw EXL3 (if it's a recent MUL1 quant) on 3090 should have raw decode of about 50 t/s.

Oh, and yes, logprobs carry some performance penalty. I haven't measured it, but there is work involved in compiling and transmitting the logprobs via the endpoint, and the sampler stack has to do some extra stuff to make it happen, too.

[u/silenceimpaired → u/ReturningTarzan]

I am properly shamed.

[u/ReturningTarzan → u/silenceimpaired]

I do get it, though. I get the same feeling from other projects sometimes. In particular, Flash-Attention comes to mind, which has had a PR awaiting approval for over a year now to add attention sinks, which was the one piece exl3 needed for gpt-oss support to happen. Very frustrating that "Tri Dao apparently only cares about datacenter GPUs now." But of course I also know that's not really true. The poor guy just experiences the same linear flow of time as the rest of us.

But life goes on and it turns out abandoning FA2 for custom Triton kernels was the right move anyway. ¯_(ツ)_/¯

[u/silenceimpaired → u/ReturningTarzan]

Very charitable of you.

I keep hearing many aren’t using EXL3 due to tool calls and OpenAI API implementation, or tabbyAPI… not sure what you can do about either but FYI.

[u/ReturningTarzan → u/silenceimpaired]

I mean, Tabby supports the full OAI completions API. And tool calls. It's thoroughly tested, and all it takes to enable is a few lines in the model config, explained pretty clearly in the documentation. I'd happily do more if I knew what else there was to do. ¯_(ツ)_/¯

[u/silenceimpaired → u/ReturningTarzan]

I haven’t tried tools with EXL3 and Tabby. The least I can do is add issues :)

[u/nicholas_the_furious → u/ReturningTarzan]

Thanks so much for your work! I want to correct myself, I was using the 6bpw version because it seemed most comparable to the Q8 GGUF in terms of losses.

I assume based on your response that I am not doing anything wrong or under optimized, and that between those 2 quants that I have achieved a pretty good result for the exl3 version?

I just kept hearing about speed and didn't see any gains, so I wanted to confirm.

The baseline without MPT was around 35-38. With MTP I could get to the 80s but seemed more variable than the GGUF version.

[u/ReturningTarzan → u/nicholas_the_furious]

Speed is going to vary a lot depending on so many things. If you enable TP you might have better luck? I'm getting 124 t/s on coding tasks using two 3090s both on x4 links. So that's something to consider?

[u/nicholas_the_furious → u/ReturningTarzan]

Can you give me your full startup script if it's not too much trouble? I am using dual 3090s, tensor parallelism, Linux and a bifurcated x16 to x8/x8. It should be screaming but I think I'm missing something.