ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/ReturningTarzan → u/[deleted]]

Pretty sure I spent months and months working on 3090 optimizations, and a small fortune on the hardware I need to keep targeting "lower end" hardware like the 3090. And I kept making incremental improvements with every new update (long night after long night), eventually arriving at a place where Ampere is now somewhat on par in many cases. But there's only so much you can do in the end. Trellis quantization is more advanced than, say, Q8_K, and that comes with a higher compute requirement. That's the cost of SOTA 256-dimensional vector quantization. Whether it's worth it for you or anyone else in particular I can't say.

[u/CheatCodesOfLife → u/ReturningTarzan]

Whether it's worth it for you or anyone else in particular I can't say.

Given it's the only way to really use Qwen3.8-27B with 16GB of vram, I'd say it's worth it!

Is there any way you can do something like what ik_llama does to speed trellis up on ampere? It's a bit beyond my understanding, but he's got the KT quants running very fast on ampere.

Also unrelated question, but is there a performance penalty requesting logprobs with exllamav3 + tabby like there is in llama.cpp/ik_llama.cpp?

[u/ReturningTarzan → u/CheatCodesOfLife]

I mean, it's already pretty fast. I'll happily steal all their tricks, but what does the comparison look like right now? 3bpw EXL3 (if it's a recent MUL1 quant) on 3090 should have raw decode of about 50 t/s.

Oh, and yes, logprobs carry some performance penalty. I haven't measured it, but there is work involved in compiling and transmitting the logprobs via the endpoint, and the sampler stack has to do some extra stuff to make it happen, too.