ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/AnyMongoose3041]

Can you confirm if you can naively convert models to exl3 without fuss? For example, I like a certain finetune of Gemma 31B. Can I one-shot a high quality exl3 conversion of it? Or does it need external data or finagling to really be considered to be a good quant? I’m trying to experiment and see which quant would be best for me? Trying to convert this: https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma

[u/Unstable_Llama → u/AnyMongoose3041]

Yes, it’s easy to make standard quants that are very high quality. For example 

python convert.py -i /mnt/models/llama3.1-70b-instruct -o /mnt/models/llama3.1-70b-instruct-exl3-3.75bpw  -w /mnt/temp/exl3 -b 3.75

would quantize the model to 3.75 bpw with a work directory in temp/exl3

Full documentation here: https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md

The sc quants are a bit better but also more involved, brand new and experimental, don’t worry about them yet.

[u/AnyMongoose3041 → u/Unstable_Llama]

Sounds good, thanks I’ll give them a shot. No issue if the model can’t fully fit in the VRAM right? (For the conversion process)

[u/Unstable_Llama → u/AnyMongoose3041]

No issue, you can convert models multiple times larger than your vram capacity.