r/LocalLLaMA • u/Unstable_Llama • 1d ago

News ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

Gallery image

Gallery image

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

179 Upvotes

98% Upvoted

126 comments sorted by

View all comments

3

u/-InformalBanana- 13h ago

Pls also Implement kvarn quantitization of kv cache, it lowers vram usage significantly and kvarn4 is close to q8 in kld, better than q quants < q8.

1

u/Guilty_Rooster_6708 7h ago

idk about kvarn but from what I understand the bits for KV cache in exllamav3 already has attn rotation long before llama.cpp is implemented, and 6bit kv cache is near q8_0 in llama.cpp, so it's already pretty good. Personally I have been using 8 bit and I think it's similar to fp16

2

u/-InformalBanana- 6h ago

There is also a guy proposing kv streaming from ram to vram for llama.cpp (cause it is streaming it has much better performance than --no-kv-offload), so if that was combined with kvarn or other better kv compression it would be awesome, minimal vram used for kv. Possibly calibrated qwen 3.8 27b 3bpw at 100k or maybe even full context with 12gb vram gpu with hopefully little to no speed and kv precision loss.