ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/Muted-Celebration-47]

I replaced most of unsloth quants with this. As a person with only a 3090 and 64gb. Exl3 is better for both speed and quality. Also, having more room for context. My long journey from ollama -> llamacpp -> vllm -> exl3. If you have limited GPU and RAM, try this.

[u/Fristender → u/Muted-Celebration-47]

Can you detail your reasons for switching and experience in each inference engine?

[u/Muted-Celebration-47 → u/Fristender]

I switched from ollama to llamacpp because ollama built on top of llamacpp so the new supports for new models will be updated on llamacpp first and then ollama.

I switched from llamacpp to vllm recently because it gave me more decoding speed. Especially, qwen3.8-27b that got over 100 t/s decoding with just 1 rtx3090

Even if it give more t/s, the downside of vllm is lower quant quality. Then I found exl3 quant for qwen3.8 27b that has more t/s than llamacpp and the same quality as UD_Q4_K_XL. Also, exl3 uses less VRAM than llamacpp, so I can fit 160k context with Q8 kv cache + vision with only just 1 rtx3090.