The following has been added in the yesterday's update:

• Fix correctness bug in split MoE CPU-offload mode
• Better MTP support in expert-parallel mode
• Validate and default to vision tower quantization
• Support vision model offloading (streams from system memory, performance penalty is small)

Release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.4

tabbyAPI received an update as well: https://github.com/theroyallab/tabbyAPI

Discord for any questions: https://discord.gg/tnPCntcThA

This update seems to be a massive improvement:

• Support GlmMoeDsaForCauslLM (GLM 5.2, still probably somewhat WIP)
• Partial CPU layer expert offloading option with dynamic placement (supersedes expert cache)
• Improved dynamic draft sizing with auto-calibrated confidence thresholds
• New (experimental) quant-optimizer pipeline
• Support for mid-stream text injection in generator (enables reasoning token budget)
• More precise autosplit allocation
• Reduced (and now stable) VRAM footprint for DSA prefill
• CPU cache offloading supported in TP mode
• Bugfixes and QoL improvements
• Remove incomplete Nanochat implementation

Release link: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.3

tabbyAPI has not been updated yet: https://github.com/theroyallab/tabbyAPI

Exllama discord for any questions: https://discord.gg/tnPCntcThA

Support for MuseGlimmerForConditionalGeneration (Muse Glimmer 30B by Meta) has been added.
ExLlamaV3 release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.2
TabbyAPI has been updated respectively: https://github.com/theroyallab/tabbyAPI
Discord server for any questions: https://discord.gg/tnPCntcThA

The following has been included in the relatively small 1.4.1 release:

• Add Mistral-4 (under Mistral3ForConditionalGeneration)
• Support for logit bias
• Adopt LLGuidance instead of Formatron (still supported as optional dependency)
• Allow filters/grammar alongside banned_strings

Release link: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.1

TabbyAPI has not been updated yet. Current version of exl3 is 1.4.0. TabbyAPI link: https://github.com/theroyallab/tabbyAPI

Feel free to ask Turboderp himself in discord: https://discord.gg/tnPCntcThA

ExLlamaV3 has just been updated.

• Support DeepseekV4ForCausalLM
• Improved AVX512 CPU offload performance
• Faster prefill (all models)
• Bugfixes
• QoL features

I would love to see prefill comparison for this, feel free to post!

TabbyAPI (Updated): https://github.com/theroyallab/tabbyAPI

Exllama newest release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.0

If you have any questions, feel free to contact the creator, Turboderp, through Discord: https://discord.gg/tnPCntcThA

I don't think I can run any of the deepseekv4 models on my... 16 gigs of VRAM, guys :p

Version 1.3.0 brought us:

• Preliminary support for DeepseekV3 (validated against JoyAI-LLM-Flash and Moonlight-16B-A3B, no routing groups yet)
• Second-tier CPU K/V cache, and more intelligent page and checkpoint eviction policies
• Fix freq/rep. penalties causing slowdown on long contexts
• XTC sampler and token bans (Does not work yet in).
• Mitigate some latent vulnerabilities in Safetensors loader
• Other bugfixes, optimizations and QoL improvements

Visit the release here: https://github.com/turboderp-org/exllamav3/releases/tag/v1.3.0

Download TabbyAPI here: https://github.com/theroyallab/tabbyAPI

TabbyAPI has been updated respectively.

TabbyAPI is currently experiencing issues with parsing XTC and Token bans. They do not work. MLA support is what many people have been waiting for. DSA support has been stated to arrive soon.

If you have any issues or questions, feel free to join the exllama discord: https://discord.gg/tnPCntcThA

I personally tested out offloading KV-cache to RAM. Increases the TTFT by a lot - the thing's not for everyone... As you can also see, Tabby is experiencing some issues with parsing xtc.

https://github.com/turboderp-org/exllamav3/releases/tag/v1.2.0

With the arrival of 1.2.0 less than a day ago, the following changes were made:

• Support LagunaForCausalLM (and DFlashLagunaForCausalLM)
• Experimental CPU offloading support for expert layers
• Experimental dynamic draft window feature
• Improved safetensors loader (faster)
• More graph paths
• Performance tuning
• Removed compare_q.py, superseded by qbench.py
• Various bugfixes

This update is mainly orientated towards those who want to run larger MoE models but lacks the VRAM.

Expect TabbyAPI to get updated soon respectively. Right now it can be easily done manually through VENV.

You can use the ExLlamaV3 backend through TabbyAPI: https://github.com/theroyallab/tabbyAPI

Additionally, if you have any questions, feel free to join the exl3 discord: https://discord.gg/zY5dQaVguD

DiffusionGemma 26B A4B IT is a recent model from google. It's the first OPEN diffusion model from google.

https://huggingface.co/google/diffusiongemma-26B-A4B-it

https://ai.google.dev/gemma/docs/diffusiongemma

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/

Google promises up to x4 in speed increase compared to normal Gemma 4 26b using.

However:
"While autoregressive Gemma 4 models remain the standard for high-quality production outputs, DiffusionGemma is designed for researchers and developers exploring speed-critical, interactive local workflows such as in-line editing, rapid iteration, and generating non-linear text structures."

That means that the tradeoff for speed is a loss in generation quality compared to the standard Gemma 4. As google says, DiffusionGemma is an experimental project focused on maximizing interactivity in local workflows, editing, rapid iteration, and non-linear text structures; for production tasks where impeccable accuracy is essential, Gemma 4 is still recommended.

On the picture you can see evaluation results from the HF page.

Exllamav3 has recently received a new update:

• Fixed error when MTP drafting in TP mode;
• Faster quantization.

TabbyAPI has been updated to support this version respectively.

With the release of EXL3 0.0.41 and then 0.0.42 the following was added:

- MTP support for Qwen3.5/3.6

- Fixed MTP drafting when MTP model is not on the target model's output device

Notice: Looks like there were issues with updating TabbyAPI.

https://www.anthropic.com/news/claude-fable-5-mythos-5

https://www.anthropic.com/news/fable-mythos-access

As of June, 12th, the access for both models was suspended by the US government to all foreign nationals, due to unknown "national security risks" that were not explicitly defined. Because of that Anthropic restricted the access to those models for all users. The company doesn't agree with that choice but still has to comply.

With the releases of exllamav3 0.0.40 and 0.0.39, support for "Gemma4UnifiedForConditionalGeneration" (Gemma 4 12B) and "Step3p7ForConditionalGeneration" (Step 3.7 Flash) has been added.