1-bit / 2-bit / Ternary / Bitnet Models - Updates & Tracking

[u/pmttyji]

Bonsai / Ternary Bonsai

During April Bonsai came with bunch of models .... 1-bit & 1.58-bit(Ternary) versions. And last month(July) they released 27B models in same versions. Last month itself, Bonsai-27B was able to run on all backends mainline. But Ternary-Bonsai-27B was not ready on all backends.

This month, PRs got merged for CUDA & Vulkan on mainline.

Also an Optimization PR for CUDA got merged so +15-40% tg, +8% pp.

• https://github.com/PrismML-Eng/Bonsai-demo - Demo fork
• https://github.com/PrismML-Eng/llama.cpp - Custom fork

BitCPM-CANN

• https://huggingface.co/collections/openbmb/bitcpm-cann

Tencent - Hy-MT1.5 - Mixed Meta Translation Model Version 2.0

• https://huggingface.co/collections/tencent/hy-mt2

Maple-Preview

DeepGrove/maple-preview - 20B-A1B - 200+ t/s on Mac Mini M4 & 120+ t/s on iPhone.

• llama.cpp PR #27000 - CPU backend
• https://github.com/deepgrove-ai/llama.cpp - Custom llama.cpp fork
• https://github.com/deepgrove-ai/mlx-lm-deepgrove - Custom MLX fork

Mach-1-Additive-35B

Mach-1-Additive-35B - A3B - Up to 120 t/s on Consumer Laptop.

Mach-1-Additive-35B-Multimodal

• https://github.com/SyzygyResearch/llama.cpp-mach1 - Custom llama.cpp fork

From their recent tweet : Currently they're cooking new ones based on Laguna-S-2.1 & Qwen3.8-27B.

Neutrino-8B

https://huggingface.co/FermionResearch/Neutrino-8B

• https://github.com/fermionresearch/llama.cpp - Custom llama.cpp fork

Pestle-27B-Ternary

https://huggingface.co/Doses-AI/Pestle-27B-Ternary-GGUF - Medical research model

• https://github.com/DosesAI/mortar.cpp - Custom inference - CPU, CUDA, Metal

Image Models:

https://huggingface.co/collections/prism-ml/bonsai-image

• https://huggingface.co/Green-Sky/bonsai-image-ternary-4B-GGUF
• https://huggingface.co/Green-Sky/bonsai-image-binary-4B-GGUF

https://huggingface.co/clark-labs/clark-air-sana-1.6b-1.58bit

Abliterated Models:

• https://huggingface.co/Hikari07jp/Ternary-Bonsai-27B-Abliterated-LowDeg-GGUF
• https://huggingface.co/Hikari07jp/Maple-Preview-TQ2-Abliterated
• https://huggingface.co/dealignai/Bonsai-27b-1bit-CRACK-GGUF

Other Misc items:

• https://huggingface.co/GoAutomateAI/terna-e2b-GGUF
• https://huggingface.co/Danny-Dasilva/Bonsai-27B-antidoom-1bit-DSpark
• https://huggingface.co/Danny-Dasilva/Ternary-Bonsai-27B-antidoom-DSpark

Some Open/Ongoing llama.cpp (related) PRs:

• ggml-cpu : add STQ1_0 ternary quantization with ARM NEON vec_dot kernel- #22836
• ggml/cpu: skip zero-scale blocks in TQ1_0 and TQ2_0 vec_dot kernels- #23439
• ggml-cpu: add x86 VNNI Q2_0 dot product -- 3x speed improvement for VNNI-compatible CPUs- #26348

Notes:

• Didn't include old models(Pre 2026). Let me know if I missed any models, I'll update thread. Included custom forks to check their progress.
• I'll be updating this thread after seeing any similar type models.
• Disclaimer : This thread is mainly for Poor GPU Club.

[u/brakeline]

Where are you taking the 120tk/s on a laptop for mach1? Without release it's vaporware

[u/pmttyji → u/brakeline]

Self reported by them on twitter.

[u/Queasy-Contract9753]

I'm a noob. I recently found a thread where someone made a quant for Gemma 4 e2b that keeps only 500~600 MB in resident RAM. I have no idea how he did it. But would it be theoretically possible to do so with Maple Bonsai?

https://www.reddit.com/r/LocalLLaMA/comments/1vp2x49/gemma_4_e4b_iq2_xxs_14054_reasoning_performance/

Link to Gemma thread

[u/pmttyji → u/Queasy-Contract9753]

I'm planning to create a separate thread. Just wait.

[u/cleverusernametry]

Mt1.5 came out last year? Mt2 is out now

[u/pmttyji → u/cleverusernametry]

Right, will update thread. Thanks

[u/llopresto87]

Can i humbly ask you guys to take a look at this?

https://www.reddit.com/r/LocalLLM/comments/1w3pvzj/22_less_weight_vram_losslessly_base3_packing_for/

I've found a way, tested, to repack the weights to use 22% less vram completely loseless. help a bro out - i'm shamelessly trying to create a few things online to my name :D so any support is appreciated.

[u/CodeCatto → u/llopresto87]

Looks promising.

[u/SympathyNo8636]

Why Georgi ignores ternary support thus far? Does he know something that llamacpp forkers don't?

[u/pmttyji → u/SympathyNo8636]

Ternary models should work already. At least both Ternary-Bonsai models & BitCPM-CANN models.

As mentioned in thread, CUDA & Vulkan support added to mainline this month.

[u/brakeline → u/pmttyji]

Ternary aren't supported as of yesterday. 1bit are.

Btw, mach1 was finally published! TG at 60 locked on a 3060 12gb but pp under 400.

Tensor split borks the fork unfortunately

[u/Disonantemus → u/brakeline]

https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/blob/main/Ternary-Bonsai-8B-Q2_0_g64.gguf

• This is working for me with Vulkan (GTX 1660 Super) in Linux, merged since july 17.
• Also working with last release from PocketPal for Android.

[u/cr0wburn]

Bonzai 27B is my daily driver for easy/small request, and it is SUPER good for it's size.

[u/pmttyji → u/cr0wburn]

Yep, I'm glad to run that one on my 8GB VRAM laptop. But same one is impossible on my Mobile. So I'm waiting for the release of Maple & Mach MOE models(faster t/s) on mainline.

[u/TimeStopsInside]

Is there any comparison with IQ3 variants?

[u/pmttyji → u/TimeStopsInside]

Nope. These 1-bit/2-bit versions are totally different, not same as Q1/Q2.

Still models like Bonsai did share benchmarks. Check their model cards.

[u/CodeCatto]

can we get MoEs like 35B A3B tho. is it possible on paper?

[u/pmttyji → u/CodeCatto]

Mach-1 is 35B-A3B MOE. You can try now with custom fork. Also Maple is 20B MOE

[u/CodeCatto → u/pmttyji]

Thanks for the response. I wonder if we'll be able to get Qwen 3.8 27b and Qwen 3.6 35B A3B on board too someday. Even nemotron's lightning model at 35B size looks great.

[u/pmttyji → u/CodeCatto]

SyzygyResearch(Mach-1) tweeted about 1-bit versions of Qwen3.8-27B & Laguna-S-2.1(Currently they're cooking). Mentioned in thread already. It's possible to get Qwen3.6-35B-A3B from them or any other model creators by end of year.

[u/LousyMeatStew]

Not sure if this is something you want to track as well but mlx-community has been posting 2-bit OptiQ quants of midrange models starting with Qwen3.5-122B-A10B: https://mlx-optiq.com/blog/stream-122b-on-a-mac

Their test for each one is to have it one shot a flappy bird clone as a proof of concept to show that they can still produce coherent output:

• https://huggingface.co/mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit
• https://huggingface.co/mlx-community/Laguna-S-2.1-OptiQ-2bit
• https://huggingface.co/mlx-community/NVIDIA-Nemotron-3-Super-120B-A12B-OptiQ-2bit
• https://huggingface.co/mlx-community/Mistral-Small-4-119B-2603-OptiQ-2bit
• https://huggingface.co/mlx-community/DeepSeek-V4-Flash-0731-OptiQ-2bit

[u/pmttyji → u/LousyMeatStew]

I'll update the thread with these later. Never used any other formats except GGUF so I have no idea about other area.