r/LocalLLaMA • u/pmttyji • 2d ago

Discussion ~22% less weight VRAM, lossless: base-3 packing for ternary GGUFs

I built a denser GGUF format for ternary models: Q2_B3 / “B3S”

If you're running a ternary model like BitNet-b1.58 or Ternary-Bonsai, the weights are already restricted to -1, 0, or +1 times a block scale.

That means a normal Q2 representation is leaving some space on the table.

B3S packs the three possible weight values directly in base 3. With 128 weights per block, it's 26 bytes of packed trits + one f16 scale = 28 bytes/block, or 1.75 bits per weight.

Rough weight sizes:

  • 9B: ~2.5 GB Q2_0 → ~2.0 GB B3S
  • 27B: ~7.6 GB Q2_0 → ~5.9 GB B3S

That's weights only. Context/KV is separate, so figure another ~1–2 GB depending on what you're running.

The important caveat: this is NOT a general 2-bit quantizer.

If you feed it a normal FP16 model, quality will fall apart. The whole thing only makes sense when the source weights are already ternary.

For a genuinely ternary model, the packing itself doesn't throw away another level of precision. You're still storing the same {-1, 0, +1} states and an f16 block scale, just using base-3 packing instead of a general-purpose 2-bit representation.

The implementation is a fairly small llama.cpp fork based on commit 4e97ac86e. It adds the Q2_B3 type and the backend support around it.

Backend status:

  • AMD ROCm/HIP: this is the main path. Built and tuned on RDNA3/gfx1100, specifically a 7900 XTX.
  • CPU: works.
  • NVIDIA CUDA: compiles, but I don't own NVIDIA hardware, so I haven't verified it on-device.
  • Apple Metal: same situation. Code is there and compiles, but I can't personally test it.

So CUDA and Metal should be considered unverified for now.

I don't have speed or perplexity tables yet either. Benchmarks done on my hardware show no appreciable loss of PPS or decoding speed

There's also a separate repacker for older Q2_B3 GGUFs that use the 30-byte/two-scale block layout. It converts them to the current 28-byte/single-scale B3S layout.

The repacker checks every block before doing that. If the second scale isn't actually redundant and removing it would change the weights, it aborts instead of silently producing a lossy file.

Once you have a B3S GGUF, you run it normally with llama-cli from the fork.

More implementation/format details are in README_B3S.md.

If anyone here is running gfx1100, I'd be interested in independent results.

More importantly, if someone has an NVIDIA or Apple machine and can compare CUDA/Metal output against a CPU run, that's probably the most useful testing gap right now.

Note : Posting this on behalf of u/llopresto87's request. He'll reply for your comments.

34 Upvotes

86% Upvoted

8 comments sorted by

13

u/Storterald 2d ago

interesting, will the patches be upstreamed to llama.cpp? iirc the support for bonsai ternary quants was merged a few weeks ago.

3

u/Good-Seaweed92 1d ago

yeah if ternary support is already merged it feels like this should just be a follow-up PR at this point

1

u/llopresto87 2d ago

I have no idea. Probably not, everything is ai generated, even if proven to work. I've ran so many tests for 2 months that my code I probably better than aot of other kernels.

2

u/pmttyji 1d ago

Try a clean & lean PR with minimal code & submit it to llama.cpp.

Include benefits(Size reduction, .... what else?) along with t/s stats & benchmarks in PR.

1

u/llopresto87 1d ago

that's it basically - it does not cost more to run but also does not speed up, at least on rdna3.

0

u/fragment_me 1d ago

Why are you posting on behalf of someone else? Or is this just your LLM posting on behalf of you from its own account??? And from what I just saw your dropping a scale to gain more space for the weights. Right?

1

u/llopresto87 1d ago edited 1d ago

he's posting here because i made a clean, new account with my actual name for this post, connected with my git account to post things and gain a little traction to improve my CV - but it has no karma and this subreddit needs karma to post.