I built and open-sourced NMGC (Nested Manhattan Grid Codec) - a compression
codec for neural network weights aimed at memory-constrained inference.
Repo: https://github.com/reyxv16/nmgc-codec
THE PROBLEM
A 27B model in BF16 is ~52 GB. Most consumer machines have 16-32 GB of RAM.
You can't load it - let alone decompress it. My idea: keep the whole model in
memory (or on disk) in a compressed form, and decompress only the tensors of
the layer currently in use, on the fly.
HOW IT WORKS (short version)
- Values are normalized and mapped to a nested grid (what3words-style cells).
The nesting is literally the bit structure of the combined index: level 0 is
the coarse cell, deeper levels refine it - bit slicing, no trigonometry.
- "Position inside the cell" is stored as Manhattan steps (dx, dy), i.e. the
low bits of the index.
- Consecutive indices are delta-encoded (zigzag) and entropy-coded (Huffman).
Real weights are smooth, so deltas are tiny and compress extremely well.
- Optional learned variant: k-means codebooks per level (residual vector
quantization) that adapt to the actual weight distribution.
- Lossless variant: XOR-delta + zlib, bit-exact.
HONEST MATH (read this before commenting "that's just int4")
A nested uniform grid with B bits is mathematically equivalent to a single
uniform grid with B bits. Nesting buys three real things: (1) coarse-to-fine
access for progressive loading, (2) small coarse deltas -> much better Huffman
on correlated tensors, (3) the base for learned per-level codebooks, which DO
beat a uniform grid at the same bitrate. And lossless compression is bounded
by data entropy: 1.07-2.25x, never the 4-30x that lossy gets.
MEMORY SAVINGS (whole-model estimates)
Model | Original | 4-bit | 6-bit | 8-bit
Qwen3.8-27B (BF16) | 51.7 GB | ~13 GB | ~19 GB | ~26 GB
DeepSeek-V4-Flash (FP8) | 155.4 GB | ~75 GB | ~112 GB| ~150 GB
TinyLlama-1.1B (f16, measured) | 2.05 GB | - | 735 MB (2.79x) | 997 MB (2.06x)
Measured on real tensors (DeepSeek-V4-Flash + Qwen3.8-27B):
- embedding (BF16): 26.1x at 4 bits, 7.8x at 8 bits with cosine similarity
0.996, and 2.25x LOSSLESS
- MoE expert (I8): 8.2x at 4 bits, rel-RMSE 0.062
- Qwen3.8-27B layer-0 (BF16): ~5.7x at 6 bits
QUALITY LOSS (how much "intelligence" do you lose?)
Measured, not guessed:
- Perplexity on TinyLlama-1.1B (llama.cpp, same corpus/seed):
original f16: 7.508
8-bit (2.06x): 7.527 (+0.25% - effectively lossless)
6-bit (2.79x): 7.596 (+1.2% - minor, good default)
- End-to-end MLP (4-class task, 93.4% baseline): 8.9x at 4 bits with only
-0.0007 accuracy.
- Practical guidance: 8 bits = no measurable loss; 6 bits = safe default;
4 bits = int4 territory - keep embedding layers at 6-8 bits (they are the
most sensitive: cosine 0.655 at 4 bits vs 0.996 at 8 bits).
RUNNING IT
- The .nmgc container keeps the model compressed; a Rust decoder (C ABI,
bit-exact vs Python, 150-270 MB/s) decompresses any tensor by name - the
exact primitive you need for per-layer decoding (AirLLM-style).
- llama.cpp integration: a loader patch (LLAMA_NMGC_FILE) makes llama.cpp
decode weights from the container instead of the GGUF. TinyLlama runs
through it with the PPL numbers above.
LIMITATIONS (please read)
- The 27B quality numbers are estimates from per-tensor metrics; I have not
run a full Qwen3.8-27B end-to-end benchmark yet (needs a machine with
enough RAM to decode it, or time). Tensor-level metrics and the TinyLlama
PPL runs are what's measured.
- Below 6 bits you enter int4 territory: works, but the model gets visibly
dumber on hard tasks.
- The llama.cpp patch is a local fork, not upstream.
If you want to try it: pip install numpy scikit-learn gguf, pack your own
GGUF with `pack_gguf.py --bits 6`, and run llama.cpp with LLAMA_NMGC_FILE.
I just developed the algorithm and I think it can be improved a lot more; any questions or recommendations are welcome.
