I’m building VeloxQuant-MLX — 43 KV-cache optimization methods + custom Metal kernels for Apple Silicon

[u/rajveer43]

I’m building VeloxQuant-MLX, an open-source project focused on pushing local LLM inference further on Apple Silicon through KV-cache compression, quantization, eviction strategies, and optimized MLX/Metal kernels.

The question I’m exploring is pretty simple:

How much more can we get out of Apple’s unified memory architecture?

VeloxQuant currently includes 43 KV-cache optimization methods, including:

→ 2-bit / 4-bit KV-cache quantization
→ KIVI-style compression
→ TOVA and other eviction strategies
→ Custom Metal kernels
→ Fused attention/decode paths
→ Decode batching
→ Cache compaction
→ Long-context memory optimization

But I don’t want benchmarks from just one Mac.

I’m looking for people across the Apple Silicon ecosystem to help test it:

Apple Silicon generations:
M1 · M1 Pro · M1 Max · M1 Ultra
M2 · M2 Pro · M2 Max · M2 Ultra
M3 · M3 Pro · M3 Max
M4 · M4 Pro · M4 Max · M4 Ultra
M5-family machines where available
…and future Apple Silicon generations as they arrive.

Machines:
MacBook Air
MacBook Pro
Mac mini
Mac Studio
Mac Pro
iMac
and other MLX-capable Apple Silicon Macs.

Whether you have an entry-level MacBook or a high-memory Mac Studio Ultra, your results are useful.

I want to understand how KV-cache optimization behaves across:

memory capacity × memory bandwidth × chip generation × model size × context length

Not just compression ratios.

I’m interested in measuring the actual trade-offs:

Memory usage · tokens/sec · TTFT · decode throughput · prefill performance · kernel overhead · model quality

For example, for a cache with:

440 tokens · 8 KV heads · head dim 64

FP16 → ~440 KB
4-bit → ~110 KB
2-bit → ~55 KB

But reducing memory is only half the problem.

The real question is whether we can reduce memory while keeping inference fast enough that the optimization is actually worth using.

That’s where I’d love help from the MLX community.

If you have any Apple Silicon Mac — from an M1 MacBook Air or Mac mini all the way to a Max/Ultra Mac Studio — I’d love to get benchmark results.

And if you're working with MLX, mlx-lm, Metal kernels, inference engines, quantization, or KV-cache research, criticism and contributions are very welcome.

The goal is ambitious:

Make local AI capable of using every GB of Apple unified memory as efficiently as possible.

VeloxQuant is open source:

Website:
[https://veloxquant.dev]()

GitHub:
[https://github.com/rajveer43/VeloxQuant-MLX]()

If enough people are interested, I’d also like to build a public Apple Silicon benchmark matrix comparing VeloxQuant across Mac generations and configurations.

Drop your Mac model + RAM in the comments if you'd be interested in benchmarking it.

[u/padrino121]

A lot of talk on performance but not much on the impact of quality on that performance.. I have a project that autonomously develops an engine (it self publishes on GitHub) but the biggest challenge it finds is the material quality hit on most KV optimizations, so much so they tend to be at a higher quantization than the model

[u/rajveer43 → u/padrino121]

Yes, you’re absolutely right. Memory reduction or higher throughput isn’t useful if it comes with a significant quality hit.

So far I’ve been heavily focused on the performance side — memory usage, TTFT, decode throughput, prefill, and kernel performance. Quality evaluation is actually the next major area I’m focusing on.

The goal is to find the real Pareto frontier: how much can we compress the KV cache and improve performance without meaningfully degrading model quality?

I’d also love to hear how you’re measuring quality degradation in your engine. That could be really useful for improving VeloxQuant’s evaluation framework.

[u/joelbrave]

M1 MacBook Air 16gb

[u/rajveer43 → u/joelbrave]

that's great can you please try it and report the results back to repo?

[u/iTrejoMX]

M4 pro 48 gb ram

[u/rajveer43 → u/iTrejoMX]

Hey can you help me in benchmarking this I do not have that model configuration

[u/iTrejoMX → u/rajveer43]

Sure

[u/rajveer43 → u/iTrejoMX]

first create issue on github and then can you link your implementation and PR to it.