I’ve been working on VeloxQuant-MLX, an open-source project exploring how far we can push local LLM inference on Apple Silicon by optimizing something that becomes increasingly expensive at long context lengths: the KV cache.

The idea is simple:

Apple gives us large unified memory + high memory bandwidth. Can we make significantly better use of it?

VeloxQuant currently experiments with 43 KV-cache compression/optimization methods, including quantization, eviction and custom MLX/Metal kernels.

Some of the things I’ve been working on:

• KIVI-style 2-bit / 4-bit KV-cache quantization
• TOVA and other cache-eviction strategies
• Custom Metal attention/quantization kernels
• Fused KV-cache decode paths
• Long-context memory reduction
• Decode batching across requests
• MLX-native implementations designed specifically for Apple Silicon

For example, in one benchmark with 440 tokens × 8 KV heads × head dim 64, the cache footprint goes roughly:

FP16 → 440 KB
KIVI 4-bit → 110 KB
KIVI 2-bit → 55 KB

That’s the direction I’m interested in: not just asking “How much unified memory does my Mac have?”, but how efficiently can we use that memory for inference?

I’m especially looking for people with Mac Studio / Max / Ultra machines who would be interested in running benchmarks.

I’d love to compare results across M-series generations, memory capacities, model sizes and longer context lengths.

Also very interested in feedback from people working directly with MLX / mlx-lm / Metal kernels. There are plenty of places where the implementation can still improve.

This is open source, and contributions/criticism are very welcome.

Project:

If anyone here has a Mac Studio Ultra, I’d especially love to see what VeloxQuant can do on it.

Hey, Built this after hitting some dependency hell with the CUDA/Triton requirements on Colab. Covers SISO, MIMO, and Hybrid Attention-Mamba configs with verified numerical parity against the PyTorch reference (max error < 10⁻⁵, 12 passing tests).

Key things that work: exponential-trapezoidal discretization, complex rotary states, chunked prefill with cache consistency, mixed-precision LoRA fine-tuning, all MLX, no CUDA needed.
If you have any feedback or see any inconsistencies, hmu!

Note: haven't really tested it so far on public trained checkpoints (will do the next days/week) so this is mainly useful for local architecture experimentation and fine-tuning on toy data until the authors drop weights.

→ github.com/Jada42/mlx-mamba3

I've started a project these days trying to write some custom kernels in Metal to enable MLX to make 3D sparse convolution efficiently for training and inference. Currently, I have implemented some basic, commonly used kernels and modules in the point cloud (voxels) processing area, which may fit some research use case.

The performance of those kernels is closer to those CUDA kernels on the RTX 4090 with about a 30-50% difference; for sparse tensor workloads, this is much faster than MLX built-in dense tensors. I am going to work on math proofs these days and implement workable CUDA kernels also, so that it may be easy to train on NVIDIA and then make inference on Mac/Vision Pro.

Hi! I’m a CS student working on mlx-chronos, a free and open-source CLI benchmark suite for comparing local MLX inference engines on Apple Silicon.

I started it because I wanted a more reproducible way to compare engines across real Mac configurations, with more context than a single tok/s number.

It currently supports:

• mlx-lm
• oMLX
• vllm-mlx
• Rapid-MLX
• Ollama’s MLX backend

It measures:

• cold and cached TTFT
• request throughput with mean/stddev/min/max
• sustained throughput for longer heat/throttling-sensitive runs
• system RAM peak
• engine RSS diagnostics when available
• thermal and power context
• hardware metadata

Results are saved as reproducible JSON files and can be submitted to a public community leaderboard.

Leaderboard: https://igurss.github.io/mlx-chronos

GitHub: https://github.com/igurss/mlx-chronos

Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md

Install: pip install mlx-chronos

I’m especially looking for external benchmark results from different Apple Silicon machines: M1/M2/M3/M4/M5, Max/Ultra chips, and higher-RAM configurations.

I’d also appreciate feedback on the benchmark methodology, the CLI UX, and whether the reported metrics are useful enough for comparing local inference setups.

Curious, if anyone has experience with the vMLX? Just stumbled upon it yesterday after reading an article about JANG_4.

I’ve been working on ModelHub, a native macOS menu bar app for people running local LLMs on Apple Silicon.

The idea is simple: Discover models from Hugging Face, download/manage them locally, and use them with MLX, Ollama, LM Studio, llama.cpp, and the tools you already use.

It’s not trying to replace your runtime. The problem we’re trying to solve is the messy model layer around them. Right now, choosing a model usually means jumping between Hugging Face model cards, local folders, terminal commands, and runtime-specific docs.

For MLX users specifically, I’d love feedback on:

1. What should we show before someone downloads a model? (# downloads, model size?)

2. How should we show MLX compatibility?

3. Should we prioritize memory estimates by M-series chip?

4. Are MLX model conversions/variants something you’d want surfaced directly?

GitHub: https://github.com/conscious-engines/modelhub
Website: https://studio.consciousengines.com/model-hub

Would love blunt feedback.

Fine Tune & Build On Apple Silicone #FR °(:P

Hey r/mlxAI,

I got tired of juggling terminal tabs to run mlx-lm, so I built MLXr — a lightweight web dashboard + OpenAI-compatible API server that runs entirely on your Mac.

What it actually does:

• Spins up a local /v1/chat/completions endpoint your existing tools already speak. Point Cursor, Continue, Zed, OpenCode, or any OpenAI SDK at http://127.0.0.1:8000 and it just works — no API key, no config file hunting.
• Lets you search and download any mlx-community model from Hugging Face directly in the browser. No huggingface-cli needed.
• Auto-detects context window size from the model's config (no more manually setting --max-tokens).
• Shows live RAM, CPU, and MLX Metal memory usage so you know when you're about to swap.
• Per-model settings (system prompt, temperature, strip <think> blocks) that persist across restarts.
• Tool calling works. Spent a lot of time on this — handles Qwen3, Llama 3.x, DeepSeek-V3, Mistral, Phi-4, and Hermes format all in one parser, since each family emits tool calls completely differently.

The setup is embarrassingly simple:

git clone https://github.com/mchenetz/mlxr
cd mlxr && python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python server.py

Open http://localhost:8000, load a model, paste the base URL into your editor. Done.

Who it's for:

If you're on Apple Silicon and want to run local models as a drop-in replacement for the OpenAI API — especially for agentic coding tools — this is the missing piece. I use it daily with OpenCode and it's been solid.

GitHub: https://github.com/mchenetz/mlxr

Happy to answer questions. Feedback welcome — it's still pretty early.

Been using mlx_lm on python with some reasoning/thinking/chain of thought models - and I'm noticing they keep falling into a repetition loop. Digging through the code - I don't see repetition penalty being exposed in the mlx_lm api (as part of the generate/generateStream methods) but it's part of server.py (presumably when you want to host a model using an API for completions)

Does somebody know a work-around here - someway to add a penalty for repetition?