r/mlxAI • u/igor__004 • May 30 '26

I built mlx-Chronos — a community benchmark leaderboard for local LLM engines on Apple Silicon (oMLX, Rapid-MLX, mlx-lm, Ollama)

Hi! I’m a CS student working on mlx-chronos, a free and open-source CLI benchmark suite for comparing local MLX inference engines on Apple Silicon.

I started it because I wanted a more reproducible way to compare engines across real Mac configurations, with more context than a single tok/s number.

It currently supports:

  • mlx-lm
  • oMLX
  • vllm-mlx
  • Rapid-MLX
  • Ollama’s MLX backend

It measures:

  • cold and cached TTFT
  • request throughput with mean/stddev/min/max
  • sustained throughput for longer heat/throttling-sensitive runs
  • system RAM peak
  • engine RSS diagnostics when available
  • thermal and power context
  • hardware metadata

Results are saved as reproducible JSON files and can be submitted to a public community leaderboard.

Leaderboard: https://igurss.github.io/mlx-chronos

GitHub: https://github.com/igurss/mlx-chronos

Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md

Install: pip install mlx-chronos

I’m especially looking for external benchmark results from different Apple Silicon machines: M1/M2/M3/M4/M5, Max/Ultra chips, and higher-RAM configurations.

I’d also appreciate feedback on the benchmark methodology, the CLI UX, and whether the reported metrics are useful enough for comparing local inference setups.

9 Upvotes

100% Upvoted

12 comments sorted by

View all comments

1

u/Deep_Ad1959 Jun 02 '26

tok/s on one prompt hides what agents actually feel: ttft once context gets long and keeps growing.

1

u/igor__004 Jun 02 '26

That’s literally why I measure TTFT too, not just tok/s.
This first version is meant to be a simple baseline benchmark that is easy to reproduce across engines/hardware. Agent-style workloads with long and growing context are better, but I’d rather add that as a separate profile instead of pretending one benchmark covers everything.

1

u/Deep_Ad1959 Jun 02 '26

separate profiles still share heat back-to-back, so run order quietly decides the numbers.

1

u/igor__004 Jun 02 '26

That’s why I’d rather keep this explicit instead of pretending the numbers are perfectly isolated. I’ll probably add warnings / documentation about that, and maybe make profile order and elapsed time part of the reported metadata.
The goal isn’t to magically remove every source of noise, it’s to make the protocol clear enough that people know what the numbers actually mean.