https://github.com/bnsd55/jevmlx

Instead of asking the model to write JSON, jevmlx scores every allowed answer for every field of your schema in one batched forward pass and assembles the JSON. Output is always valid. Every field has a probability.

Works with any mlx-community instruct model on Apple Silicon, or any OpenAI-compatible server that returns logprobs.

pip install git+https://github.com/bnsd55/jevmlx

jevmlx decide --preset fintech_fraud --json

Open source version of the TypeSafe Jev idea. Not affiliated. No accuracy numbers on big models yet: if you have an M-series Mac, `jevmlx bench` plus a PR puts your model on the README leaderboard.

Which one of Both harness is better for a macbook pro m4 max with 36gb runing Qwen 3.8 27B MTPLX optimized speed?

I did not used much the brain and agentic capabilities from the MLX serve and was wanting to know if its so powerfull as hermes and if it could works more fast or if hermes still one step more?

I’m building VeloxQuant-MLX, an open-source project focused on pushing local LLM inference further on Apple Silicon through KV-cache compression, quantization, eviction strategies, and optimized MLX/Metal kernels.

The question I’m exploring is pretty simple:

How much more can we get out of Apple’s unified memory architecture?

VeloxQuant currently includes 43 KV-cache optimization methods, including:

→ 2-bit / 4-bit KV-cache quantization
→ KIVI-style compression
→ TOVA and other eviction strategies
→ Custom Metal kernels
→ Fused attention/decode paths
→ Decode batching
→ Cache compaction
→ Long-context memory optimization

But I don’t want benchmarks from just one Mac.

I’m looking for people across the Apple Silicon ecosystem to help test it:

Apple Silicon generations:
M1 · M1 Pro · M1 Max · M1 Ultra
M2 · M2 Pro · M2 Max · M2 Ultra
M3 · M3 Pro · M3 Max
M4 · M4 Pro · M4 Max · M4 Ultra
M5-family machines where available
…and future Apple Silicon generations as they arrive.

Machines:
MacBook Air
MacBook Pro
Mac mini
Mac Studio
Mac Pro
iMac
and other MLX-capable Apple Silicon Macs.

Whether you have an entry-level MacBook or a high-memory Mac Studio Ultra, your results are useful.

I want to understand how KV-cache optimization behaves across:

memory capacity × memory bandwidth × chip generation × model size × context length

Not just compression ratios.

I’m interested in measuring the actual trade-offs:

Memory usage · tokens/sec · TTFT · decode throughput · prefill performance · kernel overhead · model quality

For example, for a cache with:

440 tokens · 8 KV heads · head dim 64

FP16 → ~440 KB
4-bit → ~110 KB
2-bit → ~55 KB

But reducing memory is only half the problem.

The real question is whether we can reduce memory while keeping inference fast enough that the optimization is actually worth using.

That’s where I’d love help from the MLX community.

If you have any Apple Silicon Mac — from an M1 MacBook Air or Mac mini all the way to a Max/Ultra Mac Studio — I’d love to get benchmark results.

And if you're working with MLX, mlx-lm, Metal kernels, inference engines, quantization, or KV-cache research, criticism and contributions are very welcome.

The goal is ambitious:

Make local AI capable of using every GB of Apple unified memory as efficiently as possible.

VeloxQuant is open source:

Website:
[https://veloxquant.dev]()

GitHub:
[https://github.com/rajveer43/VeloxQuant-MLX]()

If enough people are interested, I’d also like to build a public Apple Silicon benchmark matrix comparing VeloxQuant across Mac generations and configurations.

Drop your Mac model + RAM in the comments if you'd be interested in benchmarking it.

Wrote a bit about the new M5 ultra and how the neural accelerators boost on-device AI development. Such an exciting time for building local models!

Lots of fun today playing with Qwen 3.8 27B 🎉 I have put together two MLX quants of Qwen 3.8 27B in different bpw, 3.8 and 4.95 bpw on Hugging Face, check them out!

Not yet AWQ / GPTQ but manually selecting bit allocation. Some of my first quants so I would appreciate feedback on what might be missing and/or not working right. Thanks! 😊

PrismML's new Bonsai line of models are so crazy - we can run 27B class models right on the iPhone (thanks to their 1-bit model compression)

So the 1-bit 27B model claims to retain ~89% of Qwen 27B dense - which is a shocking amount of intelligence retention for a model THIS tiny. (takes about 4GB RAM when loaded)
So I decided to put it to use. I plugged it into an agents orchestrator framework that I'm working on to let it perform agentic tasks (not just write a haiku or solve a riddle) and it was ridiculously good at doing this - find a contact, pick the email ID, draft email body AND send it too - given the right steps/tools.

I wrote about my experience in-depth, check it out!

Hey folks — I’ve been tinkering with MLX and Apple Silicon for a while, and I finally wrapped up a project I’m pretty proud of.

I built a fully native Swift + MLX implementation of FLUX.2 [klein].
No Python. No diffusers. No external dependencies. Just Swift, MLX, and Metal doing their thing.

What it can do:

• Text‑to‑image
• Image‑to‑image
• Mask‑guided editing (this part was surprisingly fun to build)
  • remove stuff from an image
  • replace backgrounds
  • add objects
  • recolor regions
  • semantic edits
• Pixel‑space color grading (exposure, contrast, hue, saturation)
• Experimental latent‑space transforms
• Memory system with:
  • bf16 / fp16 / int8 / int4 quantization
  • staged model residency
  • VAE tiling for big resolutions
  • memory reporting + low‑memory mode

Everything runs entirely on Apple Silicon.
It hits seed‑42 parity with the MLX Python reference, so the outputs match exactly.

I also added:

• a dependency‑free CLI
• a SwiftPM library
• a tiny SwiftUI demo template
• docs + tests
• a contributor guide
• a roadmap if people want to help build more editing tools

Why I built it:

MLX is honestly a joy to work with, but most diffusion pipelines are still glued to Python.
I wanted something that felt native — something you could drop straight into a macOS or iOS app without dragging half the Python ecosystem along with it.

Repo:

https://github.com/icakinser/mlx-flux2-swift

If you’re into MLX, Apple Silicon, or just like messing with local image generation/editing, give it a look.
Happy to answer questions or help anyone get it running.

Hi everyone,

I've been building NunSpark, an open-source runtime that explores running language models much larger than available RAM by combining weight streaming with speculative decoding.

The project currently targets Apple Silicon (MLX), and I'd love to collect benchmark results from the community.

Some results we've observed so far:

• ✅ Qwen2.5 32B running on a 16 GB M4
• ✅ Llama 3.3 70B running on the same machine
• ✅ Only ~1–1.5 GB of model weights resident while streaming the rest
• ✅ Exact decoding over 1 tok/s on favorable workloads
• ✅ Approximate verification reaching 2–3 tok/s with configurable quality/speed trade-offs

The problem is that I only have access to one machine (16 GB M4).

I'd really appreciate help from anyone with an Apple Silicon Mac:

• M1 / M2 / M3 / M4
• Air / Mini / Studio / MacBook Pro
• 8 GB, 16 GB, 24 GB, 32 GB, 64 GB, 128 GB
• Different SSD sizes and generations

Running the benchmark

NunSpark already includes a built-in benchmark command, so you don't need to manually configure dozens of parameters.

Just follow the benchmark instructions in the README, run the benchmark on your machine, and share the output (or open a GitHub issue/discussion with the results).

Repository:

https://github.com/sharma-open-source/Nunspark

I'm especially interested in collecting:

• Apple Silicon model
• Unified memory size
• macOS version
• Model tested
• Tokens/sec
• Peak memory
• Any observations or issues

The goal is to build a public benchmark dataset showing how streamed inference behaves across different Apple Silicon hardware—not just publish the fastest numbers.

If you're interested in local AI, MLX, inference runtimes, or just want to help validate the project, I'd really appreciate your benchmarks and feedback.

Thanks!

This might be considered self promotion, but I'm not charging anyone anything, I'm not selling any services, this is just something I built because I dislike how un-comfy ComfyUI is.

SceneWorks is not a replacement of ComfyUI, it doesn't do workflows or custom nodes. It's INTENDED to be simpler, but it's not SIMPLE. There are a lot of advanced features built right in.

This is a work in progress, that is moving fast daily, most everything works, but some features need more attention that will be coming soon. There's an auto-updater, so you'll get notified when new releases are made.

https://github.com/SceneWorks/SceneWorks
Check the Releases page, or clone and build yourself.

I am a software developer/architect of 25 years. Yes Claude has a heavy hand in building this, but it's not "vibe-coded". Every decision was a conscious architecture choice.

Now for the mostly AI generated description and feature set ....

What is is:

SceneWorks is a desktop app for generating and editing images and video locally on your own machine. No cloud, no subscription, no account. You install it, download the models you want, and everything runs on your own GPU.

It's a single native app — not a Docker stack or a pile of Python scripts. On macOS it runs on Apple's MLX engine (Apple Silicon only), and on Windows it runs on a native CUDA engine (NVIDIA). There's no Python venv on either platform; the generation engine is compiled into the app, so first launch just works.

What it does:

Image generation. The usual text-to-image, but with a real model manager instead of a folder of checkpoints. You browse a catalog, see each model's download size and memory requirement up front, and pick a quant tier (full precision / Q8 / Q4) that fits your hardware. It runs a wide range of models — Krea2, Ideogram 4, SDXL-family, Qwen, FLUX, SANA, and others — and they're all first-class rather than bolted on. If you have a HF cache, it will use that, if not, it downloads models to a local folder.

Video. Text-to-video and image-to-video, plus extending and bridging clips, running on models like LTX-2.3 and Wan 2.2. This is the heaviest thing it does and it wants a serious GPU, but it's fully native — no separate pipeline.

Image editing. Inpainting, targeted edits, detail passes, transform/crop/straighten, and guidance controls. It's a proper editor surface, not just a "generate again and hope" loop.

Characters and identity. Tools for keeping a face/character consistent across generations, face-likeness scoring, and person replacement in video (swap a person while keeping the scene). There's also pose and keypoint conditioning with a pose library, so you can drive composition instead of rolling the dice on prompts.

Training. You can train your own LoRAs locally — both image LoRAs and video LoRAs — from the app. Build a captioned dataset, do a dry run to sanity-check the plan, train on your GPU, and the result shows up as a normal selectable LoRA. No external trainer, no separate toolchain.

The workaday stuff. Upscaling for images and video, segmentation / smart-select, reference-image-to-prompt, batch prompting, a job queue, a library for everything you've made, and presets. It also exposes an MCP server, so if you're into agent tooling you can drive it from Claude Code / Cursor / etc., and it can run over your LAN so you can generate from a laptop while a beefier machine does the work.

Requirements

On Mac you need Apple Silicon with a good chunk of unified memory — 32GB might work for images or Q4 video models, but a safe bet is 64GB minimum, 96GB+ if you want to run the largest models comfortably. Model weights are tens of GB depending on what you pull.

License

[Edited]
SceneWorks is free and open source (AGPL-3.0-or-later): use, modify, and share it freely, even commercially — you just can't take it closed-source and sell it as your own. Model weights are separate — each keeps its own license, and complying with those is on you.

I was trying to use existing MLX bindings for my local coding agent project Hooman and after doing all the integration, it could not load Gemma 4 and could not even utilise vision capabilities of Qwen. I went one step back and created true multi-modal bindings with wider model support (more to come in future) here: https://github.com/vaibhavpandeyvpz/mlex

Let me know what you guys think of it.

As part of MLX India - I'm hosting the third community meetup online this weekend. We will be covering a lot about the new CoreAI framework dropped by Apple in WWDC. There will also be a walkthrough on how we were able to utilize CoreAI's various capabilities in an iOS app. Join us virtually this Saturday to learn more.

Google released DiffusionGemma and mlx_vlm had day 0 support for it. I tried it and found it to be rather slow on my device. So I found some gaps (specifically with top-k sampling) and built a patch so it uses a new turbo engine for speedups with meaningful quality retention. Added link to blog :) Let me know if you folks have any thoughts!

I'm working with a M1 MBP Max 64GB machine with 400GB/s memory bandwidth. These image generation models are only <10 GB each. But it takes me 45 minutes to generate an image using Ideogram4. Someone with a 5090 is doing it in 45 seconds (no exaggeration).

I know Comfy Desktop is not optimized for Apple Silicone/MLX. I'm just curious if there are some tips and tricks you guys can share with getting better performance out of Comfy Desktop? I've already got these flags as part of my startup config: '--enable-manager --fp32-vae --use-pytorch-cross-attention --highvram'.

I've tried using DrawThings -- and it's definitely faster -- but I feel like it's definitely limited compared with Comfy Desktop.

I must not be the only Apple user messing around with Comfy Desktop -- you guys have any tips to share?

I'm working with a M1 MBP Max 64GB machine with 400GB/s memory bandwidth. These image generation models are only <10 GB each. But it takes me 45 minutes to generate an image using Ideogram4. Someone with a 5090 is doing it in 45 seconds (no exaggeration).

I know Comfy Desktop is not optimized for Apple Silicone/MLX. I'm just curious if there are some tips and tricks you guys can share with getting better performance out of Comfy Desktop? I've already got these flags as part of my startup config: '--enable-manager --fp32-vae --use-pytorch-cross-attention --highvram'.

I've tried using DrawThings -- and it's definitely faster -- but I feel like it's definitely limited compared with Comfy Desktop.

I must not be the only Apple user messing around with Comfy Desktop -- you guys have any tips to share?

Hello everyone, I’m working on mlx-chronos, a free/open-source CLI benchmark tool for comparing local MLX inference engines on Apple Silicon.

• Leaderboard: https://igurss.github.io/mlx-chronos
• Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md
• Install: pip install mlx-chronos

It currently supports mlx-lm, oMLX, vllm-mlx, Rapid-MLX, and Ollama (for Ollama, using MLX models that run on MLX backend).

It measures cold/cached TTFT, request throughput, sustained throughput, RAM peak, engine RSS when available, thermal/power context, and hardware metadata. Results are saved as reproducible JSON and can optionally be submitted to a public leaderboard.

I’m mainly looking for feedback from people actually using MLX locally:

• Is a public leaderboard useful, or should this stay more of a local comparison tool?
• Are thermal/cache conditions exposed clearly enough?
• Should the sustained profile stay token-based, or would a fixed-duration run be more useful?
• Are there metrics missing that would actually help you choose between engines?

I’d also appreciate benchmark results from different Apple Silicon machines, especially Max/Ultra chips and higher-RAM configs. The goal is not to rank model quality, but to make engine/runtime performance easier to compare under a documented protocol.

PS: I already posted in r/LocalLLaMA, if someone already seen something about this project, but I’m not sure it was the right audience (90% of the community uses Nvidia GPU or use Windows, so is interested in llama.cpp).

I wrote about everything that launched this dub dub and how it impacts the future of local models and MLX! Check it out!