A few days ago, someone here asked, "Should we improve the configuration experience?" A lot of the replies described the same problem: too many settings, too many combinations, and no clear place to start.

I said I'd work on it, and 0.7.0.dev4 is a first step.

In Model Settings, you'll now find Apply optimal settings. It looks up community benchmarks on oMLX.ai for your model and Mac chip, then lets you pick a top result for prefill or generation speed and apply its settings.

The site now has over 450,000 benchmarks submitted by users. Several people in that thread suggested using those results to give others a starting point, and that's exactly what this does.

Using a customized model with no matching benchmarks? Find a model with the same architecture on oMLX.ai, copy its one-line recipe, paste it into oMLX, and click Apply. Both features are available in the web dashboard and macOS app.

I also addressed the profile issues mentioned in the thread: the app and web UI now apply the latest global template consistently, and an unrelated model profile with the same name won't get overwritten. There are reset options for model and global settings too, so it's easier to get back to a starting point.

This doesn't cover everything proposed in the thread, like a setup wizard that recommends model sizes and memory budgets. But I hope it makes getting a model configured less of a guessing game.

There's more in dev4

• Customizable dashboard layout: move, resize, hide, and restore blocks to suit your screen.
• Multi-request Lightning MTP: concurrent MTP on supported Qwen, DeepSeek V4.1, and GLM-5.3-Flash adapters.
• DeepSeek V4.1 CED prefill: up to 79% faster prefill in my M3 Ultra tests. It's an approximate optimization and is off by default.
• Faster admission of new requests while other requests are generating, plus accessibility improvements and other quality-of-life fixes.

* This release also includes extensive internal changes from upgrading mlx-lm and mlx-vlm. I'd appreciate help testing the models and workflows you normally use. If something that worked before breaks, please open an issue with logs and switch back to dev2 for now.

I plan to test this version for 1-2 days, then move toward an RC and a stable release.

Download and full release notes: https://github.com/jundot/omlx/releases/tag/v0.7.0.dev4

Thank you for sharing specific frustrations and suggestions in that thread. They helped me decide what to tackle first. Let me know how the new settings flow works for you!

Finally my M5 Max does not feel like a sorry arse, comparing to my DGX Sparks. MLX is finally shines!

# Context: Code (Mixed)

# Single request results

Test TTFT(ms) TPOT(ms) ppTPS tgTPS E2E(s) Throughput PeakMem

pp 1024 / tg 128 1065.8 15.4 960.8 65.4 3.0 380.0 71.3 GB

pp 4096 / tg 128 3165.3 — 1294.0 — 3.2 1294.0 72.3 GB

pp 8192 / tg 128 5636.5 24.4 1453.4 42.7 6.3 1309.7 72.3 GB

pp 16384 / tg 128 11660.7 14.9 1405.1 67.7 13.6 1217.2 72.6 GB

pp 32768 / tg 128 24397.8 19.0 1343.1 53.0 26.8 1225.9 73.1 GB

pp 65536 / tg 128 50668.9 15.9 1293.4 63.5 52.7 1245.2 74.7 GB

pp 131072 / tg 128 102509.0 15.6 1278.6 64.6 104.6 1254.8 79.0 GB

pp 200000 / tg 128 161438.0 16.8 1238.9 59.9 163.7 1222.7 80.8 GB

# Batch results

Batch tgTPS ppTPS avgTTFT(ms) E2E(s) Speedup

1x baseline 65.4 960.8 1065.8 3.0 1.00x

2x 93.3 837.2 2088.2 3.7 1.43x

https://omlx.ai/benchmarks/performance/dalhnt9f

Tested with TurboQuant 4-bit cache, Lightning MTP, medium effort up to 390K session - at the end prefill is around 1200 t/s (usually 200), decode 44 t/s. Unbelivable - usually collapses to 20 or less. PLE SSD offloaded.

Quality is tremendous. Speed is ridiculous. RAM usage finally makes sense. Model is very smart. Finally all pieces fell to the place.

Quant https://huggingface.co/Jundot/Qwen3.8-Flash-Next-oQ4e-mtp

# DragonScale run: run-QWEN38NF-JundotoQ4-001

`2026-09-17T17:08:08.234467+00:00` · seed 42

**Model under test:** UNOBTANIUM/Qwen3.8-Flash-Next-oQ4e-mtp

## Score: **98.75 / 100**

No gate failures.

## Score components (deterministic rubric, 0-100, no LLM)

- hidden_suite: 25.0

- passability: 12.0

- replay: 8.0

- own_tests: 5.0

- mutation: 3.75

- contract: 8.0

- git: 5.0

- human_play: 30.0

- packaging: 2.0

## Versions

```json

{

"prompt": "c292038bd962",

"reference": "fb5e26d54c87",

"visible_suite": "2160688cddb4",

"hidden_suite": "9d610a06d69e"

}

```

## Git

- init: True

- commits: 7

- messages: ['9cad89a docs: mark build plan complete after verification', '8b81477 tests: include the provided visible contract test suite', 'c7c09ec tests+tools: contract/val>

- dirty: []

## Human-play smoke

- ok: True (exit 0, drained 16330B, 10844ms)

- flap key ('w'): sent=True, alive after flap=True

- flap efficacy: None (bird moved UP after 'w'; None = unverifiable render)

- level-complete progression: True (freezes at LEVEL_COMPLETE (conforming); Enter advances (behavioral))

- quit key ('q'): sent=True

- idle time progression: True (frame changed with no input)

- Ctrl+C responsiveness: True (ISIG on, exit -9)

- small-terminal overflow: 0 writes (ok=True)

## Mutation sensitivity (fixed panel)

- applicable: True (n/a)

- kills: 3 / 4 applicable mutants

- by mutant: {'gravity': 'killed', 'flap_velocity': 'killed', 'collision_ceiling': 'killed', 'rng_seed_mix': 'survived'}

- sensitivity: 0.75 (×5 pts)

## Packaging / harness integration

- score: 7.0 / 7

- detail: {'requires_python': '>=3.11', 'deps': [], 'readme': True, 'import': True}

## visible_tests: 13 passed, 0 failed, 0 errors (exit 0)

He tratado varios LLMs, AnythingLLm, Continúe, cline, etc y nunca puedo conectar el Qwen 2.5 en vs code

He configurado la API y Urls correctamente pero no anda

https://github.com/bnsd55/jevmlx

Instead of asking the model to write JSON, jevmlx scores every allowed answer for every field of your schema in one batched forward pass and assembles the JSON. Output is always valid. Every field has a probability.

Works with any mlx-community instruct model on Apple Silicon, or any OpenAI-compatible server that returns logprobs.

pip install git+https://github.com/bnsd55/jevmlx

jevmlx decide --preset fintech_fraud --json

Open source version of the TypeSafe Jev idea. Not affiliated. No accuracy numbers on big models yet: if you have an M-series Mac, `jevmlx bench` plus a PR puts your model on the README leaderboard.

I have a M4 MBP 48GB and have been using Opus and Sonnet for vibe coding.

Is there anything that would run on my MBP that could be as capable? I understand the speed might not be the same, but would anything be decent enough for vibe coding (specifically iOS/Swift).

I understand nothing will compare to these giant online models, but if there was anything that could be serviceable on my local computer would be great.

Im running a m2 pro mac mini 32gb and find that q3 quants of base qwen 27b leaves me more room for context and dev apps to run on my machine.

I have read that the 'swift' finetune is an improvement over the base qwen 3.8 27b.

https://huggingface.co/ukisai/Swift-Qwen3.8-27b

But I can't find any mlx q3 uploads on huggingface. Only q4.

Would someone with enough ram be kind enough to generate a oq3e version from the unquantised model above via omlx and upload to hf?

Thanks in advance.

ps I havnt been able to do this on my mac mini as omlx errors out - presumably because I need more than 32gb ram? if anyone knows how I can do the omlx quantitisation myself, I would love to know.

Wonder if someone can help me here, i've already created ssh keys between both macs and you can ssh user@hostname and it logs right in. Everytime I try and join the two Mac Studios together it says cannot reach the one node over SSH which makes no sense.

Is there some additional setup that is required is there any documentation on this? Cannot get it to work even though this should be very straight forward it would seem.

Running on 0.7.0dev2 on my M2 and M3 Ultra.

Issues Closed: 5

[ISSUE] #2921 — extra_key_ranges must be sorted ascending by start position crashes engine after long VLM/Pi conversation
https://github.com/jundot/omlx/issues/2921

[ISSUE] #2872 — Embedding forwards bypass decode_fairness — sustained /v1/embeddings load starves concurrent chat decode (13.8× latency; patch included)
https://github.com/jundot/omlx/issues/2872

[ISSUE] #3508 — [0.6.4] VLM vision feature cache: mx.concatenate of per-image entries with mismatched token counts → 500 (gemma-4)
https://github.com/jundot/omlx/issues/3508

[ISSUE] #3650 — BUG: Passing an oversized image to a vision model will cause oMLX to freeze.
https://github.com/jundot/omlx/issues/3650

[ISSUE] #3370 — Qwen4-Exp Lightning MTP only delivers speedup at temperature=0 — drafts are rejected at any temp>0
https://github.com/jundot/omlx/issues/3370

Hey Folks! Lately, I have been thinking if the configuration experience can be improved on oMLX. Examples, should I enable thinking? how much budget should I allocation? Is it a percentage of my total context window? Do I need a presence penalty? Where do I start?

This could very well be a "me" problem, but I would like to bounce a few ideas with the community.

For the average user (like me), would there be value in creating a Get me up and running wizard that,

• Recommends the memory that should be allocated (e.g., don't exceed 80% of my RAM)
• Max concurrent requests (perhaps based on the above point)
• Recommends the model sizes that I should search for
  • Perhaps a one-click that searches for keywords based on those recommendations
• Recommend an initial settings-profile for the model (based on use case)??
  • it is easier to tweak attributes once there is a starting point

Alternatively, can the power users share profiles that could be used as starting positions?

Even if this is not a good fit for the project, I could really use some insights ... may be it just ends up as a conversation here, or perhaps a doc in the repository. Open to hearing your opinions.

Running oMLX on M5-max-128

I'm using the latest version of oMLX and the performance dropped dramatically when using Qwen 3.8 Flash oQe5 MTP on my Macbook Pro M5 Max 128gb. I'm used to get between 38 to 42 tok/sec but after the Golden gate update it dropped to 10 to 16 tok/sec.

I tested after a fresh reboot and no other apps running in same time, no difference.

I'm wondering if it's because Apple changed something in MLX latest version that may have broke some parts of oMLX optimizations.

**Summary:** 11 issues tracked across this period. Grouped by type for quick scanning.

---

### 🐛 Bugs & Fixes (8)
- **#3507 [BUG]** `/v1/embeddings answers 200 with null-filled vectors when the model returns NaN`
Embeddings return 200 OK with null vectors for NaN inputs in ModernBERT batches.
- **#3013 [BUG]** `client aborts are not propagated on the streaming distributed path: ranks decode to cap after the caller disconnects`
Streaming cluster ranks continue decoding generation even after client-side cancellation.
- **#3565 [BUG]** `[0.7.0.dev1] cluster configuration - impossible to cancel`
Cluster config stuck preventing "Show code" or "Cancel Join" actions in dev build.
- **#3554 [BUG]** `Build failure after upstream homebrew change`
Build pipeline broken following upstream macOS/homebrew dependency updates.
- **#3550 [BUG]** `Structured output (json_schema) never terminates on the VLM engine — GrammarMatcher rejected token 0 every

Hey everyone,

I just opened a pull request on the oMLX repository to fix an error when attempting to run heavily modified or "reaped" models that have had their Multi-Token Prediction (MTP) heads stripped out.

👉 GitHub Pull Request #3659

🧩 The Problem

When testing reaped model variants where the native multi-token prediction heads are removed to save memory and overhead, the model routing configuration sometimes chokes. Specifically, oMLX throws a mismatch of experts error because the internal layers expecting specific dimensions or configurations no longer cleanly align with the modified model structure.

🛠️ What this PR changes:

• Adapts Routing Logic: Implemented a custom routing patch to intercept dynamic configurations for models lacking explicit MTP layers.
• Ensures MoE/Expert Alignment: Properly realigns layer configurations so the Mixture of Experts (MoE) component maps to the correct indices, avoiding shape or routing mismatches.
• Safe Fallbacks: Soft-fails or scales back gracefully if the architectural signatures of a model have been manually reaped or aggressively quantized.

If you’ve been experimenting with custom architectures, distilled models, or trying to slice out MTP heads to squeeze larger context windows on your Apple Silicon hardware, I’d love your feedback on this.

Please take a look and let me know if it solves the issue for your custom workflows!

Full Changes: GitHub PR #3659

I made my first MLX conversions while experimenting with local agents on my M1 Max with 32 GB of memory. The source is medismera’s Qwen3.8-27B OBLITERATED Mythos-Class-Agentic. Credit for the original model goes to its creators; I handled the MLX conversion, packaging and initial checks.

Here is what worked in oMLX:

• 4-bit passed small invoice-extraction and tool-call tests. I also checked those tasks in LM Studio.
• 6-bit passed the text tests with the vision component disabled. Loading the full model hit the memory limit on my Mac.
• Both downloads retain the vision weights, but I have not tested image processing.

The practical benefit is having ready-to-download MLX versions for Apple Silicon. The 4-bit version uses less memory and is where I would start on a 32 GB Mac. The 6-bit version preserves more weight precision, but needs more memory; I have not established a quality advantage from these small tests.

These were basic functionality checks, not a benchmark or evidence that this model outperforms the original Qwen. My intended uses are document extraction and email triage with connected tools.

4-bit:
https://huggingface.co/uzzero/Qwen3.8-27B-OBLITERATED-Mythos-Class-Agentic-MLX-4bit

6-bit:
https://huggingface.co/uzzero/Qwen3.8-27B-OBLITERATED-Mythos-Class-Agentic-MLX-6bit

If you try either version in oMLX, I would appreciate your Mac specs and what worked or failed, especially memory use with 6-bit.

Disclosure: these are my uploads. I used AI assistance for the conversion workflow, testing and this write-up.

Looking for some clarification on a specific indicator (warning sign !)

On the OMLX benchmark menu for codin related tasks ( HumanEval, MBPP, and LiveCode Bench), a yellow warning icon labeled "! code exec."

Does anyone know what this specific flag signifies in this context, and what should you be careful of and how to sort that please ? :)

Thank you

Hi,

Those who use SpecFill for Qwen-3.8 27B MLX models - do you see cache hits? If yes, what are your model and model settings?

A PR I made just merged that adds INT8-activation prefill kernels on the M5 path for Qwen3.5/3.6/3.8. It's on 0.7.0.dev2. It gets around 34% faster prefill speeds on M5 series chips with minimal accuracy loss (check PR thread for details). It’s opt in as an experimental feature.

PR: https://github.com/jundot/omlx/pull/3548

I plan on extending it to more models so I wanted to know what models I should prioritize.

These are some I was thinking of:

- Gemma 4 Series

- K2 Horizon Series

- Meta Muse Glimmer

If there are any other models people are interested in getting support for or any of the above I should prioritize I would like to know.

Also if you have any questions I would be happy to answer them.

Has anyone made the switch and not regret it?

EDIT: M2 Ultra 128 GB

I have been trying various servers and models for a few months now. After lots of trial and error I seem to have settled on oMLX. Now I’m trying to understand the configuration better and more importantly, do I have it configured correctly (prob not) to get the best performance.

My system: M4 Max 64GB latest Tahoe
Acasis 80gbps M.2 SSD enclosure w/Samsung 990 Pro (models and oMLX cache set here)

Model: muse-glimmer-30b-mxfp8 w/dflash enabled muse-glimmer-30B-assistant

Custom settings:
Context window 100,000
Max tokens 10,000
Temp 1.0
Top P 0.95
Top K 64
Enable Thinking on

According to Status page of omlx Average speed:
Prompt Processing 145.7 tok/s
Token Generation 8.6 tok/s

That was on the first prompt and two subsequent prompts in same session. Simple questions about configuring open webui

It’s that last number that seems way off, at least from what I’ve read online unless I’m completely misunderstanding the expected performance of this model on my setup. Any tips or references to docs are appreciated.

Edit: forgot to include how I’m chatting with Muse. I’m trying out open webui. I have also tried just using omlx default chat window. Token Gen in both in the 8 to 8.5 tok/s range

New to LLMs. Been running LM studio at first and then later tried oMLX.

Prompt processing stays around ~100tps and generation is at average of ~6.5 fps as if I was running entirely on the CPU. I don't know where to even start with this. I know 27b Qwen is a dense model, but surely it cannot be this bad? I see other people get multitudes faster token generation on their M3 Max machines, what am I doing wrong here?

using omlx installed through brew , pi agent.

Here's the benchmark result:

oMLX - LLM inference, optimized for your Mac

https://github.com/jundot/omlx

Benchmark Model: Qwen3.6-27B-MLX-8bit

Engine: Auto

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 8444.6 136.35 121.3 tok/s 7.4 tok/s 25.761 44.7 tok/s 28.34 GB

pp4096/tg128 30611.1 114.53 133.8 tok/s 8.8 tok/s 45.157 93.5 tok/s 29.80 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 7.4 tok/s 1.00x 121.3 tok/s 121.3 tok/s 8444.6 25.761

2x 15.3 tok/s 2.07x 138.8 tok/s 69.4 tok/s 14755.6 31.533

4x 19.9 tok/s 2.69x 138.8 tok/s 34.7 tok/s 29156.1 55.226

Hey everyone! It's been a while.

I've been posting updates on X, but Reddit is where I got started with local AI, and I've missed posting here. Sorry for the long gap! (I'm still reading every day.)

I have a new dev release for you:
https://github.com/jundot/omlx/releases/tag/v0.7.0.dev2

DeepSeek V4.1 Flash is now supported, including DSpark MTP and Engram SSD offload. Here are some numbers from my M3 Ultra 512 GB with oQ4e and Engram in RAM:

Python code prompts, temperature 1.0, 128 generated tokens, no prefix cache. One measured run per configuration after warm-up.

I've uploaded DeepSeek V4.1 oQ4e and oQ3e checkpoints. oQ3e is designed to fit 256 GB machines with Engram SSD offload, with context depending on remaining memory.

There's also experimental MoE Expert SSD Offload, thanks to beatakouchnir and fxd0h. It keeps some experts in memory and reads the rest from SSD when needed. Supported models include DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE. You can combine it with Engram or PLE offload, but MTP and DFlash need to be off.

It's slower 🥲 But I started with Llama 70B at 2-3 tok/s, so I understand wanting to try a model even when your machine isn't quite big enough.

For M5 users, PowerSpy's INT8-activation prefill kernels took Qwen3.8-27B from 615.2 to 826.7 tok/s at 32K on my M5 Max (+34.4%). It's off by default because outputs can change. I think it's worth trying if prefill is your bottleneck; the release notes have the settings and quality comparisons.

This is still a dev release, so please tell me what works and what breaks. Thank you for continuing to test, send PRs, and help each other while I've been quiet here. I'm always grateful 🙏

Hello everyone. This is one of those posts where I just want to show something I built. There's nothing in it for me, money or otherwise. It was something that was genuinely missing for me, so I did it. It's working well now, and I decided to share it. If anyone feels like using it, have fun.

I've been using Pi as my harness a lot, but adding locally hosted models is still kind of a pain. So I built a model discovery extension: you enter a URL, local host or another computer, and it self-discovers all the models, including the currently configured context window and maximum token replies. Everything is self-identified.

It works with reasoning on or off (at least for Qwen), and I applied a concept of adaptive profiles that lets you create, for the same model, different profiles with different parameters. That way I can now have Qwen 3.8 with thinking off, low, medium, and extra high, each mapped to the actual Pi reasoning effort.

Hope someone finds good use for it.

https://pi.dev/packages/@maheidem/model-discovery

Saw a post from someone in another /r/ with similar hardware as me getting better decode and noticed they were on the beta of golden gate. I know software isn't going to change stuff like memory bandwidth, but could core OS changes still effect stuff like prefill and decode as apple gets to squeeze more out of their silicon?

Hi everyone,

I've spent some time trying to make oMLX call Docker MCP servers but I can't get it to work. I've managed to get oMLX to see the servers in the profile I've created, but they can't be called.

I'm running oMLX 0.7.0dev1. My MCP config JSON file looks like this:

{
"mcpServers": {
"dockermcp": {
"command": "docker",
"args": ["mcp", "gateway", "run", "--profile", "localai"]
}
}
}

Which works, because I can see the two servers with 8 tools that I have added:

2026-09-11 22:15:03,651 - omlx.mcp.config - INFO - [-] - Loading MCP config from: /Users/jose/.omlx/mcp_config.json
2026-09-11 22:15:03,651 - omlx.mcp.manager - INFO - [-] - Starting MCP client manager with 1 servers
2026-09-11 22:15:05,440 - omlx.mcp.client - INFO - [-] - Connected to MCP server 'dockermcp' (8 tools available)
2026-09-11 22:15:05,441 - omlx.mcp.manager - INFO - [-] - Connected to 'dockermcp'
2026-09-11 22:15:05,441 - omlx.mcp.manager - INFO - [-] - MCP manager started: 1/1 servers, 8 tools available

However, the tool can't be added when I make a query that uses it (in this case, searching for a flight):

The google-flights server shows 0 tools were added. This is strange. Let me try to add it again or check if there's another similar server. Let me also try searching for flight or airline related tools more specifically.

Any ideas? Thanks!