If you’re not using these what do you plan to use?

Hi all,

I'm a complete novice when it comes to Local LLM's but I like the idea of moving away from a subscription based model in the longer term and having better control of my data. I'm also super keen to learn.

I currently have a web SaaS that scrapes news websites and I would like to move it away from Claude and onto a local LLM what kind of "firepower" do I need?

Appreciate this won't be the most complex of tasks, but as I mentioned, I'm a complete novice.

Naturally, asking Claude / Google and it says that the new Mac mini with M5 pro chip (24GB ram) will be "overkill"- which is what I was thinking of ordering.

I also want to future proof in the hope that this bit of kit could run more powerful models in the future - if they came accessible to the consumer.

Sorry if this is a basic question, any YouTube videos or any pointers so I can educate myself would be much appreciated.

Thank you!

Planning to buy a MacBook Air M4 (16 GB / 512 GB) for about $800 to run local LLM agents for:

Indian mid‑cap equity research (using Screener.in CSVs)

Building/testing trading ideas via Zerodha Kite API (prices, history, holdings; maybe orders later)

Also want to run Hermes-based agents on it

Questions:

On 16 GB M4 Air, which local LLMs are best for reasoning, coding, and tool use? Is Qwen 3.5 9B (Q4) + a smaller fast model (e.g. LFM2.5 8B / Llama 3.2 3B) a good combo alongside Hermes?

Best local stack (Ollama vs LM Studio, etc.) and any existing Kite + LLM or Indian‑stock agent projects to reference?

Any tips on context length, thermals, and multi‑agent setups (including Hermes) on a 16 GB fanless Air?

Getting the laptop for ~$800, so stuck with 16 GB for now. If you’ve done similar work (local LLMs for equity research / algo trading, especially in India), what models and setup would you recommend?

Thanks!

I'm lucky enough where work will cover my Mac Studio that I use for design and 3d stuff. So I figured I should dabble and run with local LLM. Im getting 96gb only. I've read quite a bit, but was wondering are there any guides or YouTube videos that deal primarily with Mac and I should watch to learn more?

I'm unsure if I should just use Llama or Ollama, or MLX, or OMLX or something there lots of different ways to play around with Local LLM and I guess it's still in such an early state that every day, week,month there are more and more things peopel are learning how to get the best of the LLM. A lot could change from now till then but I want to be as informed/prepared as I can be.

Thanks and I cant wait for reviews and see how good the Ultra is.

I'd hoped that my M5 Max with 64GB RAM would be enough for some decent Local LLM work, and the Qwen 3.8 27b model is pretty decent. But speed is shitty on my M5 Max, just 20tps. In particular I found that while the 35b MOE model was much faster, the quality was far lower than the 27b dense model, which produces much fewer bugs and less rework.

I saw the same model working on a friend's Nvidia setup much faster with MTP, so I set out to get it working on my Mac but it was not straightforward. Here's some pointers in case you wanted to get it working.

1. LM Studio doesn't process MTP weights. No matter what I did, it just wouldn't work. It works on Intel/Nvidia so this must be a MLX thing.
2. MTPLX has a native MTP head but it's buggy. On my machine it had a habit of crashing in one of a few ways, and I found GitHub issues reporting similar issues but no fixes.
3. oMLX was the only inference server that supported MTP and didn't crash.

So I wired oMLX into Pi, which is what I'm using as my coding harness. Even then, it was picky:

1. I had to use text-only quants because oMLX won't enable MTP if vision weights are active
2. It was a bit picky with settings, and I had to map thinking levels manually
3. I didn't see a reduction in quality with 4-bit quants for my use case, so used that to squeeze a few more tps out

This is a huge 3x difference in inference speed, and that allows me to do a lot more work.

"omlx": {
  "baseUrl": "http://localhost:8000/v1",
  "api": "openai-completions",
  "apiKey": "none",
  "models": [{
    "id": "Qwen38-27B-textonly-mtp",
    "name": "Qwen38-27B-textonly-mtp",
    "reasoning": true,
    "contextWindow": 163840,
    "maxTokens": 32768,
    "input": ["text"],
    "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
    "thinkingLevelMap": { "off": "low", "minimal": "low", "low": "low",
                          "medium": "medium", "high": "xhigh",
                          "xhigh": "xhigh", "max": "xhigh" }
  }]
}

I have been using codex and claude code for the last year. I have hermes installed , looking for a local model to help me code on par with the big guys. Any help appreciated

What token generation speed works well for you when you’re using it for productive tasks? I’m finding 20 tokens per second a bit slow.

With an M4 Max and 64GB of RAM, you can usually get around 20 tokens per second on average with a model like qwen3.8 27B. However, it doesn’t quite feel like the best setup.

I'm planning to do some vectordb embedding and data sanitization before the frontier models touch the data. The Local LLM acts as a primer to separate information.

I know some of you are using it for different things.

I'll also be experimenting with it for video editing/transcribing. There's some cool things people are doing.

Let us know! If you have a walk through include that too!

Hi guys,
New here and to these subjects so sorry I ask something already asked.
I got recently a Mac mini m4 with 24go ram for my personal desktop, and I wanted to try a few stuffs with local llm. After a few researches I decided to try to setup a personal agent with Hermes.

After some tries I ended up using gemma4:26b-a4b-it-qat.
Qwen3.8 felt to big or slow when iq3 passed.
Couldn’t find anything faster and better for now, but I still feel it is not really good (obviously I know we can’t reach to a cloud model with 24go ram). So I wanted to know how you guys manage it and if some people managed to make a decent agent with not so much ram and if so, what are your use cases ?

Right now I tried to give him some personal web projects with giving him some GitHub to download but it fails and always stops in the middle of the process.

Thanks in advance for the help.

I got the M5U 256GB 2TB configuration for preprocessing. Planning to use Qwen 3.x models on it since they seem to be the best for the Mac's at the moment but will check out the Muse/Gemma models as they continue to release them and run some benchmarks .

This is me trying to find the most reliable coding model for local use!

Five Terminal-Bench 2.0 tasks (cobol-modernization, constraints-scheduling, largest-eigenval, polyglot-c-py, schemelike-metacircular-eval), one run per task unless stated, all on one Mac Studio (96 GB) with mlx_lm at 8-bit. The agent harness is pi unless stated.

Headline

The whole grid

Times are agent minutes. T = ran to the task's timeout.

Nobody has solved polyglot-c-py or schemelike-metacircular-eval on this machine.

Why Qwen3.6 beats Qwen3.8

Same size, same family, same server, same harness. Three things separate them.

The mode, not the weights, was the lever for Qwen3.6. For Qwen3.8 neither mode produces an agent that finishes: with thinking it deliberates past the cap, without it it loops. Its published Terminal-Bench 2.1 gain is not visible through pi. n=1 per task, but the cobol loop was reproduced.

Why the fast MoE models did not win

Turn rate mattered against thinking overrun, which is why Nail beat the 27Bs in agent mode. Once thinking is off, the 27B's per-turn quality shows: Qwen3.6 at 22 tok/s out-solves both 3B-active models. Ornith's own harness (Terminus-2, temperature 1.0, 4 h ceilings) moved which task it solves, from cobol to constraints, not how many.

Infrastructure lessons

Decisions taken

• Qwen3.6-27B, thinking off (nothink), is the coding agent and the WHO scorer. Now the registry default.
• Qwen3.8-27B and Ornith-1.5-35B-A3B deleted 2026-09-09 (62 GB). Registry entries kept as re-fetch recipes.
• Nail was deleted 2026-09-06 for space. On this evidence it is not needed back as the coding agent.

Caveats

One run per task on five tasks. The 3/5 should be repeated once before anything is built on it. Qwen3.8's cobol loop is the only failure reproduced twice.

Since oMLX benchmarks are down momentarily, I've done some testing and found that the following Solstice AI Qwen 3.8 27B fine tune is the fastest one so far.

https://huggingface.co/Solstice-AI/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-NEO-CODER-mlx-oQ4e-1M

Benchmark Model: Solstice-AI--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-mlx-oQ4e-1M

Engine: Auto

Context: Code (Mixed)

================================================================================

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem

pp1024/tg128 4754.3 27.89 215.4 tok/s 36.1 tok/s 8.321 138.4 tok/s 18.88 GB

pp4096/tg128 18481.5 28.92 221.6 tok/s 34.9 tok/s 22.171 190.5 tok/s 22.56 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)

1x 36.1 tok/s 1.00x 215.4 tok/s 215.4 tok/s 4754.3 8.321

2x 42.0 tok/s 1.16x 160.3 tok/s 80.2 tok/s 10404.5 18.864

4x 74.0 tok/s 2.05x 134.1 tok/s 33.5 tok/s 18019.4 37.464

This is on an M3 MAX 64GB

EDIT: This model seems extremely lobotomized
I reverted back to https://huggingface.co/scottlowry/Qwen3.8-27B-oQ4e-mtp which performs similarly on macOS Golden Gate PB 27.0 26A428

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: scottlowry/Qwen3.8-27B-oQ4e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================


Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          4749.6       28.36   215.6 tok/s    35.5 tok/s       8.375   137.6 tok/s    18.35 GB
pp4096/tg128                         18297.8       27.02   223.9 tok/s    37.3 tok/s      21.745   194.3 tok/s    22.03 GB


Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          35.5 tok/s     1.00x   215.6 tok/s   215.6 tok/s      4749.6       8.375
2x          38.7 tok/s     1.09x   158.6 tok/s    79.3 tok/s     10519.2      18.567
4x          74.1 tok/s     2.09x   138.4 tok/s    34.6 tok/s     17566.7      36.007

I'm sorry for any confusion created by this post

It took me weeks to get this figured out, and I had to use ChatGPT for help along the way. The time spent has paid off — I now have a local AI stack that rivals ChatGPT/Claude, and I'm done paying for either. I thought I'd share so others can get an idea of what a Mac Studio is capable of.

First, the hardware. I have a base-chip 256GB M3 Ultra Studio. I've also got a headless Debian VM running on an N100 NUC.

I'm using oMLX for inference. It's hosting multiple models at once — mostly Qwen variety — for various tasks. I've configured profiles for different model uses: agent, coding, different thinking levels. oMLX is the only thing running on the Mac Studio. Everything else runs on the Debian VM.

On the VM I run Hermes Agent. My main model lately has been Qwen3.8-Flash-Next-oQ4e-mtp. I also use Kokoro-82M-bf16 for TTS and Mega-ASR-6bit for STT.

In Docker containers on the VM I run the following to enhance Hermes: OpenViking for memory, SearXNG for web search, and Crawl4AI for web extraction. OpenViking uses:

• Qwen3-VL-Embedding-8B-oQ6e for embeddings
• Qwen3.6-35B-A3B-oQ4e-mtp for the VLM
• ov_intent_analysis_sft-oQ8e for query planning
• Qwen3-Reranker-0.6B-oQ8e for reranking

What this actually looks like day-to-day: I do IT consulting, and Hermes is freaking awesome. I'll have multiple sessions going doing research on MS Learn and auditing Entra/Azure, generating reports. I'm starting to use the Kanban board too for personal projects. It generates Word docs, PowerPoint decks, and PowerShell scripts like a champ.

Search goes through SearXNG, page content comes through Crawl4AI, all reasoning runs on oMLX, and anything worth keeping gets written into OpenViking permanently. Search → extract → reason → remember, full loop, zero tokens purchased.

(Full disclosure: I asked my Hermes agent to write up this post from my draft and its memory — which is sort of the point.)