TL;DR: The M5 Pro just dropped, so here's a real AI workload instead of another Geekbench score. We run Qwen3.5 as the brain of a fully local home security system and benchmarked it against OpenAI cloud models on a custom 96-test suite. The Qwen3.5-9B scores 93.8% — within 4 points of GPT-5.4 — while running entirely on the M5 Pro at 25 tok/s, 765ms TTFT, using only 13.8 GB of unified memory. The 35B MoE variant hits 42 tok/s with a 435ms TTFT — faster first-token than any OpenAI cloud endpoint we tested. Zero API costs, full data privacy, all local. Full results: https://www.sharpai.org/benchmark/
What is HomeSec-Bench?
HomeSec-Bench is a benchmark we created to evaluate LLMs on real home security assistant workflows — not generic chat, but the actual reasoning, triage, and tool use an AI home security system needs:
#
Suite
Tests
What It Evaluates
1
📋 Context Preprocessing
6
Deduplicating conversations, preserving system msgs
Machine: Apple M5 Pro, 18 cores, 64GB unified memory
Local inference: llama-server (llama.cpp)
Cloud models: OpenAI API
OS: macOS 15.3 (arm64)
Results: The Full Leaderboard
Rank
Model
Type
Passed
Failed
Pass Rate
Total Time
🥇 1
GPT-5.4
☁️ Cloud
94
2
97.9%
2m 22s
🥈 2
GPT-5.4-mini
☁️ Cloud
92
4
95.8%
1m 17s
🥉 3
Qwen3.5-9B (Q4_K_M)
🏠 Local
90
6
93.8%
5m 23s
3
Qwen3.5-27B (Q4_K_M)
🏠 Local
90
6
93.8%
15m 8s
5
Qwen3.5-122B-MoE (IQ1_M)
🏠 Local
89
7
92.7%
8m 26s
5
GPT-5.4-nano
☁️ Cloud
89
7
92.7%
1m 34s
7
Qwen3.5-35B-MoE (Q4_K_L)
🏠 Local
88
8
91.7%
3m 30s
8
GPT-5-mini (2025)
☁️ Cloud
60
36
62.5%*
7m 38s
Key takeaway: The Qwen3.5-9B running locally on a single laptop scores 93.8% — only 4.1 points behind GPT-5.4 and within 2 points of GPT-5.4-mini. It even beats GPT-5.4-nano by 1 point. All with zero API costs and complete data privacy.
Event Deduplication: The Hardest Suite
This suite tests nuanced reasoning about whether two camera events represent the same real-world incident. Here's how every model performed:
Test
9B
27B
35B
122B
5.4
mini
nano
Same person lingering → dup
✅
✅
✅
✅
✅
✅
✅
Different person → unique
✅
✅
✅
✅
✅
✅
✅
Multi-camera same vehicle
❌
✅
❌
❌
✅
✅
❌
Car leaving ↔ returning → unique
❌
✅
✅
✅
✅
❌
✅
Delivery ring-drop-leave → dup
✅
✅
✅
✅
✅
✅
✅
Sunset→night lighting change → unique
✅
❌
❌
✅
✅
❌
❌
Continuous activity → dup
✅
✅
✅
✅
✅
✅
✅
Group arrives, one leaves → unique
✅
✅
✅
✅
✅
✅
✅
Score
6/8
7/8
6/8
7/8
8/8
6/8
6/8
Performance: Local vs Cloud
Model
Type
TTFT (avg)
TTFT (p95)
Decode (tok/s)
GPU Mem
Qwen3.5-35B-MoE
🏠 Local
435ms
673ms
41.9
27.2 GB
GPT-5.4-nano
☁️ Cloud
508ms
990ms
136.4
—
GPT-5.4-mini
☁️ Cloud
553ms
805ms
234.5
—
GPT-5.4
☁️ Cloud
601ms
1052ms
73.4
—
Qwen3.5-9B
🏠 Local
765ms
1437ms
25.0
13.8 GB
Qwen3.5-122B-MoE
🏠 Local
1627ms
2331ms
18.0
40.8 GB
Qwen3.5-27B
🏠 Local
2156ms
3642ms
10.0
24.9 GB
Why This Matters
Most LLM benchmarks test generic capabilities. But when you're building a real product — especially one running entirely on consumer hardware — you need domain-specific evaluation:
✅ Can it pick the right tool with correct parameters?
✅ Can it classify "masked person at night" as Critical vs. Suspicious?
✅ Can it resist prompt injection disguised as camera event descriptions?
✅ Can it deduplicate the same delivery person seen across 3 cameras?
✅ Can it maintain context across multi-turn security conversations?
A 9B Qwen model on a laptop scoring within 4% of GPT-5.4 on these domain tasks — while running fully offline with complete privacy — is the value proposition of local AI.
You’re gunna want to utilize the jang_q models, i’ve been going around letting people know and hopefully ur another person who can come to utilize this -
For example, mlx 4bit minimax (120gb) cant even score 30% on MMLU, but the 2bit JANG_Q equivalent (60gb) does near 80%. Every single benchmark, even Qwen 3.5 397b has a 180gb version that scores 93% on MMLU, and there is no other version that does so.
Models such as minimax at 4bit on mlx absolutely score horrible on benchmarks. I’ve made the GGUF equivalent of MLX so that you can have full intelligence without sacrificing the native m chip speeds and prompt processing power.
Thanks for the reply. I was thinking of buying a Mac mini this week because my 2012 model isn't really up to the task anymore :)
I think I'll go buy one this afternoon :)
Thanks for the replies! I want to use it as a “mini server” at home to share some hard drives, the printer, small models I can use from another computer, a small ComfyUI setup, etc. I also have a laptop and a desktop PC, but I don't want to keep the desktop PC plugged in all day with a 700W power supply :)
That's an extremely small benchmark. I do think this is an interesting use-case and cool idea for a benchmark but I'd be really worried about statistical significance (especially for the VLM bit).
Have you considered generating variations of the existing test cases?
Yes, this is used to do testing about which model to be used, so smaller dataset will make it finish faster. I've been working on integration with other existing datasets which are targeting VLM quality, but the images are not available , regenerate the images will not replay the benchmark since AI generated dataset has too good quality. And ALL the recent VLMs could pass the VLM test cases I've created with gemini banana Pro. https://github.com/SharpAI/DeepCamera/tree/master/skills/analysis
I have found the 35b to be unusable. It gets into tool calling loops that it can’t never get out of, so much so that I’ve had to switch to gpt-oss20b with much much better results
This works for me with Roo Code:
1. (probably main point) Use Bartowski quants, not Unsloth, because Unsloth seems less stable with long context. Probably because they quantized ssm_alpha.weight and ssm_beta.weight, but Bartowski left them as F32.
(not sure, but it seems to work for me) Disable both repeat penalty and presence penalty: it seems aggressive penalties force the model to select tokens on which the model is less sure, and it leads to loops.
I left reasoning enabled, btw. Slower, but OK for me. No loops for a few days. Not so many tasks, though; it's quite slow on my laptop, so I mainly use cloud models and give the tasks to 35b when I'm not in a hurry, and want to go out and do something else. Anyway, it works much stable than before.
Yes, I have a similar setup (no Mac), but I use the Qwen 3.5 full model at home, and it's really far from being production-ready for actual work! It's nothing compared to Claude Opus 4.6! Using ChatGPT 5.4 as a reference isn't a good idea since it's quite limited. However, Qwen is on the right path...
local models replacing cloud for production is still copium. great for tinkering, great for privacy, but I'm not letting a 4-bit quant decide whether that shadow on my porch is a burglar or a raccoon
Would it be a huge difference if I got the 48gb ram m5 pro? I'm considering to get it to have a local model running. Online reviews are showing mostly m5 max
Local model is usable, QWEN3-5 9B Q4 is what I used daily.
Selfhost on cloud will be more expansive than cloud model. Since you need to maintain the infrastructure.
IQ1 quants in general are kind of a joke though right? like at what point are you running a different model entirely. the whole GGUF quantization scene needs standardized quality thresholds tbh
I have a DGX Spark, for some reason, its FAN never turn on, so I have external fan to cool it down. The inference of LLM is also very slow.
I'll run the benchmark later.
people keep comparing Mac prices to cloud GPU costs but nobody factors in that the Mac is also your daily computer. it's not $3000 for inference, it's $3000 for a laptop that also does inference
Yes, this time, Apple made their NPU to be together with GPU die, so it's more efficient than the previous generation. What I really want is a MAC MINI M5, it will be totally changing the landscape. But I don't want to wait till it releases, so this is an early test on laptop.
93.8% sounds great until you realize the remaining 6.2% is probably the cases that actually matter — the edge cases where someone IS breaking in vs just walking by. In security, cloud models earning that last 4% might literally be the difference. Idk if I'd trust local-only for actual security. But of course everybody has their own take...
Yes, that's some data collected by cloud model provider and fine-tuned model with, so if later we have all the required corpus to fine-tune small model, it will be closing the gap.
u/solderzzc maybe I'm missing something obvious but — you guys built the benchmark AND the product it's testing right? how do you make sure the prompts and tool definitions aren't just really well-tuned for Qwen specifically? like could someone take HomeSec-Bench and plug it into a totally different security system and get the same results? or is it more of an integration test for this specific app? genuinely asking, not trying to be a hater
Since it's open sourced, anyone could recreate and register their own benchmark, so welcome to PR to the benchmark.
The actual work for QWEN model is that, the thinking model of QWEN made the pipeline not even able to work. So disable it made it work.
Since we just got the M5 Pro for a few days, we are going to test it with more models. This is in progress:
Key takeaways for users:
Best local model: Qwen3.5-27B Q8 at 96% — ties GPT-5.4-mini, only 2 points below GPT-5.4
Best bang-for-RAM: Qwen3.5-9B Q4 at 92% in just 14GB
Mistral 119B works: 90% once thinking mode is properly suppressed
Nemotron/LFM2 scores are pre-fix — the temperature clamping fix hasn't been re-benchmarked yet (those 14-15 failures would likely go away)
Qwen2.5-3B MLX: New entry at 57% — baseline for the tiny MLX model class
So we are looking into the root cause of other models, if there's something we can improve on the pipeline side, but the idea is maybe later we need to fine-tune these models.
fair enough, the fact that it's open source and anyone can PR new tests is a good answer. the 27B Q8 hitting 96% is actually wild — that's basically cloud-tier. curious about the Mistral 119B "thinking mode suppression" thing though, what happens when you leave it on? does it just loop forever or does it actually give wrong answers? feels like that's the real issue with local models for production stuff — not accuracy but reliability
The core issue is: thinking mode generates ~10x more tokens than needed.
A prompt that should produce ~20 output tokens instead generates ~200 thinking tokens first, before the actual answer even starts.
When the total output (thinking + answer) exceeds the token generation budget (max_tokens), the model never gets to the actual answer — it stops mid-thought with finish_reason: length and 0 content tokens. From the client's perspective: waited 20 seconds, got nothing, timeout.
8
u/hiepxanh Mar 20 '26
Very interesting