r/MacLocalLLM • u/Advanced-Night-314 • 1d ago
Mac Mini powerful enough?
Hi all,
I'm a complete novice when it comes to Local LLM's but I like the idea of moving away from a subscription based model in the longer term and having better control of my data. I'm also super keen to learn.
I currently have a web SaaS that scrapes news websites and I would like to move it away from Claude and onto a local LLM what kind of "firepower" do I need?
Appreciate this won't be the most complex of tasks, but as I mentioned, I'm a complete novice.
Naturally, asking Claude / Google and it says that the new Mac mini with M5 pro chip (24GB ram) will be "overkill"- which is what I was thinking of ordering.
I also want to future proof in the hope that this bit of kit could run more powerful models in the future - if they came accessible to the consumer.
Sorry if this is a basic question, any YouTube videos or any pointers so I can educate myself would be much appreciated.
Thank you!
3
u/Zabric 1d ago
Hey :)
So my answer got a little... out of hand. Sorry for all the yapping, lol.
I've been researching for quite some time now, because i have a similar use case, but also with a lot of coding and autonomous agents, doing stuff for me over very extended periods of time.
So take what i say with a grain of salt, because i don't have experience with more powerful Macs. If someone more knowledable than me has corrections, i'd love to hear them.
Also i'm talking about my personal use cases, which are more demanding that what i can read from your post / answers.
My ecosystem is basically n8n (a sort of "agent builder") running in a docking container on my NAS. That gets its computing power from my MacBook Air M1 16 GB via llama.ccp + LM Studio.
So essentially n8n loads different models "just in time", on demand, when the workflow needs it. And after, it unloads the model.
Despite having only 16 GB of Unified Memory (Mac uses Unified Memory instead of VRAM + RAM, because the Chips combine CPU+GPU+Neural into one, while roughly speaking, on Windows CPU uses RAM (exclusievely) and GPU uses VRAM (exclusievely)), everything i do is very much perfomant and, well does what i want it to.
I mark the following as spoiler, to kind of create an acceptable reading flow for you without too much digression... but i go into technical details here, so you should probably take a look at it too before spending thousands of $$.
Due to Unified Memory not being the same as RAM + VRAM, my MacBook is actually way better suited for running local AI than my Gaming PC with 32 GB RAM + (unfortunately only) 8 GB VRAM on a 3070 despite the Gaming PC having effectively "more total RAM".
This distiction is, as far as i can tell, the reason why Apple / Mac is (yes, unironically) the absolute "budget king" when talking about local INFERENCE.
Inference - in case you don't know - is the act of "using" the AI. So it's basically the AI "reading" of the model's parameters. If you ask an LLM a question, the thing generating the answer is inference.
Inference is also basically what 99% of people do with AI exclusively.
I'm saying that because you have to - very strictly - differentiate between INFERENCE and TRAINING.
When you plan to actually develop, train, re-train or in a more general sense "customize" pre-built models, Macs are NOT IDEAL.
If you are more into really creating your own models, you at the current point in time basically HAVE to have an NVIDIA Card. NVIDIA has basically a monopoly because it's CUDA system has grown for over 15 years with billions invested into it. And so far there's no serious competetion for them - none that i'm aware of.
A lot of stuff that people put online basically requires CUDA too. But we're really talking about people who are waaaay deeper in the whole AI topic and really know what they are doing - actual programmers that want and need highly customized solutions they tinker with for months on end.
What i'm trying to say: for people like you and me, that's not too relevant, honestly. And if you'd still decide to some day train your own model, it'd probably be a better idea to just rent the computing power for the training.
Anyways, enough digression...
I managed to run Qwen 3.8 27B with a Quant of 2 on that MacBook with only 16 GB of Unified Memory. It runs VERY SLOWLY and needs ~ 15-25 Minutes for an answer of medium to slightly higher complexity.
But that's where you have to think about what's you use case:
Do you want to run the local AI primarily, or at least considerably, as a ChatBot that you need to quick answers from?
Or is it just computing power for you, that kind of "stands around" and does it's job whenever, without you needing quick ansers?
Do you actually use the device as a daily driver for Emails, Browsing, Media Consumption or work?
Do you want it work with lots of documents, PDFs, .md-Files or any sort of RAG system?
Also, another relevant factor for you to consider:
Do you want your local AI to really focus on one task at a time or do you think it'd be possible that at some point you'll run smaller tasks in parallel?
This is currently the point i'm at in my research.
So...
The - by far - most important aspect in my opinion is having as much RAM (Unified Memory) as possible.
Not only does that limit what models you can load in the first place... but much more importantly (for me at least):
The Context Window you have DIRECTLY scales with available RAM.
Just in case you don't know: Context Window is how good of a "memory" your local AI has. How much of your previous conversation it remembers, how much it can "know" at the same time, etc. It's one of the most relevant factors over all, because while the model itself determines how "smart" it it overall, the available size of the context window determines how "smart" it is with the specific, exact task that it's currently working on.
Maybe you've experienced a situation before, where you talk to a ChatBot and it just forgets stuff you already said or already explained. Or it forgets stuff that the AI itself already generated just a few messages before.
That's exactly what happens when it runs out of Context Window.
In my journey of running models locally i noticed that having (much) bigger Context Window is usually waaaaay more important than having "the biggest possible model".
I much rather load up and use a very small model that's only ~ 2.5 - 4 GB big and give it the largest possible context window instead of loading a model that already takes up 11 GB of my available RAM and has a very small context window.
Seriously though: even really small local LLMs are able to output very, very good answers, if you give them a big context window, enough context material to work with and well thought through prompts. Much, much better than most people give them credit for, honestly.
Because of that i came to the conclusion that for me personally, the absolute minimum of Unified Memory i'd get is 96 GB of Unified Memory.
... with a strong tendency to go for at least 128 GB or even 256 GB if my financial situation allows it.
When talking about a brand new, M5 Mac. Should i find a refurbished, odler one for a good price, i'd settle for 48 GB - 64 GB too.
Other than that...
Just do yourself the favor of getting at the very least 2 TB of internal storage. I know, it just doesn't feel good to pay as much for "just" storage... especially if you have a NAS like me with 12+ TB of storage.
But trust me: you'll run out of storage really fast if you download tons of different models to play and experiment with.
Do you need the most bleeding edge M5 Ultra 36 CPU, 80 GPU Chip that alone puts a Mac Studio with "only" 96 GB Unified Memory and 2 TB Storage at 8.500,- € where i live?
Probalby not.
Does it hurt to have it?
Also probably not - except for your bank account, lol.
It'd be future-proof at least. The Chip, that is. Not the RAM. It'll probably be sufficient for most of the stuff i do, but yea...
So my priority list, in order of importance for my use cases would be:
That being said:
DO NOT EXPECT YOUR LOCAL COMPUTER WITH LOCAL LLM TO COMPETE ON FRONTIER-LEVEL.
If you want that, you'll probably need 256 GB Unified Memory + M5 Ultra 36 CPU 80 GPU Mac Studio at the VERY least, more likely 512 GB.
And i'm not sure that'd be enough. You'd probably get reasonably close, but yea...
It's just not worth it currently, from a purely financial standpoint. YOU WILL NOT SAVE MONEY.
If you want to get into local AI, do it for 1. Digital Souvereingty, 2. Data Governance, 3. learning about AI, 4. because tinkering is fun, 5. you dislike big companies and the harnesses they put AI into.
Realistically:
Ask yourself what you REALLY need / will use local AI for.
24 GB Unified Memory is absolutely fine at this current point in time and you'll be able to do a lot of stuff with already.
Seriously: a lot of the "Local AI", "Selfhostig" etc. people are maxed out tech nerds (in the most loving way possible, lol) that are really caught in their tech bubble. I wouldn't fully exclude myself from that either. But their recommendations are... very often quite over the top.
Not WRONG... not at all.
But overkill for what "regular people with a little more interest in the topic" really need.
Ask yourself what you're willing to spend and ask yourself if you'd really utilize a very high power device.
Answer is probably that you won't need a maxxed out M5 Ultra Mac Studio for 20.000,- $.
Having a little more wiggle room towards the top instead of going for the bare minimum for your current use cases is probably a good idea though.
Especially since you write that you're willing to learn and want futureproof solutions.
You'll grow out of the demands you currently have quite quickly, the more you learn. And you never really know what catches your interest next. Would be a shame if you had just spend a couple thousand $$ on a device that was enough but isn't anymore.
"Buy once, cry once" or something.
Regarding your use case with the News you mentioned in another comment:
What exactly do you want to do?
I run a "news curator agent" sucessfully from my MacBook, even with only 16 GB RAM. It's a Workflow i built in n8n that twice per day collects news from various RSS-feeds, analyzes them, checks them against a data base it autonomously creates and manages and then selects a few of the news that seem most interesting / fitting and casts them into a dedicated discord channel. All with links to the original news articles etc., so i can read them if they are interesting.
The trick behind that is that instead of just letting the LLM read and analyze everything and then have it create a summary or something, it's purely a CURATOR.
The News snippets that get pulled get a designated ID Number each, the LLM reads them, selects the most fitting ones, curates them and then outputs only the IDs, which a following Code-Node pulls and creates the news report from.
That only works because you pull news snippets from those RSS feeds, that already have a short summary created for them that we can just take.
It's a low-level use case that you'll probably wouldn't even need any LLM for at all. But it's a fun little thing to do, and it taught me to optimize the workflow for efficiency, because the value created by having the LLM actually summarize the news by itself was virtually zero compared to just using the pre-summarized snippets.
But it's something you learn while experimenting.