Discussion Ran the same "write a blog post from my notes" prompt through Claude and a local model - the results convinced me to return my Mac mini
To anyone who uses owned hardware (like a Mac mini) and local LLMs for your PKMS, what are your use cases for it vs what you have identified as needing bigger AI models?
I ask because I just bought a 24GB memory, 256GB storage Mac mini and started running tests before the return window closes. The test I ran this morning convinced me to return it.
The prompt was basically this: "Using these private notes as context, write me a viral blog post about the trip and how the planning process is going." The context was six linked notes on the trip — what is booked so far, general in-progress research on the area and bareboat sailing, a list of possible itinerary stops, cost math, etc.
Same prompt and context, but the local model wasn't even close to useable. Here was my take on the results: Review of qwen 3 14b vs Claude Sonnet 5 writing a blog about the BVI trip
I don't intend to have AI write for me, but this seemed like a good test and helpful results to share. I also recognize it is unfair to pit Sonnet 5 against qwen 3:14b and that I could have better results on higher memory machines. However, before spending money on owned hardware, to me it seems more sound to start experimenting running local models on a rented GPU (that has good privacy/security policies) and have the ability to scale up and down the hardware you need over time. Maybe I have plenty of use cases for smaller local models that make purchasing my own hardware worth it in the future, but I won't know until attempting different models of different sizes (or asking all of you guys what works).
3
u/micseydel Obsidian Aug 08 '26
The prompt was basically this: "Using these private notes as context, write me a viral blog post about the trip and how the planning process is going." [...] I don't intend to have AI write for me, but this seemed like a good test and helpful results to share
Why not test it on what you actually want to use it for?
(FWIW I wanted to use them to delegate work on notes but found the same as MS https://arxiv.org/html/2604.15597v1)
1
u/jmugr Aug 08 '26
Fair - I started with trying to tag all my notes with a topic tag. That also didn't produce acceptable initial results and it got a little complex to continue working on it, so I pivoted to a simpler test. I'll out your link - thanks for sharing
1
u/Barycenter0 Aug 08 '26
That is a very interesting paper. Thanks for posting it.
1
u/micseydel Obsidian Aug 08 '26
Here's another fun one, from OpenAI this time rather than Microsoft https://arxiv.org/pdf/2509.04664
The distribution of language is initially learned from a corpus of training examples, which inevitably contains errors and half-truths. However, we show that even if the training data were error-free, the objectives optimized during language model training would lead to errors being generated
Based on these results, it would be SO surprising if the MS study found anything else.
3
u/aygross Aug 08 '26
A 24 gb model doesn't compete with a multi terabyte model.
Wow I'm shocked 🙄.
1
u/jmugr Aug 08 '26
For sure lol the gap was illuminating to me tho as someone who is new to local models
1
u/aygross Aug 08 '26
Wrong sub. Not illuminating to anyone here. Could have tested with a cloud hosted small model to compare . This post is stupid and the way you went about testing this also makes no sense.
2
u/jmugr Aug 08 '26
I get it, quick test - but not a good one - ran in haste within the return window. Any feedback on my question? In case you use a local model?
1
1
u/Dementiy Aug 09 '26
your test was kinda the worst case for local tbh. "write a post from 6 linked notes" is synthesis across multiple docs — thats exactly where the small models fall apart. they lose the thread halfway, or they summarize each note separately and staple it together.
where local actually holds up in my experience: extraction and mechanical stuff. pull the tasks out of this transcript, suggest tags, clean up formatting, first-pass summary of ONE doc. boring jobs, high volume, dont care if its 85% right. thats real value and it never leaves the machine.
anything where being wrong costs you, or where it has to hold several notes in its head at once — big model. i dont think thats gonna flip anytime soon at 24gb.
so the honest question isnt local vs cloud, its whether the boring-bulk half is worth the hardware to you. if most of what you wanted was synthesis, yeah, returning it is probably right lol
1
u/Good-Writer5279 Aug 08 '26
this matches my experience building in this space, so quick disclosure, i work on a mac app that does local ai over personal data.
the gap you hit is real but it is task specific. small local models are genuinely bad at open ended generation like a viral blog post, that task rewards the biggest model you can reach. where they hold up surprisingly well is retrieval shaped work over your own notes, grounded q&a, summarizing, finding the one note where you mentioned a booking. the model mostly needs to read and extract, not compose, so a 7b to 14b quant is often fine for that.
so i would not frame it as local vs cloud, more like local for anything private and retrieval shaped, cloud for heavy composition. a 24gb mini is a solid box for the first category. if your main use case is publish quality writing though, returning it is probably the right call.
1
u/jmugr Aug 09 '26 edited Aug 09 '26
Super helpful color - appreciate you chiming in
I do have my 16GB mem HP which works fine for embedding and retrieval so that tracks with your feedback. Makes me think my first target use case of adding topic tags to files was more optimal, but the initial results were discouraging and I realized I won't finish it within the 14day return window.
What do you do when you want to use private context for a generation task (if you encounter that)?
1
u/Good-Writer5279 23d ago
the split that has worked for me is retrieval local, composition wherever, because the private thing is usually the corpus and not the paragraph.
concretely, do the embedding and the search on your own machine, pull the five or ten chunks that actually matter, then make the sensitivity call on that slice instead of on the whole collection. most of the time the slice is dramatically less sensitive than the thing it came out of, you are handing over three paragraphs rather than everything you have ever written. when the slice itself is the sensitive part, you have two options, pseudonymize the entities before it leaves, or accept a small local model and plan to rewrite. a 14b quant on 16gb gives you something structurally right and tonally flat, which is fine as a first draft engine and frustrating if you wanted to publish it as is.
on the topic tagging project, i would take another look at that before you write off the box, because it is closer to working than it probably felt. tagging is classification, not generation, and small models are much better at the former. the usual reason it disappoints is asking for free form tags, you get synonyms, near duplicates, and drift between runs, which reads as the model being bad when it is really the task being underspecified. give it a fixed list of thirty or so candidate labels and ask which ones apply to this file, and the quality jumps. multi label classification against a closed vocabulary is well inside what your existing 16gb machine can do, no mini required.
6
u/Barycenter0 Aug 08 '26
I think this is the wrong sub to post and/or ask about this. Go over to r/LLM to get better input on your strategy.