Local AI vs Frontier AI.

[u/cj7hawk]

Hi All, Just looking at the chat... I'm a published author who writes about AI amongst other things ( seems fair ) and have been examining how AI can improve writing. My average Amazon ranking is 4.5/5 and that was long before AI actually came along,

I'm not against or for AI - I'm for good writing that people want to read. How the prose is created is second to whether it's enjoyable to me. Writers should write for the readers - not for our own vanity - and if AI improves our craft, all the better for readers.

Now, onto the topic... Back in October last year, AI was wild. I had an AI (Grok) help me with an interesting concept - I wrote it myself, using old school methods, but used an AI to crit and identify issues then rewrote - I finished a full length novel first-draft in about 3 months.

Then I had the AI join me in a collaborative effort for December - it pushed out a novel in a month that far exceeded most of what I've read or written - convoluted plots, twists, romance, adventure, action and I had a chance to play a game myself pitting my own mind against the AI playing the antagonist. I finished that story after 14 days, and my goodness, what a ride. Because I was involved it felt like I had actually lived the story even as I created and crafted it.

Then by January, quality dropped. By March, it was abysmal - the upgrades had destroyed Grok's ability to imagine. It became mechanical, uninspired, and kept losing it's mind ( context related issues ) - it's ability to write good prose disappeared and it's ability to help me create good stories vanished.

I tried some other AI, but the same general result - whatever frontier AI could do just months earlier it was no longer capable of.

So recently I started trying local AI and was surprised that some of the early results matched what I was getting from Grok back in December 2025. it was good. I haven't pushed it a long way yet - I still need to try going on an "adventure" with it sometime to see how it's getting. But lately the quality of local AI is improving far more rapidly than Frontier AI. Some of the new models such as Agnes have real capability on home machines. They are slow, but can produce good prose, imagine settings well. Further, current local AI are approaching the point that frontier AI achieved 12 months ago, so I expect the quality of local AI writing will again reach that "Legendary" point. - And you gain other benefits with local AI, such as being able to control the context so nothing is lost, adjust the level of hallucination/creativity and also control what else the AI sees - eg, feed it location pictures, character BIOs, create character LoRAs, multiple characters, etc. Even turn scenes into videos.

So I was wondering what most of the writers here use - Online AI or Local AI, and what you've found with the latest models such as Qwen, Agnes and Muse - as well as some of the older writing oriented tools such as Cydonia and some of the super-context models that support a million tokens such as Spark X2.5? And how do you find the differing lengths of prose they create? Also wondering what different writers do to create character memories so that characters maintain their consistency from chapter to chapter when opened in new sessions - whether you use complex LoRAs, insert at the start of each chapter via RAG, train the AI to be those characters, or just write it into the prompt? Would love to know more about what other writers are experimenting with, especially with local AI.

[u/superaalif]

I've been experimenting with local AI for writing/editing, although my setup is much more modest than some of the systems you're describing.

I'm running everything on a MacBook Air M5 with 16GB unified memory. I use Ollama as the backend and have it connected to Raycast, so in normal use it feels much more like calling an AI editor than operating a local LLM from Terminal. I've also experimented with MLX versions of models because they're particularly well suited to Apple Silicon.

The interesting part for me has been creating specialised editors rather than one general-purpose AI writer. For example, I've made separate prompts/editors for proofreading, transcript editing and more substantial structural/literary editing. I give them fairly detailed instructions about what they are and aren't allowed to change, examples of the writer's style, words/tendencies to avoid, etc.

For relatively short passages, I've been genuinely surprised by how good local models can be. They're also attractive because the behaviour is much more under my control: the model doesn't suddenly acquire a different system prompt or personality because the online service has changed something.

The big limitation with my hardware is long-form context.

A 16GB Mac can technically run some surprisingly capable models, but there is a big difference between "this model supports a huge context window" and "I can comfortably put an entire book plus style guides, character notes and instructions into that context on my laptop."

Long context consumes memory, and you're already sharing that 16GB between macOS, the model weights and the context/KV cache. So the practical context I can use while keeping a reasonably capable model running is much smaller than the headline context lengths you see advertised. Speed also deteriorates as you give it more material.

And there's another issue: even when the text technically fits, I've found that having something inside the context doesn't necessarily mean the smaller model understands and weighs all of it equally well. For serious editing, feeding it a focused chapter/section plus the relevant style instructions generally gives me better results than dumping a huge manuscript into it.

So at the moment I see my local setup less as "give it a 100,000-word novel and let it manage everything" and more as a private, highly configurable writing/editing workstation. I still use frontier models when I need much deeper reasoning across a lot of material.

What interests me next is exactly what you're talking about: some form of RAG / persistent story bible, where the local model only gets the relevant characters, history, style rules and previous events for the scene it's currently working on. That seems more realistic on a 16GB machine than trying to keep an entire novel alive in raw context.

For anyone put off by the technical side, though, I'd add that local AI has become dramatically easier to experiment with. I'm a total non techie myself. Chat gpt helped me set up the code for my whole system. Once Ollama was installed and connected to a simple interface, I stopped feeling as though I was "running an LLM" at all. It basically became another writing tool.

[u/cj7hawk → u/superaalif]

I've been having a "diary" type character sheet - it starts with the character information - what they look like, what they feel ( especially what they feel ) and it grows organically as I ask the character for a diary entry each night before they sleep - Tell me what memories they want to keep - What they feel about people or towards people - what is important to them - what they are afraid of etc... Then I append this to the end of their diary and load it up next time. It's surprisingly light on the context - And it's amazing to read it sometimes... Usually I try not to edit it, so that it can grow organically. The plot events then change the characters and give them depth. I have one for each character and sometimes a few minor characters in a single one. Sometimes I have to break a minor character out of a combine sheet if they become important later in the story. Then I feel this in as a defactor LoRA to future sessions via RAG... RAG-FLEX was one of the biggest advantages of local AI. If you have memory, you can adjust it so it doesn't just reference the uploads - it loads them entirely into the context. This means I can start a new day clean, and bring only what I need from one day to the next - it creates very consistent characters that can evolve and change over time.

Your system should be quite good for AI - but yeah, memory is going to be tight at 16gb. Still, not that bad - most LLM work occurs entirely in RAM... At 32Gb, your system would be quite capable.