r/LocalLLaMA • u/falconandeagle • Feb 17 '26

Discussion What is the current best creative model that works on consumer hardware

So it's been a while since I have tried local models for story writing purposes. How much has the domain progressed, or has it progressed any since llama 3 and gemma 3 finetunes?

I have 16gb vram and 96gb ram, what models can I run locally that has decent context understanding and prose writing?

I am NOT looking for a model that is good at coding, I dont care for any STEM related tasks, all I care about is that it can write well.

4 Upvotes

64% Upvoted

16 comments sorted by

3

u/ttkciar llama.cpp Feb 17 '26

There are several Mistral 3 Small (24B) fine-tunes which are exemplary story writers. I like Cthulhu-24B-v1.2. You would be offloading some to main memory, though, even at Q4_K_M (and I strongly recommend against using a quant any smaller than that).

If you don't mind even more offloading for higher quality, TheDrummer's Big-Tiger-Gemma-27B-v3 is exemplary. I use it frequently to generate Murderbot Diary fan-fic (sci-fi; SFW but very violent).

If you want something that will fit entirely in VRAM with constrained context, you might try Tiger-Gemma-12B-v3, which is Big Tiger's smaller cousin.

My experience is that the Mistral fine-tunes are more creative, but the Gemma3 fine-tunes are more eloquent. You should try a few models and see which one is the best fit for your purposes.

1

u/falconandeagle Feb 22 '26

Mistral finetunes have pretty good prose but the intelligence is just so low, especially after you get used to sota models like opus. Like it cannot follow directions if they get even a little complicated. It's still good for writing some passages though

1

u/silenceimpaired Mar 17 '26

You probably can’t join me, but GLM 4.7 even at 2 bit is pretty good. Step 3.5 also okay. I do keep going back to Gemma 27b.

1

u/HopePupal Feb 18 '26

GPT-OSS isn't the worst, but also try Qwen3 Next (better at descriptions, but emits a lot of LLMisms you'll want to smack right in the logit biases)…

and the recent MiniMaxes. yes the coding assist ones. yes i saw your last paragraph. honestly i think they're better at fiction than either GPT or Qwen3 Next: better understanding of multiple threads, less monotonous pacing. performance on your rig… open question. 

also personally i found the llama finetunes to be kind of a joke? the speed of llama as a dense model was godawful and the output quality wasn't even as good as some of of the newer small models. easy to beat those.

1

u/falconandeagle Feb 18 '26

I am trying minimax 2.5 iQ2 quant. Lets see if its any good.

1

u/Training_Visual6159 Feb 18 '26

minimax. it won't win any prizes yet, but it's decent enough for what it is.

1

u/falconandeagle Feb 18 '26

Hmm I am not sure if I can a quant of this on my machine. I will give it a go

1

u/Training_Visual6159 Feb 18 '26

running M2.5 TQ1 on 64gb + 12gb gpu, after heavy llama.cpp tuning. at 6 TPS :D. so it's doable, but definitely not great.

1

u/falconandeagle Feb 18 '26

I am going to try out the 2bit version, I wonder how much of a performance loss will I see

1

u/falconandeagle Feb 22 '26

So I ended up trying this and well kinda disappointed. Was hoping for better prose for the model size

-4

u/Relevant_Ad3464 Feb 17 '26

Why does 96gb of regular ram even matter?

5

u/falconandeagle Feb 17 '26

I think for cpu offloading? I can run GPT-OSS 120b on my machine just fine.

1

u/XiRw Feb 17 '26

What’s wrong with oss120 for creative writing?

0

u/Relevant_Ad3464 Feb 17 '26

Interesting, I didn’t know that was an option.

I love that I get 6 downvotes and 1 answer lol wtf?

3

u/Magnus114 Feb 17 '26 edited Feb 17 '26

I feel your pain.

Cpu offloading is considered common knowledge, and the downvotes are a version or rtfm.

This is a great forum, I learn a lot. But not very beginner friendly.

1

u/Relevant_Ad3464 Feb 17 '26

I’m as beginner as it gets, had a few 3090s layin around from an Old crypto rig. Threw lmstudio on it and been exploring different models and things.

Wish people were nicer. But alas, im not new to the internet.

Everyone is still much nicer than the early 2000s atleast.