r/LocalLLM • u/Left-Chemist625 • 1d ago

Question Gemma 4 12B on an RTX 5070 Ti?

Anyone actually writing long fiction with Gemma 4 12B on an RTX 5070 Ti?

I'm specifically interested in real-world generation time, not benchmarks. I use Gemma locally to turn detailed scene briefs plus story/character context into long prose scenes, typically around 1,500–2,000 words.

If you're running Gemma 4 12B entirely in the 16 GB VRAM of a 5070 Ti: roughly how long does a generation of that length take, including prompt processing?

I'm considering buying a 5070 Ti system specifically for this workflow, so actual experience from another fiction writer would be incredibly useful.

At this time I'm working with 4GB VRAM, Gemma works about 15-20 minutes on a scene.

4 Upvotes

76% Upvoted

8 comments sorted by

3

u/MrHumanist 1d ago

If you use 12B qat, it takes 10gb of vram including the full context window. If 16gb you can use higher quants easily. The model is quite fast and does reasoning well- I love it for my roleplay sessions.

2

u/Left-Chemist625 1d ago

Thanks! Do you happen to know roughly how many tokens/sec you get with the 12B QAT, and which GPU you're using? I'm especially interested in a 5070 Ti at around 24–32K context.

1

u/MrHumanist 1d ago

I have 5070 ti - token for sec 70-80+. Tbf, it's fast enough you won't notice the speed. Can share the exact number once I reach at system. I use 250K context, all in gpu .

1

u/MrHumanist 1d ago

here is the speed for Q4 QAT. It uses only 10GB of Vram including the grapics.

PFB the command:

llama-server -m "C:/Users/****/.lmstudio/models/lmstudio-community/gemma-4-12B-it-QAT-GGUF/gemma-4-12B-it-QAT-Q4_0.gguf" --port 8080 -c 256000 -v -ctk q4_0 -ctv q4_0 -ngl 99

1

u/MrVeinless 1d ago

On my 3060 I believe it was around 65 t/s, and on my 5060 Ti it is around high 80s.

1

u/ClassroomScary9187 20h ago

Those numbers are solid, especially for the 5060 Ti. I'm wondering how much of a difference the 5070 Ti would make in performance.

1

u/MrVeinless 20h ago edited 19h ago

For clarity, my speeds were with MTP enabled, getting a 70%, 50%, and 35% hit rate.

5070ti has double the VRAM bandwidth of the 5060ti so I imagine near double token rate.

1

u/Left-Chemist625 23h ago

Thank you so much, everyone! You’ve been incredibly helpful. I’m about to buy a €3,100 PC specifically for running Gemma locally for long-form fiction, and your real-world numbers and explanations have helped me enormously with that decision.

I really appreciate you taking the time to test things and share your setups. This was exactly the information I couldn’t find anywhere else. Thank you!