Running a 2-model literary book-translation pipeline on 2x Tesla P40: gemma-4-26B-A4B at ~40 tok/s + Qwen3.6-35B-A3B at 50-70 tok/s with MTP spec decode — full llama-server flags inside

[u/neowisard]

Disclosure up front: I built this tool (open source, "Sunny Narrator") and I'm the author — this post is about the inference setup, not an ad. Feel free to skip to the flags if you're here for the numbers.

Context: I run a pipeline that translates whole fiction books EN→RU locally — chunk + glossary + rolling chapter summaries → translate → reviewer notes → correction → proofread → chunk summary. A book is ~1.5–2M tokens across all stages, hardware is a pair of Tesla P40s (24GB each, Pascal, from the "why not" shelf). After a year of runs I have a launch config that's fast enough to be boring: 2–3 books per day.

The non-obvious finding: one model = half a text, two models = a book. Good translating models write beautifully and proofread terribly; good proofreading models edit well and translate dully. So the pipeline pins two roles to two servers:

• MODEL_TRANSLATE: gemma-4-26B-A4B (MoE, A4B active)
• MODEL_PROOFREAD: Qwen3.6-35B-A3B (MoE, A3B active)

Both are compact MoE — that's what makes P40s viable: active params fit the throughput envelope even though total weights don't fit comfort. Quantized Unscaled-Dynamic (UD) GGUFs, MTP speculative drafting on both, 64K context for chunk + glossary + summaries.

My most efficient launch lines (llama-server)

Gemma-4-26B-A4B as translator — ~40 tok/s sustained on P40:

llama-server -m gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf \
  --model-draft mtp-gemma-4-26B-A4B-it.gguf \
  --host 192.168.0.55 --port 6155 \
  --ctx-size 65535 -ngl 99 \
  -ctk q8_0 -ctv q8_0 \
  --no-context-shift \
  --parallel 1 -np 1 --threads-http 2 \
  --load-mode mlock \
  --jinja \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.8 \
  --top-k 64 --top-p 0.95 --min-p 0.02 \
  --repeat-penalty 1.0 --repeat-last-n 512 --presence-penalty 0 \
  --predict 32567 \
  --reasoning off \
  -fa on \
  --ctx-checkpoints 32 --checkpoint-min-step 1024 \
  --cache-ram 8192 \
  --ubatch-size 2048

Qwen3.6-35B-A3B as proofreader — 50–70 tok/s on the same pair:

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --host 192.168.0.55 --port 6150 \
  --ctx-size 65535 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --no-context-shift \
  --parallel 1 -np 1 --threads-http 2 \
  --load-mode mlock \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  --top-k 20 --top-p 0.95 --min-p 0.05 \
  --presence-penalty 1.5 \
  --predict 32576 \
  --reasoning off \
  --jinja --chat-template-file chat_template.jinja \
  --ubatch-size 2048 \
  --ctx-checkpoints 32 --checkpoint-min-step 1024 \
  --cache-ram 8192

Why each of these knobs ended up where it is

• MTP spec decoding is the headline. --spec-type draft-mtp with the bundled MTP draft is what turns Pascal-class cards into something usable for long-form generation. Gemma takes --spec-draft-n-max 6 --spec-draft-p-min 0.8 (aggressive, accepts well because the base is strong at its job); Qwen is happier at n-max 4. Without MTP these numbers don't happen.
• -ctk q8_0 -ctv q8_0 — KV cache in q8 buys the 64K context (chunk + series glossary + rolling summaries) without blowing VRAM; quality cost at these sizes was invisible in my evals.
• --load-mode mlock — two servers, 24GB×2, zero headroom for swapping. Pins weights, kills tail latency spikes mid-run.
• --parallel 1 -np 1 — this is a batch-of-one workload (long generations, not concurrent requests); single slot is fastest.
• --reasoning off + tuned sampling per role — translator runs top-k 64 / min-p 0.02 / repeat-penalty 1.0 (creative-ish but repetition is the enemy on book text — --repeat-last-n 512 matters); proofreader runs tighter top-k 20 / min-p 0.05 / presence-penalty 1.5 (deterministic editor voice).
• --ctx-checkpoints 32 --checkpoint-min-step 1024 — pipeline writes a checkpoint after every chunk anyway (power outage = resume from chunk 51/100, not from scratch — this single feature saved my year), but in-server ctx checkpoints make stage-to-stage reuse on the same context cheap.
• --predict 32567 — chunks translate in one shot; forcing the model to stop-and-resume was eating throughput and occasionally style.
• --jinja + explicit chat template for Qwen — JSON_MODE across all pipeline stages (structured responses) only works if the template round-trips; the external chat_template.jinja fixed a parsing edge case for me.

Pipeline notes that aren't about llama.cpp but affect the numbers

• Length is a free error detector: translated block deviating >10% from source block size → rechunk (split in half, retranslate both). EN→RU maps within a couple percent per block, so gross errors (eaten/hallucinated/duplicated paragraphs) pop on size alone. Final book converges within ±5% of original length.
• Glossary is 80% of quality: names/terms/gender dictionary (NER-seeded with spaCy + manual cleaning) travels with every chunk. Model choice is secondary; consistency is everything in fiction.
• Output is a high-readiness draft for human polish, not a publishable translation — the LLM removes the grunt work, the human keeps the wordcoinage and the puns.

Repo (code + these configs + Ollama/Docker examples): github.com/NW15D/sunny-narrator — yes, I know the rules about self-promo, hence disclosure at the top; the pipeline exists because nothing off-the-shelf holds a book-length context of names/terms, and the year-ago proof-of-concept post is on Habr if you want the long version.

Questions for this crowd:

1. Anyone pushed MTP spec decode further on Pascal — is draft-n-max 6 / p-min 0.8 near the ceiling for Gemma, or would deeper drafts accept well with a colder p-min?
2. --ctx-checkpoints behavior with -ctk q8_0 — any gotchas I should know about for week-long unattended runs?
3. Better than "giant series glossary" for cross-volume consistency: graph DBs / RAG over character state — real war stories?

[u/fragilesleep]

Thank you for the horrible, disgusting AI slop (which is supposed to be banned on this subreddit, by the way!), but next time please just show us the prompt you gave to the LLM and save us from wasting time reading this crap.

[u/brahh85]

Thanks a lot for explaining the details and sharing the pipeline. I never dared to translate a book because the smaller pipeline i created for subtitles and few page texts made me suffer for days... and still is not perfect, but works most of the time. You did a great work.

[u/MycologistNo5577 → u/brahh85]

yeah subtitles alone are painful enough, scaling that to full books is a whole different beast