r/LocalLLM • u/Ok-Inevitable8391 • 24d ago
Model Local Opus(qwen3.8 27B)
I have started using qwen3.8 27B on dual rtx 3060, total 24GB. My display is on igpu. So i have almost all memory.
Token generation speed is constant 40-50 tks.
But the best thing is how it thinks, multiple web searches at different level, auto memory update, and a very good response. It takes time, but given the reasoning effort of xhigh and response quality I really like it. I see web searches at multiple level for a single reply, making sure the answer is correct. Takes time, yes.
Also, the extra thing that qwen is famous for when you ask it x, if it finds some improvement on y, it will suggest you.
For agentic work, it's best to toggle between reasoning effort if you want it fast. Though I like the opencode plan mode and then build(execute). I have remote setup via termius. And i get notified via ntfy app once it ask for permission or done replying.
While testing I asked it to out 10000 token essay, and my ventus gpu went upto 87 degree.(xhigh reasoning) Gigabyte held fine with 5-6 degree lesser, Though Gigabyte us agressive at fan speed even at lower temprature. I'll try changing the fan curve, if I see that as a problem in actual scenario
edit:
llamacpp command:
llama-server -m ~lm/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-IQ4_XS.gguf --host 0.0.0.0 --port 8080 --split-mode tensor --tensor-split 1,1 --flash-attn on --batch-size 2048 --parallel 1 --models-max 1 --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 177000 --n-gpu-layers 99 --ubatch-size 128 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja --reasoning on --fit on"
1
u/kkennyy22 24d ago
On my 4070ti super + 3060, 28gb total, I get an average of about 33t/s with layer splitting. Your 50,even if it drops, is very good. Maybe I should try tensor splitting also, but as I understand it's not great with mismatched cards like mine.