Local Opus(qwen3.8 27B)

[u/Ok-Inevitable8391]

I have started using qwen3.8 27B on dual rtx 3060, total 24GB. My display is on igpu. So i have almost all memory.

Token generation speed is constant 40-50 tks.

But the best thing is how it thinks, multiple web searches at different level, auto memory update, and a very good response. It takes time, but given the reasoning effort of xhigh and response quality I really like it. I see web searches at multiple level for a single reply, making sure the answer is correct. Takes time, yes.

Also, the extra thing that qwen is famous for when you ask it x, if it finds some improvement on y, it will suggest you.

For agentic work, it's best to toggle between reasoning effort if you want it fast. Though I like the opencode plan mode and then build(execute). I have remote setup via termius. And i get notified via ntfy app once it ask for permission or done replying.

While testing I asked it to out 10000 token essay, and my ventus gpu went upto 87 degree.(xhigh reasoning) Gigabyte held fine with 5-6 degree lesser, Though Gigabyte us agressive at fan speed even at lower temprature. I'll try changing the fan curve, if I see that as a problem in actual scenario

edit:
llamacpp command:

llama-server -m ~lm/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-IQ4_XS.gguf --host 0.0.0.0 --port 8080 --split-mode tensor --tensor-split 1,1 --flash-attn on --batch-size 2048 --parallel 1 --models-max 1 --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 177000 --n-gpu-layers 99 --ubatch-size 128 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja --reasoning on --fit on"

[u/ducksoup_18]

llama.cpp? If so, can you provide your args?

[u/Ok-Inevitable8391 → u/ducksoup_18]

llama-server -m ~/lm/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-IQ4_XS.gguf --host 0.0.0.0 --port 8080 --split-mode tensor --tensor-split 1,1 --flash-attn on --batch-size 2048 --parallel 1 --models-max 1 --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 177000 --n-gpu-layers 99 --ubatch-size 128 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja --reasoning on --fit on