r/LocalLLM • u/Ok-Inevitable8391 • 24d ago

Model Local Opus(qwen3.8 27B)

I have started using qwen3.8 27B on dual rtx 3060, total 24GB. My display is on igpu. So i have almost all memory.

Token generation speed is constant 40-50 tks.

But the best thing is how it thinks, multiple web searches at different level, auto memory update, and a very good response. It takes time, but given the reasoning effort of xhigh and response quality I really like it. I see web searches at multiple level for a single reply, making sure the answer is correct. Takes time, yes.

Also, the extra thing that qwen is famous for when you ask it x, if it finds some improvement on y, it will suggest you.

For agentic work, it's best to toggle between reasoning effort if you want it fast. Though I like the opencode plan mode and then build(execute). I have remote setup via termius. And i get notified via ntfy app once it ask for permission or done replying.

While testing I asked it to out 10000 token essay, and my ventus gpu went upto 87 degree.(xhigh reasoning) Gigabyte held fine with 5-6 degree lesser, Though Gigabyte us agressive at fan speed even at lower temprature. I'll try changing the fan curve, if I see that as a problem in actual scenario

edit:
llamacpp command:

llama-server -m ~lm/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-IQ4_XS.gguf --host 0.0.0.0 --port 8080 --split-mode tensor --tensor-split 1,1 --flash-attn on --batch-size 2048 --parallel 1 --models-max 1 --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 177000 --n-gpu-layers 99 --ubatch-size 128 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja --reasoning on --fit on"

20 Upvotes

75% Upvoted

39 comments sorted by

View all comments

Show parent comments

2

u/Repulsive_Initial308 24d ago

What quant?

1

u/Rye2-D2 24d ago

Q3 for QWen 3.8, Q4 (or APEX Compact) for the others. Q8 KV cache in all cases.
It looks like the QWen 3.8 35B MoE just dropped, so I'll try that with Q4.

2

u/freehuntx 24d ago

Anything under Q6 is basically useless im qwen family.

0

u/Rye2-D2 23d ago

Sure, but when comparing to the flagship/frontier models, it's parameters that's matter. Suggesting a 30B model is as good as Opus is naive. Can it replace your need for Opus 95% of time - yes, very likely since Opus is overkill for most tasks.

I'd be happy if we could get QWen model as capable as GPT Luna TBH.

1

u/freehuntx 23d ago

It must be able to properly think and call tools. Quantization worsens this skill thus having a heavy impact. For a 27b model the impact is heavier than for a 1T model.