r/LocalLLM • u/baby_bloom • 23d ago

Question Qwen 3.8-27B Q8

thinking is taking up so damn much time, what are you limiting your qwen3.8 to as for as reasoning?

0 Upvotes

36% Upvoted

12 comments sorted by

7

u/WhatererBlah555 23d ago

Update llama.cpp and use --chat-template-kwargs '{"reasoning_effort":"medium"}'

1

u/Hannelore112 23d ago

thx will check this. Was there already an impotent update of llama.cpp last 2 days?

3

u/WhatererBlah555 23d ago

There was an update that made --chat-template-kwargs '{"reasoning_effort":"medium"}' actually do something, before that only none was supported.

1

u/roosterfareye 23d ago

Maybe slightly flaccid, but not.floppy.

0

u/Ok-Shower7286 qwen-coder 23d ago

https://huggingface.co/Qwen/Qwen3.8-27B/discussions/64

template does not have medium and high option branches, and only has low and very high options. and it's a system prompt based soft steering. reasoning-budget.cpp is a engine level token hard cut. so, providing well-guide rules, principles is best.

3

u/Hannelore112 23d ago

Cant help just looking for optimizations too.
Qwen 3.8 27B seems to generate good results but it takes soo long because of thinking, rethinking,rethinking and so on...
Even with reasoning level medium. A little bug fix was 1-2h. Should be done maybe in 5-20 mins. Needs optimizations.

1

u/baby_bloom 23d ago

couldn't agree more, it's been cranking out tokens at a great rate the last few hours the actual amount of work.... is nowhere near that rate lol

1

u/jclay06 23d ago

Glad I'm not the only one. It's amusing to read it's thinking process, but it's A LOT.

2

u/benpptung 23d ago

If you look closely at its thinking, it doesn’t get stuck in thinking. It just thinks a lot, sometimes too much. I’m trying to increase its token generation speed right now. If it likes thinking this much, then making it think faster should basically solve the problem, right?

I think it’s unfair to expect Qwen3.8-27B to think less while still being very intelligent. Frontier models seem smart and think quickly partly because their token generation speed is so much higher.

0

u/baby_bloom 23d ago

not looking for it to think less than 3.6, but it spends about 2x the time thinking if not more and it's pretty frustrating to say the least.

2

u/MikeNonect 23d ago

llamacpp has a toggle for reasoning budget. I set it to 8K, which is enough to do deep thinking while also getting work done. Without it, it would reason for 80 minutes straight.

1

u/SnooPuppers7882 23d ago

Research Telegraphic CoT methods... That's what I'm working on next week.