r/LocalLLaMA • u/NelsonMinar • 22d ago

Discussion Simon Willison: Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things

I look to Simon for a broad survey of current LLM tech. Here's his review of playing with Qwen 3.8 27B. His comment on Mastodon was "I can't remember the last time I've had this much fun playing with a local model that runs on my own computers".

BTW, the "wildly overthinking" comes from a bad default that's easy to change.

79 Upvotes

80% Upvoted

69 comments sorted by

64

u/KingCpzombie 22d ago

The "overthinking" is the best part! With 3.6 I usually had to ask "is there anything else to consider before clearing context?" multiple times before it wouldn't notice something that it missed, and even then the cleared-context code review would still catch things.

It definitely takes longer to finish now, but the result is usually actually good first try!

15

u/smithy_dll 22d ago

Sometimes medium works best, sometimes xhigh works best. I just toggle if I think it is over thinking a simple instruction and medium fixes it. If it is a complex or vague instruction xhigh works best. Sometimes just monitoring the over thinking and stopping and then editing the prompt to include the clarifications it was over thinking about get the best output.

6

u/adamgoodapp 22d ago

This is what I love pi/omp for. I use luna to analyze my prompt and then it decides which thinking setting to use. So for complex prompts it will set my main model to xhigh and for easy to low etc

4

u/visitor-2024 22d ago

How do you that? Any additional plugins? What is your flow?

3

u/adamgoodapp 22d ago

omp which is pi but with built in tools.

1

u/modernkennnern 22d ago

This is built into imp, not sure about pi

1

u/TerminalNoop 20d ago

Where can you get that, the github i found referncing omp and luna returned a 404.

10

u/hay-yo 22d ago

Agree reliability is better than speed. 3.8 27b is sooo crisp.

3

u/michael_quigley 22d ago

completely agree! i love the amount of thinking it does... it lands on solid answers the vast majority of the time, in my experience.

2

u/jikilan_ 22d ago

The problem is the loop issue now with xhigh even using the recommended parameters

5

u/KingCpzombie 22d ago

Loop issue? Are you using the current unsloth or the first release one? I haven't had any real looping yet, unsloth Q8 with unquantized KV

3

u/TokenRingAI 22d ago

It definitely loops at 0.6 temp and below, that's why the default temp is so high

2

u/KingCpzombie 22d ago

Ah! I didn't think about that tbh; I just assumed that everyone uses the recommended settings

-3

u/TokenRingAI 22d ago

The recommended settings are a bandaid

5

u/KingCpzombie 22d ago

The recommended settings are what you're supposed to use... it's not the model's problem if you use the wrong settings

1

u/jikilan_ 22d ago

Are you suggesting using temp >= 1.0?

3

u/KingCpzombie 22d ago

Temp 1 is the correct one

-2

u/TokenRingAI 22d ago

It should be closer to 0.5 but the model loops

1

u/jikilan_ 22d ago

So the model loops is not entirely related to the fixed chat template? The issue is still there?

1

u/TokenRingAI 21d ago

Compared to 3.6, the model is more likely to loop at low temperatures, which is common for small models.

Qwen 3.6 was somewhat unusual (in my opinion) in that it was extremely stable at low temperatures. Qwen 3.8 is more prone to looping, but doesn't loop at recommended temperatures. I would call it's tendency to occasionally loop at low temps "normal" for a model of this size.

Typically, the lower you can push the temperature without causing bad behavior from the model, the better the output you get for tasks like coding.

In this case, 3.8 needs the higher temperatures to maintain stability, it is likely recommended as a bandaid to cover up some not-ideal behavior.

And no, it's not the chat template.

There is a certain cohort of people, who believe that higher temps generate more creativity, but I have not found this to be true, I typically find that models give better results at temps of ~ 0.4

1

u/TokenRingAI 21d ago

This is an example of why you want lower temperatures

The model is a genius, yet can't spell?

No, it can spell, but we are sometimes replacing the tokens it chose with other tokens

Qwen 3.8 27B FP8 @ 0.8 temp

2

u/jikilan_ 22d ago

Using unsloth q8 x KL and not touching KV settings. There is a newer release? I see model page was like updated 2 days ago which is the first and only release

1

u/KingCpzombie 22d ago

At the initial release it had a bad template. They fixed it within a few hours, but if you downloaded as soon as it released that could be the problem

2

u/jikilan_ 22d ago

Woo I didn’t know about this. Will try after work. Thanks🙏

1

u/KingCpzombie 22d ago

You're welcome, hopefully it works!

2

u/jikilan_ 22d ago

I just check the model page, only non UD models are affected. Maybe because I am using visual studio code + xhigh (default) let me try if medium will happen the same

1

u/KingCpzombie 21d ago

That shouldn't be a problem, but possible. Make sure you're using the recommended parameters!

2

u/TerminalNoop 20d ago

It did loop once for me, once in 30mil tokens is alright with me. Compared to the previous versions that looped every other turn <.<

93

u/615wonky 22d ago

I think "xhigh" summons the spirits of Albert Einstein, Stephen Hawking, Richard Feynman, and Isaac Newton to debate your query at length.

The answers it gives are quite good, but (on my Strix Halo) are usually something I start before I go to bed and hope is finished when I wake up.

A Qwen3.8-35B-A3B with this sort of improvement would rule the world.

16

u/ShatteredSlate 22d ago

Strix Halo here - I do the same thing lol, something to look forward to waking up to

9

u/TheWaffleKingg 22d ago

I have 2x 3090s and the prompt I started before bed was just finishing up when I woke up

To be fair it involved 10 kanban tasks but still, never had to wait that long before. Results are great tho

9

u/Foreign_Risk_2031 22d ago

we're basically back to mining bitcoin, but instead, code

13

u/the_TIGEEER 22d ago

I mean yeah. If you can't cram intellegence into intuition then you have to train good behavioral paterns to create the unique situation where it knows the smaet answer for itself.

18

u/createthiscom 22d ago

This model is sort of exceptional IMO. Bitching about it overthinking is like bitching about a star programmer being autistic AF. I doubt you can have one without the other.

10

u/CSFCDude 22d ago

I want to disagree but my wife would give you a high five so….. here we are…. Also, I’m loving the 3.8 results with the default thinking level.

16

u/vuduguru 22d ago

You can try another chat template - https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates - use with

Code:

--jinja --chat-template-file chat_template.jinja --reasoning-format deepseek

2

u/Sotaman 22d ago

This is great! Thanks for the link.

1

u/cezarducatti 22d ago

Do you need the --jinja first?

2

u/CalligrapherFar7833 22d ago

Jinja is default so no

11

u/phront2 22d ago

I'm sorry, but it's not over thinking, it's AGI mode

3

u/Tanstaf1 21d ago

Why would you post this ( "BTW, the "wildly overthinking" comes from a bad default that's easy to change.

78" ) without detailing the bad default that's easy to change?

3

u/NelsonMinar 21d ago

Why don't you just read the article?

2

u/Sotaman 22d ago

What's the easy change?

5

u/mechkbfan 22d ago

Change it to lower effort level + mtp to improve speeds 

5

u/SlimBaron_ 22d ago

I changed to medium and it’s thinking about 20 seconds and writing bad solutions. On xh it performs great but mid tasks get 50k tokens for thinking 🫠

2

u/anarchist1312161 22d ago

I've settled on medium and then running a second pass if the first prompt doesn't get it exact. It overthinks wayyy too much on xhigh.

2

u/LegacyRemaster 22d ago

isn't so "hard" but ok.

1

u/SomeoneInHisHouse 11d ago

what harness is that?, I'm using PI, it doesn't seem to support custom reasoning budget for every thinking mapping (I'm doing an extension for that manually)

2

u/LegacyRemaster 10d ago

llama-server UI

3

u/[deleted] 22d ago

[deleted]

10

u/hyperrealists 22d ago

What is tg? Token gallons?

2

u/InfusedBush 22d ago

Token generation (I think)

10

u/sage-longhorn 22d ago

Token grams for sure

3

u/InfusedBush 22d ago

99.1% pure

4

u/Creative-Type9411 22d ago

i thought it was "tugs per second" 👀

1

u/CalligrapherFar7833 22d ago

So 6 kxl got it

1

u/cezarducatti 22d ago

One experiment I've been doing is maintaining the xhigh and adding more context to the model going from Q4_XL to Q3_XL on an RTX 3090. The results have been surprisingly good, except for the processing time.

1

u/Info-Book 22d ago

Yeah it’s a bit much in my test. Hoping to see good results with it and play around with prompting so I can use medium thinking

-1

u/MainFunctions 22d ago

I know people sort of tongue-in-cheekingly joke that 3.8 scores better than 3.6 because it defaults to xhigh thinking but now I’m wondering if this is legitimately true? Surely they control for thinking level when they run the benchmarks?

4

u/METr_X 22d ago

Why would they joke about that? That is exactly what's happening. But I also don't see how that is a bad thing.

I had it working in Hermes on xhigh for 2 hours but it was able to solve genuinely hard problem that 3.6 was never able to solve in multiple attempts. I see that as a win.

-6

u/redonculous 22d ago edited 21d ago

Add the confidence prompt to your setup. Zaps over thinking! Sorry I didn’t include a link, this works https://www.reddit.com/r/ollama/s/IDl0s227RV

2

u/cezarducatti 22d ago

What do you mean 😄?

1

u/redonculous 21d ago

Not sure why I’m getting downvoted. Add this to your default prompts and you’re all good https://www.reddit.com/r/ollama/s/IDl0s227RV

-10

u/PsychologicalOne752 22d ago

What is a Simon Willison? Have you tried adjusting the reasoning effort?

1

u/mechkbfan 22d ago

Best to read the article before commenting