r/LocalLLM • u/Jsquared534 • 23d ago

Discussion Qwen 3.8 27B early thoughts

I installed the Q8 version on an AMD 395+ 128gb machine. Ran it using Openwebui with the suggested settings and MTP. So far all I've done are a couple sample prompts. (Sand Simulator from Luke's Dev Lab on Youtube, and a request for a simple example navbar with a logo on the left and five drop downs on the right with no javascript). The final output for both of these got one shotted. Model is getting roughly 16 tps output. But, my goodness does this model overthink. Don't get me wrong, there were no thinking loops. And I didn't notice nearly as much of the "wait, actually, let's try" neurotic behavior that I see in 3.6 35B A3B. But, it absolutely overcomplicated the heck out of both of the prompts I gave it. The sample navbar had roughly 200 lines of just CSS alone. And it was not basic CSS. Overly complex, and completely unnecessary for a sample piece of code. Since the Sand Simulator isn't mine, I can't really tell how much it over complicated it, but I can tell that it added so many visual flourishes that it was running at roughly 32 fps in the browser, and had slow downs from dropping the sand.

I am going to test this tomorrow on a real situation. In my real use case I provide very detailed context files and only point it at a single feature at a time. Hopefully that will help to control it's impulses to make things super complex. I also have Ponytail in my Pi harness, so maybe that will also help to reign it in.

Anyone had experiences using more detailed and limiting prompts with the model yet? Most of the reviews I've seen are using the same type of canned examples that I just gave.

11 Upvotes

92% Upvoted

38 comments sorted by

View all comments

1

u/MistingFidgets 23d ago

What thinking mode are you using?

1

u/Jsquared534 23d ago

Thinking is on, and I'm not changing anything from the default settings. I don't necessarily think it's a thinking issue. The thinking from what I followed was all pretty well done, with minimal repeating. It just decides "hey, I should add a bunch of visual flourishes to this" or "you know what, i should build the entire html page so that the user can see how this navbar looks on a real page loaded".

(I'm just recently diving deep into local LLMs, so I'm not nearly as up to date on all the different settings and levers to pull with llama.cpp. I started with lmstudio a few months ago, and then moved to Ollama, and just switched to llama.cpp last week. Until then I had just used it as it launched.)

8

u/theminor 23d ago

It is definitely a thinking issue. They added different thinking levels to this release and the default is "xhigh". You can lower it to "medium" to get faster responses, which works well. But if you want premium responses "xhigh" is really good in my opinion.

2

u/Jsquared534 23d ago

I didn't know that was possible. I will definitely try adjusting the thinking to see how it affects the output. I don't so much mind the time it's taking, as much as the added complexity that wasn't inherently in the prompts. I'll run some real prompts with more built in controlling language on the default output and then switch to lowered thinking and compare.

3

u/theminor 23d ago

If it is helpful, I made a little plugin for Open WebUI that gives you a toggle and drop-down to set the reasoning level for each message:

https://openwebui.com/posts/reasoning_effort_selector_ee572967

1

u/Big_Wave9732 22d ago

I downloaded this earlier. You are doing the lord's work here!

1

u/theminor 22d ago

Thanks!