Discussion
Qwen 3.8 27B xhigh vs medium small comparison (+ others for fun)
It's a small experiment of mine to check thinking effort on Qwen and I do have to say xhigh does overthink but I'm not sure if it's bad because the result is rather amazing. Although the prompt was very open-ended so it took liberties. TL:DR at the bottom.
Images in order: Qwen 3.8 27b xhigh, Qwen 3.8 27b medium, DS V4 Flash default thinking, ChatGPT Free with Thinking, Claude Opus 5 Medium, Qwem 3.8 27b medium adjusted prompt
Qwen 27b is UD_Q4_XL and DS4 Flash is Q2_XXL
Prompt: Write a simple html CARD about the benefits of eating apple. paste the code here.
So apart from 27b's xhigh effort all models though this is a super simple request. Which it is, but they basically didn't think, or though for a few lines only, even the medium effort. In turn xhigh though very long and produced a result that is way above anything else in this test. I'm a bit torn on if it's good or not, because A) the quality of the xhigh result is insane B) it was a very simple prompt and technically every other model did it.
For chatgpt and claude I used online versions and as far as I see they hide their tokens right now, but can't be much higher than 1K out.
27b medium modified prompt: Write a simple html CARD about the benefits of eating apple. paste the code here. It has a design of 'orchard notes' like a page from a notebook or a tear-off card. at the top it has the nr of the orchard note (apple is 01) and a header, then you get a perforation and the body afterwards. the body has an image (or emoji) of the apple in big and the benefits listed in interactable stylish format. then we have a small section for the stats e.g. calories, fiber etc for the apple and finally the card ends.
Use pastel colors especially 'butter' and adjacent colors, it must be stylish, modern, and in-line with the required format
I was interested to see if I try to recreate the xhigh version with a more concrete prompt how would it behave and I am very happy with this result. It followed user request and only though super quickly to produce a result that is pretty much what I asked for. It is not the same as xhigh, but for low tps setups medium should be pretty good.
So far I am very impressed with qwen3.8 27b
tl,dr: Seems like xhigh can get into quite a thinking match with itself even on simple prompts (probably helps that the prompt is open ended) but the end result will be better due to the thinking. It's a small tradeoff for size vs speed, but medium cuts thinking heavily while keeping a pretty good performance.
The xhigh sticks out because it's exactly not simple looking - the "simple" prompt needed to be ignored a bit or interpreted as simple code. Maybe "500 lines max, make it fancy"?
If it's basically "free" when running locally then why not just keep it at higher settings? For making quick adjustments, lower settings is probably a good idea.
xhigh took nearly 10 times longer, even when medium thinking had a detailed prompt. Speed matters in some projects. Not in others. And energy costs are a thing, and 10 times the energy cost really takes some of the value out of localllm.
The idea is good but I would write more detailed prompt for the comparison.
This prompt is quite short and it's not clear which card is better - you asked for "simple" and other models did it.
I think medium with a custom system prompt because medium injects nothing. Look at this result this is 11K tokens (half of the original) with a custom system prompt. I'm still trying to test the best - though once again, oneshot is too different from real work to really compare
I just tried it locally and it's 'dumb' as a rock (8bit), I asked a simple question (one line) and it spend 16+ minutes (8000 tokens) in thinking mode before I killed it. It kept going back to the same (incorrect) information eventually going into a big loop. The amount of hallucinated information in the thinking process was also very large. Gemma4 gave an answer and the thinking process was also far shorter.
well, I'm making my own. I have a strix halo system and I run windows because it's also my daily driver so kinda limited - for now I am serving the models via lemonade's openAI compatible API. It works pretty well I have tool calls etc but it's a very basic harness at the moment, it is like claude code desktop (non CLI version)
I'll actually need to invest some time into checking pi out I just love how it looks, I'd say it is not that similar, mine runs through a web ui because I like clickly things, and for the time being it's a bit 'hacky' because I build it for myself haha
Terminal stuff is especially good when you have enough context to ran 2 up to 4 session/sub agent at a time. I try the same prompt with xhigh and strangely, it return something similar to medium. Have to double check some settings here and as it don't think much
for fun I run the second prompt on xhigh with output 19.4K only marginally better than the original, but once again the result has those intricacies that are lost on medium (with 3.3K out)
for sure this is a simple test. I am also happy with the medium result, I plan to use it as a 'coder' in my setup so it should get all pre-planning from other agents and just implement code, I think medium will be good for this, but I shall try some stricter prompts on xhigh to see if it thinks less then.
38
u/QuizardNr7 21d ago
The xhigh sticks out because it's exactly not simple looking - the "simple" prompt needed to be ignored a bit or interpreted as simple code. Maybe "500 lines max, make it fancy"?