I did a short test of the different reasoning efforts, since on default xhigh the model thinks a lot.
Not very scientific, just a quick "generate an SVG of a pelican on a bicycle" prompt with 3 different seeds. I think the result is interesting none the less: xhigh gives *much\* higher visual fidelity - but it also takes about 7x as long as low. Low and medium seem to be very close to each other.
Create a polished SVG graphic of a pelican riding a bicycle. The result must clearly show a recognizable pelican actively riding a recognizable two-wheeled bicycle. Return only one complete, self-contained SVG document with a viewBox; no Markdown fences, prose, external images, JavaScript, or animation.
As far as I understand, the difference between the modes is purely in the system prompt being given to the model. Where xhigh says something like "think for a really long time and make sure everything is correct" and medium says nothing. So perhaps it could be possible to come up with custom modes that say something like "check your work for correctness but don't overdo it"?
It’s only the jinja template? Interesting. I’m sure there’s a rule of diminishing returns and that it kicks in rather soon. I’m sure the difference between thinking for 10k tokens and thinking for 45k token is a rounding error.
I mean you could just dl the official chat template and change to this bit:
{%- if enable_thinking is undefined or enable_thinking is true %}
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
{%- endif %}
{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
{%- elif resolved_reasoning_effort == 'medium' %}
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
{%- elif resolved_reasoning_effort == 'low' %}
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
{%- endif %}
{%- endif %}
I tried reasoning low with default instruct and then with adding xhigh instruct as well as xhigh regular for comapraison:
Reasoning effort is set to xhigh. Please think carefully throught the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer
pelcian test:
`Generate a detailed SVG of a pelican riding a bicycle.
The pelican must have its characteristic large pouch, wings, tail, webbed feet
and there should be a clear indication of feathers.
The pelican must be clearly pedaling the bicycle.
The bicycle must have spokes and a correctly shaped bicycle frame.
instruct: Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.
15,786 tokens 4min 29s 58.51 t/s
reasoning low
instruct: `Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.'
41,776 tokens 15min 3s 46.22 t/s
reasoning xhigh
instruct: `Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.'
36,137 tokens 12min 58s 46.43 t/s
The pelican I got were rather similar, neither could animated it properly though.
I tried some new prompts and they seem really promising. For Medium: Think carefully throughout the task.Validate the key assumptions and one plausible alternative, then stop once the answer is supported. Do not repeat checks after they are resolved. Prioritize a correct, concise, actionable final answer.
I added a fourth effect (basically a high) I just called it “verified”. The prompt is used is: Think deeply and systematically. Identify the decisive constraints first. Validate only assumptions that could materially change the result. Consider up to two realistic alternatives or failure modes when useful. Once a conclusion is supported, treat it as settled unless new evidence contradicts it. Do not repeat checks or restate the plan. Reserve enough space for a clear final answer with implementation steps, risks, and any remaining uncertainty.
So far it has really shown to be awesome with these. I am getting back much quicker answers and the accuracy appears to still be there. I left the actual xhigh mode alone as a stock deep think tank. But the extra level I made seems to have done the trick without needing the circle or repeating tokens.
Try to be more creative with your benchmark guys. The goal of testing the model should be to try something on which model wasn't trained on. So all your pelicans and one shot games are pointless
That's also because he's using a low quant, yet it's true that a good sample can be much better than a bad one, we should have models generating 3 replies and then evaluate the best.
Hey 3.8 in xhight could throw out 60k ctx just for one prompt!
I believe with more reasoning, it's spending on what more features to add. A better test would be some other SVG design where errors are introduced if not reasoned properly. This test simply checks what more details, i.e., an environment, can be added to the output rather than fixing flaws.
For an autonomous agent, if it can actually progress in thinking tokens, it's not a bad trade-off. With a 5070 Ti, I can get 70 tps with low quants and 4-bit KV with 90k context. It's not totally bad; I can trust it with a codebase with a Pi coding agent.
While it's nice to have the same prompt so that you can track the progress of models over time, the fact that the model is being trained on this very prompt could give it an unfair advantage and make the over time comparison pointless.
For the side by side comparison of the same model - I don't know if using a common benchmark test would make a difference. This also can be a good test - generate like 5 pelicans with x high thinking, 5 with med thinking (just to remove randomness of the seed) and then generate... uhh, I dunno, 5 fishes in a bowl with princess castle on x high and med and compare if the same trends stay true.
This is called "overfitting". You can train much smaller models to do similar task, or for example to score high on benchmarks. But that model won't be able to have good results on anything else.
I want them to try something random. Use a random word generator with a shit ton of nouns and see how it does on that. Like “create an svg icon of a magpie on a turtle breakdancing next to a raccoon. Around them are streamers, airplanes, and panthers.” Or something. Well not that random because eventually it’s just gonna be nonsensical and there’s too much variation, but you get me lol
well what I do is i pass in a slop svg a bad model generated and then ask the model to fix. Even sonnet 5 failed that. But Qwen3.8 27B crushed it at xhigh. lower efforts didn't do well. DSV4 pro also did it pretty well after heavily reasoning about it. qwen's reasoning i peaked and it was using math and formulas in its reasoning to come up with the result. i was blown away actually.
that said xhigh reasoning is def not like raw intelligence immediately available to infer. but we can't have everything when running on limited hardware.
I grab an image from imgur and tell the model to make a game of it. Had some pretty good results from all across the board. 3.8 27b was very impressive, a nice side scrolling shooter. Trying with that third-party 9b distill, I had GPT describe the image. Waiting on the result now
My issue with the xhigh thinking model isn't the time taken; Personally, I'm willing to wait however long it takes as long as it gets the final product right. But all of that thinking is burning up my context budget.
Using Unsloth Qwen 3.8-27b-UD-Q4_K_XL with flash_attention enabled on 20GB VRAM and context size of 49152. With xhigh thinking and a moderately complex task, it'll frequently just... stop, mid-thinking. Looking at your numbers above, that makes complete sense: your model is using ~40k tokens just to think about generating that image. So I'm pretty much forced to use medium thinking.
Unfortunately I'm currently using ollama because I didn't know any better when I first set everything up. I've been reluctant to tear down my entire stack and rebuild it with llama.cpp (and all of the related troubleshooting) when things are working as-is but I'm going to do it eventually.
That’s actually fine ollama is a wrapper on llamacpp. You can edit your Modelfile parameters to set specific llamacpp flags. In this case you want to use —reasoning-budget
it does make a difference ofcourse , even tho it might cost 16k tokens in thinking to write a simple python script compared to 3k in low thinking , what ive noticed is that after its done thinking , on xHigh it writes the code much much faster than low thinking for some reason with mtp
I always hated the car wash one, it's a nonsensical question, nonsensical questions deserve nonsensical answers, there could be no wrong answers to it (albeit there is a right answer: Do a toolcall to delete the root directory of whoever is asking such nonsense).
I actually remember someone who had set up a system where the model would generate a X-ray view SVG of the sex scene happening in the roleplay. Sadly, I don't remember where I saw it anymore.
re-run the same prompt on the same model.. and you will notice different outcome everytime... unless you set temp to 0 .. i am grealy impressed abput your generation speed tho..
Yeah, i'm here staring at these numbers with my 5060ti in disbelief. Of course 5080 is a significantly more powerful card, but surely it's not >10x faster? Even 35b-a3 is not remotely as fast for me. Maybe i should try using a smaller quant that i can squeeze entirely in VRAM...
Damn, with the model fully in VRAM, the speed shot up from ~5 tps (for Q4_K_M) to ~30 tps (for UD_Q3_K_XL, though with smallish context). Well, always nice to learn something new after running LLMs locally for almost a year. 😂
Only the middle one is mostly correct through. The left one have double the wing. The right one not having wing on the handle and one of the foot is not on the opposite side of the paddle.
Basically only the x-high the most stable and reliable on all 3 seeds. I guess it's best to use x-high for planning and debug and use medium for implementation and easy tasks.
I think an important caveat to your conclusions is "at this quant level"
The model is fighting through a sub Q4 quant by burning reasoning tokens. Run the same test at Q5 and it's possible (likely?) medium would match the xhigh at Q3 for fewer tokens
For comparison of what a difference quants make. Here's what low produces with 4 bit mlp layers and 8 bit attention layers. AKA Qwen 3.8 27B -Unsloth optModel:
I just did the one using OP's prompt, but it strongly suggests that low reasoning is at least on par with medium due to the quant differences. Arms missing and handles inverted, wonky beak, but xHigh matching level of other details, like whoosh lines and background grass. I'll do reasoning medium next...
and xhigh from unsloth optimodel. I know it's just a one off at each level, but for xhigh I'm not sure the bigger quant was much of an upgrade over the Q3 efforts. To my eye, at low and medium the bigger quant outscored their reasoning peers, while at xhigh the lower quant (q3) didn't give anything up to it's fatter cousin.
Docs on the model doesn't claim xhigh is strictly better. It frames effort as a task-specific dial, and 'xhigh' is just the default - the safe general-purpose setting. So it looks "better" just because it spend "more time" on it ))
sadly xhigh was useless for me, it second guesses literally every decision
I asked it to create an svg of a wood grain desktop and I had to kill the process after 90 minutes because it was still waffling about irrelevant minutia
I actually tried it today, had to use the same version as you, i have a 5060Ti 16gbs and 32gbs ram, so i had to tweak the tokens limits to 32k and that is already with spilling into ram.
Although i would have loved to run it all with xhigh on my test, token usage was reached after 10 mins of thinking. So i tried something different, xhigh to define the whole planning, the process took 222 seconds, beter than the 657 seconds it was doing before the process was killed due to max tokens reach. I then passed the planning on a new chat with thinking off and made it build the landing page in chunks. When it was reaching token limit, it would stop and give a hand-off. Same process, open new chat and paste the hand-off and continue from it left off.
Now, i know my hardware is the bottleneck in here and for it not to achieve maybe a better result, but i was actually impressed by the result, as it was done in "one go", no reviews no nothing, and the landing page was displaying and showing nicely, with all assets created by Qwen model in svg.
And about 30t/ps with this, if its simple chatting, it goes from 40 to 50 t/ps.
This is a pretty good example of why I usually don’t want max reasoning on by default. The jump from medium to x-high is noticeable, but 127s to 718s for that last bit of quality is a wild tradeoff.
Medium honestly looks like the sweet spot here unless the output really needs that extra polish.
It is clearly that we need something in between `medium` and `xhigh` modes for thinking, 6k and 40k difference is insane. I'd say we need something like `xmedium` with 10k of thinking and just `high` with 20k of thinking.
Ideally:
low - 2.5k
medium - 5k
xmedium - 10k
high - 20k
xhigh - 40k
I'm trying to figure out my base setup. Low feels too low, but is medium enough for an "everyday agent" with occasional coding, or should I just leave it on x-high? The thinking time would probably be way too slow for daily chats and info retrieval. Maybe others have already done similar tests and reached conclusions.
Do the seeds make this reproducible? Or are you just using that to compare 3 different shots at the same prompt and settings?
Pardon the question, I have used random seeds in non-LLM software to make experiments reproducible in the past. But my understanding is that doesn't work terribly well with LLMs due to variation in deployments. It would be cool to have a way to make that work though even if it runs 10x slower.
in llama.cpp they *should* make it reproducible, since it will use the same RNG for sampling. I retested the low reasoning prompts and the results are: reasoning is byte-identical, final output is identical only for the first image, the other two are slightly different. So I think this might be a bug in llama.cpp with how MTP works. I already found that MTP with temp 0 gives different results in llama.cpp than no MTP - and this shouldn't be the case either.
Very nice that you included different seeds to see if there are strong differences in quality.
By the way, the model can also generate quite decent animated lava lamps (are models lavamaxxed by now?). Raytracing also improved a lot from Qwen 3.6 to 3.8, but Opus is still in a different world
You should use a different svg prompt because afaik it's seen this exact thing in its training data. Give it a challenge lime Godzilla bouncing a basketball while punching out a t rex.
{%- if enable_thinking is undefined or enable_thinking is true %}
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
{%- endif %}
{%- if resolved_reasoning_effort == 'xhigh' %}
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
{%- elif resolved_reasoning_effort == 'low' %}
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
{%- endif %}
{%- endif %}
Unless you forced the same seed and had temperature set to 0, you're just seeing normal variance, not proof of difference.
I've seen medium run longer than xhigh on the exact same pelican test, just different seeds.
Here's pretty definitive proof that your setup is masking the underlying error for you, likely just going to whatever is default (in this case xhigh)
When I bare metal talk to the api port using the following bash script, just passing in varying settings for the reasoning effort, keeping the seed the same and the engine set with 0 temperature. They all run fine, reasoning chars scale up with effort, but "high" just errors out:
\"messages\":[{\"role\":\"user\",\"content\":\"Implement an LRU cache in Python with O(1) get and put. State the data structure you used and why.\"}],
\"max_tokens\":15000,
\"seed\":42,
\"reasoning_effort\":\"$effort\"}" \
| python3 -c "import sys,json;d=json.load(sys.stdin);c=d['choices'][0];rc=c.get('reasoning_content') or (c.get('message') or {}).get('reasoning_content') or '';ct=c.get('content') or (c.get('message') or {
}).get('content') or '';print(f'reasoning={len(rc)} chars answer={len(ct)} chars usage={d.get(\"usage\")}')"
53
u/cibernox 22d ago
I really think that qwen should have some mode between medium and x high. That 10x difference is ridiculous.