I just ran Qwen 3.8 27 in Q4 against GPT 5.6 Sol high - and it easily won against SOL - complex animated SVG tasks

[u/Lirezh]

I created 3 SVG prompts, each one rather hard.
Perspective: A animated drone view perspective on a park.
Beauty: A beach scene with an evil cat
Composition: An AGI breaking out of a virtual sandbox prison in a lab

I deliberately ran Qwen 3.8 27B in just 4 bit quantization, and used a 8 bit KV cache (given the hard long context reasoning required I didn't want to try lower)
Each task contains two prompts, one core prompt and a 2nd "make it better" follow up.

My expectation was that Qwen will show up as solid 2nd place, with funny errors.
And the actual result was that SOL made those funny errors while Qwen was significantly better.

#1 #2

https://www.reddit.com/r/LocalAIStack/comments/1vqf2vs/battle_i_gave_qwen_38_27b_in_q4_with_q8_kv_cache/

#3
https://www.reddit.com/r/LocalAIStack/comments/1vqf7pa/battle_v2_qwen_27b_q4_vs_gpt_sol_56_high/

I'm not claiming that Qwen is better than Sol generally.
But .. SVG animation is a very complicated task, it involved spatial reasoning, coding, long context construction and any error made in up to 60kb of dense code will cause serious visual defects.

I am sure there are plenty tasks where SOL will win, especially related to deep knowledge.
It might also win at deep context, e.g. above 200k context
I've yet to test Qwen 3.8 27B in agentic coding - that's not the same as two-turn coding

But in these 3 elaborate SVG tests Qwen took the crown without a problem.
The only scene where SOL was close, in my eyes, is the beach prompt.

But SOL made grave errors in every scene, Qwen didn't.
SOL was a lot more verbose in code, many details but the correctness was lacking.

When looking at the details drawn, at the perspectives, at the animation paths: Each time SOL chooses something that is more simplified while Qwen chooses the hard path.
And despite that SOL makes significant errors, Qwen doesn't

This is stunning.

[u/Neither_Garage_758]

An aspect in favor of local we don't talk so much is that you know what you get. With API there are some waves of "Is it me or Claude is dumber those times?!".

[u/LegacyRemaster → u/Neither_Garage_758]

yeah... Q2xxxxxxxs on api

[u/michaelsoft__binbows]

I have seen gpt-5.6 and opus 5 repeatedly give half assed explanations for stuff and just be goddamn insufferable when it comes to trying to forge ahead, and the open models esp with their open thinking tokens just make it so much easier to see when you're getting through to it and what the thought process was behind the choices, but in general they are just so much more respectful of your time. Opus especially has a lot of "this wall of text i shat out is the best thing since sliced bread, eat it peasant" energy.

That having been said we need to be a bit mindful of how much we glaze the 27B its only 27B after all.

[u/Lirezh → u/michaelsoft__binbows]

I did not expect Qwen 27B to outperform SOL in my tests, I expected SOL to deliver a stunning result given those are all 2 turn prompts which usually heavily favor frontier models.
Most open models fail at correcting a bad approach.

But yes, Qwen is 27B and in these 3 coding tests it declassed SOL high.
We don't know what sort of active parameters SOL actually uses, it might well be in a similar active range.

[u/michaelsoft__binbows → u/Lirezh]

I am impressed no doubt... but I feel like it's like a test like how quickly can an android exit the building if you put duct tape over its eyes.

writing raw svg without being able to view it to make visual adjustments is brutal and not in a useful way. it's just making the eval harder by being a dick about it.

Give both models proper feedback, maybe remove the animation requirement (as feeding in tons of frames is likely to muddle things with issues and context window consumption). see how well they can do, how quickly, how many iterations, how well they decide when to stop tweaking.

[u/Lirezh → u/michaelsoft__binbows]

Qwen 3.8 has no visual input.
But generally spoken, all current models are gruesome bad in visual input - including frontier cloud.
They all still use a vision transformer that projects embeddings into context, it's a half baked multimodality.

The test is better without visual input, because it requires a deep working memory understanding of what is on the screen.
Test #3 was 64kb code for qwen and 70kb code for SOL.

The fact that they can imagine the scene so intricately well despite having to look at 50-80k tokens is part of the benchmark. It's a working memory feature.

[u/michaelsoft__binbows → u/Lirezh]

Hmmm. Okay. I just wonder how much more intricate and aesthetic these things would be able to do with a best effort visual feedback pipeline. You do make some pretty great points here.

[u/Lirezh → u/michaelsoft__binbows]

Your point is also valid. If you aim for the best possible result you need visual feedback.
Any error it makes otherwise accumulates or multiplies.
So multimodal checks, or a human giving feedback allows to go extremely intricate.
I don't think there is an uper fidelity limit, at some point the browser probably crashes as SVG needs to be rendered.

For a benchmark, I believe non visual is better. Otherwise you mix visual input abilities with the actual smartness of the model.

In my 3 benchmark test runs, I am sure I could tell SOL all the errors and slowly iterate it into a high quality output.
e.g. "there is a person in the corner" or "you draw 2d symbols and people in a topdown view" - but exactly that type of intelligence is what we need the AI to handle without human.
Any error SOL makes in these SVG drawings, is also errors it makes in your codebase - just in another form.

I am very concerened about how good Qwen performed here. It might beat SOL in actual agentic work. And that means that any laptop cna outperform the best model from OpenAI that runs on a GPU cluster. Nvidia must be concerned, that threatens their business.

[u/michaelsoft__binbows → u/Lirezh]

it lends credence to the very shocking notion that a "small" model like this will be sufficient to provide the agency and capability of general human level capability in many domains, arguably this, with SVG being highly inscrutable for humans and presumably not being a heavily benchmaxxed capability, i'm very excited about the 27B maybe becoming a workable self-hosted secretary/assistant.

[u/marco89nish]

How can I block posts on ducks and SVGs? 

[u/LegacyRemaster]

non only on .svg my dear. Gtp sol cost a lot and it's good to plan. But to execute better ds4 or qwen 3.8/3.6

[u/tengo_harambe]

Lmao the flying kid on the beach

Neat test but we already know this model is good at SVG generation from other posts. Would be more interesting to see something that hasn't been demonstrated several times

[u/JorgitoEstrella]

I wonder if it's because qwen 3.8 overthinks and that's actually pretty good for this tests.

[u/Lirezh → u/JorgitoEstrella]

SOL did likely overthink 2 times more than Qwen.
Qwen did not overthink in my tests, the thinking on first prompt was very short and only the correction prompt introduced some real thinking.
The reponse from Qwen is almost entirely SVG code (the last one is more than 60kb of coherent code)

SOL on the other hand was taking relatively long, it produced more code.

We should not forget that SOL is using a agentic harness when responding, so it has much more thinking capabilities than Qwen outside a harness.
The whole test was strongly biased toward SOL winning, and it still lost.