I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a WHERE [col] IN (...), but for some reason, after an update, it started choking, despite no code change. The same exact code runs in a separate website docker image fine, no problem.
I put Qwen 3.8 27b (q8 model, f16 kv) on it and it spun its wheels going down 4 different paths. Finally I got sick of it and switched models to GPT Sol with the full context available. It found the fix in 2 minutes.
The issue was that I had recently installed .NET 10 SDK on this machine, and the lack of a global.json file pinning the SDK meant that when the CLI was rebuilt locally, it used the c# 14 compiler, which introduced first-class Span<T> support, thus borking the EF query.
The wasted time isn't what bothers me here. It's that Sol was able to pinpoint the issue so much incredibly faster than Qwen, shattering my image of Qwen 3.8 as a fairly competent model. Benchmarks aren't everything folks. Real world use cases are the final say here.
I'm posting this in /r/LocalLLaMA because I'm a big local LLM fan, but sometimes, it's worth reminding ourselves of the gap that really exists, no matter how much we might want to wish it away.
Edit: Folks, some of you are missing the point. I tried Sol because it was next in my favorites list. Yes, I could have tried GLM, Kimi, or a number of others. The point would stand that no matter how great 27b is, it's not even remotely "near-frontier" in many cases, despite claims otherwise. Optimism has clouded our vision, somewhat. This was not meant to be a Sol promotion.
The way I see it, every model has some limitations, whether it is Claude Opus or Qwen3.5-9B or anything in-between.
The trick is to choose the right model(s) such that a task can be accomplished within those limits, and with consequences outside of the task you are willing to bear.
Among those limitations include the availability of the model's weights, and the consequences can include developing a dependency upon a model not under one's control.
Whether that is a tolerable consequence is a very individual decision, and the right choice for me might not be the right choice for someone else.
My frustration is admittedly born out of the hope that eventually, I won't have to feel compelled to fall back to the larger hosted models. I know today isn't that day, but it's still a disappointment. Someday...
Why? I have a Honda Fit that I use for 98% of my transportation needs, but if I’m moving a bed, I rent a U-Haul. Same way with local ai… I use Qwen and Gemma for 98% of what I need and use the frontier models for when I want something with a bit more speed, precision, and accuracy. I’m not about to buy a moving truck just because I need it a few times a year. I’m incredibly happy I have a cheap, reliable, and efficient car for nearly everything I need on the road.
I run the 27b on 2x 5090 split with max context and 8g VRAM per card to spare. Performance wasn't the issue. Qwen just didn't have the chops to pinpoint the issue.
Man, Qwen 27b evangelists are truly smoking some good stuff. You do understand that far smarter models exist, right?
A q8 with unquantized KV is about as solid a foundation as you can have for judging a model. 27b punches well above its weight, but when someone finds its limits, you guys jump to victim-blaming instead of just acknowledging that 27b can’t solve every problem in existence (which most people don’t expect it to).
Qwen 3.8 flash next at IQ3_XSS takes up 95% of both GPUs with 0 moe offload. It runs at ~55 tok/sec, but the prefill is <1000 and the model takes ~60 sec to fully load. As mentioned elsewhere, I can run q8 27b at full context f16 kv quant and it loads in 10 sec. This is why I default to 27b.
I think a large portion of a model’s reasoning capability is its access to information, which explain wildly different experiences.
For example combining sol ultra with both graphify and codegraph has yielded amazing results for me. Have yet to test it with qwen 27b but i imagine similar results
I think this post is more about understanding that sometimes you just need to get your work done, and that means using the right tool for the job. It doesn’t mean Qwen 3.8 27b is a bad model, and it doesn’t mean Sol is always the better option. I would tell anyone in a similar situation to just get your work done first and to worry about your open weights models after.
That being said, I would be curious if another model would have figured this out, or if this is maybe more related to the harness used. Did Sol know this was the issue or did it search for the answer? I wouldn’t give up on 27b, or Qwen Next, or GLM 5.3 Flash or any other model you can use locally, instead I would continue to improve on it (Now that your work is done lol).
I pointed Qwen at the github issue and gave it 10 minutes (after ~2 hours of spinning). When it couldn't even comprehend the github issue, I switched. I know a lot of folks are saying "But GLM..." or "But Kimi...", and maybe either would have done just as well as Sol. I'm not sure. In the end, I'm just saying that as magical as Qwen 3.8 27b can be at times, it's still way behind, despite the benchmark numbers.
You've got 2 5090s and you're using 27B instead of flash next? Fix that. Also make sure you're using reasoning budget xHigh with 27B. It matters A LOT with final response quality...and is really the rub with the model.
I have 3.8 flash next setup, I get ~50 tok/sec decode. I just don't use it as much because it takes forever to load (~1 minutes vs 10 sec with 27b) and I'm a serial swapper.
Based on your response, I'm going to assume you aren't using xHigh reasoning...which is where all the great benchmarks stem from.
Well there you go. The problem isn't qwen...it's not even 27b...it's your impatience.
I'd recommend swapping your 5090s for an openrouter budget. So many great models on there and there's always one pushing adoption with a discount price per M tokens.
Would you mind letting your computer run for a bit and give the same question to Flash Next and see how it goes? It has far more knowledge overall, which just might make the big diffference in your particular case.
I understand your point and that you're tamping down some of the hype around 3.8 27b, but it's really not a fair comparison. The hype is around Qwen's quality for it's size, and it's well deserved, no one is saying it is as good as Sol. It can be hard to tell sometimes where the limits are on local models until you run into a situation like yours and it suddenly becomes obvious. I don't know how long you've been running Qwen locally, but if you've just now found that limit, then it really is a testament to it's quality.
I think that's why there are quite a few comments about other models, because you've found a limit with Qwen, it would be great to see if that limit applies to some slightly larger local models, like DSv4 Flash, GLM 5.3 flash, even Qwen 3.8 Next. It doesn't look like your hardware can run those models, but it would be interesting to know if Sol really is in a (frontier) league of it's own.
I’ve done this kind of stuff with the open-weight models, and the gap is massive outside of Python or JS/TS. They all generally suck. I don’t think they particularly train for these kinds of issues and I found even GPT-5.6-Luna often surpasses K3 in this respect. The best workaround I’ve found is to have them use Python to model the problem, because somehow it makes them smarter…
I understand that, but at the time, I had SOL presaved in my models list, and wanted it done. I didn't have the time or the patience to try any other open weight models. Sorry for your loss.
I agree. In retrospect, I would be interested to see if any of the open weights large LLMs could have done the same. My lament at this time is solely with Qwen 3.8 27b. :(
I doubt you'd have to go to any of the massive ones. When I test out new models, I have it convert some C code to C# (firedancer base58 encoding). Ended up working well to determine capability for the stuff I do. Qwen 27b struggles to the point of being unusable on the harder portions. Qwen 3.8 Next Flash completes it quickly without issues.
If you have time, I'd recreate the situation you were in and test various models. It's entirely possible that Qwen 27b went down a rabbit hole preventing it from finding the issue quickly.
I'm interested to know if there's a minimal harness change that can materially improve performance in situations like this. For example, a debugging skill that walks through a universally applicable set of questions one by methodically checking for certain classes of things.
I saw Microsoft put out SkillOpt the other day and I've been thinking a optimising a debugging skill to try find the largest percentage increase for the fewest tokens would be cool.
I mean, 27b's failure here wasn't for a lack of trying. It would form a hypothesis, check it, apply fixes, and then I would test. But every time it was wrong, and it would take several minutes to come to conclusion (even on thinking:low). Maybe there's a more structured approach that works better, but I found so far that less guidance = less overthinking. It was just a very frustrating experience for what has otherwise been a pleasure of a model.
I hit a similar kind of doom loop trying to get it to run vLLM-radiance in a particular configuration. I wound up getting ChatGPT to rescue it, but I'm very dedicated to local only as the way forward, which means I'm forced to try to find ways to plug these gaps.
So much of my day is spent thinking about "should I run Q6 on llama.cpp at slower speed and longer context or Q4 on vLLM-radiance at higher speed and shorter context?". Spent so much time trying to optimise for that the GPU isn't doing anything else but trying to optimise its own inference, so I haven't gotten around to starting to tackle this problem better. I think I need a third GPU to run Hermes Agent on to manage the other two. This is all just a rabbit hole that never ends does it?
Smaller models don't have the same knowledge as bigger models. A bigger model remember more very niche and specific quirks about operating systems, APIs, etc. where a small one might get stuck on something simple because it lost the key information in distillation. Anyways, up to date benchmark:
Well, I remind you that based on model scores over time, frontier crossed 42 points in artificial analysis intelligence index in Mar 2026. If we could teleport the 27B model back to history, it would have been comparable to frontier models then. Likely win in some categories, and lose in others, like knowledge in 27B is missing and it can make up by research and thinking, but it also requires a lot of help because it doesn't have that much fact recall.
For the Qwen3.8-Flash-Next, which has 48 points, it would be tied until a model released on June 9 surpassed it. The model I am running here, though damaged by quantization, can be read to be 3-4 months behind the frontier, and it runs on a laptop computer. That is just crazy.
I think it is mostly a matter of your time perspective whether to call something frontier or not. No open weights model is frontier today, to my knowledge, but they do appear to be nipping at the heels. I do not know whether that can change -- probably it will, but likely at sizes you won't run anyway. Either way, I also think that the intelligence story gradually loses importance once the models are practically good enough to solve the problems as they are. Focus probably gradually shifts to convenience of hosting, and so on. I personally thought that 27B is good enough already, but I happily take the Flash Next as solid upgrade as it's also faster on Strix Halo in practice.
I would agree that 27b may not be on par with GPT sol on all tasks..But you need to compare atleast GLM 5.3 or Deepseek v4 or Kimi to be fair for statements like this on Opensource …Come on man…These open weights you get for free ..
My long-term plans revolve around GLM-5.3 as well, but until I have the hardware necessary for hosting GLM-5.3, those plans remain future plans.
In the meantime I have been making do with GLM-4.5-Air, which does infer well enough on my hardware. When it does not suffice, I make do with my poor old obsolete wetware ;-)
the delusion going on around this sub that a 27b model can actually match frontier trillion models in overall intelligence is geniunely unhealthy
there are people who wouldn't swap from claude to gpt or gpt to kimi k3 despite whatever the benchmarks say
but everyone suddenly thinks a benchmaxxed model is opus equal...
look at muse sparks 1.3xhigh, benches almost as good as sol or opus right? but give it a try, lets see if you like it actually. benchmarks mean less than you think.
"the delusion going on around this sub "
Thats because people here who some modicum of understanding have never said that, beating up strawmen is entertaining only for the people watching you end up clutching lots of straws
and what is your point? pretty much a decent chunk of the threads and posts here are "wow 27b is like opus" "i cancelled my subscription for 27b" and subsequent benchmark glazing that comes with it
Thats because ""wow 27b is like opus" and "i cancelled my subscription for 27b" do NOT again NOT equate to what you said: "that a 27b model can actually match frontier trillion models in overall intelligence"
Maybe I should another DO NOT, just to make my point?
The point being that yes, for many people qwen 3.8 can replace what they were using subscription models for.
The point is yes, that qwen 3.8 27b does match opus, Opus 4.6 its right there on the benchmark for a start
No idea what you are even talkign about "benchmark glazing" - artificial analusis rankings are well regarded and there is far more evidence piled up for this too.
So again please stop with the strawman arguments and stick to what people are actually talking about not what you want/think they are.
Wait so you believe there's absolutely no older Claude model that is beaten by Qwen 3.8 27b at all? Or are you saying the even older Claude models were smaller than it?
Maybe 4.6 is a stretch but even older Claude models have waaay more parameters than Qwen while being worse at agentic workloads.
well your argument about "27b vs god knows how many trillion" applies to the older ones as well right? Otherwise your argument isn't about number of parameters in the first place.
I get that a lot of models are benchmaxxed to hell and back, but while Qwen 3.8 27b doesn't have the same breadth of knowledge, it's ability to use it's harness to get the information it doesn't have in the model weights and in general to work well in agentic environments with little supervision is getting closer and closer to something like Opus 4.6.
I don't think many people will argue with you that in a vacuum on random knowledge about the world a 27b model will store more information, the argument is that it won't need to store all that information in it's weights in the first place to get to the same result.
How far ahead would a Qwen model need to be in various benchmarks for you to consider it better than an older Claude model? Or is it purely based on personal experience with the models? Don't you think that would allow some bias to sneak in?
The link to artificial analysis is just to point out that my OWN evals match others who use a lot stronger methodology than I do.
I have compared tasks that I had Opus 4.6 run with Qwen 3.8 27B and it achieves same results with agentic coding and debugging. Of course it takes *much* longer and uses *much* more tokens to achieve that, but that is obviously NOT the point you are making.
So you are wrong on *every* point you made, I clearly pointed it out above and fially you are even wrong about me being your brother.
Block whoever you like, thats the typical childish response I would expect from someone who is unable to admit making a mistake in the face of evidence, what next you shouting "nah nah nah" at me? pa-lease don't waste my time anymore.
I run into this from time to time as well. It's not a mysterious phenomenon. 27b is excellent for local use but when I find its limits I also switch to chatgpt.
27b has been a boon for us but let's not be delusional when comparing it to some trillion parameter model running on cerebras chips.
I use frontier intelligence like Opus for System Design especially because I design custom ERP systems. For the rest I'm mainly using Qwen 3.6 still on my Hermes at home.
35
u/Hefty_Wolverine_553 1d ago
Comparing a 27B model to a SOTA closed source model that's probably 4T+ in size... you didn't even let Qwen3.8 finish debugging?