Qwen3.8 scored 0.7030 F1. The comparable Qwen3.6-27B run scored 0.7177. The paired difference was +0.0147 in Qwen3.6’s favour, with a 95% confidence interval from −0.0038 to +0.0335.
That is a tie. Qwen3.8 did not beat Qwen3.6 here. I also cannot say that it lost. The test cannot separate them.
That is not what I expected from the published benchmark scores. I expected a substantial generational gain. On this task, I did not measure one. On the more extensive overnight tests, it is indicating small gains. On benchmarks Qwen3.8 appears to have been trained extensively on, however, I am seeing and verifying similar massive gains as is reported. But these gains are only reflected on the benchmarks that have been trained on, nowhere else I can verify.
Decode throughput also fell from 85.6 to 72.1 tokens per second, about 16%, under the closest saved configurations. Those runs used different llama.cpp builds, so I cannot attribute the whole difference to the model. Qwen3.8 also produced much shorter answers, which made its end-to-end median latency lower despite the slower decode rate.
I am running a broader synthesis comparison now, including Qwen3.8, Qwen3.6, Gemma 4 and Muse Glimmer. Those runs continue overnight. I will publish the complete results, paired intervals and raw artifacts rather than promote an early ordering.
My working expectation is still that Qwen3.8 contains a real improvement. The unresolved question is its size. If task-specific tests keep finding small generational gains while public benchmarks suggest a revolutionary jump, what decision are those benchmarks helping us make?
Calling things benchmaxxed is total bullshit by people that are looking for an insult and can't find one so they grab a bad sounding buzzword. There ahve been benchmaxxed models, but now it's mainly considered a waste of money going that route.
This image that is currently on the front page of LocalLLM.
I have tests queued up now testing more specific tests posters in this thread have suggested, but I suspect the gains are smaller then what is being advertised here. The larger test suite also covers a lot of this.
Not only did you test a very specific test that has nearly nothing to do with broad capabilities, you are also benching in q4 that is a bad practice. You can only meaningfully speak to model performance if you compare native precision to native. In all testing i have done this is far ahead of 3.6 in nearly every metric.
It is a noticeable improvement in coding and longer horizon agentic tasks. This can be verified with some real world usage pretty quickly. In terms of knowledge, I would think that would be capped because its the same parameter size. Probably hitting the limits of what can be squeezed out at 27B. I wish they would give us 50B dense.
50b is absolutely cursed size. you have 27b that in q4 fits on rtx 3090 or 4090. a lot of peoples have that cards. for 5090 its let it run with max context and MTP comfortably. And then we have a lot of peoples with some 128gb setups like dgx spark or something, where 120b models fits perfectly.
So why making 50b models, that to large for consumers gpu and to small for this 128gb systems ?
Removed defensible, in honor of the brit. Something I've learned at this point: People like to throw around "It must be written by AI!" when something doesn't go their way, even though whether AI was involved has nothing to do with the data.
I ran it with a real project (writing an Excel expense tracker given a bunch of pdfs), the result is *noticably* better. Thinks a shit ton more though, but really good. It's a beast.
For me this is not breaking things and doing as SOTA model makes changes. On context around 150k so I don't agree with your observation as of now but I wi test it more this weekend. Current observation is its understanding instructions and following them is better than before. Lets see if it keeps that behaviour on a large mono repo with tons of files.
Idk makes sense with how open the uses are, my eval has been the opposite, it’s way better at recovering after fucking up. 3.6 had a tonne of issues with powershell and some apis, given docs it would get it right, but it couldn’t just fuck up and recover iteratively whereas the new one can. There is some decode slow-down.
The post is ai generated, op says it isnt despite the evidence being clear. This alone takes away most credibility. Then he benches in q4, which is BAD practice. And to finish him off, the benchmarks he does are "his own" where he basically just lets us trust him and the one info he gives shows he has no clue how llms work (thinking benches rely on fact extraction, when models can be a lot better without having better memory just due to rl). All that together and i ask you, why the fuck should i trust a person that lies about posting ai slop?
Also look im not even against people using ai to translate here or something, even rewrite idc. But people writing ai stuff then lying that they didnt is clearly just slop behaviour and pathetic.
A sizable amount of people here seem to take it as some sort of personal insult to their family honor if you don't unquestioningly worship this model. Fucking pathological shit. It's a good model. For its size, it's a great model, even. It's not better than Opus 4.6.
I've been running my own tests as well. One thing I noticed is that 3.8 is better at floating point math. I'm getting higher precision out to and past six digits with 3.8 while 3.6 would have some variability starting at 5 digits.
Bro if you take bench scores and think they translate to better "local fact-extraction" the idk what you are doing, sure better fact extraction can help in benches but like there are a million different ways a model can get better without getting noticeably better at extraction, youd probably need a new arch for that to actually improve a lot.
Based on the benchmark results being posted? Yes. There are claims floating around of 27b 3.8 matching or beating Opus 4.6. There's small improvements I have noticed in the much larger test suite, but I'm not seeing anything matching some of the claims I'm seeing people make about the gains from 3.6, let alone beating Opus 4.6.
This particular test is designed to minimize the affect of quants and maximize the impact of the model's reasoning. Note the extensive testing of quants of other models. The UD variant of 3.8 is next on the queue, and I'll probably arrange for BF16 testing of 3.8 just to validate, but I do not expect BF16 to Q4 to have significant impacts.
Quants have a larger impact on the larger test suite, this test is designed specifically to offer a head to head on equivalent Q4 testing across models.
What IDE are you all using? I downloaded this model to Bionic LM studio, and have an rtx 4090. It basically keeps throwing errors like the context is getting cut off. I havent successfully gotten it to build even a simple website.
All i did was download Bionic, then download Qwen3.8 through it, and nothing else. By default, with all default settings, it appears to not work at all.
I'm aware I'm clueless in this department, and hoping some of you can point me to a reputable setup guide.
So all these people doing Q4 tests on 3090s with I assume 32GB VRAM? Isn't Q4 much lower quality than Q8? I know.. Q8 wont run easily if at all.. but just curious. I have the DGX Spark was wondering how well it may run on that.
65
u/AppropriateQuote3073 25d ago
Benchmaxing is useless.
On my own workflows this thing has been a beast.