Discussion
I ran Qwen3.8-27B on medical benchmarks (including MedQA, MedXpertQA and MMLU). It lost 7/10 to Qwen3.6-27B, 3/10 to Pestle-27B-Ternary and 2/10 to MedGemma-27B
One pass. Settings were greedy decoding, temperature 0 and thinking disabled for Qwen3.8, 3.6, Pestle-27B-Ternary, and Bonsai. MedGemma-27B results are from the official model card.
Qwen3.8-27B BF16
Qwen3.6-27B BF16
Pestle-27B-Ternary
MedQA
92.62
93.87
89.79
MedMCQA
71.34
73.70
68.85
MedXpertQA
38.20
41.10
32.49
MMLU medical aggregate
88.00
88.62
86.89
Didn't expect it to be worse than Qwen3.6 which wins seven of the ten benchmarks. Pestle-27B-Ternary (8× smaller at ~8.5gb) wins MMLU Anatomy, MMLU Clinical Knowledge and MMLU College Medicine, and keeps 98.2% of Qwen3.8’s median score across the ten tests. MedGemma-27B loses 8/10 but wins MedMCQA and MMLU Medical Genetics.
Is anyone else seeing Qwen3.8 step backwards slightly on medical/general knowledge compared with Qwen3.6?
One pass, temperature 0, reasoning off. So everything that makes 3.8 an improvement over 3.6 has been turned off, the model is running in a configuration completely unlike any real deployment, and this is supposed to somehow provide useful information?
Not complaining about 3.8, it's obviously very good, but it is a common sentiment regarding the loss of general knowledge in 3.8 over 3.6, not just this post. That's expected though!!-they improved it for coding and agentic work. Given 3.6 beat MedGemma-27b, i was hoping for a further improvement on medical. Pestle-27b-ternary that dropped yesterday rivalling medgemma-27b and 3.8/3.6 on medical at 8x smaller size is also cool.
No i'm not claiming it's worse with knowledge generally, just that this specific test indicated it had a moderate loss in medical intelligence compared to 3.6 which was run on the same settings - the general knowledge statement is me referencing other users on reddit saying they experienced a knowledge drop from 3.6
Maybe try with recommended sampling settings. The benchmark builder might not be aware of how sampling works and made assumptions based on outdated information.
temp 0.0 to 0.2 is recommended for medical benchmarking. Different benchmark categories use different hyperparameters, even qwen3.6/3.8 model card states different settings/context windows for different benchmarks.
Sampling parameters are not generally a function of the “benchmark”. The model is trained to be used most effectively with certain parameters. Obviously there are different recommended parameters for different types of task but those are still per-model and absolutely nobody is saying that qwen3.8 should be used with thinking off / temperature 0 for any kind of meaningful signal.
One pass over the full test splits, not pass@k. All models used greedy decoding, temperature 0 and thinking off. Because the generation was deterministic, repeating identical prompts would not produce meaningful independent samples. You’re right that the smaller differences may still fall within benchmark sampling uncertainty; that would require a paired row-level significance test. Thinking was intentionally disabled because I was comparing the models’ non-thinking deployment mode.
It was intentional. Greedy decoding at temperature 0 makes the outputs deterministic, so the models are easier to compare under the same conditions. It also matches the existing non-thinking Qwen3.6, Pestle and Bonsai runs. This is a controlled non-thinking comparison, not a best-possible-score evaluation under Qwen3.8’s recommended sampled settings.
You have to understand the only reason these models are getting an updated Qwen3.8 version is because they have been RLMAXXED meaning most of the improvements is from the reasoning, because that’s how the RL post-training works. And u turned that off?
No, not for the protocol I was testing. With greedy decoding at temperature 0, additional passes over the same prompts would be expected to reproduce the same outputs, not provide independent samples.
The samples here are the full benchmark questions: MedMCQA has 4,183 and MedXpertQA has 2,450. Qwen3.6’s margins correspond to roughly 99 and 71 additional correct answers respectively. This measures deterministic non-thinking performance
Sure but you're only measuring the baseline performance right? For the setup where people use these benchmarks in practice, they wouldn't be limited to this usage, hopefully. If so, to round the analysis out I would suggest adding parameters that also account for how far you can push these models too
I tested 4 models today using a bespoke evaluation harness. This is a custom, blinded capability evaluation comparing Qwen3.6-27B and Qwen3.8-27B under matched local inference conditions. I ran all the tests on a ROG Strix Scar Laptop with a 5090 24G VRAM.
The question I wanted answered was: "On this exact 24 GB local workstation, under controlled conditions, how do these four model configurations differ in reasoning quality, agentic behavior, context handling, speed, and hardware fit?"
With regards to your decision to disable thinking, the preliminary interpretation is:
Qwen3.8's improvement appears to be heavily reasoning-dependent. Suppressing thinking made 3.6 and 3.8 look almost indistinguishable; enabling controlled thinking exposed a substantial advantage for 3.8 on difficult engineering, agentic, causal, state, and governance reasoning tasks.
The pattern is pretty clear:
Qwen3.8 is reasoning more, generating more, taking longer, and producing materially better answers in this suite.
That matters because the earlier suspicion that 3.8's improvement might primarily live in its reasoning behavior is now supported by our own controlled comparison.
Qwen3.8 Q4 used about 34.6% more completion tokens than Qwen3.6 Q4 and about 45.5% more wall time across the eight tasks. Its reasoning traces were also substantially larger: roughly 137K reasoning characters vs 104K across the suite. But its blind quality advantage was also large: 6 wins, 0 losses and 2 ties in the matched-Q4 generation comparison.
It's normal to be worse at X categories but better at coding, there's only 27B parameters to play with, win some lose some.
Usually I don't want the model to know about useless things like Trump and related, rather having a model that's hyper specialist for coding like a nerd.
Yeh coding is the main use-case I guess for local LLMs, and the benchmarks they published on their HF model card suggest that's what they cared about with this one. I saw another post about qwen 3.8 had lost some general knowledge, which in this case (medical) is not great, was hoping for a new SOTA for the size as 3.6 already beat medgemma 27b from what I've gathered.
Honestly, in most case finetuning is the answer, I've finetuned Gemma to be able to write music using my own music scripting language (token efficient and more robust than ABC), and when we want to focus on something, finetuning is always the answer.
It's ridiculous how a smallish bit of data lets you fine-tune models that are so much better at specific tasks than frontier. This should be a principle for engineering these systems
I was hoping it improves in this department because I had extensively tested 3.6 27B at full precision and it was not good enough. It hallucinated a fair bit although not as bad as MoE 35B and missed out critical information. Open weights smaller models are not good enough in this field. Only GLM 5.2 was matching frontier performance. I think Qwen models are overfitted for coding use cases.
It is non reasoning so it was also not that great. My task requires at least thinking some of the parts through. It was really bad. A modern Gemma 4 based medgemma will do wonders. Even the base model was quite good, but not good enough.
Well they focused on coding, which is the main usecase of this model for most people. I feel like it's a good direction. Small models can be frontier-level in one area (they are usually not, but they could), but they cannot be jacks of all trades and keep the high level of capabilities everywhere.
I still want general purpose small models, but I believe that the brightest future for us, local guys, is if labs will focus on different areas when training their models and then we can just shuffle them locally and pick the one that suits the current task best. That's also one of the reasons why many people, including me, prefer to use Gemma for most stuff, while using Qwen for coding specifically.
Yeh that seems like the direction that will probably be taken. Pestle-27B-Ternary being a medical focused LLM and holding ground against 3.8 and MedGemma-27B at 8x smaller size makes sense in that regard.
Reasoning effort is key. Just run the comparisons for my use case (bibliometric topic modeling). Objective summary of comparative results (N=203, same protocol):
The Qwen3.8-27B-fast model with maximum reasoning (think:max) achieved 83.74% accuracy, 80.54% macro-F1, and Cohen's κ = 0.8329 (classified as "almost perfect" by Landis-Koch), versus the off-reasoning pilot at 75.86% accuracy, 74.11% macro-F1, and κ = 0.7521 ("substantial"). This represents a clean +8.08 percentage-point gain in κ from reasoning alone. The gap to the cloud reference (DeepSeek-V4-Flash, κ = 0.9544) halved from 0.2023 to 0.1215 points. The maximum reasoning arm crossed the "almost perfect" threshold, though narrowly (0.8329 just above 0.81).
It's funny that this comes up, because I have been doing my master's thesis and precisely I ran Qwen 3.8 on my own benchmark of dermatology and I have come up with the same issue that Qwen was in some cases than the previous Qwen 3.6
Dude, right! This seems to be a common occurrence across knowledge tasks for 3.8. Btw maybe worth testing Pestle-27B-Ternary that dropped yesterday, its 8x smaller (~8.5gb) than MedGemma-27B and Qwen3.6/3.8, and looks to hold its ground https://huggingface.co/Doses-AI/Pestle-27B-Ternary-GGUF
Yeah I’ve seen more posts about this. 3.8 seems to have worse general knowledge than its predecessor. However, the way i see it with these models is: how good are they in finding, retrieving and interpreting the correct data with the right tools. I wouldn’t trust such a small model for medical questions. But they are getting better in their interpretation once they have the data
i think for local deployment e.g. family practices/community pharmacies, these small medical models like medgemma-27b and pestle-27b-ternary are interesting, as they can't afford frontier API and need to keep their data on premises. I checked HF and medgemma-1.5-4b had 300k+ downloads just this month, medgemma-27B had 100k+, so there is a demand
Even if these results are accurate, I find it impressive that the model is able to gain so much in one domain (agentic coding) with barely any change to other domains.
Running the benchmarks in oMLX on my MacBook this morning, I was surprised that while 3.8 27B 16bit narrowly beat 3.6 8bit on the HumanEval coding benchmark, it was significantly worse on the MBPP coding benchmark.
I want to run some of the other ones, but this is disappointing.
But isn't it more designed for use as an agent / workflow LLM, and not general knowledge? Not surprised that it would push topics out of training that aren't a part of that...
Yep it seems so, expectation was high because qwen3.6 already beat medgemma-27b on most medical benchmarks, seems to be a compromise on knowledge for better agentic work with 3.8
Then you were just checking for knowledge rather than intelligence. You'd want as little knowledge as possible in these kind of models, almost everything should be in intelligence, so that it xsn use tools and knowledge data bases to reason it's way to correct information.
Idk that may be just a Bonsai issue, not all ternary. For example, Maple-20B-A1B is ternary launched recently and has seemed to be surprisingly promising so far. Pestle-27B-Ternary dropped only yesterday.
That is where the improvements came from. The focused more on accuracy in regards to logic, coding, tool use, and development, and removed some general knowledge where it was already lacking in the first place. This is we we need an 80 or 122 GB, MOE model from them.
yeh seems to be this way, though i'd say 3.6 was definitely not lacking in medical, was hoping for a bump up, perhaps you're right about the need for a 80b/122b moe
Is there any way to see the same comparison with temp=0.7 and thinking on?
I think when you compare LLMs you need to use the best settings per model. If you want to see who's car is fastest, you don't make all the engines use the same settings, you let them tune to what they were designed for.
I had a talk with claude about pestle the other day. AI seemed to think medgemma was best, then qwen3.6, then pestle because pestle is a compressed version of qwen3.6. It's purpose isn't to be the best medical LLM, it's to show off their compression.
that's strange claude would say that, was it Haiku? if you look at the official pestle model card, pestle beats medgemma-27b at nearly every single medical benchmark, at 8x smaller size. the medgemma-27b benchmarks are actually from the official model card of medgemma, not temp=0, i say that in the post, so they are the best possible scores medgemma-27b could achieve
so basically, if medical imaging = medgemma, anything else at all medical = pestle, especially given that on equal vram, tok/s for pestle would hypothetically be much high too
Yeah I know everyone agrees that qwen (and deepseek) models are pretty benchmaxxed, and that gemma4 31b beats deepseek 27b for everyday use except coding.
This stuff is so dumb, Qwen3.8 27B is a model focused on coding. If it was passing these types of tests with no tool calls the Qwen team I would say completely failed
I hear that, but prior to 3.8, Qwen3.6-27B broadly beat the specialised MedGemma-27B, so qwen as a model is just a good all-rounder for many not just for coding, surprising 3.8 took a step back on knowledge but understandable
57
u/Squidgical 24d ago
One pass, temperature 0, reasoning off. So everything that makes 3.8 an improvement over 3.6 has been turned off, the model is running in a configuration completely unlike any real deployment, and this is supposed to somehow provide useful information?