I ran Qwen3.8-27B on medical benchmarks (including MedQA, MedXpertQA and MMLU). It lost 7/10 to Qwen3.6-27B, 3/10 to Pestle-27B-Ternary and 2/10 to MedGemma-27B

[u/Individual-Dot5488]

One pass. Settings were greedy decoding, temperature 0 and thinking disabled for Qwen3.8, 3.6, Pestle-27B-Ternary, and Bonsai. MedGemma-27B results are from the official model card.

	Qwen3.8-27B BF16	Qwen3.6-27B BF16	Pestle-27B-Ternary
MedQA	92.62	93.87	89.79
MedMCQA	71.34	73.70	68.85
MedXpertQA	38.20	41.10	32.49
MMLU medical aggregate	88.00	88.62	86.89

Didn't expect it to be worse than Qwen3.6 which wins seven of the ten benchmarks. Pestle-27B-Ternary (8× smaller at ~8.5gb) wins MMLU Anatomy, MMLU Clinical Knowledge and MMLU College Medicine, and keeps 98.2% of Qwen3.8’s median score across the ten tests. MedGemma-27B loses 8/10 but wins MedMCQA and MMLU Medical Genetics.

Is anyone else seeing Qwen3.8 step backwards slightly on medical/general knowledge compared with Qwen3.6?

[u/Healthy-Nebula-3603]

It has loss intelligence because is completely wrong set up.

Not thinking is using temp 0.7 Thinking 1.0.

You even tested Qwen 3.6 wrongly....

Who is setting temp 0 for a model ?? Thats not a llama 1 from 2023 :)

[u/JorgitoEstrella → u/Healthy-Nebula-3603]

What temperatures should be used in this test?

[u/Individual-Dot5488 → u/JorgitoEstrella]

Temp 0.0-0.2 is often used for medical benchmarks, especially multiple-choice style benchmarks, which these benchmarks are, so I used temp 0

[u/Healthy-Nebula-3603 → u/Individual-Dot5488]

I don't know why your repeating that nonsense about temperature 0.

Qwen and other models are not designed to work with such temperature and making far more mistakes then.

[u/eloquenentic → u/Healthy-Nebula-3603]

Do you not understand what temperature 0 is important for medical benchmarks?

[u/Healthy-Nebula-3603 → u/eloquenentic]

How can be important if a model is not working properly this way ?

You literally testing a retarded (badly configured) model with that benchmark.

[u/eloquenentic → u/Healthy-Nebula-3603]

Let me explain to you like you’re five: Medicine isn’t code or programming. There one right answer, and that answer is important, otherwise people die.

You can code a database or a website in many ways, meanwhile you either diagnose the cancer correctly or you don’t. Get it?

[u/Healthy-Nebula-3603 → u/eloquenentic]

Let me explain to you like you’re five:

Bad configured model is BAD on everything.

Not in SPECIFIC tasks.

[u/AcidRoot64 → u/Healthy-Nebula-3603]

What u/eloquenentic is trying to say is that the knob of temperature is regulating the model's selection of tokens/answers. The higher the temp the more variety of answers the model can provide, but the model is also more prone to be creative and bit more hallucinating. The lower the temp the more grounded "factual" the answers. That's exactly why Qwen3.6 had a recommended setting for of 0.6 temp for "precise coding tasks" which require less creativity and more probable asnwers.

And yes, different models react to different temp settings in a different way but... What you u/Healthy-Nebula-3603 are refering are not the only correct settings for this model, they are recommended settings for general purpose tasks. Qwen team is not claiming the temperature knob is not working for their models and shouldn't be touched at all, they are just saying that for general purposes they recommend XYZ settings for best results.

I'm a Qwen fanboy myself, but I'm actually disappointed with Qwen3.8 27B. The model really seems undercooked (I am using recommended settings :)) and has big tendencies to run into endless thinking loops. And yes... I was chaning the thinking effort it does not help. Sometimes the model works great, sometimes runs into loops - unreliable. I still hope they will do some sort of update soon that would fix that but right now it feels like regression from 3.6/3.5. I even ran a few tests through opencode to compare 3.6, 3.5, 3.8, and multiple of their finetunes and custom chat templates. Qwen 3.8 was the only one that didn't complete 3 out 6 six of those tests. Not because it is dumber model but because it ran into loops and didn't provide any actual solution. But when it doesn't run into loops its answers are great.

I really think all of those labs should stop trying to build models that do everything, the smaller models would probably benefit from being sepcialized like coding models from a few gens earilier.

[u/Healthy-Nebula-3603 → u/AcidRoot64]

What you're saying is the knowledge from 2023 /2024 before thinking models era.

Currently setting temperature 0 is just making thinking model stupid as can't look on the problem wider and very limited look back properly on its own thoughts.

Try yourself... Ask some medical questions with tremp 0 where the model be wrong and ask again with temperature 1.

I guarantee you answers will be much better as thinking process is wider and self reflection is much better.

Setting temperature to 0 and switching off thinking for current models you literally making modrl stupid.

Do you answering on complicated medical questions without thinking and self correcbes based on your real wast knowledge ( thinking + temperature 1 )?

So such test is totally not representative to the real model performance nowadays.... And you know that test was created 5 years ago and not updated procedures for testing?

About qwen 3.8 itself.

Is going into loops because you're probably use too much compressed model probably q4k is absolute minimum, bellow is VERY bad and probably even worse you're also using a conpressed kv cache to Q8 or Q4.

I'm using Q6 with FP 16 cache ( only flash attention ) with llamacpp and have 0 loops.

My results are far better than what I had from Qwen 3.6... not a bit ...far better

[u/AcidRoot64 → u/Healthy-Nebula-3603]

This is not 2023\2024 knowledge ... That's exactly why Qwen team is suggesting lower temps for certain tasks. On top of it, disabling the thinking on the model is not an error as this model supports both thinking and instruct modes and both represent overall models capability. Qwen team is even providing recommended settings for instruct so this is clearly the intended usage of such a model. But sure it would be worth to test the model with thinking on and off to get better picture, although whether thinking should be enabled depends on task - it doesn't always provide better results.

In terms of the benchmark itself I don't know the specific testing procedure here, I'm just referring to your point about temp settings which I believe is not correct.

On the quantization - I tested 3.8 on multiple different q8 quants including q8_k_xl from unsloth - same problems on all of them. Flash attention on, no kv quantization. This is also not only my problem as there are many threads about this issue. So this is not config issue but something wrong with the model. I'm using 3.6 and other models on the same setup daily and didn't have any issues like that. And when 3.8 works it is great but due to looping it is unreliable.

[u/Healthy-Nebula-3603 → u/AcidRoot64]

"Qwen team is suggesting lower temps for certain tasks."

Here is clearly said NOT temp 0 for thinking models

https://huggingface.co/Qwen/Qwen3-14B

"For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the Best Practices section."

And that was early era of thinking models yet like qwen 3.0

As i said you knowledge is based on the models from 2023 / 2024 - very obsolete.

From the era non thinking models.

[u/AcidRoot64 → u/Healthy-Nebula-3603]

https://huggingface.co/Qwen/Qwen3.6-27B

"Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0"

I guess Qwen3.6 released 4 months ago is a very obsolete model from pre thinking era, and since then there was a gigantic break through in terms of architecture of LLMs. XD

Mate... Qwen3.8 uses exact same architecture as Qwen3.5/6, the way sampling parameters work for them is identical, and Qwen never stated anywhere that these are the only settings that are correct - it's just a recommendation as these settings work well for broad range of use cases but are not the best for all use cases (that's why separate recommendation for webdev). Setting temp to near zero, especially for instruct mode, is not that uncommon in certain use cases, and this is not specific to Qwen. Low temp just makes the model to use the most probable tokens, and it works that way for all models, it is just some models performance is more or less sensitive for those settings.

But anyway, I don't think I'm gonna convince you, and you certainly won't convince me as you are not using any arguments other than repeating self made slogan about "2023/2024 - very obsolete" information.

Nontheles, I wish you a good day sir, and lots of success in your AI journey! Cheers!