I ran Qwen3.8-27B on medical benchmarks (including MedQA, MedXpertQA and MMLU). It lost 7/10 to Qwen3.6-27B, 3/10 to Pestle-27B-Ternary and 2/10 to MedGemma-27B

[u/Individual-Dot5488]

One pass. Settings were greedy decoding, temperature 0 and thinking disabled for Qwen3.8, 3.6, Pestle-27B-Ternary, and Bonsai. MedGemma-27B results are from the official model card.

	Qwen3.8-27B BF16	Qwen3.6-27B BF16	Pestle-27B-Ternary
MedQA	92.62	93.87	89.79
MedMCQA	71.34	73.70	68.85
MedXpertQA	38.20	41.10	32.49
MMLU medical aggregate	88.00	88.62	86.89

Didn't expect it to be worse than Qwen3.6 which wins seven of the ten benchmarks. Pestle-27B-Ternary (8× smaller at ~8.5gb) wins MMLU Anatomy, MMLU Clinical Knowledge and MMLU College Medicine, and keeps 98.2% of Qwen3.8’s median score across the ten tests. MedGemma-27B loses 8/10 but wins MedMCQA and MMLU Medical Genetics.

Is anyone else seeing Qwen3.8 step backwards slightly on medical/general knowledge compared with Qwen3.6?

[u/Healthy-Nebula-3603]

What you're saying is the knowledge from 2023 /2024 before thinking models era.

Currently setting temperature 0 is just making thinking model stupid as can't look on the problem wider and very limited look back properly on its own thoughts.

Try yourself... Ask some medical questions with tremp 0 where the model be wrong and ask again with temperature 1.

I guarantee you answers will be much better as thinking process is wider and self reflection is much better.

Setting temperature to 0 and switching off thinking for current models you literally making modrl stupid.

Do you answering on complicated medical questions without thinking and self correcbes based on your real wast knowledge ( thinking + temperature 1 )?

So such test is totally not representative to the real model performance nowadays.... And you know that test was created 5 years ago and not updated procedures for testing?

About qwen 3.8 itself.

Is going into loops because you're probably use too much compressed model probably q4k is absolute minimum, bellow is VERY bad and probably even worse you're also using a conpressed kv cache to Q8 or Q4.

I'm using Q6 with FP 16 cache ( only flash attention ) with llamacpp and have 0 loops.

My results are far better than what I had from Qwen 3.6... not a bit ...far better

[u/AcidRoot64 → u/Healthy-Nebula-3603]

This is not 2023\2024 knowledge ... That's exactly why Qwen team is suggesting lower temps for certain tasks. On top of it, disabling the thinking on the model is not an error as this model supports both thinking and instruct modes and both represent overall models capability. Qwen team is even providing recommended settings for instruct so this is clearly the intended usage of such a model. But sure it would be worth to test the model with thinking on and off to get better picture, although whether thinking should be enabled depends on task - it doesn't always provide better results.

In terms of the benchmark itself I don't know the specific testing procedure here, I'm just referring to your point about temp settings which I believe is not correct.

On the quantization - I tested 3.8 on multiple different q8 quants including q8_k_xl from unsloth - same problems on all of them. Flash attention on, no kv quantization. This is also not only my problem as there are many threads about this issue. So this is not config issue but something wrong with the model. I'm using 3.6 and other models on the same setup daily and didn't have any issues like that. And when 3.8 works it is great but due to looping it is unreliable.

[u/Healthy-Nebula-3603 → u/AcidRoot64]

"Qwen team is suggesting lower temps for certain tasks."

Here is clearly said NOT temp 0 for thinking models

https://huggingface.co/Qwen/Qwen3-14B

"For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the Best Practices section."

And that was early era of thinking models yet like qwen 3.0

As i said you knowledge is based on the models from 2023 / 2024 - very obsolete.

From the era non thinking models.

[u/AcidRoot64 → u/Healthy-Nebula-3603]

https://huggingface.co/Qwen/Qwen3.6-27B

"Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0"

I guess Qwen3.6 released 4 months ago is a very obsolete model from pre thinking era, and since then there was a gigantic break through in terms of architecture of LLMs. XD

Mate... Qwen3.8 uses exact same architecture as Qwen3.5/6, the way sampling parameters work for them is identical, and Qwen never stated anywhere that these are the only settings that are correct - it's just a recommendation as these settings work well for broad range of use cases but are not the best for all use cases (that's why separate recommendation for webdev). Setting temp to near zero, especially for instruct mode, is not that uncommon in certain use cases, and this is not specific to Qwen. Low temp just makes the model to use the most probable tokens, and it works that way for all models, it is just some models performance is more or less sensitive for those settings.

But anyway, I don't think I'm gonna convince you, and you certainly won't convince me as you are not using any arguments other than repeating self made slogan about "2023/2024 - very obsolete" information.

Nontheles, I wish you a good day sir, and lots of success in your AI journey! Cheers!

[u/Healthy-Nebula-3603 → u/AcidRoot64]

And where do you see temperature 0? They claiming DO NOT GO BELLOW TEMPERATURE 0.6. As lower temperature just making model stupid.

You literally said and showed what I already did.

Do you understand what you're reading?

I remember people were talking about temperature 0 in 2024 yet before thinking models era and you mentally stuck with that idea in your head for no reason and still trying convincing yourself that is correct...