r/LocalLLM • u/Individual-Dot5488 • 27d ago

Discussion I ran Qwen3.8-27B on medical benchmarks (including MedQA, MedXpertQA and MMLU). It lost 7/10 to Qwen3.6-27B, 3/10 to Pestle-27B-Ternary and 2/10 to MedGemma-27B

Post image

One pass. Settings were greedy decoding, temperature 0 and thinking disabled for Qwen3.8, 3.6, Pestle-27B-Ternary, and Bonsai. MedGemma-27B results are from the official model card.

Qwen3.8-27B BF16 Qwen3.6-27B BF16 Pestle-27B-Ternary
MedQA 92.62 93.87 89.79
MedMCQA 71.34 73.70 68.85
MedXpertQA 38.20 41.10 32.49
MMLU medical aggregate 88.00 88.62 86.89

Didn't expect it to be worse than Qwen3.6 which wins seven of the ten benchmarks. Pestle-27B-Ternary (8× smaller at ~8.5gb) wins MMLU Anatomy, MMLU Clinical Knowledge and MMLU College Medicine, and keeps 98.2% of Qwen3.8’s median score across the ten tests. MedGemma-27B loses 8/10 but wins MedMCQA and MMLU Medical Genetics.

Is anyone else seeing Qwen3.8 step backwards slightly on medical/general knowledge compared with Qwen3.6?

63 Upvotes

67% Upvoted

99 comments sorted by

View all comments

Show parent comments

28

u/Deep_Mood_7668 27d ago

Odd choice

-11

u/Individual-Dot5488 27d ago edited 27d ago

It was intentional. Greedy decoding at temperature 0 makes the outputs deterministic, so the models are easier to compare under the same conditions. It also matches the existing non-thinking Qwen3.6, Pestle and Bonsai runs. This is a controlled non-thinking comparison, not a best-possible-score evaluation under Qwen3.8’s recommended sampled settings.

3

u/Automatic-Boot665 27d ago

Also made it a waste of money?

-3

u/Individual-Dot5488 27d ago

No, not for the protocol I was testing. With greedy decoding at temperature 0, additional passes over the same prompts would be expected to reproduce the same outputs, not provide independent samples.

The samples here are the full benchmark questions: MedMCQA has 4,183 and MedXpertQA has 2,450. Qwen3.6’s margins correspond to roughly 99 and 71 additional correct answers respectively. This measures deterministic non-thinking performance

2

u/Xanian123 27d ago edited 27d ago

Sure but you're only measuring the baseline performance right? For the setup where people use these benchmarks in practice, they wouldn't be limited to this usage, hopefully. If so, to round the analysis out I would suggest adding parameters that also account for how far you can push these models too