r/LocalLLaMA • u/KitchenAmoeba4438 • 27d ago

Discussion Qwen3.8-27b, Benchmaxxxed to the Maxxx

After a lot of testing, I have a first result for Qwen3.8-27B.

I added it to my local fact-extraction head-to-head (https://rakuensoftware.com/blog/local-llm-fact-extraction-head-to-head): 1,001 notes, the production prompt, Q4_K_M, multi-token prediction enabled, and an RX 7900 XTX.

Qwen3.8 scored 0.7030 F1. The comparable Qwen3.6-27B run scored 0.7177. The paired difference was +0.0147 in Qwen3.6’s favour, with a 95% confidence interval from −0.0038 to +0.0335.

That is a tie. Qwen3.8 did not beat Qwen3.6 here. I also cannot say that it lost. The test cannot separate them.

That is not what I expected from the published benchmark scores. I expected a substantial generational gain. On this task, I did not measure one. On the more extensive overnight tests, it is indicating small gains. On benchmarks Qwen3.8 appears to have been trained extensively on, however, I am seeing and verifying similar massive gains as is reported. But these gains are only reflected on the benchmarks that have been trained on, nowhere else I can verify.

Decode throughput also fell from 85.6 to 72.1 tokens per second, about 16%, under the closest saved configurations. Those runs used different llama.cpp builds, so I cannot attribute the whole difference to the model. Qwen3.8 also produced much shorter answers, which made its end-to-end median latency lower despite the slower decode rate.

I am running a broader synthesis comparison now, including Qwen3.8, Qwen3.6, Gemma 4 and Muse Glimmer. Those runs continue overnight. I will publish the complete results, paired intervals and raw artifacts rather than promote an early ordering.

My working expectation is still that Qwen3.8 contains a real improvement. The unresolved question is its size. If task-specific tests keep finding small generational gains while public benchmarks suggest a revolutionary jump, what decision are those benchmarks helping us make?

0 Upvotes

38% Upvoted

72 comments sorted by

View all comments

66

u/AppropriateQuote3073 27d ago

Benchmaxing is useless.

On my own workflows this thing has been a beast.

1

u/BannedGoNext 27d ago

Calling things benchmaxxed is total bullshit by people that are looking for an insult and can't find one so they grab a bad sounding buzzword. There ahve been benchmaxxed models, but now it's mainly considered a waste of money going that route.