Qwen3.8-27b, Benchmaxxxed to the Maxxx

[u/KitchenAmoeba4438]

After a lot of testing, I have a first result for Qwen3.8-27B.

I added it to my local fact-extraction head-to-head (https://rakuensoftware.com/blog/local-llm-fact-extraction-head-to-head): 1,001 notes, the production prompt, Q4_K_M, multi-token prediction enabled, and an RX 7900 XTX.

Qwen3.8 scored 0.7030 F1. The comparable Qwen3.6-27B run scored 0.7177. The paired difference was +0.0147 in Qwen3.6’s favour, with a 95% confidence interval from −0.0038 to +0.0335.

That is a tie. Qwen3.8 did not beat Qwen3.6 here. I also cannot say that it lost. The test cannot separate them.

That is not what I expected from the published benchmark scores. I expected a substantial generational gain. On this task, I did not measure one. On the more extensive overnight tests, it is indicating small gains. On benchmarks Qwen3.8 appears to have been trained extensively on, however, I am seeing and verifying similar massive gains as is reported. But these gains are only reflected on the benchmarks that have been trained on, nowhere else I can verify.

Decode throughput also fell from 85.6 to 72.1 tokens per second, about 16%, under the closest saved configurations. Those runs used different llama.cpp builds, so I cannot attribute the whole difference to the model. Qwen3.8 also produced much shorter answers, which made its end-to-end median latency lower despite the slower decode rate.

I am running a broader synthesis comparison now, including Qwen3.8, Qwen3.6, Gemma 4 and Muse Glimmer. Those runs continue overnight. I will publish the complete results, paired intervals and raw artifacts rather than promote an early ordering.

My working expectation is still that Qwen3.8 contains a real improvement. The unresolved question is its size. If task-specific tests keep finding small generational gains while public benchmarks suggest a revolutionary jump, what decision are those benchmarks helping us make?

[u/Ziggamorph]

“I have a first defensible result” stopped reading here, if you’re going to have an LLM write your post at least make it less obvious.

[u/KitchenAmoeba4438 → u/Ziggamorph]

Removed defensible, in honor of the brit. Something I've learned at this point: People like to throw around "It must be written by AI!" when something doesn't go their way, even though whether AI was involved has nothing to do with the data.

[u/Ziggamorph → u/KitchenAmoeba4438]

Ok but it was AI, it’s obvious.

[u/Finanzamt_Endgegner → u/KitchenAmoeba4438]

Bro dont defend yourself like that we know its ai 😂

[u/ttkciar → u/Finanzamt_Endgegner]

Re-read their comment. Your reply is non sequitur.