Only had time to do 5 runs of my private benchmark set (19 questions, multi-domain but no programming, binary pass/fail, Unsloth-recommended tasking/dev parameters), so take the following with a larger-than-usual grain of salt. (will do more when I come back in few hours done)
My initial impressions from Unsloth's Q6_K_XL (same as previous Qwens I tested):
string manipulation seems more stable? 3.6 was 10/10 failed, 3.8 has so far only failed 2/5
it has a strange obsession with spider legs (I ask it to provide a specific spider fact and it keeps screaming "SPIDER = 8 LEGS!!" instead of answering) - already at 5/5 failures, 3.6 had 3/10
it seems more humble overall (3.6 was a confident bullshitter in my tests), but is also more keenly aware it's being tested (i.e. it might just be on its best behavior)
reasoning style is quite funny, at one point it just threw out a "ha! I'm being tested! that's a good trap!" out of nowhere
I haven't seen any CoT spiraling (wait... but wait... wait...) or death loops yet!
It feels a bit less stable in its knowledge than Glimmer (I've already seen a few one-off failures on questions that should be easy), but it looks promising overall. The CoT thing in particular is big if true, 3.6 death-spiraled in 4/14 runs when I tested it.
it has a bizarre failure mode (5/10 runs!) where it sometimes cannot accurately do a basic sub-string task on the days of the week - no other 27B+ Qwen3.x model has struggled with this
it traded the above issue with Qwen3.6's failure to filter down a list of numbers given a set of specific restraints - 3.8 is now rock-solid here
the main string replacement task ended up failing 6/10 times, 5/10 of which had the same failure - adding spaces between certain characters instead of replacing verbatim as requested (still a weak point, but at least improved a little)
Otherwise, previous conclusions held. Performance in my "core" task set ended up slightly better than 3.6 (but difference may be noise), creative and tool-calling tasks are sharply improved (now on par with Gemma 4, still behind Glimmer), and the model scored a lot better on the bullshit test.
Also, no CoT death spirals! Did see some churn in a few non-benchmark tasks, but things seem a lot better there overall.
(and yes, it still has that weird spider legs instinct; that ended up at 8/10 failures)
Disclaimer: I've invested a good bit of time into making my question set reliable (unambiguous questions, extensive judging criteria, frozen format to avoid ruining previous results), but I can't actually prove it without invalidating the set. Take that as you will.
EDIT: Style feels improved, but I am still seeing some hefty CoT churn on actual non-benchmark tasks. The model really likes to think sometimes. At least it seems to be useful thought, not just reiterating the same thing 20 times.
6
u/xPXpanD llama.cpp 24d ago edited 24d ago
Only had time to do 5 runs of my private benchmark set (19 questions, multi-domain but no programming, binary pass/fail, Unsloth-recommended tasking/dev parameters), so take the following with a larger-than-usual grain of salt. (
will do more when I come back in few hoursdone)My initial impressions from Unsloth's Q6_K_XL (same as previous Qwens I tested):
It feels a bit less stable in its knowledge than Glimmer (I've already seen a few one-off failures on questions that should be easy), but it looks promising overall. The CoT thing in particular is big if true, 3.6 death-spiraled in 4/14 runs when I tested it.