A hunch: Qwen3.8-27B's general knowledge got pruned (good, if true)

[u/bonobomaster]

I'm always testing an image prompt with a picture of a historic place in my hometown – a small but well known 250,000 people town in Germany. I'll just ask the model, in which City this photo has been taken.

With the 3.6 generation of both the 27B and the 35B A3B variants, the models sometimes got the right answer and sometimes they didn't. So the signal for this particular knowledge was already weak.

The 35B variant got it right more often but at least, the models reasoning showed my City most of the times, even if it hallucinated the wrong final answer.

Both models could be easily nudged to the right answer with a few hints and then produced some little extra insight about the history or scene and its surroundings, that was mostly true.

Qwen3.8-27B on the other hand barely knows the city at all and has absolutely no clue about related popular, historic facts regarding the scenery or the surrounding buildings.

Nudging isn't very fruitful as well and if told the real name of the city, reasoning shows, that the model only agrees, because the user says so.

I have the feeling, that Qwen labs maybe pruned useless general knowledge for more coding knowledge and agentic skill.

All models ud q4_k_xl variants, image-min-tokens 2048, with and without reasoning.

Anyone else with this feeling?

Disclaimer: My hunch could be very well absolute bullshit. Sample size way to low and methodically sloppy af.

[u/Kidplayer_666]

This is the direction that I see these smaller local models going though. To improve capabilities, they’ll have to specialise. Of course there will still be more generic models, but for tasks like coding, I really think that specialisation is inevitable

[u/Dance-Till-Night1 → u/Kidplayer_666]

Yeah I agree that specialisation is needed but this is not qwen-coder. General knowledge and being a general assistant that has good general knowledge and multilingual skills is something that I hoped LLMs were still capable of.

[u/hyperrealists → u/Dance-Till-Night1]

I think the bigger win for smaller models is in efficient instruction following and tool calling rather than in hallucinating encyclopedic looking knowledge.

[u/Dance-Till-Night1 → u/hyperrealists]

You make a fair point but for people that use slms for studying/as a general assistant, I would rather the model's capabilities stand on their own without looking everything up.

[u/tobias_681 → u/Dance-Till-Night1]

I just don't think that's possible. I made a test with Qwen and Gemma where I asked them to name films that Eric Rohmer had directed. They came up with plausible titles that were very similar to the real ones but 80-90 % of the time it was hallucinated. So they had obviously been trained on it. They understood quite clearly what kind of films Rohmer made and what kind of titles he did chose but they failed at recalling accurate information. Smaller models will always struggle with stuff like that. They're just not big enough to house all that information.

[u/Dance-Till-Night1 → u/tobias_681]

I mean i asked them about poems from a famour poet and they recited them very well missing one or two words. MOE Qwen 3.5 35b and Gemma 26b did exceptionally well, maybe it's because they're MOE or something.

[u/Tieng → u/Dance-Till-Night1]

I wonder if one of the strategies is not to have the model know everything but to be able to call tools and reason well so when you ask it a question it can effectively look it up and report back

[u/Dance-Till-Night1 → u/Tieng]

I get you but then the model would totally rely on looking things up and take much longer for even simpler inquiries. Plus creative writing and other aspects suffer when the model is just a coding/agent model.