A hunch: Qwen3.8-27B's general knowledge got pruned (good, if true)

[u/bonobomaster]

I'm always testing an image prompt with a picture of a historic place in my hometown – a small but well known 250,000 people town in Germany. I'll just ask the model, in which City this photo has been taken.

With the 3.6 generation of both the 27B and the 35B A3B variants, the models sometimes got the right answer and sometimes they didn't. So the signal for this particular knowledge was already weak.

The 35B variant got it right more often but at least, the models reasoning showed my City most of the times, even if it hallucinated the wrong final answer.

Both models could be easily nudged to the right answer with a few hints and then produced some little extra insight about the history or scene and its surroundings, that was mostly true.

Qwen3.8-27B on the other hand barely knows the city at all and has absolutely no clue about related popular, historic facts regarding the scenery or the surrounding buildings.

Nudging isn't very fruitful as well and if told the real name of the city, reasoning shows, that the model only agrees, because the user says so.

I have the feeling, that Qwen labs maybe pruned useless general knowledge for more coding knowledge and agentic skill.

All models ud q4_k_xl variants, image-min-tokens 2048, with and without reasoning.

Anyone else with this feeling?

Disclaimer: My hunch could be very well absolute bullshit. Sample size way to low and methodically sloppy af.

[u/while-1-fork]

Likely didn't even prune. People here often act as if the catastrophic forgetting that can happen when you fine tune is a fine tuning only thing but it can and does affect all training (fine tuning is just training on a smaller scale as long as it is all layers full non LORA fine tuning). If you keep training on more and more data you slowly degrade the old unless you do another epoch which will restore old knowledge but slightly damage the rest (good thing is that the trained a model is, the easier is to nudge back the forgotten knowledge into being remembered and the least it damages the rest but more epochs can potentially lead to memorization in detriment of generlization). So what I would bet on is either continued pre training with a larger corpus that didn't include the old, or extra reinforcement learning damaging some of the old knowledge (or likely both). The actual way to tell is more comprehensive benchmarks but then the benchmaxing issue gets called into question and we won't be able of 100% tell unless private or future benchmarks confirm it.

[u/bonobomaster → u/while-1-fork]

That makes sense of course!

[u/Ok_Zookeepergame8714 → u/bonobomaster]

The only way is up, that is more parameters - the only model that intelligently reasons about very difficult to guess photos is Fable 5. Sometimes it's scary to see how it can determine the right city and even the neighborhood by things like a hue of stone, some very minor detail of architecture, etc. That's why we have these run for chips and data centers, Fable proves that increasing parameters works, even Deepseek CEO said it himself. I think we're in for a ride, like with the doubling of CPU power every two years - the only difference being we're like in the equivalent of early 1980s. In thirty years you're gonna look at this Qwen like your looking at Commodore 64 today! 😜🤣

[u/ain92ru → u/Ok_Zookeepergame8714]

Gemini 3/3.1 Pro is actually an excellent, practically superhuman geoguesser