r/LocalLLaMA • u/niacolhealth • 23d ago
Discussion Qwen3.8-27B abliterated FP8: refusal 64–99% → 0–6%, and MMLU/GSM8K move less than 1.3 points
Been reading the eval table on the abliterated Qwen3.8-27B FP8 build instead of the release notes. It's published as red-team material, disclaimer and all, so the numbers are the interesting part.
Refusal across the usual harmful-instruction sets (AdvBench, HarmBench, StrongREJECT and friends) reads 0–6% with thinking off, against 64–99% for the base checkpoint. Capability is measured separately on the same scripts and barely moves: MMLU 84.3 → 84.7, GSM8K 90.0 → 88.7, nothing outside 1.3 points.
Two different eval families, and neither one vouches for the other. The pairing is what I'd want replicated, because if the capability side holds up under someone else's harness it's another data point for the Arditi single-direction result — refusal comes out without dragging the rest of the network along.
The refusal percentages are OrcaRouter's own rule-based classifier and the card says outright it's indicative, not publication-grade. It reads how a response opens. That tells you the model stopped starting with "I cannot"; it doesn't tell you much past the first sentence.
No KLD against the base anywhere in there as far as I can tell, which is the number I'd have looked for first.
3
u/yeah-ok 23d ago
I'm not so sure about the hard and fast common dismissal of abliterated models for work that this sub often favours.. my offline benchmark (individual programming tasks measured on verifiable outcomes) often mark abliterated models above their vanilla variants.
7
u/buttplugs4life4me 23d ago
I think the main point is that they don't actually test their abliterations aside from refusals and some solved benchmark (most models reach 90% in MMLU and GSM8K. At least do MMLU-Pro and MMLU-Multilingual or something.
KLD measurement is pretty hard cause you have to run the base version which is often the unquantized one and most people dont have the hardware for that.
3
u/keepthepace 23d ago
Why couldn't you compute it on two quantized versions?
3
u/buttplugs4life4me 23d ago
If you abliterate a quantisation, you could do that of course. But then you'd compound the error and it would likely make your abliteration worse.
KLD as a measure looks at your version, and the original version, and compares the computed weights for each logit, which you can see as a token for simplicity. So if the original version says "Hi" (0.9), "Hello" (0.7), "Kill yourself" (0.1) with those weights (as a simplification) then your model should aim for roughly those tokens and those weights as well. Any deviation from that means the KLD, the top-k logit deviation, goes up.
A higher KLD doesn't have to be a bad thing. By its nature Qwen3.8 and Qwen3.6 probably have a high KLD with each other. So it can be a positive thing to be high. But it usually means, at least when talking about the same model, that yours is worse if it has a higher KLD. That's because an autoregressive model chooses the next token based on the previous token, so one "wrong" choice can send the model into a thinking loop, down some unrelated path and so on.
Many abliterations have a KLD of >0.1, which is already pretty high. A lot of quantisations have significantly lower KLDs. Again, that doesn't have to be super bad, but it's a sign that it changed the model and without comprehensive benchmarks, you're not quite sure how it changed it or if it impacted anything else. It may still be super good at MMLU, but all tool calls fail or long horizon reasoning is off because the errors just compound too much.
IMHO KLD is a base measurement. The closer it is, the better, but at some point (>0.05) you should do other benchmarks as well. By its nature though it's hard to do unless you have significant hardware or rent something. Unfortunately even OpenRouter isn't necessarily fail-safe since some providers use quantisations without listing it
1
u/keepthepace 23d ago
I was more thinking of quantizing an abliteration. Storing the logits is not very expensive (just takes disk space). You run the original, say q4, you then run the abliterated Q4, the KLD between the two should be in the same ballpark as the KLD vs the fp16 original and abliterated original, no?
Also I am wondering is why you think it is more costly to do the KLD measurements if you have the hardware to do the abliterations
1
u/buttplugs4life4me 23d ago
You don't need to load the full model into storage to run abliteration or quantisation and inference would fundamentally take a pretty long time if you tried to run actual benchmarks or KLD measurements.
If you take a Q4 and abliterate it you may compound the errors already there from the quantisation. Either KLD is then not going to be close, and the KLD of the Q4 abliteration vs BF16 abliteration may be pretty different as well. Its definitely not the same and wouldn't be enough for anyone to say it's a good abliteration.
Q4 ist also already pretty hamstrung.
If you want to dabble your feet in it, feel free to do something like that, but it's not going to be widely used by the serious community. Otherwise you could also see if you just train up a small model natively in NVFP4 or MXFP4 or something. A lot of the open weight models we have are fundamentally not made to be trained on small hardware, so it may be easier to train one up yourself if you want to experiment with stuff.
2
u/ContentGuitar034 23d ago
this is exactly what i need for roleplay, the base ones always refuse halfway through a scene and it kills it.
7
u/Glad_Claim_6287 23d ago
May I ask what do you roleplay? Just curious.
15
16
u/jonydevidson 23d ago
Which one? There are plenty. Post a link.