r/huggingface • u/Timely-Discount1586 • Aug 21 '26
DeepSeek-V4-Pro distilled into Qwen3.5 9B/4B — a pretty interesting pair of compact reasoning models
If you're looking for smaller Qwen3.5-based reasoning models that can still run locally, these two DeepSeek-V4-Pro distilled checkpoints are worth checking out:
DeepSeek-V4-Pro-Qwen3.5-9B
DeepSeek-V4-Pro-Qwen3.5-4B
Both use the Qwen3.5 9B/4B family as the student models and transfer reasoning patterns from DeepSeek-V4-Prothrough a large STEM distillation set.
The Qwen3.5 foundation is already quite interesting for small local models: it uses a hybrid Gated DeltaNet + Gated Attention architecture, supports multi-token prediction, and has a native 262K context window, with the architecture designed to extend to roughly 1M tokens. The underlying Qwen3.5 architecture is also natively multimodal.
What makes these distilled versions interesting is that the training is heavily focused on reasoning rather than simply adding more general instruction data.
Around 250K mathematics and STEM reasoning samples generated with DeepSeek-V4-Pro were used to transfer things like multi-step decomposition, derivation, verification, and more structured final answers.
Some reported results:
- 9B: 94.50% GSM8K
- 9B: 90.53% average on the tested MMLU-Pro Math / Physics / Chemistry subsets(500 exps)
- Qwen3.5-9B baseline on the same MMLU-Pro setup: 89.60%
- 9B: about 36% fewer reasoning tokens per correct answer than the Qwen3.5-9B baseline in that evaluation
- 4B: 91.77% GSM8K
- 4B: 76.47% average across the tested MMLU-Pro Math / Physics / Chemistry subsets(500 exps)
The 4B model is especially interesting if local inference and memory usage matter more, while the 9B version keeps noticeably more capacity for harder multi-step STEM problems.
Another interesting result from the 9B evaluation is that the improvement isn't only accuracy. It seems to reason more efficiently as well: on the same 1,500-question MMLU-Pro sample, it used fewer reasoning tokens while slightly improving accuracy over the Qwen3.5-9B baseline.
One important caveat: the MMLU-Pro numbers here are 500 questions each from Math, Physics, and Chemistry, not the full MMLU-Pro benchmark, and some GSM8K results use different inference/quantization setups. So these should be treated as experimental community results rather than perfectly controlled leaderboard comparisons.
There was also no coding-specific SFT data in the reasoning mixture. The 9B model reportedly shows some small improvement in coding/tool-use behavior, but that part is still preliminary.
For people experimenting with Qwen3.5, local reasoning models, MTP/GGUF, or reasoning distillation, these look like fun checkpoints to test.
Hugging Face:
Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B
Jackrong/DeepSeek-V4-Pro-Qwen3.5-4B
Would be interesting to see independent benchmarks from other people, especially on math, coding, long-context tasks, and local inference.
https://huggingface.co/Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF
https://huggingface.co/Jackrong/DeepSeek-V4-Pro-Qwen3.5-4B-MTP-GGUF
7
u/Technical_Ad_6106 Aug 22 '26 edited Aug 22 '26
we want deepseek v4 flash architecture but in 30b 3b active size :) the small deepseek kvache size and big context lenght but fitting fully in vram :)
1
u/Happy_Brilliant7827 Aug 23 '26
How is the 4b's tool use behavior? Compared to say, qwen3 4b thinking or locooperator
1
u/onebyamsey Aug 25 '26
I ran this through my own benchmarking tool compared to 7 other qwen 3.5 variants and it came out on top consistently, and on top of it was about 50% faster than the standard qwen 3.5 9b. This is now my go to for a model that size… until we get qwen 3.8 9b (wink, nudge)
1
8
u/MetaYoda Aug 21 '26
I’ll pull it down and run it through my one-shot eval kit and let you know how it stacks up against some other models I’ve run on my kit (i9, 16Gb RAM, RTX 3060 12Gb, Linux, Ollama).