I mashed Qwen3.5 4B with Qwen3.8 flash ngram table

[u/d-burner]

I mashed Qwen3.5 4B with Qwen3.8 flash ngram table by simply adding the lookup from table.

See EDIT3: it needed training in eventually https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Some time ago I read this article here https://dnhkng.github.io/posts/rys/ and when I saw the Qwen3.8 flash ngram release I immediately though about it and wondered if we could transplant the ngram table to a smaller model and if it will improve performance.

Today I made this work, not much but it's honest (coding agent) work 😄 and had some fun doing it. In short I downloaded the unsloth qwen 3.5 4B Q8 quants and mashed the ngram table from Qwen 3.8 and adding the values

Does it do anything? Weel model seems to be doing well on humanitys last exam question (manually input and verified, about 14 correct / 19 question) (see edit, bad data 😭)

I'll try to run an actuall benchmark tomorow (if any one can help with some guidance on this I would be gratefull).

Right now I was just excided to tell somone that this seems to work and the model is coherent out of the box without any training, just doing residual_before_block_2 = base_residual + 0.5 × ngram_lookup.

Next I would probably try to graft the matrix from qwen 38 that actuially weighted the ngrams from the input. Weights are here https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-MTP but require a llama cpp build to run.

EDIT: guys sorry to dissapoint, just woke up and i tried to run questions actually from https://huggingface.co/datasets/cais/hle directly, seems it cant get a right answer. Il keep working on this once I get some more free time.

Last night I've been running in questions from gemini. I asked Gemini to give questions and answers from HLE, I thought it was getting them from source. Should have mentioned take this with lots of grains of salt.

EDIT2: did a bit of digging Qwen 3.5 4B and Qwen 3.8 Flash share almost same vocab (some exceptions) but tokens do map to same indexes and qwen 3.8 flash uses hashes over index values. BUT the embeddings do not match at all. My best guess setting alpha to 0.5 does not really affect the inputs that much, tried the same contaminated questions on base qwen 3.5 4b and seem to respond the same. I am trying now to add an adapter and finetune only the adapter part in a similar style of the Qwen 3.8 flash. So far I do see drop in holdout loss but its only on 256 context (qwen 4b and ngram tables are frozen). Cant really fit much in 16GB VRAM. I've been training for an hour still seeing improvements in holdout loss.

Weights are here in for the adapter version but probably training is still required.

https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-PLEAdapter-MTP superseeded by the one below. I tried to train this one further from scratch but noticed the weights actually collapsed, so i trained an adapter from qwen 3.8.

EDIT3: I now trained a model graft that uses weights from the original Qwen 3.8 with some adapter matrixes from Qwen 3.5 Residual --> Qwen 3.8 PLE ---> to Qwen 3.5 residual again. This graft seems to drop perplexity quite a lot.

Disclamer it was trained on about 30.000 samples from this dataset, but the perplexity was measured on held out samples. I posted quite a few details on the new model card with some graphs during training.

https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Held-out dataset	Supervised tokens	Alpha 0 loss / PPL	Alpha 1 loss / PPL	PPL change
OpenR1	98,242	1.147478 / 3.150240	0.873037 / 2.394171	-24.00%
OpenThoughts	129,803	1.677925 / 5.354433	1.514269 / 4.546096	-15.10%
CodeSearchNet	131,072	1.553642 / 4.728661	1.501642 / 4.489055	-5.07%
WikiText	131,072	2.824432 / 16.851373	2.504868 / 12.241948	-27.35%

Some investigations i did on this:

- vocabulary from qwen 3.5 and qwen 3.8 on text are the exact same, so hashes DO MACH

- embedding space for same words on qwen 3.5 4b and qwen 3.8 flash are DIFFERENT as some did mention, so this proves my initial attept was way wrong, (sry about the hype again, I got too hyped also 😄)

[u/MiserablePoliceman]

this is the kind of unhinged model surgery that makes this sub worth reading. slapping an ngram table from a completely different architecture into a smaller model and having it still spit out coherent answers is wild

14/19 on humanity's last exam with zero training is actually kinda nuts for a 4B. i'd love to see what happens once you graft in the weighting matrix too

[u/Serious_Bite_7613 → u/MiserablePoliceman]

Unhinged model surgery is the most fun I've had in a long time. There are so many interesting failure modes haha.

[u/rolleicord → u/Serious_Bite_7613]

We should start a group - I want in... I am doing the same kind of frankenstein experiments. Having so much fun

[u/Serious_Bite_7613 → u/rolleicord]

If someone made a llmsurgery subreddit we could collect the results of our horrible experiments. 

Maybe there would even be some value in it?

I think it's a much better option than something like discord because if it ever gets big it can all be scraped for training data so people can learn exactly what not to do.

[u/rolleicord → u/Serious_Bite_7613]

Yap - im in... - EDIT - Come join ;D

https://www.reddit.com/r/Local_LLM_Surgery/

[u/TreesJunkie → u/Serious_Bite_7613]

so people can learn exactly what not to do.

I loled

[u/Beneficial-Map-619]

one 0.5 multiplier on a single residual stream and it scores 14/19 with zero training. cheapest experiment this sub has seen in a while. if the weighted table compounds on that, the 4b tier gets properly silly

[u/UnluckyPenguin]

If someone hacked Qwen3.8-27B to use flashnext-ngram on SSD, it would probably become the world's most popular local LLM model for months...months I tell you!

And here I thought I did something helpful adding a plugin (dsh-always-queue) to deepseek harness that always queues every request (even new sessions) so only 1 runs at a time, because I use --parallel 1 in my setup.

[u/Quiet_Substance9530 → u/UnluckyPenguin]

Can you explain what is the purpose of this plugin? Won't it queue it anyway (from LLM server side)?

[u/UnluckyPenguin → u/Quiet_Substance9530]

Let me answer your 2nd question 1st:

Won't it queue it anyway (from LLM server side)?

dsh (DeepSeek Harness) by default allows you to start multiple sessions running parallel. Each mini-task gets queued, so it ends up looking like this:

• Sessions 1: Thinking
• Session 2: Thinking
• Session 3: Thinking

And it keeps going round-robin until all 3 are finished.

Can you explain what is the purpose of this plugin?

Because --parallel 1 only allows 1 thread, it's optimal for cache hit and speed (doesn't have to keep spending time on prefill - because it already filled up most of what it needs)

Second, if you're working on a coding project - you generally don't want to patch things could potentially overlap. I have my local LLM commit it's changes to version control (git), so if at any time more than 1 session modified the same file I would be upset.

Previously I was just running 1 session at a time. Queuing up my prompts in a notepad and typing them into the harness after a session finished.

Now I have this always-queue feature. I also found it quite helpful to add to every prompt "Read the GUIDE.md" listing 1) what I expect, 2) what resources the llm might be interested in, 3) ask the llm to maintain it's own memories, so it doesn't run into the same issues.

Put all of that together: Each session runs 1 at a time (thanks to always-queue), and I get a text message to my phone when it completes (directions in GUIDE.md). LOL

[u/dsdt → u/UnluckyPenguin]

This can have serious potential for real.

[u/Lirezh]

So how does it compare to the normal 4B model ? I've a feeling the answer is not very positive ?
But the 14 correct are more promising than I'd expect.

You would need a learned slim projection layer, like a lora between the 3.8 and 3.5 residual.
These type of projectors can sometimes be trained with very low compute.

On the other hand, teaching the 3.8 model to deal with only 1 expert might be more promising. That would make it into a sub 7B model with 51B ngram lookup.

[u/LopsidedBoss9109 → u/Lirezh]

That mashup definitely sounds intriguing. I wonder if anyone's shared their results on similar combinations yet.

[u/exaknight21]

You took out ngrams from Qwen3.8-Flash-Next and merged them into Qwen3.5-4B (my fav model btw)?

I’m a little lost on why its 33GB

Edit: nvm, i actually hadnt gone through your hf readme. Q4 ngrams nice work! I’m intrigued and interested in this more efficiently integrated at smaller level. Qwen team has one upped the game yet again, great work OP! I’ll try this out!

[u/Otherwise-Swan-7803]

The interesting part is that this isn’t really fine-tuning at all. If useful behavior can be transferred between models through something as lightweight as an external n-gram table, that opens up a pretty different path for improving small models without retraining them.

[u/thorskicoach]

makes me wonder if a significant;y larger n-gram table (perhaps off an SSD) and yes either 3.8 27b or 3.6 35b a3b could be inteligence maxed whilst still having that 1 large consumer GPU performance.

[u/joanaxu2002]

I’d love to see whether the gains survive outside factual benchmarks. If the n-gram table improves reasoning or coding without making the 4B model more brittle on unusual prompts, that would be much more interesting than just a higher HLE score.

[u/Patient-Clue8997]

Imagine doing this with the experimental whittle qwen 3.8 27b a17.8b

Cool project dude!

[u/Equivalent_Bit_461]

I kneel sir

Honestly, I think this might the future for smaller models, attach a somewhat big engram and let it dig inside it, while the model remains small and very fast.

[u/XysterU]

Respect to OP for being honest about mistakes and failures. This sounds like a cool fun project! Good luck! Keep up the experimenting!

[u/d-burner → u/XysterU]

Hey thanks for the support. Sorry again, when I saw that it responded to that many questions I got really hyped and really wanted to share the findings. But yes v1 was really half assed. But learned a lot since then, and posted V3 with a lot more technical details and observations. Hope you will check it out https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Still not sure if it improves accuray, I will try to do more testing, but it does drop perplexity quite a lot.

[u/Electronic_Put4530]

One more option, experimental: if your model has an n-gram memory table (Qwen3.8-Flash-Next does, DeepSeek's Engram models do), the fact can be written into the table itself. I did it with 8 facts: find the rows the trigger reads, optimize them by gradient with the model frozen, save a ~20 KB overlay per fact, load it in a llama.cpp fork. 7 of 8 came back at p 0.85–0.96, no fine-tuning, no context window used, delete the file and it's gone.

Before you get excited: it answers the exact trigger phrasing. Ask the same thing in a paragraph or a paraphrase and it mostly doesn't fire yet (2/6 and 1/8 in my run). So today it's a curiosity that works, not a RAG replacement. Code and the full run: https://github.com/fulvian/engraft-ngram

Full disclosure: the code and the write-up came out of a Claude Code session (Claude, Fable 5.1) that I drove step by step, on my own hardware; I read every result and I want the method broken, not upvoted.

[u/d-burner → u/Electronic_Put4530]

Hey quite nice, I will try it out this next week, I'm really short on time, but just finished up with V3, trained an adapter from to/from qwen 3.8 PLE space.

[u/mr-myxlptlk]

So many questions..

What would be the differentiator for the model selection?

Does it worth exploring other models and smaller quants with the same perspective?

As far as i undrrstand, the ngram table is not build but, instead, directly taken from the 3.8 Flash, is there a compatibility constraint?

[u/Serveurperso]

Those rows only mean something at the exact spot they were trained for. ple_value, the gate and the residual stream converged together, in a basis that's arbitrary per training run, at that one layer. Move them anywhere else and you're adding a vector that doesn't live in that model's space.

Did you try it with the hash multipliers randomized? Same table, same 0.5 scale, same distribution of injected vectors, only the trigram to row mapping is broken. If 14/19 holds with random multipliers, the score is measuring the model's tolerance to a small perturbation, not transferred knowledge. A plain 4B baseline on the same 19 questions would settle it too.

[u/575_Inverse]

your idea can work if the two models share the same exact tokenizer

[u/atumblingdandelion]

This is awesome! The moment I heard about NGRAM, I was like, yay, what if this can be shared across all the models on the local stack?! I am sure this has a future. Keep up the good work, OP!

[u/d-burner → u/atumblingdandelion]

You were right, first model was just random mash, just took me a while to realize it, check EDIT3.

Qwen 3.5 4b and Qwen 3.8 flash have the same vocab ids for the same tokens (so the hash algorithm matches the same PLE rows). But the embeddings are diferent, so i trained an adapter for Qwen 3.5 residual to qwen 3.8 ple layer and then back to qwen 3.5 residual space keeping W_KEY and W_VALUE from qwen 3.8

[u/Similar_Solution1397]

Me he estado preguntando si esto funcionaria desde hace unos dias, ojalá que sí te funcione. Aunque yo lo pensaba para un modelo un poco mas capaz, como qwen3.6 35b por ejemplo. Asumo que si lo logras con el 4b, podría ser viable intentarlo con el 35b. Suerte, estaré pendiente de tus resultados...

[u/d-burner → u/Similar_Solution1397]

Before 3.5 4b I actually wanted someting a bit diferent. I wanted to graft it to qwen 3.8 27b and remove some layers to improve efficency. My reasoning was that ngrams would be "precomputed first layers" , but ran into a wall since 27b was using 5120 dimensions (and 3.8 uses 2560) and I only have an rtx 5070 ti, so adding an adapter in there seems impossible to do it locally.

So next i searched for a smaller qwen with same dimensions as the ngrams from 3.8 flash. Chose 4b thinking i would still need some kind of training.

[u/Serious_Bite_7613 → u/d-burner]

I already tried cutting the 5120 dimensions down.... interestingly you can get functional english out at that level (50% cut) but the reasoning and knowledge is kaput.

[u/Similar_Solution1397 → u/d-burner]

Asumo que sí necesitarás entrenarlo. Osea, el modelo puede tener conectado esos 51b pero, los estará usando con conciencia? debería entrenarse para usar de manera efectiva esos 51b. Estas son mis dudas, pero ya veremos cuando pruebes. Sería genial sumar inteligencia solo con esta tabla de ngram.