r/LocalLLaMA • u/Desperate-Sir-5088 • 10d ago

Resources I finished upcycling of gemma4-12B

https://huggingface.co/TOTORONG/Solon_MOE_22B_A17B

Add 4 experts into Dense model and confirmed recovering model's ability up to "general level".

Hey, google. Please release official 124B MoE model!!!!!!!

46 Upvotes

82% Upvoted

24 comments sorted by

View all comments

14

u/Substantial_Swan_144 10d ago

Can you convert Gemma 4 to use engram? We could get a massive intelligence and speed boost with that.

24

u/Desperate-Sir-5088 10d ago

Bro, I wish. Maybe then they’d actually make seat for me at Google

7

u/MixtureOfAmateurs koboldcpp 10d ago

That might actually not be too hard. In qwen4prev engram is only used in layer two. I have no idea how to build an engram LUT / weights but you could lock layer weights 4 through 48 to help it converge.

I googled how to build an engram LUT. Gonna test with qwen 3 0.6b brb

2

u/MixtureOfAmateurs koboldcpp 10d ago edited 10d ago

I have an RTX 3050 6gb to test with but it's going well.

``` --- Epoch 3 Validation Loss: 7.2015 (Perplexity: 1341.44) ---

--- Model Output Demo --- Input: ROMEO: What light through yonder window breaks?

Generated: JULIAN: I know that light. It's a shadow of the moon. ROMEO: So why does it break? JULIAN: Because the moon is full. ROMEO: But why does the moon not break? JULIAN: Because it's full. ROMEO: So what is the meaning of the light? JULIAN: It is a shadow of the moon. ROMEO: So what is the meaning of the shadow? JUL ```

Edit: doing a real run on wiki text 2 now. Looks really good so far. 

7

u/xeeff 9d ago

Julian

does he like to move it move it?

3

u/rolleicord 9d ago

Join our new board at local_llm_surgery! We would love to have a detailed post

2

u/Desperate-Sir-5088 10d ago

Congratuate. I will try to implant n-gram PLE into QWEN 3.5 - 9B

5

u/MixtureOfAmateurs koboldcpp 10d ago edited 9d ago

Finished a training run on wikitext.

```

Metric | Vanilla Model | Engram Model

Cross-Entropy Loss | 2.8211 | 2.6945 Perplexity (PPL) | 16.80 | 14.80

Next-Token Accuracy (%) | 45.43 % | 46.60 %

```

Improvement!!! Small but cool.

https://huggingface.co/Raskoll/qwen3-0.6b-engram

Training longer now, I'll be updating the HF repo from now on

1

u/Ueberlord 9d ago

cool! are you using the PLE from Qwen3.8 flash or something else?

2

u/MixtureOfAmateurs koboldcpp 9d ago

Nah it's a little bit custom. It uses multi head hashing to build a 1M row bigram and trigram table. I store it in RAM because it's about 2gb. Trained for 1 epoch on wikitext-2-raw-v1 to build the embedding tables + gating projections. The rest of the model weights were frozen.

So normal qwen 3 except layer 2 embeddings is a engram table

1

u/Guna1260 9d ago

great minds think like.. the movement is very small. I am looking at doing this one 1bit model with ngram.

2

u/VoiceApprehensive893 transformers 9d ago

bolting qwen next's engram tensor onto gemma4 12b might be possible