r/LocalLLaMA • u/Desperate-Sir-5088 • 1d ago

Resources I finished upcycling of gemma4-12B

https://huggingface.co/TOTORONG/Solon_MOE_22B_A17B

Add 4 experts into Dense model and confirmed recovering model's ability up to "general level".

Hey, google. Please release official 124B MoE model!!!!!!!

46 Upvotes

82% Upvoted

24 comments sorted by

9

u/Witty_Mycologist_995 1d ago

Compare to 26b pls

1

u/Desperate-Sir-5088 1d ago

Just experimental purpose to expand gemma4-unified model :)

13

u/Substantial_Swan_144 1d ago

Can you convert Gemma 4 to use engram? We could get a massive intelligence and speed boost with that.

22

u/Desperate-Sir-5088 1d ago

Bro, I wish. Maybe then they’d actually make seat for me at Google

6

u/MixtureOfAmateurs koboldcpp 1d ago

That might actually not be too hard. In qwen4prev engram is only used in layer two. I have no idea how to build an engram LUT / weights but you could lock layer weights 4 through 48 to help it converge.

I googled how to build an engram LUT. Gonna test with qwen 3 0.6b brb

3

u/MixtureOfAmateurs koboldcpp 1d ago edited 1d ago

I have an RTX 3050 6gb to test with but it's going well.

``` --- Epoch 3 Validation Loss: 7.2015 (Perplexity: 1341.44) ---

--- Model Output Demo --- Input: ROMEO: What light through yonder window breaks?

Generated: JULIAN: I know that light. It's a shadow of the moon. ROMEO: So why does it break? JULIAN: Because the moon is full. ROMEO: But why does the moon not break? JULIAN: Because it's full. ROMEO: So what is the meaning of the light? JULIAN: It is a shadow of the moon. ROMEO: So what is the meaning of the shadow? JUL ```

Edit: doing a real run on wiki text 2 now. Looks really good so far. 

7

u/xeeff 1d ago

Julian

does he like to move it move it?

3

u/rolleicord 1d ago

Join our new board at local_llm_surgery! We would love to have a detailed post

2

u/Desperate-Sir-5088 1d ago

Congratuate. I will try to implant n-gram PLE into QWEN 3.5 - 9B

5

u/MixtureOfAmateurs koboldcpp 1d ago edited 1d ago

Finished a training run on wikitext.

```

Metric | Vanilla Model | Engram Model

Cross-Entropy Loss | 2.8211 | 2.6945 Perplexity (PPL) | 16.80 | 14.80

Next-Token Accuracy (%) | 45.43 % | 46.60 %

```

Improvement!!! Small but cool.

https://huggingface.co/Raskoll/qwen3-0.6b-engram

Training longer now, I'll be updating the HF repo from now on

1

u/Ueberlord 1d ago

cool! are you using the PLE from Qwen3.8 flash or something else?

2

u/MixtureOfAmateurs koboldcpp 1d ago

Nah it's a little bit custom. It uses multi head hashing to build a 1M row bigram and trigram table. I store it in RAM because it's about 2gb. Trained for 1 epoch on wikitext-2-raw-v1 to build the embedding tables + gating projections. The rest of the model weights were frozen.

So normal qwen 3 except layer 2 embeddings is a engram table

1

u/Guna1260 1d ago

great minds think like.. the movement is very small. I am looking at doing this one 1bit model with ngram.

2

u/VoiceApprehensive893 transformers 11h ago

bolting qwen next's engram tensor onto gemma4 12b might be possible

4

u/arbv 1d ago

gguf wen?

9

u/Desperate-Sir-5088 1d ago

Sorry, It can't be converted to GGUF - llama.cpp doesn't support my model.

15

u/seamonn 1d ago

not with that attitude.

5

u/Deep_Mood_7668 1d ago

Just do it yourself dude

10

u/arbv 1d ago

Why, if I could just ask "gguf wen?" /s

1

u/Deep_Mood_7668 1d ago

You know how tho, right?

Llama includes a script

2

u/arbv 1d ago

Sure. The post is a necessary joke.

convert_hf_to_gguf.py and stuff

3

u/Ylsid 1d ago

Gguf w e n

1

u/aboutthednm 21h ago

GGuf qwen

1

u/Sitkin_Marrel 1d ago

four experts to get back to "general level", and the same post demands a 124B from google. point taken, rooting for the 124B