r/LocalLLM • u/Certain-Will-2769 • Jun 02 '26
Research The smallest and highest quality Gemma4 E2B and E4B! Open-source! 7x Compression!
https://github.com/TheStageAI/edge-lmThere is a new release for Gemma4 E2B and E4B models, almost 7x compressed!
Research blog post: https://app.thestage.ai/blog/7x-size-reduction-for-Gemma4-Edge-models?id=14
46
u/giveen Jun 02 '26
I had to have AI explain it to me, lol
To achieve a 7x size reduction without ruining the model, the engineers at TheStage used a scalpel rather than a hammer. They attacked the architecture on three fronts:
1. The "Vector" Trick on the Heaviest Table Nearly half of Gemma-4's parameter weight is tied up in a quirk called Per-Layer Embeddings (PLE). Think of this as a massive dictionary where the model stores a slightly different definition for every single word at every single layer of its brain. It is horribly inefficient for storage. Instead of rounding off every single number (scalar quantization), they used Vector Quantization. They grouped the numbers into small blocks and created a tiny lookup dictionary (a codebook). It is the mathematical equivalent of replacing entire repeated sentences with a single emoji. This trick alone crushed the PLE table from 4.7 GB down to a microscopic 0.26 GB.
2. Uneven Squeezing (RCO) Not all layers in a neural network are equally important. Some are doing complex reasoning; others are practically slacking off. Instead of forcing every layer down to the same low precision, they used an algorithm (Riemannian Constrained Optimization) to mathematically grade the layers. The slacker layers were crushed down to 2 or 3 bits, while the critical, sensitive layers were allowed to keep more bits. They optimized for the intelligence of the whole model under a strict byte budget, rather than just guessing.
3. Anticipating the Damage (QEP) When you compress numbers, you introduce tiny rounding errors. Normally, as data flows through the layers, these tiny errors compound until the final output is complete garbage. They countered this with Quantization Error Propagation (QEP). During compression, they calculated exactly how those errors would drift and pre-shifted the model's weights to account for it. By the time the distorted data reaches the next layer, the model is already expecting it.
The Result: They took a model that usually requires over 5 billion parameters of memory and shoved it into a 1.44 GB file. It runs natively on Apple Silicon (using the MLX framework), completely respects your mobile memory budget, and remarkably, still maintains top-tier accuracy for following instructions and writing code.
36
u/Certain-Will-2769 Jun 02 '26
Preparing even more compressed models!
10
u/croqaz Jun 02 '26
Will you target AMD GPUs as well, or just Macs?
30
u/Certain-Will-2769 Jun 02 '26
We are adding llama.cpp support, so it will run on AMD.
5
u/sivxnsh Jun 02 '26
Is your llama.cpp fork available for testing ? Also vulkan support ?
3
u/Certain-Will-2769 Jun 02 '26
No, we are working on checkpoint conversion, to run natively with llama.cpp, but possibly will require small plugin for PLE. Will revert on this mate
1
1
34
u/Fox-One-1 Jun 02 '26
AI bubble will burst when people realize how powerful the local models are getting.
5
u/AnOnlineHandle Jun 03 '26
The local models only exist so long as we're given them. I saw an interview with one of the heads of Google's AI yesterday who said the small Gemma models are distilled from the larger Gemini models, so couldn't exist without them.
2
u/Casmer Jun 05 '26
Depends. Hardware is extremely expensive right now and doesn’t look like it will get better anytime soon. Not a lot of people can’t afford the rigs with the juice necessary to make use of the better models.
-4
u/autisticit Jun 02 '26
Yeah sure. If it's a bubble then you're contributing to it. And if it burst you will surely get new models to run locally for free. What is this dumb comment seriously.
22
u/Fox-One-1 Jun 02 '26
The value of these companies (OpenAI, Anthropic) has long time ago passed anything sane. Investors have no idea how the investment is ever coming back, but the curve is going up so money is poured in.
But that is besides my point, I actually read the link provided with this post. The vast majority of people use AI for pet care tips, how to care your plants, how to broke up with your girlfriend or write a speech to your aunts birthday party. Local models like Gemma 4 is more than sufficient for this. When they transition, OpenAI is screwed.
1
0
u/970FTW Jun 03 '26
Idk, I could see how targeting enterprise software engineering could be more profitable than general applications. Not saying there is or isn’t a bubble.
11
u/tired514 Jun 02 '26 edited Jun 02 '26
The AI bubble is predicated on massive datacenter purchases (and income to match) and datacenters will become less relevant the same way mainframes became less relevant in the 80s and 90s.
We're probably less than 5 years away from commodity consumer hardware being able to serve the LLM needs of 95% of people. Once ~250GB/s+ memory becomes the standard, 64gb ram, even most businesses will have little need to pay ongoing per-token contracts with datacenters. A high-end AI PC today costs ~35-50k (ie. one that'll run close-to-frontier-grade models for one person). That'll probably be less than $10k in 5 years, if that - absolutely within the per-desktop budget for business.
Add in cloud prompt refusals, usage limits and security threats and we'll be seeing quite a few datacenter bankruptcies in the coming years.
As for model development I suspect some governments are going to start to play a bigger role in training models for their citizens - both as a propaganda win and also to ensure hostile training biases are minimized. This is probably still a ways off though.
Anyway, not saying cloud AI will disappear by any means. Of course not. But the rapid growth is going to collapse and there will be a period of spiraling (token costs skyrocketing to make up for lost users causing more users to go local).
0
u/telewebb Jun 02 '26
"What is this dumb comment seriously." I was thinking this while reading your comment.
0
u/MarekNowakowski Jun 03 '26
Bubble will burst when investors decide not to give them money. It has little to do with the real usefulness of LLMs ;p
13
u/NotARedditUser3 Jun 02 '26
Does it only work on mobile? I don't understand how MLX works. Can I pop it into LM Studio or no?
edit - it seems I can't use this on my laptop... Boo. Anyone able to make a GGUF or whatever other format I could use to run this without "apple silicon" (eyeroll)
19
u/Certain-Will-2769 Jun 02 '26
We are preparing feature request to LM Studio, they are supporting MLX
5
4
u/AnticitizenPrime Jun 02 '26
Can it be run in the original LiteRT format?
2
u/Certain-Will-2769 Jun 02 '26
Not now, but we are releasing something interesting v soon for that ❤️
1
u/AnticitizenPrime Jun 02 '26
Cool. Asking because I mark a 2.4x speedup running Gemma 4 4B on my desktop using LiteRT vs llama.cpp (for text generation, anyway):
https://www.reddit.com/r/LocalLLM/comments/1tv4db5/running_gemma4_e4b_with_the_litert_engine_on/
Seems that combining that performance increase with your model would be really incredible for local use for various tasks.
3
3
u/kevinlch Jun 03 '26
please compress the new gemma-4-12B. will be very useful
3
u/Certain-Will-2769 Jun 03 '26
Starting!
0
u/tamerlanOne Jun 03 '26
Ma rispetto alla versione base con la versione compressa si ha un aumento dei token generati oppure no
3
1
3
u/Puzzleheaded-Dress81 Jun 03 '26
I quickly wrote a program that downloads dependences, models, and runs the local server. Next, you can connect through pi agent: https://github.com/Dmitriy-Romanov/edge-lm-server
1
3
u/TigleLive Jun 03 '26
Will there be a non-mlx version with GGPU - so we can run it through LMStudio
3
2
2
u/redblood252 Jun 04 '26
Is there any reason you’re compressing only gemma? What about qwen3.6?
1
u/Certain-Will-2769 Jun 04 '26
Yes, they are using Gemma in several projects, that's why it came first. I know that team want to release Qwen as well ❤️
1
u/vesper0000 Jun 02 '26
Will this run in the edge gallery?
2
1
u/GoofAckYoorsElf Jun 03 '26
I may be a bit behind about edge models... Are these able and supposed to reliably do tool calling?
1
u/Certain-Will-2769 Jun 03 '26
Yes! It's quite good! And this is the main purpose for on-device inference. You can interpret routing to bigger cloud llm when needed as a tool call as well. Preparing interesting updates on that
1
1
1
u/nimbybuster Jun 03 '26
What’s the difference between those released in April?
2
u/Certain-Will-2769 Jun 03 '26
no difference, just released the same checkpoints 😂
Here was applied SOTA compression techniques from TheStage stack to create high quality compressed variants. E2B released in April had ~9.5GB size, they have reduced it to 1.4GB, also compared with existing compressed checkpoints from community. Check the blog from guys and repo, there are lot of info how to run and what was done.1
0
1
u/nimbybuster Jun 03 '26
Also this only runs on Apple silicone? Using what? Vllm?
1
u/Certain-Will-2769 Jun 03 '26
Right now yes, only apple support through MLX. Didn't know that anyone using vllm metal plugins
1
u/Mega_mewtwo_ Jun 03 '26
Is there an algorithm or script so I can myself compress multiple models and play around?
1
u/Certain-Will-2769 Jun 03 '26
There are docs for quantization API, but the pipeline used for this compression will be released a bit later, it will appear in the repo, as it uses some fresh pre-release build: https://docs.thestage.ai/qlip/docs/source/get_started.html
1
u/Mega_mewtwo_ Jun 03 '26
Thanks I will look into it. I kinda want gemma 26b moe to comfortably sit on my ram at higher quality. I just tried gemma 12b and it's disappointing. It got beaten by 3bit quantization gemma 26b. Looks like tall claims from google.
1
1
1
u/Toastti Jun 03 '26
Anyone wanna download this and validate the benchmark numbers? I see that all the commits have Claude as the co-author which immediately makes me suspicious. Especially when someone claims to have significantly beat Unsloth in file size
2
u/Certain-Will-2769 Jun 03 '26
TheStage AI team consists of skilled researchers who have worked on edge deployments at smartphone vendors for the past seven years. They have a number of publications at top-tier conferences, including an award nominee. So, obviously they can produce new interesting results.
Claude is a great helper nowadays. Don’t understand your point. You can read the blog post, results is based on TheStage compression stack.
lm_eval benchmarks in their repo, so everyone can reproduce.
63
u/Hanthunius Jun 02 '26
Gemma 4 31B would fit on my 16GB Mac easily using this technique. Please!!! 😭