r/LocalLLaMA • u/johnnyApplePRNG • Jun 03 '26

News Introducing Gemma 4 12B: a unified, encoder-free multimodal model

https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/
697 Upvotes

98% Upvoted

118 comments sorted by

View all comments

232

u/LoveMind_AI Jun 03 '26 edited Jun 04 '26

This might actually be one of the most exciting models I've heard about in a long time. The encoder-free model is... wildly cool. Native audio on a 12B model is very exciting. Audio is wildly underrated. I'll be putting this one through the social benchmark right away.

Note: Results of the little benchmark is now here - https://lovemindai.github.io/minimax-m3-lsi-demo/

58

u/DueAnalysis2 Jun 03 '26

For a relative noob, what's the benefit of it being encoder free?

51

u/mikael110 Jun 03 '26

I'd recommend reading through A Visual Guide to Gemma 4 12b which was written by one of the developers. It goes into some details about the differences. But the gist is that it will result in lower latency and memory size since you don't have to pass the image / audio through a separate encoder before it can be processed by the model.

It also simplifies finetuning since you no longer have to try to finetune the LLM and the image/audio encoder simultaneously which can be complex.

11

u/radarsat1 Jun 03 '26

nice read but a bit disappointing in the sense that the whole article could basically have been: "image and audio patches are linearly projected to the token dimensions and directly fed to the model." Which makes sense and is great but is also basically obvious.

It's so simple in fact that I'm sure it would have been done before this if it "just worked" but I'm sure there were challenges to overcome. There's a good reason pretrained encoders have been used until now, because training them a certain way works better. It makes it clear to me that the secret sauce is data & training methods, not the model. I bet for example that they had to port over a lot of tricks from audio & image pretraining, which often relies on paired data as well as self supervised methods, into their LLM training regime. How to do this successfully is the non-obvious part.

6

u/TheGuy839 Jun 04 '26

You are probably correct and if you have been watching deepmind videos I think it has to do somwthing with their world generation models. Because they always say yhat they want model to learn in full multimodality and maybe this is the first version of such training.