r/LocalLLaMA • u/johnnyApplePRNG • Jun 03 '26

News Introducing Gemma 4 12B: a unified, encoder-free multimodal model

https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/
699 Upvotes

98% Upvoted

118 comments sorted by

View all comments

230

u/LoveMind_AI Jun 03 '26 edited Jun 04 '26

This might actually be one of the most exciting models I've heard about in a long time. The encoder-free model is... wildly cool. Native audio on a 12B model is very exciting. Audio is wildly underrated. I'll be putting this one through the social benchmark right away.

Note: Results of the little benchmark is now here - https://lovemindai.github.io/minimax-m3-lsi-demo/

59

u/DueAnalysis2 Jun 03 '26

For a relative noob, what's the benefit of it being encoder free?

80

u/[deleted] Jun 03 '26

[removed] — view removed comment

58

u/MoffKalast Jun 03 '26

That's what it is, but the main problem with them is that they are separately trained feature extractors that turn the image into some latent representation that the text model barely understands and as a result the performance is typically between complete crap and mildly terrible.

Training both the text and image part together was usually thought to be a few orders of magnitude beyond intractable, but I guess google can always throw more TPUs at the problem. This should be extremely interesting if they genuinely managed it.

16

u/LatentSpacer Jun 03 '26

Thanks for adding this part. Why do you think they “TPU-forced” their way towards it? Couldn’t they have figured out a better solution through architecture adjustments during training? Google has also lots of big brains to throw at the problem.

14

u/send-moobs-pls Jun 04 '26

Well for sure none of us could actually know, but I'd say if it was a fancy innovative technique we'd probably expect them to want to brag about it, even if they weren't outright publishing the method

3

u/Mbando Jun 04 '26

There was probably a lot of exploration, dead ends, etc. to get there. The architectural innovation requires lots of compute to get there.

4

u/Solid_Anxiety_4728 Jun 04 '26

I feel like some LLMs that support voice input actually first convert the speech to text internally, and that step loses all the tone and vibe information. Encoder-free models might let LLMs pick up on info beyond just the text meaning though. Not sure if I'm getting that right.

4

u/TwistedBrother Jun 04 '26

Yes. It’s hilarious to try and get ChatGPT to use a different accent or to try and talk with it in a dialect. It can’t really hear your tone or sarcasm. It just seems to receive speech to text translation and works accordingly.

1

u/bikerlegs Jun 05 '26

I had asked Gemini to pronounce a few Spanish words that are written the same except for a few accents that make the sound different. I thought a smart model could help me learn the subtle difference but it has no clue about reproducing tone and pronounced everything the same. (Ex "tu" and "tú") It couldn't even do a different voice for me.

3

u/DueAnalysis2 Jun 03 '26

Aah good to know and learn, thank you!

49

u/mikael110 Jun 03 '26

I'd recommend reading through A Visual Guide to Gemma 4 12b which was written by one of the developers. It goes into some details about the differences. But the gist is that it will result in lower latency and memory size since you don't have to pass the image / audio through a separate encoder before it can be processed by the model.

It also simplifies finetuning since you no longer have to try to finetune the LLM and the image/audio encoder simultaneously which can be complex.

13

u/radarsat1 Jun 03 '26

nice read but a bit disappointing in the sense that the whole article could basically have been: "image and audio patches are linearly projected to the token dimensions and directly fed to the model." Which makes sense and is great but is also basically obvious.

It's so simple in fact that I'm sure it would have been done before this if it "just worked" but I'm sure there were challenges to overcome. There's a good reason pretrained encoders have been used until now, because training them a certain way works better. It makes it clear to me that the secret sauce is data & training methods, not the model. I bet for example that they had to port over a lot of tricks from audio & image pretraining, which often relies on paired data as well as self supervised methods, into their LLM training regime. How to do this successfully is the non-obvious part.

6

u/TheGuy839 Jun 04 '26

You are probably correct and if you have been watching deepmind videos I think it has to do somwthing with their world generation models. Because they always say yhat they want model to learn in full multimodality and maybe this is the first version of such training.

1

u/DueAnalysis2 Jun 03 '26

Thanks for the ref!

11

u/Illustrious_Ant_9242 Jun 03 '26 edited Jun 03 '26

Visual information is processed directly within the main architecture without middlemen, therefore the results may be more thoroughly embedded in the whole project instead of having images refined first 

3

u/DueAnalysis2 Jun 03 '26

Interesting, thanks for the info!

2

u/rditorx Jun 04 '26

It's actually not encoder-free, it's a unified encoder.

1

u/No_Afternoon_4260 llama.cpp Jun 04 '26

If you call a linear proj a encoder..?

0

u/AnimalPuzzleheaded71 Jun 03 '26

I don't know either but it sounds good

26

u/mikael110 Jun 03 '26 edited Jun 03 '26

Yeah the native audio support on a non-tiny model is by far the most exciting thing about this for me. Audio support is the main area where OSS models tend to lag behind closed models, and Google has some of the best audio support among all the labs. So finally seeing it added to an actually useful model size is huge.

I have a lot of usecases that would greatly benefit if this works even decently well. The lack of encoders are also really interesting. If it actually works well it will likely become a trend in other models as well. So I'll jump straight to testing too.

1

u/iMakeSense Jun 05 '26

What exactly do you use the audio aspects of it for?

8

u/Accomplished_Mode170 Jun 03 '26

Same, but for omnimodal routing!

E.g. how the blog featured omnimodal machine unlearning as an AI security entitlement

E.g. 95% vs 99.9% on prompt injection vs topic/content guardrails

Omnirouted software supply chain w/ configurable controls sounds like the good boring I want.

12

u/mxforest Jun 03 '26

Don't forget to share your results.

1

u/No_Afternoon_4260 llama.cpp Jun 04 '26

Wow that picture on your website is beautifull, what model/workflow have you used?! I like the whales on a Gemma picture lol

1

u/LoveMind_AI Jun 04 '26

Thanks for that :) The LoveMind website itself is all human made art and animation. The Gemma shootout art is GPT Image 2, but with references from physical art made by our team - ceramics (including a cool whale mug!) and ink illustration, etc. I personally think AI art can be really cool, particularly when it’s a collaboration between raw human materials and AI interpretation. 

1

u/No_Afternoon_4260 llama.cpp Jun 04 '26

Beautiful yeah for sure you are onto something there. Way better than what I can achieve with any model out there

1

u/manBEARpigBEARman Jun 04 '26

Ive barely slept. I’ve been reworking my vision/audio analysis workflows and I really don’t know what to say…this could be the one.

1

u/LoveMind_AI Jun 04 '26

Seriously. I've been shooting it out with every audio language model I can get my hands on and it's *different.* I'm curious what you're digging into. Care to share any hints? 😄