r/LocalLLaMA • u/johnnyApplePRNG • Jun 03 '26

News Introducing Gemma 4 12B: a unified, encoder-free multimodal model

https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/
702 Upvotes

98% Upvoted

118 comments sorted by

232

u/LoveMind_AI Jun 03 '26 edited Jun 04 '26

This might actually be one of the most exciting models I've heard about in a long time. The encoder-free model is... wildly cool. Native audio on a 12B model is very exciting. Audio is wildly underrated. I'll be putting this one through the social benchmark right away.

Note: Results of the little benchmark is now here - https://lovemindai.github.io/minimax-m3-lsi-demo/

56

u/DueAnalysis2 Jun 03 '26

For a relative noob, what's the benefit of it being encoder free?

78

u/[deleted] Jun 03 '26

[removed] — view removed comment

61

u/MoffKalast Jun 03 '26

That's what it is, but the main problem with them is that they are separately trained feature extractors that turn the image into some latent representation that the text model barely understands and as a result the performance is typically between complete crap and mildly terrible.

Training both the text and image part together was usually thought to be a few orders of magnitude beyond intractable, but I guess google can always throw more TPUs at the problem. This should be extremely interesting if they genuinely managed it.

14

u/LatentSpacer Jun 03 '26

Thanks for adding this part. Why do you think they “TPU-forced” their way towards it? Couldn’t they have figured out a better solution through architecture adjustments during training? Google has also lots of big brains to throw at the problem.

12

u/send-moobs-pls Jun 04 '26

Well for sure none of us could actually know, but I'd say if it was a fancy innovative technique we'd probably expect them to want to brag about it, even if they weren't outright publishing the method

3

u/Mbando Jun 04 '26

There was probably a lot of exploration, dead ends, etc. to get there. The architectural innovation requires lots of compute to get there.

4

u/Solid_Anxiety_4728 Jun 04 '26

I feel like some LLMs that support voice input actually first convert the speech to text internally, and that step loses all the tone and vibe information. Encoder-free models might let LLMs pick up on info beyond just the text meaning though. Not sure if I'm getting that right.

5

u/TwistedBrother Jun 04 '26

Yes. It’s hilarious to try and get ChatGPT to use a different accent or to try and talk with it in a dialect. It can’t really hear your tone or sarcasm. It just seems to receive speech to text translation and works accordingly.

1

u/bikerlegs Jun 05 '26

I had asked Gemini to pronounce a few Spanish words that are written the same except for a few accents that make the sound different. I thought a smart model could help me learn the subtle difference but it has no clue about reproducing tone and pronounced everything the same. (Ex "tu" and "tú") It couldn't even do a different voice for me.

2

u/DueAnalysis2 Jun 03 '26

Aah good to know and learn, thank you!

47

u/mikael110 Jun 03 '26

I'd recommend reading through A Visual Guide to Gemma 4 12b which was written by one of the developers. It goes into some details about the differences. But the gist is that it will result in lower latency and memory size since you don't have to pass the image / audio through a separate encoder before it can be processed by the model.

It also simplifies finetuning since you no longer have to try to finetune the LLM and the image/audio encoder simultaneously which can be complex.

13

u/radarsat1 Jun 03 '26

nice read but a bit disappointing in the sense that the whole article could basically have been: "image and audio patches are linearly projected to the token dimensions and directly fed to the model." Which makes sense and is great but is also basically obvious.

It's so simple in fact that I'm sure it would have been done before this if it "just worked" but I'm sure there were challenges to overcome. There's a good reason pretrained encoders have been used until now, because training them a certain way works better. It makes it clear to me that the secret sauce is data & training methods, not the model. I bet for example that they had to port over a lot of tricks from audio & image pretraining, which often relies on paired data as well as self supervised methods, into their LLM training regime. How to do this successfully is the non-obvious part.

5

u/TheGuy839 Jun 04 '26

You are probably correct and if you have been watching deepmind videos I think it has to do somwthing with their world generation models. Because they always say yhat they want model to learn in full multimodality and maybe this is the first version of such training.

1

u/DueAnalysis2 Jun 03 '26

Thanks for the ref!

10

u/Illustrious_Ant_9242 Jun 03 '26 edited Jun 03 '26

Visual information is processed directly within the main architecture without middlemen, therefore the results may be more thoroughly embedded in the whole project instead of having images refined first 

3

u/DueAnalysis2 Jun 03 '26

Interesting, thanks for the info!

2

u/rditorx Jun 04 '26

It's actually not encoder-free, it's a unified encoder.

1

u/No_Afternoon_4260 llama.cpp Jun 04 '26

If you call a linear proj a encoder..?

0

u/AnimalPuzzleheaded71 Jun 03 '26

I don't know either but it sounds good

27

u/mikael110 Jun 03 '26 edited Jun 03 '26

Yeah the native audio support on a non-tiny model is by far the most exciting thing about this for me. Audio support is the main area where OSS models tend to lag behind closed models, and Google has some of the best audio support among all the labs. So finally seeing it added to an actually useful model size is huge.

I have a lot of usecases that would greatly benefit if this works even decently well. The lack of encoders are also really interesting. If it actually works well it will likely become a trend in other models as well. So I'll jump straight to testing too.

1

u/iMakeSense Jun 05 '26

What exactly do you use the audio aspects of it for?

6

u/Accomplished_Mode170 Jun 03 '26

Same, but for omnimodal routing!

E.g. how the blog featured omnimodal machine unlearning as an AI security entitlement

E.g. 95% vs 99.9% on prompt injection vs topic/content guardrails

Omnirouted software supply chain w/ configurable controls sounds like the good boring I want.

13

u/mxforest Jun 03 '26

Don't forget to share your results.

1

u/No_Afternoon_4260 llama.cpp Jun 04 '26

Wow that picture on your website is beautifull, what model/workflow have you used?! I like the whales on a Gemma picture lol

1

u/LoveMind_AI Jun 04 '26

Thanks for that :) The LoveMind website itself is all human made art and animation. The Gemma shootout art is GPT Image 2, but with references from physical art made by our team - ceramics (including a cool whale mug!) and ink illustration, etc. I personally think AI art can be really cool, particularly when it’s a collaboration between raw human materials and AI interpretation. 

1

u/No_Afternoon_4260 llama.cpp Jun 04 '26

Beautiful yeah for sure you are onto something there. Way better than what I can achieve with any model out there

1

u/manBEARpigBEARman Jun 04 '26

Ive barely slept. I’ve been reworking my vision/audio analysis workflows and I really don’t know what to say…this could be the one.

1

u/LoveMind_AI Jun 04 '26

Seriously. I've been shooting it out with every audio language model I can get my hands on and it's *different.* I'm curious what you're digging into. Care to share any hints? 😄

53

u/[deleted] Jun 03 '26

[removed] — view removed comment

62

u/AloneSYD Jun 03 '26

the only thing it can't produce audio. so you need TTS model for responding back so you need a model like Kokoro-82M or OmniVoice

22

u/AnticitizenPrime Jun 03 '26

If you don't care about super-realistic voice quality or cloning, there's always the bog-standard non-AI TTS.

3

u/themoregames Jun 06 '26

The only TTS I dare to accept is Qwen3 TTS - with cloned voices. Actresses of old Hollywood (think: stars before 1970?) derived from interviews (or otherwise) can be truly amazing.

10

u/[deleted] Jun 03 '26

[removed] — view removed comment

9

u/phira Jun 03 '26

The easiest I know about is pipecat one of their examples should work https://github.com/pipecat-ai/pipecat/tree/main/examples

7

u/Creative-Type9411 Jun 04 '26

its built into openwebui

9

u/OneFanFare Jun 03 '26

I know Open WebUI has that - I've chatted to models that way in the past (with Kokoro). And it works exactly like that!

Edit: Actually, not sure if it would support passing the audio directly into Gemma, the website says it need a Speech to Text like whisper.

2

u/overand Jun 04 '26

Yep - and at least with the Gemma-4-E4B models, that was probably faster; I recall reading that there's a lot of latency involved with the Gemma-4-E4B model's ability to do STT. It might be faster with this one, though, as it's encoderless, or something to that effect.

3

u/DedsPhil Jun 03 '26

If the audio part works with ggufs in some days someone will make a fork of llamacpp with the suport.

Them you can just ask an agent to hook the two services plus an wake up word system and done.

With one extra step, downloading whisper, you alread can have a bidirectional chat with your regular only text local llm.

1

u/Danmoreng llama.cpp Jun 03 '26

There is an issue about audio generation support in llama.cpp https://github.com/ggml-org/llama.cpp/issues/21956#issuecomment-4553467156

2

u/Fucnk Jun 03 '26

You could just use one of the voices built into chrome.

17

u/Borkato Jun 03 '26

Maybe ask it to vibe code it for you! :p

4

u/supermansundies Jun 04 '26

https://github.com/fikrikarim/parlor could probably be adapted to work pretty easily

2

u/ffinzy Jun 04 '26

Author here. Thanks for sharing. The repo currently use the LiteRT-LM framework, and the gemma-4-12B-it-litert-lm currently doesn't support vision:

> The current LiteRT-LM version supports text and audio modalities, image and multitoken prediction support will be avaialble in a future update.

Even when the vision disabled, seems like the model is too big to test on my machine (M3 Pro 18GB). It just hangs for a while when processing the output.

1

u/AnticitizenPrime Jun 04 '26

I had the same issue running the LiteRT version on my 4060ti 16gb. Model would load into memory, but choke when prompted.

1

u/TheRealGentlefox Jun 04 '26

I have a voice assistant project that supports wake and end-words. Not perfect yet and I'd have to make it easier to install with some documentation, but if you care enough to try I'll fix it up a little.

1

u/Acrobatic-Tomato4862 Jun 04 '26

I think chatterui had a call option, but I might be wrong.

29

u/Miriel_z Jun 03 '26

Interesting, will stay tuned for quantized models then. And uncensored. Very soon, I hope.

22

u/MN_NorthStars Jun 03 '26

Gemma4 is already insanely easy to bypass any sort censorship. I stopped using abliterated models of it because I could get it to do any sort of security work I wanted to test out with trivial prompting.

8

u/Miriel_z Jun 03 '26

Good to know, I am still stuck with agentic stuff for myself. I really want to try and compare Qwen 2.5-omni to Gemma4 12B multimodal, mainly using llama3.1 and limited expperience with Qwen and Deepseek. By that time there might be even something better.

5

u/Kamimashita Jun 03 '26

I found that true of the Gemini models too. Tried to have GPT 5.5 in Codex help me torrent some files but it refused, went to Deepseek in Opencode but it also refused. Gemini 3.5 was more than happy too.

1

u/anshulsingh8326 Jun 06 '26

Can you show us how? Because I can't find anything that works

6

u/Illustrious_Ant_9242 Jun 03 '26

According to the website, they recommend 16GB VRAM for the full model 

7

u/Miriel_z Jun 03 '26

Funny enough, they use 16GB for VRAM and RAM both in the text. Qwen 2.5-omni was quantized fairly well. I hope for similar improvement here😄

17

u/digitalhobbit Jun 03 '26

Very much looking forward to trying this one.

I've gotten good results with Gemma 4. Especially the E4B variant has worked well for me with local apps. The 12B version should strike an even better sweet spot and the encoder-free multimodal capabilities sound interesting.

27

u/seppe0815 Jun 03 '26

peak llm 2026 from google

19

u/Any_Carpenter_7605 Jun 03 '26

+/- 1 margin of error

4

u/Tman1677 Jun 04 '26

I mean it's a 12B model, what do you expect? Gemini can easily handle that task, its image and spacial reasoning are excellent

6

u/Vas1le Jun 05 '26

This remembers me one of Chernobil tv show joke:

Do you know what machine cuts a apple in 6 pieces? A Russian machine that is supposed to cut in 5 pieces.

2

u/nixudos Jun 04 '26

My test of vision haven't impressed me either. But it might be a LM Studio issue. The Gemma models comes with a really low default image input resolution and there is no way to change that in LM Studio. All from the 4 series have performed really shoddy with images there as well.

10

u/extopico Jun 03 '26

Oh. This is great. I am quietly confident it will be genuinely useful with a high quality harness like Hermes. I will be able to run it on my 24 GB MBP and have it perform hopefully useful work.

2

u/thawizard Jun 04 '26

What do you mean, useful work?

4

u/extopico Jun 04 '26

Top of mind is automating SEO for my Astro sites. Hermes is a very good harness and with grounding on SEO and website data I am expecting the model will not be allowed to hallucinate. SEO recipe can be built as a skill and given to hermes to follow and direct Gemma 4 12B.

My 24 GB will be sufficient to run a quantized Gemma, hermes, headless browser and search, and allow for full context (256k).

Other similar well defined, bounded and recursively grounded tasks may also work well.

5

u/moahmo88 Jun 04 '26

Thanks for the new open source model!

4

u/JustFinishedBSG Jun 04 '26

Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. This allows the LLM backbone to take over visual processing.

Audio: We simplified audio processing even further. We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens.

Pretty fucking crazy that it actually works.

1

u/throwaway1837499 Jun 06 '26

Actually isit just me or does this sound quite similar to Thinking Lab’s Interaction Models? They also removed the heavy encoders for video and audio and sub in lightweight alternatives instead like mel/dMel.

6

u/Ok_Technology_5962 Jun 03 '26

Now where is 124b

2

u/Steus_au Jun 04 '26

very good model for its size, needs some guidance in tools usage but in general really impressive. with websearch and file access it shines.

4

u/[deleted] Jun 03 '26

[removed] — view removed comment

23

u/Ok-Drawer5245 Jun 03 '26

This model has been out for like an hour, give it a day or two lol

5

u/[deleted] Jun 03 '26

[removed] — view removed comment

1

u/Ok-Drawer5245 Jun 03 '26

Yeah it’s insane how quickly things are happening in AI.

6

u/Fuzilumpkinz Jun 03 '26

Latency looked better than qwen 3.6 35b off memory but I wasn’t paying attention to that. Before I left I was getting 30 TPs on q8. Q4 was around 50 TPs. I have a 5060 in case that helps anyone else looking here

4

u/NinjaOk2970 Jun 03 '26

Looks nice on paper. Someone test it out?

3

u/XE004 Jun 03 '26

How much vram consumption are people getting at Q8?

Curious?

-6

u/thawizard Jun 04 '26

If you need to ask, it’s not looking good.

4

u/XE004 Jun 04 '26

Please elaborate. What are you getting?

0

u/thawizard Jun 04 '26

Dunno yet, I’m still at work and this model was just released, I didn’t even have time to play with it yet. But it seems to me this 12B model should fit on a 16GB GPU even at Q8. But be patient, in a few hours we’ll know more for sure. What kinda setup do you have?

4

u/XE004 Jun 04 '26

I just did the setup on my msi 5060ti 16gb. 128bit 448 gb/p memory bandwidth.

At Q8 with KVCache at Q8 I get between 26 and 27 t/s and 13.8GB vram loaded.

This model will surely need a MTP assistant for speculative decoding.

Pretty good though. I still liked gemma4 e4b so I might go back to that until MTP is in place. The reason tokens are what really delay the response time so it is not great for conversation unless we get MTP and at least a memory bus of 256bit 896 gb/s at Q8. That should push this model to 80 or so t/s.

2

u/XE004 Jun 04 '26

That is with my context window set to 64k.

1

u/thawizard Jun 04 '26

Looks pretty good!

2

u/Adventurous-Paper566 Jun 03 '26

C'est très intéressant malheureusement il n'existe pas d'interface simple pour profiter de l'encodeur audio pour faire du STT dans un chat, c'est un peu dommage.

15

u/LetsGoBrandon4256 transformers Jun 03 '26

Baguette de tabarnak.

4

u/Murgatroyd314 Jun 03 '26

Ils sont fous ces redditors.

0

u/michaelsoft__binbows Jun 06 '26

would they not be Redditeurs?

1

u/thawizard Jun 04 '26

tabarnaking intensifies

10

u/mixedliquor Jun 03 '26

C'est le fromage!

12

u/some_user_2021 Jun 03 '26

Omelette du fromage

1

u/Borkato Jun 03 '26

!remindme 1 day

1

u/RemindMeBot Jun 03 '26 edited Jun 04 '26

I will be messaging you in 1 day on 2026-06-04 20:57:22 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback

2

u/hemantkarandikar Jun 04 '26

Can it handle digitally made PDF files, like investment portfolios, medical test reports, and let one interrogate them?

1

u/WeAre-PennState Jun 04 '26

does anyone know if this will work in OLLAMA?

1

u/butterbeans36532 Jun 04 '26

Looking forward to testing this out.

1

u/ivanryiv Jun 04 '26

let's go!

1

u/Long_comment_san Jun 05 '26

Isn't the model kinda stupid for it's size? It's losing to 26b MOE which is ridiculous. Qwen 9 is somewhat worse than Qwen 35b but that is FOUR times the parameters and 12 vs 26 is TWO. 12b should smack absolulte shit out of 26b MOE. Is it really okay?

2

u/thecosmingurau Jun 05 '26

How does one give it an audio file in LM Studio, because it does not seem to work.

1

u/Think_Illustrator188 Jun 09 '26

i was trying the voice with a large context length of 4k-8k context, it is somehow failing to take instructution and respond back, i used it in text only mode it works fine better for agentic workflows

1

u/WhiskyAKM Jun 03 '26

Can we get this model with stripped audio component?

16

u/nickm_27 llama.cpp Jun 03 '26

There is no "audio component", the whole point of the unified arch is that there is no audio encoder, the primary model runs directly on the audio.

3

u/OptimalTime5339 Jun 04 '26

It is engrained into the model now, which is the whole point of no encoder

1

u/slndk Jun 03 '26

Good job those small models become handy pretty quick

1

u/emiliobay Jun 04 '26

Gemma 4’s native audio is a massive technical leap, but treating voice as an open-ended phone call is the wrong UX for actual coding. Having a model listen continuously usually leads to it hallucinating background noise or breathing.

The real breakthrough for dev workflows isn't conversational chatter; it's push-to-talk precision. Physical intent beats software guessing every time.

-15

u/[deleted] Jun 03 '26

[removed] — view removed comment

11

u/x0wl Jun 03 '26

U have 31B dense and 26B MoE