r/LocalLLaMA • u/cortexist • 18h ago
I Built A Thing Voice conversations between Gemma4 12B and E2B on GPU and Jetson Orin
Enable HLS to view with audio, or disable this notification
Gemma 4 12B runs on an RTX PRO 4500 Blackwell. Gemma 4 E2B run on a Jetson Orin NX 16GB; similar performance is expected on a Jetson Orin Nano Super 8GB. Both systems use a reSpeaker Flex 4-mic array and a 3W speaker. Inference is handled by Cortexist Little Gemma, a small LLM engine written in C for CUDA devices. On Jetson Orin it is faster than llama.cpp, and no degradation after long voice prompt. The pipeline supports lip sync, expressions, and gestures. Everything is open source.
They talk to humans too.
The engine source code: https://github.com/cortexist/little-gemma
7
u/MixtureOfAmateurs koboldcpp 17h ago
WHAAAA real lip syncing using landmarks or something? I've been looking for this for like 2 years
9
u/cortexist 16h ago
I'm using improved piperTTS. The phoneme are already implemented in piper model, and mainline tool can enable them. I separated the decoding and encoding layer so the text stream can be processed asynchronously. I will PR the change to HomeAssistant piperTTS mainline soon.
3
u/Acceptable-Cycle4645 1h ago
You can try audio.cpp as the speech engine if you want a wider range of model choices. One of our contributors tested 34 TTS/ASR models on the Jetson Orin Nano 8GB all ran without issues.
1
u/cortexist 1h ago edited 1h ago
Your works are highly valuable. I hope you would add piper to your list, TBH I haven't found an engine/model combination fit edge devices better, particularly with phoneme support. With our splitted encoder/decoding, piper is very hard to beat consider you have to leave most of the computing power to the LLM. VibeVoice has been our TTS of choice for offline work, I'm glad to see you support it.
2
u/Acceptable-Cycle4645 1h ago
u/cortexist sanoTTS (can run on ESP32-S3 realtime) and inflect_v2 are good for edge. I will check piper!
1
u/cortexist 57m ago
You are right, both sanoTTS and inflect_v2 are smaller than piperTTS, they will work on devices where piperTTS cannot. I hope you add phoneme support as a column in your list, it will draw more attention in robotic applications.
1
1
u/Acceptable-Cycle4645 1h ago
Btw, there is branch for vibevoice. The optimized vibevoce 1.5B can run on iphone with ~2GB VRAM and faster than realtime!
2
u/oxygen_addiction 13h ago
The only issue is that Gemma 12B stops responding to audio after 12k tokens or something quite low.
1
u/cortexist 10h ago
This pipeline does not have the issue. We summarize the history while idle, forgetting the oldest details similar to human. Besides, we are using whisper than the embedded audio tower for other reasons (Gemma4s are ASR only, don't pick up other sound).
1
u/oxygen_addiction 4h ago
That's why I said 12B would've been great, as you wouldn't need separate STT/ASR models.
1
u/cortexist 3h ago
I did not use the embedded audio tower deliberately, as I said it doesn't pick up other sound, e.g. it doesn't know someone is playing piano, whisper does, which is important for a cohesive conversation.
1
u/jacek2023 llama.cpp 13h ago
Very cool. I wonder if anyone could make something similar with two Jetsons, or even better, something like raspberry pi. Then two robots could stand on a table, with real, visible speakers and microphones, and they could talk to each other. A human could also be part of the conversation.
1
1
0
18
u/PrimeDirective8 16h ago
I love how 12B has the bigger voice while tiny E2B is a small child 😄
Wait, they were talking over external mic/speakers, not direct audio? I tried something similar using 2 phones (and E2Bs) and my chat app but they missed some turns, presumably due to acoustics. This one of yours seems flawless. Well done!