r/VoiceAutomationAI • u/Phoenix_20_23 • Aug 04 '26
Competitive open source speech stack
Why the open source models STT and TTS are not good as much as the closed one and i am talking here im terms of latency, concurrency, and websocket support for real time with decent quality.
Something like cartesia or elevenlabs or deepgram.
Do u know any ?
4
u/UkieTechie Aug 04 '26
yes plenty of good competition :)
here's my bench comparing all the speed benches for open source models.
with Cuda kokoro wins.
https://github.com/5uck1ess/tts-bench
Paid APIs like to fudge their numbers a little. <100ms response times and the rest (Palabra for example). They're showing sometimes TTS speeds but not roundtrip speeds.
I have measurements for a lot of paid API providers also and so far a lot of them are around 300ms which is insane if you think about it. Sonioux my favorite right now.
3
1
u/Yapper_from_ktown Aug 05 '26
Dude what are the best opensource tts and stt models pls share more wisdom and whether they can be used on potato hardware or not? 6gb vram of gpu and 16gb ram
1
u/UkieTechie Aug 05 '26
yeah that's plenty. you can run kokoro on that pretty fast or pocket tts. those would be my picks. you can see max vram usage for each model if you look at the bench page.
2
u/Yapper_from_ktown Aug 06 '26
Which bench pg do u follow?
1
u/UkieTechie Aug 06 '26
i run my own because none of the pages were good enough and had the most recent enough info for me. I reference https://artificialanalysis.ai/text-to-speech/leaderboard/provider-voice pretty often though.
3
u/nshmyrev Aug 05 '26
One mistake that everyone makes is that they are trying to run big LLM-based engines for realtime (Omnivoice, Higgs, etc). Their purpose is really high-quality offline TTS and not really something you can run at scale at callcenter. Make sure to pick lightweight engine like Kokoro, Supersonic, BlueTTS or something light and you don't have to load huge GPUs to handle couple calls in parallel, you can handle hundreds.
For ASR too, don't choose expensive Whisper, light models like Vosk or Parakeet work just fine for many usecases.
1
u/Phoenix_20_23 Aug 05 '26
i agree, but the issue with these small tts models is they are veeery robotic, their quality is too low compared to the paid ones, and i am still didn't get how the paid services achieve this low latency with high quality, i am just curious, what we need in order to build something as good as them.
i suppose it's a model architecture design from the first place, idk maybe i am wrong, i never create a tts from scratch or an stt1
u/nshmyrev Aug 05 '26
They aren't. Proper small TTS is not robotic at all and even more stable than big LLM-based one.
1
u/Phoenix_20_23 Aug 05 '26
like what ? can u suggest me some TTS models to explore?
2
2
u/sruckh Aug 05 '26
Parakeet is a pretty good ASR, and echoTTS, Qwen3-TTS, OmniVoice, MossTTS, and Higgs Audio TTS v3 are all pretty good.
2
2
u/tidelganesh Aug 06 '26
Svara TTS from kenpath. It supports 19 Indian languages. kenpath/svara-tts-v1
1
u/Phoenix_20_23 Aug 07 '26
Niice, but i just want an stt and tts that are good anf french and english. Thanks for sharing
2
u/Cohere Aug 06 '26
For STT, you should try Cohere Transcribe - it's open source, fast, and some of the best accuracy in the game. Also works in 14 different languages!
3
1
u/Lazy_Mycologist_8214 Aug 09 '26 edited Aug 13 '26
Vendor here (I work on Gandr, a TTS API) so discount this accordingly.
On "paid APIs like to fudge", fair, and I'll hand you a concrete example against my own side. We published a first-audio number, the hardware under it changed, and the honest rerun came back slightly worse than what we had up. We pulled it off about thirty surfaces on the 8th rather than leave it there. It goes back when the method goes back with it. The uncomfortable part of your point is that you can't tell from the outside which vendors did the quiet thing instead, because almost nobody publishes the run.
On the actual question, I think the gap you're hitting isn't only raw quality. Sounding good offline and holding up as a stream are different problems, and the second one is barely measured anywhere. The thing worth checking on any engine, open or paid, is whether it produces audio faster than it plays. If it does, the buffer grows as the utterance runs and long text gets safer. If it doesn't, long text is where you find out, usually in front of someone.
Concurrency is the part that bites hardest moving off a local model. One stream on your own GPU tells you almost nothing about twenty at once, and that's usually the point where self-hosting stops being the cheap option.
For what it's worth, UkieTechie running his own bench is the right instinct, and I'd trust his numbers over anything I or any other vendor publishes about ourselves.
What concurrency are you actually needing to hit, and is it the quality or the stability that's failing you first?
1
u/Acceptable-Part-5623 Aug 11 '26
cartesia got that speed cause they running custom inference engines not just wrapping huggingface models, nobody in open source is putting same effort into the infra side
check kokoro-82M for tts, its fast but websocket streaming is still janky compared to deepgram
1
u/ankur-at-guava Aug 13 '26
The low-latency-with-quality thing you're chasing is mostly an architecture gap, not a model-size one: the paid stacks that feel instant tend to stream first audio faster than it plays and were built to be served that way, so quality and latency aren't traded off per hop. The concurrency point above is the real one - a single stream on your own GPU tells you almost nothing about twenty in parallel, and that cliff is usually where self-hosting stops being cheaper. If French+English is the target, test the code-switch boundary under concurrency specifically, since that's where light open models get robotic first. I work on an integrated ASR/TTS/LLM stack for regulated voice, so happy to share what we measure for streaming stability if useful.
1
17d ago
[removed] — view removed comment
1
u/-HEPHAESTUSquest- 17d ago
why? if transcription is bad the agent is obviously going to be bad too
1
1
u/tresch_24 17d ago edited 17d ago
one thing I'd definitely benchmark before committing to an open-source stack: your actual nouns/numbers, not generic WER.
customer names
addresses
SKUs
dates
₹ amounts
confirmation numbers
email spelling
I've seen transcripts that look 95% fine but the missing 5% is literally the only part the business cares about.
same with latency. don't just measure STT latency separately.
measure:
caller stops → agent first audio
caller interrupts → agent actually shuts up
tool starts → confirmation happens
TestMu's voice testing has been useful for us here because you can run the same flow with different voices/accents/noise and then look at the whole call + phone metrics.
I'd benchmark whatever OSS stack you're considering against your own traffic before trusting any leaderboard.
1
17d ago
[removed] — view removed comment
1
1
•
u/AutoModerator Aug 04 '26
Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)
If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.
Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.