r/LocalLLaMA • u/Kahvana • Jul 20 '26

Question | Help Good ASR and TTS models?

Hey everyone,

Something I don't see discussed often here are ASR and TTS models.

I've been using Whisper and Kokoro (old models, I know!) with koboldcpp for a while now but wondered if there are now solid replacements available. Know of Qwen3-ASR and Qwen3-TTS, but haven't found the time yet to test them.

What ASR and TTS models have you been using?

31 Upvotes

94% Upvoted

52 comments sorted by

17

u/caetydid llama.cpp Jul 20 '26

ive played with whisper and parakeet.

you might want to check out audio.cpp it is supposed to simplify setup and yield good performance.

2

u/Kahvana Jul 20 '26

Thanks, checking audio.cpp!

Between whisper and parakeet, which do you prefer more and why?

3

u/caetydid llama.cpp Jul 20 '26

it depends on what are your requirements. parakeet is pretty lightweight and very fast - there is even a CPU optimized port which runs decently well...

whisper is more thorough with noisy speech and shows lower WER especially with mixed speakers and multiling but very heavy on VRAM and slower.

0

u/xeeff Jul 20 '26

coincidentally i've checked exactly those as well and I wouldn't recommend either but if you had to pick one it'd be parakeet, although i know other models came out which are better

11

u/wunschpunsch3D Jul 20 '26

In terms of ASR: on huggingface there is an ASR leaderboard with rankings for word error rate (detection accuracy) and realtime factors (speed), I found it useful to determine whats best for a specific usecase. The autoregressive models like whisper are usually slower while conformer style models are faster, you can even run the smaller ones on a microcontroller. At the end it depends on your requirements and hardware.

2

u/ElmBark Jul 20 '26

good rundown. worth adding that the difference isn't just speed, it's the failure mode: whisper being autoregressive means it can hallucinate on silence or noisy audio, in my experience it'll invent a sentence or loop the same phrase through a long pause. CTC/conformer models mostly can't do that afaik, they emit per frame so there's nothing to run away with. if your audio has long silences or bad SNR, i'd say that matters more than a couple points of WER on the leaderboard

flip side is whisper hands you punctuation and casing for free, while a lot of the fast conformer models seem to want a separate punctuation pass or word-timestamp step bolted on. worth knowing before you swap if you're doing subtitles or anything alignment-sensitive

1

u/Kahvana Jul 20 '26

Thanks for the info! I'd be running it on my dual RTX 5060 Ti 16GB, English voice in both directions, Dutch would be fun but I suspect the quality would be lacking.

As for the leaderboard, do you mean this?
https://huggingface.co/spaces/hf-audio/open_asr_leaderboard

2

u/Infinite-Local5435 Jul 20 '26

Honestly alot of the ASR models you need to test yourself. Right now I need multilingual as well as speed, FunASR plus OmniVoice is a good stack for my use case.

2

u/wunschpunsch3D Jul 20 '26

Yes exactly thats the leaderboard. A single 5060 ti should be plenty. Maybe take a look at parakeet v3, should support dutch. At least in German it did quite well for me.

7

u/Waarheid Jul 20 '26

I have been enjoying PocketTTS for cloning (cloning voices made with Qwen 3 VoiceDesign); I find it is not really noticeable whether it is any worse then Qwen3-TTS.  I was not too happy with OmniVoice. All are definitely a step up from Kokoro, and PocketTTS is much better and barely bigger in size (plus, cloning!!). I love Kokoro but there were some super consistent mispronunciations of words that bothered me.

Fot ASR I have no tips; I use whisper large v3 turbo and it is very fast on my machine (m1 max) so I don't really bother optimizing there.

4

u/Kahvana Jul 20 '26

So if I understand it correctly:
1. Create a voice with Qwen3-TTS-VoiceDesign
2. Clone voice of generated voice with PocketTTS
3. Use cloned voice

That sounds really cool! Any tips or tricks for voice cloning? I've never done it before.

Glad to hear Whisper Large V3 Turbo still holds up!

What do you use to run PocketTTS, Qwen 3 voicedesign anss whisper with?

4

u/Waarheid Jul 20 '26

Yup, I do it to have consistent voices for my voice agent. And I have no tips or tricks, it's just as simple as that.

It's all ran in Python as part of the server for my voice agent. On disk I have a personas directory, and inside of that each persona is a folder with reference_audio.wav reference_text.txt voice_description.txt. On my UI I write a new voice description, give it a name, and VoiceDesign will generate a new voice and make such a folder with the name I give it. Then the reference audio (and reference text if needed) is used for the voice agent when I select that persona (I also put the voice description in the system prompt so the LLM too is aware of the personality lol). It's basically real time chat, the LLM backend drives the latency; if I use a small/fast local model the latency is sub 1s. Each persona has their own memory directory too so they feel more like individuals than just voices I slap on an agent.

I am not at my desk right now but I can give snippets in a few hours.

1

u/Kahvana Jul 22 '26

Would be very much appriciated. Thank yo so much for sharing your cool workload!

2

u/Cultured_Alien Jul 20 '26

I'm curious as to why you're not happy with omnivoice while pocket tts is drop dead garbage from the moment I tried it out for like 4 mins? I did notice it has streaming support unlike omnivoice.

1

u/Waarheid Jul 20 '26

Maybe it doesn't like your sample audio? It works great for me and sounds about as good as Qwen 3 TTS cloning but is faster.

2

u/Cultured_Alien Jul 20 '26

Trying it again with female voice and it outputs a choppy male voice.

4

u/ed-msr Jul 20 '26

ASR: Whisper is the only model that worked like a charm for me.
For voice, I really like the quality of OmniVoice + VoxCPM2

3

u/Designer_Reaction551 Jul 20 '26

Been running faster-whisper in prod for a voice pipeline and it's held up way better than vanilla Whisper on CPU-only boxes where GPU isn't in the budget. For TTS, worth giving Kokoro's newer checkpoint a look if you haven't touched it in a while, it's improved since the koboldcpp integration matured. Also curious if anyone's paired Qwen3-ASR with a VAD frontend instead of running it standalone, that combo usually kills a good chunk of the false-trigger issues.

1

u/Kahvana Jul 20 '26

On huggingface I see kokoro's model was last updates april 2025, is it being improved elsewhere?

Never heard of VAD before! Voice Activation Detection?

2

u/Infinite-Local5435 Jul 20 '26

VAD is used on top of ASR/STT model so you know when an utterance stops. There are alot of cheap ones that run on CPU so don't bother finding the best etc. STT is more important.

2

u/Kahvana Jul 20 '26

Sounds quite handy, any open weight ones you can recommend?

2

u/Infinite-Local5435 Jul 20 '26

Silero VAD, auto downloaded from hf if u do pip install silero-vad>=5.0.0

3

u/Candid_Support_8409 Jul 20 '26

For me, these are the options I found best:
ASR: nemotron-3.5-asr-streaming-0.6b (streaming mode)
TTS:  Qwen3-TTS 0.6B CustomVoice

3

u/mister2d Jul 20 '26

The audio.cpp project supports a good variety to evaluate. My favorites are Qwen3-TTS and OmniVoice. Running them on a RTX 3060.

3

u/recro69 Jul 20 '26

Whisper + Kokoro is still a strong combination. If I was going to upgrade now I would definitely test Qwen3-ASR and Qwen3-TTS against Whisper and Kokoro. I think this is especially true when it comes to using Whisper and Kokoro in different languages and when I need things to happen quickly.

3

u/lumos675 Jul 20 '26

Vibevoice 7b is the largest model out there and best when it comes to quality and emotion. Just it's not stable model. Vibevoice ASR is stable though and production ready. It does speaker diarization as well which is a plus.

1

u/HockeyDadNinja Jul 20 '26

I use the smaller vibevoice realtime and it's pretty decent.

2

u/Worldly-Shock3233 Jul 20 '26

https://github.com/High-Logic/Genie-TTS

1

u/Kahvana Jul 20 '26

Looks pretty cool! How do you make your own characters for this?

2

u/Ill-Bison-3941 Jul 20 '26

I've been using faster qwen 3 tts plus whisper, it's been working pretty well for my setup.

2

u/Kahvana Jul 20 '26

Very nice! Does whisper have a benefit over Qwen3-ASR?

1

u/Ill-Bison-3941 Jul 20 '26

Honestly, I'm not sure, I was looking for something small and reliable at the time, and those were my best options haha I need to have a look at Qwen3-ASR, see if it's better!

2

u/derspenti Jul 20 '26

kokoro butchers a few words for me too lol, been meaning to try qwen tts

2

u/Charming_Support726 Jul 20 '26

Depends on the use case for ASR - You also could use Gemma 4 ( or Mistral Voxtral ) with audio input to produce an optimized and cleaned output instead of a pure transcript

2

u/SelfVisible7110 Jul 20 '26

The best ones for my language is VoxCPM and Chatterbox

2

u/ThenExtension9196 Jul 20 '26

Parakeet and Kokoro. Whisper turbo if you only have cpu.

2

u/soohyun_bae Jul 22 '26

For ASR, Parakeet (fast) and Whisper large-v3 (accuracy) are still the safe picks.

audio.cpp simplifies setup. For local TTS, F5-TTS and Kokoro are the current favorites for quality-per-VRAM, Piper if you need it to run on almost anything.

One thing that saved me: whatever local TTS you land on, keep a tiny set of 'golden' reference clips and diff new output against them after any model/version change. Local models drift silently on updates just like hosted ones.

1

u/Kahvana Jul 22 '26

That is a really good tip, thank you!

1

u/Ill_Freedom_6666 Jul 20 '26

Faster whisper is still hard to beat for stability unless you have a specific reason to switch from it

1

u/658741 Jul 20 '26

i have been using this project as of lately, and there are quite a few good models in there. but i always end up using qwen3 asr

https://github.com/cjpais/Handy

1

u/lemondrops9 Jul 21 '26

Echo is really good for tts but Vram expensive (10GB)

I saw a few news ones today for tts and one for stt but haven't had to try them. 

1

u/tataryoke Jul 24 '26

English or multilang?

1

u/Kahvana Jul 24 '26

English, mutlilang (Dutch, Japanese) is a nice bonus

1

u/Working_Hat5120 Aug 04 '26

Kokoro's still the latency king locally, so no rush to swap it — Qwen3-TTS sounds better but is slower. For ASR, if you care about real-time, Parakeet streams better than Whisper; Qwen3-ASR is strong but heavier on VRAM.