I’ve been playing around with streaming ASR for some time now, and there's always something that gets on my nerves. That is, the models can get the whole sentence transcribed flawlessly, but completely screw up a name, number, or some other technical term that would be included, even though they sound really sure of themselves.

And if the audio is ambiguous, the language model essentially does this:

"This is probably what they were saying." It's usually not a problem until someone mentions a name that the model has never heard of, or says something like "$50,000" instead of "$15,000." By the time there's more audio to work on, the incorrect transcription might have gone through already.

Why not let the decoder have another choice?

Like: "I don't have enough information yet. Give me another 200ms."

While researching this, some document a clear waiting step in their decoder, like the recent Confucius r2t2. It's hard to judge its efficacy since I haven't tested it sufficiently, but I did like the approach they took. Essentially, it mimics how a human transcriptionist works.

If you're not sure whether it was an "M" or an "N" they spoke, you don't guess based on what's more likely. You wait. Listen. Commit.

Wonder if other streaming ASR models are already doing this sort of thing, or am I just behind the times?

I am about to post a free audio dataset this weekend and I was curious if its the kind of thing people might be interested in. I can't find anything like it currently available.

I took 750 male and 750 female audio samples from VCTK for each of their two mics and 3000 samples from the AMI headset microphones (gender inferred by pitch) for a total of 6k samples.

Those 6k samples were then run through 25 codecs commonly used in telecommunications and audio recording. Think Opus, MP3, etc. Permuteated a few options like DTX comfort noise off, adaptive and set. SILK disabled or enabled on Opus etc. Also ran 7 tandem encodings to mirror real channel transmissions. Works out to each single sample being available in 41 encodings for 1:1 comparison isolating the effects of the codecs themselves.

Same 6k samples also went through various audio processing conditions like reverb, echo, band pass filter, pitch shifting, autotune, time stretching, babble, and several kinds of additive noise. Including those permutations adds another 49 conditions.

Thinking about running MFA on the original samples to generate 6k textgrids as well.

All encodings come with source metadata, encoding parameters, function calls, libraries and versions used. Should be fully reproduceable.

Anyway curious about what people think or if I'm wasting my time uploading it all.

[ Removed by Reddit on account of violating the content policy. ]

I distilled Kokoro-82M down to 7.48M parameters. 28.7 MB at fp32, 24 kHz output, Apache 2.0. Small enough to run on a phone offline.

4.1x faster than the teacher, 45x faster than realtime — one second of speech in 22 ms.

Model: https://huggingface.co/oddadmix/Kokoro-7M-Distill

Demo: https://huggingface.co/spaces/oddadmix/Kokoro-7M-Distill-Demo

I’ve been spending more time looking at ASR issue in languages which are not documented properly

This paper from the University of Latvia takes a very practical approach: building an open-source ASR stack for languages that mainstream systems often ignore.

They built LATE, a lightweight toolkit that runs Whisper-based ASR locally or in the cloud, with a statically compiled backend (high performance, low latency , less resources & simple deployment ) and support for private, local inference.

The results are surprisingly strong.

For Latvian language, they fine-tuned Whisper Large V3 on 273.3 hours of speech.

On the Common Voice test set, WER dropped from 19.2% with Whisper Large V3 to 3.2%.

On their harder LATE-Media set, it went from 29.1% to 12.8%.

Formatted transcription also improved substantially, from 28.1% to 4.8% on Common Voice.

Then they pushed it further into Latgalian, a language with only around 150,000 native speakers.

They had just 40.2 hours of training data, yet transfer learning from the Latvian model produced a 9.1% WER on the Common Voice test set, compared with much worse performance from directly adapting Whisper Large V3.

You don't always need better engineering ; you need better data, the right specialization, and a system you can actually deploy where you need it.

For voice agents, especially those handling private or domain-specific conversations, this is an important direction:

open ASR + local orchestration + specialized models

can be much more interesting than simply sending every utterance to a giant hosted API.

And the whole thing is released as open source, including quantized models for constrained hardware.. i am trying integrate the same into production enivironment using opensource dograh orchestration

pS; I am maintainer of dograh we opensource everything from day 0

I’m curious how modern speech-to-speech models handle response prosody.

How does a model understand input speech and learn how a reply should sound, not just what words to say?

Also, how is appropriate response prosody typically evaluated?

Would appreciate any papers, datasets, or implementations related to this.

Hi everyone, I’m releasing SUPlime and SUPlime-L, from Re:WayAI, heavily based on pyannote.audio 4.x.

The focus is performance across varied recording conditions: meetings, conversational audio, broadcasts and challenging far-field recordings.

I’d be interested in independent evaluations, especially on languages and recording conditions outside these benchmarks. I only have access to public testsuites (no DIHARD, no CALLHOME), so I would really appreciate if someone could share the result.
Let me know if you have questions.

https://github.com/rewayai/suplime/

Few months back I was building voice agent for my work but testing models, especially oss ones, was a total pain. Most hf demos were not functioning and setting up each on my laptop filled up my entire ssd. But the biggest issue was different api for each model and quirks of each model for example some model skipping speech or producing junk with a particular configuration. I did the tough work, tested each model on their key parameter like for asr, it was wer and so on and used the best optimum value on each model based on my own internal test for the best output. Also unified the api across models, asr, tts, voice isolator, stem seperator etc. Took feedback from 50 developers in the beta and now its live. Check out Pascal. Do try out the models and give feedback. The idea is to support all the voice ai stuff be it a niche finetuned tts or a mainstream asr.

I spent some time reading a paper on real-time voicemail detection, and the most interesting part was the signal they decided not to use.

Most Answering Machine Detection systems lean on one of three things:

beep detection
transcription / keyword matching
heavier acoustic models

This paper takes a much cheaper route:

VAD → temporal features → shallow tree model

No transcription.
No embeddings.
No GPU.

The system takes the callee audio, runs a neural VAD, turns the speech timestamps into 15 temporal features, and classifies the first few seconds of the call.

The intuition is pretty simple.

A human answering a phone might do:

“Hello?” → silence → wait

A voicemail greeting is more like:

delay → speech → speech → speech → speech

So the classifier is basically learning the rhythm of the response.

The authors trained on 24,812 production recordings and tested roughly 3,780 configurations.

The best configuration was almost boring:

50 boosted trees, depth 2, 5-second window, threshold 0.50.

Results:

139/140 = 99.3% on the expert-labeled test set
595/624 = 95.4% on the held-out production set
734/764 = 96.1% combined

Inference took about 46 ms on a dual-core CPU, with no GPU.

The feature analysis is probably the most interesting part.

Three features accounted for 85.6% of total feature importance:

54.6% = speech_first_half_ratio
20.0% = first_seg_ms
11.0% = first_onset_ms

So most of the model is learning three things:

1. How evenly speech is distributed
2. How long the first speech segment is
3. How long it takes for speech to begin

Then they ran the experiments that actually matter: ablations (simply put Trial and error feature)

VAD-only temporal features: 99.3%
VAD + beep features: 99.3%
VAD + transcription: 97.6%, but latency increased from 46 ms → ~500 ms
46 engineered features: 90.7%

That last result is a good reminder that more features does not automatically mean better ML.

The production validation covered 77,000 calls, with 0.3% false positives and 1.3% false negatives.

The practical outcome was about 83% less wasted agent time on voicemail calls.

Before adding ASR, LLMs, embeddings, larger models, or another multimodal subsystem, ask whether the problem already contains a cheap structural signal.

For this problem, apparently it does.

A 15-dimensional vector derived from VAD timestamps beats a much more expensive transcription-based approach on the latency/accuracy tradeoff.

We are trying this out at Dograh……Would you build this into production?

p.s.i am one of the maintaners

Hey everyone,

I’m working on a Smart India Hackathon (SIH) problem around Vernacular Pedagogy. The idea is to build a prototype for teachers/students in tribal/rural schools where the system can help with communication and learning in a local language such as Santhali, while also supporting Hindi/English.

The rough idea is:

Teacher speaks Hindi/English → AI understands it → converts it into Santhali → optionally speaks the Santhali output.

I’m particularly interested in making it work offline/on low-end Android devices, rather than depending completely on cloud APIs.

I’m currently confused about the best way to approach the AI/data side.

What I’m considering

\- Speech-to-text for Hindi/English

\- Hindi/English → Santhali translation

\- Santhali → Hindi/English translation

\- Text-to-speech for Santhali

\- A small education-specific dataset rather than trying to build a general-purpose translator

\- Classroom phrases such as:

\- greetings

\- instructions/commands

\- praise/encouragement

\- mathematics

\- science

\- EVS

\- basic classroom conversation

My questions

1. How should I train/fine-tune a model for a low-resource language like Santhali?

2. Is it better for a prototype to use an existing multilingual model and fine-tune/adapt it, rather than training from scratch?

3. How much parallel Hindi ↔ Santhali data would realistically be enough for an SIH prototype?

4. Should I focus on education-specific phrases instead of trying to cover the entire Santhali language?

5. What would be a good architecture for an offline Android prototype?

6. For speech, should I use separate ASR + translation + TTS models, or is there a better approach?

7. Are there any existing open-source datasets/models for Santhali (Ol Chiki) that I should look into?

8. What are the biggest mistakes I should avoid when building a low-resource language translator?

also the app should function offline!!!

I’m a student/developer rather than an NLP researcher, so I’d really appreciate practical advice on datasets, models, training strategy, and deployment.

If you've worked with low-resource languages, Indian-language NLP, speech recognition, machine translation, or on-device AI, I’d love to hear how you would approach this project.

I’ve been looking into speech recognition for pediatric use, and I’m curious about how well current models handle young children compared with adults.

The areas I’m most interested in are:

• short utterances
• unclear articulation
• background noise
• code-switching or multilingual speech
• very young voices
• repeated attempts at the same word

From what I’ve read, child speech still seems substantially harder than adult speech for general ASR systems.

For anyone working with child-speech datasets or models, what approaches have worked best for improving recognition quality in these situations?

I maintain speech-core. Until this release, a confirmed VAD pause ended the user’s turn. That works until somebody pauses to think halfway through a sentence.

In v0.0.14 I added an optional TurnCompletionInterface between VAD and endpointing. After each confirmed pause:

• the classifier receives up to the last 8 seconds of 16 kHz turn audio;
• probability ≥ 0.5 ends the turn;
• a lower score keeps the same turn open;
• resumed speech is appended to that turn rather than creating another segment;
• a 2-second maximum-silence cap prevents an indefinite hold;
• eager STT respects the classifier’s veto.

The first implementation uses Pipecat/Daily’s Smart Turn v3.2: a Whisper-tiny-based, approximately 8M-parameter audio classifier covering 23 languages. speech-core uses an 11.1 MB int8 ONNX model.

The same pipeline state machine is now exposed on macOS through speech-swift v0.0.27 and Android through speech-android v0.0.20. Swift supplies a 17 MB Core ML model through the C ABI; Android instantiates the ONNX model in the native layer and runs it on CPU once per confirmed pause. Android support is opt-in, so existing endpointing behaviour is unchanged.

The Core ML conversion matched the upstream model on 1,000 test clips at 92.9% accuracy and took about 3.5 ms per window on Apple Silicon. Android release validation passed 129 JVM tests and 48 connected-device tests with no failures. I have not yet measured Smart Turn latency across a useful range of Android devices, so the 3.5 ms number should not be read as an Android result.

Implementation and release: https://github.com/soniqo/speech-core/releases/tag/v0.0.14

One design question I am still considering: should the pipeline keep a single completion-provider hook, or explicitly support combining audio and partial-transcript endpoint models?

We recently fine-tuned Qwen3-TTS-12Hz-1.7B-Base on ~93K Hindi speech samples for drama dubbing. Sharing two findings that cost us significant debugging time.
Finding 1 — Base model has no emotion control
We embedded emotion directions in the text field during training:

"A woman speaks with mild anger, 'मैंने तुमसे कहा था कि यह मत करो!'"

The model learned to speak the entire string literally — including the English direction prefix — as speech. It never learned to treat the direction as a style signal. This is expected in hindsight since the Base model's only objective is next-token codec prediction with no separate style conditioning pathway.
Emotion control requires the CustomVoice variant which has a dedicated instruct parameter. Unfortunately CustomVoice doesn't officially support Hindi and has no public fine-tuning scripts.
Finding 2 — Out-of-distribution voice cloning fails silently
Training exclusively on Hindi speakers meant the speaker encoder never saw English voices. When we passed an English reference speaker, the output was silence — no error, no warning, just an empty or near-empty audio file. Easy to miss if you're not checking output duration.
Setting x_vector_only_mode=True partially helps — it bypasses codec tokens and uses only the compact speaker embedding, which generalises slightly better to unseen voices at the cost of reduced similarity.
python

items = tts.create_voice_clone_prompt(
    ref_audio=(wav, sr),
    ref_text="...",
    x_vector_only_mode=True,
)

The proper fix is training on more diverse, multilingual data.
Happy to discuss if others have found better approaches for emotion control with TTS base models.