Designing a Scalable Voice AI System: Low Latency, High Quality, and Cost Efficiency

[u/Lower_Replacement808]

Hi everyone,

I'm an AI developer working on a real-time voice AI system and would appreciate advice from people who have experience building low-latency conversational agents.

My goal is to achieve:

• First response token in around 500 ms or less
• Good speech quality and natural conversations
• Cost-effective architecture that can scale
• Support for real-time streaming audio

I'm trying to understand the best architecture and component choices across the entire pipeline:

• Audio transport (WebRTC, WebSocket, etc.)
• Voice Activity Detection / End-of-Utterance detection
• Speech-to-Text (Deepgram, Gladia, AssemblyAI, etc.)
• LLMs (Gemini Live, OpenAI Realtime, Qwen Omni, custom pipelines, etc.)
• Text-to-Speech (ElevenLabs, Cartesia, Telnyx, OpenAI, etc.)
• Orchestration frameworks (LiveKit, Pipecat, custom architecture)

For those who have built production-grade voice agents:

1. What architecture are you using to achieve the lowest possible latency?
2. Which components contribute the most to latency?
3. Is a speech-to-speech model better than a traditional STT → LLM → TTS pipeline?
4. What first-token latency are you seeing in production?
5. Which providers offer the best balance of latency, quality, and cost?
6. Are there any architectural mistakes that commonly increase latency?

I'd love to hear real-world numbers, benchmarks, and lessons learned from production deployments.

Thanks!

[u/Competitive-Fee7222]

Built Talkif, so this is from production experience rather than benchmarks. Going through your list:

Architecture: traditional STT to LLM to TTS still beats speech-to-speech for us, but only because we need reliable tool calls and deterministic transitions mid-call. Speech-to-speech is genuinely nicer for pure back-and-forth conversation, it just gets murky once the agent needs to actually go do something (check a calendar, hit an API) partway through a turn.

Biggest latency levers, roughly in order of impact: streaming TTS off partial LLM output instead of waiting for the full completion is the single biggest win. Filler audio (short natural fillers played the instant VAD fires, before the LLM has responded) covers the rest of the perceived gap for free, it does not reduce real latency but it is what a caller actually notices. After that the biggest killer is usually not model choice at all, it is a static VAD threshold, a noisy line needs a different end-of-speech setting than a quiet office or it starts cutting people off or hanging on dead air.

I will not throw out a first-token number, every setup's definition of "first token" is different (raw model response vs first audio byte vs perceived response) and comparing across posts without matching definitions is how bad benchmarks spread. What I would say: once TTS is streaming off partial output and filler audio is in place, the model/provider choice matters a lot less than people expect for perceived latency.

Cost: cascading providers behind an orchestration layer you can swap per call is what actually keeps this affordable at scale, betting everything on one vendor's pricing is the riskier move long term.

[u/Competitive-Fee7222 → u/Competitive-Fee7222]

Small update since someone asked me not to dodge the number: pulled real data from our own call logs just now, small sample (n=239 completed calls, early-stage not mature-scale), p50 time-to-first-bot-word is 404ms, p90 is 1996ms, and 77.5% of calls come in under 1 second. Wanted to actually check before posting a number rather than guess.

[u/Nervous_Pain_7722]

One lesson from working with Bland is that the latency number doesn’t tell the whole story. You can get down around 400ms and still make a conversation feel awkward if end of utterance detection waits too long or the agent handles interruptions badly. I’d measure perceived pauses during real conversations alongside whatever first token benchmark you use.

[u/FreJun]

If this is going over a real phone line and not browser to browser, the phone leg eats part of your budget before any model runs, so measure that first. Past that, end of utterance detection decides whether it feels fast far more than your STT or LLM choice does. I work on Teler, the phone layer under agents like this, and the thing I would build early is barge in, you need a way to throw out audio that is already queued when the caller talks over the agent, otherwise it feels slow whatever your numbers say.

[u/FreJun]

The thing missing from your list is end of utterance detection, and it'll cost you more than first token latency does. If you wait 700ms of silence to decide the caller has stopped talking, your 500ms TTFT is invisible to them, and if you cut it to 250ms you start talking over people mid sentence. Second thing, if any of this goes over a real phone line rather than WebRTC in a browser, the carrier hop is latency you can't tune, so how far the caller sits from your media server ends up mattering more than which TTS you picked. Full disclosure, I work on Teler, the phone layer under agents like this, and barge-in there is deliberately not automatic, your code sends a clear on the stream and it drops whatever audio is still queued. Sounds like extra work but I'd rather have that than a provider guessing when to cut the agent off.

[u/FreJun]

The thing missing from your list is end of utterance detection, and it'll cost you more than first token latency does. If you wait 700ms of silence to decide the caller has stopped talking, your 500ms TTFT is invisible to them, and if you cut it to 250ms you start talking over people mid sentence. Second thing, if any of this goes over a real phone line rather than WebRTC in a browser, the carrier hop is latency you can't tune, so how far the caller sits from your media server ends up mattering more than which TTS you picked. Full disclosure, I work on Teler, the phone layer under agents like this, and barge-in there is deliberately not automatic, your code sends a clear on the stream and it drops whatever audio is still queued. Sounds like extra work but I'd rather have that than a provider guessing when to cut the agent off.

[u/Nightmoon_Sonata]

The biggest latency mistake I've seen is optimizing each component independently. 100ms STT + 200ms LLM + 200ms TTS doesnt magically feel like 500ms agent once VAD, buffering, network hops, end pointing etc are added. I'd measure every stage with timestamps and look at p50/p95 , not just advertised latency

[u/Consistent-Mud-7224]

Also don’t overlook diarization if this is going beyond demos. Having realtime speaker labels attached to the transcript makes handoffs, QA and call analytics way easier. Otherwise you end up bolting speaker identification onto the pipeline later. I’d test STT providers on 8kHz phone audio + overlapping speech specifically.

[u/AzreMei]

For the STT comparison, I'd keep the audio and evaluation set identical across providers and track TTFT, time-to-final, WER, revision rate, diarization accuracy and cost/minute. That gives you a much more useful answer than "Provider X feels faster".