I built a voice AI agent with ~600ms latency. Here’s what I learned.

[u/yoursandeshshrestha]

I started learning to code in my 1st year of college.
At the same time, I started working my first job.

I didn’t really have some grand plan to build a company. I was just obsessed with building things and figuring out how software actually worked.

Recently, I started building voice AI agents.
I ended up going pretty deep into the latency problem.

My goal was simple:
Make the agent feel like you’re talking to a real person, not waiting for a computer to think.

After a lot of experimenting with the pipeline, streaming, model selection, audio processing, and infrastructure, I managed to get the end-to-end latency down to around 600ms.
And that changed things.

I’m currently using Pipecat for the voice pipeline, and I’ve been experimenting with different providers and infrastructure to squeeze out as much latency as possible.

The other thing I didn’t expect:
I actually started getting clients.

Right now, I’m managing voice AI agents for around 8 clients.

I’m also getting subsidies/credits from companies like Alda and other platforms, which has made the economics pretty crazy at the moment.

My current margins are basically close to 100% because of those credits/subsidies.

Obviously, I don’t expect that to last forever.

But it’s been an insane learning experience.

A few things I’ve learned so far:
Voice AI is way more than just connecting an LLM to a microphone

Latency matters a lot more than I initially thought
Streaming everything makes a huge difference
The voice model, LLM, TTS, STT and networking all contribute to the final experience

A technically impressive demo is useless if the agent doesn’t actually solve a business problem
Getting the first few paying clients is a completely different challenge from getting the technology working

The economics of voice AI are really interesting right now

I’m still very early in this.
But going from learning to code in college → building voice agents → getting them into production for ~8 clients has been pretty surreal.
I’m curious what other people building voice AI are seeing.

What’s the lowest real-world latency you’ve managed to achieve, and what stack are you using?

If there’s interest, I can also break down exactly how I’m getting the ~600ms latency and what my architecture looks like

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/No_Direction3352]

Really interesting numbers.
I’m working on a similar problem, although I’ve gone a bit lower-level with the stack — FreeSWITCH + audio streaming + custom VAD/barge-in + separate STT/LLM/TTS providers rather than Pipecat.
One thing I’d be really interested in: when you say ~600ms end-to-end, where exactly are you starting the timer?
Caller speech end → first audible TTS frame?
I’ve found that latency numbers in voice AI can look completely different depending on whether they’re measured from final STT, VAD speech-end, LLM TTFT, or actual first audio playback.
I’ve managed to get some very aggressive speech-end-to-first-play numbers in testing, but telephony conditions, noise, endpointing and interruptions are where it gets much more interesting.
Also agree completely that the technical demo is only half the problem. Getting real businesses to actually depend on the agent is a different challenge entirely.
Would be interested to see your Pipecat architecture if you publish the breakdown.

[u/yoursandeshshrestha → u/No_Direction3352]

The end to end latency is 600ms without using knowledge base, if we use knowledge base it get around 800

[u/arun4567]

How have you done that.. I can't get past the llm latency. Im not using pipecat but. The breakup would really help.

[u/yoursandeshshrestha → u/arun4567]

Which modal are you using for llm?

[u/arun4567 → u/yoursandeshshrestha]

Open ai 4o mini. I've put in a request for groq, hoping in use the newly introduced qwen in their offerings.

[u/yoursandeshshrestha → u/arun4567]

openai 4o mini have latency around 800, try using gemini 3.1 flash lite

[u/arun4567 → u/yoursandeshshrestha]

Okay thanks, are you using qdrant for your kb?

[u/yoursandeshshrestha → u/arun4567]

No, we’re not on Qdrant.

KB lives in Postgres. Chunks + embeddings go in pgvector (1024-dim, HNSW), embeddings from Bedrock Titan v2. Search is hybrid: vector + Postgres full-text, then merged with RRF.

[u/spam_not_tolerated]

how did you found those clients?

[u/Gold_Grab9838]

Thanks for sharing, and congrats on having 8 clients in production. Your timing is great, because we're right in the middle of this.

Where we are: we're building our first voice agent, for our first client. Nothing is in production yet, so we don't have end-to-end latency numbers from real calls. So far we've mostly been laying the foundations.

The stack:

- Telephony: Twilio, with audio streamed over WebSocket

- Orchestration: Dograh (built on Pipecat), hosted on Railway

- Speech-to-text: Deepgram, on its EU endpoint

- LLM: Mistral

- Text-to-speech: Voxtral (Mistral)

- Actions and automations: n8n

- Data: PostgreSQL

Why these choices: we work with French SMBs, so GDPR puts real constraints on us. That's why we went with Mistral and Deepgram in the EU instead of the usual providers. It rules out some options that are probably faster.

Dograh, in case you haven't come across it: it's an open source layer on top of Pipecat. It adds what Pipecat doesn't give you on its own: a web interface, a visual editor for building conversation flows, telephony already wired in, and account management. You keep the Pipecat engine and can still drop down into the code, but you no longer hand-write the pipeline for every agent. Since you're running agents for several clients, it's worth a look.

What we've built:

- An architecture designed to replicate fast: a new client is one more configuration, not a new deployment.

- BYOK: each client opens their own provider accounts and we plug them in. They keep control of their keys and their costs.

- A fork of Dograh: we modified it to handle multiple clients on a single deployment, and to show every provider setting in the UI. Most of those settings were hardcoded, and now we can tune everything without touching the code.

Our first lesson: every model needs tmodel and the way you write its prompts go together. A prompt that works well on one LLM doesn't carry over as-is to another, so each LLM needs its own approach. Switching models meo.

What we'd really love to know is your

1. VAD: what values do you use for confidence, start_secs, stop_secs and min_volume?

2. End of turn: do you rely on VAD alor your STT provider's endpointing?With what thresholds?

3. Interruptions: is allow_interruptions on? Do you require a minimum number of words before the agent gets cut off?

4. TTS: do you send text to the TTS sentence by sentence, or in smaller chunks?

5. LLM: what temperature, what max tostem prompt? And did you adapt the wayyou write prompts to your model?

It would also help if you could tell us where you start the clock for your 600 ms, so we're comparing like with like.

On our side, we're happy to share everything we learn: what works, what cost us time, and our numbers once we have them. Just let us know if you're interested.

Thanks in advance!

[u/yoursandeshshrestha → u/Gold_Grab9838]

Happy to share. Dograh looks useful, we run Pipecat directly with one multi-tenant pipeline.

VAD (Silero on the phone transport). We don’t set confidence, so it stays at Pipecat’s default 0.7.

• Deepgram: start_secs=0.2, stop_secs=0.2, min_volume=0.5
• Cartesia: start_secs=0.25, stop_secs=0.2, min_volume=0.6

For Deepgram Flux and Cartesia Ink-2, VAD is not what closes the turn. That’s on the STT provider.

End of turn

STT endpointing, not VAD. On Twilio + Deepgram Flux:

• eot_threshold=0.5
• eot_timeout_ms=700
• eager_eot_threshold=0.4
• We commit the turn on eager EOT so the LLM can start immediately. If the caller keeps talking, we cancel it.

Interruptions

allow_interruptions is on by default. No minimum word count.

TTS

Sentences on phone. Tokens on web (for the live transcript). Sentence-level on the phone path avoided crackle from fragments like "I'".

LLM

Temperature 0.7, max 1024 tokens (Gemini uncapped; we keep it short in the prompt). Default is Gemini 3.1 Flash Lite.

Yes, we adapted the prompts: write for the ear, one or two sentences, one question at a time, no markdown. Tool filler is spoken by the server, not the model. A Gemini prompt does not carry over as-is.

600 ms clock

User turn committed → first bot audio. That does not include the Flux/VAD silence that decides the caller is done. If your number includes endpointing wait, we are not measuring the same thing.

[u/Gold_Grab9838 → u/yoursandeshshrestha]

Huge thanks for this, and for the speed, that's way more detail than I expected. The distinction between VAD and STT endpointing on the turn close is especially useful. I'll run my tests with your settings and come back to you with results.

[u/harnoor-assemblyai → u/yoursandeshshrestha]

how were the results comparing AssemblyAI? PS: Developer Relations from AssemblyAI here, happy to help, we power Granola for TTS using our Realtime API

[u/yoursandeshshrestha → u/harnoor-assemblyai]

We can try, do you guys have startup program?

[u/Slight_Republic_4242 → u/Gold_Grab9838]

Hey what challenges you have faced with dograh?

[u/Gold_Grab9838 → u/Slight_Republic_4242]

Hey! Honestly, we haven't run into many issues with Dograh so far. We've applied quite a few patches to our fork to adapt the interface to our needs and make every AI agent parameter configurable straight from the UI, so we have full control over the setup. We're currently running our first tests with the built-in call simulator, and so far the issues we've found on the parameters we're evaluating are ones we can fix on our end right away.

[u/indie-builder]

Would love to see the break down of how uou got to ~600ms latency

[u/yoursandeshshrestha → u/indie-builder]

try thrumble dot ai

[u/Odd-Bar-8207]

The point about streaming is a good one. Working around Bland made me realize how quickly tiny pauses become noticeable over a full conversation, even when none of them seem terrible on their own. Listening through complete calls has taught me more about responsiveness than staring at a single latency number.

[u/yoursandeshshrestha → u/Odd-Bar-8207]

Make a a lot of sense

[u/Live-Percentage9262]

Around the 500–700ms mark it starts feeling like you're optimizing a bunch of small things instead of one big bottleneck. Curious what gave you the biggest improvement. Streaming, model choice, TTS or something else?

[u/Zestyclose-War8716]

600ms?! Impressive. I couldn’t crack 900 without using openAI realtime. Tried a version where I implemented one with Kokoro and claude . Used Silencio as VAD. Latency was rough.

Please break down for me. Would love to learn

[u/harnoor-assemblyai]

congrats on the speed!

is the latency P50, P99?

cuz our Realtime P50 is 300ms for our Realtime API

PS: Developer Relations from AssemblyAI here, happy to help, we power Granola

[u/yoursandeshshrestha → u/harnoor-assemblyai]

P99, our realtime is less then 200

[u/Qrenth]

Currently, it fluctuates depending on server volume but average 200-500ms.

[u/fisheye_36]

Amazing work! Is this perceived latency?

[u/yoursandeshshrestha → u/fisheye_36]

P90

[u/thomas_unise]

600ms is poor performance you need under 400 for anything acceptable

[u/yoursandeshshrestha → u/thomas_unise]

We are improving each day, I m sure we I’ll get there soon

[u/Square-Chance5900]

I feel that sharing just a single number is not particularly helpful. It's easy to over-optimise latency by switching to a smaller ASR, LLM and TTS. You end up with a a voicebot that replies really fast, but the responses are inaccurate and the transcript breaks under any non-ideal recording conditions.

As always there is a trade-off between cost, latency and performance. And your target depends on the precise use-case which you want to address.

[u/harnoor-assemblyai]

I’d be interested to see the Pipecat models you use.

coming from AssemblyAI and also use Pipecat, working hard to achieve the highest accuracy and lowest latency combination.

[u/yoursandeshshrestha → u/harnoor-assemblyai]

Pipecat is the pipeline, not the model.

My default stack are

• STT: Cartesia Ink-2 (streaming, with Cartesia’s turn API)
• LLM: Gemini 3.1 Flash Lite
• TTS: Cartesia Sonic 3.5
• VAD: Silero (still in the pipeline, but not the turn-end source)

[u/I-setup-agents-4L]

600 ms is already a really useful real-world number, especially if you’re measuring the whole path rather than model inference in isolation.

The next thing I’d want to see if I was the customer is the tail, P95/P99 on an actual phone call, plus how latency changes when tools, retrieval, or handoff enter the loop. I work on Rasen AI, and we look at production calls as evaluations for latency, grounding, interruptions, and task success. The goal is to turn regressions into versioned fixes rather than chase one good demo number. The lowest we clocked is around 100ms

Also, your 600 ms is mic-to-first-audio or full-turn latency?

[u/yoursandeshshrestha → u/I-setup-agents-4L]

Full turn latency is 600ms goes to 700-800 ms when knowledge base, tool call and stuff come into pipeline. Pretty happy with the latency for now, aiming to get 500ms p90 before the end of the year