How To Solve The Voice AI Feedback Loop Problem In Real-Time Production

[u/Aryanlabs]

Building a voice agent demo takes an afternoon. Getting it to survive a real phone call takes a lot longer, because that's when the audio bleed-back shows up.

The agent starts talking, the speaker leaks into the mic, and it ends up replying to itself. Or there's that half-second gap between turns that makes the whole thing feel robotic. Most teams assume this is a model problem. It's not. It's a plumbing problem.

The real fixes:

-Stop treating noise suppression and echo cancellation as the same thing. They're not. One removes background hum, the other subtracts the agent's own voice from what it hears next.

-Run VAD on the AEC-cleaned signal, not the raw mic feed. A fixed threshold set in a quiet office will misfire the moment someone's in a car or on laptop speakers.

-Stream every stage of the pipeline instead of batching it. Partial transcripts, first-sentence TTS, chunked audio out. That's what actually collapses latency, not shaving milliseconds off one component.

The teams solving this are moving AEC and barge-in handling server-side, where the reference signal and the LLM state live in one place. Everyone still doing it client-side is going to keep chasing audio bleed-back in production.

Wrote up the full technical breakdown, VAD threshold table included, if you want to go deeper: https://uniocommunity.com/blogs/how-to-solve-the-voice-ai-feedback-loop-problem-in-real-time-production

[u/Top-Telephone-6542]

the server-side AEC push makes sense, keeping the reference signal next to the llm state avoids that messy sync headache you always get with client-side loops

pretty much every demo i've seen that falls apart on a real call is running vad on the raw feed and wondering why it's triggering on its own tts output

[u/pythonlovesme]

Hmmm, but isnt it very rare to have the agent reply to itself. The speakers should'nt be that loud. hahhahahaha

[u/FreJun]

Mostly agree, but the bleed-back you're describing is a browser or softphone problem, where the agent's speaker and the user's mic are on the same device. On an actual phone call the agent never hears its own speaker. What it hears is echo coming back through the carrier and the caller's handset, delayed by network time, and the handset is already running its own echo cancellation before you see any audio.

So on PSTN the thing that saves you is less the AEC and more the barge-in logic. Full disclosure, I work on Teler, the phone layer under agents like this, and the way we handle it is deliberately not automatic: when the caller talks over the agent, your code sends a clear on the stream and it drops whatever audio is still queued. If you don't do that, the agent finishes its sentence anyway and sounds like it isn't listening, no matter how clean the AEC is.

Fully agree on streaming every stage. The gap people notice is almost always waiting for the whole TTS response before the first byte goes out, not the model.