The first voice agent build usually plays audio to completion. The caller starts speaking three seconds into a fifteen-second menu explanation and is ignored until it finishes. Everyone has been on that call and everyone hates it.
What barge-in actually requires
Interrupting is not one feature. It is four things that must all work within a couple of hundred milliseconds:
Detect speech while playing. Voice activity detection running on the inbound stream at the same time as outbound audio is playing. That means full duplex — a half-duplex pipeline cannot do this at all.
Stop the audio immediately. Not at the end of the current sentence or buffer. Flush the playback queue on detection. Anything still buffered downstream will keep playing after you have stopped sending.
Cancel the generation. If the model is still producing tokens and the TTS is still synthesising, both need aborting. Otherwise you pay for output nobody hears, and the next turn arrives late because the pipeline is still draining.
Reconcile what was actually heard. The caller heard part of a sentence. The conversation state must record what was played, not what was generated, or the agent will assume it communicated something it did not.
That last point is the one most often missed and it produces the strangest failures — an agent referring back to information it never actually delivered.
Echo is what makes it hard
The microphone hears the speaker. Without acoustic echo cancellation your VAD triggers on the agent’s own voice, the agent interrupts itself, and the call degrades into stuttering.
On a telephony path AEC usually exists in the carrier leg. On WebRTC it is the browser’s job and generally works. Mixing the two — a SIP trunk bridged into a browser session — is where it silently stops working, and it is worth testing that specific path deliberately.
Tune the threshold against real audio
VAD sensitivity is a trade. Too eager and a cough, a door, or background conversation stops the agent mid-sentence. Too relaxed and the caller has to raise their voice to be heard, which they experience as being ignored.
There is no correct default. Tune it against recordings from the actual deployment — a hotel lobby, a call centre, a car — because the noise floor is what determines the right number, and it is different in every one of those.