Building a Real-Time Voice AI Agent with Barge-In (Streaming STT to LLM to TTS)

The difference between a voice bot that feels like a phone call and one that feels like a walkie-talkie is one behavior: can you interrupt it? On the voice-AI pipeline I built at Peasy, the whole thing lived or died on barge-in — the caller starts talking over the bot, and the bot stops instantly instead of steamrolling through its sentence. Miss that and every user hates it within ten seconds.

The pipeline is a streaming chain over WebSockets: Google Cloud Speech (STT) → OpenAI (streaming) → MiniMax (TTS). Audio in, audio out, all three stages overlapping in time. The hard part isn't wiring the APIs together — it's making them overlap without turning into a laggy mess.

Don't wait for full turns — stream every stage

The naive version does this in strict steps: record the caller's whole utterance, transcribe it, send the full text to the model, wait for the full completion, synthesize the whole thing, play it. Every step blocks the next. The caller sits in silence for seconds. It feels dead.

The version that feels alive overlaps all of it. Audio streams into STT continuously. The moment the model starts emitting tokens, I don't wait for the full response — I chunk it into sentences and start synthesizing and playing the first sentence while the model is still writing the second.

// As the LLM streams tokens, flush a chunk to TTS at each sentence boundary.
let buffer = "";
for await (const delta of llmStream) {
  buffer += delta;
  if (/[.!?]\s$/.test(buffer)) {   // a sentence just finished
    ttsQueue.enqueue(buffer.trim()); // synthesize + play now, don't wait for the rest
    buffer = "";
  }
}
if (buffer.trim()) ttsQueue.enqueue(buffer.trim());

Starting TTS at the first sentence boundary is the single biggest win for perceived latency. The caller hears the bot begin answering while the model is still thinking about the end of its own reply. That overlap is what collapses the dead air.

Barge-in: a state machine, not an afterthought

Barge-in is where it gets genuinely tricky, because now you have audio flowing both ways at once. The bot is speaking (TTS audio going out) while the caller's mic is still open (audio coming in). When incoming speech is detected during playback, three things have to happen immediately, in order:

1. Stop the TTS playback — cut the current audio out.
2. Flush the queued sentences — everything the bot was about to say is now stale; drop it.
3. Cancel the in-flight LLM stream — the answer it was generating is answering a question the caller just changed.

function onCallerSpeechDetected() {
  if (state === "speaking") {
    ttsPlayer.stop();       // 1. shut up now
    ttsQueue.clear();       // 2. drop what we were about to say
    llmStream?.abort();     // 3. the old answer is obsolete
    state = "listening";
  }
}

If you skip step 2, the bot goes quiet, the caller finishes their new sentence, and then the bot resumes vomiting the old answer out of the queue — which is worse than no interruption at all. The queue has to be treated as disposable the instant the human speaks.

The real enemy is buffering, not model speed

People assume the model is the bottleneck. In practice, the thing that wrecks a voice pipeline is buffering at the boundaries — an audio chunk waiting for a full buffer to fill, a TTS call that won't start until it has the whole sentence, a WebSocket frame sized too large. Every stage has to be tuned to emit small and emit early. I care more about how fast the first audio sample comes out than how fast the whole response completes, because humans measure a conversation by the gap before the reply starts, not its total length.

My honest opinion after building this: get barge-in and first-sentence streaming working before you tune anything else. A voice agent that talks fast but can't be interrupted feels robotic; one that's a touch slower but yields the floor instantly feels human. Turn-taking beats raw speed.

Takeaways

• Overlap every stage. Stream STT in, chunk the LLM output at sentence boundaries, and start TTS on the first sentence — don't wait for full turns.
• Treat barge-in as a state machine: on detected speech, stop playback, flush the queue, and abort the in-flight LLM stream.
• The queued response is disposable the moment the caller speaks — never resume a stale answer.
• Optimize for time-to-first-audio, not total response time. That's what makes it feel like a call.

Voice AI isn't three API calls in a row. It's three streams sharing a clock, and a strict rule that the human always gets the floor.

---

I build real-time, streaming, and AI systems end-to-end. If you're working on something similar, let's talk.