Design a Real-Time Voice Assistant

1. Quick anchor

A real-time voice assistant is not "speech-to-text, then ChatGPT, then text-to-speech." It is a pipeline of streaming stages racing a clock: the moment the user stops talking, you have roughly 800 milliseconds before the silence feels broken, and humans start to perceive lag at 300–500ms. The entire art is overlap — your speech-to-text (STT) emits partial transcripts before the user finishes, your LLM streams tokens before the sentence is done, and your text-to-speech (TTS) starts synthesizing audio off the first clause instead of waiting for the full response. Layered on top of that race are three things that make voice uniquely hard: barge-in (the user can interrupt mid-response and you must instantly stop talking), turn detection (telling "pause while thinking" apart from "I'm done"), and tool calls (where a 600ms backend round-trip blows your whole budget unless you hide it). Get the latency choreography right and the rest is plumbing; get it wrong and no model quality saves you.

2. Why interviewers probe this

• IC5 — Can you build the streaming pipeline correctly? They want to see you decompose time-to-first-audio (TTFA) into a per-stage budget, choose streaming over cascading, and handle the obvious-but-easy-to-botch cases: partial transcript instability, turn-end detection, barge-in. The tell is whether you reason in milliseconds, not in "it should be fast."
• IC5 — Do you know the failure modes that only show up live? Double-talk, transcript flicker, TTS that won't stop when interrupted, context that grows unbounded across a 10-minute call. Naming these unprompted is the signal.
• IC6 — Can you make the system-level tradeoff calls? When do you pick a smaller LLM to win TTFA over a smarter one that's slow? How do you hide tool latency with filler speech without lying to the user? How do you keep a fleet of concurrent calls cheap when each one holds open three streaming connections? They're probing whether you can reason about cost, quality, and latency as a single optimization, not three separate ones.
• IC6 — Architecture under adversarial conditions. Network jitter, a flaky STT provider, a tool that times out, a user on a bad cell connection. Staff candidates design the degradation path, not just the happy path.

3. Concept build-up

3.1 The latency budget is the spec

Before any boxes-and-arrows, write the budget. A realistic 2026 TTFA target is ~800ms, and humans perceive anything above 300–500ms of silence after their turn as unnatural. Here's where it goes:

The LLM's time-to-first-token (TTFT) is the elephant — it's half the budget. This single fact drives most of your design decisions: it's why you stream tokens (so TTS can start before generation finishes), why you reach for a smaller/faster model, and why prompt caching matters (a cached system prompt cuts prefill latency 80–90% on the reused portion). Notice TTS gets only 150ms for the first chunk — you are explicitly not waiting for the full audio, you're synthesizing the first sentence and streaming the rest behind it.

3.2 Streaming architecture, not cascading

A cascading design — STT finishes → hand full transcript to LLM → LLM finishes → hand full text to TTS — is the trap. It's easy to build and it's 300–600ms slower than it needs to be. The streaming version has every stage feeding the next as it produces output:

• STT emits partial transcripts continuously, before the user even stops. You don't wait for the full utterance to start thinking.
• The LLM streams tokens as it generates them.
• TTS begins synthesis on partial text — sentence- or clause-level chunking. The moment the LLM emits "Your order shipped this morning," TTS speaks it while the LLM is still writing "...and should arrive Thursday."

The cost of streaming is complexity: now you have three concurrent streams whose lifecycles you must manage, and you have to handle the fact that partials are unstable.

3.3 Turn detection and transcript instability

Partial transcripts change as more audio arrives — "I want to..." might become "I want two tickets." If you fire the LLM on every partial, you generate against a half-formed query, waste tokens, and produce wrong answers. Two production fixes:

1. Confidence threshold on the partial — only trigger the LLM once STT's confidence on the current transcript crosses a bar.
2. Short-silence window — wait for a brief silence (VAD goes quiet) before invoking the LLM.

The deeper problem is turn detection: distinguishing "I paused to think" from "I'm finished." Tune this wrong toward eager and you talk over the user ("...the address is — let me think — " and you've already barged in with an answer). Tune it toward patient and you feel sluggish. Modern systems use semantic end-of-turn models (is this utterance grammatically/semantically complete?) layered on top of acoustic silence detection, because raw silence alone can't tell a thinking pause from a finished thought.

3.4 Barge-in: the architectural fork

Barge-in is the single design decision that shapes your entire audio layer. The requirement: while the assistant is speaking, the user can interrupt, and the assistant must stop immediately — not finish its sentence. Concretely, when VAD detects user speech during playback you must:

1. Cancel the in-flight TTS synthesis stream.
2. Flush the audio playback buffers — there may be hundreds of milliseconds of already-synthesized audio queued; you must drop it.
3. Restart STT on the new user utterance — without losing the leading audio frames (which is why you keep a small rolling pre-buffer).

This requires tight coupling between the audio playback layer and VAD. It's also why full-duplex (simultaneously playing and listening) is non-negotiable for a good experience — a half-duplex walkie-talkie design physically cannot barge-in. The subtle failure mode is echo: your own TTS output leaks into the mic and VAD thinks the user is talking, causing the assistant to interrupt itself. You need acoustic echo cancellation (AEC) so the system doesn't hear its own voice as a barge-in.

3.5 Context carryover across a long call

A 10-minute call is dozens of turns. You can't replay the full transcript into the LLM every turn — latency and cost grow with context length. Three strategies, often combined:

• Sliding window — keep the last N turns verbatim. Cheap, simple, loses early context.
• Summarization — periodically compress old turns into an LLM-generated summary, keep recent turns raw. Preserves continuity at the cost of a background LLM call and some fidelity loss.
• Retrieval-augmented — fetch relevant past context on demand (e.g. "what did the user say about their address earlier?"). Best for long calls and known-facts recall, adds retrieval latency.

The pragmatic production pattern is sliding window + rolling summary: last ~6 turns verbatim, everything older folded into a running summary updated off the critical path. Keeping context tight also directly helps TTFT — a shorter prompt prefills faster. See /context-engineering for the compression tradeoffs in depth.

3.6 Tool use mid-call

This is where voice agents get genuinely hard. The LLM decides it needs to call a tool — "look up order #4412" — and that tool takes 200–600ms. That round-trip lands inside your 800ms budget and blows it. Three patterns, none free:

• Pre-fetch / parallelize — speculatively fetch likely context (the caller's recent orders) before you know you need it, or fire multiple tool calls in parallel rather than serially.
• Cache — semantic or exact-match caching of tool results that repeat within or across calls.
• Filler speech / acknowledgment — the human trick: say "Let me check that for you..." (which TTS speaks immediately) while the tool call runs in the background. This buys you 1–2 seconds of perceived-natural time. The honesty caveat: the filler must be true — don't say "I found it" before you have.

Tool use also forces a sizing decision: a smaller LLM with good prompts often beats a large one on TTFA, and for tool-routing specifically you can use a cheap model to decide which tool, then a stronger one (or none) to phrase the answer. See /agents for tool-call orchestration patterns.

3.7 The cost and quality knobs

Every quality dial trades against latency or cost:

• LLM size — bigger = smarter but slower TTFT and pricier. On voice, TTFA usually wins, so default smaller-and-well-prompted.
• STT model — larger models are more accurate but slower; voice tolerates slightly noisier transcripts because the LLM is robust to typos.
• TTS — neural voices sound fresh and natural but cost more latency; pre-generated/cached audio for fixed phrases (greetings, fillers) is instant and free.
• Routing — send simple turns ("yes," "no," confirmations) to a tiny model or skip the LLM entirely with rule-based handling.

4. Minimal implementation

This is a shape-accurate (not toy) orchestrator for the streaming loop: it shows the overlap, the barge-in coupling, the partial-transcript gating, and clause-level TTS handoff. The transport/provider clients are abstracted, but the control flow is the part that matters in an interview and in production.

import asyncio
import re
 
# --- Abstracted streaming clients (provider-agnostic) ---
# stt.stream()  -> async iterator of (text, is_final, confidence)
# llm.stream(messages) -> async iterator of token strings
# tts.synthesize(text) -> async iterator of audio chunks (bytes)
# player.play(chunk) / player.flush() control the speaker
# vad.speech_detected() -> bool, on raw mic frames
 
CLAUSE_BOUNDARY = re.compile(r"[.!?,;]\s")
 
class VoiceSession:
    def __init__(self, stt, llm, tts, player, vad):
        self.stt, self.llm, self.tts = stt, llm, tts
        self.player, self.vad = player, vad
        self.history = []            # rolling context (window + summary)
        self.assistant_speaking = False
        self.cancel = asyncio.Event()  # barge-in signal
 
    async def run(self):
        # Two concurrent loops: one transcribes + responds,
        # one watches for barge-in while we speak.
        await asyncio.gather(self._dialog_loop(), self._barge_in_watch())
 
    async def _dialog_loop(self):
        async for text, is_final, conf in self.stt.stream():
            # Gate on turn-end: only respond on a finalized, confident turn.
            # This is what prevents firing on unstable partials.
            if not is_final or conf < 0.6:
                continue
            self.history.append({"role": "user", "content": text})
            await self._respond(self._build_context())
 
    async def _respond(self, messages):
        self.assistant_speaking = True
        self.cancel.clear()
        buffer, spoken = "", ""
        try:
            # Stream LLM tokens; hand each completed clause to TTS immediately.
            async for token in self.llm.stream(messages):
                if self.cancel.is_set():
                    return  # barged-in: abandon generation
                buffer += token
                if CLAUSE_BOUNDARY.search(buffer):
                    clause, buffer = self._split_clause(buffer)
                    spoken += clause
                    await self._speak(clause)   # overlaps with next-token gen
            if buffer.strip() and not self.cancel.is_set():
                spoken += buffer
                await self._speak(buffer)
        finally:
            self.assistant_speaking = False
            # Record only what we actually said (barge-in may have cut us off).
            self.history.append({"role": "assistant", "content": spoken})
 
    async def _speak(self, text):
        async for audio_chunk in self.tts.synthesize(text):
            if self.cancel.is_set():
                self.player.flush()   # drop queued audio instantly
                return
            self.player.play(audio_chunk)
 
    async def _barge_in_watch(self):
        # Tight coupling: VAD + playback. If the user talks while we speak,
        # cancel TTS and flush buffers so we stop within ~one frame.
        while True:
            await asyncio.sleep(0.02)  # 20ms audio frame cadence
            if self.assistant_speaking and self.vad.speech_detected():
                self.cancel.set()
                self.player.flush()
 
    def _build_context(self):
        # Sliding window of recent turns; older turns would be folded
        # into a rolling summary off the critical path (omitted here).
        window = self.history[-6:]
        return [{"role": "system", "content": "You are a concise voice assistant."}] + window
 
    def _split_clause(self, buf):
        m = CLAUSE_BOUNDARY.search(buf)
        idx = m.end()
        return buf[:idx], buf[idx:]

The load-bearing details: _respond hands each clause to TTS the instant it's complete instead of waiting for the full response (3.2); the cancel event is checked in both the LLM loop and the TTS loop so barge-in stops us within one audio frame (3.4); _barge_in_watch is the tight VAD-playback coupling running concurrently; and we record only what was actually spoken so the context reflects reality after an interruption (3.5). The turn-end gate (is_final and conf >= 0.6) is the transcript-instability fix from 3.3.

5. Production tradeoffs

Cost reality. LLM API per-token prices dropped ~80% from 2025→2026, but a voice agent makes many calls per conversation (every turn, plus tool-routing, plus summarization), so the per-call cheapness turns into meaningful per-conversation cost at fleet scale. The dominant cost levers are: route simple turns to cheap models (classification/confirmation turns to a Haiku-class model can be ~12x cheaper than a Sonnet-class model with minimal quality loss), cache the system prompt (80–90% latency cut on the cached prefix, which also helps TTFT), and cache fixed TTS phrases.

Latency reality. Your p90 TTFA is what users feel, not your median. The tail is dominated by LLM TTFT variance and tool-call timeouts. Set hard timeouts on every external call and have a degradation path: if the tool times out, say "I'm having trouble pulling that up — can I take a message?" rather than dead air.

Failure modes that only appear live. (1) Echo/self-barge-in — TTS leaks into the mic, VAD self-interrupts; needs acoustic echo cancellation. (2) Transcript flicker firing the LLM on unstable partials — fixed by the confidence/silence gate. (3) Double-talk — both parties speak; you must decide who yields. (4) Context drift over long calls — summary errors compound. (5) Cascade collapse — one slow stage (a hiccupping STT provider) stalls the whole pipeline; isolate with timeouts and fallbacks. (6) Voice inconsistency — switching TTS voices mid-session breaks the user's mental model; pin one voice per session.

Observability. Per the LLMOps discipline, monitor (predefined metrics) and observe (full traces). Trace each turn end-to-end — VAD → STT → turn-detect → LLM → TTS → playback — with component-level latency so you can attribute a slow turn to a stage. Track p50/p90/p95 TTFA, barge-in rate, transcript revision count, tool-call latency, and per-turn cost. Sample ~10–20% of calls for detailed traces; log basic metrics (tokens, cost, latency) for all. See /inference for the serving-side levers (continuous batching, speculative decoding) that move TTFT.

6. How it's asked

7. Pitfalls & flashcards

• Designing cascading, not streaming. If you wait for STT to fully finish before the LLM, and the LLM to fully finish before TTS, you're 300–600ms slow for free. Overlap or lose.
• Firing the LLM on unstable partials. Gate on confidence or a silence window; partials flicker.
• Forgetting echo cancellation. Without AEC, TTS leaks into the mic and the assistant barges in on itself.
• No barge-in, or barge-in that doesn't flush buffers. Canceling synthesis but still playing queued audio means the user keeps hearing the old response for hundreds of milliseconds.
• Unbounded context. Replaying the full transcript every turn bloats TTFT and cost on long calls. Window + summary.
• Putting tool calls on the critical path with no filler and no timeout. Either hide the latency or own the dead air.
• Optimizing median latency. Users feel the p90. The tail is TTFT variance and tool timeouts — instrument and bound it.
• Switching voices mid-session. Pin one voice; consistency is part of the product.

Flashcard. Voice = overlap stages to hit ~800ms TTFA (LLM TTFT is half the budget); barge-in means cancel TTS + flush buffers within one audio frame; gate the LLM on turn-end, not partials; hide tool latency with filler + pre-fetch + timeout; carry context with window + rolling summary.

8. Further reading

• Smallest.ai — Designing Voice Assistants — the STT-LLM-TTS pipeline, latency budget breakdown, and barge-in handling that anchors this lesson.
• AssemblyAI — Voice Agent Architecture — streaming vs. cascading patterns and orchestration.
• Morph — LLM Inference Optimization — the TTFT levers (prompt caching, continuous batching, speculative decoding) behind the LLM stage of the budget.
• Redis — LLMOps Guide 2026 — model routing and semantic caching for the cost knobs.
• arXiv 2603.04445 — Dynamic Model Routing Survey — cost-quality routing mechanisms for sending simple turns to cheap models.
• Langfuse — LLM Observability — monitoring vs. observability and end-to-end tracing for per-stage latency attribution.

Next: /inference — the serving-side optimizations (continuous batching, speculative decoding, PagedAttention) that directly move the LLM TTFT term in your voice budget.