Latency is a feature: the engineering of real-time speech

Most discussions of speech synthesis focus on how a voice sounds. In a conversation, how quickly it sounds matters just as much. A reply that arrives a moment too late reads as hesitation, confusion or indifference, and no amount of acoustic quality can rescue it. We have come to treat latency not as a performance metric to be optimized at the end, but as a product feature that shapes the architecture from the first design decision. This post walks through where the time goes and the trade-offs we weigh at each step.

Why delay breaks conversation

Human conversation runs on remarkably tight timing. Across many languages, the typical gap between one person finishing a turn and the next person starting is on the order of a few hundred milliseconds, often shorter than the time it would take to plan a sentence from scratch. People manage this by predicting where a turn will end and preparing their response while the other person is still talking.

Machines break that rhythm in ways listeners feel before they can name. A short delay reads as thoughtfulness. A longer one reads as a system that did not hear you, and people start to repeat themselves, or talk over the response when it finally arrives. Past a certain point the interaction stops feeling like a dialogue and starts feeling like leaving voicemails. The threshold is not a single number; it depends on context, on what was asked, and on whether the system signals that it is working. But the qualitative pattern is consistent enough that we design against it deliberately.

Latency is not a property of the server. It is a property of the conversation.

Where the time goes

A voice exchange is a pipeline, and every stage spends part of the budget. Audio has to be captured and sent; speech recognition has to decide what was said and, harder, that the person has actually finished; a language model has to produce a response; the synthesizer has to turn that text into sound; and the audio has to travel back and start playing. Network hops sit between most of these stages.

The metric we watch most closely is time-to-first-audio: the interval between the end of the user's speech and the first sound of the reply. Total generation time matters much less, because once audio starts flowing, the listener is occupied. A system that begins speaking quickly and keeps up with real-time playback will feel faster than one that produces the whole answer marginally sooner but stays silent until it is done.

That observation drives almost everything else. It means every stage should stream, passing partial results forward as soon as they are usable rather than waiting for completion. Recognition streams partial transcripts, the language model streams tokens, and synthesis must be able to begin on a fragment of text and continue as more arrives.

Chunking and lookahead

Streaming synthesis sounds simple until you try it. Speech is not a sequence of independent words. The way a word is pronounced, its pitch and duration and stress, depends on what comes after it. A sentence that ends in a question rises at the end, and a good speaker anticipates that rise several words earlier. A model that synthesizes each word the instant it arrives will sound choppy and flat.

So the central trade-off is lookahead. The more upcoming text the synthesizer can see before committing to a sound, the more natural the prosody, and the longer the listener waits. We think about this in terms of chunks: how much text to gather before starting, where to break, and how much context to carry across boundaries. Breaking at punctuation and clause boundaries tends to preserve natural phrasing; breaking mid-phrase saves time but risks audible seams.

One approach we find promising is to make the first chunk small and subsequent chunks larger. The opening words get out fast, which is what the listener notices, and later chunks have the luxury of more context because playback is already buying time. Another is to let the model condition on text it has not yet spoken, so that a chunk can be rendered with awareness of what follows even when the following audio has not been generated. Neither is free: both require the model and the serving system to be designed together, which is one of the reasons we build our own.

There is also a subtle interaction with language models. Their output is generated token by token and is not always well-formed until a clause is complete. Numbers, abbreviations and names may arrive in pieces. Text normalization, which converts "3.5" or "Dr." into speakable words, has to decide when it has seen enough to be confident, and a wrong early guess cannot be taken back once it has been spoken.

Barge-in and turn-taking

Real conversations are not strictly alternating. People interrupt, add a correction halfway through, or say "mm-hm" without meaning to take the floor. A system that cannot handle this feels rigid in a way that is hard to fix after the fact.

Barge-in, letting the user interrupt the system mid-reply, requires several things working together. The system has to keep listening while it speaks, which means separating the user's voice from its own output echoing back through the microphone. It has to decide quickly whether a sound is a genuine interruption or a cough, a backchannel or background noise. And when it does stop, it has to stop cleanly, cut playback promptly, cancel generation upstream, and keep track of how much of its reply was actually heard, so the next turn does not assume the user absorbed words they never received.

The mirror problem is endpointing: knowing when the user has finished. Waiting for a fixed stretch of silence is simple and robust, but it adds that silence directly to every response. People pause mid-thought, especially when working through something complicated, and a system that jumps in at every pause is worse than one that waits. We're exploring endpointing that uses more than silence, drawing on intonation, syntactic completeness and the content of the partial transcript, so the system can respond quickly when a turn is clearly finished and wait patiently when it is not. It is possible to begin preparing a response speculatively before the endpoint is certain and discard it if the user keeps talking, trading some wasted compute for responsiveness.

Infrastructure choices

Much of the latency budget is decided by infrastructure rather than models. Physical distance matters: every round trip between the user and a distant data centre adds delay that no model optimization can recover, which argues for running inference in regions close to the people using it. With teams and users spread across North America and Europe, that is a practical concern for us rather than a theoretical one.

Transport matters too. Long-lived, bidirectional connections avoid the overhead of setting up a fresh request for each chunk, and protocols designed for real-time media handle jitter and packet loss more gracefully than those designed for documents. On the serving side, keeping models warm, avoiding cold starts, and batching requests without letting any individual stream wait in a queue are all small decisions whose costs add up.

We also measure carefully and at the right place. Averages hide the experience that drives people away; the slow tail is what users remember. We track distributions of time-to-first-audio as experienced at the client, not just inside our servers, because the network and device contribute delay that server-side timing never sees.

What we're working toward

None of these problems has a single clean solution, and the right balance shifts with the product. A reading assistant can afford more lookahead than a live conversational agent; a noisy environment demands more conservative barge-in than a quiet one. What we are building is less a fixed pipeline than a set of components designed to stream, to be interrupted, and to expose their trade-offs as explicit settings. Our early experiments suggest that the biggest gains come not from making any single stage dramatically faster, but from making every stage willing to start before it is finished. That, more than any benchmark, is what makes a voice feel present.