Voice AI Latency: What Happens Between Speaking and Hearing a Response
The gap between finishing your sentence and hearing "her" begin is the single most studied number in voice AI. This article breaks that gap into its actual components, explains what each costs and why, and describes the real tradeoffs we make at AyraVerse — including why the honest answer about our own latency is "it depends, and here is what it depends on."
TL;DR
- A voice round-trip is seven stages: recording, upload, transcription, retrieval, generation, synthesis, playback — and each adds its own delay.
- The two dominant costs are transcription and first-sentence generation; synthesis matters most for how early it can start.
- Streaming is the whole game: parallel, incremental stages are what make voice feel instant.
- Latency numbers depend on network, reply length, and provider load — be suspicious of any fixed "2-second" marketing claim, including about us.
- Every optimization trades something: accuracy, cost, naturalness, or battery.
The seven stages of a spoken reply
Where the time goes
- Recording — audio is captured from the microphone
- Upload — audio travels to a server (or browser STT engine)
- Transcription — speech becomes text
- Retrieval — relevant memories are fetched and injected
- Generation — the model writes the reply, token by token
- Synthesis — text becomes audio, sentence by sentence
- Playback — the browser plays it while the avatar animates
What follows is what each stage actually costs and the non-obvious parts we have learned maintaining them.
Recording and upload: the part nobody counts
Before any AI runs, there is pure physics: microphone buffering, encoding, and network transit. Audio at conversational quality is a modest stream, but on a weak mobile connection upload can quietly dominate the budget — and it is invisible in any server-side measurement, which is why products feel slower than their dashboards claim.
Streaming upload (sending audio while you speak, not after) is the standard fix. It overlaps upload with the only thing that must happen anyway: you talking.
Transcription: accuracy vs speed
Speech-to-text is the first big AI stage. The tradeoff space here is well-defined: larger models are more accurate but slower; streaming models return partial results while you speak; batch models return one polished result after you finish. AyraVerse uses Google's live transcription (Gemini-family) for streaming sessions and Groq's Whisper as our batch path — each chosen for where it sits in that tradeoff space.
The under-discussed latency factor in STT is accent robustness. A model that mishears Indian-accented English creates invisible latency of a different kind: repeated corrections, rephrasing, and user fatigue. Speed that sacrifices comprehension is not speed.
Retrieval: small but nonzero
Memory retrieval — embedding your message and searching your memory store — is the cheapest AI stage in the pipeline, typically tens of milliseconds. It matters for a different reason: it can run while transcription is still finishing, so a well-pipelined system hides it entirely.
Generation: the token tax
The language model writes the reply token by token, and tokens per second is the hard floor you cannot engineer around — only choose (bigger models write slower; "fast" model tiers trade a little quality for speed). The product-level lever is reply length: a companion that answers in three sentences is structurally faster than one that answers in fifteen, which is a personality design decision as much as an engineering one.
The stage that matters is not the full reply, though. It is the first sentence. Everything after generation of sentence one can overlap with synthesis and playback of sentence one.
Synthesis: start early, stream always
Text-to-speech per sentence, streamed as chunks, is what lets the first sentence of audio play while the rest of the reply is still being written. Ayra's synthesis (Fish Audio, with her custom voice) starts the moment a sentence closes. A batch-TTS design — wait for the whole reply, synthesize, then play — would roughly double perceived latency for long replies, and no amount of model speed fixes that.
Playback: the last 100ms
Browser audio scheduling has its own small costs (AudioContext warm-up, buffer priming), and the avatar's lip-sync must line up with the audio timeline. This stage is engineering hygiene rather than a frontier — but the first time a user clicks a voice button, AudioContext warm-up can add a beat of silence that feels like the AI thinking. Pre-warming the audio context on page load is the kind of invisible fix that users experience as "this app is just faster."
The tradeoff table
There is no free latency win. Every optimization below buys speed by spending something else:
| Optimization | Buys you | Costs you |
|---|---|---|
| Streaming STT | Transcription overlaps your speech | Partial results that may be revised |
| Streaming upload | Upload disappears from the budget | Slightly more complex client code |
| Per-sentence TTS | First audio plays early | Choppier prosody across sentence joins |
| Faster/smaller LLM tier | Lower generation floor | Some reasoning/personality quality |
| Shorter replies by design | Less to generate | Companion feels more terse |
| Bigger STT model | Fewer mishearings | Later first transcript |
Why we don't publish a fixed number
AyraVerse could market "2–4 second responses" — many voice products do — but on a weak network with a long reply, that number is fiction, and on a good network with a short reply it's pessimistic. Latency in this product depends on your connection, the reply's length, and provider load. What we commit to structurally: every stage streams, nothing waits for a full reply, and the first sentence is what we optimize, because that is what your ear judges.
The concrete implementation of all of this — our exact pipeline and what we learned building it — is in How Voice Conversations Work in AyraVerse.