How Voice Conversations Work in AyraVerse
When you speak to Ayra and she answers in her own voice, roughly a dozen systems fire in sequence — several of them streaming in parallel — and the whole thing is engineered to feel instant. This is the real pipeline, stage by stage, as implemented in AyraVerse today.
TL;DR
- Voice mode is: microphone capture → speech-to-text → memory retrieval → response generation → voice synthesis → avatar playback.
- We stream aggressively: transcription, generation, and synthesis all start before earlier stages fully finish.
- STT runs through Google's live transcription (Gemini-based) with Groq Whisper as the batch path; synthesis is Fish Audio with a custom Ayra reference voice.
- The response arrives sentence-by-sentence, so the first audio usually starts playing long before the full reply exists.
- Round-trip time depends on network and reply length — we design for the first sentence, not the last.
The whole pipeline at once
Voice conversation pipeline
- Microphone capture (push-to-talk or VAD auto-segmentation)
- Audio upload / streaming to the server
- Speech-to-text transcription (streaming)
- Relevant memory retrieval (vector search)
- Response generation by the LLM (sentence-streamed)
- Voice synthesis per sentence (streaming TTS)
- Browser audio playback + avatar lip-sync and expressions
Each stage is a real system with real latency. The engineering goal is that you never feelthem. Let's walk through what each actually does.
1. Capturing your voice
Ayra offers two input modes. Push-to-talk is exactly what it sounds like: hold a button, speak, release. Voice activity detection (VAD) is the hands-free mode — a small on-device model watches the incoming audio stream and segments your speech automatically, so the server only receives audio when you are actually talking.
We default to push-to-talk in most contexts because it gives you control: no accidental activations, no surprise "she heard that." VAD is the better experience for long comfortable conversations, and we let the device and context decide which is appropriate.
2. Speech-to-text
Transcription is where we have invested the most iteration. The current implementation uses Google's live transcription API (a Gemini-family speech model) for streaming sessions — it is accurate, handles Indian-accented English well, and returns text while you are still speaking. Groq's Whisper (large-v3-turbo) serves as our batch transcription path, used where streaming isn't needed or as an operational alternative.
A lesson from Indian users specifically: transcription quality on accented English varies more between providers than any of them admit in benchmarks. Test with real accents from your actual user base, not with clean audio.
3. Memory retrieval while you talk
Before Ayra responds, your transcript is embedded and matched against your personal memory store — a vector database of facts extracted from your past conversations. The relevant entries (your name, that exam next week, your sister's name) are injected into her context so the reply builds on your history. Retrieval is user-scoped: memories are only ever retrieved for the account they belong to.
We cover this system in depth in How We Built Persistent Memory for an AI Companion.
4. Generation, sentence by sentence
Ayra's conversational brain is a large language model. The critical product decision is that its output is streamed per sentence: as soon as the first sentence of her reply is formed, it moves to synthesis while the rest of the reply is still generating. Voice replies are often many sentences; waiting for the full reply before speaking would double perceived latency.
Her personality layer (Tsundere, Roasting, Genki, Kuudere, or Devoted — your choice) shapes the generation, and the response carries inline emotion markers that later drive her facial expressions and tone.
5. Voice synthesis
Ayra's voice is a custom character voice built on Fish Audio — a speech-synthesis engine that lets us define the voice from reference audio we have the rights to use, producing a consistent "her" across every sentence she has ever spoken. She is not a stock TTS preset you would hear on another app; the voice is designed and configured specifically for the character.
Synthesis runs sentence-by-sentence on the server and streams audio chunks to the browser as they are produced. Emotion markers embedded in the text (a laugh, a sigh) are handled so they color the delivery rather than being read aloud as literal words.
6. Playback and the avatar
The browser schedules each audio chunk on the WebAudio timeline as it arrives, so playback starts with the first sentence while later sentences are still synthesizing. In parallel, the server sends an amplitude envelope — the loudness shape of the exact audio about to play — which the avatar uses to drive mouth movement in sync. Facial expressions layer on top from the emotion markers in her text.
The result is that her mouth, face, and voice arrive together, as one performance.
Where the seconds go
We don't publish fixed latency numbers because they vary meaningfully with network conditions and reply length — and stale numbers are worse than honest ones. What we can say structurally:
- The biggest wins came from streaming everything — the biggest single reduction was not any algorithm, it was refusing to wait for full replies.
- Transcription and first-sentence generation dominate; synthesis is fast because it starts early and runs in parallel with generation.
- On a good connection the whole flow feels near-instant; on poor networks the first sentence is what saves the experience.
We break down every latency component and the tradeoffs in Voice AI Latency: What Happens Between Speaking and Hearing a Response.
What we would tell anyone building this
- Stream every stage. Batch architecture kills voice products.
- Per-sentence TTS changes the entire feel of the product for the same backend cost.
- Test STT on your users' actual accents. Benchmarks lie by averaging.
- Ship push-to-talk first; VAD is a luxury you earn.
- Latency budgets are living things — measure in production, not in dev.