How Real-Time Lip Sync Works for a Browser AI Avatar
Ayra's mouth moves while she speaks. It looks effortless and is not: it is a two-path system with a subtle timing problem at its heart, and we rebuilt it once because the first design was structurally late. This is how the shipped version works, including what it gets wrong.
TL;DR
- Ayra's lip sync drives mouth morphs from audio loudness/shape — amplitude envelopes precomputed by the server, with a live FFT analyser as fallback.
- It is not phoneme-based: we do not map individual sounds (P, O, M...) to mouth shapes. It is expressive, not frame-accurate articulation.
- The naive approach (just read a WebAudio analyser) is always ~30–60ms behind the audio the viewer hears — structurally, not incidentally. That reads as dubbing.
- Our fix: the backend computes the envelope of the exact PCM it is about to stream, and the browser player schedules mouth shapes against its own audio timeline.
- Honest limitation: some synthesis paths (MPEG audio) ship no envelope and fall back to the analyser.
What lip sync actually has to solve
The requirement is deceptively simple: when sound leaves the speakers, the mouth should be moving as if it made that sound. The moment the mouth visibly lags — even by a few tens of milliseconds — the brain stops believing the character is speaking and starts seeing a puppet. Voice-actor dubbing in films fails at exactly this threshold, for exactly this reason.
So the problem is not "make the mouth move." It is "make the mouth move at the right time, continuously, on consumer hardware, for streaming audio."
Path 1: the live analyser (and why it's late)
The obvious implementation is a WebAudio AnalyserNode attached to the audio element: every frame, read the frequency data, drive the mouth open with loudness. We started there. It works — and it is always late:
- An FFT window of 1024 samples at 32kHz is ~32ms of history — the analyser describes the recent past, not the present.
- The analyser's smoothing constant adds more effective delay.
- The viseme interpolation (how fast the mouth opens/closes) adds more again.
Stack those up and the mouth is reacting to sound the viewer already heard. Each delay is small; the sum is visible. On a stream it reads as dubbing.
Path 2: server envelopes, scheduled on the audio timeline
The fix we shipped uses a fact that was sitting in front of us: the backend synthesizes Ayra's voice from raw text, which means it holds the exact PCM before the browser hears any of it. It can therefore compute an amplitude envelope — the loudness contour over time — of the audio it is about to send, and ship it alongside the audio stream.
Envelope lip-sync flow
- Backend synthesizes a sentence as raw PCM
- Backend computes the amplitude envelope of that exact PCM
- Envelope streams to the browser alongside the audio
- Browser schedules audio chunks on the AudioContext timeline
- Each animation frame: sample the envelope at the exact playback position
- Mouth morphs follow — in sync with what is audible, frame for frame
Because the browser knows precisely where each audio chunk sits on its own timeline, the viseme for any moment is a lookup — currentTime - chunkStart — not a guess from a lagging analyser. This path is more accurate than the fallback in every dimension, and it costs the server one cheap pass over audio it already had in memory.
The general lesson we took from this: when you control the audio source, sync data should ride with the audio, not be re-derived at the far end. Anything re-derived inherits every delay of the derivation pipeline.
When the envelope doesn't exist: the fallback is the point
Some synthesis paths return compressed audio (MPEG) rather than raw PCM — and computing an envelope for those would require a server-side decode first, which we don't currently do. So every frame, the lip-sync system asks: is there an envelope for what's playing? If yes, use it; if no, defer to the FFT analyser for that frame.
The design rule we extracted: null means "fall back," never "close the mouth." A fallback that silently degrades reads as a slightly looser sync; a mouth that snaps shut reads as a bug.
What it looks like on the avatar
Visually, the mouth is morph targets on the VRM model — an "aa" shape for open/loud, an "ou" rounding for certain frequency character, closed at silence — with asymmetric interpolation: the mouth opens fast (it must be open by the time the vowel lands) and closes slower (a natural jaw fall). Blink, brow, and emotion expressions layer on top independently, so a smile arrives on the right sentence while the mouth keeps tracking every syllable underneath it.
Honest limitations
- Not phoneme-accurate: you will never see a precise P or F shape. Amplitude-driven sync is expressive, not articulation-correct.
- Envelope sync only covers synthesis paths that ship raw PCM; other paths use the (slightly late) analyser.
- AudioContext clock jitter on overloaded tabs can momentarily displace chunk starts; the interpolation absorbs most of it, not all.
- Cross-browser timeline behavior differs at the margins; we tune on the major engines.
The avatar this drives is its own story — Building a Real-Time VRM Anime Avatar in the Browser covers the model, expressions, and the mobile budget.