Building a Real-Time VRM Anime Avatar in the Browser
Ayra is a real-time 3D anime character who lives in your browser tab. There is no video file behind her — she is a rigged 3D model rendered by your GPU every frame, blinking and breathing and mouthing words as she speaks to you. This article is how that actually works, written by the team that built it.
TL;DR
- Ayra is a VRM model — an open avatar format built on glTF — rendered with Three.js and @pixiv/three-vrm.
- Expressions (blinks, smiles, blushing) are morph targets on the model, blended at runtime; they are not pre-rendered video.
- The same renderer runs on the marketing homepage and inside the product, with a static poster fallback for weak devices.
- The biggest constraint is mobile GPUs and a ~19 MB model — we budget for it instead of pretending it away.
- It is not magic: there are real limits, and we list ours at the end.
Why VRM and not a video or Live2D sprite
A companion needs to react in real time — to speak with synced mouth movement, to look at you when you move your cursor, to smile mid-sentence. Pre-rendered video cannot do any of that. 2D sprite rigs (Live2D-style) can, but every expression must be hand-authored per layer, and head-turn freedom is limited.
VRM is an open 3D avatar format from the VTuber ecosystem, built on top of glTF 2.0. It was designed for exactly this use case: humanoid anime-style characters, real-time rendering, standardized facial expressions (via VRM blendshapes), and a documented bone layout. Because it is an open standard, models work across tools — the same file an artist rigs in one editor can be loaded in a browser, a VR app, or a streaming avatar tool.
We render with Three.js (the WebGL library most of the 3D web runs on) and @pixiv/three-vrm, the reference VRM runtime maintained under pixiv. That combination is what the entire AyraVerse renderer is built on — from her idle breathing to her lip-synced speech.
What loads when
Avatar load pipeline
- Browser fetches the .vrm file (glTF binary with rigged mesh)
- GLTFLoader parses the binary; VRMLoaderPlugin resolves VRM extensions
- Skeleton, morph targets, and materials become Three.js objects
- First frame renders; idle systems (blink, breath, look-at) start
The model is a single binary asset. Once parsed, we have a full skeleton (hips, spine, arms, fingers…), a set of morph targets for facial expressions, and skinned mesh materials. VRM's loader normalizes the model to a known coordinate system so head-tracking and expression code can be written once against any VRM model.
Expressions are morph targets, blended at runtime
VRM defines standardized expression channels — happy, angry, sad, surprised, blink, look-at, and more. Each is a morph target: a stored deformation of the face mesh. When Ayra smiles, the renderer is blending the "happy" morph weight up over a few frames; when she blushes, that is another channel, often driven as a texture overlay.
Ayra's responses carry emotional intent. Her reply text includes markers for emotional state, and the renderer maps them to expression weights — smiling on a happy line, softening on a serious one. Blinks are procedural: on a timer with randomized intervals, closing and opening over roughly a tenth of a second, because a character that never blinks reads as dead.
A detail that surprised us: blinking matters more than almost any "fancy" expression. Users forgive a lot when the character blinks naturally, and notice immediately when she doesn't.
Idle motion: the low-cost trick that sells presence
A body frozen between inputs feels like a statue. Our idle system layers a handful of cheap procedural motions on the skeleton each frame:
- Breathing — a slow sine driving small spine/chest rotations.
- Gentle sway and head bob — low-frequency offsets so she is never perfectly still.
- Look-at — her head and eyes track your cursor (desktop) using VRM's look-at rig.
- One-shot greetings — a short authored wave/tilt animation when she first loads.
None of this requires animation clips. It is all math on bone rotations — cheap enough to run alongside everything else.
Mouth movement while speaking
When Ayra speaks, her mouth needs to move with the audio. We do not run phoneme-perfect viseme mapping; we drive mouth morphs from the audio signal itself — a server-computed amplitude envelope when it is available, and a WebAudio analyser as fallback. The mouth opens with loudness and rounds with frequency character. It reads convincingly at conversational speed, and it costs almost nothing.
We cover the full mechanism — including why an analyser alone is always slightly late — in How Real-Time Lip Sync Works.
The performance budget, honestly
Here is the part marketing pages skip. A rigged, expressive VRM model is not free:
- The model file is around 19 MB — noticeable on slow connections, so we treat it as a progressive enhancement, not a gate.
- Skinned-mesh rendering plus morph targets is real GPU work. Fine on modern laptops; the limiting factor is weak mobile GPUs.
- We skip heavy post-processing. The look comes from lighting and the model's materials, not expensive shaders.
On the marketing site, the live avatar is desktop-only: mobile devices, reduced-motion users, and low-core-count devices get a static poster instead. That is a deliberate tradeoff — the homepage must load fast for everyone; the 3D experience is for devices that can actually render it. Inside the product, where the avatar is the experience, she loads for everyone, with quality expectations set honestly.
If we could redo one thing: ship a decimated (lower-poly) variant from day one rather than treating model size as a later optimization. "Optimize later" tends to mean "never."
Known limitations
- WebGL is at the mercy of the device. On very old GPUs the fallback path is a static image, not a degraded 3D scene.
- Lip sync is amplitude-driven, not phoneme-driven — it is expressive, not frame-accurate articulation.
- Physics (hair, cloth) is minimal; full soft-body simulation is not worth the frame budget on mid-range phones.
- Browsers differ in WebGL implementation details; we test on the major engines but exotic configurations exist.
Why this matters for a companion
Every choice above serves one goal: presence. A companion you can see, who looks at you, breathes, blinks, and speaks with a moving mouth, is a categorically different experience from a text box — even with an identical language model behind it. The 3D avatar is not decoration for Ayra; it is half the product.