Visible Speech

MouthSample

Face landmarks and blendshapes, ~30 times a second, into the ring buffer above. Everything is divided by the inter-ocular distance, so leaning toward the camera should move nothing. Marks on the aperture bar are the spec's acceptance thresholds: under 0.1 at rest, over 0.7 on an open "ah". Width wants a few seconds of talking before it means anything — it calibrates against the range your own face actually covers.

Transcript

Each word is bound to the moment it was spoken, not the moment the recogniser reported it, so its glyphs carry the mouth state from ~300 ms ago. Weight is jaw opening; width is smile against pucker. Two axes from two independent signals — which is what makes the line read as a mouth rather than as one value smeared across it.

Say "papa" then "meep": the weight contrast should be obvious without measuring. Then say "and then I scream" and let it revise to "ice cream" — the prefix must not change shape at all.

Channel A sends your microphone to Google. Everything else in this project stays on the device; transcription cannot.

The dashed marker is the moment being rendered. Say a vowel and watch whether it crosses the aperture peak as the letters swell — if it does not, LAG is wrong. The green bars along the bottom are the spans each word claims.

AudioFrame

Channel B's front end. The spectrum, zero-crossing rate and pitch are computed on every animation frame; the level envelope comes off the audio thread every 2.7 ms, which is the only reason a ten-millisecond burst can be seen at all. Nothing here leaves the device.

The bars hold their extremes — the thin tick is the highest (or, for flatness, the lowest) value since the last reset. A plosive is over before the screen has refreshed once, so a meter showing only the current value cannot show you one.

acceptance

Session recorder

Records every MouthSample the landmarker produces and every AudioFrame the analyser produces, plus the derived intermediates the thresholds are tuned against and the raw ~375 Hz level envelope. Mark segments while you record — a labelled recording is a fixture, and Phase 5 detects trills by FFT-ing that envelope, so a marked trill in a recording is a detector that can be written and tested without going back to the microphone.

record everything opens the camera, the microphone and the recogniser and starts recording, then stops all four together — the raw video is never stored, only landmarks. The individual buttons in the panels above still work; this is for when you want a take rather than an experiment.

Every recording is sent to R2 when you press stop. Download is there for when the upload fails, not as the normal path. Nothing is kept in browser storage.

RingBuffer.sampleAt

The acceptance ramp from the spec: 100 samples at 10 ms spacing, value = i. Marks are stored samples; the line is sampleAt swept continuously across a range wider than the data, so the flat shoulders are the clamp — queries outside the retained window return the nearest end rather than extrapolating a value that was never observed.

Timeline

Every event is pushed with the time it was noticed and how far behind reality its detector runs. The timeline subtracts that latency to recover the time it actually happened, then holds everything back by LAG. Fire the burst and watch the fast detector lose the race it should never have been in.