Visible Speech

MouthSample

Face landmarks and blendshapes, ~30 times a second, into the ring buffer above. Everything is divided by the inter-ocular distance, so leaning toward the camera should move nothing. Marks on the aperture bar are the spec's acceptance thresholds: under 0.1 at rest, over 0.7 on an open "ah". Width wants a few seconds of talking before it means anything — it calibrates against the range your own face actually covers.

Transcript

Each word is bound to the moment it was spoken, not the moment the recogniser reported it, so its glyphs carry the mouth state from ~300 ms ago. Weight is jaw opening; width is smile against pucker. Two axes from two independent signals — which is what makes the line read as a mouth rather than as one value smeared across it.

Say "papa" then "meep": the weight contrast should be obvious without measuring. Then say "and then I scream" and let it revise to "ice cream" — the prefix must not change shape at all.

Channel A sends your microphone to Google. Everything else in this project stays on the device; transcription cannot.

The dashed marker is the moment being rendered. Say a vowel and watch whether it crosses the aperture peak as the letters swell — if it does not, LAG is wrong. The green bars along the bottom are the spans each word claims.

Session recorder

Records every MouthSample the landmarker produces, plus the derived intermediates the thresholds are tuned against. Mark segments while you record — a labelled recording is a fixture, and Phases 5 and 6 will want fixtures rather than another afternoon in front of the camera.

Every recording is sent to R2 when you press stop. Download is there for when the upload fails, not as the normal path. Nothing is kept in browser storage.

RingBuffer.sampleAt

The acceptance ramp from the spec: 100 samples at 10 ms spacing, value = i. Marks are stored samples; the line is sampleAt swept continuously across a range wider than the data, so the flat shoulders are the clamp — queries outside the retained window return the nearest end rather than extrapolating a value that was never observed.

Timeline

Every event is pushed with the time it was noticed and how far behind reality its detector runs. The timeline subtracts that latency to recover the time it actually happened, then holds everything back by LAG. Fire the burst and watch the fast detector lose the race it should never have been in.