Visible Speech

MouthSample

Face landmarks and blendshapes, ~30 times a second, into the ring buffer above. Everything is divided by the inter-ocular distance, so leaning toward the camera should move nothing. Marks on the aperture bar are the spec's acceptance thresholds: under 0.1 at rest, over 0.7 on an open "ah". Width wants a few seconds of talking before it means anything — it calibrates against the range your own face actually covers.

Transcript

Each word is bound to the moment it was spoken, not the moment the recogniser reported it, so its glyphs carry the mouth state from ~300 ms ago. Weight is jaw opening; width is smile against pucker. Two axes from two independent signals — which is what makes the line read as a mouth rather than as one value smeared across it.

Say "papa" then "meep": the weight contrast should be obvious without measuring. Then say "and then I scream" and let it revise to "ice cream" — the prefix must not change shape at all.

Non-lexical sounds are drawn rather than spelled, inline and at the moment they were made. Say "and then — pow — it stopped": the burst must land between then and it, not before and. ASR reports a word ~300 ms late and the plosive detector ~15 ms late, so the burst is genuinely detected first — the timeline puts it back where it happened. A faded mark is one the camera could not confirm.

Channel A sends your microphone to Google. Everything else in this project stays on the device; transcription cannot.

The dashed marker is the moment being rendered. Say a vowel and watch whether it crosses the aperture peak as the letters swell — if it does not, LAG is wrong. The green bars along the bottom are the spans each word claims.

AudioFrame

Channel B's front end. The spectrum, zero-crossing rate and pitch are computed on every animation frame; the level envelope comes off the audio thread every 2.7 ms, which is the only reason a ten-millisecond burst can be seen at all. Nothing here leaves the device.

The bars hold their extremes — the thin tick is the highest (or, for flatness, the lowest) value since the last reset. A plosive is over before the screen has refreshed once, so a meter showing only the current value cannot show you one.

acceptance

Trill

The amplitude envelope is transformed, not the audio. FFT-ing audio finds pitch; a trill's identity is how fast its loudness pulses — a 683 ms window, every 85 ms, looking between 18 and 36 Hz. cycles is then a real count of how many times the lips closed, which is what stops the number of r's in brrrrr from being a guess about emphasis.

A peak alone means nothing — noise always has a largest bin somewhere. What marks an oscillator is that the peak stays put: five consecutive windows agreeing within 1.5 Hz. Watch spread rather than rate.

Not yet trustworthy in a live room. Replayed against a recording made with music playing, this fires roughly once every five seconds on things that are not trills. It finds real ones; it does not yet reject rhythmic backgrounds.

Session recorder

Records every MouthSample the landmarker produces and every AudioFrame the analyser produces, plus the derived intermediates the thresholds are tuned against and the raw ~375 Hz level envelope. Mark segments while you record — a labelled recording is a fixture, and Phase 5 detects trills by FFT-ing that envelope, so a marked trill in a recording is a detector that can be written and tested without going back to the microphone.

record everything opens the camera, the microphone and the recogniser and starts recording, then stops all four together — the raw video is never stored, only landmarks. The individual buttons in the panels above still work; this is for when you want a take rather than an experiment.

Every recording is sent to R2 when you press stop. Download is there for when the upload fails, not as the normal path. Nothing is kept in browser storage.

Controls

Every threshold in the project, live. Make the sound, drag the control, listen to whether it was right — which is the whole point of this phase. Four of the Phase 6 defaults contradict the spec because the spec was wrong about them, and each of those cost a recording and a rebuild to find. Ranges are wide enough to get back to the spec's value and see for yourself.

A moved control shows ↺ to put it back, and its group counts how many have moved so a collapsed one cannot hide the change that is confusing you. The address bar carries the whole tuning: copy the URL to keep a setting, to reopen it on the phone, or to send it to me. Nothing is stored on the machine.

Two of these change only what happens next. The type axes do not reshape words already on the line — finalised text is frozen, which is rule 4 — and latency.asr is applied when a word arrives, so both need something said after the change to show it.

RingBuffer.sampleAt

The acceptance ramp from the spec: 100 samples at 10 ms spacing, value = i. Marks are stored samples; the line is sampleAt swept continuously across a range wider than the data, so the flat shoulders are the clamp — queries outside the retained window return the nearest end rather than extrapolating a value that was never observed.

Timeline

Every event is pushed with the time it was noticed and how far behind reality its detector runs. The timeline subtracts that latency to recover the time it actually happened, then holds everything back by LAG. Fire the burst and watch the fast detector lose the race it should never have been in.