diff --git a/quest/m0/2981-moq-audio-nothing-in-the-decode-or-playback-path-models-a.md b/quest/m0/2981-moq-audio-nothing-in-the-decode-or-playback-path-models-a.md index 9b00ca786c2..4e26143d61d 100644 --- a/quest/m0/2981-moq-audio-nothing-in-the-decode-or-playback-path-models-a.md +++ b/quest/m0/2981-moq-audio-nothing-in-the-decode-or-playback-path-models-a.md @@ -9,7 +9,7 @@ timestamps say where its samples actually belong. ## Plan -Gaps are routine: `latency_max` skipping a stalled group, `moq play` dropping +Gaps are routine: `max_age` skipping a stalled group, `moq play` dropping an undecodable packet, an ingest that resyncs. Below the catalog they are all treated as contiguous audio. @@ -44,7 +44,7 @@ reads as discontinuous under a strict rule. belong before the hole) or drop them, reset it, and stamp the next output from the new packet rather than by rewinding. - `play_audio` writes silence for the missing duration, which also covers - `latency_max` skips. + `max_age` skips. - Tests: frames at 0 and 2048 samples across a gap produce a hole; a 2.5 ms Opus packet lost after a 20 ms one is a gap; millisecond-stamped 1024-sample AAC packets detect no gap; the encoder's first PTS matches its first input. diff --git a/quest/m0/README.md b/quest/m0/README.md index d8952b30738..e68819938e5 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -19,6 +19,7 @@ regression test per Root Cause First. - [Rust log redaction](/quest/m0/rust-log-redaction.md) - moq-native logs relay URLs with their query and moq-rtmp logs stream keys - [#3139](/quest/m0/3139-moqsrc-a-rendition-nobody-answers-keeps-the-session-alive.md) - moqsrc: a rendition nobody answers keeps the session alive after the catalog closes - [#2981](/quest/m0/2981-moq-audio-nothing-in-the-decode-or-playback-path-models-a.md) - moq-audio: a media gap is a hole in the output, not a splice +- [Playout clock](/quest/m0/playout-clock.md) - moq play presents on a clock it controls, with a `--delay` offset and forward re-anchoring - [#3080](/quest/m0/3080-fix-watch-audio-ring-truncate-can-race-the-worklet-reader.md) - watch: an audio ring truncate can race the worklet reader for one quantum - [#2833](/quest/m0/2833-moq-export-ts-a-rewound-timeline-stalls-the-si-table.md) - moq export ts: a rewound timeline stalls SI tables, PCR, and pacing until the media clock catches up - [#2806](/quest/m0/2806-js-net-the-draft-14-15-adapter-keeps-one-request-per.md) - js/net: a duplicate PUBLISH_NAMESPACE on draft-14/15 strands the first request diff --git a/quest/m0/playout-clock.md b/quest/m0/playout-clock.md new file mode 100644 index 00000000000..79e0de5b9b9 --- /dev/null +++ b/quest/m0/playout-clock.md @@ -0,0 +1,127 @@ +# [M] play: a real playout clock, with a --delay offset + +## Goal + +`moq play` presents against a clock it controls. `--delay ` (default +`100ms`) is how far playback trails the live edge, and a frame that arrives +earlier re-anchors playback forward instead of leaving it wherever the first +frame landed. + +Today there is no offset anywhere in the path, and two clocks disagree about +what time it is. With audio, `rs/moq-cli/src/play.rs` re-derives `Clock { media: +end - sink.buffered(), wall: now }` on every audio frame, so the offset is +whatever the speaker happens to hold, bounded by a hardcoded `AUDIO_BUFFER_MAX` +of one second. Video-only, `video_clock` is anchored once on the first frame and +never re-anchored (its sole write is gated on `is_none()`), so a first frame that +arrives late appears to leave playback behind live for the rest of the session. +`js/watch/src/sync.ts` handles exactly this by lowering its reference whenever a +frame arrives earlier. Reproduce that case before treating it as a defect. + +Not in scope: the `"auto"` (RTT-derived) and `"instant"` modes of `moq-watch`'s +`Delay`, and any FFI surface. `libmoq` and `moq-ffi` hand decoded frames to a +caller callback and have no playout stage, so there is nothing there to delay +yet. + +## Plan + +Branch from main: nothing here breaks a published API. + +### One primitive, not two + +`moq_mux::pace::Pacer` is already most of `Sync`. Its `anchor: Option<(Instant, +u128)>` is `Sync`'s `reference`, it re-anchors forward only, and `with_lead` is +nearly `moq-watch`'s `buffer`. The only missing piece is a delay offset, so +extend the primitive rather than adding a parallel one beside it. + +It keeps the name `Pacer`. `moq_mux::Clock` is taken by the capture-side shared +epoch (`rs/moq-mux/src/clock.rs`, used across `moq-audio`'s capture path and +`moq-cli publish`), which is a different concept, and the rename's only +justification was the clarity that name would have bought. Staying put also keeps +this additive, so `rs/moq-srt/src/server.rs` and `rs/moq-cli/src/subscribe.rs` +need no change at all. + +Three things the offset has to get right, none of which fall out of adding it to +`send_at`: + +- **The lead comparison must exclude the delay.** `pace` re-anchors when + `send_at` leads `now` by more than `lead`, and `lead` defaults to zero, so a + delay folded into `send_at` makes the very first delayed frame overshoot and + re-anchor, discarding the offset. `hurry` has the same problem: it returns + `now`, undelayed, so every later re-anchor drops the delay again. Compare and + re-anchor on the undelayed instant and apply the delay last. +- **A moved anchor has to wake pending waits.** With audio and video sharing one + `Pacer`, a task can already be parked waiting on a frame when an earlier frame + on the other stream moves the anchor. `Sync.#setReference` resolves `#update` + to wake every parked wait so it recomputes; `Pacer` has no notification at all. + + This is a re-anchor notification, not a timed wait: `kio` has no time module + (`rs/CLAUDE.md`), so a `poll_*` that had to fire at a deadline would park + forever with nothing to arm it, and taking a `moq_net::Timers` the way + `origin::Driver::run` does would drag a runtime into a type that has never + needed one. Keep `pace` returning the instant and let the caller sleep, which + is the contract the export path already relies on, and add only the + `poll_*`/`kio::Waiter` pair that reports the anchor moving. +- **Audio must not take the delay twice.** If audio waits for the delayed + instant and then hands samples to a sink that also holds a delay's worth, it + trails video by roughly another delay. Either feed the ring ahead of the + deadline and let it only update the shared anchor, or subtract the sink depth + from the audio write deadline. + +### play.rs + +- Delete the local `Clock` struct and `video_clock`, and route both audio and + video through the one `Pacer`. The reference anchor becomes the single + authority; the speaker-derived clock goes away. +- Honoring a delay in audio means configuring the sink, not the throttle. + `AUDIO_BUFFER_MAX` in `play.rs` only gates writes against `sink.buffered()`; + the ring itself is sized by the private `LATENCY` (50ms) and `CAPACITY` (3s) + constants in `rs/moq-audio/src/playback/sink.rs`, so one of them has to become + configurable. Keep a floor the way `ringSamples` does in + `js/watch/src/audio/latency.ts`: a zero-depth ring can never be read from. +- `--delay ` replaces `--max-age` on `play`, with no alias. The two are + one number here, since nothing older than `delay` is worth presenting, so + `delay` is what goes on the wire as the subscription's max age. `--max-age` + stays unchanged on `import`, the stdout containers, and `rtmp export`. +- Reconcile `doc/bin/cli.md`, which documents `play --max-age` and carries an + example invocation using it, plus any other `moq play` example in the tree. + +### Open questions + +Left open deliberately: they are A/V policy that wants the code in front of it, +and the answer changes what the tests should assert. + +- **Which stream owns the anchor when both exist.** Video taking a tune-in burst + can `hurry` the shared anchor past what the speaker is still draining, and + because the anchor only moves forward, later audio cannot pull it back. Making + the speaker the sole re-anchor source while audio exists is one answer; + discarding audio to the same edge is another. +- **What the speaker contributes.** `last_timestamp - buffered()` is the sample + sounding now, so feeding it in as the anchor while every result is shifted by + `delay` would schedule video a full delay behind it. If the speaker position + stays in the calculation it has to enter at the live edge, not the playing + edge. +- **Whether a per-write bound is needed.** `play_audio` splits PCM into chunks of + up to a second and checks `sink.buffered()` only between whole writes, so a + first chunk can overshoot a 100ms delay outright. Sizing each part to the + remaining headroom is the obvious fix; whether the sink should grow an + operation for it is not. + +### Verification + +- Unit tests on `Pacer` with `tokio::time::pause()`: a late first frame followed + by an earlier one re-anchors forward; a frame that merely arrives late does + not; the delay survives a re-anchor through both `pace` and `hurry`; a wait + parked on one stream shortens when another stream moves the anchor; and + `delay = 0` paces exactly as today, which is what keeps the export path + unchanged. +- Manually: `moq play` against the demo relay, with audio and video-only, + checking that a late start catches up to live rather than staying behind it, + and that audio and video stay aligned at a non-zero delay. + +## Required + +- [#2981](/quest/m0/2981-moq-audio-nothing-in-the-decode-or-playback-path-models-a.md) - settles `play_audio`'s gap and silence handling first, so the clock rework is not rebased onto it + +## Related + +- [#2278](/quest/m2/2278-watch-absolute-wall-clock-latency-target-for-synchronized.md) - the same anchoring model in js/watch, gaining an absolute wall-clock target diff --git a/quest/m2/2278-watch-absolute-wall-clock-latency-target-for-synchronized.md b/quest/m2/2278-watch-absolute-wall-clock-latency-target-for-synchronized.md index ea6a9630f2d..3bfd3331d91 100644 --- a/quest/m2/2278-watch-absolute-wall-clock-latency-target-for-synchronized.md +++ b/quest/m2/2278-watch-absolute-wall-clock-latency-target-for-synchronized.md @@ -48,8 +48,8 @@ The codebase actively fights wall-clock interpretation of frame timestamps. `Tim **Latency is relative (buffer depth), never absolute.** `js/watch/src/sync.ts`: ```ts -export type Bound = "real-time" | Time.Milli; -export type Latency = Bound | { min?: Bound; max?: Bound }; +export type Delay = "instant" | "auto" | Time.Milli; +// plus a separate `buffer: Getter` input on Sync ``` `Sync.received(timestamp)` computes `ref = Time.Milli.now() - timestamp`, and `Time.Milli.now()` is `performance.now()` (`js/net/src/time.ts:68`) - monotonic since page load, explicitly "not wall-clock time". So `reference` is an arbitrary local-clock↔PTS offset anchored by whichever frame happened to arrive first. Two viewers who tune in 3 seconds apart are 3 seconds apart forever, and nothing in the model can tell. @@ -59,7 +59,7 @@ export type Latency = Bound | { min?: Bound; max?: Bound }; #### Proposed shape 1. **A JS `Timeline.Consumer`** in `js/hang` (needed by the DVR work too). -2. **A third `Bound` variant, or a new absolute mode**: something like `latency: { absolute: Time.Milli }` meaning "render pts P at wall time `wall + P + absolute`". `Sync` then anchors on `Timeline.wall` instead of first-frame arrival, and `now()` becomes a wall-clock computation rather than an offset. +2. **A third mode alongside `delay` and `buffer`**: something like `delay: { absolute: Time.Milli }` meaning "render pts P at wall time `wall + P + absolute`". `Sync` then anchors on `Timeline.wall` instead of first-frame arrival, and `now()` becomes a wall-clock computation rather than an offset. 3. **Client clock sync.** This is the real work. `Timeline.wall` is the *publisher's* wall clock; the viewer's `Date.now()` can be off by seconds. Options, roughly in order of cost: - trust `Date.now()` (fine for a watch party, useless for betting) - estimate offset from the session RTT (we already track min RTT via PROBE in `#runJitter` for exactly this kind of anchoring) @@ -71,13 +71,13 @@ export type Latency = Bound | { min?: Bound; max?: Bound }; #### Open questions -- Absolute latency and the existing `min`/`max` bounds interact awkwardly: an absolute target is a *point*, the bounds are a *range*. Does absolute override the range, or clamp within it? +- An absolute target is a *point* while `delay` and `buffer` are two spans either side of the live edge. Does absolute replace `delay`, or sit alongside it with `buffer` still capping the lookahead? - `Timeline.wall` is per-rendition. Are audio and video guaranteed consistent? A rendition switch must not re-anchor. - Does this need anything on the publisher, or is `set_wall` sufficient? Today nothing in `moq-video`/`js/publish` calls `set_wall` as far as I can tell - so step 0 might be "actually populate `wall`". #### Naming note -`Latency` is already doubly overloaded: the `Bound | {min,max}` type (`js/watch/src/sync.ts:16`) and `class Latency` (`js/hang/src/util/latency.ts:21`, jitter+buffer). Different packages, both exported. Worth resolving before adding a third meaning. +Mostly resolved: `js/watch`'s `Latency` / `Bound` types are gone, replaced by `Delay` plus a separate `buffer`, leaving `class Latency` (`js/hang/src/util/latency.ts`, jitter+buffer) as the only remaining use. Pick a name for the absolute mode that does not reintroduce the collision. #### Branch @@ -90,3 +90,7 @@ export type Latency = Bound | { min?: Bound; max?: Bound }; ## Closes - [#2278](https://github.com/moq-dev/moq/issues/2278) - close this issue when the quest finishes + +## Related + +- [Playout clock](/quest/m0/playout-clock.md) - ports the same anchoring model to Rust as `moq_mux::Clock`