From fe7d07d68911a33a19e941cfd68dcc1b3f93ba1a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 15:14:24 -0700 Subject: [PATCH 1/2] Plan per-frame arrivals in @moq/watch Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/watch-arrivals.md | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 quest/m1/watch-arrivals.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 7e4ab09148f..2439c9cd6a8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -47,6 +47,7 @@ were done or not worth their cost. - [moq-bot may push workflow changes](/quest/m1/bot-workflows-permission.md) - condition: the maintainer grants moq-bot's GitHub App the `workflows` permission, so back-merges carrying workflow changes go through - [main merges through a squash queue](/quest/m1/merge-queue-settings.md) - condition: once `release` carries the new back-merge script, the maintainer enables the squash merge queue and moq-bot's pull_request bypass together - [Binding audio delay](/quest/m1/binding-surface.md) - moq-ffi and every wrapper configure and observe audio playout delay +- [Per-frame arrivals in @moq/watch](/quest/m1/watch-arrivals.md) - the audio and video decoders expose a window of recent frame arrivals, late and skipped marks included, as a signal - [Bump web-transport-iroh for the capsule close](/quest/m1/iroh-capsule-bump.md) - condition: moq-dev/web-transport#419 ships in a release, then moq's iroh HTTP/3 client reports a peer's close capsule - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it - [lite-07 Live flag](/quest/m1/lite-live.md) - a separate `Live` field on lite-07 SUBSCRIBE, so merged floors never starve a subscriber; late lower groups build on it diff --git a/quest/m1/watch-arrivals.md b/quest/m1/watch-arrivals.md new file mode 100644 index 00000000000..ebb68a9a822 --- /dev/null +++ b/quest/m1/watch-arrivals.md @@ -0,0 +1,29 @@ +# [S] Per-frame arrivals in @moq/watch + +## Goal + +The audio and video decoders in `@moq/watch` each expose an `arrivals` +signal: a bounded window of recent frames as `{timestamp, arrival, late, +skipped}`, where `arrival` is when the frame came off the transport, `late` +means it missed its playout time, and `skipped` means the consumer dropped +its group to stay within the latency budget. A page can draw per-frame +arrival against playout without patching `Sync.received`. + +## Plan + +Decided 2026-10-09: + +- **A signal, not a callback.** A signal holding only the newest arrival + would coalesce and lose frames between reads, so it holds a window (time- + or count-bounded, chosen when implementing) that updates per frame. +- **Feed it from where the data already exists.** `Sync.received()` + computes lateness per frame and only logs it today; the container consumer + knows which groups it skipped and only warns. Neither has a signal. +- The first consumer is moq.dev's Voice AI demo, which draws each word's + frames and marks the ones a WebRTC jitter buffer would have dropped + ([moq.dev's condition quest](https://github.com/moq-dev/moq.dev/blob/main/quest/m1/watch-arrivals-release.md)). +- Test with recorded arrival traces through `js/watch/src/audio/replay.ts`. + +## Related + +- [Browser stats](/quest/m1/stats/js.md) - its late-frame and skip counters can be derived from this window From 359f56cfbc3fa20f3ea5787dae755d133a599f84 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 17:35:34 -0700 Subject: [PATCH 2/2] quest(m1): fold review corrections into watch-arrivals Arrival is the container's ingress stamp, late is the sync deadline, skipped groups need an explicit representation, the window covers a transcript word, and stats counters stay at the producer. Co-Authored-By: Claude Opus 5.5 --- quest/m1/watch-arrivals.md | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/quest/m1/watch-arrivals.md b/quest/m1/watch-arrivals.md index ebb68a9a822..b0f86f09474 100644 --- a/quest/m1/watch-arrivals.md +++ b/quest/m1/watch-arrivals.md @@ -5,25 +5,40 @@ The audio and video decoders in `@moq/watch` each expose an `arrivals` signal: a bounded window of recent frames as `{timestamp, arrival, late, skipped}`, where `arrival` is when the frame came off the transport, `late` -means it missed its playout time, and `skipped` means the consumer dropped -its group to stay within the latency budget. A page can draw per-frame -arrival against playout without patching `Sync.received`. +means it arrived after its sync deadline (not that it rendered late), and +`skipped` marks content the consumer dropped to stay within the latency +budget. A page can draw per-frame arrival against playout without patching +`Sync.received`. ## Plan Decided 2026-10-09: - **A signal, not a callback.** A signal holding only the newest arrival - would coalesce and lose frames between reads, so it holds a window (time- - or count-bounded, chosen when implementing) that updates per frame. + would coalesce and lose frames between reads, so it holds a window that + updates per frame. It must reach back at least as far as the span the + first consumer draws (a transcript word); time or count bound is chosen + when implementing. - **Feed it from where the data already exists.** `Sync.received()` computes lateness per frame and only logs it today; the container consumer knows which groups it skipped and only warns. Neither has a signal. +- **`arrival` is the ingress time.** `Container.Consumer` stamps each frame + as `readFrame()` returns, but the `Frame` it returns drops that stamp, and + `Sync.received()` samples the clock later, after reordering and any + decoder backpressure. Carry the ingress stamp through to the decoder and + compute lateness from it. +- **Skipped content never reaches the decoder.** A latency-skipped group's + frames are not decoded, so they cannot appear as ordinary entries. Decide + how the window represents the gap (a mark on the first frame after it, or + a separate entry), keeping it distinct from other `continuous: false` + causes such as markers. - The first consumer is moq.dev's Voice AI demo, which draws each word's frames and marks the ones a WebRTC jitter buffer would have dropped ([moq.dev's condition quest](https://github.com/moq-dev/moq.dev/blob/main/quest/m1/watch-arrivals-release.md)). -- Test with recorded arrival traces through `js/watch/src/audio/replay.ts`. +- Test with recorded arrival traces through `js/watch/src/audio/replay.ts`, + including a case where delivery is delayed but the recorded arrival is + unchanged. ## Related -- [Browser stats](/quest/m1/stats/js.md) - its late-frame and skip counters can be derived from this window +- [Browser stats](/quest/m1/stats/js.md) - its late-frame and skip counters share these sources, but count at the producer since a reader of an evicting window can miss entries