From 6197bd026edcf7a7830c83758c33d7ad3f9e0995 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 10:26:57 -0700 Subject: [PATCH 1/2] quest: plan ascending order for buffered playback, and a buffered video poster Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 + quest/m1/buffered-ascending.md | 82 +++++++++++++++++++++++++++++++ quest/m1/buffered-video-poster.md | 39 +++++++++++++++ 3 files changed, 123 insertions(+) create mode 100644 quest/m1/buffered-ascending.md create mode 100644 quest/m1/buffered-video-poster.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 24fdc76a867..85987aec04f 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -113,6 +113,8 @@ tracks, so per-group stats stop costing cluster announces. - [AAC alias classification](/quest/m1/aac-he-alias.md) - every accepted AAC spelling gets the right frame duration from one classification - [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles - [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust +- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - a buffered `` asks for the oldest group first, so a voice reply's opening arrives first, and audio's head hold shrinks to the floor +- [Buffered video poster](/quest/m1/buffered-video-poster.md) - buffered startup paints the first video frame at once and starts A/V together at a shared floor - [qmux local lifecycle](/quest/m1/qmux-local-lifecycle.md) - an abort blocked on send credit resets at once, and a close behind a backed-up socket still sends APPLICATION_CLOSE - [JS release preflight](/quest/m1/js-publish-preflight.md) - `release.ts` refuses a package whose `@moq/*` dependency is not on npm, so no dependent publishes uninstallable - [Stream sockets disable Nagle](/quest/m1/tcp-nodelay.md) - every qmux TCP, TLS, and WebSocket socket, accepted or dialed, sets `TCP_NODELAY` diff --git a/quest/m1/buffered-ascending.md b/quest/m1/buffered-ascending.md new file mode 100644 index 00000000000..6debf92154d --- /dev/null +++ b/quest/m1/buffered-ascending.md @@ -0,0 +1,82 @@ +# [M] Buffered playback subscribes in ascending order + +## Goal + +A `` with a nonzero `buffer` subscribes to its audio and video +tracks with `order: "ascending"`, so the oldest group in its window is sent +first and a voice reply's opening never arrives after its live edge. The audio +decoder's head hold from [moq-dev/moq#6549](https://github.com/moq-dev/moq/pull/6549) +shrinks to one rule: hold the head until it spans the floor, then release it +sorted. Live playback (`buffer` of zero) stays newest-first. + +Not here: the protocol's `Group Order` (landed in +[moq-dev/moq#5385](https://github.com/moq-dev/moq/pull/5385)), the buffered +video poster ([Buffered video poster](/quest/m1/buffered-video-poster.md)), and +ordering for the Python or FFI audio consumers. + +## Plan + +Motivated by the moq.dev Voice demo: each TTS reply is a fresh track whose +head arrives newest-first, which shifted Chrome's decoder timeline and +overflowed the ring (moq-dev/moq#6523), then clipped each reply's opening +until #6549 held and sorted the head with heuristics. + +Decided 2026-10-23 by the maintainer: + +- The goal: a subscriber asks for ascending order, so the head-hold heuristics + from #6549 can go. Voice AI motivates it but is not the only user. +- ✅ All buffered media goes ascending, audio and video. Not chosen: audio + only; an explicit opt-in independent of buffering. +- ✅ The opt-in is buffered mode itself: a nonzero `buffer` makes every media + subscription ascending. A buffered player always wants its window from the + start, and a live one never does, so a second knob adds nothing. +- ✅ Keep a head hold, sized by the floor (the minimum latency): hold frames + until they span the floor (`delay`), then release sorted. Playback cannot + start before the floor anyway, so the hold costs no startup latency, and it + absorbs residual reordering between in-flight streams, which the draft still + requires a subscriber to tolerate. #6549's "a newer group than the first" and + max-delay release rules go. Not chosen: delete the hold outright; keep + #6549's rules unchanged. +- ✅ The hold stays audio-only. Video's startup hold is the poster quest's. +- ✅ Placed in m1, requiring [Finalize moq-lite-07](/quest/m1/lite07-finalize.md): + lite-06 and earlier serve an ascending request newest-first, so this is a + no-op on default sessions until lite-07 negotiates by default. `release` + has no `order` field, so it ships in the next minor, not a 0.6.x patch. + Not chosen: m2; wiring it now behind the lite-07 gate. + +Implementation: + +- `subscribeMedia` (`js/watch/src/media.ts`) takes the order. Audio and video + pass `"ascending"` when `sync.out.buffered` is true, and re-subscribe or + update when buffering toggles. Check how `Subscription.order` updates over + SUBSCRIBE_UPDATE on lite-07, and that moq-transport fixes it at SUBSCRIBE. +- A relay requests descending upstream unless every subscription it serves is + ascending. A voice reply track has one subscriber, so it goes ascending end + to end. Say in the watch docs that a mixed audience gets newest-first + upstream. +- The draft says an ascending subscriber SHOULD raise `Subscriber Max Age`. + Buffered mode already subscribes with `delay + buffer`; confirm that covers + it. +- Replace `Order`'s release rules in `js/watch/src/audio/order.ts` with the + floor span, keeping `admit()` and the epoch reset ordering from #6549. Keep + the decoder tests for a reversed head (now released at the floor) and the + fresh-encoder case. +- Interplay with [Container reads drop backfill](/quest/m1/read-backfill.md): + once the JS consumer drops groups below its cursor, an ascending head + arrives above it, so nothing valid is dropped. Whichever lands second + re-runs the other's tests. +- Update `doc/concept/use-case/ai.md` and the `buffer` docs in + `js/watch/src/element.ts` and `js/watch/src/sync.ts` in the same PR. +- Verify against a voice agent on a lite-07 relay: every reply decodes from + its first frame, plain and barge-in, with no overflow. moq.dev's local + Pipecat stack (moq-dev/moq.pro#2785) is one way. + +## Required + +- [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - an ascending request is served newest-first until lite-07 negotiates by default + +## Related + +- [Buffered video poster](/quest/m1/buffered-video-poster.md) - buffered startup paints the first video frame and holds video until the shared floor +- [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS consumer stops delivering groups below its cursor +- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - owns the wire's `Group Order` and the ordered cursor's floor diff --git a/quest/m1/buffered-video-poster.md b/quest/m1/buffered-video-poster.md new file mode 100644 index 00000000000..24bc8d282e0 --- /dev/null +++ b/quest/m1/buffered-video-poster.md @@ -0,0 +1,39 @@ +# [S] Buffered video paints a poster while it fills + +## Goal + +A `` with a nonzero `buffer` renders its first decoded video frame +as soon as it decodes, as a poster, and holds video there until playback +starts. Synchronized audio and video share one floor: playback starts once +every enabled track has buffered the floor, and a video-only broadcast waits +on video's own span. Live playback is unchanged. + +## Plan + +Decided 2026-10-23 by the maintainer, while planning +[Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md): + +- ✅ Video holds before playback starts too, but the first frame renders right + away as a poster while the rest loads. +- ✅ Synchronized A/V shares one floor. Not chosen: video starting on its own + floor; audio alone deciding the start. +- ✅ Its own quest, ready now: it has no protocol dependency. Not chosen: one + quest with the ascending work, blocked on lite-07. + +Implementation: + +- First record what buffered video does at startup today: when the first + frame paints, and whether it moves before the shared clock starts. +- Reuse the paused poster path (`renderer.out.frame`, read by `Player`) + rather than adding a second one. +- Gate the shared clock's start on every enabled track reaching the floor, + consistent with [A/V clock](/quest/m1/av-clock.md), where the audio + playhead drives `Sync.reference` once audio plays. +- Tests: the poster renders before the floor; video does not advance until + every enabled track has buffered the floor; video-only starts on its own + span; live mode is unchanged. + +## Related + +- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - buffered subscriptions ask for the oldest group first +- [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives `Sync.reference` while audio plays From e17e9ed2a1c66ba677e37b0c5ec56bf691fb8095 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 10:56:57 -0700 Subject: [PATCH 2/2] quest: describe the voice verification without a private reference Co-Authored-By: Claude Opus 5.5 --- quest/m1/buffered-ascending.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/quest/m1/buffered-ascending.md b/quest/m1/buffered-ascending.md index 6debf92154d..b47f4748e1d 100644 --- a/quest/m1/buffered-ascending.md +++ b/quest/m1/buffered-ascending.md @@ -67,9 +67,10 @@ Implementation: re-runs the other's tests. - Update `doc/concept/use-case/ai.md` and the `buffer` docs in `js/watch/src/element.ts` and `js/watch/src/sync.ts` in the same PR. -- Verify against a voice agent on a lite-07 relay: every reply decodes from - its first frame, plain and barge-in, with no overflow. moq.dev's local - Pipecat stack (moq-dev/moq.pro#2785) is one way. +- Verify against a voice agent on a lite-07 relay, such as a Pipecat pipeline + publishing each TTS reply as a fresh track with one Opus frame per group: + every reply decodes from its first frame, plain and barge-in, with no + overflow. ## Required