Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@ tracks, so per-group stats stop costing cluster announces.
- [AAC alias classification](/quest/m1/aac-he-alias.md) - every accepted AAC spelling gets the right frame duration from one classification
- [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles
- [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust
- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - a buffered `<moq-watch>` asks for the oldest group first, so a voice reply's opening arrives first, and audio's head hold shrinks to the floor
- [Buffered video poster](/quest/m1/buffered-video-poster.md) - buffered startup paints the first video frame at once and starts A/V together at a shared floor
- [qmux local lifecycle](/quest/m1/qmux-local-lifecycle.md) - an abort blocked on send credit resets at once, and a close behind a backed-up socket still sends APPLICATION_CLOSE
- [JS release preflight](/quest/m1/js-publish-preflight.md) - `release.ts` refuses a package whose `@moq/*` dependency is not on npm, so no dependent publishes uninstallable
- [Stream sockets disable Nagle](/quest/m1/tcp-nodelay.md) - every qmux TCP, TLS, and WebSocket socket, accepted or dialed, sets `TCP_NODELAY`
Expand Down
83 changes: 83 additions & 0 deletions quest/m1/buffered-ascending.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# [M] Buffered playback subscribes in ascending order

## Goal

A `<moq-watch>` with a nonzero `buffer` subscribes to its audio and video
tracks with `order: "ascending"`, so the oldest group in its window is sent
first and a voice reply's opening never arrives after its live edge. The audio
decoder's head hold from [moq-dev/moq#6549](https://github.com/moq-dev/moq/pull/6549)
shrinks to one rule: hold the head until it spans the floor, then release it
sorted. Live playback (`buffer` of zero) stays newest-first.

Not here: the protocol's `Group Order` (landed in
[moq-dev/moq#5385](https://github.com/moq-dev/moq/pull/5385)), the buffered
video poster ([Buffered video poster](/quest/m1/buffered-video-poster.md)), and
ordering for the Python or FFI audio consumers.

## Plan

Motivated by the moq.dev Voice demo: each TTS reply is a fresh track whose
head arrives newest-first, which shifted Chrome's decoder timeline and
overflowed the ring (moq-dev/moq#6523), then clipped each reply's opening
until #6549 held and sorted the head with heuristics.

Decided 2026-10-23 by the maintainer:

- The goal: a subscriber asks for ascending order, so the head-hold heuristics
from #6549 can go. Voice AI motivates it but is not the only user.
- ✅ All buffered media goes ascending, audio and video. Not chosen: audio
only; an explicit opt-in independent of buffering.
- ✅ The opt-in is buffered mode itself: a nonzero `buffer` makes every media
subscription ascending. A buffered player always wants its window from the
start, and a live one never does, so a second knob adds nothing.
- ✅ Keep a head hold, sized by the floor (the minimum latency): hold frames
until they span the floor (`delay`), then release sorted. Playback cannot
start before the floor anyway, so the hold costs no startup latency, and it
absorbs residual reordering between in-flight streams, which the draft still
requires a subscriber to tolerate. #6549's "a newer group than the first" and
max-delay release rules go. Not chosen: delete the hold outright; keep
#6549's rules unchanged.
- ✅ The hold stays audio-only. Video's startup hold is the poster quest's.
- ✅ Placed in m1, requiring [Finalize moq-lite-07](/quest/m1/lite07-finalize.md):
lite-06 and earlier serve an ascending request newest-first, so this is a
no-op on default sessions until lite-07 negotiates by default. `release`
has no `order` field, so it ships in the next minor, not a 0.6.x patch.
Not chosen: m2; wiring it now behind the lite-07 gate.

Implementation:

- `subscribeMedia` (`js/watch/src/media.ts`) takes the order. Audio and video
pass `"ascending"` when `sync.out.buffered` is true, and re-subscribe or
update when buffering toggles. Check how `Subscription.order` updates over
SUBSCRIBE_UPDATE on lite-07, and that moq-transport fixes it at SUBSCRIBE.
- A relay requests descending upstream unless every subscription it serves is
ascending. A voice reply track has one subscriber, so it goes ascending end
to end. Say in the watch docs that a mixed audience gets newest-first
upstream.
- The draft says an ascending subscriber SHOULD raise `Subscriber Max Age`.
Buffered mode already subscribes with `delay + buffer`; confirm that covers
it.
- Replace `Order`'s release rules in `js/watch/src/audio/order.ts` with the
floor span, keeping `admit()` and the epoch reset ordering from #6549. Keep
the decoder tests for a reversed head (now released at the floor) and the
fresh-encoder case.
- Interplay with [Container reads drop backfill](/quest/m1/read-backfill.md):
once the JS consumer drops groups below its cursor, an ascending head
arrives above it, so nothing valid is dropped. Whichever lands second
re-runs the other's tests.
- Update `doc/concept/use-case/ai.md` and the `buffer` docs in
`js/watch/src/element.ts` and `js/watch/src/sync.ts` in the same PR.
- Verify against a voice agent on a lite-07 relay, such as a Pipecat pipeline
publishing each TTS reply as a fresh track with one Opus frame per group:
every reply decodes from its first frame, plain and barge-in, with no
overflow.

## Required

- [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - an ascending request is served newest-first until lite-07 negotiates by default

## Related

- [Buffered video poster](/quest/m1/buffered-video-poster.md) - buffered startup paints the first video frame and holds video until the shared floor
- [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS consumer stops delivering groups below its cursor
- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - owns the wire's `Group Order` and the ordered cursor's floor
39 changes: 39 additions & 0 deletions quest/m1/buffered-video-poster.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# [S] Buffered video paints a poster while it fills

## Goal

A `<moq-watch>` with a nonzero `buffer` renders its first decoded video frame
as soon as it decodes, as a poster, and holds video there until playback
starts. Synchronized audio and video share one floor: playback starts once
every enabled track has buffered the floor, and a video-only broadcast waits
on video's own span. Live playback is unchanged.

## Plan

Decided 2026-10-23 by the maintainer, while planning
[Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md):

- ✅ Video holds before playback starts too, but the first frame renders right
away as a poster while the rest loads.
- ✅ Synchronized A/V shares one floor. Not chosen: video starting on its own
floor; audio alone deciding the start.
- ✅ Its own quest, ready now: it has no protocol dependency. Not chosen: one
quest with the ascending work, blocked on lite-07.

Implementation:

- First record what buffered video does at startup today: when the first
frame paints, and whether it moves before the shared clock starts.
- Reuse the paused poster path (`renderer.out.frame`, read by `Player`)
rather than adding a second one.
- Gate the shared clock's start on every enabled track reaching the floor,
consistent with [A/V clock](/quest/m1/av-clock.md), where the audio
playhead drives `Sync.reference` once audio plays.
- Tests: the poster renders before the floor; video does not advance until
every enabled track has buffered the floor; video-only starts on its own
span; live mode is unchanged.

## Related

- [Buffered playback subscribes in ascending order](/quest/m1/buffered-ascending.md) - buffered subscriptions ask for the oldest group first
- [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives `Sync.reference` while audio plays
Loading