From 10d0b045c1d86f609d56bb90518ca7e4d45a464e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 10:33:45 -0800 Subject: [PATCH] quest: replan the layers epic around hang compositions Co-Authored-By: Claude Opus 5.5 --- quest/m2/catalog-at-timestamp.md | 33 ++++++++++++++ quest/m3/layers/README.md | 77 ++++++++++++++++++++------------ quest/m3/layers/play.md | 21 ++++----- quest/m3/layers/spec.md | 51 ++++++++++++++------- quest/m3/layers/watch.md | 19 ++++---- 5 files changed, 136 insertions(+), 65 deletions(-) create mode 100644 quest/m2/catalog-at-timestamp.md diff --git a/quest/m2/catalog-at-timestamp.md b/quest/m2/catalog-at-timestamp.md new file mode 100644 index 00000000000..b1f883cf3af --- /dev/null +++ b/quest/m2/catalog-at-timestamp.md @@ -0,0 +1,33 @@ +# [S] hang applies catalog updates at their timestamp + +## Goal + +A hang catalog update takes effect at its timestamp on the broadcast clock, +not when a consumer reads it. `drafts/draft-lcurley-moq-hang.md` says so, and +`rs/hang` and `js/hang` hand each update to the application with its +timestamp, so a rendition added or removed, or a change in a section such as +a composition's, lines up with the frame stamped alike. + +## Plan + +Decided 2026-11-12 (maintainer, moq-dev/studio /quest-plan): ✅ catalogs are +timed from lite-07 on, and every update applies at its timestamp, as one rule +for the whole catalog (rejected: timing only a composition section, which +leaves two timing rules in one catalog). Its first consumer is +[the composition](/quest/m3/layers/spec.md). It lands in m2, beside lite-07 +(rejected: m3 with the composition), so lite-07 ships with the rule. + +[The model](/quest/m2/time-addressing/model.md) already has moq-mux's catalog +producer stamp each catalog from the broadcast clock. Today +`rs/hang/src/catalog/consumer.rs` drops that stamp (`.map(|value| +value.value)`), and `js/hang` does likewise; both return it with the +catalog. A consumer that reads an update ahead of its timestamp holds it +until media reaches that time; one that reads it late applies it at once. + +A scale-less broadcast's catalog has no media time to wait for, so it applies +on arrival, as today. + +## Required + +- [Rust groups are addressed by their start](/quest/m2/time-addressing/model.md) - the catalog producer stamps from the broadcast clock +- [@moq/net addresses groups by start](/quest/m2/time-addressing/js.md) - the same for js/hang diff --git a/quest/m3/layers/README.md b/quest/m3/layers/README.md index 30ca6309dc4..f23a69fda12 100644 --- a/quest/m3/layers/README.md +++ b/quest/m3/layers/README.md @@ -1,37 +1,56 @@ -# Layered broadcasts +# Composed broadcasts ## Goal -hang specifies a layer layout track: a broadcast's main picture and sound plus -other tracks layered over it, each placed, timed, and mixed on the media -clock. `moq play` and `@moq/watch` composite those layers over the main -picture and mix their audio, so a moq-dev/studio broadcast plays as its -creator composed it, while sources published on their own tracks stay -separately fetchable and toggleable. +hang specifies a composition: a broadcast whose catalog lists other +broadcasts, each on its own timeline with an offset onto the composition's +clock, and says where and when each shows and how its sound mixes. `moq play` +and `@moq/watch` play a composition, so a moq-dev/studio broadcast plays as +its creator composed it, while each source stays its own broadcast that a +viewer can toggle and fetch at its own priority. -This README's own work is the end-to-end proof: one fixture broadcast with a -picture layer and an audio layer plays in `moq play` and `@moq/watch` with the -same placement and timing, within a stated tolerance, and with the layer's -sound mixed in. +This README's own work is the end-to-end proof: one fixture composition with +a main broadcast, a picture child, and a sound child plays in `moq play` and +`@moq/watch` with the same placement and timing, within a stated tolerance, +and with the child's sound mixed in. ## Plan -Decided 2026-11-08 (maintainer, moq-dev/studio /quest-plan): +Replanned 2026-11-12 (maintainer, moq-dev/studio /quest-plan), after the +studio chose a composition over a layer layout track for naming guests (its +`quest/m2/compose/layout.md`). hang's `include` requires an included +broadcast to share the listing catalog's PTS timeline, `media.clock`, and +`media.join.interval`, which a guest's broadcast does not. -- The studio publishes the program plus each independently published source - as its own track, and players composite those sources over the program. - Rejected: viewers rebuilding the whole scene from every source's track, - which would take more encoder sessions than consumer GPUs have. -- hang specifies the layout and each player implements it with its own code. - The studio's compositor (moq-scene) stays in moq-dev/studio, since moq - never depends on that repository, so parity comes from the shared fixture, - not shared code. Rejected: moving moq-scene into moq. -- The studio prototypes the format first (`moq-studio --watch`, reading a - provisional `dev.moq.studio.layers` track), and the spec starts from what - that prototype settles (rejected: drafting the spec in parallel). -- Players mix layers' audio in, at the gain, mute, pan, and delay the layout - carries (pan and delay added 2026-11-09, as the spec quest records). -- Each layer carries a priority and whether a viewer may toggle it. A player - maps priority onto moq-lite's subscription priorities and unsubscribes a - toggled-off layer. -- m3: moq-dev/studio is the named consumer, and its side sits in its m2. +- **What:** ✅ a composition (rejected: a layer layout track naming tracks of + one broadcast, the 2026-11-08 plan). Every independently published source + is its own broadcast, and the composition is a broadcast of its own that + lists them; the studio's program is one child. +- **Generality:** ✅ a general way to compose broadcasts, such as multiview or + picture-in-picture, keeping only what every player can reproduce + (rejected: exactly what the studio emits). The studio is its first + producer. +- **Shape:** ✅ all in the composition broadcast's catalog, its updates timed + (rejected: the catalog listing children with a separate timed track for + placement; everything in one track). A catalog update applies at its + timestamp ([that rule](/quest/m2/catalog-at-timestamp.md), m2). +- **Clock:** ✅ each child carries the composer's offset from its PTS to the + composition's (rejected: aligning children by their `media.clock` wall + times, which depends on publishers' clocks agreeing and drops the latency + the composer saw). A child that restarts gets a new offset in a new update. +- **Older players:** ✅ the composition names one child as main, and its + catalog lists the main's renditions, so a player that knows no + compositions plays the main picture alone (rejected: an empty catalog for + them). +- **Untimed:** ✅ a composition requires a timed catalog; in a scale-less one + it is a publisher error, and a player plays the main alone (rejected: + applying updates on arrival). + +Still settled from 2026-11-08: hang specifies the format and each player +implements it with its own code, so parity comes from the shared fixture, not +shared code (moq never depends on moq-dev/studio); the studio prototypes the +format first. Players mix each child's sound at the gain, mute, pan, and +delay the composition carries. Each child carries a priority and whether a +viewer may toggle it; a player maps priority onto moq-lite's subscription +priorities and unsubscribes a toggled-off child. m3: moq-dev/studio is the +named consumer, and its side sits in its m2. diff --git a/quest/m3/layers/play.md b/quest/m3/layers/play.md index e40cb13ca48..81938a7091b 100644 --- a/quest/m3/layers/play.md +++ b/quest/m3/layers/play.md @@ -1,17 +1,18 @@ -# [M] moq play composites layers +# [M] moq play plays compositions ## Goal -`moq play` plays a broadcast that carries a layer layout track: it composites -each visible layer over the main picture where and when the layout says, -mixes each audio layer in at its gain, mute, pan, and delay, and lets a -viewer toggle each layer the layout marks toggleable. A broadcast without one -plays as today. +`moq play` plays a composition: it opens each child broadcast the +composition lists, places each visible picture child where and when the +composition says, on the composition's clock through the child's offset, +mixes each sound child in at its gain, mute, pan, and delay, and lets a +viewer toggle each child the composition marks toggleable. A broadcast +without a composition plays as today. ## Plan -Fetch only visible layers, at the layout's priorities through moq-lite's -subscription priorities, and unsubscribe a toggled-off layer. Prove placement +Fetch only visible children, at the composition's priorities through moq-lite's +subscription priorities, and unsubscribe a toggled-off child. Prove placement and timing against the epic's fixture. Decided 2026-11-09 (maintainer, moq-dev/studio /quest-audit): the viewer @@ -20,9 +21,9 @@ toggle and the priority mapping are this quest's work, not only `quest/m2/compose/controls.md`) that viewer controls land in the native player, and its `hang-layers` quest deletes its prototype player once this lands, so this quest is their home. Subscriptions are per track, not per -layer, as that quest settled: a track several layers show stays subscribed +child, as that quest settled: a track several children show stays subscribed while any of them is on, at the highest priority among them. ## Required -- [hang specifies a layer layout track](/quest/m3/layers/spec.md) - what `moq play` composites +- [hang specifies a composition](/quest/m3/layers/spec.md) - what `moq play` plays diff --git a/quest/m3/layers/spec.md b/quest/m3/layers/spec.md index d59c2608e8c..74fd1af25d0 100644 --- a/quest/m3/layers/spec.md +++ b/quest/m3/layers/spec.md @@ -1,31 +1,48 @@ -# [M] hang specifies a layer layout track +# [M] hang specifies a composition ## Goal -`drafts/draft-lcurley-moq-hang.md`, `rs/hang`, and `js/hang` define a layer -layout track: a versioned JSON track in snapshot mode on the media clock that -names each layer's track and gives its transform, crop, z-order, opacity, -visibility, keyframes, priority, and whether a viewer may toggle it. An audio -layer also carries its gain, mute, pan, and delay. +`drafts/draft-lcurley-moq-hang.md`, `rs/hang`, and `js/hang` define a +composition: a versioned catalog section that lists child broadcasts, marks +one as main, and gives each child its offset from its PTS onto the +composition's clock, its priority, and whether a viewer may toggle it. A +picture child also carries its transform, crop, z-order, opacity, visibility, +and keyframes; a sound child carries its gain, mute, pan, and delay. The +composition's catalog also lists the main child's renditions, so a consumer +that knows no compositions plays the main alone. ## Plan -Start from the format moq-dev/studio's watch prototype settled, and keep only -what every player can reproduce. A consumer refuses an unknown version loudly -and plays the main picture alone. Layers name tracks and catalog references, -never files. +The [epic](/quest/m3/layers/README.md) holds the 2026-11-12 decisions: the +composition lives in its own broadcast's catalog, applied at each update's +timestamp; offsets are the composer's; a composition needs a timed catalog, +and a consumer plays the main alone in a scale-less one. -Decided 2026-11-09 (maintainer, moq-dev/studio /quest-audit): an audio layer +Start from the format moq-dev/studio's prototype settles, and keep only what +every player can reproduce. A consumer refuses an unknown version loudly and +plays the main alone. Children name broadcasts, relative like `include`'s and +contained the same way (`check_resolvable` in +`rs/hang/src/catalog/consumer.rs`), never files. Unlike an include, a child +keeps its own PTS timeline and root settings, and listing one does not +subscribe to it. The composition's clock is the main's, so the main's offset +is zero and the catalog lists its renditions by their `broadcast` reference, +on the same timeline. + +Decided 2026-11-09 (maintainer, moq-dev/studio /quest-audit): a sound child carries pan and delay as well as gain and mute. The studio's strip has all four (`rs/moq-scene/src/collection.rs` in moq-dev/studio; pan shipped in [moq-dev/studio#116](https://github.com/moq-dev/studio/pull/116)), and a -source's own track is taken before the strip applies them. Without the delay -in the layout, a microphone the creator delayed for sync and published on its -own track would play out of sync for viewers. +source's own broadcast is taken before the strip applies them. Without the +delay, a microphone the creator delayed for sync would play out of sync for +viewers. ## Waiting -- moq-dev/studio: its `moq-studio --watch` prototype plays program plus - layers from the provisional `dev.moq.studio.layers` track, planned as - `quest/m2/compose/watch.md` there. Check: that quest's file is gone from +- moq-dev/studio: the studio publishes a provisional composition and its + `moq-studio --watch` prototype plays it, planned as + `quest/m2/compose/layout.md` there. Check: that quest's file is gone from moq-dev/studio's `main` because its PR merged. + +## Required + +- [hang applies catalog updates at their timestamp](/quest/m2/catalog-at-timestamp.md) - a composition's changes land on the frame stamped alike diff --git a/quest/m3/layers/watch.md b/quest/m3/layers/watch.md index f2d1059db0c..515afc245bd 100644 --- a/quest/m3/layers/watch.md +++ b/quest/m3/layers/watch.md @@ -1,20 +1,21 @@ -# [M] The watch player composites layers +# [M] The watch player plays compositions ## Goal -`@moq/watch` plays a broadcast that carries a layer layout track: it -composites each visible layer over the main picture where and when the layout -says, mixes each audio layer in at its gain, mute, pan, and delay, and lets a -viewer toggle each layer the layout marks toggleable. A broadcast without one -plays as today. +`@moq/watch` plays a composition: it opens each child broadcast the +composition lists, places each visible picture child where and when the +composition says, on the composition's clock through the child's offset, +mixes each sound child in at its gain, mute, pan, and delay, and lets a +viewer toggle each child the composition marks toggleable. A broadcast +without a composition plays as today. ## Plan -Fetch only visible layers, at the layout's priorities through moq-lite's -subscription priorities, and unsubscribe a toggled-off layer. Prove placement +Fetch only visible children, at the composition's priorities through moq-lite's +subscription priorities, and unsubscribe a toggled-off child. Prove placement and timing against the epic's fixture, matching `moq play` within its tolerance. ## Required -- [hang specifies a layer layout track](/quest/m3/layers/spec.md) - what `@moq/watch` composites +- [hang specifies a composition](/quest/m3/layers/spec.md) - what `@moq/watch` plays