Repository navigation
quest: replan the layers epic around hang compositions #8576
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # [S] hang applies catalog updates at their timestamp | ||
|
|
||
| ## Goal | ||
|
|
||
| A hang catalog update takes effect at its timestamp on the broadcast clock, | ||
| not when a consumer reads it. `drafts/draft-lcurley-moq-hang.md` says so, and | ||
| `rs/hang` and `js/hang` hand each update to the application with its | ||
| timestamp, so a rendition added or removed, or a change in a section such as | ||
| a composition's, lines up with the frame stamped alike. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-12 (maintainer, moq-dev/studio /quest-plan): ✅ catalogs are | ||
| timed from lite-07 on, and every update applies at its timestamp, as one rule | ||
| for the whole catalog (rejected: timing only a composition section, which | ||
| leaves two timing rules in one catalog). Its first consumer is | ||
| [the composition](/quest/m3/layers/spec.md). It lands in m2, beside lite-07 | ||
| (rejected: m3 with the composition), so lite-07 ships with the rule. | ||
|
|
||
| [The model](/quest/m2/time-addressing/model.md) already has moq-mux's catalog | ||
| producer stamp each catalog from the broadcast clock. Today | ||
| `rs/hang/src/catalog/consumer.rs` drops that stamp (`.map(|value| | ||
| value.value)`), and `js/hang` does likewise; both return it with the | ||
| catalog. A consumer that reads an update ahead of its timestamp holds it | ||
| until media reaches that time; one that reads it late applies it at once. | ||
|
|
||
| A scale-less broadcast's catalog has no media time to wait for, so it applies | ||
| on arrival, as today. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Rust groups are addressed by their start](/quest/m2/time-addressing/model.md) - the catalog producer stamps from the broadcast clock | ||
| - [@moq/net addresses groups by start](/quest/m2/time-addressing/js.md) - the same for js/hang | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,37 +1,56 @@ | ||
| # Layered broadcasts | ||
| # Composed broadcasts | ||
|
|
||
| ## Goal | ||
|
|
||
| hang specifies a layer layout track: a broadcast's main picture and sound plus | ||
| other tracks layered over it, each placed, timed, and mixed on the media | ||
| clock. `moq play` and `@moq/watch` composite those layers over the main | ||
| picture and mix their audio, so a moq-dev/studio broadcast plays as its | ||
| creator composed it, while sources published on their own tracks stay | ||
| separately fetchable and toggleable. | ||
| hang specifies a composition: a broadcast whose catalog lists other | ||
| broadcasts, each on its own timeline with an offset onto the composition's | ||
| clock, and says where and when each shows and how its sound mixes. `moq play` | ||
| and `@moq/watch` play a composition, so a moq-dev/studio broadcast plays as | ||
| its creator composed it, while each source stays its own broadcast that a | ||
| viewer can toggle and fetch at its own priority. | ||
|
|
||
| This README's own work is the end-to-end proof: one fixture broadcast with a | ||
| picture layer and an audio layer plays in `moq play` and `@moq/watch` with the | ||
| same placement and timing, within a stated tolerance, and with the layer's | ||
| sound mixed in. | ||
| This README's own work is the end-to-end proof: one fixture composition with | ||
| a main broadcast, a picture child, and a sound child plays in `moq play` and | ||
| `@moq/watch` with the same placement and timing, within a stated tolerance, | ||
| and with the child's sound mixed in. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-08 (maintainer, moq-dev/studio /quest-plan): | ||
| Replanned 2026-11-12 (maintainer, moq-dev/studio /quest-plan), after the | ||
| studio chose a composition over a layer layout track for naming guests (its | ||
| `quest/m2/compose/layout.md`). hang's `include` requires an included | ||
| broadcast to share the listing catalog's PTS timeline, `media.clock`, and | ||
| `media.join.interval`, which a guest's broadcast does not. | ||
|
|
||
| - The studio publishes the program plus each independently published source | ||
| as its own track, and players composite those sources over the program. | ||
| Rejected: viewers rebuilding the whole scene from every source's track, | ||
| which would take more encoder sessions than consumer GPUs have. | ||
| - hang specifies the layout and each player implements it with its own code. | ||
| The studio's compositor (moq-scene) stays in moq-dev/studio, since moq | ||
| never depends on that repository, so parity comes from the shared fixture, | ||
| not shared code. Rejected: moving moq-scene into moq. | ||
| - The studio prototypes the format first (`moq-studio --watch`, reading a | ||
| provisional `dev.moq.studio.layers` track), and the spec starts from what | ||
| that prototype settles (rejected: drafting the spec in parallel). | ||
| - Players mix layers' audio in, at the gain, mute, pan, and delay the layout | ||
| carries (pan and delay added 2026-11-09, as the spec quest records). | ||
| - Each layer carries a priority and whether a viewer may toggle it. A player | ||
| maps priority onto moq-lite's subscription priorities and unsubscribes a | ||
| toggled-off layer. | ||
| - m3: moq-dev/studio is the named consumer, and its side sits in its m2. | ||
| - **What:** ✅ a composition (rejected: a layer layout track naming tracks of | ||
| one broadcast, the 2026-11-08 plan). Every independently published source | ||
| is its own broadcast, and the composition is a broadcast of its own that | ||
| lists them; the studio's program is one child. | ||
| - **Generality:** ✅ a general way to compose broadcasts, such as multiview or | ||
| picture-in-picture, keeping only what every player can reproduce | ||
| (rejected: exactly what the studio emits). The studio is its first | ||
| producer. | ||
| - **Shape:** ✅ all in the composition broadcast's catalog, its updates timed | ||
| (rejected: the catalog listing children with a separate timed track for | ||
| placement; everything in one track). A catalog update applies at its | ||
| timestamp ([that rule](/quest/m2/catalog-at-timestamp.md), m2). | ||
| - **Clock:** ✅ each child carries the composer's offset from its PTS to the | ||
| composition's (rejected: aligning children by their `media.clock` wall | ||
| times, which depends on publishers' clocks agreeing and drops the latency | ||
| the composer saw). A child that restarts gets a new offset in a new update. | ||
| - **Older players:** ✅ the composition names one child as main, and its | ||
| catalog lists the main's renditions, so a player that knows no | ||
| compositions plays the main picture alone (rejected: an empty catalog for | ||
| them). | ||
| - **Untimed:** ✅ a composition requires a timed catalog; in a scale-less one | ||
| it is a publisher error, and a player plays the main alone (rejected: | ||
| applying updates on arrival). | ||
|
|
||
| Still settled from 2026-11-08: hang specifies the format and each player | ||
| implements it with its own code, so parity comes from the shared fixture, not | ||
| shared code (moq never depends on moq-dev/studio); the studio prototypes the | ||
| format first. Players mix each child's sound at the gain, mute, pan, and | ||
| delay the composition carries. Each child carries a priority and whether a | ||
| viewer may toggle it; a player maps priority onto moq-lite's subscription | ||
| priorities and unsubscribes a toggled-off child. m3: moq-dev/studio is the | ||
| named consumer, and its side sits in its m2. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,17 +1,18 @@ | ||
| # [M] moq play composites layers | ||
| # [M] moq play plays compositions | ||
|
|
||
| ## Goal | ||
|
|
||
| `moq play` plays a broadcast that carries a layer layout track: it composites | ||
| each visible layer over the main picture where and when the layout says, | ||
| mixes each audio layer in at its gain, mute, pan, and delay, and lets a | ||
| viewer toggle each layer the layout marks toggleable. A broadcast without one | ||
| plays as today. | ||
| `moq play` plays a composition: it opens each child broadcast the | ||
| composition lists, places each visible picture child where and when the | ||
| composition says, on the composition's clock through the child's offset, | ||
| mixes each sound child in at its gain, mute, pan, and delay, and lets a | ||
| viewer toggle each child the composition marks toggleable. A broadcast | ||
| without a composition plays as today. | ||
|
|
||
| ## Plan | ||
|
|
||
| Fetch only visible layers, at the layout's priorities through moq-lite's | ||
| subscription priorities, and unsubscribe a toggled-off layer. Prove placement | ||
| Fetch only visible children, at the composition's priorities through moq-lite's | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fetch sound children, not only visible picture children. The fetch rule reads "visible children." The spec defines visibility only for picture children ( Suggested fix
|
||
| subscription priorities, and unsubscribe a toggled-off child. Prove placement | ||
| and timing against the epic's fixture. | ||
|
|
||
| Decided 2026-11-09 (maintainer, moq-dev/studio /quest-audit): the viewer | ||
|
|
@@ -20,9 +21,9 @@ toggle and the priority mapping are this quest's work, not only | |
| `quest/m2/compose/controls.md`) that viewer controls land in the native | ||
| player, and its `hang-layers` quest deletes its prototype player once this | ||
| lands, so this quest is their home. Subscriptions are per track, not per | ||
| layer, as that quest settled: a track several layers show stays subscribed | ||
| child, as that quest settled: a track several children show stays subscribed | ||
| while any of them is on, at the highest priority among them. | ||
|
|
||
| ## Required | ||
|
|
||
| - [hang specifies a layer layout track](/quest/m3/layers/spec.md) - what `moq play` composites | ||
| - [hang specifies a composition](/quest/m3/layers/spec.md) - what `moq play` plays | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,31 +1,48 @@ | ||
| # [M] hang specifies a layer layout track | ||
| # [M] hang specifies a composition | ||
|
|
||
| ## Goal | ||
|
|
||
| `drafts/draft-lcurley-moq-hang.md`, `rs/hang`, and `js/hang` define a layer | ||
| layout track: a versioned JSON track in snapshot mode on the media clock that | ||
| names each layer's track and gives its transform, crop, z-order, opacity, | ||
| visibility, keyframes, priority, and whether a viewer may toggle it. An audio | ||
| layer also carries its gain, mute, pan, and delay. | ||
| `drafts/draft-lcurley-moq-hang.md`, `rs/hang`, and `js/hang` define a | ||
| composition: a versioned catalog section that lists child broadcasts, marks | ||
| one as main, and gives each child its offset from its PTS onto the | ||
| composition's clock, its priority, and whether a viewer may toggle it. A | ||
| picture child also carries its transform, crop, z-order, opacity, visibility, | ||
| and keyframes; a sound child carries its gain, mute, pan, and delay. The | ||
| composition's catalog also lists the main child's renditions, so a consumer | ||
| that knows no compositions plays the main alone. | ||
|
|
||
| ## Plan | ||
|
|
||
| Start from the format moq-dev/studio's watch prototype settled, and keep only | ||
| what every player can reproduce. A consumer refuses an unknown version loudly | ||
| and plays the main picture alone. Layers name tracks and catalog references, | ||
| never files. | ||
| The [epic](/quest/m3/layers/README.md) holds the 2026-11-12 decisions: the | ||
| composition lives in its own broadcast's catalog, applied at each update's | ||
| timestamp; offsets are the composer's; a composition needs a timed catalog, | ||
| and a consumer plays the main alone in a scale-less one. | ||
|
|
||
| Decided 2026-11-09 (maintainer, moq-dev/studio /quest-audit): an audio layer | ||
| Start from the format moq-dev/studio's prototype settles, and keep only what | ||
| every player can reproduce. A consumer refuses an unknown version loudly and | ||
| plays the main alone. Children name broadcasts, relative like `include`'s and | ||
| contained the same way (`check_resolvable` in | ||
| `rs/hang/src/catalog/consumer.rs`), never files. Unlike an include, a child | ||
| keeps its own PTS timeline and root settings, and listing one does not | ||
| subscribe to it. The composition's clock is the main's, so the main's offset | ||
| is zero and the catalog lists its renditions by their `broadcast` reference, | ||
| on the same timeline. | ||
|
|
||
| Decided 2026-11-09 (maintainer, moq-dev/studio /quest-audit): a sound child | ||
| carries pan and delay as well as gain and mute. The studio's strip has all | ||
| four (`rs/moq-scene/src/collection.rs` in moq-dev/studio; pan shipped in | ||
| [moq-dev/studio#116](https://github.com/moq-dev/studio/pull/116)), and a | ||
| source's own track is taken before the strip applies them. Without the delay | ||
| in the layout, a microphone the creator delayed for sync and published on its | ||
| own track would play out of sync for viewers. | ||
| source's own broadcast is taken before the strip applies them. Without the | ||
| delay, a microphone the creator delayed for sync would play out of sync for | ||
| viewers. | ||
|
|
||
| ## Waiting | ||
|
|
||
| - moq-dev/studio: its `moq-studio --watch` prototype plays program plus | ||
| layers from the provisional `dev.moq.studio.layers` track, planned as | ||
| `quest/m2/compose/watch.md` there. Check: that quest's file is gone from | ||
| - moq-dev/studio: the studio publishes a provisional composition and its | ||
| `moq-studio --watch` prototype plays it, planned as | ||
| `quest/m2/compose/layout.md` there. Check: that quest's file is gone from | ||
| moq-dev/studio's `main` because its PR merged. | ||
|
|
||
| ## Required | ||
|
|
||
| - [hang applies catalog updates at their timestamp](/quest/m2/catalog-at-timestamp.md) - a composition's changes land on the frame stamped alike |
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,20 +1,21 @@ | ||||||||||
| # [M] The watch player composites layers | ||||||||||
| # [M] The watch player plays compositions | ||||||||||
|
|
||||||||||
| ## Goal | ||||||||||
|
|
||||||||||
| `@moq/watch` plays a broadcast that carries a layer layout track: it | ||||||||||
| composites each visible layer over the main picture where and when the layout | ||||||||||
| says, mixes each audio layer in at its gain, mute, pan, and delay, and lets a | ||||||||||
| viewer toggle each layer the layout marks toggleable. A broadcast without one | ||||||||||
| plays as today. | ||||||||||
| `@moq/watch` plays a composition: it opens each child broadcast the | ||||||||||
| composition lists, places each visible picture child where and when the | ||||||||||
| composition says, on the composition's clock through the child's offset, | ||||||||||
| mixes each sound child in at its gain, mute, pan, and delay, and lets a | ||||||||||
| viewer toggle each child the composition marks toggleable. A broadcast | ||||||||||
| without a composition plays as today. | ||||||||||
|
|
||||||||||
| ## Plan | ||||||||||
|
|
||||||||||
| Fetch only visible layers, at the layout's priorities through moq-lite's | ||||||||||
| subscription priorities, and unsubscribe a toggled-off layer. Prove placement | ||||||||||
| Fetch only visible children, at the composition's priorities through moq-lite's | ||||||||||
| subscription priorities, and unsubscribe a toggled-off child. Prove placement | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Specify shared-track handling in the watch plan. The watch plan says to unsubscribe a toggled-off child. It does not say what happens when a track is shown by more than one child. If an implementer reads this literally, toggling off one child can unsubscribe a track that another visible child still needs. The Suggested fix
Committable suggestion
|
||||||||||
| and timing against the epic's fixture, matching `moq play` within its | ||||||||||
| tolerance. | ||||||||||
|
|
||||||||||
| ## Required | ||||||||||
|
|
||||||||||
| - [hang specifies a layer layout track](/quest/m3/layers/spec.md) - what `@moq/watch` composites | ||||||||||
| - [hang specifies a composition](/quest/m3/layers/spec.md) - what `@moq/watch` plays | ||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Read timed catalog updates without skipping.
The planned rule applies every timed catalog update at its timestamp. The catalog
Consumer::poll_nextreads throughpoll_latest, which discards older states. Stamp 10 and stamp 20 can both arrive before playback reaches 10. The consumer then reads only the update at 20, and the update at 10 is never applied.The plan should require timed catalog updates to be read in order and without skipping. It should not reuse the latest-value reader for this path.
LanguageTool
[style] ~24-~24: Consider using “who” when you are referring to a person instead of an object.
Context: ... return it with the catalog. A consumer that reads an update ahead of its timestamp ...
(THAT_WHO)