Repository navigation
quest: plan moqsink's first catalog and moqsrc A/V alignment #5189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
1bdd1a9
7b02baf
f12839d
a44ce19
76503ab
e863e76
2191f37
ad04544
12e7dc3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # [S] moqsink's first catalog lists every requested pad | ||
|
|
||
| ## Goal | ||
|
|
||
| When a `moqsink` run starts, its first catalog lists every pad requested so | ||
| far, not only the pads whose caps arrived first. A `moqsrc` following a | ||
| restarted `moqsink` then resumes each rendition on its held pad, instead of | ||
| ending a late-listed one with EOS and adding a new, unlinked pad for it. A | ||
| rendition the new run really drops still ends with EOS. | ||
|
|
||
| Non-goals: other publishers. The CLI importers reserve their track set up | ||
| front, and `<moq-publish>`'s default `source` mode waits for its enabled | ||
| tracks to settle before announcing. `@moq/publish` with `announce="always"`, | ||
| or its `Broadcast` class used directly, announces before its renditions | ||
| resolve and has the same gap, which is planned separately. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-10-10, planning the follow-ups of moqsrc's restart work | ||
| ([#5181](https://github.com/moq-dev/moq/pull/5181), superseded by | ||
| [#5191](https://github.com/moq-dev/moq/pull/5191)): | ||
|
|
||
| - The fix lives in `moqsink`, and `moqsrc`'s rule stays: a rendition the new | ||
| run's first catalog does not list ends with EOS. Holding unlisted | ||
| renditions in `moqsrc` instead would let one that never returns stall a | ||
| downstream muxer. | ||
| - A browser publisher that announces before its tracks settle stays out of | ||
| scope (decided in #5189's review), so this quest stays a `moqsink` change. | ||
| - Each pad reserves its catalog slot (`moq_mux::catalog::Producer::reserve`) | ||
| when it is requested in `request_new_pad` (`rs/moq-gst/src/sink/imp.rs`), | ||
| instead of when its caps arrive (`catalog.reserve()` in | ||
| `rs/moq-gst/src/sink/pad.rs`), hands it to its importer once the caps | ||
| resolve the rendition, and drops it if the pad is released first or its | ||
|
Comment on lines
+31
to
+33
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an H.264 pad receives changed CAPS before its first SPS while a sibling is already catalog-ready, this move-only handoff leaves a gate gap: renegotiation is supported ( AGENTS.md reference: AGENTS.md:L16-L18 Useful? React with 👍 / 👎. |
||
| caps are opaque, which the catalog never lists. The catalog withholds its | ||
|
Comment on lines
+32
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a requested pad receives structurally accepted but unusable caps, such as AAC without AGENTS.md reference: AGENTS.md:L17-L18 Useful? React with 👍 / 👎.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed. (Written by Claude Opus 5.5) |
||
| first snapshot until every reservation resolves, so that snapshot lists | ||
| every pad. The catalog producer exists only during a run (`READY` to | ||
| `PAUSED`), so each run reserves for the pads that exist when it starts | ||
| (`start_session`), and a pad requested mid-run reserves at once. | ||
| - A pad admitted while a run starts must not miss the gate. `request_new_pad` | ||
| counts the admission under the control lock before `add_pad`, so | ||
| `start_session` can run in between and miss the pad in `sink_pads`, and a | ||
| sibling can publish the first snapshot before the pad confirms. So key the | ||
| slot by pad, not by a count: each pad's lifecycle holds at most one | ||
| reservation, and filling it is idempotent. The control state tracks | ||
| admitted-but-unconfirmed pads instead of only `control.admissions`; | ||
| `start_session` fills the slot of every pad in `sink_pads` or that set, a | ||
| mid-run request fills it when it is admitted, and `cancel_admission` or | ||
| release drops it. A pad already visible in `sink_pads`, or whose CAPS | ||
| arrived through a reentrant `child_added` handler, is then never reserved | ||
| twice. | ||
|
|
||
| Decided 2026-10-11 by the maintainer, in #5189: | ||
|
|
||
| - A pad requested but never fed holds back the run's first catalog, so its | ||
| reservation drops when the pad is released, reaches EOS without caps, or | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a video pad has accepted caps but has not yet resolved its catalog config, such as H.264 byte-stream caps followed by EOS before an SPS, the reservation is already owned by the importer, so the stated AGENTS.md reference: AGENTS.md:L18-L18 Useful? React with 👍 / 👎.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed. Once caps arrive the importer owns the reservation, so "EOS without caps" misses a pad that ends before its config resolves, such as H.264 byte-stream ending before its SPS. In the new head the Plan drops the reservation on every terminal path before the rendition resolves, including EOS after caps, and the test covers that case while a sibling streams. (Written by Claude Opus 5.5) |
||
| gets opaque caps. Every other terminal path before the rendition resolves | ||
| drops it too: caps that fail (`CapsOutcome::Failed`, such as AAC without | ||
| `codec_data` or a duplicate track name), and EOS after caps but before the | ||
| importer resolved a config (H.264 byte-stream that ends before its SPS), | ||
| whose importer owns the reservation by then. Today `handle_event(Eos)` | ||
| only marks the pad ended, and producers finalize once every pad ends, so a | ||
| streaming sibling would keep the gate shut. Dropping reservations once | ||
| another pad's first frame is ready was rejected: it brings back the race | ||
| this quest removes. The initial gate only holds the first snapshot, so | ||
| later catalogs never wait. | ||
| - m1, ranked after #3021: only a restart of a `moqsink` whose pads negotiate | ||
| at different times is affected, so it does not gate the release under | ||
| [Broadcast epochs](/quest/m0/broadcast-epoch/README.md). | ||
| - Docs stay inline: the implementing PR updates the `moqsink` and `moqsrc` | ||
| paragraphs of `doc/bin/gstreamer.md`, including that every linked pad must | ||
| negotiate before the broadcast lists anything. No separate docs quest. | ||
|
|
||
| Tests: a `moqsink` whose audio pad gets caps after its video pad's first | ||
| buffer publishes one first catalog listing both; a pad that is released, | ||
| ends without caps, fails its caps, or ends after caps but before its config | ||
| lets the first catalog publish while a sibling streams; a pad requested while | ||
| the run starts is listed even when a sibling resolves before that pad | ||
| confirms, and reserved once even when it is already in `sink_pads`; and a | ||
| `moqsrc` following that `moqsink` across a restart keeps both pads without | ||
| EOS. | ||
|
|
||
| ## Related | ||
|
|
||
| - [#3115](/quest/m2/3115-moqsink-the-publication-has-no-generation-so-a-flush.md) - a new generation re-reserves for its pads too | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # [M] moqsrc keeps audio and video aligned | ||
|
|
||
| ## Goal | ||
|
|
||
| All of a `moqsrc` run's pads share one timestamp reference and one segment | ||
| base, so audio and video keep the relative timing the broadcast gives them, | ||
| at the first start and after every switch. Today each pump starts its PTS at | ||
| its own first frame (`reference_ts` in `rs/moq-gst/src/source/imp.rs`) and | ||
| bases the segment it pushes with its first buffer on the running time that | ||
| buffer arrived, so two tracks line up only as well as their first frames' | ||
| arrivals happen to match their timestamps. A video track joining at a | ||
| keyframe and an audio track joining at its latest group start apart. | ||
|
|
||
| Non-goals: aligning to wall time across hosts, which | ||
| [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) | ||
| settles on the sink side; jitter buffering. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-10-10, planning the follow-ups of moqsrc's restart work | ||
| ([#5181](https://github.com/moq-dev/moq/pull/5181), superseded by | ||
| [#5191](https://github.com/moq-dev/moq/pull/5191)): the tracks of one | ||
| broadcast share one timeline, which the catalog's one `clock` maps to wall | ||
| time, so equal times on different tracks present together. They do not | ||
| share one scale: CMAF keeps each track's `mdhd` timescale, and LOC can carry | ||
| a per-frame one (`moq_mux::container::Frame::timestamp`). `moqsink` already | ||
| publishes aligned tracks (`two_pads_keep_av_aligned_through_real_segments`); | ||
| only `moqsrc` loses the alignment. | ||
|
|
||
| Decided 2026-10-11 by the maintainer, in #5189: | ||
|
|
||
| - Per run, the first frame any pump reads fixes the run's reference | ||
| timestamp and segment base, and every pad of the run maps through both. | ||
| Mapping through the catalog's wall `clock` was rejected: the clock is | ||
| optional, assumes synchronized hosts, and only means something for | ||
| `moqsink` once #3021 lands. | ||
| - Compare times converted to one scale, such as nanoseconds: | ||
| `moq_net::Timestamp::checked_sub` refuses mismatched scales, and | ||
| `relative_pts` turns that refusal into zero. | ||
| - A frame before the run's reference must land before the segment start, | ||
| where the segment clips it. Today `relative_pts` clamps it to zero instead. | ||
| For example, use the converted time as the PTS and the reference as the | ||
| segment start. | ||
| - With one base, a pump whose first frame arrives late in wall time but | ||
| early in media time reaches a clock-synced sink late. That is synced | ||
| playback working as intended, not something to compensate for here. | ||
| - m1, ranked after #3021. | ||
| - Docs stay inline: the implementing PR updates the `moqsrc` paragraph of | ||
| `doc/bin/gstreamer.md`. No separate docs quest. | ||
|
|
||
| Test: an audio and a video frame with the same broadcast time reach | ||
| `moqsrc`'s two pads at the same running time, at the first start and after a | ||
| restart, whichever pump reads first and whichever track's first frame is | ||
| earlier. A frame before the run's reference reaches its pad with a PTS before | ||
| the segment start, for downstream to clip. One case is a CMAF broadcast whose | ||
| tracks use different timescales. | ||
|
|
||
| ## Related | ||
|
|
||
| - [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - the wall epoch on the sink side |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fresh evidence in this final tree: a repo-wide
rgacrossquest/finds no browser first-catalog follow-up beyond the two statements in this file.Broadcaststill defaultsannounceto true (js/publish/src/broadcast.ts:91-96) and serves its current catalog before announcing (:238-264), so a direct API user orannounce="always"publisher can still send a partial first snapshot; on restart,moqsrcthen EOSes the held rendition and later exposes an unlinked replacement pad. Add and link the separate JS quest promised here so this production path is not left unowned. (Written by GPT-5.6 Sol)AGENTS.md reference: AGENTS.md:L48-L48
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct, there is no quest yet for the browser case (
announce="always", orBroadcastused directly). The maintainer chose to plan it separately rather than widen thismoqsinkquest, and planning a new quest belongs to a maintainer-run/quest-planinterview. I'm reporting it back as a follow-up for that, not adding it here.(Written by Claude Opus 5.5)