diff --git a/quest/m1/README.md b/quest/m1/README.md index efb4f35a3f4..363a1fad4d1 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -182,6 +182,8 @@ bench all require its crate. - [Audio warmup](/quest/m1/audio-warmup.md) - a viewer joining an Opus rendition mid-stream never hears the unconverged first 80 ms - [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - moq-gst picks the broadcast wall epoch; a restarted source is a new epoch, not a forward re-anchor - [TS stitch catalog bound](/quest/m1/ts-follow-catalog-bound.md) - `--linger` bounds a mid-stream `--stitch` until the replacement's catalog arrives, so it ends loudly instead of stalling +- [moqsink first catalog](/quest/m1/gst-first-catalog.md) - a restarted `moqsink` lists every requested pad in its first catalog, so `moqsrc` resumes each on its held pad +- [moqsrc A/V alignment](/quest/m1/gst-src-av-sync.md) - every `moqsrc` pad of a run shares one timestamp reference and segment base - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - `export ts --passthrough` writes the `m2ts` track back byte-identical (less late drops) on a fixed delay - [FLV and MKV export delay](/quest/m1/export-delay.md) - FLV and MKV interleave through the shared jitter buffer on a fixed delay, breaking the CLI once - [MKV lacing](/quest/m1/mkv-lacing.md) - laced MKV blocks import as one timed frame each, refused without DefaultDuration diff --git a/quest/m1/gst-first-catalog.md b/quest/m1/gst-first-catalog.md new file mode 100644 index 00000000000..eceaecfb46e --- /dev/null +++ b/quest/m1/gst-first-catalog.md @@ -0,0 +1,84 @@ +# [S] moqsink's first catalog lists every requested pad + +## Goal + +When a `moqsink` run starts, its first catalog lists every pad requested so +far, not only the pads whose caps arrived first. A `moqsrc` following a +restarted `moqsink` then resumes each rendition on its held pad, instead of +ending a late-listed one with EOS and adding a new, unlinked pad for it. A +rendition the new run really drops still ends with EOS. + +Non-goals: other publishers. The CLI importers reserve their track set up +front, and ``'s default `source` mode waits for its enabled +tracks to settle before announcing. `@moq/publish` with `announce="always"`, +or its `Broadcast` class used directly, announces before its renditions +resolve and has the same gap, which is planned separately. + +## Plan + +Decided 2026-10-10, planning the follow-ups of moqsrc's restart work +([#5181](https://github.com/moq-dev/moq/pull/5181), superseded by +[#5191](https://github.com/moq-dev/moq/pull/5191)): + +- The fix lives in `moqsink`, and `moqsrc`'s rule stays: a rendition the new + run's first catalog does not list ends with EOS. Holding unlisted + renditions in `moqsrc` instead would let one that never returns stall a + downstream muxer. +- A browser publisher that announces before its tracks settle stays out of + scope (decided in #5189's review), so this quest stays a `moqsink` change. +- Each pad reserves its catalog slot (`moq_mux::catalog::Producer::reserve`) + when it is requested in `request_new_pad` (`rs/moq-gst/src/sink/imp.rs`), + instead of when its caps arrive (`catalog.reserve()` in + `rs/moq-gst/src/sink/pad.rs`), hands it to its importer once the caps + resolve the rendition, and drops it if the pad is released first or its + caps are opaque, which the catalog never lists. The catalog withholds its + first snapshot until every reservation resolves, so that snapshot lists + every pad. The catalog producer exists only during a run (`READY` to + `PAUSED`), so each run reserves for the pads that exist when it starts + (`start_session`), and a pad requested mid-run reserves at once. +- A pad admitted while a run starts must not miss the gate. `request_new_pad` + counts the admission under the control lock before `add_pad`, so + `start_session` can run in between and miss the pad in `sink_pads`, and a + sibling can publish the first snapshot before the pad confirms. So key the + slot by pad, not by a count: each pad's lifecycle holds at most one + reservation, and filling it is idempotent. The control state tracks + admitted-but-unconfirmed pads instead of only `control.admissions`; + `start_session` fills the slot of every pad in `sink_pads` or that set, a + mid-run request fills it when it is admitted, and `cancel_admission` or + release drops it. A pad already visible in `sink_pads`, or whose CAPS + arrived through a reentrant `child_added` handler, is then never reserved + twice. + +Decided 2026-10-11 by the maintainer, in #5189: + +- A pad requested but never fed holds back the run's first catalog, so its + reservation drops when the pad is released, reaches EOS without caps, or + gets opaque caps. Every other terminal path before the rendition resolves + drops it too: caps that fail (`CapsOutcome::Failed`, such as AAC without + `codec_data` or a duplicate track name), and EOS after caps but before the + importer resolved a config (H.264 byte-stream that ends before its SPS), + whose importer owns the reservation by then. Today `handle_event(Eos)` + only marks the pad ended, and producers finalize once every pad ends, so a + streaming sibling would keep the gate shut. Dropping reservations once + another pad's first frame is ready was rejected: it brings back the race + this quest removes. The initial gate only holds the first snapshot, so + later catalogs never wait. +- m1, ranked after #3021: only a restart of a `moqsink` whose pads negotiate + at different times is affected, so it does not gate the release under + [Broadcast epochs](/quest/m0/broadcast-epoch/README.md). +- Docs stay inline: the implementing PR updates the `moqsink` and `moqsrc` + paragraphs of `doc/bin/gstreamer.md`, including that every linked pad must + negotiate before the broadcast lists anything. No separate docs quest. + +Tests: a `moqsink` whose audio pad gets caps after its video pad's first +buffer publishes one first catalog listing both; a pad that is released, +ends without caps, fails its caps, or ends after caps but before its config +lets the first catalog publish while a sibling streams; a pad requested while +the run starts is listed even when a sibling resolves before that pad +confirms, and reserved once even when it is already in `sink_pads`; and a +`moqsrc` following that `moqsink` across a restart keeps both pads without +EOS. + +## Related + +- [#3115](/quest/m2/3115-moqsink-the-publication-has-no-generation-so-a-flush.md) - a new generation re-reserves for its pads too diff --git a/quest/m1/gst-src-av-sync.md b/quest/m1/gst-src-av-sync.md new file mode 100644 index 00000000000..a739984957a --- /dev/null +++ b/quest/m1/gst-src-av-sync.md @@ -0,0 +1,60 @@ +# [M] moqsrc keeps audio and video aligned + +## Goal + +All of a `moqsrc` run's pads share one timestamp reference and one segment +base, so audio and video keep the relative timing the broadcast gives them, +at the first start and after every switch. Today each pump starts its PTS at +its own first frame (`reference_ts` in `rs/moq-gst/src/source/imp.rs`) and +bases the segment it pushes with its first buffer on the running time that +buffer arrived, so two tracks line up only as well as their first frames' +arrivals happen to match their timestamps. A video track joining at a +keyframe and an audio track joining at its latest group start apart. + +Non-goals: aligning to wall time across hosts, which +[#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) +settles on the sink side; jitter buffering. + +## Plan + +Decided 2026-10-10, planning the follow-ups of moqsrc's restart work +([#5181](https://github.com/moq-dev/moq/pull/5181), superseded by +[#5191](https://github.com/moq-dev/moq/pull/5191)): the tracks of one +broadcast share one timeline, which the catalog's one `clock` maps to wall +time, so equal times on different tracks present together. They do not +share one scale: CMAF keeps each track's `mdhd` timescale, and LOC can carry +a per-frame one (`moq_mux::container::Frame::timestamp`). `moqsink` already +publishes aligned tracks (`two_pads_keep_av_aligned_through_real_segments`); +only `moqsrc` loses the alignment. + +Decided 2026-10-11 by the maintainer, in #5189: + +- Per run, the first frame any pump reads fixes the run's reference + timestamp and segment base, and every pad of the run maps through both. + Mapping through the catalog's wall `clock` was rejected: the clock is + optional, assumes synchronized hosts, and only means something for + `moqsink` once #3021 lands. +- Compare times converted to one scale, such as nanoseconds: + `moq_net::Timestamp::checked_sub` refuses mismatched scales, and + `relative_pts` turns that refusal into zero. +- A frame before the run's reference must land before the segment start, + where the segment clips it. Today `relative_pts` clamps it to zero instead. + For example, use the converted time as the PTS and the reference as the + segment start. +- With one base, a pump whose first frame arrives late in wall time but + early in media time reaches a clock-synced sink late. That is synced + playback working as intended, not something to compensate for here. +- m1, ranked after #3021. +- Docs stay inline: the implementing PR updates the `moqsrc` paragraph of + `doc/bin/gstreamer.md`. No separate docs quest. + +Test: an audio and a video frame with the same broadcast time reach +`moqsrc`'s two pads at the same running time, at the first start and after a +restart, whichever pump reads first and whichever track's first frame is +earlier. A frame before the run's reference reaches its pad with a PTS before +the segment start, for downstream to clip. One case is a CMAF broadcast whose +tracks use different timescales. + +## Related + +- [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - the wall epoch on the sink side