diff --git a/quest/m1/README.md b/quest/m1/README.md index 6d42809dc9f..290891ab9a5 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -41,6 +41,11 @@ Keep one owner per shared code area. Order each prerequisite before its consumer; targeted benchmarks and controlled-time regressions remain part of the feature that needs them. +Added 2026-10-13 by the redundant TS alignment plan: epoch refusals, the +epoch anchor, and raw TS cuts on the epoch timeline, ranked just above +same-epoch importers, which requires them. Catalog clock adoption moved up +above them, since the epoch anchor requires it. + ## Required - [Epoch correctness](/quest/m0/broadcast-epoch/README.md) - publisher identity, duplicate suppression, and catalog pinning gate the breaking cut @@ -77,12 +82,14 @@ the feature that needs them. - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away - [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared source coordinates and distinguish TS identity from coordinated RTP recovery +- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused +- [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, archive, and transcode refuse `--epoch`, since two instances never publish identical tracks +- [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break +- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and a joiner adopts the live M2ts section, so instances started apart publish identical groups and catalogs - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest - [Data consumer timestamps](/quest/m1/data-consumer-timestamps.md) - json and binary consumers return each value's timestamp, in Rust and every binding; snapshots add `latest()` beside an in-order `next()` - [JSON window timestamps](/quest/m1/json-window-timestamps.md) - Rust and JS window consumers return each event's frame timestamp like the snapshot and stream consumers -- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused - [Exact catalog wall clock](/quest/m1/catalog-wall-clock-precision.md) - `hang::catalog::Clock::wall_clock` keeps sub-millisecond precision - [JavaScript answers TRACK_STATUS](/quest/m1/js-track-status.md) - a browser publisher answers IETF TRACK_STATUS like Rust, so a native fetch-only request needs no SUBSCRIBE fallback - [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface diff --git a/quest/m1/archive/replay-catalog.md b/quest/m1/archive/replay-catalog.md index 91464e74d5f..522b33e0177 100644 --- a/quest/m1/archive/replay-catalog.md +++ b/quest/m1/archive/replay-catalog.md @@ -55,8 +55,9 @@ this by hand-building a catalog. catalog groups, so the same name, epoch, and group number could carry different bytes. Revisit only once replayed metadata is byte- and sequence-identical to live. So `moq import archive` refuses `--epoch` - (`takes_epoch` false for `Archive`), and any `moq-archive` host mints its - own epoch or takes one from its caller, never defaulting to `.info`'s. + ([Epoch refusals](/quest/m1/epoch-refusals.md) lands that), and any + `moq-archive` host mints its own epoch or takes one from its caller, never + defaulting to `.info`'s. - The recorded catalog's `archive` entry describes the source's live timelines, not the recording's, so replace it with the timelines the reader replays rather than trusting the recorded ones. diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md new file mode 100644 index 00000000000..8aab48851a1 --- /dev/null +++ b/quest/m1/epoch-anchor.md @@ -0,0 +1,142 @@ +# [M] Same-epoch importers share one anchor and era + +## Goal + +Every importer publishing one path under one explicit `--epoch` places its +timestamps on one coordinate: the catalog root clock that the epoch's first +importer fixed, and the timestamp era that clock implies. A later importer +adopts that clock instead of placing its own, and shifts each timestamp +base by the whole number of the format's timestamp periods that lands it on +the incumbent's live timeline. Two +importers of one stream then publish identical timestamps and catalogs +whenever each started, including on opposite sides of a 33-bit TS wrap or of +a flagged forward clock break. + +Today each importer places its clock at its own first frame's arrival, or +shifts onto a clock already in use by `clock.now()` minus its first +timestamp (#5082), and each TS importer unwraps from its own first PCR or +PTS. Two importers agree only if they started at the same packet. + +## Plan + +Decided 2026-10-13 by the maintainer, over one importer per epoch and over a +contract narrowed to what one input proves: a shared epoch carries no +source-to-wall mapping or era, and only one observation of the source can +supply them, so the epoch fixes them once and nothing re-derives them. The +maintainer accepted the follow-on rules below the same day: eras from the +incumbent's live timeline, joiners adopting history-derived catalog fields, +and the subscribe grant a joiner needs. + +- The epoch's first importer places the clock at its first frame, as today. + A later importer reads the epoch's live catalog before it announces and + adopts its root clock exactly before publishing a frame, through the path + that [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds. + `catalog::Timebase::place` will not do: it takes a `SystemTime`, which + `Clock::wall_clock` floors to milliseconds, so the re-advertised clock + would differ by up to 999 µs. It reads what its relay has now and never + waits on the cluster for completeness; an incumbent it cannot see yet is + the co-start case below. +- The era comes from the incumbent's published timeline, not from the + clock's wall reading. A flagged forward break advances media time without + wall time, so after two accepted +10 h breaks the clock reads about 10 h + while the incumbent publishes 30 h, and a joiner choosing against the + clock lands a full wrap off. Each independent timestamp base (in TS the + PCR, and each stream's PTS and DTS, which unwrap separately) takes the + whole number of periods (2^33 at 90 kHz for TS) that lands its first value + nearest the incumbent's latest published timestamp. One offset taken from + the first frame would misplace a stream whose first value sits across a + wrap from it. Delivery skew alone is far below half a period, so it + changes nothing. Add the periods in source ticks before converting to + microseconds, since a period is not a whole microsecond count. +- The first importer has no incumbent, so it applies the same rule with its + own first value as the reference. Today each stream's `PtsUnwrap` starts + at era 0, so a first importer started at a wrap, with one stream's first + PTS still just before it, places that stream 26.5 h from the others, and + every joiner would adopt the broken coordinate. Timestamps are unsigned, + so the pre-wrap stream's nearest candidate can be negative: resolve the + initial bases together before placing the clock, and lift the whole + coordinate by a period when one would land below zero. A base created + later, such as a PID a PMT revision adds after a wrap (today + `ensure_stream` starts it at `PtsUnwrap::default()`), resolves against + the program's current verified timeline, on every importer alike. +- The joiner verifies its era against the incumbent's objects before + announcing, one per independently mapped timestamp base (each track): the + incumbent's object at the joiner's first group's time must match it in + timestamp and payload. One object would let a matching video track + approve an audio track a period off. A sparse base with no cadence + (verbatim subtitles or teletext, each with its own `PtsUnwrap`) must not + hold the announcement until its next cue: resolve it against its + program's verified PCR base instead. A mismatch exits with an error, and + so does an incumbent that leaves before the check. Decided + 2026-10-13 by the maintainer, over documenting a bound: the latest + timestamp alone is wrong when flagged breaks totalling half a period fall + within the delivery skew (a joiner one second behind, while the incumbent + takes two +10 h breaks, lands a wrap ahead). +- The era choice alone cannot detect a different stream, since a nearest + whole period always exists; the verification above refuses one at join. + Past that, same stream stays the operator's precondition, unchecked like + the rest of the same-epoch contract (maintainer, 2026-10-13). +- Every catalog field an importer derives from its history rather than the + content at hand is the epoch's too. The joiner adopts the live value as of + its first group, and later changes depend only on content on the epoch + timeline. In TS passthrough that is the M2ts section's `random_access` and + recorded `mux_rate`, owned by + [raw TS cuts](/quest/m1/ts-passthrough-epoch-cuts.md). +- Two importers that both start before either is live would place two + clocks under one epoch. One must exit, so that its supervisor's restart + adopts; settle how. Detection by each importer through its own relay + fails in a cluster: a replica on each relay reads before its peer's route + arrives, and `route_order` keeps preferring each relay's own route + afterwards, so neither sees the other. A co-start is also routine, since + both importers of a pair restart together under a successor epoch. + Candidates: an external leader or shared anchor; detection through state + both can see, such as the announcement of the peer's route; or co-start + as an unsupported precondition, which the successor-epoch flow would have + to avoid. +- A flagged forward break must be carried as the same function of the + source's clock values on every importer, so media time stays congruent to + the raw clock modulo the period and the era choice above holds after it. + The raw importer's clamp after a small flagged jump + (`(last + step).max(self.written)` in `passthrough.rs`) breaks that and + depends on history; remove or bound it. +- A rewind stays fatal (#4543), so both importers of a pair exit together + and restart under a successor epoch. Settle how the successor gets its + anchor without the co-start race above. The successor epoch is proposed to + derive from the old epoch and the last frame's catalog time; the same + inputs and the first source timestamp after the step can fix its anchor. +- Nothing outlives the live route, so a total outage loses the anchor: the + next importer under the same explicit epoch is a first importer, and its + era and clock can differ from the old run's. Decided 2026-10-13 by the + maintainer: an explicit epoch names one continuous run, and a supervisor + restarting after every importer stopped mints a successor epoch, as for a + rewind; the `--epoch` docs say so. Rejected: persisting the anchor beyond + the live route, and a flag marking the first importer. +- The mapping drifts from UTC at the source's clock error for the epoch's + life, as one importer's does today. Nothing re-measures it. + +Tests, on mocked time where possible: for each importer that takes +`--epoch`, two instances started at different source offsets publish +identical catalogs and timestamps; the same with the second started after a +33-bit wrap the first crossed, at a wrap with its streams' first values on +opposite sides of it, after flagged forward breaks that together exceed half +a period, with observations straddling such breaks (the verification +refuses a wrong era, on every track), and with different delivery skew into +each; a first importer started at a wrap with its streams on opposite sides +of it; a stream a PMT revision adds after a wrap; a different stream under +the same epoch fails verification; a cold +co-start, including one across two clustered relays, leaves one publisher. + +Public API: no new items, but `--epoch` on an importer now reads the live +broadcast before announcing, so its token needs subscribe rights on its own +path. A publish-only token that works with `--epoch` today fails there: a +breaking auth requirement, documented with `--epoch` in `doc/bin/cli.md` and +`doc/bin/relay/auth.md`. Wire: none. + +## Required + +- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - the exact adoption path a joining importer uses + +## Related + +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair on this coordinate +- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - same-epoch routes resume each other's subscriptions diff --git a/quest/m1/epoch-refusals.md b/quest/m1/epoch-refusals.md new file mode 100644 index 00000000000..54aa86292b8 --- /dev/null +++ b/quest/m1/epoch-refusals.md @@ -0,0 +1,43 @@ +# [S] Publishers that cannot align refuse a shared epoch + +## Goal + +The CLI accepts `--epoch` only where two instances fed one stream can +publish identical tracks. It refuses it, naming the reason, for publishers +whose timestamps or bytes come from the instance rather than the input: + +- `import capture`, since two encoders never produce the same bytes; +- `import avc3`, which stamps frames from their arrival on stdin + (`Split::pts`); +- `import rtc --connect`, since each pull's RTP session is its own; +- `transcode`, for the same reason as capture; +- `import archive`, whose replay rewrites catalog and timeline groups under + the live names, so one name, epoch, and group number could carry + different bytes. + +All five accept it today, so a redundant pair built from them fails at its +first failover instead of at start. + +## Plan + +Decided 2026-10-13 by the maintainer: same-epoch routes resume each other's +subscriptions, so a publisher that cannot meet the +[same-epoch contract](/quest/m1/hop-aligned-import.md) refuses the epoch +rather than documenting a hazard. `takes_epoch` in `rs/moq-cli/src/args.rs` +returns false for the four import sources, and `validate` refuses `--epoch` +with `transcode`. The archive refusal moved here from +[Replay catalog](/quest/m1/archive/replay-catalog.md) so it lands with the +others, ahead of the same-epoch contract. + +Update the `--epoch` paragraph in `doc/bin/cli.md` and the `--epoch` doc +comment to say what accepts it. + +Tests: extend `epoch_applies_only_to_a_once_per_run_publisher` with each +refusal, and keep the other sources accepted. + +Public API: `--epoch` is refused for these five, where it was accepted. + +## Related + +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the contract the accepted importers meet +- [Replay catalog](/quest/m1/archive/replay-catalog.md) - why archive replay mints its own epoch diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 9f2d7fcac16..ef6dfadc297 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -1,4 +1,4 @@ -# [L] Same-epoch importers publish identical tracks +# [M] Same-epoch importers publish identical tracks ## Goal @@ -12,56 +12,63 @@ failover `export ts` sees a timestamp rewind and exits (#4354). ## Plan -The 2026-10-11 audit adds a planning prerequisite for the shared source -coordinate. The group-number and wall-anchor proposals below are provisional -until it settles how the input supplies a common anchor and wrap era. A -shared epoch or raw PTS/PCR value alone does not supply that mapping. Also -resolve raw passthrough explicitly: today's `--epoch` accepts whole MPTS -there, whereas the semantic importer scope below says one program. - Decided: every publisher of one path and epoch MUST publish the same broadcasts and tracks, since routes with the same epoch resume each other's subscriptions. A redundant pair is keyed by a shared explicit epoch (decided -2026-10-03: `--hop` no longer exists to key it). Make every importer that -`moq import --epoch` accepts for one encoded stream (ts with one program, -fmp4, flv, avc3, and the HLS pull that reuses them) meet the contract when -fed one encoded stream. RTMP and SRT imports, the WHIP listener, and -`ts --program all` are out: they mint an epoch per ingest connection -(#4962), so two of them never share one. The WHEP pull and archive replay -are out too (each pull's RTP session and each replay is its own instance), -and so is capture: two encoders never align. Since the CLI still accepts -`--epoch` for `rtc --connect`, make it refuse one (`takes_epoch` false, as -[Replay catalog](/quest/m1/archive/replay-catalog.md) does for archive), with -a regression test. +2026-10-03: `--hop` no longer exists to key it). Every importer that +`moq import --epoch` accepts meets the contract when fed one encoded stream: +ts with one program, ts `--passthrough` (a whole multiplex included), fmp4, +flv, and the HLS pull that reuses them. RTMP and SRT imports, the WHIP +listener, and `ts --program all` are out: they mint an epoch per ingest +connection (#4962), so two of them never share one. Publishers whose +timestamps or bytes come from the instance refuse a shared epoch +([Epoch refusals](/quest/m1/epoch-refusals.md)). + +Decided 2026-10-13 by the maintainer: an epoch has one +coordinate, which its first importer fixes and every later importer adopts +([Epoch anchor](/quest/m1/epoch-anchor.md)). This quest numbers groups on it +and proves the pair end to end. A shared epoch or a raw PTS or PCR value +alone does not supply that coordinate, and arrival time never does. +Considered instead: one importer per epoch with redundancy only downstream, +which stays the supported 1+1 until this lands, and a contract narrowed to +what one input proves, which fails at the first failover across a timestamp +wrap. -- A group's sequence derives from its keyframe's media timestamp (PTS in TS), - not a per-process counter. Decide how both processes agree across a - timestamp wrap (TS PTS wraps every 26.5 h) when they started on opposite - sides of it. -- Frame timestamps derive from the input alone too, and importers refuse a - rewind. Any offset its `catalog::Timebase` applies is input-derived for a - same-epoch importer, never from - `clock.now()` (decided in the 2026-10-06 audit), so two instances shift - identically. -- The catalog's root `clock` must agree too. Today each importer places it - at its own first-frame arrival (`Clock::arrival` in - `rs/moq-mux/src/catalog/producer.rs`), which is also the first published - clock, since each importer holds its catalog until that frame, and a - joining importer is offset by its arrival time. Decided in the 2026-10-05 - audit: redundant importers derive the wall anchor from the input (its PTS - or PCR) or from the shared epoch, never from arrival, so two catalogs of - one stream are identical. They pass it through `catalog::Timebase::place` - rather than a second anchoring path (2026-10-06 audit). Rejected: narrowing - the contract to exclude the catalog clock. +- A group's sequence derives from its first frame's time on the epoch + timeline (the keyframe in the demultiplexed lane, the group's first object + in passthrough), not a per-process counter, so two importers number the + same group alike wherever each started. A source may start consecutive + groups at one timestamp (`container::Producer` accepts a keyframe at the + previous group's start), which time alone cannot number without a + history-derived counter: settle a tie-breaker the source carries, or + refuse equal starts under a shared epoch, and cover it in the pair test. +- Frame timestamps derive from the input and the epoch's coordinate alone, + never from `clock.now()` (2026-10-06 audit), and importers refuse a rewind. +- The catalog's root `clock` is the epoch's, never an importer's arrival + (2026-10-05 audit; narrowing the contract to exclude the catalog clock was + rejected). So is every other catalog field an importer derives from its + history, which a joiner adopts from the live catalog. +- The tracks an importer generates are tracks too: the catalog tracks + (hang, compressed, MSF) and `*.timeline`. Today catalog groups are + numbered locally (`Outputs::emit`), the MSF frame is stamped with + `Timestamp::now()`, and timeline records start at zero, so a joiner would + reuse earlier group or record numbers for newer bytes. Their numbering + and timestamps come from the live track at join or from the epoch + timeline, never the instance. - An importer announces only once it knows its tracks, so it never refuses a track the incumbent serves. -- Docs (`doc/bin/cli.md` "Redundant publishers", the `--epoch` doc comment) - narrow the contract to one encoded stream fed to each publisher. +- Docs (the `--epoch` paragraph in `doc/bin/cli.md`, the `--epoch` doc + comment) state the contract: one encoded stream fed to each publisher, and + a later importer that can reach the live broadcast before it announces. -Tests: per importer, two instances started at different offsets into the same -input produce the same group sequences and timestamps. End to end: the -issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, two -`export ts`) survives the standby joining and the incumbent stopping. +Tests: per importer, two instances started at different offsets into one +input, including the later one starting after a timestamp wrap the earlier +crossed, publish identical catalogs, group sequences, and timestamps, on +the generated catalog and timeline tracks too, joining after earlier +catalog updates and timeline records. End to +end: the issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, +two `export ts`) survives the standby joining, before and after a wrap, and +the incumbent stopping; repeat it with `--passthrough`. ## Closes @@ -70,9 +77,12 @@ issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, two ## Required -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle the source-to-wall anchor, wrap era and raw group coordinate that a shared epoch alone does not provide +- [Epoch refusals](/quest/m1/epoch-refusals.md) - publishers that cannot meet the contract refuse a shared epoch +- [Epoch anchor](/quest/m1/epoch-anchor.md) - one anchor and timestamp era per epoch, adopted by every later importer +- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough groups start where every instance cuts ## Related - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch - [E2EE](/quest/m2/e2ee/README.md) - an encrypting publisher refuses a shared epoch, so a redundant pair is plaintext only +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - what two exports of the pair render from the common coordinate diff --git a/quest/m1/plan-ts-alignment.md b/quest/m1/plan-ts-alignment.md deleted file mode 100644 index 14cc553acd1..00000000000 --- a/quest/m1/plan-ts-alignment.md +++ /dev/null @@ -1,46 +0,0 @@ -# [S] Plan: shared coordinates for redundant TS publishers and exporters - -## Goal - -Settle what shared information makes redundant TS publishers and exporters -identify the same content and schedule. Rewrite the implementation quests -around that contract before promising failover or ST 2022-7 compatibility. - -## Plan - -The 2026-10-11 audit found two missing prerequisites: - -- Same epoch identity does not establish a source PCR/PTS-to-wall mapping, - a 33-bit timestamp wrap era, or a common raw-packet position. Importers - starting at different offsets cannot infer these from the UUID alone. - `catalog::Timebase::place` applies a supplied mapping; it does not discover - one. Raw fallback's timeout cuts also depend on acquisition history, even - though the CLI currently accepts a shared `--epoch` there. -- Identical TS packets are insufficient for ST 2022-7, which requires the - matching RTP packets, including headers. Complementary losses on redundant - paths are a reason to merge them; requiring the two legs to drop the same - content defeats that benefit. - -Present the maintainer with the actual shared-coordinate choices: an -explicit common source anchor and era, a common upstream packetizer, or an -honestly narrowed importer contract. Include what each can prove when the -standby starts after a timestamp wrap, a clock break, or a fallback group cut. -Do not derive a media clock from wall arrival and call it deterministic. - -For egress, separate deterministic TS slot/counter identity for common -content from coordinated RTP packetization, sequence/timestamp/SSRC identity -and any receiver merge. Decide the intended scope and API before implementing -`--sync`. NTP-synchronized hosts alone do not supply the missing source map. - -Produce focused implementation quests and explicit refusals for unsupported -shared-epoch modes. Their automated tests start peers at different source -offsets, cross wraps, vary delivery skew, and introduce complementary loss; -compare common content in its shared coordinate rather than whole files -whose delivered content differs. Actual ST 2022-7 claims require RTP-level -tests against the [standard](https://pub.smpte.org/latest/st2022-7/st2022-7-2019.pdf). - -## Related - -- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - currently assumes an input-derived wall anchor is available -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - TS identity and RTP recovery need separate acceptance evidence -- [TS passthrough export](/quest/m2/ts-passthrough-export.md) - unchanged packets do not establish identical groups from independent starts diff --git a/quest/m1/ts-passthrough-epoch-cuts.md b/quest/m1/ts-passthrough-epoch-cuts.md new file mode 100644 index 00000000000..8c6c7c76f94 --- /dev/null +++ b/quest/m1/ts-passthrough-epoch-cuts.md @@ -0,0 +1,60 @@ +# [S] Raw TS groups and their catalog section follow the epoch timeline + +## Goal + +Two `moq import ts --passthrough` instances sharing one `--epoch` and fed +one stream cut identical groups wherever each started, including on a +source that never marks random access and on a multi-program stream. +Objects already run from PCR to PCR, so only group starts differ: the +fallback cut counts `FALLBACK` from the current group's start, and before +the first group from when the importer began waiting, so on a source with no +random access points each importer keeps its own group phase for good. +Their catalogs' M2ts sections match too, though both fields depend on +history today: one non-random-access cut clears `random_access` for good, +and the recorded `mux_rate` is judged over windows that start at the +importer's first PCR. + +## Plan + +Decided 2026-10-13 by the maintainer: every group start is a function of +the content and the [epoch's timeline](/quest/m1/epoch-anchor.md), never of +when the importer started. + +- Fallback cuts fall on a grid of the epoch timeline, so a joiner's first + group starts where the incumbent cut. A grid on the raw 27 MHz count will + not do: a 33-bit wrap is not a whole number of seconds, so such a grid + shifts by about 0.72 s at each wrap. +- An importer's first group starts only where the incumbent's rules also + cut: a random access point, a flagged forward break, or the grid. +- Settle the grid rule so that a source marking random access more often + than `FALLBACK` is never cut away from its random access points, since any + other start publishes `random_access` false. +- Size cuts count from the group's start, so they follow once group starts + do. +- A cut is a position in the content, so under a shared epoch it follows + media time alone. Decided 2026-10-13 by the maintainer as an explicit + exception to timers advancing on max(wall, pts), because a wall-clock cut + would split a stalled instance from its partner. Rejected: keeping a wall + path. +- A joiner adopts the live M2ts section as of its first group, and its later + changes depend only on content on the epoch timeline: `random_access` + clears on a cut both instances make, and the rate windows sit on the + epoch's grid, or the recorded rate stays fixed for the epoch. Settle which. + +Tests: two instances started at different offsets into one capture publish +identical groups and objects from the later one's first group. Cover a +source with no random access indicator, a multi-program stream, groups that +hit the size cap, the later instance started after a 33-bit wrap the +earlier crossed, and one started after the earlier made a fallback cut and +recorded a rate; their catalogs match. + +Public API: none. + +## Required + +- [Epoch anchor](/quest/m1/epoch-anchor.md) - the timeline the grid sits on + +## Related + +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair +- [TS passthrough export](/quest/m2/ts-passthrough-export.md) - identical objects export as identical packets diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index f53a1f5a9a5..9953eae72ae 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -64,7 +64,8 @@ their current quests. ## Related -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - shared identity requires a shared source coordinate, and TS identity alone is not ST 2022-7 +- [Epoch anchor](/quest/m1/epoch-anchor.md) - shared identity across importers requires one source coordinate per epoch +- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - TS identity alone is not ST 2022-7, which also needs matching RTP headers - [TS health counters](/quest/m2/ts-health-stats.md) - ingress and egress evidence need distinct monitoring points - [MSFTS mapping](/quest/m3/msfts-convergence.md) - document the exact equivalence and differences between the carriage modes diff --git a/quest/m2/README.md b/quest/m2/README.md index a4fef66ee93..02367cda4f9 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -18,6 +18,9 @@ The first follow-through is [release execution](/quest/m2/release/README.md). Rate-grant work targets the next unpublished lite version after finalized lite-07; bitrate enforcement is not part of this release. +Added 2026-10-13 by the redundant TS alignment plan: TS over RTP legs, +ranked just below deterministic TS legs, which it requires. + ## Required - [Time test sweep](/quest/m2/time-test-sweep.md) - unrelated test conversion and the workspace lint ratchet @@ -77,7 +80,8 @@ lite-07; bitrate enforcement is not part of this release. - [One port on the io_uring workers](/quest/m2/uring-demux.md) - `moq-uring`'s workers host the UDP demux, so a ring relay keeps WebRTC media and SRT on its QUIC port - [Synced data playback](/quest/m2/watch-data-sync.md) - js/watch releases JSON and binary payloads on the media playhead, and a slow data track holds media back - [Watch decode gate](/quest/m2/watch-decode-gate.md) - video lookahead stays encoded until it is near presentation, like audio -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - two export legs preserve common TS packet identity; coordinated RTP is a separate prerequisite for ST 2022-7 +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - two `--sync` legs of one broadcast render every common slot identically, continuity counters included, through complementary loss +- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - two legs send identical RTP packets, headers included, so an ST 2022-7 receiver can merge them - [DVB E-AC-3](/quest/m2/ts-eac3.md) - E-AC-3 private data is split per access unit (independent frame plus its dependent substreams) and buffer-modelled in TS export - [T-STD controls](/quest/m2/tstd-controls.md) - the harness gains an MB-overflow control and an AAC broadcast reference - [MP4 export](/quest/m2/mp4-export.md) - `moq export mp4 --output` records crash-safe fragments, then finishes a regular MP4 with moov at the end diff --git a/quest/m2/ts-hitless.md b/quest/m2/ts-hitless.md index 4263c61f862..c0a3ddcac8c 100644 --- a/quest/m2/ts-hitless.md +++ b/quest/m2/ts-hitless.md @@ -1,13 +1,17 @@ -# [L] Redundant TS exports preserve common packet identity +# [L] Redundant TS exports render common content identically ## Goal Two `moq export ts --sync` legs of one broadcast, joined at different -moments, emit packet-identical TS from their first common packet, across -skips and source drift, continuity counters included, within one epoch. -This establishes TS identity only. ST 2022-7 also requires matching RTP -headers and payloads; coordinated RTP egress and receiver recovery need -their own scope and tests before claiming selector compatibility. +moments and fed with different delivery skew, emit identical TS for every +slot whose content both release, continuity counters included, from their +first common slot, within one epoch. A slot whose content one leg lacks +keeps its position in that leg as stuffing, so complementary losses stay +complementary. + +This is TS identity only. RTP packetization for ST 2022-7 is +[its own quest](/quest/m2/ts-rtp-egress.md), and merging the legs is the +receiver's job. Non-goal: identity across a replaced broadcast. A leg ends on a replacement, or starts a fresh stream with `--stitch`, @@ -16,40 +20,75 @@ output across restarts is not supported). ## Plan -Decided (2026-10-01), from t0ms's review of the fixed-delay export (#4645). -They call 2022-7 a differentiator for primary distribution, not a -requirement, so this waits until the T-STD line settles. - -- Measured acquisition stays the default. `--sync` opts in to anchoring on - the catalog clock: release at `clock.wall + pts + delay`, so every leg - anchors the same way however it joined. This needs NTP-synced hosts. If the - measured age is implausible (negative, or beyond the delay), the export - fails loud. -- Continuity counters restart at each group. Each media PID's first packet in - a group sets `discontinuity_indicator`, and its counter is numbered from - there, so it comes from the media and not from what a leg has sent. The PCR - moves to its own PID, because a discontinuity flag on the PCR PID would - declare a time-base break. Loss is still detected within a group. -- For content both legs release, the shared schedule must lay out the same - slots. Independent late delivery can leave complementary gaps; forcing both - legs to drop the same content would defeat redundant-path recovery. The - alignment planning prerequisite settles how the TS scheduler and any RTP - egress preserve common coordinates through those gaps. - t0ms's `two_legs_render_a_skip_the_same_way` - (`rs/moq-mux/src/container/ts/export_timing_test.rs`) stays ignored until - this lands. With the audio late, it also showed a - PES header byte that differs at the skip; find and fix that. -- Test with mocked time: - - two legs, joined a second apart, give byte-identical output including - the counters; - - the same across a mid-run skip, and under ±30 ppm drift; - - `--sync` with an implausible age fails loud. +Decided 2026-10-13 by the maintainer: the coordinate the +legs share is the broadcast's media timeline, each object's timestamp on its +catalog clock, with a constant mux rate. Two exports of one broadcast +already read the same objects and timestamps, so this needs no importer +alignment; [same-epoch importers](/quest/m1/hop-aligned-import.md) extend it +to legs fed by different importers of one epoch. Every byte a `--sync` leg +renders from the objects it has (slot, PCR, table, PES header, continuity +counter) is a function of those objects, that coordinate, the catalog, and +the rate, never of what the leg has sent or when it joined. 2022-7 is a differentiator for primary distribution, not +a requirement (t0ms's review of #4645), so this waits until the T-STD line +settles. + +- Release is unchanged: measured acquisition and the fixed-delay release's + clock recovery. This supersedes the 2026-10-01 plan to release at + `clock.wall + pts + delay`. The catalog's wall mapping is fixed for the + epoch, so a wall-scheduled release spends its delay on the source's clock + error (a 1 s delay lasts about 9 h at 30 ppm), and NTP on the hosts does + not change that. The legs' differential delay stays their delivery skew, + which the receiver absorbs. +- Continuity counters restart at each group. Each media PID's first packet + in a group sets `discontinuity_indicator` and is numbered from there. The + PCR moves to its own PID, because the flag on the PCR PID would declare a + time-base break. Loss is still detected within a group. +- Content one leg misses never changes what the other renders, and the leg + that misses it renders stuffing in its slots. Forcing both legs to drop the + same content would defeat redundant paths. +- A leg that lacks an access unit cannot know how many packets it fills, so + no later packet's slot may depend on an earlier access unit's size. A + constant rate fixes capacity, not placement. Settle the placement rule + before building the scheduler: fixed per-PID reservations on the timeline, + a packet plan carried with the objects, or each group re-anchored at a + slot derived from its timestamp, with headroom. +- t0ms's + `two_legs_render_a_skip_the_same_way` + (`rs/moq-mux/src/container/ts/export_timing_test.rs`) becomes this + complementary case and stops being ignored. With the audio late, it also + showed a PES header byte that differs at the skip; find and fix that. +- `--sync` refuses what it cannot keep: an output with no constant rate + (neither `--mux-rate` nor the catalog's recorded rate), since a measured + rate gives each leg its own grid; `--stitch`, since identity is per epoch; + and `--passthrough`, whose output is already the source's packets. + +Tests on mocked time compare slots keyed by their media time, not by output +position, and assert continuity counters too (`assert_same_packets` stops +masking them under `--sync`): + +- legs joined at different source offsets, mid-group and a group apart; +- different delivery skew and read cadence, each track lagging in turn; +- complementary loss, each leg missing groups the other has; +- a broadcast timeline crossing the 33-bit PCR and PTS wrap while both run, + and a leg joining just after it; +- ±30 ppm source drift; +- each refusal. + +The validation work (#5328) found the `just test ts --pair` arm at 86.96% +SDT agreement over 41 seconds of clean delivery, below its 90% gate. Its +grader associates a table with the next PES PTS; check that association +against the slot scheduler before attributing mismatches to table emission. +Do not loosen the threshold, and do not claim packet identity from table +agreement alone. + +Public API: `moq export ts --sync`. ## Required -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared coordinates and coordinated RTP scope before implementing the public sync contract - [T-STD TS export](/quest/m1/tstd/README.md) - the export, clock recovery, and schedule these legs share ## Related -- [TS passthrough export](/quest/m2/ts-passthrough-export.md) - byte-identical legs for free in its own lane; adopts `--sync` for alignment +- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - packetizes these slots with matching RTP headers for ST 2022-7 +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - extends the coordinate to legs fed by different importers +- [TS passthrough export](/quest/m2/ts-passthrough-export.md) - identical packets for common objects by construction, in its own lane diff --git a/quest/m2/ts-passthrough-export.md b/quest/m2/ts-passthrough-export.md index 0eaa9ea37d3..6ee9738a281 100644 --- a/quest/m2/ts-passthrough-export.md +++ b/quest/m2/ts-passthrough-export.md @@ -57,10 +57,8 @@ promise immediate decode for every program in an MPTS. - Passthrough gives 1+1 identity for free: two exporters fed the same objects emit identical TS packets for every object both release, continuity counters included; an object one leg drops late is a gap in that leg only. - Aligning the legs in time is left to the `--sync` anchor planned for the - demultiplexed export's 2022-7 legs, which passthrough can adopt. That is TS - identity, not ST 2022-7 recovery, which also needs matching RTP headers from - a coordinated RTP egress. + That is TS identity, not ST 2022-7 recovery, which also needs matching RTP + headers from [RTP egress](/quest/m2/ts-rtp-egress.md). - Rust only. This quest owns the reusable raw packet/position and release-schedule oracle @@ -88,8 +86,8 @@ Public API: a new export mode; nothing existing breaks. ## Related -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - common packet identity and the proposed `--sync` anchor +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - the demultiplexed lane's common packet identity under `--sync` - [T-STD TS export](/quest/m1/tstd/README.md) - the jitter buffer and clock recovery this release shares - [TS validation](/quest/m1/ts-reliability/validation.md) - shared automatic graders and controls - [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual missing content and reacquisition proof -- [Redundant alignment](/quest/m1/plan-ts-alignment.md) - identical common objects do not prove independent importers publish identical groups +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - identical common objects do not prove independent importers publish identical groups diff --git a/quest/m2/ts-rtp-egress.md b/quest/m2/ts-rtp-egress.md new file mode 100644 index 00000000000..db3599f4fa5 --- /dev/null +++ b/quest/m2/ts-rtp-egress.md @@ -0,0 +1,60 @@ +# [M] TS over RTP legs that an ST 2022-7 receiver can merge + +## Goal + +`moq export ts` can send its output as RTP (MPEG-2 TS over RTP, RFC 2250), +and two `--sync` legs of one broadcast send the same RTP packets, headers +included, for every datagram both can fill: the same sequence number, +timestamp, SSRC, and payload. A receiver applying +[ST 2022-7](https://pub.smpte.org/latest/st2022-7/st2022-7-2019.pdf) then +reconstructs the common content from whichever leg delivers each datagram, +through complementary loss on the two paths. + +Non-goals: the receiver's merge; a MoQ-side merge of two subscriptions, +which a relay's same-epoch failover already does at the object level; and +identity across a replaced broadcast. + +## Plan + +Decided 2026-10-13 by the maintainer: RTP identity is a +function of [TS slot identity](/quest/m2/ts-hitless.md) and the epoch, and of +nothing a leg mints. + +- Datagrams are cut on the slot grid, so their boundaries fall at the same + packets on every leg. The sequence number and the 90 kHz timestamp come + from the datagram's position on the broadcast's media timeline, modulo + their field widths, so they survive the 33-bit source wrap and a late + join. The SSRC derives from the broadcast path and the route's epoch, so + a replacement is a new RTP stream and two broadcasts sharing one epoch (a + multi-stage import clones its process epoch across paths) differ, barring + a 32-bit hash collision. RFC 3550's collision resolution would split the + legs, so it does not apply. An epochless route (a session whose version carries no + epoch) gives two legs nothing shared to derive it from, so synchronized + RTP refuses one; settle whether an explicit SSRC lifts that. +- A leg never sends a datagram it cannot fill exactly. A packetizer that + completes a datagram with stuffing, or primes a cushion with null-only + datagrams after a stall, sends them under sequence numbers its partner + fills with programme, and a receiver keeps whichever copy arrives first. + Silence lets the partner's copy through. The renderer hands the + packetizer which slots it filled with substitute stuffing, since the + bytes alone cannot tell them from the source's own null packets. +- Datagrams go out on the export's own release; no wall-clock scheduling. +- Settle the sink's shape (a destination on `export ts`, or its own sink), + and whether the passthrough export can be packetized the same way from a + constant-rate source. + +Tests: two legs fed one broadcast with different join offsets, delivery +skew, and complementary group loss send byte-identical RTP packets for every +datagram both send, and a reference ST 2022-7 selection over the two +reproduces every datagram either sent. A datagram neither leg can fill is +absent from both, never stuffing in one, and a datagram holding a slot one +leg substituted is absent from that leg. Sequence numbers and timestamps +wrap identically, a replacement or another path changes the SSRC, and an +epochless route is refused. Claims of ST 2022-7 compatibility cite the standard and an +RTP-level test, not TS identity. + +Public API: an RTP output for TS export, in the shape settled above. + +## Required + +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - the slots and counters these datagrams carry