From 287da9a60d31632127d8ef7bc372e6466c525420 Mon Sep 17 00:00:00 2001 From: t0ms Date: Tue, 13 Oct 2026 14:24:02 +0100 Subject: [PATCH 1/8] quest(m1): settle the shared coordinate for redundant TS Retire the alignment plan. Same-epoch importers adopt the epoch's first catalog clock and timestamp era, passthrough cuts fallback groups on the epoch's grid, and publishers that cannot align refuse --epoch. TS legs render from the broadcast's media timeline under --sync, and RTP packetization for ST 2022-7 is its own quest. Co-authored-by: Claude Opus 5.5 --- quest/m1/README.md | 8 +- quest/m1/epoch-anchor.md | 69 +++++++++++++++++ quest/m1/epoch-refusals.md | 39 ++++++++++ quest/m1/hop-aligned-import.md | 86 ++++++++++----------- quest/m1/plan-ts-alignment.md | 46 ------------ quest/m1/ts-passthrough-epoch-cuts.md | 47 ++++++++++++ quest/m1/ts-passthrough-export.md | 10 +-- quest/m1/ts-reliability/README.md | 3 +- quest/m2/README.md | 6 +- quest/m2/ts-hitless.md | 104 +++++++++++++++++--------- quest/m2/ts-rtp-egress.md | 51 +++++++++++++ 11 files changed, 333 insertions(+), 136 deletions(-) create mode 100644 quest/m1/epoch-anchor.md create mode 100644 quest/m1/epoch-refusals.md delete mode 100644 quest/m1/plan-ts-alignment.md create mode 100644 quest/m1/ts-passthrough-epoch-cuts.md create mode 100644 quest/m2/ts-rtp-egress.md diff --git a/quest/m1/README.md b/quest/m1/README.md index e7f6d811a19..a043b6f4761 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -63,6 +63,10 @@ disabled renditions, ranked just above simulcast rung disable, which publishes them. Rung disable does not wait on it, but until it lands a disabled rung strands WHEP, HLS, and moq-mux export viewers. +Added 2026-10-13 by the redundant TS alignment plan: epoch refusals, the +epoch anchor, and raw TS cuts on the epoch timeline, ranked just above +same-epoch importers, which requires them. + ## Required - [Interop close logs](/quest/m1/interop-close-log.md) - logging fix landed; delete once the required Interop run is green @@ -99,7 +103,9 @@ disabled rung strands WHEP, HLS, and moq-mux export viewers. - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away - [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared source coordinates and distinguish TS identity from coordinated RTP recovery +- [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, and transcode refuse `--epoch`, since two instances never publish identical tracks +- [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break +- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid, so instances started apart cut identical groups - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [moq.sh deploys from CI](/quest/m1/moq-sh-deploy.md) - the first `release` run of the moq.sh workflow deploys with the Workers Editor token - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md new file mode 100644 index 00000000000..048d5ff72eb --- /dev/null +++ b/quest/m1/epoch-anchor.md @@ -0,0 +1,69 @@ +# [M] Same-epoch importers share one anchor and era + +## Goal + +Every importer publishing one path under one explicit `--epoch` places its +timestamps on one coordinate: the catalog root clock that the epoch's first +importer fixed, and the timestamp era that clock implies. A later importer +adopts that clock instead of placing its own, and shifts its timestamps by a +whole number of the format's timestamp periods chosen against it. Two +importers of one stream then publish identical timestamps and catalogs +whenever each started, including on opposite sides of a 33-bit TS wrap or of +a flagged forward clock break. + +Today each importer places its clock at its own first frame's arrival, or +shifts onto a clock already in use by `clock.now()` minus its first +timestamp (#5082), and each TS importer unwraps from its own first PCR or +PTS. Two importers agree only if they started at the same packet. + +## Plan + +Decided 2026-10-13 with the redundant TS alignment plan, over one importer +per epoch and over a contract narrowed to what one input proves: a shared +epoch carries no source-to-wall mapping or era, and only one observation of +the source can supply them, so the epoch fixes them once and nothing +re-derives them. + +- The epoch's first importer places the clock at its first frame, as today. + A later importer reads the epoch's live catalog before it announces and + adopts its root clock before publishing a frame, through + `catalog::Timebase::place` or the adoption path that + [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds. Its token + needs subscribe rights on its own path for that. +- The later importer's offset is the whole number of periods (2^33 at + 90 kHz for TS) that lands its first frame nearest the adopted clock's + reading. Wall time only selects among candidates a period apart, so any + error short of half a period changes nothing, and it never sets the media + clock. No candidate within that bound means a different stream: refuse to + announce. +- Two importers that both start before either is live would place two + clocks under one epoch. The later must detect it and exit, so that its + supervisor's restart adopts; settle how it detects it. +- A flagged forward break must be carried as the same function of the + source's clock values on every importer, so a standby started after it + lands on the incumbent's timeline. The raw importer's clamp after a small + flagged jump (`(last + step).max(self.written)` in `passthrough.rs`) + depends on history; remove or bound it. +- A rewind stays fatal (#4543), so both importers of a pair exit together + and restart under a successor epoch. Settle how the successor gets its + anchor without the co-start race above. The successor epoch is proposed to + derive from the old epoch and the last frame's catalog time; the same + inputs and the first source timestamp after the step can fix its anchor. +- The mapping drifts from UTC at the source's clock error for the epoch's + life, as one importer's does today. Nothing re-measures it. + +Tests, on mocked time where possible: for each importer that takes +`--epoch`, two instances started at different source offsets publish +identical catalogs and timestamps; the same with the second started after a +33-bit wrap the first crossed, after a flagged forward break, and with +different delivery skew into each; a cold co-start leaves one publisher; a +different stream under the same epoch is refused. + +Public API: none new. `--epoch` on an importer reads the live broadcast +before announcing. Wire: none. + +## Related + +- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - the adoption path a joining importer can use +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair on this coordinate +- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - same-epoch routes resume each other's subscriptions diff --git a/quest/m1/epoch-refusals.md b/quest/m1/epoch-refusals.md new file mode 100644 index 00000000000..7109b7eb9ca --- /dev/null +++ b/quest/m1/epoch-refusals.md @@ -0,0 +1,39 @@ +# [S] Publishers that cannot align refuse a shared epoch + +## Goal + +The CLI accepts `--epoch` only where two instances fed one stream can +publish identical tracks. It refuses it, naming the reason, for publishers +whose timestamps or bytes come from the instance rather than the input: + +- `import capture`, since two encoders never produce the same bytes; +- `import avc3`, which stamps frames from their arrival on stdin + (`Split::pts`); +- `import rtc --connect`, since each pull's RTP session is its own; +- `transcode`, for the same reason as capture. + +All four accept it today, so a redundant pair built from them fails at its +first failover instead of at start. + +## Plan + +Decided 2026-10-13 with the redundant TS alignment plan: same-epoch routes +resume each other's subscriptions, so a publisher that cannot meet the +[same-epoch contract](/quest/m1/hop-aligned-import.md) refuses the epoch +rather than documenting a hazard. `takes_epoch` in `rs/moq-cli/src/args.rs` +returns false for the three import sources, and `validate` refuses +`--epoch` with `transcode`. Archive replay's refusal belongs to +[Replay catalog](/quest/m1/archive/replay-catalog.md). + +Update the `--epoch` paragraph in `doc/bin/cli.md` and the `--epoch` doc +comment to say what accepts it. + +Tests: extend `epoch_applies_only_to_a_once_per_run_publisher` with each +refusal, and keep the other sources accepted. + +Public API: `--epoch` is refused for these four, where it was accepted. + +## Related + +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the contract the accepted importers meet +- [Replay catalog](/quest/m1/archive/replay-catalog.md) - refuses `--epoch` for archive replay diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 75c3654ab51..165ed372c93 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -1,4 +1,4 @@ -# [L] Same-epoch importers publish identical tracks +# [M] Same-epoch importers publish identical tracks ## Goal @@ -12,56 +12,49 @@ failover `export ts` sees a timestamp rewind and exits (#4354). ## Plan -The 2026-10-11 audit adds a planning prerequisite for the shared source -coordinate. The group-number and wall-anchor proposals below are provisional -until it settles how the input supplies a common anchor and wrap era. A -shared epoch or raw PTS/PCR value alone does not supply that mapping. Also -resolve raw passthrough explicitly: today's `--epoch` accepts whole MPTS -there, whereas the semantic importer scope below says one program. - Decided: every publisher of one path and epoch MUST publish the same broadcasts and tracks, since routes with the same epoch resume each other's subscriptions. A redundant pair is keyed by a shared explicit epoch (decided -2026-10-03: `--hop` no longer exists to key it). Make every importer that -`moq import --epoch` accepts for one encoded stream (ts with one program, -fmp4, flv, avc3, and the HLS pull that reuses them) meet the contract when -fed one encoded stream. RTMP and SRT imports, the WHIP listener, and -`ts --program all` are out: they mint an epoch per ingest connection -(#4962), so two of them never share one. The WHEP pull and archive replay -are out too (each pull's RTP session and each replay is its own instance), -and so is capture: two encoders never align. Since the CLI still accepts -`--epoch` for `rtc --connect`, make it refuse one (`takes_epoch` false, as -[Replay catalog](/quest/m1/archive/replay-catalog.md) does for archive), with -a regression test. +2026-10-03: `--hop` no longer exists to key it). Every importer that +`moq import --epoch` accepts meets the contract when fed one encoded stream: +ts with one program, ts `--passthrough` (a whole multiplex included), fmp4, +flv, and the HLS pull that reuses them. RTMP and SRT imports, the WHIP +listener, and `ts --program all` are out: they mint an epoch per ingest +connection (#4962), so two of them never share one. Publishers whose +timestamps or bytes come from the instance refuse a shared epoch +([Epoch refusals](/quest/m1/epoch-refusals.md)). + +Decided 2026-10-13 with the redundant TS alignment plan: an epoch has one +coordinate, which its first importer fixes and every later importer adopts +([Epoch anchor](/quest/m1/epoch-anchor.md)). This quest numbers groups on it +and proves the pair end to end. A shared epoch or a raw PTS or PCR value +alone does not supply that coordinate, and arrival time never does. +Considered instead: one importer per epoch with redundancy only downstream, +which stays the supported 1+1 until this lands, and a contract narrowed to +what one input proves, which fails at the first failover across a timestamp +wrap. -- A group's sequence derives from its keyframe's media timestamp (PTS in TS), - not a per-process counter. Decide how both processes agree across a - timestamp wrap (TS PTS wraps every 26.5 h) when they started on opposite - sides of it. -- Frame timestamps derive from the input alone too, and importers refuse a - rewind. Any offset its `catalog::Timebase` applies is input-derived for a - same-epoch importer, never from - `clock.now()` (decided in the 2026-10-06 audit), so two instances shift - identically. -- The catalog's root `clock` must agree too. Today each importer places it - at its own first-frame arrival (`Clock::arrival` in - `rs/moq-mux/src/catalog/producer.rs`), which is also the first published - clock, since each importer holds its catalog until that frame, and a - joining importer is offset by its arrival time. Decided in the 2026-10-05 - audit: redundant importers derive the wall anchor from the input (its PTS - or PCR) or from the shared epoch, never from arrival, so two catalogs of - one stream are identical. They pass it through `catalog::Timebase::place` - rather than a second anchoring path (2026-10-06 audit). Rejected: narrowing - the contract to exclude the catalog clock. +- A group's sequence derives from its first frame's time on the epoch + timeline (the keyframe in the demultiplexed lane, the group's first object + in passthrough), not a per-process counter, so two importers number the + same group alike wherever each started. +- Frame timestamps derive from the input and the epoch's coordinate alone, + never from `clock.now()` (2026-10-06 audit), and importers refuse a rewind. +- The catalog's root `clock` is the epoch's, never an importer's arrival + (2026-10-05 audit; narrowing the contract to exclude the catalog clock was + rejected). - An importer announces only once it knows its tracks, so it never refuses a track the incumbent serves. -- Docs (`doc/bin/cli.md` "Redundant publishers", the `--epoch` doc comment) - narrow the contract to one encoded stream fed to each publisher. +- Docs (the `--epoch` paragraph in `doc/bin/cli.md`, the `--epoch` doc + comment) state the contract: one encoded stream fed to each publisher, and + a later importer that can reach the live broadcast before it announces. -Tests: per importer, two instances started at different offsets into the same -input produce the same group sequences and timestamps. End to end: the -issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, two -`export ts`) survives the standby joining and the incumbent stopping. +Tests: per importer, two instances started at different offsets into one +input, including the later one starting after a timestamp wrap the earlier +crossed, publish identical catalogs, group sequences, and timestamps. End to +end: the issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, +two `export ts`) survives the standby joining, before and after a wrap, and +the incumbent stopping; repeat it with `--passthrough`. ## Closes @@ -70,9 +63,12 @@ issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, two ## Required -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle the source-to-wall anchor, wrap era and raw group coordinate that a shared epoch alone does not provide +- [Epoch refusals](/quest/m1/epoch-refusals.md) - publishers that cannot meet the contract refuse a shared epoch +- [Epoch anchor](/quest/m1/epoch-anchor.md) - one anchor and timestamp era per epoch, adopted by every later importer +- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough groups start where every instance cuts ## Related - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch - [E2EE](/quest/m1/e2ee/README.md) - an encrypting publisher refuses a shared epoch, so a redundant pair is plaintext only +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - what two exports of the pair render from the common coordinate diff --git a/quest/m1/plan-ts-alignment.md b/quest/m1/plan-ts-alignment.md deleted file mode 100644 index 7a53c2dbd61..00000000000 --- a/quest/m1/plan-ts-alignment.md +++ /dev/null @@ -1,46 +0,0 @@ -# [S] Plan: shared coordinates for redundant TS publishers and exporters - -## Goal - -Settle what shared information makes redundant TS publishers and exporters -identify the same content and schedule. Rewrite the implementation quests -around that contract before promising failover or ST 2022-7 compatibility. - -## Plan - -The 2026-10-11 audit found two missing prerequisites: - -- Same epoch identity does not establish a source PCR/PTS-to-wall mapping, - a 33-bit timestamp wrap era, or a common raw-packet position. Importers - starting at different offsets cannot infer these from the UUID alone. - `catalog::Timebase::place` applies a supplied mapping; it does not discover - one. Raw fallback's timeout cuts also depend on acquisition history, even - though the CLI currently accepts a shared `--epoch` there. -- Identical TS packets are insufficient for ST 2022-7, which requires the - matching RTP packets, including headers. Complementary losses on redundant - paths are a reason to merge them; requiring the two legs to drop the same - content defeats that benefit. - -Present the maintainer with the actual shared-coordinate choices: an -explicit common source anchor and era, a common upstream packetizer, or an -honestly narrowed importer contract. Include what each can prove when the -standby starts after a timestamp wrap, a clock break, or a fallback group cut. -Do not derive a media clock from wall arrival and call it deterministic. - -For egress, separate deterministic TS slot/counter identity for common -content from coordinated RTP packetization, sequence/timestamp/SSRC identity -and any receiver merge. Decide the intended scope and API before implementing -`--sync`. NTP-synchronized hosts alone do not supply the missing source map. - -Produce focused implementation quests and explicit refusals for unsupported -shared-epoch modes. Their automated tests start peers at different source -offsets, cross wraps, vary delivery skew, and introduce complementary loss; -compare common content in its shared coordinate rather than whole files -whose delivered content differs. Actual ST 2022-7 claims require RTP-level -tests against the [standard](https://pub.smpte.org/latest/st2022-7/st2022-7-2019.pdf). - -## Related - -- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - currently assumes an input-derived wall anchor is available -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - TS identity and RTP recovery need separate acceptance evidence -- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - unchanged packets do not establish identical groups from independent starts diff --git a/quest/m1/ts-passthrough-epoch-cuts.md b/quest/m1/ts-passthrough-epoch-cuts.md new file mode 100644 index 00000000000..09ada415ba5 --- /dev/null +++ b/quest/m1/ts-passthrough-epoch-cuts.md @@ -0,0 +1,47 @@ +# [S] Raw TS groups cut on the epoch timeline + +## Goal + +Two `moq import ts --passthrough` instances sharing one `--epoch` and fed +one stream cut identical groups wherever each started, including on a +source that never marks random access and on a multi-program stream. +Objects already run from PCR to PCR, so only group starts differ: the +fallback cut counts `FALLBACK` from the current group's start, and before +the first group from when the importer began waiting, so on a source with no +random access points each importer keeps its own group phase for good. + +## Plan + +Decided 2026-10-13 with the redundant TS alignment plan: every group start +is a function of the content and the +[epoch's timeline](/quest/m1/epoch-anchor.md), never of when the importer +started. + +- Fallback cuts fall on a grid of the epoch timeline, so a joiner's first + group starts where the incumbent cut. A grid on the raw 27 MHz count will + not do: a 33-bit wrap is not a whole number of seconds, so such a grid + shifts by about 0.72 s at each wrap. +- An importer's first group starts only where the incumbent's rules also + cut: a random access point, a flagged forward break, or the grid. +- Settle the grid rule so that a source marking random access more often + than `FALLBACK` is never cut away from its random access points, since any + other start publishes `random_access` false. +- Size cuts count from the group's start, so they follow once group starts + do. + +Tests: two instances started at different offsets into one capture publish +identical groups and objects from the later one's first group. Cover a +source with no random access indicator, a multi-program stream, groups that +hit the size cap, and the later instance started after a 33-bit wrap the +earlier crossed. + +Public API: none. + +## Required + +- [Epoch anchor](/quest/m1/epoch-anchor.md) - the timeline the grid sits on + +## Related + +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair +- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - identical objects export as identical packets diff --git a/quest/m1/ts-passthrough-export.md b/quest/m1/ts-passthrough-export.md index a69e43db5fe..34ed2fff1eb 100644 --- a/quest/m1/ts-passthrough-export.md +++ b/quest/m1/ts-passthrough-export.md @@ -47,10 +47,8 @@ promise immediate decode for every program in an MPTS. - Passthrough gives 1+1 identity for free: two exporters fed the same objects emit identical TS packets for every object both release, continuity counters included; an object one leg drops late is a gap in that leg only. - Aligning the legs in time is left to the `--sync` anchor planned for the - demultiplexed export's 2022-7 legs, which passthrough can adopt. That is TS - identity, not ST 2022-7 recovery, which also needs matching RTP headers from - a coordinated RTP egress. + That is TS identity, not ST 2022-7 recovery, which also needs matching RTP + headers from [RTP egress](/quest/m2/ts-rtp-egress.md). - Rust only. This quest owns the reusable raw packet/position and release-schedule oracle @@ -78,8 +76,8 @@ Public API: a new export mode; nothing existing breaks. ## Related -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - common packet identity and the proposed `--sync` anchor +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - the demultiplexed lane's common packet identity under `--sync` - [T-STD TS export](/quest/m1/tstd/README.md) - the jitter buffer and clock recovery this release shares - [TS validation](/quest/m1/ts-reliability/validation.md) - shared automatic graders and controls - [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual missing content and reacquisition proof -- [Redundant alignment](/quest/m1/plan-ts-alignment.md) - identical common objects do not prove independent importers publish identical groups +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - identical common objects do not prove independent importers publish identical groups diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index ac20be15b2d..79c6714324c 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -65,6 +65,7 @@ their current quests. ## Related -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - shared identity requires a shared source coordinate, and TS identity alone is not ST 2022-7 +- [Epoch anchor](/quest/m1/epoch-anchor.md) - shared identity across importers requires one source coordinate per epoch +- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - TS identity alone is not ST 2022-7, which also needs matching RTP headers - [TS health counters](/quest/m2/ts-health-stats.md) - ingress and egress evidence need distinct monitoring points - [MSFTS mapping](/quest/m3/msfts-convergence.md) - document the exact equivalence and differences between the carriage modes diff --git a/quest/m2/README.md b/quest/m2/README.md index a6a79c37fb6..5020513f96c 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -18,6 +18,9 @@ and bench readiness. Refusal reasons folded the same day into [Session outcomes](/quest/m1/session-outcomes.md) and [Typed refusal reason](/quest/m1/refusal-reason.md). +Added 2026-10-13 by the redundant TS alignment plan: TS over RTP legs, +ranked just below deterministic TS legs, which it requires. + ## Required - [Grant stats label](/quest/m2/grant-label.md) - an auth grant's optional `label` names what its session's presence and live bytes count under in stats, apart from the path scope it may access @@ -31,7 +34,8 @@ and bench readiness. Refusal reasons folded the same day into - [One port on the io_uring workers](/quest/m2/uring-demux.md) - `moq-uring`'s workers host the UDP demux, so a ring relay keeps WebRTC media and SRT on its QUIC port - [Synced data playback](/quest/m2/watch-data-sync.md) - js/watch releases JSON and binary payloads on the media playhead, and a slow data track holds media back - [Watch decode gate](/quest/m2/watch-decode-gate.md) - video lookahead stays encoded until it is near presentation, like audio -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - two export legs preserve common TS packet identity; coordinated RTP is a separate prerequisite for ST 2022-7 +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - two `--sync` legs of one broadcast render every common slot identically, continuity counters included, through complementary loss +- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - two legs send identical RTP packets, headers included, so an ST 2022-7 receiver can merge them - [DVB E-AC-3](/quest/m2/ts-eac3.md) - E-AC-3 private data is split per access unit (independent frame plus its dependent substreams) and buffer-modelled in TS export - [T-STD controls](/quest/m2/tstd-controls.md) - the harness gains an MB-overflow control and an AAC broadcast reference - [MP4 export](/quest/m2/mp4-export.md) - `moq export mp4 --output` records crash-safe fragments, then finishes a regular MP4 with moov at the end diff --git a/quest/m2/ts-hitless.md b/quest/m2/ts-hitless.md index dc041cd3d0e..4e698bdc3d4 100644 --- a/quest/m2/ts-hitless.md +++ b/quest/m2/ts-hitless.md @@ -1,13 +1,17 @@ -# [L] Redundant TS exports preserve common packet identity +# [M] Redundant TS exports render common content identically ## Goal Two `moq export ts --sync` legs of one broadcast, joined at different -moments, emit packet-identical TS from their first common packet, across -skips and source drift, continuity counters included, within one epoch. -This establishes TS identity only. ST 2022-7 also requires matching RTP -headers and payloads; coordinated RTP egress and receiver recovery need -their own scope and tests before claiming selector compatibility. +moments and fed with different delivery skew, emit identical TS for every +slot whose content both release, continuity counters included, from their +first common slot, within one epoch. A slot whose content one leg lacks +keeps its position in that leg as stuffing, so complementary losses stay +complementary. + +This is TS identity only. RTP packetization for ST 2022-7 is +[its own quest](/quest/m2/ts-rtp-egress.md), and merging the legs is the +receiver's job. Non-goal: identity across a replaced broadcast. A leg ends on a replacement, or starts a fresh stream with `--stitch`, @@ -16,40 +20,68 @@ output across restarts is not supported). ## Plan -Decided (2026-10-01), from t0ms's review of the fixed-delay export (#4645). -They call 2022-7 a differentiator for primary distribution, not a -requirement, so this waits until the T-STD line settles. - -- Measured acquisition stays the default. `--sync` opts in to anchoring on - the catalog clock: release at `clock.wall + pts + delay`, so every leg - anchors the same way however it joined. This needs NTP-synced hosts. If the - measured age is implausible (negative, or beyond the delay), the export - fails loud. -- Continuity counters restart at each group. Each media PID's first packet in - a group sets `discontinuity_indicator`, and its counter is numbered from - there, so it comes from the media and not from what a leg has sent. The PCR - moves to its own PID, because a discontinuity flag on the PCR PID would - declare a time-base break. Loss is still detected within a group. -- For content both legs release, the shared schedule must lay out the same - slots. Independent late delivery can leave complementary gaps; forcing both - legs to drop the same content would defeat redundant-path recovery. The - alignment planning prerequisite settles how the TS scheduler and any RTP - egress preserve common coordinates through those gaps. - t0ms's `two_legs_render_a_skip_the_same_way` - (`rs/moq-mux/src/container/ts/export_timing_test.rs`) stays ignored until - this lands. With the audio late, it also showed a - PES header byte that differs at the skip; find and fix that. -- Test with mocked time: - - two legs, joined a second apart, give byte-identical output including - the counters; - - the same across a mid-run skip, and under ±30 ppm drift; - - `--sync` with an implausible age fails loud. +Decided 2026-10-13 with the redundant TS alignment plan: the coordinate the +legs share is the broadcast's media timeline, each object's timestamp on its +catalog clock, with a constant mux rate. Two exports of one broadcast +already read the same objects and timestamps, so this needs no importer +alignment; [same-epoch importers](/quest/m1/hop-aligned-import.md) extend it +to legs fed by different importers of one epoch. Every byte a `--sync` leg +renders (slot, PCR, table, PES header, continuity counter) is a function of +that coordinate, the catalog, and the rate, never of what the leg has sent +or when it joined. 2022-7 is a differentiator for primary distribution, not +a requirement (t0ms's review of #4645), so this waits until the T-STD line +settles. + +- Release is unchanged: measured acquisition and the fixed-delay release's + clock recovery. This supersedes the 2026-10-01 plan to release at + `clock.wall + pts + delay`. The catalog's wall mapping is fixed for the + epoch, so a wall-scheduled release spends its delay on the source's clock + error (a 1 s delay lasts about 9 h at 30 ppm), and NTP on the hosts does + not change that. The legs' differential delay stays their delivery skew, + which the receiver absorbs. +- Continuity counters restart at each group. Each media PID's first packet + in a group sets `discontinuity_indicator` and is numbered from there. The + PCR moves to its own PID, because the flag on the PCR PID would declare a + time-base break. Loss is still detected within a group. +- Content one leg misses never changes what the other renders, and the leg + that misses it renders stuffing in its slots. Forcing both legs to drop the + same content would defeat redundant paths. t0ms's + `two_legs_render_a_skip_the_same_way` + (`rs/moq-mux/src/container/ts/export_timing_test.rs`) becomes this + complementary case and stops being ignored. With the audio late, it also + showed a PES header byte that differs at the skip; find and fix that. +- `--sync` refuses what it cannot keep: an output with no constant rate + (neither `--mux-rate` nor the catalog's recorded rate), since a measured + rate gives each leg its own grid; `--stitch`, since identity is per epoch; + and `--passthrough`, whose output is already the source's packets. + +Tests on mocked time compare slots keyed by their media time, not by output +position, and assert continuity counters too (`assert_same_packets` stops +masking them under `--sync`): + +- legs joined at different source offsets, mid-group and a group apart; +- different delivery skew and read cadence, each track lagging in turn; +- complementary loss, each leg missing groups the other has; +- a broadcast timeline crossing the 33-bit PCR and PTS wrap while both run, + and a leg joining just after it; +- ±30 ppm source drift; +- each refusal. + +The validation work (#5328) found the `just test ts --pair` arm at 86.96% +SDT agreement over 41 seconds of clean delivery, below its 90% gate. Its +grader associates a table with the next PES PTS; check that association +against the slot scheduler before attributing mismatches to table emission. +Do not loosen the threshold, and do not claim packet identity from table +agreement alone. + +Public API: `moq export ts --sync`. ## Required -- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared coordinates and coordinated RTP scope before implementing the public sync contract - [T-STD TS export](/quest/m1/tstd/README.md) - the export, clock recovery, and schedule these legs share ## Related -- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - byte-identical legs for free in its own lane; adopts `--sync` for alignment +- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - packetizes these slots with matching RTP headers for ST 2022-7 +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - extends the coordinate to legs fed by different importers +- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - identical packets for common objects by construction, in its own lane diff --git a/quest/m2/ts-rtp-egress.md b/quest/m2/ts-rtp-egress.md new file mode 100644 index 00000000000..ab1e3b77c4d --- /dev/null +++ b/quest/m2/ts-rtp-egress.md @@ -0,0 +1,51 @@ +# [M] TS over RTP legs that an ST 2022-7 receiver can merge + +## Goal + +`moq export ts` can send its output as RTP (MPEG-2 TS over RTP, RFC 2250), +and two `--sync` legs of one broadcast send the same RTP packets, headers +included, for every datagram both can fill: the same sequence number, +timestamp, SSRC, and payload. A receiver applying +[ST 2022-7](https://pub.smpte.org/latest/st2022-7/st2022-7-2019.pdf) then +reconstructs the common content from whichever leg delivers each datagram, +through complementary loss on the two paths. + +Non-goals: the receiver's merge; a MoQ-side merge of two subscriptions, +which a relay's same-epoch failover already does at the object level; and +identity across a replaced broadcast. + +## Plan + +Decided 2026-10-13 with the redundant TS alignment plan: RTP identity is a +function of [TS slot identity](/quest/m2/ts-hitless.md) and the epoch, and of +nothing a leg mints. + +- Datagrams are cut on the slot grid, so their boundaries fall at the same + packets on every leg. The sequence number and the 90 kHz timestamp come + from the datagram's position on the broadcast's media timeline, modulo + their field widths, so they survive the 33-bit source wrap and a late + join. The SSRC derives from the route's epoch, so a replacement is a new + RTP stream; settle whether an explicit SSRC is also accepted. +- A leg never sends a datagram it cannot fill exactly. A packetizer that + completes a datagram with stuffing, or primes a cushion with null-only + datagrams after a stall, sends them under sequence numbers its partner + fills with programme, and a receiver keeps whichever copy arrives first. + Silence lets the partner's copy through. +- Datagrams go out on the export's own release; no wall-clock scheduling. +- Settle the sink's shape (a destination on `export ts`, or its own sink), + and whether the passthrough export can be packetized the same way from a + constant-rate source. + +Tests: two legs fed one broadcast with different join offsets, delivery +skew, and complementary group loss send byte-identical RTP packets for every +datagram both send, and a reference ST 2022-7 selection over the two +reproduces every datagram either sent. A datagram neither leg can fill is +absent from both, never stuffing in one. Sequence numbers and timestamps +wrap identically, and a replacement changes the SSRC. Claims of ST 2022-7 +compatibility cite the standard and an RTP-level test, not TS identity. + +Public API: an RTP destination for `export ts`. + +## Required + +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - the slots and counters these datagrams carry From 0b5ef3484390ae2bdfe71dc866431c169472e252 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 09:55:10 -0700 Subject: [PATCH 2/8] quest(m1): era from the live timeline; history-derived catalog state is the epoch's Record the maintainer's choice of an explicit anchor and era per epoch. A joiner picks each timestamp base's era against the incumbent's latest published timestamp, since accepted forward breaks move media time away from the clock's wall reading. Same stream is an unchecked precondition. Passthrough joiners adopt the live M2ts section. --sync settles a loss-independent packet placement, and synchronized RTP refuses an epochless route. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/epoch-anchor.md | 62 ++++++++++++++++++--------- quest/m1/epoch-refusals.md | 4 +- quest/m1/hop-aligned-import.md | 5 ++- quest/m1/ts-passthrough-epoch-cuts.md | 26 ++++++++--- quest/m2/ts-hitless.md | 19 +++++--- quest/m2/ts-rtp-egress.md | 13 +++--- 7 files changed, 88 insertions(+), 43 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index a043b6f4761..4c08739857b 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -105,7 +105,7 @@ same-epoch importers, which requires them. - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` - [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, and transcode refuse `--epoch`, since two instances never publish identical tracks - [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break -- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid, so instances started apart cut identical groups +- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and a joiner adopts the live M2ts section, so instances started apart publish identical groups and catalogs - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [moq.sh deploys from CI](/quest/m1/moq-sh-deploy.md) - the first `release` run of the moq.sh workflow deploys with the Workers Editor token - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index 048d5ff72eb..d61de0993e3 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -5,8 +5,9 @@ Every importer publishing one path under one explicit `--epoch` places its timestamps on one coordinate: the catalog root clock that the epoch's first importer fixed, and the timestamp era that clock implies. A later importer -adopts that clock instead of placing its own, and shifts its timestamps by a -whole number of the format's timestamp periods chosen against it. Two +adopts that clock instead of placing its own, and shifts each timestamp +base by the whole number of the format's timestamp periods that lands it on +the incumbent's live timeline. Two importers of one stream then publish identical timestamps and catalogs whenever each started, including on opposite sides of a 33-bit TS wrap or of a flagged forward clock break. @@ -18,31 +19,51 @@ PTS. Two importers agree only if they started at the same packet. ## Plan -Decided 2026-10-13 with the redundant TS alignment plan, over one importer -per epoch and over a contract narrowed to what one input proves: a shared -epoch carries no source-to-wall mapping or era, and only one observation of -the source can supply them, so the epoch fixes them once and nothing -re-derives them. +Decided 2026-10-13 by the maintainer, over one importer per epoch and over a +contract narrowed to what one input proves: a shared epoch carries no +source-to-wall mapping or era, and only one observation of the source can +supply them, so the epoch fixes them once and nothing re-derives them. - The epoch's first importer places the clock at its first frame, as today. A later importer reads the epoch's live catalog before it announces and adopts its root clock before publishing a frame, through `catalog::Timebase::place` or the adoption path that [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds. Its token - needs subscribe rights on its own path for that. -- The later importer's offset is the whole number of periods (2^33 at - 90 kHz for TS) that lands its first frame nearest the adopted clock's - reading. Wall time only selects among candidates a period apart, so any - error short of half a period changes nothing, and it never sets the media - clock. No candidate within that bound means a different stream: refuse to - announce. + needs subscribe rights on its own path for that. It reads what its relay + has now and never waits on the cluster for completeness; an incumbent it + cannot see yet is the co-start case below. +- The era comes from the incumbent's published timeline, not from the + clock's wall reading. A flagged forward break advances media time without + wall time, so after two accepted +10 h breaks the clock reads about 10 h + while the incumbent publishes 30 h, and a joiner choosing against the + clock lands a full wrap off. Each independent timestamp base (in TS the + PCR, and each stream's PTS and DTS, which unwrap separately) takes the + whole number of periods (2^33 at 90 kHz for TS) that lands its first value + nearest the incumbent's latest published timestamp. One offset taken from + the first frame would misplace a stream whose first value sits across a + wrap from it. Delivery skew between the two inputs is far below half a + period, so it changes nothing. Add the periods in source ticks before + converting to microseconds, since a period is not a whole microsecond + count. +- Same stream is the operator's precondition, unchecked like the rest of the + same-epoch contract. A nearest whole period always exists, so the era + choice cannot detect a different stream. Not planned: comparing the + joiner's first group with the incumbent's before announcing; propose it if + misconfigured pairs show up. +- Every catalog field an importer derives from its history rather than the + content at hand is the epoch's too. The joiner adopts the live value as of + its first group, and later changes depend only on content on the epoch + timeline. In TS passthrough that is the M2ts section's `random_access` and + recorded `mux_rate`, owned by + [raw TS cuts](/quest/m1/ts-passthrough-epoch-cuts.md). - Two importers that both start before either is live would place two clocks under one epoch. The later must detect it and exit, so that its supervisor's restart adopts; settle how it detects it. - A flagged forward break must be carried as the same function of the - source's clock values on every importer, so a standby started after it - lands on the incumbent's timeline. The raw importer's clamp after a small - flagged jump (`(last + step).max(self.written)` in `passthrough.rs`) + source's clock values on every importer, so media time stays congruent to + the raw clock modulo the period and the era choice above holds after it. + The raw importer's clamp after a small flagged jump + (`(last + step).max(self.written)` in `passthrough.rs`) breaks that and depends on history; remove or bound it. - A rewind stays fatal (#4543), so both importers of a pair exit together and restart under a successor epoch. Settle how the successor gets its @@ -55,9 +76,10 @@ re-derives them. Tests, on mocked time where possible: for each importer that takes `--epoch`, two instances started at different source offsets publish identical catalogs and timestamps; the same with the second started after a -33-bit wrap the first crossed, after a flagged forward break, and with -different delivery skew into each; a cold co-start leaves one publisher; a -different stream under the same epoch is refused. +33-bit wrap the first crossed, at a wrap with its streams' first values on +opposite sides of it, after flagged forward breaks that together exceed half +a period, and with different delivery skew into each; a cold co-start leaves +one publisher. Public API: none new. `--epoch` on an importer reads the live broadcast before announcing. Wire: none. diff --git a/quest/m1/epoch-refusals.md b/quest/m1/epoch-refusals.md index 7109b7eb9ca..5a6f49958d4 100644 --- a/quest/m1/epoch-refusals.md +++ b/quest/m1/epoch-refusals.md @@ -17,8 +17,8 @@ first failover instead of at start. ## Plan -Decided 2026-10-13 with the redundant TS alignment plan: same-epoch routes -resume each other's subscriptions, so a publisher that cannot meet the +Decided 2026-10-13 by the maintainer: same-epoch routes resume each other's +subscriptions, so a publisher that cannot meet the [same-epoch contract](/quest/m1/hop-aligned-import.md) refuses the epoch rather than documenting a hazard. `takes_epoch` in `rs/moq-cli/src/args.rs` returns false for the three import sources, and `validate` refuses diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 165ed372c93..c498e318713 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -24,7 +24,7 @@ connection (#4962), so two of them never share one. Publishers whose timestamps or bytes come from the instance refuse a shared epoch ([Epoch refusals](/quest/m1/epoch-refusals.md)). -Decided 2026-10-13 with the redundant TS alignment plan: an epoch has one +Decided 2026-10-13 by the maintainer: an epoch has one coordinate, which its first importer fixes and every later importer adopts ([Epoch anchor](/quest/m1/epoch-anchor.md)). This quest numbers groups on it and proves the pair end to end. A shared epoch or a raw PTS or PCR value @@ -42,7 +42,8 @@ wrap. never from `clock.now()` (2026-10-06 audit), and importers refuse a rewind. - The catalog's root `clock` is the epoch's, never an importer's arrival (2026-10-05 audit; narrowing the contract to exclude the catalog clock was - rejected). + rejected). So is every other catalog field an importer derives from its + history, which a joiner adopts from the live catalog. - An importer announces only once it knows its tracks, so it never refuses a track the incumbent serves. - Docs (the `--epoch` paragraph in `doc/bin/cli.md`, the `--epoch` doc diff --git a/quest/m1/ts-passthrough-epoch-cuts.md b/quest/m1/ts-passthrough-epoch-cuts.md index 09ada415ba5..1a7bf155736 100644 --- a/quest/m1/ts-passthrough-epoch-cuts.md +++ b/quest/m1/ts-passthrough-epoch-cuts.md @@ -1,4 +1,4 @@ -# [S] Raw TS groups cut on the epoch timeline +# [S] Raw TS groups and their catalog section follow the epoch timeline ## Goal @@ -9,13 +9,16 @@ Objects already run from PCR to PCR, so only group starts differ: the fallback cut counts `FALLBACK` from the current group's start, and before the first group from when the importer began waiting, so on a source with no random access points each importer keeps its own group phase for good. +Their catalogs' M2ts sections match too, though both fields depend on +history today: one non-random-access cut clears `random_access` for good, +and the recorded `mux_rate` is judged over windows that start at the +importer's first PCR. ## Plan -Decided 2026-10-13 with the redundant TS alignment plan: every group start -is a function of the content and the -[epoch's timeline](/quest/m1/epoch-anchor.md), never of when the importer -started. +Decided 2026-10-13 by the maintainer: every group start is a function of +the content and the [epoch's timeline](/quest/m1/epoch-anchor.md), never of +when the importer started. - Fallback cuts fall on a grid of the epoch timeline, so a joiner's first group starts where the incumbent cut. A grid on the raw 27 MHz count will @@ -28,12 +31,21 @@ started. other start publishes `random_access` false. - Size cuts count from the group's start, so they follow once group starts do. +- A cut is a position in the content, so it follows media time alone. A + wall-clock cut path would put a stalled instance's groups apart from its + partner's, so this lane is an exception to timers advancing on + max(wall, pts). +- A joiner adopts the live M2ts section as of its first group, and its later + changes depend only on content on the epoch timeline: `random_access` + clears on a cut both instances make, and the rate windows sit on the + epoch's grid, or the recorded rate stays fixed for the epoch. Settle which. Tests: two instances started at different offsets into one capture publish identical groups and objects from the later one's first group. Cover a source with no random access indicator, a multi-program stream, groups that -hit the size cap, and the later instance started after a 33-bit wrap the -earlier crossed. +hit the size cap, the later instance started after a 33-bit wrap the +earlier crossed, and one started after the earlier made a fallback cut and +recorded a rate; their catalogs match. Public API: none. diff --git a/quest/m2/ts-hitless.md b/quest/m2/ts-hitless.md index 4e698bdc3d4..7d80dc4ac95 100644 --- a/quest/m2/ts-hitless.md +++ b/quest/m2/ts-hitless.md @@ -1,4 +1,4 @@ -# [M] Redundant TS exports render common content identically +# [L] Redundant TS exports render common content identically ## Goal @@ -20,15 +20,15 @@ output across restarts is not supported). ## Plan -Decided 2026-10-13 with the redundant TS alignment plan: the coordinate the +Decided 2026-10-13 by the maintainer: the coordinate the legs share is the broadcast's media timeline, each object's timestamp on its catalog clock, with a constant mux rate. Two exports of one broadcast already read the same objects and timestamps, so this needs no importer alignment; [same-epoch importers](/quest/m1/hop-aligned-import.md) extend it to legs fed by different importers of one epoch. Every byte a `--sync` leg -renders (slot, PCR, table, PES header, continuity counter) is a function of -that coordinate, the catalog, and the rate, never of what the leg has sent -or when it joined. 2022-7 is a differentiator for primary distribution, not +renders from the objects it has (slot, PCR, table, PES header, continuity +counter) is a function of those objects, that coordinate, the catalog, and +the rate, never of what the leg has sent or when it joined. 2022-7 is a differentiator for primary distribution, not a requirement (t0ms's review of #4645), so this waits until the T-STD line settles. @@ -45,7 +45,14 @@ settles. time-base break. Loss is still detected within a group. - Content one leg misses never changes what the other renders, and the leg that misses it renders stuffing in its slots. Forcing both legs to drop the - same content would defeat redundant paths. t0ms's + same content would defeat redundant paths. +- A leg that lacks an access unit cannot know how many packets it fills, so + no later packet's slot may depend on an earlier access unit's size. A + constant rate fixes capacity, not placement. Settle the placement rule + before building the scheduler: fixed per-PID reservations on the timeline, + a packet plan carried with the objects, or each group re-anchored at a + slot derived from its timestamp, with headroom. +- t0ms's `two_legs_render_a_skip_the_same_way` (`rs/moq-mux/src/container/ts/export_timing_test.rs`) becomes this complementary case and stops being ignored. With the audio late, it also diff --git a/quest/m2/ts-rtp-egress.md b/quest/m2/ts-rtp-egress.md index ab1e3b77c4d..02bed5a5791 100644 --- a/quest/m2/ts-rtp-egress.md +++ b/quest/m2/ts-rtp-egress.md @@ -16,7 +16,7 @@ identity across a replaced broadcast. ## Plan -Decided 2026-10-13 with the redundant TS alignment plan: RTP identity is a +Decided 2026-10-13 by the maintainer: RTP identity is a function of [TS slot identity](/quest/m2/ts-hitless.md) and the epoch, and of nothing a leg mints. @@ -25,7 +25,9 @@ nothing a leg mints. from the datagram's position on the broadcast's media timeline, modulo their field widths, so they survive the 33-bit source wrap and a late join. The SSRC derives from the route's epoch, so a replacement is a new - RTP stream; settle whether an explicit SSRC is also accepted. + RTP stream. An epochless route (a session whose version carries no + epoch) gives two legs nothing shared to derive it from, so synchronized + RTP refuses one; settle whether an explicit SSRC lifts that. - A leg never sends a datagram it cannot fill exactly. A packetizer that completes a datagram with stuffing, or primes a cushion with null-only datagrams after a stall, sends them under sequence numbers its partner @@ -41,10 +43,11 @@ skew, and complementary group loss send byte-identical RTP packets for every datagram both send, and a reference ST 2022-7 selection over the two reproduces every datagram either sent. A datagram neither leg can fill is absent from both, never stuffing in one. Sequence numbers and timestamps -wrap identically, and a replacement changes the SSRC. Claims of ST 2022-7 -compatibility cite the standard and an RTP-level test, not TS identity. +wrap identically, a replacement changes the SSRC, and an epochless route is +refused. Claims of ST 2022-7 compatibility cite the standard and an +RTP-level test, not TS identity. -Public API: an RTP destination for `export ts`. +Public API: an RTP output for TS export, in the shape settled above. ## Required From 6ca0e8a83c81d7aed5106163040f361264752e0b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 10:07:11 -0700 Subject: [PATCH 3/8] quest(m1): exact clock adoption, archive refusal, and outage limits for epoch anchors The epoch anchor requires exact catalog clock adoption, ranked above it, since Timebase::place floors the wall to milliseconds. It names the skew window in which breaks can still mislead era selection, the anchor lost in a total outage, and the subscribe grant a joining importer now needs. Archive replay's --epoch refusal moves into epoch refusals. RTP SSRCs key on path and epoch, and the packetizer learns which slots were substituted. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 7 +++-- quest/m1/archive/replay-catalog.md | 5 +-- quest/m1/epoch-anchor.md | 50 +++++++++++++++++++++--------- quest/m1/epoch-refusals.md | 18 ++++++----- quest/m2/ts-rtp-egress.md | 17 ++++++---- 5 files changed, 64 insertions(+), 33 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 4c08739857b..fd154c27b50 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -65,7 +65,8 @@ disabled rung strands WHEP, HLS, and moq-mux export viewers. Added 2026-10-13 by the redundant TS alignment plan: epoch refusals, the epoch anchor, and raw TS cuts on the epoch timeline, ranked just above -same-epoch importers, which requires them. +same-epoch importers, which requires them. Catalog clock adoption moved up +above them, since the epoch anchor requires it. ## Required @@ -103,7 +104,8 @@ same-epoch importers, which requires them. - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away - [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` -- [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, and transcode refuse `--epoch`, since two instances never publish identical tracks +- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused +- [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, archive, and transcode refuse `--epoch`, since two instances never publish identical tracks - [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break - [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and a joiner adopts the live M2ts section, so instances started apart publish identical groups and catalogs - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives @@ -111,7 +113,6 @@ same-epoch importers, which requires them. - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest - [Data consumer timestamps](/quest/m1/data-consumer-timestamps.md) - json and binary consumers return each value's timestamp, in Rust and every binding; snapshots add `latest()` beside an in-order `next()` - [JSON window timestamps](/quest/m1/json-window-timestamps.md) - Rust and JS window consumers return each event's frame timestamp like the snapshot and stream consumers -- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused - [Exact catalog wall clock](/quest/m1/catalog-wall-clock-precision.md) - `hang::catalog::Clock::wall_clock` keeps sub-millisecond precision - [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface - [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native FILL_TIMEOUT bounds upstream waiting on drafts 18-22, alongside local content-age enforcement diff --git a/quest/m1/archive/replay-catalog.md b/quest/m1/archive/replay-catalog.md index c61f9d820c3..cf496bcac7a 100644 --- a/quest/m1/archive/replay-catalog.md +++ b/quest/m1/archive/replay-catalog.md @@ -55,8 +55,9 @@ this by hand-building a catalog. catalog groups, so the same name, epoch, and group number could carry different bytes. Revisit only once replayed metadata is byte- and sequence-identical to live. So `moq import archive` refuses `--epoch` - (`takes_epoch` false for `Archive`), and any `moq-archive` host mints its - own epoch or takes one from its caller, never defaulting to `.info`'s. + ([Epoch refusals](/quest/m1/epoch-refusals.md) lands that), and any + `moq-archive` host mints its own epoch or takes one from its caller, never + defaulting to `.info`'s. - The recorded catalog's `archive` entry describes the source's live timelines, not the recording's, so replace it with the timelines the reader replays rather than trusting the recorded ones. diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index d61de0993e3..0420a6fae65 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -26,12 +26,13 @@ supply them, so the epoch fixes them once and nothing re-derives them. - The epoch's first importer places the clock at its first frame, as today. A later importer reads the epoch's live catalog before it announces and - adopts its root clock before publishing a frame, through - `catalog::Timebase::place` or the adoption path that - [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds. Its token - needs subscribe rights on its own path for that. It reads what its relay - has now and never waits on the cluster for completeness; an incumbent it - cannot see yet is the co-start case below. + adopts its root clock exactly before publishing a frame, through the path + that [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds. + `catalog::Timebase::place` will not do: it takes a `SystemTime`, which + `Clock::wall_clock` floors to milliseconds, so the re-advertised clock + would differ by up to 999 µs. It reads what its relay has now and never + waits on the cluster for completeness; an incumbent it cannot see yet is + the co-start case below. - The era comes from the incumbent's published timeline, not from the clock's wall reading. A flagged forward break advances media time without wall time, so after two accepted +10 h breaks the clock reads about 10 h @@ -41,15 +42,21 @@ supply them, so the epoch fixes them once and nothing re-derives them. whole number of periods (2^33 at 90 kHz for TS) that lands its first value nearest the incumbent's latest published timestamp. One offset taken from the first frame would misplace a stream whose first value sits across a - wrap from it. Delivery skew between the two inputs is far below half a - period, so it changes nothing. Add the periods in source ticks before + wrap from it. Delivery skew alone is far below half a period, so it + changes nothing. Add the periods in source ticks before converting to microseconds, since a period is not a whole microsecond count. +- The latest timestamp is still wrong when flagged breaks totalling half a + period fall within the delivery skew between the two observations: a + joiner one second behind, while the incumbent takes two +10 h breaks, + lands a wrap ahead. Settle whether the joiner confirms its era on an + overlapping observation before announcing (the incumbent's object at its + own first group's time, which would also catch a different stream), or + the docs state that bound. - Same stream is the operator's precondition, unchecked like the rest of the same-epoch contract. A nearest whole period always exists, so the era - choice cannot detect a different stream. Not planned: comparing the - joiner's first group with the incumbent's before announcing; propose it if - misconfigured pairs show up. + choice cannot detect a different stream. No check is planned for identity + alone; the overlapping observation above would catch one if chosen. - Every catalog field an importer derives from its history rather than the content at hand is the epoch's too. The joiner adopts the live value as of its first group, and later changes depend only on content on the epoch @@ -70,6 +77,12 @@ supply them, so the epoch fixes them once and nothing re-derives them. anchor without the co-start race above. The successor epoch is proposed to derive from the old epoch and the last frame's catalog time; the same inputs and the first source timestamp after the step can fix its anchor. +- Nothing outlives the live route, so a total outage loses the anchor: the + next importer under the same explicit epoch is a first importer, and its + era and clock can differ from the old run's. Persisting the anchor is out + of scope, so an explicit epoch names one continuous run, and a supervisor + restarting after every importer stopped mints a successor epoch, as for a + rewind. Settle how that is surfaced with the successor-epoch item above. - The mapping drifts from UTC at the source's clock error for the epoch's life, as one importer's does today. Nothing re-measures it. @@ -78,14 +91,21 @@ Tests, on mocked time where possible: for each importer that takes identical catalogs and timestamps; the same with the second started after a 33-bit wrap the first crossed, at a wrap with its streams' first values on opposite sides of it, after flagged forward breaks that together exceed half -a period, and with different delivery skew into each; a cold co-start leaves +a period, with observations straddling such breaks (per the rule settled +above), and with different delivery skew into each; a cold co-start leaves one publisher. -Public API: none new. `--epoch` on an importer reads the live broadcast -before announcing. Wire: none. +Public API: no new items, but `--epoch` on an importer now reads the live +broadcast before announcing, so its token needs subscribe rights on its own +path. A publish-only token that works with `--epoch` today fails there: a +breaking auth requirement, documented with `--epoch` in `doc/bin/cli.md` and +`doc/bin/relay/auth.md`. Wire: none. + +## Required + +- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - the exact adoption path a joining importer uses ## Related -- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - the adoption path a joining importer can use - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair on this coordinate - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - same-epoch routes resume each other's subscriptions diff --git a/quest/m1/epoch-refusals.md b/quest/m1/epoch-refusals.md index 5a6f49958d4..54aa86292b8 100644 --- a/quest/m1/epoch-refusals.md +++ b/quest/m1/epoch-refusals.md @@ -10,9 +10,12 @@ whose timestamps or bytes come from the instance rather than the input: - `import avc3`, which stamps frames from their arrival on stdin (`Split::pts`); - `import rtc --connect`, since each pull's RTP session is its own; -- `transcode`, for the same reason as capture. +- `transcode`, for the same reason as capture; +- `import archive`, whose replay rewrites catalog and timeline groups under + the live names, so one name, epoch, and group number could carry + different bytes. -All four accept it today, so a redundant pair built from them fails at its +All five accept it today, so a redundant pair built from them fails at its first failover instead of at start. ## Plan @@ -21,9 +24,10 @@ Decided 2026-10-13 by the maintainer: same-epoch routes resume each other's subscriptions, so a publisher that cannot meet the [same-epoch contract](/quest/m1/hop-aligned-import.md) refuses the epoch rather than documenting a hazard. `takes_epoch` in `rs/moq-cli/src/args.rs` -returns false for the three import sources, and `validate` refuses -`--epoch` with `transcode`. Archive replay's refusal belongs to -[Replay catalog](/quest/m1/archive/replay-catalog.md). +returns false for the four import sources, and `validate` refuses `--epoch` +with `transcode`. The archive refusal moved here from +[Replay catalog](/quest/m1/archive/replay-catalog.md) so it lands with the +others, ahead of the same-epoch contract. Update the `--epoch` paragraph in `doc/bin/cli.md` and the `--epoch` doc comment to say what accepts it. @@ -31,9 +35,9 @@ comment to say what accepts it. Tests: extend `epoch_applies_only_to_a_once_per_run_publisher` with each refusal, and keep the other sources accepted. -Public API: `--epoch` is refused for these four, where it was accepted. +Public API: `--epoch` is refused for these five, where it was accepted. ## Related - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the contract the accepted importers meet -- [Replay catalog](/quest/m1/archive/replay-catalog.md) - refuses `--epoch` for archive replay +- [Replay catalog](/quest/m1/archive/replay-catalog.md) - why archive replay mints its own epoch diff --git a/quest/m2/ts-rtp-egress.md b/quest/m2/ts-rtp-egress.md index 02bed5a5791..ec608fbb36e 100644 --- a/quest/m2/ts-rtp-egress.md +++ b/quest/m2/ts-rtp-egress.md @@ -24,15 +24,19 @@ nothing a leg mints. packets on every leg. The sequence number and the 90 kHz timestamp come from the datagram's position on the broadcast's media timeline, modulo their field widths, so they survive the 33-bit source wrap and a late - join. The SSRC derives from the route's epoch, so a replacement is a new - RTP stream. An epochless route (a session whose version carries no + join. The SSRC derives from the broadcast path and the route's epoch, so + a replacement is a new RTP stream and two broadcasts sharing one epoch + (a multi-stage import clones its process epoch across paths) never + collide. An epochless route (a session whose version carries no epoch) gives two legs nothing shared to derive it from, so synchronized RTP refuses one; settle whether an explicit SSRC lifts that. - A leg never sends a datagram it cannot fill exactly. A packetizer that completes a datagram with stuffing, or primes a cushion with null-only datagrams after a stall, sends them under sequence numbers its partner fills with programme, and a receiver keeps whichever copy arrives first. - Silence lets the partner's copy through. + Silence lets the partner's copy through. The renderer hands the + packetizer which slots it filled with substitute stuffing, since the + bytes alone cannot tell them from the source's own null packets. - Datagrams go out on the export's own release; no wall-clock scheduling. - Settle the sink's shape (a destination on `export ts`, or its own sink), and whether the passthrough export can be packetized the same way from a @@ -42,9 +46,10 @@ Tests: two legs fed one broadcast with different join offsets, delivery skew, and complementary group loss send byte-identical RTP packets for every datagram both send, and a reference ST 2022-7 selection over the two reproduces every datagram either sent. A datagram neither leg can fill is -absent from both, never stuffing in one. Sequence numbers and timestamps -wrap identically, a replacement changes the SSRC, and an epochless route is -refused. Claims of ST 2022-7 compatibility cite the standard and an +absent from both, never stuffing in one, and a datagram holding a slot one +leg substituted is absent from that leg. Sequence numbers and timestamps +wrap identically, a replacement or another path changes the SSRC, and an +epochless route is refused. Claims of ST 2022-7 compatibility cite the standard and an RTP-level test, not TS identity. Public API: an RTP output for TS export, in the shape settled above. From e1a7e95105dce9be98e807a11c97a1cc6b6888b0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 10:22:40 -0700 Subject: [PATCH 4/8] quest(m2): scope the RTP SSRC uniqueness claim Co-Authored-By: Claude Opus 5.5 --- quest/m2/ts-rtp-egress.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/quest/m2/ts-rtp-egress.md b/quest/m2/ts-rtp-egress.md index ec608fbb36e..db3599f4fa5 100644 --- a/quest/m2/ts-rtp-egress.md +++ b/quest/m2/ts-rtp-egress.md @@ -25,9 +25,10 @@ nothing a leg mints. from the datagram's position on the broadcast's media timeline, modulo their field widths, so they survive the 33-bit source wrap and a late join. The SSRC derives from the broadcast path and the route's epoch, so - a replacement is a new RTP stream and two broadcasts sharing one epoch - (a multi-stage import clones its process epoch across paths) never - collide. An epochless route (a session whose version carries no + a replacement is a new RTP stream and two broadcasts sharing one epoch (a + multi-stage import clones its process epoch across paths) differ, barring + a 32-bit hash collision. RFC 3550's collision resolution would split the + legs, so it does not apply. An epochless route (a session whose version carries no epoch) gives two legs nothing shared to derive it from, so synchronized RTP refuses one; settle whether an explicit SSRC lifts that. - A leg never sends a datagram it cannot fill exactly. A packetizer that From 7931d3669723a88e91e1b1f0fc474a316d9a23f2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 10:41:35 -0700 Subject: [PATCH 5/8] quest(m1): record the maintainer's era check, outage, and cut-timing decisions A joiner verifies its era against one of the incumbent's objects before announcing, which also refuses a different stream. An explicit epoch names one continuous run, so a restart after a total outage mints a successor. Passthrough cuts under a shared epoch follow media time only, an explicit exception to max(wall, pts). Co-Authored-By: Claude Opus 5.5 --- quest/m1/epoch-anchor.md | 47 +++++++++++++++------------ quest/m1/ts-passthrough-epoch-cuts.md | 9 ++--- 2 files changed, 31 insertions(+), 25 deletions(-) diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index 0420a6fae65..c6cd7c04fa0 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -22,7 +22,10 @@ PTS. Two importers agree only if they started at the same packet. Decided 2026-10-13 by the maintainer, over one importer per epoch and over a contract narrowed to what one input proves: a shared epoch carries no source-to-wall mapping or era, and only one observation of the source can -supply them, so the epoch fixes them once and nothing re-derives them. +supply them, so the epoch fixes them once and nothing re-derives them. The +maintainer accepted the follow-on rules below the same day: eras from the +incumbent's live timeline, joiners adopting history-derived catalog fields, +and the subscribe grant a joiner needs. - The epoch's first importer places the clock at its first frame, as today. A later importer reads the epoch's live catalog before it announces and @@ -43,20 +46,20 @@ supply them, so the epoch fixes them once and nothing re-derives them. nearest the incumbent's latest published timestamp. One offset taken from the first frame would misplace a stream whose first value sits across a wrap from it. Delivery skew alone is far below half a period, so it - changes nothing. Add the periods in source ticks before - converting to microseconds, since a period is not a whole microsecond - count. -- The latest timestamp is still wrong when flagged breaks totalling half a - period fall within the delivery skew between the two observations: a - joiner one second behind, while the incumbent takes two +10 h breaks, - lands a wrap ahead. Settle whether the joiner confirms its era on an - overlapping observation before announcing (the incumbent's object at its - own first group's time, which would also catch a different stream), or - the docs state that bound. -- Same stream is the operator's precondition, unchecked like the rest of the - same-epoch contract. A nearest whole period always exists, so the era - choice cannot detect a different stream. No check is planned for identity - alone; the overlapping observation above would catch one if chosen. + changes nothing. Add the periods in source ticks before converting to + microseconds, since a period is not a whole microsecond count. +- The joiner verifies its era against one of the incumbent's objects + before announcing: the incumbent's object at the joiner's first group's + time must match it in timestamp and payload. A mismatch exits with an + error, and so does an incumbent that leaves before the check. Decided + 2026-10-13 by the maintainer, over documenting a bound: the latest + timestamp alone is wrong when flagged breaks totalling half a period fall + within the delivery skew (a joiner one second behind, while the incumbent + takes two +10 h breaks, lands a wrap ahead). +- The era choice alone cannot detect a different stream, since a nearest + whole period always exists; the verification above refuses one at join. + Past that, same stream stays the operator's precondition, unchecked like + the rest of the same-epoch contract (maintainer, 2026-10-13). - Every catalog field an importer derives from its history rather than the content at hand is the epoch's too. The joiner adopts the live value as of its first group, and later changes depend only on content on the epoch @@ -79,10 +82,11 @@ supply them, so the epoch fixes them once and nothing re-derives them. inputs and the first source timestamp after the step can fix its anchor. - Nothing outlives the live route, so a total outage loses the anchor: the next importer under the same explicit epoch is a first importer, and its - era and clock can differ from the old run's. Persisting the anchor is out - of scope, so an explicit epoch names one continuous run, and a supervisor + era and clock can differ from the old run's. Decided 2026-10-13 by the + maintainer: an explicit epoch names one continuous run, and a supervisor restarting after every importer stopped mints a successor epoch, as for a - rewind. Settle how that is surfaced with the successor-epoch item above. + rewind; the `--epoch` docs say so. Rejected: persisting the anchor beyond + the live route, and a flag marking the first importer. - The mapping drifts from UTC at the source's clock error for the epoch's life, as one importer's does today. Nothing re-measures it. @@ -91,9 +95,10 @@ Tests, on mocked time where possible: for each importer that takes identical catalogs and timestamps; the same with the second started after a 33-bit wrap the first crossed, at a wrap with its streams' first values on opposite sides of it, after flagged forward breaks that together exceed half -a period, with observations straddling such breaks (per the rule settled -above), and with different delivery skew into each; a cold co-start leaves -one publisher. +a period, with observations straddling such breaks (the verification +refuses a wrong era), and with different delivery skew into each; a +different stream under the same epoch fails verification; a cold co-start +leaves one publisher. Public API: no new items, but `--epoch` on an importer now reads the live broadcast before announcing, so its token needs subscribe rights on its own diff --git a/quest/m1/ts-passthrough-epoch-cuts.md b/quest/m1/ts-passthrough-epoch-cuts.md index 1a7bf155736..cbbbab518ae 100644 --- a/quest/m1/ts-passthrough-epoch-cuts.md +++ b/quest/m1/ts-passthrough-epoch-cuts.md @@ -31,10 +31,11 @@ when the importer started. other start publishes `random_access` false. - Size cuts count from the group's start, so they follow once group starts do. -- A cut is a position in the content, so it follows media time alone. A - wall-clock cut path would put a stalled instance's groups apart from its - partner's, so this lane is an exception to timers advancing on - max(wall, pts). +- A cut is a position in the content, so under a shared epoch it follows + media time alone. Decided 2026-10-13 by the maintainer as an explicit + exception to timers advancing on max(wall, pts), because a wall-clock cut + would split a stalled instance from its partner. Rejected: keeping a wall + path. - A joiner adopts the live M2ts section as of its first group, and its later changes depend only on content on the epoch timeline: `random_access` clears on a cut both instances make, and the rate windows sit on the From 7cd7893708149fe19b1f68824353d7c6e17878db Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 10:57:04 -0700 Subject: [PATCH 6/8] quest(m1): per-track era checks, first-importer bases, generated tracks, clustered co-start A first importer resolves each stream's first timestamp against its own first value. The joiner verifies one object per timestamp base. Generated catalog and timeline tracks take their numbering from the epoch, not the instance. Cold co-start detection must work across clustered relays, whose route order hides a peer's route. Co-Authored-By: Claude Opus 5.5 --- quest/m1/epoch-anchor.md | 32 ++++++++++++++++++++++++-------- quest/m1/hop-aligned-import.md | 11 ++++++++++- 2 files changed, 34 insertions(+), 9 deletions(-) diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index c6cd7c04fa0..770abea0ec3 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -48,9 +48,16 @@ and the subscribe grant a joiner needs. wrap from it. Delivery skew alone is far below half a period, so it changes nothing. Add the periods in source ticks before converting to microseconds, since a period is not a whole microsecond count. -- The joiner verifies its era against one of the incumbent's objects - before announcing: the incumbent's object at the joiner's first group's - time must match it in timestamp and payload. A mismatch exits with an +- The first importer has no incumbent, so it applies the same rule with its + own first value as the reference. Today each stream's `PtsUnwrap` starts + at era 0, so a first importer started at a wrap, with one stream's first + PTS still just before it, places that stream 26.5 h from the others, and + every joiner would adopt the broken coordinate. +- The joiner verifies its era against the incumbent's objects before + announcing, one per independently mapped timestamp base (each track): the + incumbent's object at the joiner's first group's time must match it in + timestamp and payload. One object would let a matching video track + approve an audio track a period off. A mismatch exits with an error, and so does an incumbent that leaves before the check. Decided 2026-10-13 by the maintainer, over documenting a bound: the latest timestamp alone is wrong when flagged breaks totalling half a period fall @@ -67,8 +74,16 @@ and the subscribe grant a joiner needs. recorded `mux_rate`, owned by [raw TS cuts](/quest/m1/ts-passthrough-epoch-cuts.md). - Two importers that both start before either is live would place two - clocks under one epoch. The later must detect it and exit, so that its - supervisor's restart adopts; settle how it detects it. + clocks under one epoch. One must exit, so that its supervisor's restart + adopts; settle how. Detection by each importer through its own relay + fails in a cluster: a replica on each relay reads before its peer's route + arrives, and `route_order` keeps preferring each relay's own route + afterwards, so neither sees the other. A co-start is also routine, since + both importers of a pair restart together under a successor epoch. + Candidates: an external leader or shared anchor; detection through state + both can see, such as the announcement of the peer's route; or co-start + as an unsupported precondition, which the successor-epoch flow would have + to avoid. - A flagged forward break must be carried as the same function of the source's clock values on every importer, so media time stays congruent to the raw clock modulo the period and the era choice above holds after it. @@ -96,9 +111,10 @@ identical catalogs and timestamps; the same with the second started after a 33-bit wrap the first crossed, at a wrap with its streams' first values on opposite sides of it, after flagged forward breaks that together exceed half a period, with observations straddling such breaks (the verification -refuses a wrong era), and with different delivery skew into each; a -different stream under the same epoch fails verification; a cold co-start -leaves one publisher. +refuses a wrong era, on every track), and with different delivery skew into +each; a first importer started at a wrap with its streams on opposite sides +of it; a different stream under the same epoch fails verification; a cold +co-start, including one across two clustered relays, leaves one publisher. Public API: no new items, but `--epoch` on an importer now reads the live broadcast before announcing, so its token needs subscribe rights on its own diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index c498e318713..9d8b1428109 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -44,6 +44,13 @@ wrap. (2026-10-05 audit; narrowing the contract to exclude the catalog clock was rejected). So is every other catalog field an importer derives from its history, which a joiner adopts from the live catalog. +- The tracks an importer generates are tracks too: the catalog tracks + (hang, compressed, MSF) and `*.timeline`. Today catalog groups are + numbered locally (`Outputs::emit`), the MSF frame is stamped with + `Timestamp::now()`, and timeline records start at zero, so a joiner would + reuse earlier group or record numbers for newer bytes. Their numbering + and timestamps come from the live track at join or from the epoch + timeline, never the instance. - An importer announces only once it knows its tracks, so it never refuses a track the incumbent serves. - Docs (the `--epoch` paragraph in `doc/bin/cli.md`, the `--epoch` doc @@ -52,7 +59,9 @@ wrap. Tests: per importer, two instances started at different offsets into one input, including the later one starting after a timestamp wrap the earlier -crossed, publish identical catalogs, group sequences, and timestamps. End to +crossed, publish identical catalogs, group sequences, and timestamps, on +the generated catalog and timeline tracks too, joining after earlier +catalog updates and timeline records. End to end: the issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, two `export ts`) survives the standby joining, before and after a wrap, and the incumbent stopping; repeat it with `--passthrough`. From 635e1282439f22ead793aee0cd5b684901a53afa Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 11:11:33 -0700 Subject: [PATCH 7/8] quest(m1): nonnegative first-importer era, sparse bases, equal group starts The first importer resolves its initial bases together and lifts the coordinate by a period rather than going negative. Sparse verbatim bases resolve against the program's verified PCR base instead of holding the announcement. Equal-timestamp group starts need a source tie-breaker or a refusal under a shared epoch. Co-Authored-By: Claude Opus 5.5 --- quest/m1/epoch-anchor.md | 12 +++++++++--- quest/m1/hop-aligned-import.md | 6 +++++- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index 770abea0ec3..a786f959cb5 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -52,13 +52,19 @@ and the subscribe grant a joiner needs. own first value as the reference. Today each stream's `PtsUnwrap` starts at era 0, so a first importer started at a wrap, with one stream's first PTS still just before it, places that stream 26.5 h from the others, and - every joiner would adopt the broken coordinate. + every joiner would adopt the broken coordinate. Timestamps are unsigned, + so the pre-wrap stream's nearest candidate can be negative: resolve the + initial bases together before placing the clock, and lift the whole + coordinate by a period when one would land below zero. - The joiner verifies its era against the incumbent's objects before announcing, one per independently mapped timestamp base (each track): the incumbent's object at the joiner's first group's time must match it in timestamp and payload. One object would let a matching video track - approve an audio track a period off. A mismatch exits with an - error, and so does an incumbent that leaves before the check. Decided + approve an audio track a period off. A sparse base with no cadence + (verbatim subtitles or teletext, each with its own `PtsUnwrap`) must not + hold the announcement until its next cue: resolve it against its + program's verified PCR base instead. A mismatch exits with an error, and + so does an incumbent that leaves before the check. Decided 2026-10-13 by the maintainer, over documenting a bound: the latest timestamp alone is wrong when flagged breaks totalling half a period fall within the delivery skew (a joiner one second behind, while the incumbent diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 9d8b1428109..eaefb6ca030 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -37,7 +37,11 @@ wrap. - A group's sequence derives from its first frame's time on the epoch timeline (the keyframe in the demultiplexed lane, the group's first object in passthrough), not a per-process counter, so two importers number the - same group alike wherever each started. + same group alike wherever each started. A source may start consecutive + groups at one timestamp (`container::Producer` accepts a keyframe at the + previous group's start), which time alone cannot number without a + history-derived counter: settle a tie-breaker the source carries, or + refuse equal starts under a shared epoch, and cover it in the pair test. - Frame timestamps derive from the input and the epoch's coordinate alone, never from `clock.now()` (2026-10-06 audit), and importers refuse a rewind. - The catalog's root `clock` is the epoch's, never an importer's arrival From 44c5c3c01fdceebdffcae6382d03e878fa10f199 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 11:22:21 -0700 Subject: [PATCH 8/8] quest(m1): bases added after a wrap resolve against the program timeline Co-Authored-By: Claude Opus 5.5 --- quest/m1/epoch-anchor.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index a786f959cb5..8aab48851a1 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -55,7 +55,10 @@ and the subscribe grant a joiner needs. every joiner would adopt the broken coordinate. Timestamps are unsigned, so the pre-wrap stream's nearest candidate can be negative: resolve the initial bases together before placing the clock, and lift the whole - coordinate by a period when one would land below zero. + coordinate by a period when one would land below zero. A base created + later, such as a PID a PMT revision adds after a wrap (today + `ensure_stream` starts it at `PtsUnwrap::default()`), resolves against + the program's current verified timeline, on every importer alike. - The joiner verifies its era against the incumbent's objects before announcing, one per independently mapped timestamp base (each track): the incumbent's object at the joiner's first group's time must match it in @@ -119,7 +122,8 @@ opposite sides of it, after flagged forward breaks that together exceed half a period, with observations straddling such breaks (the verification refuses a wrong era, on every track), and with different delivery skew into each; a first importer started at a wrap with its streams on opposite sides -of it; a different stream under the same epoch fails verification; a cold +of it; a stream a PMT revision adds after a wrap; a different stream under +the same epoch fails verification; a cold co-start, including one across two clustered relays, leaves one publisher. Public API: no new items, but `--epoch` on an importer now reads the live