From b6c67a249b7827afdd7f74f87d7805e430c1cb10 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 12:02:39 -0700 Subject: [PATCH 1/3] quest(m1): split announce-once-tracks-are-known out of hop-aligned-import Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/hop-aligned-import.md | 7 +++--- quest/m1/import-announce-tracks.md | 37 ++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 3 deletions(-) create mode 100644 quest/m1/import-announce-tracks.md diff --git a/quest/m1/README.md b/quest/m1/README.md index fb2da76e2ca..4f25efc6fdf 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -86,6 +86,7 @@ above them, since the epoch anchor requires it. - [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, archive, and transcode refuse `--epoch`, since two instances never publish identical tracks - [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break - [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and a joiner adopts the live M2ts section, so instances started apart publish identical groups and catalogs +- [Announce once tracks are known](/quest/m1/import-announce-tracks.md) - a joining importer announces only after it can serve its tracks, so it never aborts the incumbent's subscribers (#4352) - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest - [Data consumer timestamps](/quest/m1/data-consumer-timestamps.md) - json and binary consumers return each value's timestamp, in Rust and every binding; snapshots add `latest()` beside an in-order `next()` diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index ef6dfadc297..1b62d2d36f5 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -55,8 +55,9 @@ wrap. reuse earlier group or record numbers for newer bytes. Their numbering and timestamps come from the live track at join or from the epoch timeline, never the instance. -- An importer announces only once it knows its tracks, so it never refuses a - track the incumbent serves. +- An importer announces only once it knows its tracks: split out to + [announce once tracks are known](/quest/m1/import-announce-tracks.md) + (2026-10-13), which closes #4352. - Docs (the `--epoch` paragraph in `doc/bin/cli.md`, the `--epoch` doc comment) state the contract: one encoded stream fed to each publisher, and a later importer that can reach the live broadcast before it announces. @@ -72,11 +73,11 @@ the incumbent stopping; repeat it with `--passthrough`. ## Closes -- [#4352](https://github.com/moq-dev/moq/issues/4352) - close this issue when the quest finishes - [#4354](https://github.com/moq-dev/moq/issues/4354) - close this issue when the quest finishes ## Required +- [Announce once tracks are known](/quest/m1/import-announce-tracks.md) - a joining importer never answers `not found` for the incumbent's tracks - [Epoch refusals](/quest/m1/epoch-refusals.md) - publishers that cannot meet the contract refuse a shared epoch - [Epoch anchor](/quest/m1/epoch-anchor.md) - one anchor and timestamp era per epoch, adopted by every later importer - [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough groups start where every instance cuts diff --git a/quest/m1/import-announce-tracks.md b/quest/m1/import-announce-tracks.md new file mode 100644 index 00000000000..287a9f6cb54 --- /dev/null +++ b/quest/m1/import-announce-tracks.md @@ -0,0 +1,37 @@ +# [S] An importer announces once it knows its tracks + +## Goal + +A joining importer, such as a TS standby that has not parsed its PMT yet, +announces its broadcast only once it can serve its tracks, so a relay that +queries it the moment it connects never gets `not found` and never aborts the +subscribers already watching the incumbent (#4352). + +## Plan + +Split out of [same-epoch importers](/quest/m1/hop-aligned-import.md) +(2026-10-13): it needs no epoch anchor and lands on its own. + +In #4352 the standby connected, the relay queried every track within a +millisecond, the standby answered `not found` for each before parsing its PMT, +and both `export ts` subscribers exited 15 ms later. Delay the announcement in +the importers (`rs/moq-mux` import paths behind `moq import`) until the +catalog lists the tracks the importer will publish: for TS, the first PMT; for +the other formats, whatever first declares their tracks. An importer that +never learns its tracks never announces, and fails loud at its own deadline +rather than announcing an empty broadcast. + +Test: the #4352 setup, one relay, an incumbent `import ts` with two `export +ts` subscribers, then a second `import ts` on the same path connecting before +its PMT arrives; both subscribers keep running. Repeat for one other importer +format. + +Public API: none expected. Wire: none. + +## Closes + +- [#4352](https://github.com/moq-dev/moq/issues/4352) - close this issue when the quest finishes + +## Related + +- [Same-epoch importers publish identical tracks](/quest/m1/hop-aligned-import.md) - the redundant-pair contract this is one piece of From 8934037ff37fa35c90103edca1a25120d6793b62 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 12:11:44 -0700 Subject: [PATCH 2/3] quest(m1): same-epoch importers derive one coordinate, active-active Every importer sharing an explicit epoch derives its catalog clock, eras, and history-derived catalog fields from the epoch and its source alone, so all stay live and none adopts or exits. Supersedes #5412's first-importer adoption, join-time verification, subscribe grant, and successor epoch after a total outage. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 10 +- quest/m1/epoch-anchor.md | 214 ++++++++++++-------------- quest/m1/hop-aligned-import.md | 50 +++--- quest/m1/import-announce-tracks.md | 11 +- quest/m1/ts-passthrough-epoch-cuts.md | 26 ++-- quest/m1/ts-reliability/README.md | 2 +- 6 files changed, 152 insertions(+), 161 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 4f25efc6fdf..356353f319d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -43,8 +43,8 @@ the feature that needs them. Added 2026-10-13 by the redundant TS alignment plan: epoch refusals, the epoch anchor, and raw TS cuts on the epoch timeline, ranked just above -same-epoch importers, which requires them. Catalog clock adoption moved up -above them, since the epoch anchor requires it. +same-epoch importers, which requires them. Catalog clock adoption ranks +above them. ## Required @@ -84,9 +84,9 @@ above them, since the epoch anchor requires it. - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` - [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused - [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, archive, and transcode refuse `--epoch`, since two instances never publish identical tracks -- [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break -- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and a joiner adopts the live M2ts section, so instances started apart publish identical groups and catalogs -- [Announce once tracks are known](/quest/m1/import-announce-tracks.md) - a joining importer announces only after it can serve its tracks, so it never aborts the incumbent's subscribers (#4352) +- [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers each derive the catalog clock and timestamp era from the epoch and stay live, so they agree across a 33-bit wrap, a forward clock break, and a restart +- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and the M2ts section derives from the epoch and content, so instances started apart publish identical groups and catalogs +- [Announce once tracks are known](/quest/m1/import-announce-tracks.md) - an importer announces only after it can serve its tracks, so starting beside a live partner never aborts its subscribers (#4352) - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest - [Data consumer timestamps](/quest/m1/data-consumer-timestamps.md) - json and binary consumers return each value's timestamp, in Rust and every binding; snapshots add `latest()` beside an in-order `next()` diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index 8aab48851a1..b549f52fcd4 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -1,16 +1,14 @@ -# [M] Same-epoch importers share one anchor and era +# [M] Same-epoch importers derive one coordinate from the epoch ## Goal -Every importer publishing one path under one explicit `--epoch` places its -timestamps on one coordinate: the catalog root clock that the epoch's first -importer fixed, and the timestamp era that clock implies. A later importer -adopts that clock instead of placing its own, and shifts each timestamp -base by the whole number of the format's timestamp periods that lands it on -the incumbent's live timeline. Two -importers of one stream then publish identical timestamps and catalogs -whenever each started, including on opposite sides of a 33-bit TS wrap or of -a flagged forward clock break. +Every importer publishing one path under one explicit `--epoch`, fed one +stream, derives its catalog root clock and each timestamp's era from the +epoch and the source alone. They all stay live (active-active) and publish +identical timestamps and catalogs whenever each started, including on +opposite sides of a 33-bit TS wrap or a flagged forward break, and after a +restart under the same epoch. No importer reads another's broadcast, waits +on one, or exits because one exists. Today each importer places its clock at its own first frame's arrival, or shifts onto a clock already in use by `clock.now()` minus its first @@ -19,124 +17,106 @@ PTS. Two importers agree only if they started at the same packet. ## Plan -Decided 2026-10-13 by the maintainer, over one importer per epoch and over a -contract narrowed to what one input proves: a shared epoch carries no -source-to-wall mapping or era, and only one observation of the source can -supply them, so the epoch fixes them once and nothing re-derives them. The -maintainer accepted the follow-on rules below the same day: eras from the -incumbent's live timeline, joiners adopting history-derived catalog fields, -and the subscribe grant a joiner needs. +Decided 2026-10-13 by the maintainer: importers sharing an explicit epoch +derive the same coordinate without coordination, and none is first. This +supersedes the same day's #5412 plan, where the epoch's first importer fixed +the clock and era and every later one read the live catalog and adopted +them: adoption needs a first importer, and a co-start (routine, since both +importers of a pair restart together) has none that either can name. +Rejected with it: a co-started importer that watches its peer's announcement +and exits, an external leader or anchor flag, and co-start as an unsupported +precondition. -- The epoch's first importer places the clock at its first frame, as today. - A later importer reads the epoch's live catalog before it announces and - adopts its root clock exactly before publishing a frame, through the path - that [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds. - `catalog::Timebase::place` will not do: it takes a `SystemTime`, which - `Clock::wall_clock` floors to milliseconds, so the re-advertised clock - would differ by up to 999 µs. It reads what its relay has now and never - waits on the cluster for completeness; an incumbent it cannot see yet is - the co-start case below. -- The era comes from the incumbent's published timeline, not from the - clock's wall reading. A flagged forward break advances media time without - wall time, so after two accepted +10 h breaks the clock reads about 10 h - while the incumbent publishes 30 h, and a joiner choosing against the - clock lands a full wrap off. Each independent timestamp base (in TS the - PCR, and each stream's PTS and DTS, which unwrap separately) takes the - whole number of periods (2^33 at 90 kHz for TS) that lands its first value - nearest the incumbent's latest published timestamp. One offset taken from - the first frame would misplace a stream whose first value sits across a - wrap from it. Delivery skew alone is far below half a period, so it - changes nothing. Add the periods in source ticks before converting to - microseconds, since a period is not a whole microsecond count. -- The first importer has no incumbent, so it applies the same rule with its - own first value as the reference. Today each stream's `PtsUnwrap` starts - at era 0, so a first importer started at a wrap, with one stream's first - PTS still just before it, places that stream 26.5 h from the others, and - every joiner would adopt the broken coordinate. Timestamps are unsigned, - so the pre-wrap stream's nearest candidate can be negative: resolve the - initial bases together before placing the clock, and lift the whole - coordinate by a period when one would land below zero. A base created - later, such as a PID a PMT revision adds after a wrap (today - `ensure_stream` starts it at `PtsUnwrap::default()`), resolves against - the program's current verified timeline, on every importer alike. -- The joiner verifies its era against the incumbent's objects before - announcing, one per independently mapped timestamp base (each track): the - incumbent's object at the joiner's first group's time must match it in - timestamp and payload. One object would let a matching video track - approve an audio track a period off. A sparse base with no cadence - (verbatim subtitles or teletext, each with its own `PtsUnwrap`) must not - hold the announcement until its next cue: resolve it against its - program's verified PCR base instead. A mismatch exits with an error, and - so does an incumbent that leaves before the check. Decided - 2026-10-13 by the maintainer, over documenting a bound: the latest - timestamp alone is wrong when flagged breaks totalling half a period fall - within the delivery skew (a joiner one second behind, while the incumbent - takes two +10 h breaks, lands a wrap ahead). -- The era choice alone cannot detect a different stream, since a nearest - whole period always exists; the verification above refuses one at join. - Past that, same stream stays the operator's precondition, unchecked like - the rest of the same-epoch contract (maintainer, 2026-10-13). +- The epoch is a UUIDv7, so it carries a millisecond wall time that every + importer shares exactly. Each importer reads only its own input, the epoch, + and its own wall clock, and never the relay or the cluster. +- The era comes from the epoch (maintainer, superseding #5412's era from the + incumbent's published timeline and the join-time verification against its + objects): for each timestamp base (in TS the PCR, and each stream's PTS and + DTS), the whole number of periods (2^33 at 90 kHz for TS) that lands the + value nearest its arrival on the epoch's clock, as the importer's wall clock + reads it. The source's offset from the wall clock is the same for every + importer, so they agree unless their delivery and host-clock skew straddles + a half-period boundary of that offset: about skew over the period, near + 1e-7 for 10 ms against TS's 26.5 h. Document that bound. Add the periods in + source ticks before converting to microseconds, since a period is not a + whole microsecond count. +- Recommended: derive every value's era this way, not only each base's first, + so no importer state enters the coordinate. A restart, a base a PMT + revision adds after a wrap, and a sparse base (verbatim subtitles or + teletext) then need no special case, and `PtsUnwrap`'s history goes away. + The cost is that media time stays within half a period of the epoch + clock's reading. A flagged forward break that would carry it further reads + as a step back, which stays fatal on every importer alike; so does source + drift that walks the offset to the boundary over the epoch's life (30 ppm + takes years from a random offset), and a file read faster than real time + for half a period. The raw importer's clamp after a small flagged jump + (`(last + step).max(self.written)` in `passthrough.rs`) depends on history; + remove it. +- The catalog's wall mapping derives from the epoch too (maintainer): media + time maps to wall through the epoch's wall time plus the deterministic era, + so every importer writes the same root `clock`. Rejected: no wall mapping + under a shared epoch, and quantizing each importer's measured offset, whose + rounding boundary splits importers far more often than a period's. The + accuracy is bounded by how well the epoch's wall time matches the source: + the mapping is off by the source clock's reading at the epoch's wall time, + modulo the period, which for an arbitrary TS program clock is anywhere + within half a period (13.25 h). Document that with `--epoch`. The exact rule + is open; recommended: PTS zero sits one period before the epoch's wall time + (`E - period`, deterministically rounded to a whole microsecond), and the + era is chosen against that clock's reading. Then timestamps never land + below zero for an importer whose clock reads after the epoch's wall time + minus half a period, and a source whose clock reads zero at the epoch's + wall time (a stream started when the epoch was minted) maps exactly and + sits half a period from the boundary. A format whose timestamps never wrap + has no era: PTS zero is the epoch's wall time and its timestamps stay + verbatim. Install it through `Timebase::place`, which keeps whole + microseconds; nothing reads a catalog back, so the millisecond flooring in + `Clock::wall_clock` never enters it. - Every catalog field an importer derives from its history rather than the - content at hand is the epoch's too. The joiner adopts the live value as of - its first group, and later changes depend only on content on the epoch - timeline. In TS passthrough that is the M2ts section's `random_access` and - recorded `mux_rate`, owned by - [raw TS cuts](/quest/m1/ts-passthrough-epoch-cuts.md). -- Two importers that both start before either is live would place two - clocks under one epoch. One must exit, so that its supervisor's restart - adopts; settle how. Detection by each importer through its own relay - fails in a cluster: a replica on each relay reads before its peer's route - arrives, and `route_order` keeps preferring each relay's own route - afterwards, so neither sees the other. A co-start is also routine, since - both importers of a pair restart together under a successor epoch. - Candidates: an external leader or shared anchor; detection through state - both can see, such as the announcement of the peer's route; or co-start - as an unsupported precondition, which the successor-epoch flow would have - to avoid. -- A flagged forward break must be carried as the same function of the - source's clock values on every importer, so media time stays congruent to - the raw clock modulo the period and the era choice above holds after it. - The raw importer's clamp after a small flagged jump - (`(last + step).max(self.written)` in `passthrough.rs`) breaks that and - depends on history; remove or bound it. -- A rewind stays fatal (#4543), so both importers of a pair exit together - and restart under a successor epoch. Settle how the successor gets its - anchor without the co-start race above. The successor epoch is proposed to - derive from the old epoch and the last frame's catalog time; the same - inputs and the first source timestamp after the step can fix its anchor. -- Nothing outlives the live route, so a total outage loses the anchor: the - next importer under the same explicit epoch is a first importer, and its - era and clock can differ from the old run's. Decided 2026-10-13 by the - maintainer: an explicit epoch names one continuous run, and a supervisor - restarting after every importer stopped mints a successor epoch, as for a - rewind; the `--epoch` docs say so. Rejected: persisting the anchor beyond - the live route, and a flag marking the first importer. + content at hand derives from the epoch and the content instead (maintainer, + superseding #5412's adoption from the live catalog). In TS passthrough that + is the M2ts section's `random_access` and recorded `mux_rate`, owned by + [raw TS cuts](/quest/m1/ts-passthrough-epoch-cuts.md); audit the other + importers for more. +- A restart under the same epoch re-derives the same coordinate, including + after a total outage, so the epoch keeps naming one timeline and resuming + across the outage is a gap, not a stitch (maintainer, reversing #5412's + "an explicit epoch names one continuous run; mint a successor after a total + outage"). Nothing persists beyond the live route. +- A rewind stays fatal (#4543), so both importers of a pair exit and restart + under a successor epoch. An importer never rolls to a successor in-process + (maintainer: "Never"). Open: how a supervisor obtains the successor so both + importers of a pair land on the same one without coordination. Proposed: + derive it from the old epoch and the last frame's catalog time. Its wall + time then fixes the successor's coordinate like any explicit epoch's. +- Same stream stays the operator's precondition, unchecked. Known gap, not + planned now (maintainer): nothing refuses two routes with the same epoch + but different content, and failover resumes mid-group across them + (`rs/moq-net/src/model/resume.rs`), so a misfed pair splices two streams. - The mapping drifts from UTC at the source's clock error for the epoch's - life, as one importer's does today. Nothing re-measures it. + life. Nothing re-measures it. Tests, on mocked time where possible: for each importer that takes `--epoch`, two instances started at different source offsets publish identical catalogs and timestamps; the same with the second started after a 33-bit wrap the first crossed, at a wrap with its streams' first values on -opposite sides of it, after flagged forward breaks that together exceed half -a period, with observations straddling such breaks (the verification -refuses a wrong era, on every track), and with different delivery skew into -each; a first importer started at a wrap with its streams on opposite sides -of it; a stream a PMT revision adds after a wrap; a different stream under -the same epoch fails verification; a cold -co-start, including one across two clustered relays, leaves one publisher. +opposite sides of it, after flagged forward breaks, and with different +delivery skew into each; a stream a PMT revision adds after a wrap; forward +breaks carrying media time past half a period from the epoch clock fail on +both alike; a co-start across two clustered relays leaves both live and +byte-identical; a restart under the same epoch, alone and after both +stopped, rejoins identically; the derived root `clock` matches the +recommended rule, including a source that reads zero at the epoch's wall +time. -Public API: no new items, but `--epoch` on an importer now reads the live -broadcast before announcing, so its token needs subscribe rights on its own -path. A publish-only token that works with `--epoch` today fails there: a -breaking auth requirement, documented with `--epoch` in `doc/bin/cli.md` and -`doc/bin/relay/auth.md`. Wire: none. - -## Required - -- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - the exact adoption path a joining importer uses +Public API: no new items. Under a shared epoch the root `clock` and the +timestamps move from arrival to the epoch's derivation; the `--epoch` docs +state the accuracy bound. `--epoch` needs no subscribe right (#5412's +breaking grant for publish-only tokens is dropped). Wire: none. ## Related - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair on this coordinate +- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - exact clock adoption, which a derived clock no longer needs - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - same-epoch routes resume each other's subscriptions diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 1b62d2d36f5..b29d33979e7 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -24,15 +24,15 @@ connection (#4962), so two of them never share one. Publishers whose timestamps or bytes come from the instance refuse a shared epoch ([Epoch refusals](/quest/m1/epoch-refusals.md)). -Decided 2026-10-13 by the maintainer: an epoch has one -coordinate, which its first importer fixes and every later importer adopts -([Epoch anchor](/quest/m1/epoch-anchor.md)). This quest numbers groups on it -and proves the pair end to end. A shared epoch or a raw PTS or PCR value -alone does not supply that coordinate, and arrival time never does. -Considered instead: one importer per epoch with redundancy only downstream, -which stays the supported 1+1 until this lands, and a contract narrowed to -what one input proves, which fails at the first failover across a timestamp -wrap. +Decided 2026-10-13 by the maintainer: every importer of an epoch stays live +and derives one coordinate from the epoch and its source alone +([Epoch anchor](/quest/m1/epoch-anchor.md)), superseding the same day's plan +where later importers adopted the first one's. This quest numbers groups on +that coordinate and proves the pair end to end. A raw PTS or PCR value alone +does not supply it, and arrival time chooses only the era. Considered +instead: one importer per epoch with redundancy only downstream, which stays +the supported 1+1 until this lands, and a contract narrowed to what one input +proves, which fails at the first failover across a timestamp wrap. - A group's sequence derives from its first frame's time on the epoch timeline (the keyframe in the demultiplexed lane, the group's first object @@ -46,30 +46,32 @@ wrap. never from `clock.now()` (2026-10-06 audit), and importers refuse a rewind. - The catalog's root `clock` is the epoch's, never an importer's arrival (2026-10-05 audit; narrowing the contract to exclude the catalog clock was - rejected). So is every other catalog field an importer derives from its - history, which a joiner adopts from the live catalog. + rejected). Every other catalog field an importer derives from its history + derives from the epoch and the content instead, never from the live + catalog. - The tracks an importer generates are tracks too: the catalog tracks (hang, compressed, MSF) and `*.timeline`. Today catalog groups are numbered locally (`Outputs::emit`), the MSF frame is stamped with - `Timestamp::now()`, and timeline records start at zero, so a joiner would - reuse earlier group or record numbers for newer bytes. Their numbering - and timestamps come from the live track at join or from the epoch - timeline, never the instance. + `Timestamp::now()`, and timeline records start at zero, so an importer + started later would reuse earlier group or record numbers for newer bytes. + Their numbering and timestamps come from the epoch timeline and the + content, never the instance or the live track. - An importer announces only once it knows its tracks: split out to [announce once tracks are known](/quest/m1/import-announce-tracks.md) (2026-10-13), which closes #4352. - Docs (the `--epoch` paragraph in `doc/bin/cli.md`, the `--epoch` doc - comment) state the contract: one encoded stream fed to each publisher, and - a later importer that can reach the live broadcast before it announces. + comment) state the contract: one encoded stream fed to each publisher, no + access to the live broadcast needed, and a wall mapping only as accurate as + the epoch's wall time matches the source. Tests: per importer, two instances started at different offsets into one input, including the later one starting after a timestamp wrap the earlier crossed, publish identical catalogs, group sequences, and timestamps, on -the generated catalog and timeline tracks too, joining after earlier -catalog updates and timeline records. End to -end: the issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, -two `export ts`) survives the standby joining, before and after a wrap, and -the incumbent stopping; repeat it with `--passthrough`. +the generated catalog and timeline tracks too, starting after earlier +catalog updates and timeline records. End to end: the issues' 1+1 setup (one +relay, two `import ts` sharing one `--epoch`, two `export ts`) survives the +second importer starting, before and after a wrap, and either importer +stopping and restarting; repeat it with `--passthrough`. ## Closes @@ -77,9 +79,9 @@ the incumbent stopping; repeat it with `--passthrough`. ## Required -- [Announce once tracks are known](/quest/m1/import-announce-tracks.md) - a joining importer never answers `not found` for the incumbent's tracks +- [Announce once tracks are known](/quest/m1/import-announce-tracks.md) - a starting importer never answers `not found` for tracks its partner serves - [Epoch refusals](/quest/m1/epoch-refusals.md) - publishers that cannot meet the contract refuse a shared epoch -- [Epoch anchor](/quest/m1/epoch-anchor.md) - one anchor and timestamp era per epoch, adopted by every later importer +- [Epoch anchor](/quest/m1/epoch-anchor.md) - every importer derives one catalog clock and timestamp era from the epoch - [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough groups start where every instance cuts ## Related diff --git a/quest/m1/import-announce-tracks.md b/quest/m1/import-announce-tracks.md index 287a9f6cb54..7ee342d1411 100644 --- a/quest/m1/import-announce-tracks.md +++ b/quest/m1/import-announce-tracks.md @@ -2,10 +2,11 @@ ## Goal -A joining importer, such as a TS standby that has not parsed its PMT yet, -announces its broadcast only once it can serve its tracks, so a relay that -queries it the moment it connects never gets `not found` and never aborts the -subscribers already watching the incumbent (#4352). +An importer starting beside a live partner, such as the second `import ts` +of a pair that has not parsed its PMT yet, announces its broadcast only once +it can serve its tracks, so a relay that queries it the moment it connects +never gets `not found` and never aborts the subscribers already watching the +partner (#4352). ## Plan @@ -21,7 +22,7 @@ the other formats, whatever first declares their tracks. An importer that never learns its tracks never announces, and fails loud at its own deadline rather than announcing an empty broadcast. -Test: the #4352 setup, one relay, an incumbent `import ts` with two `export +Test: the #4352 setup, one relay, a running `import ts` with two `export ts` subscribers, then a second `import ts` on the same path connecting before its PMT arrives; both subscribers keep running. Repeat for one other importer format. diff --git a/quest/m1/ts-passthrough-epoch-cuts.md b/quest/m1/ts-passthrough-epoch-cuts.md index 8c6c7c76f94..a64636c70fd 100644 --- a/quest/m1/ts-passthrough-epoch-cuts.md +++ b/quest/m1/ts-passthrough-epoch-cuts.md @@ -20,12 +20,13 @@ Decided 2026-10-13 by the maintainer: every group start is a function of the content and the [epoch's timeline](/quest/m1/epoch-anchor.md), never of when the importer started. -- Fallback cuts fall on a grid of the epoch timeline, so a joiner's first - group starts where the incumbent cut. A grid on the raw 27 MHz count will +- Fallback cuts fall on a grid of the epoch timeline, so an importer started + later first cuts where its partner did. A grid on the raw 27 MHz count will not do: a 33-bit wrap is not a whole number of seconds, so such a grid shifts by about 0.72 s at each wrap. -- An importer's first group starts only where the incumbent's rules also - cut: a random access point, a flagged forward break, or the grid. +- An importer's first group starts only where a partner running since + before it would also cut: a random access point, a flagged forward break, + or the grid. - Settle the grid rule so that a source marking random access more often than `FALLBACK` is never cut away from its random access points, since any other start publishes `random_access` false. @@ -36,13 +37,20 @@ when the importer started. exception to timers advancing on max(wall, pts), because a wall-clock cut would split a stalled instance from its partner. Rejected: keeping a wall path. -- A joiner adopts the live M2ts section as of its first group, and its later - changes depend only on content on the epoch timeline: `random_access` - clears on a cut both instances make, and the rate windows sit on the - epoch's grid, or the recorded rate stays fixed for the epoch. Settle which. +- The M2ts section derives from the epoch and the content, never from the + live catalog or what came before the importer started (maintainer, + superseding #5412's adoption by a joiner). Today both fields are sticky + history. Settle the rule; recommended: each is a function of a window on + the epoch's grid (the rate measured over it, `random_access` true while its + cuts all fell on random access points), and an importer publishes its + catalog only once it has a whole window, so it never publishes a section + its partner did not. Rejected in #5412 and still out: conservative + constants under a shared epoch, which lose `random_access` and the rate + `--sync` uses. Tests: two instances started at different offsets into one capture publish -identical groups and objects from the later one's first group. Cover a +identical groups and objects from the later one's first group, and a +restarted instance rejoins identically. Cover a source with no random access indicator, a multi-program stream, groups that hit the size cap, the later instance started after a 33-bit wrap the earlier crossed, and one started after the earlier made a fallback cut and diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index 9953eae72ae..8c9f290185e 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -64,7 +64,7 @@ their current quests. ## Related -- [Epoch anchor](/quest/m1/epoch-anchor.md) - shared identity across importers requires one source coordinate per epoch +- [Epoch anchor](/quest/m1/epoch-anchor.md) - shared identity across importers requires one source coordinate per epoch, derived from the epoch - [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - TS identity alone is not ST 2022-7, which also needs matching RTP headers - [TS health counters](/quest/m2/ts-health-stats.md) - ingress and egress evidence need distinct monitoring points - [MSFTS mapping](/quest/m3/msfts-convergence.md) - document the exact equivalence and differences between the carriage modes From 17255783b6eee8a54cdef767dca4dc9d91256457 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 12:26:30 -0700 Subject: [PATCH 3/3] quest(m1): record the era's co-start odds and mid-run drift Co-Authored-By: Claude Opus 5.5 --- quest/m1/epoch-anchor.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/quest/m1/epoch-anchor.md b/quest/m1/epoch-anchor.md index b549f52fcd4..5fcb14d7c8d 100644 --- a/quest/m1/epoch-anchor.md +++ b/quest/m1/epoch-anchor.md @@ -37,8 +37,7 @@ precondition. value nearest its arrival on the epoch's clock, as the importer's wall clock reads it. The source's offset from the wall clock is the same for every importer, so they agree unless their delivery and host-clock skew straddles - a half-period boundary of that offset: about skew over the period, near - 1e-7 for 10 ms against TS's 26.5 h. Document that bound. Add the periods in + a half-period boundary of that offset. Add the periods in source ticks before converting to microseconds, since a period is not a whole microsecond count. - Recommended: derive every value's era this way, not only each base's first, @@ -47,12 +46,18 @@ precondition. teletext) then need no special case, and `PtsUnwrap`'s history goes away. The cost is that media time stays within half a period of the epoch clock's reading. A flagged forward break that would carry it further reads - as a step back, which stays fatal on every importer alike; so does source - drift that walks the offset to the boundary over the epoch's life (30 ppm - takes years from a random offset), and a file read faster than real time - for half a period. The raw importer's clamp after a small flagged jump + as a step back, which stays fatal on every importer alike, as does a file + read faster than real time for half a period. The raw importer's clamp after a small flagged jump (`(last + step).max(self.written)` in `passthrough.rs`) depends on history; remove it. +- Two known failure cases, documented rather than guarded (maintainer, + 2026-10-13, over refusing to start inside a guard band of the halfway + point): importers that start together split onto adjacent eras when their + skew straddles the halfway point, with odds of about skew over the period + (near 1e-7 at 10 ms against TS's 26.5 h); and the gap between the source + clock and wall time drifts by seconds a day, so over weeks it can drift + onto the halfway point mid-run, where even a single importer reads a + rewind and fails. - The catalog's wall mapping derives from the epoch too (maintainer): media time maps to wall through the epoch's wall time plus the deterministic era, so every importer writes the same root `clock`. Rejected: no wall mapping