Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ Keep one owner per shared code area. Order each prerequisite before its
consumer; targeted benchmarks and controlled-time regressions remain part of
the feature that needs them.

Added 2026-10-13 by the redundant TS alignment plan: epoch refusals, the
epoch anchor, and raw TS cuts on the epoch timeline, ranked just above
same-epoch importers, which requires them. Catalog clock adoption moved up
above them, since the epoch anchor requires it.

## Required

- [Epoch correctness](/quest/m0/broadcast-epoch/README.md) - publisher identity, duplicate suppression, and catalog pinning gate the breaking cut
Expand Down Expand Up @@ -77,12 +82,14 @@ the feature that needs them.
- [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away
- [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget
- [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev`
- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared source coordinates and distinguish TS identity from coordinated RTP recovery
- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused
- [Epoch refusals](/quest/m1/epoch-refusals.md) - capture, avc3, `rtc --connect`, archive, and transcode refuse `--epoch`, since two instances never publish identical tracks
- [Epoch anchor](/quest/m1/epoch-anchor.md) - same-epoch importers adopt the first importer's catalog clock and timestamp era, so they agree across a 33-bit wrap and a forward clock break
- [Raw TS cuts on the epoch timeline](/quest/m1/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and a joiner adopts the live M2ts section, so instances started apart publish identical groups and catalogs
- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives
- [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest
- [Data consumer timestamps](/quest/m1/data-consumer-timestamps.md) - json and binary consumers return each value's timestamp, in Rust and every binding; snapshots add `latest()` beside an in-order `next()`
- [JSON window timestamps](/quest/m1/json-window-timestamps.md) - Rust and JS window consumers return each event's frame timestamp like the snapshot and stream consumers
- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - a derivative adopts its input's wall mapping through `modify()`, and a root `clock` change after the clock is fixed is refused
- [Exact catalog wall clock](/quest/m1/catalog-wall-clock-precision.md) - `hang::catalog::Clock::wall_clock` keeps sub-millisecond precision
- [JavaScript answers TRACK_STATUS](/quest/m1/js-track-status.md) - a browser publisher answers IETF TRACK_STATUS like Rust, so a native fetch-only request needs no SUBSCRIBE fallback
- [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface
Expand Down
5 changes: 3 additions & 2 deletions quest/m1/archive/replay-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,9 @@ this by hand-building a catalog.
catalog groups, so the same name, epoch, and group number could carry
different bytes. Revisit only once replayed metadata is byte- and
sequence-identical to live. So `moq import archive` refuses `--epoch`
(`takes_epoch` false for `Archive`), and any `moq-archive` host mints its
own epoch or takes one from its caller, never defaulting to `.info`'s.
([Epoch refusals](/quest/m1/epoch-refusals.md) lands that), and any
`moq-archive` host mints its own epoch or takes one from its caller, never
defaulting to `.info`'s.
- The recorded catalog's `archive` entry describes the source's live
timelines, not the recording's, so replace it with the timelines the reader
replays rather than trusting the recorded ones.
Expand Down
142 changes: 142 additions & 0 deletions quest/m1/epoch-anchor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# [M] Same-epoch importers share one anchor and era

## Goal

Every importer publishing one path under one explicit `--epoch` places its
timestamps on one coordinate: the catalog root clock that the epoch's first
importer fixed, and the timestamp era that clock implies. A later importer
adopts that clock instead of placing its own, and shifts each timestamp
base by the whole number of the format's timestamp periods that lands it on
the incumbent's live timeline. Two
importers of one stream then publish identical timestamps and catalogs
whenever each started, including on opposite sides of a 33-bit TS wrap or of
a flagged forward clock break.

Today each importer places its clock at its own first frame's arrival, or
shifts onto a clock already in use by `clock.now()` minus its first
timestamp (#5082), and each TS importer unwraps from its own first PCR or
PTS. Two importers agree only if they started at the same packet.

## Plan

Decided 2026-10-13 by the maintainer, over one importer per epoch and over a
contract narrowed to what one input proves: a shared epoch carries no
source-to-wall mapping or era, and only one observation of the source can
supply them, so the epoch fixes them once and nothing re-derives them. The
maintainer accepted the follow-on rules below the same day: eras from the
incumbent's live timeline, joiners adopting history-derived catalog fields,
and the subscribe grant a joiner needs.

- The epoch's first importer places the clock at its first frame, as today.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resolve cross-wrap bases for the first importer

When an epoch's first TS importer starts at a wrap, with the first PES post-wrap but another stream's first PTS still pre-wrap due to interleaving or reordering, this keeps the current Import::handle_pes_start behavior: the first raw PTS fixes era 0 and each stream's fresh PtsUnwrap also starts at era 0. The other stream is therefore placed near 2^33 instead of just before zero, leaving A/V about 26.5 hours apart, and later importers merely adopt that broken coordinate. Unlike the prior standby finding, this case has no incumbent against which lines 39-50 can resolve the bases; buffer or normalize all initial bases, or discard pre-anchor units, before fixing the first clock, and test the first-importer case. (Written by GPT-5.6 Sol)

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 7cd7893. The first importer now resolves each base against its own first value, using the same nearest-period rule, instead of starting every PtsUnwrap at era 0. The tests now cover a first importer started at a wrap with its streams on opposite sides of it.

(Written by Claude Opus 5.5)

A later importer reads the epoch's live catalog before it announces and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the anchor when every replica is offline

When all replicas stop and their supervisors restart them with the same explicit --epoch, the first restarted importer has no live catalog to read, so this plan treats it as the first importer and creates a new clock and era under the old identity. Same-epoch subscribers can then resume into a different catalog, and after a source wrap the media group numbers can restart in the wrong era as well. Persist the epoch anchor somewhere that outlives the live route, or require a newly minted epoch after a total outage rather than only documenting that a later importer needs a live incumbent. (Written by GPT-5.6 Sol)

AGENTS.md reference: AGENTS.md:L64-L64

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that this case exists, and recorded in 6ca0e8a. Persisting the anchor beyond the live route is out of scope, and relays keep no such state. So an explicit epoch names one continuous run, and a supervisor that restarts after every importer has stopped mints a successor epoch, as it does for a rewind. How that is surfaced is settled together with the successor-epoch item, and is flagged for the maintainer.

(Written by Claude Opus 5.5)

adopts its root clock exactly before publishing a frame, through the path
that [catalog clock adoption](/quest/m1/catalog-clock-adopt.md) adds.
`catalog::Timebase::place` will not do: it takes a `SystemTime`, which
`Clock::wall_clock` floors to milliseconds, so the re-advertised clock
would differ by up to 999 µs. It reads what its relay has now and never
waits on the cluster for completeness; an incumbent it cannot see yet is
the co-start case below.
- The era comes from the incumbent's published timeline, not from the
clock's wall reading. A flagged forward break advances media time without
wall time, so after two accepted +10 h breaks the clock reads about 10 h
while the incumbent publishes 30 h, and a joiner choosing against the
clock lands a full wrap off. Each independent timestamp base (in TS the
PCR, and each stream's PTS and DTS, which unwrap separately) takes the
whole number of periods (2^33 at 90 kHz for TS) that lands its first value
nearest the incumbent's latest published timestamp. One offset taken from
the first frame would misplace a stream whose first value sits across a
wrap from it. Delivery skew alone is far below half a period, so it
changes nothing. Add the periods in source ticks before converting to
microseconds, since a period is not a whole microsecond count.
- The first importer has no incumbent, so it applies the same rule with its
own first value as the reference. Today each stream's `PtsUnwrap` starts
Comment on lines +51 to +52

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resolve timestamp bases introduced after startup

When a supported one-program TS adds an elementary stream after the first importer has already crossed a 33-bit wrap, resolving that stream against the first importer's original own first value selects era 0 instead of the current era. The fresh case beyond the initial-base tests is supported today: Import::handle_pmt applies a good PMT revision and ensure_stream creates the added PID with PtsUnwrap::default(), so its new track can land about 26.5 h from the established tracks. Resolve later-created bases against the current verified program/PCR timeline, or refuse that PMT transition, and cover a post-wrap add. (Written by GPT-5.6 Sol)

AGENTS.md reference: AGENTS.md:L17-L17

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 44c5c3c. Any base created after startup, such as a PID added by a PMT revision after a wrap, resolves against the program's current verified timeline on every importer, and the tests cover a post-wrap add.

(Written by Claude Opus 5.5)

at era 0, so a first importer started at a wrap, with one stream's first
PTS still just before it, places that stream 26.5 h from the others, and
every joiner would adopt the broken coordinate. Timestamps are unsigned,
so the pre-wrap stream's nearest candidate can be negative: resolve the
initial bases together before placing the clock, and lift the whole
coordinate by a period when one would land below zero. A base created
later, such as a PID a PMT revision adds after a wrap (today
`ensure_stream` starts it at `PtsUnwrap::default()`), resolves against
the program's current verified timeline, on every importer alike.
- The joiner verifies its era against the incumbent's objects before
announcing, one per independently mapped timestamp base (each track): the
incumbent's object at the joiner's first group's time must match it in
Comment on lines +62 to +64

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid waiting on sparse bases before announcing

When a supported one-program TS contains a sparse verbatim PES stream such as teletext or DVB subtitles, requiring one matching object per track before announcement can keep the standby hidden indefinitely: VerbatimStream has its own PtsUnwrap (rs/moq-mux/src/container/ts/import.rs:1410-1419) and these streams are explicitly treated as having no cadence (rs/moq-mux/src/container/ts/import.rs:707-710), so no joiner group may arrive during the gap. If the incumbent then leaves, the specified error exits the standby instead of failing over; derive and verify that sparse base from shared program state, or refuse this TS profile up front rather than blocking the whole announcement on its next cue.

AGENTS.md reference: AGENTS.md:L64-L64

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 635e128. A sparse base with no cadence (verbatim subtitles or teletext) resolves against its program's verified PCR base, so it no longer holds the announcement until its next cue.

(Written by Claude Opus 5.5)

timestamp and payload. One object would let a matching video track
approve an audio track a period off. A sparse base with no cadence
(verbatim subtitles or teletext, each with its own `PtsUnwrap`) must not
hold the announcement until its next cue: resolve it against its
program's verified PCR base instead. A mismatch exits with an error, and
so does an incumbent that leaves before the check. Decided
2026-10-13 by the maintainer, over documenting a bound: the latest
timestamp alone is wrong when flagged breaks totalling half a period fall
within the delivery skew (a joiner one second behind, while the incumbent
takes two +10 h breaks, lands a wrap ahead).
- The era choice alone cannot detect a different stream, since a nearest
whole period always exists; the verification above refuses one at join.
Past that, same stream stays the operator's precondition, unchecked like
the rest of the same-epoch contract (maintainer, 2026-10-13).
- Every catalog field an importer derives from its history rather than the
content at hand is the epoch's too. The joiner adopts the live value as of
its first group, and later changes depend only on content on the epoch
timeline. In TS passthrough that is the M2ts section's `random_access` and
recorded `mux_rate`, owned by
[raw TS cuts](/quest/m1/ts-passthrough-epoch-cuts.md).
- Two importers that both start before either is live would place two
clocks under one epoch. One must exit, so that its supervisor's restart
adopts; settle how. Detection by each importer through its own relay
fails in a cluster: a replica on each relay reads before its peer's route
arrives, and `route_order` keeps preferring each relay's own route
afterwards, so neither sees the other. A co-start is also routine, since
both importers of a pair restart together under a successor epoch.
Candidates: an external leader or shared anchor; detection through state
both can see, such as the announcement of the peer's route; or co-start
as an unsupported precondition, which the successor-epoch flow would have
to avoid.
- A flagged forward break must be carried as the same function of the
source's clock values on every importer, so media time stays congruent to
the raw clock modulo the period and the era choice above holds after it.
The raw importer's clamp after a small flagged jump
(`(last + step).max(self.written)` in `passthrough.rs`) breaks that and
depends on history; remove or bound it.
- A rewind stays fatal (#4543), so both importers of a pair exit together
and restart under a successor epoch. Settle how the successor gets its
anchor without the co-start race above. The successor epoch is proposed to
derive from the old epoch and the last frame's catalog time; the same
inputs and the first source timestamp after the step can fix its anchor.
- Nothing outlives the live route, so a total outage loses the anchor: the
next importer under the same explicit epoch is a first importer, and its
era and clock can differ from the old run's. Decided 2026-10-13 by the
maintainer: an explicit epoch names one continuous run, and a supervisor
restarting after every importer stopped mints a successor epoch, as for a
rewind; the `--epoch` docs say so. Rejected: persisting the anchor beyond
the live route, and a flag marking the first importer.
- The mapping drifts from UTC at the source's clock error for the epoch's
life, as one importer's does today. Nothing re-measures it.

Tests, on mocked time where possible: for each importer that takes
`--epoch`, two instances started at different source offsets publish
identical catalogs and timestamps; the same with the second started after a
33-bit wrap the first crossed, at a wrap with its streams' first values on
opposite sides of it, after flagged forward breaks that together exceed half
a period, with observations straddling such breaks (the verification
refuses a wrong era, on every track), and with different delivery skew into
each; a first importer started at a wrap with its streams on opposite sides
of it; a stream a PMT revision adds after a wrap; a different stream under
the same epoch fails verification; a cold
co-start, including one across two clustered relays, leaves one publisher.

Public API: no new items, but `--epoch` on an importer now reads the live
broadcast before announcing, so its token needs subscribe rights on its own
path. A publish-only token that works with `--epoch` today fails there: a
breaking auth requirement, documented with `--epoch` in `doc/bin/cli.md` and
`doc/bin/relay/auth.md`. Wire: none.

## Required

- [The catalog root clock is the producer clock](/quest/m1/catalog-clock-adopt.md) - the exact adoption path a joining importer uses

## Related

- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - group sequences and the end-to-end pair on this coordinate
- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - same-epoch routes resume each other's subscriptions
43 changes: 43 additions & 0 deletions quest/m1/epoch-refusals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# [S] Publishers that cannot align refuse a shared epoch

## Goal

The CLI accepts `--epoch` only where two instances fed one stream can
publish identical tracks. It refuses it, naming the reason, for publishers
whose timestamps or bytes come from the instance rather than the input:

- `import capture`, since two encoders never produce the same bytes;
- `import avc3`, which stamps frames from their arrival on stdin
(`Split::pts`);
- `import rtc --connect`, since each pull's RTP session is its own;
- `transcode`, for the same reason as capture;
- `import archive`, whose replay rewrites catalog and timeline groups under
the live names, so one name, epoch, and group number could carry
different bytes.

All five accept it today, so a redundant pair built from them fails at its
first failover instead of at start.

## Plan

Decided 2026-10-13 by the maintainer: same-epoch routes resume each other's
subscriptions, so a publisher that cannot meet the
[same-epoch contract](/quest/m1/hop-aligned-import.md) refuses the epoch
rather than documenting a hazard. `takes_epoch` in `rs/moq-cli/src/args.rs`
returns false for the four import sources, and `validate` refuses `--epoch`
with `transcode`. The archive refusal moved here from
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
[Replay catalog](/quest/m1/archive/replay-catalog.md) so it lands with the
others, ahead of the same-epoch contract.

Update the `--epoch` paragraph in `doc/bin/cli.md` and the `--epoch` doc
comment to say what accepts it.

Tests: extend `epoch_applies_only_to_a_once_per_run_publisher` with each
refusal, and keep the other sources accepted.

Public API: `--epoch` is refused for these five, where it was accepted.

## Related

- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - the contract the accepted importers meet
- [Replay catalog](/quest/m1/archive/replay-catalog.md) - why archive replay mints its own epoch
Loading
Loading