From 60cedfedde2173daab307febedfb190f3bea1cbf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 20 Oct 2026 09:12:47 -0700 Subject: [PATCH 1/2] quest: plan resume through an undetected outage (#6062) Adds quest/m1/resume-outage.md for a regression test and doc fix, and counts an unclosed route loss as the stall route health must first reproduce. Narrows the broadcast-epoch Goal: a same-epoch resume is still bounded by the subscriber's max age. Co-Authored-By: Claude Opus 5.5 --- quest/m0/broadcast-epoch/README.md | 5 ++- quest/m1/README.md | 1 + quest/m1/resume-outage.md | 63 ++++++++++++++++++++++++++++++ quest/m2/route-health/README.md | 5 +++ quest/m2/route-health/design.md | 5 ++- quest/m2/route-health/recovery.md | 2 + 6 files changed, 78 insertions(+), 3 deletions(-) create mode 100644 quest/m1/resume-outage.md diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index 4d9d34f835c..8e7a412e0f5 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -4,8 +4,9 @@ A path and the epoch on its route are the only content identity, and no first-party publisher reuses a pair for different content. Only routes with -the same epoch resume a subscription from the first frame it lacks; a route -without one keeps its subscriptions until it goes. So epochs are what make +the same epoch resume a subscription from the first frame it lacks, within +the subscriber's max age; a route without one keeps its subscriptions until +it goes. So epochs are what make failover seamless. Routes without an epoch never splice (maintainer, 2026-10-08): a `Restart` (or an end and start on older versions) tells every downstream subscriber to stop using its copy of the old source and diff --git a/quest/m1/README.md b/quest/m1/README.md index 5be7d6e1dcc..1bb3548e3f2 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -105,6 +105,7 @@ lands before the cut. - [SUBSCRIBE_OK names where delivery starts](/quest/m1/subscribe-ok-start.md) - a warm relay names the same start group as a cold one instead of groups it skipped as stale - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track keeps its declared end across copies, and its `closed` and `largest()` settle on a clean end - [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late, and no longer than the reader's budget +- [Resume through an outage](/quest/m1/resume-outage.md) - a same-epoch resume after an unclosed route dies delivers the backlog its max age still holds, pinned by a regression test - [Recovered group readiness](/quest/m1/resume-readable.md) - a recovered JS group's `readable()` ignores frames its replacement replays - [Resumed datagrams](/quest/m1/resume-datagrams.md) - a resumed track skips only datagrams it already delivered, not every lower sequence - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops diff --git a/quest/m1/resume-outage.md b/quest/m1/resume-outage.md new file mode 100644 index 00000000000..c77594ac747 --- /dev/null +++ b/quest/m1/resume-outage.md @@ -0,0 +1,63 @@ +# [S] Resume through an undetected outage + +## Goal + +A same-epoch resume after a route dies without a close (a killed relay, +detected only at the QUIC idle timeout) delivers every group the outage +backlog still holds within the subscriber's max age, from the first frame the +reader lacks. A regression test pins this, and the docs stop promising more +than max age allows. + +## Plan + +Reported in [#6062](https://github.com/moq-dev/moq/issues/6062): a diamond +O -> M1/M2 -> E on lite-07 with `--quic-idle-timeout 6s`, M1 SIGKILLed, and +`moq export ts` on E. Each exporter logged `media skipped upstream` for +2-7 s on every track, at both the default `--delay` and `--delay 10s`. + +Triage against `5284d969bf` (2026-10-20, code reading only): + +- Resume already asks for the right floor. `Reader::sync` subscribes the new + copy at `floor = self.newest` (`rs/moq-net/src/model/resume.rs`), and + `Subscription::clamped` raises the wire start to it. A cold M2 forwards that + start to O. +- Every hop then applies the subscriber's max age, which is `--delay / 2` for + `export ts`. Under it, groups that went stale during detection are expired, + as the draft requires ("A floor is not a request", `Group Start` in + `drafts/draft-lcurley-moq-lite.md`). At the default 1 s delay, losing most + of the 6.6 s window is intended. +- At `--delay 10s` (a 5 s budget) the code predicts about 2 s lost, not the + same loss as the default. Nothing read explains the report there, so it may + be a real bug. Candidates: the wall-clock `superseded` rule in + `model/track.rs`; `sync` finding no copies and dropping the floor + (`resume.rs`, `subscription.rs` `clamped`); or the container consumer's + hole and latency skips (`rs/moq-mux/src/container/consumer.rs`). +- [#6019](https://github.com/moq-dev/moq/pull/6019) and + [#6024](https://github.com/moq-dev/moq/pull/6024) do not touch this path. + +Decided with the maintainer, 2026-10-20: + +- Keep max-age semantics. Recovering the whole gap regardless of max age + (widening max age on resume, or fetching the gap) was declined. +- Test at the model level only, in + `rs/moq-net/tests/integration/route_change.rs` under mock time, next to + `lite_resumes_after_a_lagging_route_dies`. The alternate route starts cold. + The test writes timestamped groups while A's route is silent but not yet + closed, then aborts it. With a budget above the outage, every group + arrives. With a budget below it, only the in-budget tail arrives. No + end-to-end relay kill test. +- Fix any bug the test exposes at its source in moq-net, not in the exporter. +- Correct `doc/concept/moq-lite.md` (*Same epoch resumes*): the resume point + is a floor, and the subscriber's max age still bounds the backlog. +- Faster detection of the dead route belongs to + [route health](/quest/m2/route-health/README.md), not here. + +## Closes + +- [#6062](https://github.com/moq-dev/moq/issues/6062) - same-epoch resume after a relay loss continues at a later group + +## Related + +- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - which replaced copies stay while groups below them are in flight +- [A widened max age re-offers cached groups](/quest/m1/resolved-demand.md) - the same max-age bound on a warm relay +- [Redundant failover drift](/quest/m1/test-flakes-2/redundant-failover-drift.md) - the same idle-timeout outage hitting the TS jitter clock diff --git a/quest/m2/route-health/README.md b/quest/m2/route-health/README.md index e0ccd4e6293..1d41f3a4540 100644 --- a/quest/m2/route-health/README.md +++ b/quest/m2/route-health/README.md @@ -16,6 +16,11 @@ Decided with the maintainer, 2026-10-13: - Ranked in m2, after clustering stability, federation, and P2P. Health design must first reproduce a stall that transport failure and ordinary reconnect do not recover, or the line is abandoned. +- A route that dies without a close counts as such a stall until the QUIC + idle timeout fires (decided 2026-10-20, from + [#6062](https://github.com/moq-dev/moq/issues/6062)): a killed relay + starves a same-epoch subscription for the whole timeout, and a subscriber + max age below it sheds that backlog even though resume asks for it. - Application-assigned connection priority remains ahead of additive cost. Deterministic hashing still chooses distinct equal routes; a new connection diff --git a/quest/m2/route-health/design.md b/quest/m2/route-health/design.md index 33644aebbe5..ae3030f8816 100644 --- a/quest/m2/route-health/design.md +++ b/quest/m2/route-health/design.md @@ -11,7 +11,10 @@ do not add speculative production code or choose arbitrary timing thresholds. The [line's decisions](/quest/m2/route-health/README.md) are settled. Start by reproducing a stall that transport failure and ordinary reconnect miss; if -none exists, bring abandoning the line back to the maintainer. Preserve +none exists, bring abandoning the line back to the maintainer. A killed relay +in a same-epoch diamond qualifies until the idle timeout fires +([#6062](https://github.com/moq-dev/moq/issues/6062)); measure the media it +loses and how far a standby shortens that. Preserve application priorities, automatic health-driven switching and failback, a payload-free steady standby, and bounded payload overlap to verify delivery. The mechanism is entirely excluded for epochless broadcasts. diff --git a/quest/m2/route-health/recovery.md b/quest/m2/route-health/recovery.md index 3494e972a27..f32bc0591e7 100644 --- a/quest/m2/route-health/recovery.md +++ b/quest/m2/route-health/recovery.md @@ -5,6 +5,8 @@ Endpoints and relays automatically move a stalled track to a verified working same-epoch alternate, then return to the preferred route after demonstrated recovery, while preserving logical readers and bounded control/media overhead. +This includes a route that dies without a close: the track moves before the +QUIC idle timeout declares it dead. Epochless broadcasts never enter this active/passive mechanism. ## Plan From 64e48f116e76fd8446624d6f82edf6527dd67590 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 20 Oct 2026 09:18:56 -0700 Subject: [PATCH 2/2] quest: cover the exporter consumer in resume-outage Co-Authored-By: Claude Opus 5.5 --- quest/m1/resume-outage.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/quest/m1/resume-outage.md b/quest/m1/resume-outage.md index c77594ac747..e63430d778e 100644 --- a/quest/m1/resume-outage.md +++ b/quest/m1/resume-outage.md @@ -46,6 +46,11 @@ Decided with the maintainer, 2026-10-20: closed, then aborts it. With a budget above the outage, every group arrives. With a budget below it, only the in-budget tail arrives. No end-to-end relay kill test. +- The `--delay 10s` loss must be explained before this closes #6062. If the + moq-net test passes, add a mock-time unit test in + `rs/moq-mux/src/container/consumer.rs` that feeds the consumer a resumed + backlog burst within its budget, and assert it reads every group without a + `Read::Gap`. - Fix any bug the test exposes at its source in moq-net, not in the exporter. - Correct `doc/concept/moq-lite.md` (*Same epoch resumes*): the resume point is a floor, and the subscriber's max age still bounds the backlog.