From 3bf495f833e2296aac3b379c80bfb06d9a1b2ac9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 10 Oct 2026 22:49:36 -0700 Subject: [PATCH 1/5] chore: claim TS replacement catalog bound quest Co-Authored-By: GPT-6 Astra From 5ef61aeec5f30f1b41bf051475ccaf6217327a50 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 10 Oct 2026 23:24:57 -0700 Subject: [PATCH 2/5] fix(ts): bound replacement catalogs by the linger Co-Authored-By: GPT-6 Astra --- doc/bin/cli.md | 5 +- doc/bin/srt.md | 4 +- quest/m1/README.md | 1 - quest/m1/ts-follow-catalog-bound.md | 34 ---------- rs/moq-cli/src/args.rs | 1 + rs/moq-cli/src/srt.rs | 1 + rs/moq-mux/src/container/ts/export_test.rs | 76 +++++++++++++++++++++- rs/moq-mux/src/container/ts/follower.rs | 29 ++++++++- rs/moq-srt/src/dial.rs | 1 + rs/moq-srt/src/listen.rs | 1 + rs/moq-srt/src/server.rs | 3 +- rs/moq-srt/src/ts.rs | 2 +- 12 files changed, 116 insertions(+), 42 deletions(-) delete mode 100644 quest/m1/ts-follow-catalog-bound.md diff --git a/doc/bin/cli.md b/doc/bin/cli.md index f553d53c707..f1b0bd22bf0 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -317,7 +317,10 @@ long for the same publisher instance to come back, and carries on with the same stream. A replacement (a restarted publisher, under a new [epoch](/concept/moq-lite#publisher-epochs)) exits 1 unless `--stitch` follows it as a full program switch: a new PMT from its catalog, with every PID flagging -the break. Without `--stitch` the export stays on the old publisher while it is +the break. `--linger` also bounds a mid-stream stitch from the replacement +announcement through its first catalog snapshot; expiry exits 1. The default +`--linger 0s` follows only a replacement whose catalog is already available. +Without `--stitch` the export stays on the old publisher while it is up. Only lite-07 sessions carry epochs, so on the default version every return is a replacement. An export that fails while the broadcast stays up, such as on a codec TS cannot carry, exits 1 without lingering. Only `ts` can mark a break, diff --git a/doc/bin/srt.md b/doc/bin/srt.md index 1a07d87303b..3188573aef0 100644 --- a/doc/bin/srt.md +++ b/doc/bin/srt.md @@ -30,7 +30,9 @@ while its stale connection is still open replaces it at once: viewers see the broadcast restart, and their next subscribe reaches the new feed. An `export srt` stream ends with its broadcast; `--linger` and `--stitch` follow a return or a replacement on the same connection, as they do for -[`export ts`](/bin/cli#retention-and-latency). +[`export ts`](/bin/cli#retention-and-latency). The linger also bounds a +mid-stream replacement's first catalog; expiry ends the export with an error. +With the default `--linger 0s`, that catalog must already be available. A multi-program feed is refused unless `--program` picks one: `--program 2` imports program 2 alone, and `--program all` diff --git a/quest/m1/README.md b/quest/m1/README.md index 641ec4a11a4..f90284d4f0f 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -174,7 +174,6 @@ bench all require its crate. - [Watch decode errors](/quest/m1/watch-decode-error.md) - a WebCodecs error ends the subscription and the element reports it - [Audio warmup](/quest/m1/audio-warmup.md) - a viewer joining an Opus rendition mid-stream never hears the unconverged first 80 ms - [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - moq-gst picks the broadcast wall epoch; a restarted source is a new epoch, not a forward re-anchor -- [TS stitch catalog bound](/quest/m1/ts-follow-catalog-bound.md) - `--linger` bounds a mid-stream `--stitch` until the replacement's catalog arrives, so it ends loudly instead of stalling - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - `export ts --passthrough` writes the `m2ts` track back byte-identical (less late drops) on a fixed delay - [FLV and MKV export delay](/quest/m1/export-delay.md) - FLV and MKV interleave through the shared jitter buffer on a fixed delay, breaking the CLI once - [Release profile](/quest/m1/release-profile.md) - every release build gets fat LTO, one codegen unit, and stripping from the workspace profile instead of three script exports diff --git a/quest/m1/ts-follow-catalog-bound.md b/quest/m1/ts-follow-catalog-bound.md deleted file mode 100644 index 2ab57704a0a..00000000000 --- a/quest/m1/ts-follow-catalog-bound.md +++ /dev/null @@ -1,34 +0,0 @@ -# [S] A TS export stitch is bounded by the linger until its catalog arrives - -## Goal - -When `moq export ts --stitch` (or moq-srt egress with `stitch`) switches to -a replacement mid-stream, `--linger` bounds the wait for the replacement's -catalog, as it bounds a same-epoch return. A replacement that never delivers -its first catalog snapshot ends the export with an error instead of stalling -it with nothing going out. - -## Plan - -Found while iterating #5147 (2026-10-10), planned the same day. `ts::Follower` -(`rs/moq-mux/src/container/ts/follower.rs`) carries a linger deadline for a -return, which #5147 extended to the returned catalog's first snapshot. A -stitch that starts after the old broadcast ended inherits that deadline. A -mid-stream stitch, taken from `Running` while the old instance still serves, -carries none (`settling: None` in `State::Following`), so its catalog wait is -unbounded. - -- Decided: the linger is the one bound, from the `Restart` to the - replacement's first catalog snapshot. With `--linger 0` a stitch only - succeeds when the catalog is already there. Rejected: bounding by - `--delay`, and a fixed constant. -- On expiry the export ends with an error naming the replacement that never - served its catalog. -- Tests, with mocked time: a stitch onto a replacement whose catalog never - answers ends at the linger. One whose catalog answers within it switches - as today. -- Docs: the `--stitch` and `--linger` text in `doc/bin/cli.md` and - `doc/bin/srt.md`, including that the default `--linger 0s` lets a stitch - follow only a replacement whose catalog is already there. - -Public API: none. Behavior: a stitch can now end the export. Wire: none. diff --git a/rs/moq-cli/src/args.rs b/rs/moq-cli/src/args.rs index 25ae9ccfbd2..cbeb4fb077a 100644 --- a/rs/moq-cli/src/args.rs +++ b/rs/moq-cli/src/args.rs @@ -1059,6 +1059,7 @@ pub struct Transport { /// How long to wait for the same publisher instance to come back once it ends (e.g. `10s`), /// then carry on with the same stream. A replacement exits 1 unless `--stitch` is passed. /// An export that fails while the broadcast is still up exits without waiting. + /// Also bounds a stitched replacement through its first catalog; zero requires it already available. #[usage(long, default = "0s")] pub linger: crate::duration::Duration, diff --git a/rs/moq-cli/src/srt.rs b/rs/moq-cli/src/srt.rs index 582fd3a5880..832122048ba 100644 --- a/rs/moq-cli/src/srt.rs +++ b/rs/moq-cli/src/srt.rs @@ -56,6 +56,7 @@ pub struct ExportArgs { /// How long to wait for the same publisher instance to come back once it ends (e.g. `10s`), /// then carry on with the same stream. A replacement ends the stream unless `--stitch` /// is passed. + /// Also bounds a stitched replacement through its first catalog; zero requires it already available. #[usage(long, default = "0s")] pub linger: crate::duration::Duration, diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index d77ca6e7041..517a7af367d 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -7650,7 +7650,10 @@ async fn a_follower_stitches_onto_the_next_replacement_when_one_goes_before_it_r let origin = crate::source::produce_origin(); let route = || moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()); let first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); - let mut follower = follower(&origin).await.with_stitch(true); + let mut follower = follower(&origin) + .await + .with_stitch(true) + .with_linger(Duration::from_secs(10)); // Replaced, but the replacement's catalog request is never answered before it goes. let second = origin.publish("live", route()).unwrap(); @@ -8041,3 +8044,74 @@ async fn a_follower_ignores_a_covering_prefix_while_the_exact_route_serves() { let end = follower.next().await; assert!(matches!(end, Ok(None)), "no replacement: {end:?}"); } + +/// Both the catalog subscription and its first snapshot share the restart's linger bound. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_expires_at_the_linger() { + for snapshot in [false, true] { + for linger in [Duration::ZERO, Duration::from_secs(10)] { + let origin = crate::source::produce_origin(); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let second = origin + .publish( + "live", + moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), + ) + .unwrap(); + let _unanswered = second.dynamic(); + let _silent = snapshot.then(|| second.create_track(hang::Catalog::DEFAULT_NAME, None).unwrap()); + let waiter = kio::Waiter::noop(); + if !linger.is_zero() { + assert!(follower.poll_next(&waiter).is_pending()); + tokio::time::advance(linger - Duration::from_millis(1)).await; + assert!(follower.poll_next(&waiter).is_pending()); + tokio::time::advance(Duration::from_millis(1)).await; + } + let std::task::Poll::Ready(Err(err)) = follower.poll_next(&waiter) else { + panic!("replacement catalog wait must fail at the linger, snapshot={snapshot}, linger={linger:?}"); + }; + assert!(err.to_string().contains("live"), "{err}"); + assert!(err.to_string().contains("catalog"), "{err}"); + } + } +} + +/// A catalog ready before the bound completes the switch, including zero linger. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_within_linger_clears_the_bound() { + for linger in [Duration::ZERO, Duration::from_secs(10)] { + let origin = crate::source::produce_origin(); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let epoch = moq_net::Epoch::mint(); + let (_second, mut catalog) = publish_bare(&origin, Some(&epoch)); + let waiter = kio::Waiter::noop(); + if !linger.is_zero() { + assert!(follower.poll_next(&waiter).is_pending()); + tokio::time::advance(linger - Duration::from_millis(1)).await; + } + std::ops::DerefMut::deref_mut(&mut catalog.modify().unwrap()); + // Resolve the local async serving path so zero linger sees an already available + // subscription and snapshot, rather than a request still parked on the driver. + let broadcast = origin + .consume() + .request_broadcast("live", Some(epoch.clone())) + .await + .unwrap(); + let mut available = crate::catalog::Consumer::<()>::new(&broadcast, crate::catalog::CatalogFormat::Hang) + .await + .unwrap(); + use crate::catalog::Stream; + kio::wait(|waiter| available.poll_next(waiter)).await.unwrap().unwrap(); + let next = follower.poll_next(&waiter); + assert!(next.is_pending(), "{next:?}, linger={linger:?}"); + assert!(follower.export().cataloged()); + assert_eq!(follower.export().instance(), Some(&epoch)); + tokio::time::advance(Duration::from_secs(20)).await; + assert!( + follower.poll_next(&waiter).is_pending(), + "the first snapshot clears the bound" + ); + } +} diff --git a/rs/moq-mux/src/container/ts/follower.rs b/rs/moq-mux/src/container/ts/follower.rs index 2f28b13116d..c04859405d7 100644 --- a/rs/moq-mux/src/container/ts/follower.rs +++ b/rs/moq-mux/src/container/ts/follower.rs @@ -87,6 +87,8 @@ pub struct Follower { closed: bool, linger: Duration, stitch: bool, + /// A mid-stream replacement must supply its first catalog by this instant. + switching: Option, timer: Option>>, } @@ -111,6 +113,7 @@ impl Follower { closed: false, linger: Duration::ZERO, stitch: false, + switching: None, timer: None, }; // Take the start naming the route the export resolved now, before anything else can @@ -128,14 +131,16 @@ impl Follower { /// Wait up to `linger` for the same instance to return once the broadcast ends. Defaults to /// [`Duration::ZERO`]. /// - /// It bounds the whole return, up to the returned catalog's first snapshot. + /// It bounds the whole return, and a mid-stream stitch from its restart to the + /// replacement catalog's first snapshot. Zero requires an already available catalog. pub fn with_linger(mut self, linger: Duration) -> Self { self.linger = linger; self } /// Follow another publisher instance taking the path, as a full program switch (see - /// [`Export::follow`]). Defaults to `false`. + /// [`Export::follow`]). Defaults to `false`. Its first catalog must arrive within + /// [`with_linger`](Self::with_linger), including when the old instance is still live. pub fn with_stitch(mut self, stitch: bool) -> Self { self.stitch = stitch; self @@ -170,6 +175,7 @@ impl Follower { // instance until it ends, unless asked to switch at once. if self.stitch && matches!(self.serving, Serving::Other(_)) { tracing::info!(path = %self.path, "broadcast replaced, switching the program to the new instance"); + self.switching = Some(Instant::now() + self.linger); self.state = State::Following { resolving: self.resolve(), settling, @@ -180,6 +186,7 @@ impl Follower { let next = self.export.poll_next(waiter); if self.export.cataloged() { settling = None; + self.switching = None; } let end = match next { Poll::Ready(Ok(Some(frame))) => { @@ -189,6 +196,7 @@ impl Follower { Poll::Ready(Ok(None)) => Ok(()), Poll::Ready(Err(err)) => Err(err), Poll::Pending => { + self.check_switch(waiter)?; if let Some((end, deadline)) = &settling && self.elapsed(waiter, *deadline) { @@ -277,6 +285,7 @@ impl Follower { } tracing::warn!(path = %self.path, %err, "broadcast went before it resolved, still waiting"); self.serving = Serving::Gone; + self.switching = None; self.state = match (live, settling) { (false, Some((end, deadline))) => State::Settling { end, @@ -300,6 +309,7 @@ impl Follower { self.state = State::Running { settling }; continue; } + self.check_switch(waiter)?; if let Some((end, deadline)) = &settling && self.elapsed(waiter, *deadline) { @@ -388,6 +398,21 @@ impl Follower { } } + /// Refuse a replacement that has not supplied its catalog within the configured linger. + fn check_switch(&mut self, waiter: &kio::Waiter) -> crate::Result<()> { + if let Some(deadline) = self.switching + && self.elapsed(waiter, deadline) + { + return Err(anyhow::anyhow!( + "replacement broadcast `{}` did not serve its catalog within {:?}", + self.path, + self.linger + ) + .into()); + } + Ok(()) + } + /// Whether `at` has come, registering `waiter` to wake then if not. fn elapsed(&mut self, waiter: &kio::Waiter, at: Instant) -> bool { if Instant::now() >= at { diff --git a/rs/moq-srt/src/dial.rs b/rs/moq-srt/src/dial.rs index 32f65204958..7162daa7c5d 100644 --- a/rs/moq-srt/src/dial.rs +++ b/rs/moq-srt/src/dial.rs @@ -114,6 +114,7 @@ impl Client { /// Wait up to `linger` for the same publisher instance to come back once the broadcast /// [`publish`](Self::publish) reads ends, carrying on with the same stream. Zero (the /// default) ends the push at the broadcast's end. + /// A stitched replacement must also supply its first catalog within this bound. pub fn with_linger(mut self, linger: Duration) -> Self { self.linger = linger; self diff --git a/rs/moq-srt/src/listen.rs b/rs/moq-srt/src/listen.rs index 6630c881b79..5eb55d35cde 100644 --- a/rs/moq-srt/src/listen.rs +++ b/rs/moq-srt/src/listen.rs @@ -73,6 +73,7 @@ pub struct Config { /// How long an egress waits for the same publisher instance to come back once its /// broadcast ends, carrying on with the same stream. Zero (the default) closes the SRT /// stream at the end. Only affects egress (`m=request`). + /// A stitched replacement must also supply its first catalog within this bound. pub linger: Duration, /// Whether an egress follows another publisher instance replacing its broadcast, as a full diff --git a/rs/moq-srt/src/server.rs b/rs/moq-srt/src/server.rs index 08de535efff..9757e69645c 100644 --- a/rs/moq-srt/src/server.rs +++ b/rs/moq-srt/src/server.rs @@ -454,6 +454,7 @@ impl Subscribe { /// Wait up to `linger` for the same publisher instance to come back once the broadcast /// ends, carrying on with the same stream. Zero (the default) closes the SRT stream at /// the broadcast's end. + /// A stitched replacement must also supply its first catalog within this bound. pub fn with_linger(mut self, linger: Duration) -> Self { self.0.linger = linger; self @@ -1129,7 +1130,7 @@ mod tests { let (origin, driver) = moq_net::origin::Producer::new(moq_net::origin::Config::default()); tokio::spawn(moq_net::time::run(driver)); let mut old = Instance::publish(&origin, "old.aac"); - let (mut receiver, egress) = egress(&origin, Duration::ZERO, true).await; + let (mut receiver, egress) = egress(&origin, Duration::from_secs(10), true).await; old.write(30, 600_000_000); let first = tokio::time::timeout(Duration::from_secs(10), receiver.next()) diff --git a/rs/moq-srt/src/ts.rs b/rs/moq-srt/src/ts.rs index b3fdf755bbd..29940f4b636 100644 --- a/rs/moq-srt/src/ts.rs +++ b/rs/moq-srt/src/ts.rs @@ -165,7 +165,7 @@ impl Publisher { pub(crate) struct Options { /// The muxer's jitter-buffer delay ([`Subscriber::new`]). pub latency: Duration, - /// How long to wait for the same publisher instance to come back once its broadcast ends. + /// How long a return or stitched replacement has to supply its first catalog. pub linger: Duration, /// Follow another instance replacing the broadcast, as a full program switch. pub stitch: bool, From 80bac69cbb569104eff49e7d213f1ce1df04099e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 11:29:31 -0700 Subject: [PATCH 3/5] fix(ts): carry each catalog wait's deadline in the follower state Replace the separate switching deadline with one Awaiting bound held by Running and Following, and restore the prior state (bound and all) when a switch's instance goes before it resolves. This keeps a nested switch's pending deadline, fails a replacement whose catalog closes without a snapshot, and stops a snapshot arriving after the deadline from counting. Co-Authored-By: Claude Opus 5.5 --- rs/moq-mux/src/container/ts/export_test.rs | 107 +++++++++++++- rs/moq-mux/src/container/ts/follower.rs | 162 +++++++++++++-------- 2 files changed, 205 insertions(+), 64 deletions(-) diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 517a7af367d..7f6bde8cf3a 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -7665,13 +7665,15 @@ async fn a_follower_stitches_onto_the_next_replacement_when_one_goes_before_it_r "resolving the replacement" ); drop((second, unanswered)); - let third = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut third = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + std::ops::DerefMut::deref_mut(&mut third.1.modify().unwrap()); assert!( tokio::time::timeout(Duration::from_secs(1), follower.next()) .await .is_err(), "following the next replacement" ); + assert!(follower.export().cataloged(), "the replacement served its catalog"); // Once the first instance is gone too, the third one's end ends the export. finish(first); @@ -8115,3 +8117,106 @@ async fn a_midstream_stitch_catalog_within_linger_clears_the_bound() { ); } } + +/// A stitch onto a replacement that goes before it resolves leaves the export awaiting its +/// current replacement's catalog, by that replacement's own deadline. +#[tokio::test(start_paused = true)] +async fn a_nested_stitch_keeps_the_pending_catalog_deadline() { + let origin = crate::source::produce_origin(); + let linger = Duration::from_secs(10); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let start = tokio::time::Instant::now(); + + // Switched onto a replacement whose catalog never delivers a snapshot. + let _second = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + assert!( + tokio::time::timeout(Duration::from_secs(2), follower.next()) + .await + .is_err(), + "awaiting the replacement's catalog" + ); + + // Replaced again, but that one's catalog request is never answered before it goes. + let third = origin + .publish( + "live", + moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), + ) + .unwrap(); + let unanswered = third.dynamic(); + assert!( + tokio::time::timeout(Duration::from_secs(1), follower.next()) + .await + .is_err(), + "resolving the next replacement" + ); + drop((third, unanswered)); + + let end = tokio::time::timeout(linger * 3, follower.next()) + .await + .expect("the second replacement's deadline still bounds it"); + let err = end.expect_err("a replacement without a catalog fails"); + assert!(err.to_string().contains("catalog"), "{err}"); + assert_eq!(start.elapsed(), linger); +} + +/// A replacement whose catalog closes before its first snapshot fails the stitch, rather than +/// ending the export cleanly. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_closed_without_a_snapshot_fails() { + let origin = crate::source::produce_origin(); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin) + .await + .with_stitch(true) + .with_linger(Duration::from_secs(10)); + + let (_second, mut catalog) = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + assert!( + tokio::time::timeout(Duration::from_secs(1), follower.next()) + .await + .is_err(), + "awaiting the replacement's catalog" + ); + + catalog.finish().unwrap(); + drop(catalog); + let end = tokio::time::timeout(Duration::from_secs(30), follower.next()) + .await + .expect("the closed catalog ends the export"); + let err = end.expect_err("a replacement without a catalog fails"); + assert!(err.to_string().contains("catalog"), "{err}"); +} + +/// A snapshot that arrives after the deadline, before the follower polls again, is too late. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_after_the_linger_fails() { + let origin = crate::source::produce_origin(); + let linger = Duration::from_secs(10); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let epoch = moq_net::Epoch::mint(); + let (_second, mut catalog) = publish_bare(&origin, Some(&epoch)); + let waiter = kio::Waiter::noop(); + assert!(follower.poll_next(&waiter).is_pending()); + + tokio::time::advance(linger).await; + std::ops::DerefMut::deref_mut(&mut catalog.modify().unwrap()); + // Deliver the snapshot through the local serving path, so it is ready on the next poll. + let broadcast = origin + .consume() + .request_broadcast("live", Some(epoch.clone())) + .await + .unwrap(); + let mut available = crate::catalog::Consumer::<()>::new(&broadcast, crate::catalog::CatalogFormat::Hang) + .await + .unwrap(); + use crate::catalog::Stream; + kio::wait(|waiter| available.poll_next(waiter)).await.unwrap().unwrap(); + + let std::task::Poll::Ready(Err(err)) = follower.poll_next(&waiter) else { + panic!("a snapshot after the linger must not count"); + }; + assert!(err.to_string().contains("catalog"), "{err}"); +} diff --git a/rs/moq-mux/src/container/ts/follower.rs b/rs/moq-mux/src/container/ts/follower.rs index c04859405d7..438cfda89d3 100644 --- a/rs/moq-mux/src/container/ts/follower.rs +++ b/rs/moq-mux/src/container/ts/follower.rs @@ -35,12 +35,25 @@ enum Serving { Other(Option), } +/// A first catalog snapshot the export awaits, since a subscribed catalog that never answers is +/// no return or switch. +struct Awaiting { + /// When the linger runs out. + deadline: Instant, + missing: Missing, +} + +/// What a catalog still missing at the deadline means. +enum Missing { + /// A return that never resumed: the export ends as it did before it. + Return(crate::Result<()>), + /// A replacement switched to mid-stream that never served its catalog: a failure. + Switch, +} + enum State { - /// Exporting. A return carries how the export ended and when the linger runs out until its - /// catalog delivers a snapshot, since a subscribed catalog that never answers is no return. - Running { - settling: Option<(crate::Result<()>, Instant)>, - }, + /// Exporting, until the deadline when its catalog is yet to deliver a snapshot. + Running { awaiting: Option }, /// The export ended with `end`, waiting for the announcements to say what follows. Settling { end: crate::Result<()>, @@ -52,10 +65,10 @@ enum State { /// Resolving the instance serving the path now, to carry the export on into it. Following { resolving: Resolving, - /// How the export ended and when the linger runs out, as in `Running` or `Settling`. - settling: Option<(crate::Result<()>, Instant)>, - /// The export is still running, a stitch mid-stream, so it carries on if the switch does not. - live: bool, + /// Bounds the resolution and then the catalog's first snapshot. + awaiting: Awaiting, + /// Where the export carries on, bound and all, if the instance goes before it resolves. + fallback: Box>, }, /// Nothing follows. Done, @@ -87,8 +100,6 @@ pub struct Follower { closed: bool, linger: Duration, stitch: bool, - /// A mid-stream replacement must supply its first catalog by this instant. - switching: Option, timer: Option>>, } @@ -103,7 +114,7 @@ impl Follower { let epoch = export.instance().cloned(); let mut follower = Self { export, - state: State::Running { settling: None }, + state: State::Running { awaiting: None }, announced, path, epoch, @@ -113,7 +124,6 @@ impl Follower { closed: false, linger: Duration::ZERO, stitch: false, - switching: None, timer: None, }; // Take the start naming the route the export resolved now, before anything else can @@ -166,44 +176,67 @@ impl Follower { } fn poll_state(&mut self, waiter: &kio::Waiter) -> Poll>> { + // A deadline that passed between polls ends the wait before a late snapshot can count. + // Within one poll a ready catalog still makes it, which zero linger relies on. + let awaiting = match &self.state { + State::Running { awaiting } => awaiting.as_ref(), + State::Following { awaiting, .. } => Some(awaiting), + State::Settling { .. } | State::Done => None, + }; + if awaiting.is_some_and(|awaiting| Instant::now() >= awaiting.deadline) { + let (State::Running { + awaiting: Some(awaiting), + } + | State::Following { awaiting, .. }) = std::mem::replace(&mut self.state, State::Done) + else { + unreachable!("only a running or following state awaits a catalog"); + }; + return Poll::Ready(self.expire(awaiting)); + } + loop { match std::mem::replace(&mut self.state, State::Done) { State::Done => return Poll::Ready(Ok(None)), - State::Running { mut settling } => { + State::Running { mut awaiting } => { self.poll_announced(waiter); // Subscriptions are sticky, so a replacement leaves the export on the old // instance until it ends, unless asked to switch at once. if self.stitch && matches!(self.serving, Serving::Other(_)) { tracing::info!(path = %self.path, "broadcast replaced, switching the program to the new instance"); - self.switching = Some(Instant::now() + self.linger); self.state = State::Following { resolving: self.resolve(), - settling, - live: true, + awaiting: Awaiting { + deadline: Instant::now() + self.linger, + missing: Missing::Switch, + }, + fallback: Box::new(State::Running { awaiting }), }; continue; } let next = self.export.poll_next(waiter); if self.export.cataloged() { - settling = None; - self.switching = None; + awaiting = None; } let end = match next { Poll::Ready(Ok(Some(frame))) => { - self.state = State::Running { settling }; + self.state = State::Running { awaiting }; return Poll::Ready(Ok(Some(frame))); } - Poll::Ready(Ok(None)) => Ok(()), + // A switch whose catalog closed before its first snapshot never served one. + Poll::Ready(Ok(None)) => match awaiting { + Some(Awaiting { + missing: Missing::Switch, + .. + }) => Err(self.missing_catalog()), + _ => Ok(()), + }, Poll::Ready(Err(err)) => Err(err), Poll::Pending => { - self.check_switch(waiter)?; - if let Some((end, deadline)) = &settling - && self.elapsed(waiter, *deadline) + if let Some(awaiting) = awaiting.take_if(|awaiting| self.elapsed(waiter, awaiting.deadline)) { - tracing::info!(path = %self.path, linger = ?self.linger, "broadcast did not return"); - return Poll::Ready(end.clone().map(|()| None)); + return Poll::Ready(self.expire(awaiting)); } - self.state = State::Running { settling }; + self.state = State::Running { awaiting }; return Poll::Pending; } }; @@ -251,8 +284,15 @@ impl Follower { if follow { self.state = State::Following { resolving: self.resolve(), - settling: Some((end, deadline)), - live: false, + fallback: Box::new(State::Settling { + end: end.clone(), + grace: None, + deadline, + }), + awaiting: Awaiting { + deadline, + missing: Missing::Return(end), + }, }; continue; } @@ -265,8 +305,8 @@ impl Follower { } State::Following { mut resolving, - settling, - live, + awaiting, + fallback, } => { // Announcements wait until the export is on the instance they describe. if let Poll::Ready(resolved) = waiter.poll_future(resolving.as_mut()) { @@ -274,8 +314,9 @@ impl Follower { Ok(resolved) => resolved, // An instance that went before it resolved is no return or switch yet. // Until the next announcement says otherwise, nothing serves the path: - // a stitch stays on the export it has, and a return keeps lingering. - // A refusal from an instance still up fails loud instead. + // a stitch stays on the export it has, still awaiting what it was, and a + // return keeps lingering. A refusal from an instance still up fails loud + // instead. Err(err) => { if !matches!( err, @@ -285,15 +326,7 @@ impl Follower { } tracing::warn!(path = %self.path, %err, "broadcast went before it resolved, still waiting"); self.serving = Serving::Gone; - self.switching = None; - self.state = match (live, settling) { - (false, Some((end, deadline))) => State::Settling { - end, - grace: None, - deadline, - }, - (_, settling) => State::Running { settling }, - }; + self.state = *fallback; continue; } }; @@ -306,20 +339,18 @@ impl Follower { epoch = self.epoch.as_ref().map(tracing::field::display), "exporting broadcast" ); - self.state = State::Running { settling }; + self.state = State::Running { + awaiting: Some(awaiting), + }; continue; } - self.check_switch(waiter)?; - if let Some((end, deadline)) = &settling - && self.elapsed(waiter, *deadline) - { - tracing::info!(path = %self.path, linger = ?self.linger, "broadcast did not return"); - return Poll::Ready(end.clone().map(|()| None)); + if self.elapsed(waiter, awaiting.deadline) { + return Poll::Ready(self.expire(awaiting)); } self.state = State::Following { resolving, - settling, - live, + awaiting, + fallback, }; return Poll::Pending; } @@ -398,19 +429,24 @@ impl Follower { } } - /// Refuse a replacement that has not supplied its catalog within the configured linger. - fn check_switch(&mut self, waiter: &kio::Waiter) -> crate::Result<()> { - if let Some(deadline) = self.switching - && self.elapsed(waiter, deadline) - { - return Err(anyhow::anyhow!( - "replacement broadcast `{}` did not serve its catalog within {:?}", - self.path, - self.linger - ) - .into()); + /// How a wait for the catalog's first snapshot ends once the linger runs out. + fn expire(&self, awaiting: Awaiting) -> crate::Result> { + match awaiting.missing { + Missing::Return(end) => { + tracing::info!(path = %self.path, linger = ?self.linger, "broadcast did not return"); + end.map(|()| None) + } + Missing::Switch => Err(self.missing_catalog()), } - Ok(()) + } + + fn missing_catalog(&self) -> crate::Error { + anyhow::anyhow!( + "replacement broadcast `{}` did not serve its catalog within {:?}", + self.path, + self.linger + ) + .into() } /// Whether `at` has come, registering `waiter` to wake then if not. From 950ddcb7743d6f796ed354e20ef5df7fdf35e7a1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 18:35:41 -0700 Subject: [PATCH 4/5] fix(ts): expire a restored fallback's catalog wait before resuming it A nested stitch restoring an earlier replacement after its deadline continued inside the poll loop, so a late first snapshot could clear the bound. Check the restored wait like the poll-entry check does. Co-Authored-By: Claude Opus 5.5 --- rs/moq-mux/src/container/ts/export_test.rs | 53 ++++++++++++++++++++++ rs/moq-mux/src/container/ts/follower.rs | 45 +++++++++++------- 2 files changed, 82 insertions(+), 16 deletions(-) diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index e4f609c7cd9..c391ff6bb21 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -8209,6 +8209,59 @@ async fn a_nested_stitch_keeps_the_pending_catalog_deadline() { assert_eq!(start.elapsed(), linger); } +/// A replacement restored after its own deadline, when the next one goes before it resolves, +/// fails even with its late first snapshot ready. +#[tokio::test(start_paused = true)] +async fn a_nested_stitch_restored_after_its_deadline_fails() { + let origin = crate::source::produce_origin(); + let linger = Duration::from_secs(10); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let start = tokio::time::Instant::now(); + + // Switched onto a replacement whose catalog has no snapshot yet, bounded at 10s. + let epoch = moq_net::Epoch::mint(); + let (_second, mut catalog) = publish_bare(&origin, Some(&epoch)); + assert!( + tokio::time::timeout(Duration::from_secs(2), follower.next()) + .await + .is_err(), + "awaiting the replacement's catalog" + ); + + // Replaced again at 2s, bounded at 12s, but that one's catalog request is never answered. + let third = origin + .publish( + "live", + moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), + ) + .unwrap(); + let unanswered = third.dynamic(); + assert!( + tokio::time::timeout(Duration::from_secs(1), follower.next()) + .await + .is_err(), + "resolving the next replacement" + ); + + // At 11s, the third goes and the second's first snapshot is ready, both past its deadline. + tokio::time::advance(linger + Duration::from_secs(1) - start.elapsed()).await; + drop((third, unanswered)); + std::ops::DerefMut::deref_mut(&mut catalog.modify().unwrap()); + // Deliver the snapshot through the local serving path, so it is ready on the next poll. + let broadcast = origin.consume().request_broadcast("live", Some(epoch)).await.unwrap(); + let mut available = crate::catalog::Consumer::<()>::new(&broadcast, crate::catalog::CatalogFormat::Hang) + .await + .unwrap(); + use crate::catalog::Stream; + kio::wait(|waiter| available.poll_next(waiter)).await.unwrap().unwrap(); + + let std::task::Poll::Ready(Err(err)) = follower.poll_next(&kio::Waiter::noop()) else { + panic!("a snapshot after the restored replacement's deadline must not count"); + }; + assert!(err.to_string().contains("catalog"), "{err}"); +} + /// A replacement whose catalog closes before its first snapshot fails the stitch, rather than /// ending the export cleanly. #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/container/ts/follower.rs b/rs/moq-mux/src/container/ts/follower.rs index 438cfda89d3..21a2592a30c 100644 --- a/rs/moq-mux/src/container/ts/follower.rs +++ b/rs/moq-mux/src/container/ts/follower.rs @@ -176,22 +176,8 @@ impl Follower { } fn poll_state(&mut self, waiter: &kio::Waiter) -> Poll>> { - // A deadline that passed between polls ends the wait before a late snapshot can count. - // Within one poll a ready catalog still makes it, which zero linger relies on. - let awaiting = match &self.state { - State::Running { awaiting } => awaiting.as_ref(), - State::Following { awaiting, .. } => Some(awaiting), - State::Settling { .. } | State::Done => None, - }; - if awaiting.is_some_and(|awaiting| Instant::now() >= awaiting.deadline) { - let (State::Running { - awaiting: Some(awaiting), - } - | State::Following { awaiting, .. }) = std::mem::replace(&mut self.state, State::Done) - else { - unreachable!("only a running or following state awaits a catalog"); - }; - return Poll::Ready(self.expire(awaiting)); + if let Some(expired) = self.expired() { + return Poll::Ready(expired); } loop { @@ -327,6 +313,10 @@ impl Follower { tracing::warn!(path = %self.path, %err, "broadcast went before it resolved, still waiting"); self.serving = Serving::Gone; self.state = *fallback; + // The fallback's wait predates this poll, so it gets no same-poll grace. + if let Some(expired) = self.expired() { + return Poll::Ready(expired); + } continue; } }; @@ -429,6 +419,29 @@ impl Follower { } } + /// End a catalog wait already past its deadline, before a late snapshot can count. + /// + /// Only waits from an earlier poll are checked: within one poll a ready catalog still makes + /// it, which zero linger relies on. + fn expired(&mut self) -> Option>> { + let awaiting = match &self.state { + State::Running { awaiting } => awaiting.as_ref(), + State::Following { awaiting, .. } => Some(awaiting), + State::Settling { .. } | State::Done => None, + }; + if !awaiting.is_some_and(|awaiting| Instant::now() >= awaiting.deadline) { + return None; + } + let (State::Running { + awaiting: Some(awaiting), + } + | State::Following { awaiting, .. }) = std::mem::replace(&mut self.state, State::Done) + else { + unreachable!("only a running or following state awaits a catalog"); + }; + Some(self.expire(awaiting)) + } + /// How a wait for the catalog's first snapshot ends once the linger runs out. fn expire(&self, awaiting: Awaiting) -> crate::Result> { match awaiting.missing { From 80c6364de8aef802e05edcb60e6e2c5881cce8fc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 18:46:38 -0700 Subject: [PATCH 5/5] quest: point links to the finished catalog-bound quest at #5305 Co-Authored-By: Claude Opus 5.5 --- quest/m1/initial-catalog-routing.md | 2 +- quest/m1/ts-same-instance-resume.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/quest/m1/initial-catalog-routing.md b/quest/m1/initial-catalog-routing.md index a747ec0997e..628ed95df5f 100644 --- a/quest/m1/initial-catalog-routing.md +++ b/quest/m1/initial-catalog-routing.md @@ -58,5 +58,5 @@ decision rather than an invented retry or compatibility layer. - [Fallback PR #5284](https://github.com/moq-dev/moq/pull/5284) - check its actual effect on this reproduction before duplicating work - [Transport upgrade](/quest/m1/transport-upgrade/README.md) - existing handover and epoch contracts - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - previously captured only a short suffix, not zero-packet initial acquisition -- [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - bounds a midstream stitch, not initial discovery +- [Replacement catalog linger #5305](https://github.com/moq-dev/moq/pull/5305) - bounds a midstream stitch, not initial discovery - [Pair acquisition](/quest/m1/ts-pair-acquisition.md) - independent grader/mux finding diff --git a/quest/m1/ts-same-instance-resume.md b/quest/m1/ts-same-instance-resume.md index fbab9ce0504..62c7fa1ec74 100644 --- a/quest/m1/ts-same-instance-resume.md +++ b/quest/m1/ts-same-instance-resume.md @@ -59,7 +59,7 @@ required behavior or API change returns to the maintainer for a decision. - [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328#issuecomment-6114456593) - failing full-check evidence - [Catalog identity controls #5333](https://github.com/moq-dev/moq/pull/5333) - initial acquisition preserves the selected contract - [Live clock validation](/quest/m1/ts-live-clock.md) - independent local live failure -- [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - #5305 owns the bound on a mid-stream switch's first catalog +- [Replacement catalog linger #5305](https://github.com/moq-dev/moq/pull/5305) - owns the bound on a mid-stream switch's first catalog - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - shared catalogs, tracks and source-derived group identity - [Closed fallback](/quest/m1/transport-upgrade/closed-fallback.md) - #5284 owns early WebSocket closure and its transport-test fix - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - diagnose a shared end-of-stream cause only if reproduced