diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 25a953b54c8..198c6144930 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -324,7 +324,10 @@ long for the same publisher instance to come back, and carries on with the same stream. A replacement (a restarted publisher, under a new [epoch](/concept/moq-lite#publisher-epochs)) exits 1 unless `--stitch` follows it as a full program switch: a new PMT from its catalog, with every PID flagging -the break. Without `--stitch` the export stays on the old publisher while it is +the break. `--linger` also bounds a mid-stream stitch from the replacement +announcement through its first catalog snapshot; expiry exits 1. The default +`--linger 0s` follows only a replacement whose catalog is already available. +Without `--stitch` the export stays on the old publisher while it is up. Only lite-07 sessions carry epochs, so on the default version every return is a replacement. An export that fails while the broadcast stays up, such as on a codec TS cannot carry, exits 1 without lingering. Only `ts` can mark a break, diff --git a/doc/bin/srt.md b/doc/bin/srt.md index 1a07d87303b..3188573aef0 100644 --- a/doc/bin/srt.md +++ b/doc/bin/srt.md @@ -30,7 +30,9 @@ while its stale connection is still open replaces it at once: viewers see the broadcast restart, and their next subscribe reaches the new feed. An `export srt` stream ends with its broadcast; `--linger` and `--stitch` follow a return or a replacement on the same connection, as they do for -[`export ts`](/bin/cli#retention-and-latency). +[`export ts`](/bin/cli#retention-and-latency). The linger also bounds a +mid-stream replacement's first catalog; expiry ends the export with an error. +With the default `--linger 0s`, that catalog must already be available. A multi-program feed is refused unless `--program` picks one: `--program 2` imports program 2 alone, and `--program all` diff --git a/quest/m1/README.md b/quest/m1/README.md index 589159cfc9c..b35ce39519a 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -173,7 +173,6 @@ folded into Dropped sources. - [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - a viewer joining at a recovery point drops the leading pictures it cannot decode; continuous viewers keep them - [Audio warmup](/quest/m1/audio-warmup.md) - a viewer joining an Opus rendition mid-stream never hears the unconverged first 80 ms - [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - moq-gst picks the broadcast wall epoch; a restarted source is a new epoch, not a forward re-anchor -- [TS stitch catalog bound](/quest/m1/ts-follow-catalog-bound.md) - `--linger` bounds a mid-stream `--stitch` until the replacement's catalog arrives, so it ends loudly instead of stalling - [moqsink first catalog](/quest/m1/gst-first-catalog.md) - a restarted `moqsink` lists every requested pad in its first catalog, so `moqsrc` resumes each on its held pad - [moqsrc A/V alignment](/quest/m1/gst-src-av-sync.md) - every `moqsrc` pad of a run shares one timestamp reference and segment base - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - `export ts --passthrough` writes the `m2ts` track back byte-identical (less late drops) on a fixed delay diff --git a/quest/m1/initial-catalog-routing.md b/quest/m1/initial-catalog-routing.md index 7c15563aae1..c29da821975 100644 --- a/quest/m1/initial-catalog-routing.md +++ b/quest/m1/initial-catalog-routing.md @@ -57,5 +57,5 @@ decision rather than an invented retry or compatibility layer. - [Fallback PR #5284](https://github.com/moq-dev/moq/pull/5284) - check its actual effect on this reproduction before duplicating work - [Transport upgrade](/quest/m1/transport-upgrade/README.md) - existing handover and epoch contracts - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - previously captured only a short suffix, not zero-packet initial acquisition -- [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - bounds a midstream stitch, not initial discovery +- [Replacement catalog linger #5305](https://github.com/moq-dev/moq/pull/5305) - bounds a midstream stitch, not initial discovery - [Pair validation](/test/ts/README.md#table-anchoring-table-anchorpy) - independent grader/acquisition checks diff --git a/quest/m1/ts-follow-catalog-bound.md b/quest/m1/ts-follow-catalog-bound.md deleted file mode 100644 index 2ab57704a0a..00000000000 --- a/quest/m1/ts-follow-catalog-bound.md +++ /dev/null @@ -1,34 +0,0 @@ -# [S] A TS export stitch is bounded by the linger until its catalog arrives - -## Goal - -When `moq export ts --stitch` (or moq-srt egress with `stitch`) switches to -a replacement mid-stream, `--linger` bounds the wait for the replacement's -catalog, as it bounds a same-epoch return. A replacement that never delivers -its first catalog snapshot ends the export with an error instead of stalling -it with nothing going out. - -## Plan - -Found while iterating #5147 (2026-10-10), planned the same day. `ts::Follower` -(`rs/moq-mux/src/container/ts/follower.rs`) carries a linger deadline for a -return, which #5147 extended to the returned catalog's first snapshot. A -stitch that starts after the old broadcast ended inherits that deadline. A -mid-stream stitch, taken from `Running` while the old instance still serves, -carries none (`settling: None` in `State::Following`), so its catalog wait is -unbounded. - -- Decided: the linger is the one bound, from the `Restart` to the - replacement's first catalog snapshot. With `--linger 0` a stitch only - succeeds when the catalog is already there. Rejected: bounding by - `--delay`, and a fixed constant. -- On expiry the export ends with an error naming the replacement that never - served its catalog. -- Tests, with mocked time: a stitch onto a replacement whose catalog never - answers ends at the linger. One whose catalog answers within it switches - as today. -- Docs: the `--stitch` and `--linger` text in `doc/bin/cli.md` and - `doc/bin/srt.md`, including that the default `--linger 0s` lets a stitch - follow only a replacement whose catalog is already there. - -Public API: none. Behavior: a stitch can now end the export. Wire: none. diff --git a/quest/m1/ts-same-instance-resume.md b/quest/m1/ts-same-instance-resume.md index e596d48f915..eb454055c42 100644 --- a/quest/m1/ts-same-instance-resume.md +++ b/quest/m1/ts-same-instance-resume.md @@ -59,7 +59,7 @@ required behavior or API change returns to the maintainer for a decision. - [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328#issuecomment-6114456593) - failing full-check evidence - [Catalog identity controls #5333](https://github.com/moq-dev/moq/pull/5333) - initial acquisition preserves the selected contract - [Live clock validation](/quest/m1/ts-live-clock.md) - independent local live failure -- [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - #5305 owns the bound on a mid-stream switch's first catalog +- [Replacement catalog linger #5305](https://github.com/moq-dev/moq/pull/5305) - owns the bound on a mid-stream switch's first catalog - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - shared catalogs, tracks and source-derived group identity - [Closed fallback #5284](https://github.com/moq-dev/moq/pull/5284) - early WebSocket closure and its transport-test fix - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - diagnose a shared end-of-stream cause only if reproduced diff --git a/rs/moq-cli/src/args.rs b/rs/moq-cli/src/args.rs index 8a67d6f6ab7..0bcfe61cfa0 100644 --- a/rs/moq-cli/src/args.rs +++ b/rs/moq-cli/src/args.rs @@ -1060,6 +1060,7 @@ pub struct Transport { /// How long to wait for the same publisher instance to come back once it ends (e.g. `10s`), /// then carry on with the same stream. A replacement exits 1 unless `--stitch` is passed. /// An export that fails while the broadcast is still up exits without waiting. + /// Also bounds a stitched replacement through its first catalog; zero requires it already available. #[usage(long, default = "0s")] pub linger: crate::duration::Duration, diff --git a/rs/moq-cli/src/srt.rs b/rs/moq-cli/src/srt.rs index 582fd3a5880..832122048ba 100644 --- a/rs/moq-cli/src/srt.rs +++ b/rs/moq-cli/src/srt.rs @@ -56,6 +56,7 @@ pub struct ExportArgs { /// How long to wait for the same publisher instance to come back once it ends (e.g. `10s`), /// then carry on with the same stream. A replacement ends the stream unless `--stitch` /// is passed. + /// Also bounds a stitched replacement through its first catalog; zero requires it already available. #[usage(long, default = "0s")] pub linger: crate::duration::Duration, diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 8635d0c8bc5..214d4b870c1 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -7945,7 +7945,10 @@ async fn a_follower_stitches_onto_the_next_replacement_when_one_goes_before_it_r let origin = crate::source::produce_origin(); let route = || moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()); let first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); - let mut follower = follower(&origin).await.with_stitch(true); + let mut follower = follower(&origin) + .await + .with_stitch(true) + .with_linger(Duration::from_secs(10)); // Replaced, but the replacement's catalog request is never answered before it goes. let second = origin.publish("live", route()).unwrap(); @@ -7957,13 +7960,15 @@ async fn a_follower_stitches_onto_the_next_replacement_when_one_goes_before_it_r "resolving the replacement" ); drop((second, unanswered)); - let third = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut third = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + std::ops::DerefMut::deref_mut(&mut third.1.modify().unwrap()); assert!( tokio::time::timeout(Duration::from_secs(1), follower.next()) .await .is_err(), "following the next replacement" ); + assert!(follower.export().cataloged(), "the replacement served its catalog"); // Once the first instance is gone too, the third one's end ends the export. finish(first); @@ -8338,6 +8343,233 @@ async fn a_follower_ignores_a_covering_prefix_while_the_exact_route_serves() { assert!(matches!(end, Ok(None)), "no replacement: {end:?}"); } +/// Both the catalog subscription and its first snapshot share the restart's linger bound. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_expires_at_the_linger() { + for snapshot in [false, true] { + for linger in [Duration::ZERO, Duration::from_secs(10)] { + let origin = crate::source::produce_origin(); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let second = origin + .publish( + "live", + moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), + ) + .unwrap(); + let _unanswered = second.dynamic(); + let _silent = snapshot.then(|| second.create_track(hang::Catalog::DEFAULT_NAME, None).unwrap()); + let waiter = kio::Waiter::noop(); + if !linger.is_zero() { + assert!(follower.poll_next(&waiter).is_pending()); + tokio::time::advance(linger - Duration::from_millis(1)).await; + assert!(follower.poll_next(&waiter).is_pending()); + tokio::time::advance(Duration::from_millis(1)).await; + } + let std::task::Poll::Ready(Err(err)) = follower.poll_next(&waiter) else { + panic!("replacement catalog wait must fail at the linger, snapshot={snapshot}, linger={linger:?}"); + }; + assert!(err.to_string().contains("live"), "{err}"); + assert!(err.to_string().contains("catalog"), "{err}"); + } + } +} + +/// A catalog ready before the bound completes the switch, including zero linger. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_within_linger_clears_the_bound() { + for linger in [Duration::ZERO, Duration::from_secs(10)] { + let origin = crate::source::produce_origin(); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let epoch = moq_net::Epoch::mint(); + let (_second, mut catalog) = publish_bare(&origin, Some(&epoch)); + let waiter = kio::Waiter::noop(); + if !linger.is_zero() { + assert!(follower.poll_next(&waiter).is_pending()); + tokio::time::advance(linger - Duration::from_millis(1)).await; + } + std::ops::DerefMut::deref_mut(&mut catalog.modify().unwrap()); + // Resolve the local async serving path so zero linger sees an already available + // subscription and snapshot, rather than a request still parked on the driver. + let broadcast = origin + .consume() + .request_broadcast("live", Some(epoch.clone())) + .await + .unwrap(); + let mut available = crate::catalog::Consumer::<()>::new(&broadcast, crate::catalog::CatalogFormat::Hang) + .await + .unwrap(); + use crate::catalog::Stream; + kio::wait(|waiter| available.poll_next(waiter)).await.unwrap().unwrap(); + let next = follower.poll_next(&waiter); + assert!(next.is_pending(), "{next:?}, linger={linger:?}"); + assert!(follower.export().cataloged()); + assert_eq!(follower.export().instance(), Some(&epoch)); + tokio::time::advance(Duration::from_secs(20)).await; + assert!( + follower.poll_next(&waiter).is_pending(), + "the first snapshot clears the bound" + ); + } +} + +/// A stitch onto a replacement that goes before it resolves leaves the export awaiting its +/// current replacement's catalog, by that replacement's own deadline. +#[tokio::test(start_paused = true)] +async fn a_nested_stitch_keeps_the_pending_catalog_deadline() { + let origin = crate::source::produce_origin(); + let linger = Duration::from_secs(10); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let start = tokio::time::Instant::now(); + + // Switched onto a replacement whose catalog never delivers a snapshot. + let _second = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + assert!( + tokio::time::timeout(Duration::from_secs(2), follower.next()) + .await + .is_err(), + "awaiting the replacement's catalog" + ); + + // Replaced again, but that one's catalog request is never answered before it goes. + let third = origin + .publish( + "live", + moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), + ) + .unwrap(); + let unanswered = third.dynamic(); + assert!( + tokio::time::timeout(Duration::from_secs(1), follower.next()) + .await + .is_err(), + "resolving the next replacement" + ); + drop((third, unanswered)); + + let end = tokio::time::timeout(linger * 3, follower.next()) + .await + .expect("the second replacement's deadline still bounds it"); + let err = end.expect_err("a replacement without a catalog fails"); + assert!(err.to_string().contains("catalog"), "{err}"); + assert_eq!(start.elapsed(), linger); +} + +/// A replacement restored after its own deadline, when the next one goes before it resolves, +/// fails even with its late first snapshot ready. +#[tokio::test(start_paused = true)] +async fn a_nested_stitch_restored_after_its_deadline_fails() { + let origin = crate::source::produce_origin(); + let linger = Duration::from_secs(10); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let start = tokio::time::Instant::now(); + + // Switched onto a replacement whose catalog has no snapshot yet, bounded at 10s. + let epoch = moq_net::Epoch::mint(); + let (_second, mut catalog) = publish_bare(&origin, Some(&epoch)); + assert!( + tokio::time::timeout(Duration::from_secs(2), follower.next()) + .await + .is_err(), + "awaiting the replacement's catalog" + ); + + // Replaced again at 2s, bounded at 12s, but that one's catalog request is never answered. + let third = origin + .publish( + "live", + moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), + ) + .unwrap(); + let unanswered = third.dynamic(); + assert!( + tokio::time::timeout(Duration::from_secs(1), follower.next()) + .await + .is_err(), + "resolving the next replacement" + ); + + // At 11s, the third goes and the second's first snapshot is ready, both past its deadline. + tokio::time::advance(linger + Duration::from_secs(1) - start.elapsed()).await; + drop((third, unanswered)); + std::ops::DerefMut::deref_mut(&mut catalog.modify().unwrap()); + // Deliver the snapshot through the local serving path, so it is ready on the next poll. + let broadcast = origin.consume().request_broadcast("live", Some(epoch)).await.unwrap(); + let mut available = crate::catalog::Consumer::<()>::new(&broadcast, crate::catalog::CatalogFormat::Hang) + .await + .unwrap(); + use crate::catalog::Stream; + kio::wait(|waiter| available.poll_next(waiter)).await.unwrap().unwrap(); + + let std::task::Poll::Ready(Err(err)) = follower.poll_next(&kio::Waiter::noop()) else { + panic!("a snapshot after the restored replacement's deadline must not count"); + }; + assert!(err.to_string().contains("catalog"), "{err}"); +} + +/// A replacement whose catalog closes before its first snapshot fails the stitch, rather than +/// ending the export cleanly. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_closed_without_a_snapshot_fails() { + let origin = crate::source::produce_origin(); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin) + .await + .with_stitch(true) + .with_linger(Duration::from_secs(10)); + + let (_second, mut catalog) = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + assert!( + tokio::time::timeout(Duration::from_secs(1), follower.next()) + .await + .is_err(), + "awaiting the replacement's catalog" + ); + + catalog.finish().unwrap(); + drop(catalog); + let end = tokio::time::timeout(Duration::from_secs(30), follower.next()) + .await + .expect("the closed catalog ends the export"); + let err = end.expect_err("a replacement without a catalog fails"); + assert!(err.to_string().contains("catalog"), "{err}"); +} + +/// A snapshot that arrives after the deadline, before the follower polls again, is too late. +#[tokio::test(start_paused = true)] +async fn a_midstream_stitch_catalog_after_the_linger_fails() { + let origin = crate::source::produce_origin(); + let linger = Duration::from_secs(10); + let _first = publish_bare(&origin, Some(&moq_net::Epoch::mint())); + let mut follower = follower(&origin).await.with_stitch(true).with_linger(linger); + let epoch = moq_net::Epoch::mint(); + let (_second, mut catalog) = publish_bare(&origin, Some(&epoch)); + let waiter = kio::Waiter::noop(); + assert!(follower.poll_next(&waiter).is_pending()); + + tokio::time::advance(linger).await; + std::ops::DerefMut::deref_mut(&mut catalog.modify().unwrap()); + // Deliver the snapshot through the local serving path, so it is ready on the next poll. + let broadcast = origin + .consume() + .request_broadcast("live", Some(epoch.clone())) + .await + .unwrap(); + let mut available = crate::catalog::Consumer::<()>::new(&broadcast, crate::catalog::CatalogFormat::Hang) + .await + .unwrap(); + use crate::catalog::Stream; + kio::wait(|waiter| available.poll_next(waiter)).await.unwrap().unwrap(); + + let std::task::Poll::Ready(Err(err)) = follower.poll_next(&waiter) else { + panic!("a snapshot after the linger must not count"); + }; + assert!(err.to_string().contains("catalog"), "{err}"); +} + /// The audio PTS in `frames`. fn audio_pts(frames: &[Frame]) -> Vec { let bytes: Vec = frames.iter().flat_map(|f| f.payload.iter().copied()).collect(); diff --git a/rs/moq-mux/src/container/ts/follower.rs b/rs/moq-mux/src/container/ts/follower.rs index 2f28b13116d..21a2592a30c 100644 --- a/rs/moq-mux/src/container/ts/follower.rs +++ b/rs/moq-mux/src/container/ts/follower.rs @@ -35,12 +35,25 @@ enum Serving { Other(Option), } +/// A first catalog snapshot the export awaits, since a subscribed catalog that never answers is +/// no return or switch. +struct Awaiting { + /// When the linger runs out. + deadline: Instant, + missing: Missing, +} + +/// What a catalog still missing at the deadline means. +enum Missing { + /// A return that never resumed: the export ends as it did before it. + Return(crate::Result<()>), + /// A replacement switched to mid-stream that never served its catalog: a failure. + Switch, +} + enum State { - /// Exporting. A return carries how the export ended and when the linger runs out until its - /// catalog delivers a snapshot, since a subscribed catalog that never answers is no return. - Running { - settling: Option<(crate::Result<()>, Instant)>, - }, + /// Exporting, until the deadline when its catalog is yet to deliver a snapshot. + Running { awaiting: Option }, /// The export ended with `end`, waiting for the announcements to say what follows. Settling { end: crate::Result<()>, @@ -52,10 +65,10 @@ enum State { /// Resolving the instance serving the path now, to carry the export on into it. Following { resolving: Resolving, - /// How the export ended and when the linger runs out, as in `Running` or `Settling`. - settling: Option<(crate::Result<()>, Instant)>, - /// The export is still running, a stitch mid-stream, so it carries on if the switch does not. - live: bool, + /// Bounds the resolution and then the catalog's first snapshot. + awaiting: Awaiting, + /// Where the export carries on, bound and all, if the instance goes before it resolves. + fallback: Box>, }, /// Nothing follows. Done, @@ -101,7 +114,7 @@ impl Follower { let epoch = export.instance().cloned(); let mut follower = Self { export, - state: State::Running { settling: None }, + state: State::Running { awaiting: None }, announced, path, epoch, @@ -128,14 +141,16 @@ impl Follower { /// Wait up to `linger` for the same instance to return once the broadcast ends. Defaults to /// [`Duration::ZERO`]. /// - /// It bounds the whole return, up to the returned catalog's first snapshot. + /// It bounds the whole return, and a mid-stream stitch from its restart to the + /// replacement catalog's first snapshot. Zero requires an already available catalog. pub fn with_linger(mut self, linger: Duration) -> Self { self.linger = linger; self } /// Follow another publisher instance taking the path, as a full program switch (see - /// [`Export::follow`]). Defaults to `false`. + /// [`Export::follow`]). Defaults to `false`. Its first catalog must arrive within + /// [`with_linger`](Self::with_linger), including when the old instance is still live. pub fn with_stitch(mut self, stitch: bool) -> Self { self.stitch = stitch; self @@ -161,10 +176,14 @@ impl Follower { } fn poll_state(&mut self, waiter: &kio::Waiter) -> Poll>> { + if let Some(expired) = self.expired() { + return Poll::Ready(expired); + } + loop { match std::mem::replace(&mut self.state, State::Done) { State::Done => return Poll::Ready(Ok(None)), - State::Running { mut settling } => { + State::Running { mut awaiting } => { self.poll_announced(waiter); // Subscriptions are sticky, so a replacement leaves the export on the old // instance until it ends, unless asked to switch at once. @@ -172,30 +191,38 @@ impl Follower { tracing::info!(path = %self.path, "broadcast replaced, switching the program to the new instance"); self.state = State::Following { resolving: self.resolve(), - settling, - live: true, + awaiting: Awaiting { + deadline: Instant::now() + self.linger, + missing: Missing::Switch, + }, + fallback: Box::new(State::Running { awaiting }), }; continue; } let next = self.export.poll_next(waiter); if self.export.cataloged() { - settling = None; + awaiting = None; } let end = match next { Poll::Ready(Ok(Some(frame))) => { - self.state = State::Running { settling }; + self.state = State::Running { awaiting }; return Poll::Ready(Ok(Some(frame))); } - Poll::Ready(Ok(None)) => Ok(()), + // A switch whose catalog closed before its first snapshot never served one. + Poll::Ready(Ok(None)) => match awaiting { + Some(Awaiting { + missing: Missing::Switch, + .. + }) => Err(self.missing_catalog()), + _ => Ok(()), + }, Poll::Ready(Err(err)) => Err(err), Poll::Pending => { - if let Some((end, deadline)) = &settling - && self.elapsed(waiter, *deadline) + if let Some(awaiting) = awaiting.take_if(|awaiting| self.elapsed(waiter, awaiting.deadline)) { - tracing::info!(path = %self.path, linger = ?self.linger, "broadcast did not return"); - return Poll::Ready(end.clone().map(|()| None)); + return Poll::Ready(self.expire(awaiting)); } - self.state = State::Running { settling }; + self.state = State::Running { awaiting }; return Poll::Pending; } }; @@ -243,8 +270,15 @@ impl Follower { if follow { self.state = State::Following { resolving: self.resolve(), - settling: Some((end, deadline)), - live: false, + fallback: Box::new(State::Settling { + end: end.clone(), + grace: None, + deadline, + }), + awaiting: Awaiting { + deadline, + missing: Missing::Return(end), + }, }; continue; } @@ -257,8 +291,8 @@ impl Follower { } State::Following { mut resolving, - settling, - live, + awaiting, + fallback, } => { // Announcements wait until the export is on the instance they describe. if let Poll::Ready(resolved) = waiter.poll_future(resolving.as_mut()) { @@ -266,8 +300,9 @@ impl Follower { Ok(resolved) => resolved, // An instance that went before it resolved is no return or switch yet. // Until the next announcement says otherwise, nothing serves the path: - // a stitch stays on the export it has, and a return keeps lingering. - // A refusal from an instance still up fails loud instead. + // a stitch stays on the export it has, still awaiting what it was, and a + // return keeps lingering. A refusal from an instance still up fails loud + // instead. Err(err) => { if !matches!( err, @@ -277,14 +312,11 @@ impl Follower { } tracing::warn!(path = %self.path, %err, "broadcast went before it resolved, still waiting"); self.serving = Serving::Gone; - self.state = match (live, settling) { - (false, Some((end, deadline))) => State::Settling { - end, - grace: None, - deadline, - }, - (_, settling) => State::Running { settling }, - }; + self.state = *fallback; + // The fallback's wait predates this poll, so it gets no same-poll grace. + if let Some(expired) = self.expired() { + return Poll::Ready(expired); + } continue; } }; @@ -297,19 +329,18 @@ impl Follower { epoch = self.epoch.as_ref().map(tracing::field::display), "exporting broadcast" ); - self.state = State::Running { settling }; + self.state = State::Running { + awaiting: Some(awaiting), + }; continue; } - if let Some((end, deadline)) = &settling - && self.elapsed(waiter, *deadline) - { - tracing::info!(path = %self.path, linger = ?self.linger, "broadcast did not return"); - return Poll::Ready(end.clone().map(|()| None)); + if self.elapsed(waiter, awaiting.deadline) { + return Poll::Ready(self.expire(awaiting)); } self.state = State::Following { resolving, - settling, - live, + awaiting, + fallback, }; return Poll::Pending; } @@ -388,6 +419,49 @@ impl Follower { } } + /// End a catalog wait already past its deadline, before a late snapshot can count. + /// + /// Only waits from an earlier poll are checked: within one poll a ready catalog still makes + /// it, which zero linger relies on. + fn expired(&mut self) -> Option>> { + let awaiting = match &self.state { + State::Running { awaiting } => awaiting.as_ref(), + State::Following { awaiting, .. } => Some(awaiting), + State::Settling { .. } | State::Done => None, + }; + if !awaiting.is_some_and(|awaiting| Instant::now() >= awaiting.deadline) { + return None; + } + let (State::Running { + awaiting: Some(awaiting), + } + | State::Following { awaiting, .. }) = std::mem::replace(&mut self.state, State::Done) + else { + unreachable!("only a running or following state awaits a catalog"); + }; + Some(self.expire(awaiting)) + } + + /// How a wait for the catalog's first snapshot ends once the linger runs out. + fn expire(&self, awaiting: Awaiting) -> crate::Result> { + match awaiting.missing { + Missing::Return(end) => { + tracing::info!(path = %self.path, linger = ?self.linger, "broadcast did not return"); + end.map(|()| None) + } + Missing::Switch => Err(self.missing_catalog()), + } + } + + fn missing_catalog(&self) -> crate::Error { + anyhow::anyhow!( + "replacement broadcast `{}` did not serve its catalog within {:?}", + self.path, + self.linger + ) + .into() + } + /// Whether `at` has come, registering `waiter` to wake then if not. fn elapsed(&mut self, waiter: &kio::Waiter, at: Instant) -> bool { if Instant::now() >= at { diff --git a/rs/moq-srt/src/dial.rs b/rs/moq-srt/src/dial.rs index 32f65204958..7162daa7c5d 100644 --- a/rs/moq-srt/src/dial.rs +++ b/rs/moq-srt/src/dial.rs @@ -114,6 +114,7 @@ impl Client { /// Wait up to `linger` for the same publisher instance to come back once the broadcast /// [`publish`](Self::publish) reads ends, carrying on with the same stream. Zero (the /// default) ends the push at the broadcast's end. + /// A stitched replacement must also supply its first catalog within this bound. pub fn with_linger(mut self, linger: Duration) -> Self { self.linger = linger; self diff --git a/rs/moq-srt/src/listen.rs b/rs/moq-srt/src/listen.rs index 6630c881b79..5eb55d35cde 100644 --- a/rs/moq-srt/src/listen.rs +++ b/rs/moq-srt/src/listen.rs @@ -73,6 +73,7 @@ pub struct Config { /// How long an egress waits for the same publisher instance to come back once its /// broadcast ends, carrying on with the same stream. Zero (the default) closes the SRT /// stream at the end. Only affects egress (`m=request`). + /// A stitched replacement must also supply its first catalog within this bound. pub linger: Duration, /// Whether an egress follows another publisher instance replacing its broadcast, as a full diff --git a/rs/moq-srt/src/server.rs b/rs/moq-srt/src/server.rs index 08de535efff..9757e69645c 100644 --- a/rs/moq-srt/src/server.rs +++ b/rs/moq-srt/src/server.rs @@ -454,6 +454,7 @@ impl Subscribe { /// Wait up to `linger` for the same publisher instance to come back once the broadcast /// ends, carrying on with the same stream. Zero (the default) closes the SRT stream at /// the broadcast's end. + /// A stitched replacement must also supply its first catalog within this bound. pub fn with_linger(mut self, linger: Duration) -> Self { self.0.linger = linger; self @@ -1129,7 +1130,7 @@ mod tests { let (origin, driver) = moq_net::origin::Producer::new(moq_net::origin::Config::default()); tokio::spawn(moq_net::time::run(driver)); let mut old = Instance::publish(&origin, "old.aac"); - let (mut receiver, egress) = egress(&origin, Duration::ZERO, true).await; + let (mut receiver, egress) = egress(&origin, Duration::from_secs(10), true).await; old.write(30, 600_000_000); let first = tokio::time::timeout(Duration::from_secs(10), receiver.next()) diff --git a/rs/moq-srt/src/ts.rs b/rs/moq-srt/src/ts.rs index 4017b10bd35..8a2d36cb41e 100644 --- a/rs/moq-srt/src/ts.rs +++ b/rs/moq-srt/src/ts.rs @@ -165,7 +165,7 @@ impl Publisher { pub(crate) struct Options { /// The muxer's jitter-buffer delay ([`Subscriber::new`]). pub latency: Duration, - /// How long to wait for the same publisher instance to come back once its broadcast ends. + /// How long a return or stitched replacement has to supply its first catalog. pub linger: Duration, /// Follow another instance replacing the broadcast, as a full program switch. pub stitch: bool,