From 86972e223a5cf10ec4e1aa80aeb8fff7b7b3f424 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 07:39:34 -0700 Subject: [PATCH 1/4] chore(quest): claim quest/m1/broadcast-lifecycle/retention.md Quest-Claim: 2e0d556717542da9 From 75b18c5d5cbe81c2b3a0d367ea14520ff04b1c38 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 08:22:56 -0700 Subject: [PATCH 2/4] feat(mux)!: keep a finished broadcast up for a fixed linger Add moq_mux::Linger, which holds a finished broadcast's owners and closes its broadcasts a fixed time after the end. moq-cli imports, the RTMP, SRT, WHIP and HLS gateways, and moqsink use it instead of moq-net's broadcast::Finish, with --linger (default 30s) on moq import and a linger property on moqsink. Co-Authored-By: Claude Opus 5.5 --- doc/bin/cli.md | 13 +- doc/bin/gstreamer.md | 7 +- doc/bin/hls.md | 2 +- quest/m1/broadcast-lifecycle/README.md | 1 - quest/m1/broadcast-lifecycle/retention.md | 42 ------ quest/m1/broadcast-lifecycle/stateless.md | 4 - quest/m1/broadcast-lifecycle/ts-follow.md | 6 +- rs/moq-cli/MIGRATE.md | 6 + rs/moq-cli/src/args.rs | 41 +++++- rs/moq-cli/src/hls.rs | 15 +- rs/moq-cli/src/main.rs | 5 +- rs/moq-cli/src/moq.rs | 10 +- rs/moq-cli/src/publish.rs | 114 +++++++++++----- rs/moq-cli/src/rtc.rs | 1 + rs/moq-cli/src/rtmp.rs | 5 +- rs/moq-cli/src/rtsp.rs | 3 + rs/moq-cli/src/srt.rs | 11 +- rs/moq-cli/tests/integration/export.rs | 12 +- rs/moq-cli/tests/integration/import.rs | 6 +- rs/moq-cli/tests/integration/report.rs | 6 +- rs/moq-gst/src/sink/imp.rs | 159 ++++++++++++++++++++-- rs/moq-gst/tests/element.rs | 27 +++- rs/moq-hls/MIGRATE.md | 10 ++ rs/moq-hls/src/import.rs | 24 ++-- rs/moq-mux/MIGRATE.md | 10 ++ rs/moq-mux/src/container/ts/programs.rs | 23 ++-- rs/moq-mux/src/lib.rs | 4 + rs/moq-mux/src/linger.rs | 148 ++++++++++++++++++++ rs/moq-rtc/src/ingest.rs | 19 ++- rs/moq-rtc/src/lib.rs | 19 +-- rs/moq-rtc/src/server/mod.rs | 19 ++- rs/moq-rtc/src/session.rs | 16 ++- rs/moq-rtmp/src/dial.rs | 61 +++++---- rs/moq-rtmp/src/server.rs | 67 +++++---- rs/moq-srt/MIGRATE.md | 9 ++ rs/moq-srt/src/dial.rs | 48 +++++-- rs/moq-srt/src/server.rs | 94 ++++++++----- rs/moq-srt/src/ts.rs | 48 +++---- 38 files changed, 782 insertions(+), 333 deletions(-) delete mode 100644 quest/m1/broadcast-lifecycle/retention.md create mode 100644 rs/moq-mux/src/linger.rs diff --git a/doc/bin/cli.md b/doc/bin/cli.md index bbb84ae55fd..235a015c52d 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -415,13 +415,18 @@ fetchable, which the [HLS gateway](/bin/hls) depends on. `export fmp4`, `export and `export h265` take `--max-delay` (default 500 ms): how far a stalled group may fall behind the live edge before *this* consumer skips it. Raising `--max-age` never delays playback. -At stdin EOF, `import` keeps the finished broadcast up for its max-age, so a -subscriber still catching up can drain it; Ctrl-C ends that early. A gateway +At stdin EOF, `import` keeps the finished broadcast announced for +`--linger` (default 30 s) from the end, so a subscriber that arrives late, such +as a player that read the final catalog after the end, still gets the tail; +`--linger 0s` closes it at once, and Ctrl-C ends the wait early. A gateway import does the same at a clean end: an RTMP unpublish (`deleteStream`) or completed play, the end of an SRT stream, a WHIP `DELETE`, or an HLS playlist ending with `EXT-X-ENDLIST`. A dropped connection or an error closes the -broadcast at once, and a reconnect, which publishes a newer -[epoch](/concept/moq-lite#publisher-epochs) at the same path, ends the wait. +broadcast at once. A reconnect publishes a newer +[epoch](/concept/moq-lite#publisher-epochs) at the same path, which takes over +the path while the old broadcast finishes its linger. `import rtsp` redials +after every session, so it closes each run at once and refuses `--linger`, as +do `import archive`, `import capture`, and a WHEP pull (`import rtc --connect`). `export ts`, `export flv`, `export rtmp`, and `export mkv` take `--delay` instead, like an SRT receiver's latency, all tracks in decode order, so two exporters of one broadcast diff --git a/doc/bin/gstreamer.md b/doc/bin/gstreamer.md index 7043ae249e0..2489e17118e 100644 --- a/doc/bin/gstreamer.md +++ b/doc/bin/gstreamer.md @@ -64,7 +64,7 @@ Each `sink_%u` request pad is one track. Pad properties: `track` names it (default: after the codec), `container=loc` publishes it as [LOC](/concept/standard#loc) instead of the legacy hang container, and `track-status`/`track-error` report its lifecycle. Element properties: -`url`, `broadcast`, `connect-version`, `tls-disable-verify`, `quic-idle-timeout`, `quic-keylog`, +`url`, `broadcast`, `linger`, `connect-version`, `tls-disable-verify`, `quic-idle-timeout`, `quic-keylog`, `quic-keep-alive`, and read-only `status`, `connected`, `moq-version`, and `estimated-send-rate`, `estimated-recv-rate`, `connection-stats`, and `sessions`. The sink reconnects for as long as the pipeline runs and only @@ -84,7 +84,10 @@ it is released, reaches EOS, or fails; a flush or new stream after that EOS holds it back again. Opaque data pads are not listed. Later catalog updates do not wait for newly requested pads. -Once every pad reaches EOS, the sink closes the broadcast and posts EOS. +Once every pad reaches EOS, the sink keeps the finished broadcast announced +for `linger` milliseconds (default 30000), so a subscriber that arrives late +still gets the tail, then closes it and posts EOS. `linger=0` closes it at once, +and stopping the pipeline (`READY` or `NULL`) during the wait closes it early. All pads share one media timeline on one wall clock. The first timed buffer places it using UNIX or NTP reference timestamp metadata; other clock domains use the diff --git a/doc/bin/hls.md b/doc/bin/hls.md index fc2a6d729b3..bb23154cc15 100644 --- a/doc/bin/hls.md +++ b/doc/bin/hls.md @@ -22,7 +22,7 @@ The CLI serves the one `--broadcast` it was given, at Import follows a live playlist, joining a few segments behind its live edge, until every rendition's playlist ends with `EXT-X-ENDLIST`. It then keeps the -finished broadcast up until its media is stale +finished broadcast up for `--linger` (see [retention](/bin/cli#retention-and-latency)). A playlist that has already ended, such as a VOD recording, is refused with an error rather than imported. An `EXT-X-DISCONTINUITY` keeps the source's timestamps. Timestamp resets are not diff --git a/quest/m1/broadcast-lifecycle/README.md b/quest/m1/broadcast-lifecycle/README.md index 6179c67f015..04554384a88 100644 --- a/quest/m1/broadcast-lifecycle/README.md +++ b/quest/m1/broadcast-lifecycle/README.md @@ -46,7 +46,6 @@ timer on main at 1582fdc07d: ## Required -- [Publisher retention in moq-mux](/quest/m1/broadcast-lifecycle/retention.md) - moq-cli, the gateways, and moqsink keep a finished broadcast up for a fixed `--linger` (default 30s) through a moq-mux helper instead of moq-net's `Finish` - [Stateless broadcasts](/quest/m1/broadcast-lifecycle/stateless.md) - `Finish`, its linger, and finished-only supersession are gone from moq-net, `@moq/net`, moq-ffi, moq-c, and every binding - [TS export follows by cause](/quest/m1/broadcast-lifecycle/ts-follow.md) - `export ts|srt --linger` becomes `--follow `, a replacement's catalog has no separate deadline, and a track failure is classified by its cause instead of a 1s grace - [SRT tells a clean end from a failure](/quest/m1/broadcast-lifecycle/srt-end.md) - an SRT ingest whose peer goes silent aborts instead of finishing as if it had shut down diff --git a/quest/m1/broadcast-lifecycle/retention.md b/quest/m1/broadcast-lifecycle/retention.md deleted file mode 100644 index 50d12d7184f..00000000000 --- a/quest/m1/broadcast-lifecycle/retention.md +++ /dev/null @@ -1,42 +0,0 @@ -# [M] Publisher retention in moq-mux - -## Goal - -moq-cli imports (stdin, HLS), the RTMP, SRT, RTC, and HLS gateways, and -moqsink keep a cleanly ended broadcast announced and serving its finished -tracks for a fixed duration counted from the end, then close it. They no -longer use moq-net's `broadcast::Finish`. The duration is `--linger -` on moq-cli's publishing subcommands and a `linger` duration -property on moqsink, defaulting to 30s; 0 closes at once. An interrupt -(Ctrl-C, moqsink NULL) ends it at once. - -## Plan - -Decided with the maintainer on 2026-10-23: - -- One small moq-mux helper owns it, shared by moq-cli, `moq-rtmp`, - `moq-srt`, `moq-rtc`, and moqsink. It finishes the catalog and media - tracks, holds the broadcast and track producers, and closes the broadcast - when the deadline passes. moq-net gains nothing. -- The deadline is fixed when the content ends and does not depend on track - max-age or on when anything is polled. 30s matches today's effective - behavior (media max-age) and the relay's idle-copy window. -- It ignores a newer instance announced at the path; see the epic. -- `` and the FFI/C bindings do not use it: they finish their - catalog and close, and a binding app owns its own process lifetime. -- moq-cli RTSP closes at once today (`rs/moq-cli/src/rtsp.rs`); give it the - helper too, or say why not. -- Keep each gateway's existing split between a clean end and a failure - (RTMP unpublish vs disconnect, WHIP DELETE vs a vanished peer); only a - clean end retains. SRT's split is - [its own quest](/quest/m1/broadcast-lifecycle/srt-end.md). -- Test the helper with mocked time: a late subscriber gets the whole tail - before the deadline, and the path is unroutable after it. -- Docs: `doc/bin/cli.md`, `doc/bin/gstreamer.md`, the gateway pages under - `doc/bin/`. - -Public API: moq-mux gains the helper. moq-cli gains `--linger` on publishing -subcommands; `moq export --linger` keeps its meaning until -[TS export follows by cause](/quest/m1/broadcast-lifecycle/ts-follow.md) -renames it. moqsink's `linger` returns as a duration (the boolean never -shipped in a release, so no migration note). Wire: none. diff --git a/quest/m1/broadcast-lifecycle/stateless.md b/quest/m1/broadcast-lifecycle/stateless.md index c1ef1eaab72..a44eaffae7d 100644 --- a/quest/m1/broadcast-lifecycle/stateless.md +++ b/quest/m1/broadcast-lifecycle/stateless.md @@ -37,7 +37,3 @@ Decided with the maintainer on 2026-10-23; see the epic for why. Public API: breaking in moq-net, `@moq/net`, moq-ffi, moq-c, and every binding. Wire: none. - -## Required - -- [Publisher retention in moq-mux](/quest/m1/broadcast-lifecycle/retention.md) - Rust publishers stop using `Finish` first, so none loses its tail in between diff --git a/quest/m1/broadcast-lifecycle/ts-follow.md b/quest/m1/broadcast-lifecycle/ts-follow.md index 7df7f4e6179..d9e2fab3ea7 100644 --- a/quest/m1/broadcast-lifecycle/ts-follow.md +++ b/quest/m1/broadcast-lifecycle/ts-follow.md @@ -5,8 +5,8 @@ `moq export ts --linger` and `moq export srt --linger` become `--follow `: how long to wait for the publisher, or with `--stitch` a replacement, to come back. The default 0 means don't follow. The Rust -`container::ts::Follower::with_linger` and moq-srt's `Subscribe::with_linger` -and `Client::with_linger` become `with_follow`. A replacement's first catalog +`container::ts::Follower::with_linger` becomes `with_follow`, matching +moq-srt's `Subscribe::with_follow` and `Client::with_follow`. A replacement's first catalog is awaited until it arrives or the replacement withdraws, with no deadline of its own. A track failure is classified by its cause, not by waiting `CLOSE_GRACE` (1s) to see whether the announcement also goes. @@ -36,5 +36,5 @@ Decided with the maintainer on 2026-10-23: - Docs: `doc/bin/cli.md`, plus a migration note for moq-cli, moq-mux, and moq-srt in the Impact bullet. -Public API: moq-mux and moq-srt renames (breaking); moq-cli flag rename. +Public API: moq-mux rename (breaking); moq-cli flag rename. Wire: none unless the failure cause needs a new code. diff --git a/rs/moq-cli/MIGRATE.md b/rs/moq-cli/MIGRATE.md index f015f1cb53b..334d6ad5ca8 100644 --- a/rs/moq-cli/MIGRATE.md +++ b/rs/moq-cli/MIGRATE.md @@ -4,6 +4,12 @@ How to update your invocation and config for each breaking change, newest first. ## Unreleased +### `import --linger` sets how long a finished broadcast stays up + +A cleanly ended import used to stay up for its `--max-age`; it now stays up for `--linger` (default 30 s), counted from the end, and `--max-age` only sets retention. +Replace `import --max-age 5s` used to shorten that wait with `import --linger 5s`, or `--linger 0s` to close at once. +A reconnect at the same path no longer ends the wait. `import rtsp`, `import archive`, `import capture`, and `import rtc --connect` refuse `--linger`. + ### The `nvenc` and `nvdec` Cargo features are removed ([#6483](https://github.com/moq-dev/moq/pull/6483)) Enable `nvidia` instead; the old names were aliases for it. diff --git a/rs/moq-cli/src/args.rs b/rs/moq-cli/src/args.rs index 5cbc04a7324..d9ee240b7aa 100644 --- a/rs/moq-cli/src/args.rs +++ b/rs/moq-cli/src/args.rs @@ -318,6 +318,12 @@ impl Invocation { { anyhow::bail!("--report names a track in the catalog the import builds, but {reason}"); } + if let Command::Import(import) = command + && import.linger.is_some() + && let Some(reason) = import.source.linger_refusal() + { + anyhow::bail!("--linger keeps a cleanly ended broadcast up, but {reason}"); + } let Command::Export(export) = command else { continue; }; @@ -745,11 +751,20 @@ pub struct Import { /// wait for a late group). The default suits a segmented egress (HLS/DASH), which may only /// advertise segments that are still fetchable; lower it when nothing reads history and the /// memory matters. Media tracks only -- the catalog and timeline are read at the live edge, - /// which is retained unconditionally. A stdin import also stays up this long after EOF, so - /// a subscriber still catching up can drain the finished broadcast. + /// which is retained unconditionally. #[usage(long)] pub max_age: Option, + /// How long a cleanly ended broadcast stays up, counted from its end, e.g. "30s"; "0s" + /// closes it at once. Defaults to 30s. + /// + /// Subscribers already reading get the tail either way; this serves one that arrives after + /// the end, such as a player that read the final catalog late. A clean end is stdin EOF, an + /// RTMP unpublish or completed play, the end of an SRT stream, a WHIP `DELETE`, or an HLS + /// playlist's `EXT-X-ENDLIST`. A failure, and Ctrl-C, close the broadcast at once. + #[usage(long)] + pub linger: Option, + /// The released spelling of [`Self::max_age`]. #[usage(long = "latency-max", hide = true)] latency_max: Option, @@ -841,6 +856,28 @@ impl ImportSource { Self::Avc3 | Self::Fmp4(_) | Self::Ts(_) | Self::Flv => None, } } + + /// Why `--linger` can't apply to this source, or `None` when it can: only a source with a + /// clean end has a finished broadcast to keep up. + fn linger_refusal(&self) -> Option<&'static str> { + match self { + // Every run is followed by a redial publishing the next one, so a run's end is not + // the end of the content, and its finished catalog would tell readers to stop. + Self::Rtsp(_) => Some("`import rtsp` redials after every session, so a run's end is not the content's"), + Self::Rtc(rtc) if rtc.connect.is_some() => Some("a WHEP pull has no clean end"), + Self::Archive(_) => Some("`import archive` serves its recording until stopped"), + #[cfg(feature = "capture")] + Self::Capture(_) => Some("`import capture` has no clean end"), + Self::Avc3 + | Self::Fmp4(_) + | Self::Ts(_) + | Self::Flv + | Self::Hls(_) + | Self::Rtmp(_) + | Self::Srt(_) + | Self::Rtc(_) => None, + } + } } /// The fMP4 stdin container: whether to unwrap its fragments. diff --git a/rs/moq-cli/src/hls.rs b/rs/moq-cli/src/hls.rs index 4690c27fc43..bec2b7f792f 100644 --- a/rs/moq-cli/src/hls.rs +++ b/rs/moq-cli/src/hls.rs @@ -150,13 +150,13 @@ impl Tls { } /// Pull a remote HLS/LL-HLS playlist (URL or file path) into the Origin under `target.name`, -/// announced under `epoch`. Once the playlist ends, serve the finished broadcast until its -/// media is stale. +/// announced under `epoch`. Once the playlist ends, serve the finished broadcast for the linger. pub async fn import(target: ImportTarget, args: ImportArgs, epoch: moq_net::Epoch) -> anyhow::Result<()> { let ImportTarget { origin, name, max_age, + linger, bandwidth, } = target; let mut producer = origin.create_broadcast(&name).context("failed to create broadcast")?; @@ -189,11 +189,11 @@ pub async fn import(target: ImportTarget, args: ImportArgs, epoch: moq_net::Epoc return Err(err); } - // The playlist ended: serve the finished broadcast until its media is stale, so - // subscribers still catching up drain the tail. An interrupt drops it at once. - let finish = importer.finish()?; - tracing::info!(linger = ?finish.linger(), "playlist ended; serving the finished broadcast until its media is stale"); - finish.wait().await; + // The playlist ended: serve the finished broadcast for the linger, so a subscriber that + // arrives after the end still gets the tail. An interrupt drops it at once. + importer.finish()?; + tracing::info!(?linger, "playlist ended; serving the finished broadcast"); + moq_mux::Linger::new([producer], importer, linger).wait().await; Ok(()) } @@ -424,6 +424,7 @@ mod test { origin, name: "hls".to_string(), max_age: None, + linger: moq_mux::Linger::DEFAULT, bandwidth: Default::default(), }; let args = ImportArgs { diff --git a/rs/moq-cli/src/main.rs b/rs/moq-cli/src/main.rs index f62a8158034..46763680dbf 100644 --- a/rs/moq-cli/src/main.rs +++ b/rs/moq-cli/src/main.rs @@ -707,12 +707,14 @@ fn spawn_import( } let max_age = import.max_age.map(Into::into); + let linger = import.linger.map_or(moq_mux::Linger::DEFAULT, Into::into); // The MoQ side every gateway publishes into, minted per source since each takes it // by value onto its own task. let target = |name: String| crate::moq::ImportTarget { origin: origin.clone(), name, max_age, + linger, bandwidth: bandwidth.clone(), }; @@ -733,7 +735,8 @@ fn spawn_import( } else { let broadcast = origin.create_broadcast(&name).context("failed to create broadcast")?; Publish::new(broadcast, &format, config)? - }; + } + .with_linger(linger); publish.announce(epoch)?; local = Some(publish); } else { diff --git a/rs/moq-cli/src/moq.rs b/rs/moq-cli/src/moq.rs index 7c07e61e28b..4d9ec745f2b 100644 --- a/rs/moq-cli/src/moq.rs +++ b/rs/moq-cli/src/moq.rs @@ -7,11 +7,10 @@ use hang::moq_net; /// The MoQ side of an import: where a gateway publishes, under what name, the -/// retention it declares on the media tracks it mints, and the connection -/// allocator those tracks claim on. +/// retention it declares on the media tracks it mints, how long a clean end stays +/// up, and the connection allocator those tracks claim on. /// -/// Every gateway needs all four regardless of protocol, so they travel together -/// rather than as loose arguments per entry point. +/// They travel together rather than as loose arguments per entry point. #[derive(Clone)] pub struct ImportTarget { /// The Origin the gateway publishes into. @@ -24,6 +23,9 @@ pub struct ImportTarget { /// (`--max-age`), or `None` for the publisher's own default. pub max_age: Option, + /// How long a cleanly ended broadcast stays up (`--linger`). + pub linger: std::time::Duration, + /// Connection allocator each passthrough track claims its peak-hold bitrate on, /// so a co-resident encoder targets what is left. Unlimited when there is no /// uplink estimate. diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index 865f18b44f8..a0153f925ab 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -185,9 +185,9 @@ impl PublishDecoder { /// The avc3 split holds the final access unit until the next start code, so /// stdin EOF must flush it explicitly. /// - /// `import ts --program all` owns its broadcasts, so it returns their - /// [`Finish`](moq_net::broadcast::Finish) handles; see [`ts::Programs::finish`]. - fn finish(&mut self) -> anyhow::Result> { + /// `import ts --program all` owns its broadcasts, so it returns them; see + /// [`ts::Programs::finish`]. + fn finish(&mut self) -> anyhow::Result> { match self { Self::Avc3 { split, import } => { let tail = split.flush(None)?; @@ -311,6 +311,8 @@ pub struct Publish { route: Option, // The `--report` track, written beside the decode loop; see [`Self::with_report`]. report: Option, + // How long a stdin import keeps its broadcast up after EOF; see [`Self::with_linger`]. + linger: std::time::Duration, } /// A broadcast's announcement, held until its catalog first goes out. @@ -372,6 +374,7 @@ impl Publish { broadcast: Some(broadcast), route: None, report: None, + linger: moq_mux::Linger::DEFAULT, }); } @@ -419,6 +422,7 @@ impl Publish { broadcast: Some(broadcast), route: None, report: None, + linger: moq_mux::Linger::DEFAULT, }) } @@ -437,6 +441,7 @@ impl Publish { broadcast: None, route: None, report: None, + linger: moq_mux::Linger::DEFAULT, } } @@ -477,6 +482,7 @@ impl Publish { broadcast: Some(broadcast), route: None, report: None, + linger: moq_mux::Linger::DEFAULT, }) } @@ -498,6 +504,13 @@ impl Publish { Ok(self) } + /// Keep a stdin import's broadcast up for `linger` after EOF, so a subscriber that read the + /// final catalog late still gets the tail. Zero closes it at EOF. + pub fn with_linger(mut self, linger: std::time::Duration) -> Self { + self.linger = linger; + self + } + /// Advertise the broadcast's path under `epoch`. /// /// A capture announces now. A stdin import announces from [`run`](Self::run), once the input @@ -521,15 +534,13 @@ impl Publish { (Some(broadcast), Some(route)) => Some(Pending::new(broadcast.clone(), route)?), _ => None, }; - decode( - decoder, - catalog, - tokio::io::stdin(), - self.broadcast, + let publication = Publication { + broadcast: self.broadcast, pending, - self.report, - ) - .await + report: self.report, + linger: self.linger, + }; + decode(decoder, catalog, tokio::io::stdin(), publication).await } #[cfg(feature = "capture")] Source::Capture { @@ -583,24 +594,39 @@ impl Publish { } } +/// What a stdin import does around its media: announce, report, and retain the end. +#[derive(Default)] +struct Publication { + /// The broadcast to keep up after EOF, or `None` when the decoder owns its broadcasts. + broadcast: Option, + /// The announcement held until the catalog first goes out. + pending: Option, + /// `--report`, written beside the read loop, so a stalled input keeps reporting, and ended + /// with its final counts when the media tracks are. + report: Option, + /// How long the finished broadcasts stay up after EOF. + linger: std::time::Duration, +} + /// Decode `input` into the broadcast until EOF. /// /// At EOF the media tracks finish, then the catalog does, while it still lists them: the /// renditions retire from the catalog as the decoder drops, which a subscriber would read /// as removed tracks, and a catalog dropped unfinished reads as a publisher that vanished. -/// The finished `broadcast` then stays servable until its media is stale, so a subscriber -/// that saw the final catalog only at EOF can still request it. -/// -/// `report` writes `--report` beside the read loop, so a stalled input keeps reporting, and ends -/// with its final counts when the media tracks do. +/// The finished broadcasts then stay servable for the linger, so a subscriber that saw the +/// final catalog only at EOF can still request them. async fn decode( mut decoder: PublishDecoder, mut catalog: PublishCatalog, mut input: impl tokio::io::AsyncRead + Unpin, - broadcast: Option, - mut pending: Option, - mut report: Option, + publication: Publication, ) -> anyhow::Result<()> { + let Publication { + broadcast, + mut pending, + mut report, + linger, + } = publication; // An empty BytesMut grows to only 64 bytes and clearing it keeps that capacity. // Stdin dispatches every read to a blocking thread, so those tiny reads can // backpressure a live feed. Reserve a full IO chunk; read_buf still returns @@ -679,13 +705,12 @@ async fn decode( report.finish(); } catalog.finish()?; - finished.extend(broadcast.map(moq_net::broadcast::Producer::finish)); - drop((decoder, catalog, report)); - if !finished.is_empty() { - tracing::info!("input ended; serving the finished broadcast until its media is stale"); - // Polled together, so each broadcast closes at its own expiry. - futures::future::join_all(finished.into_iter().map(moq_net::broadcast::Finish::wait)).await; - } + finished.extend(broadcast); + tracing::info!(?linger, "input ended; serving the finished broadcast"); + // The decoder and catalog hold the finished tracks, so they go only once it closes. + moq_mux::Linger::new(finished, (decoder, catalog, report), linger) + .wait() + .await; Ok(()) } Err(err) => { @@ -1489,7 +1514,9 @@ pub(crate) mod tests { remaining: BBB, reads: 0, }; - decode(decoder, catalog, &mut input, None, None, None).await.unwrap(); + decode(decoder, catalog, &mut input, Publication::default()) + .await + .unwrap(); assert_eq!( input.reads, BBB.len().div_ceil(188) + 1, @@ -1528,7 +1555,7 @@ pub(crate) mod tests { panic!("expected a stream source"); }; let start = std::time::Instant::now(); - decode(decoder, catalog, input, None, None, None).await.unwrap(); + decode(decoder, catalog, input, Publication::default()).await.unwrap(); let elapsed = start.elapsed(); let bytes = INPUT.len() * copies; println!( @@ -1557,7 +1584,7 @@ pub(crate) mod tests { let Source::Stream { decoder, catalog } = publish.source else { panic!("expected a stream source"); }; - decode(decoder, catalog, BBB, None, None, None).await.unwrap(); + decode(decoder, catalog, BBB, Publication::default()).await.unwrap(); drop(publish.broadcast); let mut last = None; @@ -1575,11 +1602,11 @@ pub(crate) mod tests { assert_eq!(last.audio.renditions.len(), 1, "the audio rendition is still listed"); } - /// After EOF the finished tracks stay servable for their max-age, so a subscriber that reads + /// After EOF the finished tracks stay servable for the linger, so a subscriber that reads /// the final catalog only then still gets the media, and the import returns once it passes. #[tokio::test(start_paused = true)] async fn a_finished_import_serves_its_tracks_for_the_linger() { - const LINGER: Duration = hang::container::MAX_AGE; + const LINGER: Duration = Duration::from_secs(30); let broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); let publish = Publish::new(broadcast, &PublishFormat::Ts { program: None }, Default::default()).unwrap(); @@ -1596,7 +1623,12 @@ pub(crate) mod tests { panic!("expected a stream source"); }; let started = tokio::time::Instant::now(); - let mut import = std::pin::pin!(decode(decoder, catalog, BBB, publish.broadcast.clone(), None, None)); + let publication = Publication { + broadcast: publish.broadcast.clone(), + linger: LINGER, + ..Default::default() + }; + let mut import = std::pin::pin!(decode(decoder, catalog, BBB, publication)); let name = tokio::select! { res = &mut import => panic!("the import returned before its linger: {res:?}"), name = async { @@ -1657,7 +1689,11 @@ pub(crate) mod tests { let route = moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()); let pending = Pending::new(broadcast, route).unwrap(); let (mut stdin, input_end) = tokio::io::duplex(input.len()); - let mut import = std::pin::pin!(decode(decoder, catalog, input_end, None, Some(pending), None)); + let publication = Publication { + pending: Some(pending), + ..Default::default() + }; + let mut import = std::pin::pin!(decode(decoder, catalog, input_end, publication)); tokio::select! { res = &mut import => panic!("the import returned before its input ended: {res:?}"), @@ -1727,7 +1763,9 @@ pub(crate) mod tests { let Source::Stream { decoder, catalog } = publish.source else { panic!("stream source"); }; - let err = decode(decoder, catalog, Broken, None, None, None).await.unwrap_err(); + let err = decode(decoder, catalog, Broken, Publication::default()) + .await + .unwrap_err(); assert_eq!(err.to_string(), "stdin failed"); assert!( matches!(catalog_reader.recv_group().await, Err(moq_net::Error::Transport(ref reason)) if reason == "stdin failed") @@ -1763,9 +1801,11 @@ pub(crate) mod tests { panic!("expected a stream source"); }; let pending = Pending::new(broadcast, Default::default()).unwrap(); - let err = decode(decoder, catalog, &input[..], None, Some(pending), None) - .await - .unwrap_err(); + let publication = Publication { + pending: Some(pending), + ..Default::default() + }; + let err = decode(decoder, catalog, &input[..], publication).await.unwrap_err(); assert!(err.to_string().contains("declared"), "{err:#}"); let event = tokio::time::timeout(Duration::from_secs(1), announced.next()).await; assert!(event.is_err(), "announced an empty broadcast: {event:?}"); diff --git a/rs/moq-cli/src/rtc.rs b/rs/moq-cli/src/rtc.rs index 269944fefcb..f3a92f65867 100644 --- a/rs/moq-cli/src/rtc.rs +++ b/rs/moq-cli/src/rtc.rs @@ -61,6 +61,7 @@ pub async fn listen_import(target: ImportTarget, listen: Listen) -> anyhow::Resu let mut config = server_config(&listen); config.max_age = target.max_age; config.bandwidth = target.bandwidth; + config.linger = target.linger; let server = moq_rtc::Server::new(config); serve(server.publish_router(publisher), "WHIP", listen).await } diff --git a/rs/moq-cli/src/rtmp.rs b/rs/moq-cli/src/rtmp.rs index 47096634c1f..2a3dd43368f 100644 --- a/rs/moq-cli/src/rtmp.rs +++ b/rs/moq-cli/src/rtmp.rs @@ -70,6 +70,7 @@ pub async fn listen_import(target: ImportTarget, addr: SocketAddr) -> anyhow::Re origin, name, max_age, + linger, bandwidth, } = target; let mut server = Server::bind(addr).await?; @@ -86,6 +87,7 @@ pub async fn listen_import(target: ImportTarget, addr: SocketAddr) -> anyhow::Re if let Err(err) = publish .with_max_age(max_age) .with_bandwidth(bandwidth) + .with_linger(linger) .accept(&origin, &name) .await { @@ -152,7 +154,8 @@ pub async fn connect_import(target: ImportTarget, url: Url) -> anyhow::Result<() let client = Client::connect(addr, &app) .await? .with_max_age(target.max_age) - .with_bandwidth(target.bandwidth); + .with_bandwidth(target.bandwidth) + .with_linger(target.linger); Ok(client.pull(&key, &target.origin, name).await?) } diff --git a/rs/moq-cli/src/rtsp.rs b/rs/moq-cli/src/rtsp.rs index d0d9326991c..532ce8e5ef6 100644 --- a/rs/moq-cli/src/rtsp.rs +++ b/rs/moq-cli/src/rtsp.rs @@ -99,6 +99,9 @@ async fn run(target: &ImportTarget, client: &moq_rtsp::Client, published: &mut b end = client.pull(&broadcast, &catalog) => end.map_err(anyhow::Error::from), err = announce => err, }; + // Closed at once even after a clean end: the pull redials and publishes the next run under a + // newer epoch, so a run's end is not the content's, and a retained run's finished catalog + // would tell readers to stop if the next run dropped. let finished = match &end { Ok(_) => catalog.finish(), Err(err) => { diff --git a/rs/moq-cli/src/srt.rs b/rs/moq-cli/src/srt.rs index 0dade67c11e..caa20cef428 100644 --- a/rs/moq-cli/src/srt.rs +++ b/rs/moq-cli/src/srt.rs @@ -99,6 +99,7 @@ pub async fn listen_import( origin, name, max_age, + linger, bandwidth, } = target; let mut server = Server::bind(addr, latency).await?; @@ -116,6 +117,7 @@ pub async fn listen_import( .with_max_age(max_age) .with_bandwidth(bandwidth) .with_program(program) + .with_linger(linger) .accept(&origin, &name) .await { @@ -144,7 +146,7 @@ pub async fn listen_export( args: ExportArgs, ) -> anyhow::Result<()> { let mut server = Server::bind(addr, Duration::from(args.endpoint.latency)).await?; - let (linger, stitch) = (args.linger.into(), args.stitch); + let (follow, stitch) = (args.linger.into(), args.stitch); tracing::info!(%addr, %name, "SRT listening (export)"); notify_ready(); @@ -154,7 +156,7 @@ pub async fn listen_export( let origin = origin.clone(); let name = name.clone(); tokio::spawn(async move { - let subscribe = subscribe.with_linger(linger).with_stitch(stitch); + let subscribe = subscribe.with_follow(follow).with_stitch(stitch); if let Err(err) = subscribe.accept(&origin, &name).await { tracing::warn!(%name, %err, "SRT request ended with error"); } @@ -188,7 +190,8 @@ pub async fn connect_import( .with_max_delay(latency) .with_max_age(target.max_age) .with_bandwidth(target.bandwidth) - .with_program(program); + .with_program(program) + .with_linger(target.linger); client.pull(&target.origin, name).await.map_err(suggest_program) } @@ -205,7 +208,7 @@ pub async fn connect_export( let client = moq_srt::Client::new(addr, resource) .with_max_delay(args.endpoint.latency.into()) - .with_linger(args.linger.into()) + .with_follow(args.linger.into()) .with_stitch(args.stitch); Ok(client.publish(&origin, &name).await?) } diff --git a/rs/moq-cli/tests/integration/export.rs b/rs/moq-cli/tests/integration/export.rs index 96e22b0a86f..48406d71a19 100644 --- a/rs/moq-cli/tests/integration/export.rs +++ b/rs/moq-cli/tests/integration/export.rs @@ -253,8 +253,8 @@ fn import_with(relay: &Relay, global: &[&str]) -> (Child, JoinHandle /// [`import_with`] in `format`. fn import_as(relay: &Relay, global: &[&str], format: Format) -> (Child, JoinHandle) { let mut child = moq(relay, global) - // A finished import lingers for its max-age; keep that short so the publisher exits promptly. - .args(["import", "--max-age", "1s", format.name]) + // A finished import lingers; keep that short so the publisher exits promptly. + .args(["import", "--max-age", "1s", "--linger", "1s", format.name]) .stdin(Stdio::piped()) .spawn() .expect("spawn import"); @@ -363,7 +363,7 @@ async fn an_export_failure_with_the_broadcast_up_exits_without_lingering() { let export = export(&relay, "20s"); // TS cannot carry AV1, so the export fails on the catalog while the publisher stays up. - let mut publisher = moq(&relay, &["import", "--max-age", "1s", "fmp4"]) + let mut publisher = moq(&relay, &["import", "--max-age", "1s", "--linger", "1s", "fmp4"]) .stdin(Stdio::piped()) .spawn() .expect("spawn import"); @@ -627,7 +627,7 @@ async fn a_standby_before_its_tracks(global: &[&str], format: Format) { second.output_past(0).await; let mut standby = moq(&relay, publish) - .args(["import", "--max-age", "1s", format.name]) + .args(["import", "--max-age", "1s", "--linger", "1s", format.name]) .stdin(Stdio::piped()) .stderr(Stdio::piped()) .spawn() @@ -739,7 +739,7 @@ async fn an_old_publisher_that_stays_up_keeps_the_export() { /// it does once its input declares a track. async fn import_announced(relay: &Relay, format: Format) -> (Child, JoinHandle) { let mut child = moq(relay, &[]) - .args(["import", "--max-age", "1s", format.name]) + .args(["import", "--max-age", "1s", "--linger", "1s", format.name]) .env("RUST_LOG", "warn,moq_net::lite::publisher=debug") .stdin(Stdio::piped()) .stderr(Stdio::piped()) @@ -861,7 +861,7 @@ fn redundant_import(relay: &Relay, feed: &tokio::sync::mpsc::UnboundedSender Child name, ]) .args(global) - .args(["import", "--max-age", MAX_AGE, "ts"]) + .args(["import", "--linger", LINGER, "ts"]) .stdin(Stdio::piped()) .kill_on_drop(true) .spawn() @@ -133,7 +133,7 @@ const SHORT: &[u8] = include_bytes!("../../../moq-mux/src/container/ts/test_data /// A subscriber already waiting when a short import ends still receives its media: the /// import announces at the finish, once its catalog lists the tracks, and keeps them servable -/// for their max-age after EOF. +/// for the linger after EOF. #[tokio::test] async fn a_short_import_drains_its_media_to_a_waiting_subscriber() { let _ = moq_tokio::crypto::install_default(); diff --git a/rs/moq-cli/tests/integration/report.rs b/rs/moq-cli/tests/integration/report.rs index 3bcb33a39a1..3188e735e82 100644 --- a/rs/moq-cli/tests/integration/report.rs +++ b/rs/moq-cli/tests/integration/report.rs @@ -217,7 +217,7 @@ async fn import_reports_count_what_each_rendition_wrote() { assert_eq!(expected.len(), 2, "the clip carries video and audio: {expected:?}"); assert_clean(&health); - let mut import = moq(&relay, "demo", &["import", "--report", "--max-age", "2s", "ts"]) + let mut import = moq(&relay, "demo", &["import", "--report", "--linger", "2s", "ts"]) .stdin(Stdio::piped()) .spawn() .expect("spawn the import"); @@ -284,7 +284,7 @@ async fn import_reports_a_silent_video_pid() { let stimulus = suppress(CLIP, video, 2.5, 4.5); let (_, health) = written(&stimulus); - let mut import = moq(&relay, "demo", &["import", "--report", "--max-age", "2s", "ts"]) + let mut import = moq(&relay, "demo", &["import", "--report", "--linger", "2s", "ts"]) .stdin(Stdio::piped()) .spawn() .expect("spawn the import"); @@ -343,7 +343,7 @@ async fn a_late_sink_reads_the_final_report() { let relay = relay().await; let (expected, health) = written(CLIP); - let mut import = moq(&relay, "demo", &["import", "--report", "--max-age", "10s", "ts"]) + let mut import = moq(&relay, "demo", &["import", "--report", "--linger", "10s", "ts"]) // Readiness is an INFO event, independent of the invoking shell's filter. .env("RUST_LOG", "info") .stdin(Stdio::piped()) diff --git a/rs/moq-gst/src/sink/imp.rs b/rs/moq-gst/src/sink/imp.rs index f5633764b1e..aea53b3e806 100644 --- a/rs/moq-gst/src/sink/imp.rs +++ b/rs/moq-gst/src/sink/imp.rs @@ -11,7 +11,6 @@ use std::collections::HashSet; use std::sync::{LazyLock, Mutex}; -#[cfg(test)] use std::time::Duration; use anyhow::{Context, Result}; @@ -28,11 +27,23 @@ use super::session::{ notify, sessions_structure, }; -#[derive(Debug, Clone, Default)] +#[derive(Debug, Clone)] struct Settings { url: Option, broadcast: Option, connection: crate::connection::Settings, + linger: Duration, +} + +impl Default for Settings { + fn default() -> Self { + Self { + url: None, + broadcast: None, + connection: Default::default(), + linger: moq_mux::Linger::DEFAULT, + } + } } impl TryFrom for ResolvedSettings { @@ -59,6 +70,12 @@ struct State { broadcast: moq_net::broadcast::Producer, catalog: Option, wall: std::sync::Arc, + /// How long a clean end keeps the finished broadcast up before its EOS. + linger: Duration, + /// Whether the finished broadcast closed, so the EOS it owes may go out. + drained: bool, + /// The task serving the finished broadcast for its linger, aborted on stop. + lingering: Option, /// Whether this publication's single EOS message already went out. One per publication, not per /// entry to PLAYING: a finished publication is terminal here, so there is nothing to announce twice. eos_delivered: bool, @@ -85,8 +102,8 @@ struct Control { /// Claim this publication's EOS, if it is owed and may go out now. /// -/// One claim per publication, so whichever arrives first (the transition into PLAYING, or the pad that -/// ended it) is the only one that posts. Called under `control`, which is what serializes the two. +/// One claim per publication, so whichever arrives first (the transition into PLAYING, or the end of +/// the linger) is the only one that posts. Called under `control`, which is what serializes the two. fn claim_eos(control: &mut Control) -> Posted { if !control.playing { return Posted::Nothing; @@ -94,7 +111,8 @@ fn claim_eos(control: &mut Control) -> Posted { let Some(state) = control.live.as_mut() else { return Posted::Nothing; }; - if state.eos_delivered || state.session.completion().get() != Completion::Eos { + // Only a clean finish drains, so a drained publication ended with EOS. + if state.eos_delivered || !state.drained { return Posted::Nothing; } state.eos_delivered = true; @@ -175,6 +193,12 @@ impl ObjectImpl for MoqSink { .blurb("The name of the broadcast to publish") .mutable_ready() .build(), + glib::ParamSpecUInt64::builder("linger") + .nick("Linger") + .blurb("At end of stream, keep the finished broadcast up this many milliseconds for late subscribers, then post EOS; 0 closes it at once") + .default_value(crate::connection::duration_millis(moq_mux::Linger::DEFAULT)) + .mutable_ready() + .build(), // Read-only, served from the live session's status. Each notifies on change. glib::ParamSpecEnum::builder::("status") .nick("Connection status") @@ -242,6 +266,7 @@ impl ObjectImpl for MoqSink { match pspec.name() { "url" => settings.url = value.get().unwrap(), "broadcast" => settings.broadcast = value.get().unwrap(), + "linger" => settings.linger = Duration::from_millis(value.get().unwrap()), name => settings.connection.set(name, value), } } @@ -273,6 +298,7 @@ impl ObjectImpl for MoqSink { match name { "url" => settings.url.to_value(), "broadcast" => settings.broadcast.to_value(), + "linger" => crate::connection::duration_millis(settings.linger).to_value(), name => settings.connection.get(name), } } @@ -537,6 +563,7 @@ impl MoqSink { /// Create the session and producers before any buffer flows. fn start_session(&self) -> Result { let settings = self.settings.lock().unwrap().clone(); + let linger = settings.linger; let settings = ResolvedSettings::try_from(settings).map_err(|err| { gst::error!(CAT, obj = self.obj(), "invalid settings: {err:#}"); gst::StateChangeError @@ -555,6 +582,9 @@ impl MoqSink { broadcast, catalog: Some(catalog), wall, + linger, + drained: false, + lingering: None, eos_delivered: false, }); let pads = self.pads_locked(&control); @@ -635,7 +665,10 @@ impl MoqSink { if let Some(err) = failure { gst::warning!(CAT, "finalize on stop: {err:?}"); } - // Close the broadcast before reaping the session task. + // Close the broadcast before reaping the session task. This also ends a linger early. + if let Some(lingering) = state.lingering.take() { + lingering.abort(); + } state.broadcast.close(); drop(closed); state.session.stop(); @@ -1016,15 +1049,56 @@ impl MoqSink { false => Posted::Nothing, }, None if completion.finish() => { - state.broadcast.close(); - drop((closed, catalog)); - claim_eos(control) + // The closed renditions and the catalog hold the finished tracks, so they go only + // once the broadcast closes. + let finished = moq_mux::Linger::new([state.broadcast.clone()], (closed, catalog), state.linger); + self.linger_locked(control, finished) } None => Posted::Nothing, }; Finished { updates, message } } + /// Serve the finished broadcast for its linger so a subscriber that arrives after the end still + /// gets the tail, and owe the EOS once it closes. With no linger, close it and owe the EOS now. + /// + /// The EOS waits for the linger because `gst-launch` exits on it, and NULL closes the broadcast. + /// Stopping the pipeline is the interrupt: it aborts the linger, which closes the broadcast. + fn linger_locked(&self, control: &mut Control, finished: moq_mux::Linger) -> Posted { + let state = control.live.as_mut().expect("a finished publication is live"); + let linger = state.linger; + if linger.is_zero() { + drop(finished); + state.drained = true; + return claim_eos(control); + } + gst::info!( + CAT, + obj = self.obj(), + "all pads ended, serving the broadcast for {linger:?}" + ); + let task = RUNTIME.spawn(linger_task( + finished, + self.obj().downgrade(), + state.session.completion(), + )); + state.lingering = Some(task.abort_handle()); + Posted::Nothing + } + + /// Owe the EOS now that the finished broadcast closed, unless a newer run replaced `session`. + fn drained(&self, session: &CompletionHandle) { + let message = { + let mut control = self.control.lock().unwrap(); + match control.live.as_mut() { + Some(state) if std::sync::Arc::ptr_eq(&state.session.completion(), session) => state.drained = true, + _ => return, + } + claim_eos(&mut control) + }; + self.post_current(message); + } + fn notify_updates(&self, updates: Vec) { for update in updates { update.pad.notify_changes(update.changes); @@ -1111,6 +1185,18 @@ impl MoqSink { } } +/// Serve a finished broadcast for its linger, then release the EOS its publication owes. +async fn linger_task( + finished: moq_mux::Linger, + element: glib::WeakRef, + session: CompletionHandle, +) { + finished.wait().await; + if let Some(element) = element.upgrade() { + element.imp().drained(&session); + } +} + /// Build a pad's producer for `caps` in the open publication, returning its status changes and whether /// the caps were accepted. fn observe_caps( @@ -1143,8 +1229,11 @@ mod tests { use super::super::session::CompletionState; use super::*; + /// A sink that closes its broadcast at the end of stream, for tests not about the linger. fn sink() -> super::super::MoqSink { - glib::Object::builder::().build() + glib::Object::builder::() + .property("linger", 0u64) + .build() } struct CatalogRun(super::super::MoqSink); @@ -1936,6 +2025,54 @@ mod tests { let _ = element.set_state(gst::State::Null); } + // A clean end keeps the broadcast up for subscribers that arrive late, and the EOS waits for it: + // `gst-launch` exits on that message, which would close the broadcast under them. The real sink + // lingers on its own runtime, so this drives the same task on a paused clock over a stand-in track. + #[tokio::test(start_paused = true)] + async fn the_eos_waits_for_the_linger() { + gst::init().unwrap(); + let sink = sink(); + sink.set_property("url", "https://127.0.0.1:1"); + sink.set_property("broadcast", "linger-test"); + let element = sink.clone().upcast::(); + let bus = gst::Bus::new(); + element.set_bus(Some(&bus)); + element.set_state(gst::State::Playing).expect("start playing"); + let (broadcast, session) = { + let control = sink.imp().control.lock().unwrap(); + let state = control.live.as_ref().expect("live session"); + (state.broadcast.clone(), state.session.completion()) + }; + let consumer = broadcast.consume(); + let track = broadcast.create_track("video", None).unwrap(); + track.append_group().unwrap().finish().unwrap(); + track.finish().unwrap(); + assert!(session.finish()); + while bus.pop().is_some() {} + + let linger = Duration::from_secs(30); + let start = moq_clock::now(); + let finished = moq_mux::Linger::new([broadcast], track, linger); + let lingering = tokio::spawn(linger_task(finished, sink.downgrade(), session)); + + moq_clock::sleep(linger - Duration::from_secs(1)).await; + let mut late = consumer.track("video").unwrap().subscribe(None).await.unwrap(); + assert!( + late.recv_group().await.unwrap().is_some(), + "a late subscriber drains the tail" + ); + assert!(bus.pop_filtered(&[gst::MessageType::Eos]).is_none(), "the EOS waits"); + + lingering.await.unwrap(); + assert_eq!(moq_clock::now() - start, linger); + assert!(consumer.is_closed()); + assert!( + bus.pop_filtered(&[gst::MessageType::Eos]).is_some(), + "the EOS follows the linger" + ); + let _ = element.set_state(gst::State::Null); + } + #[test] fn errors_are_scoped_to_the_session_that_produced_them() { gst::init().unwrap(); @@ -2006,6 +2143,7 @@ mod tests { for name in [ "url", "broadcast", + "linger", "tls-disable-verify", "quic-idle-timeout", "quic-keep-alive", @@ -2020,6 +2158,7 @@ mod tests { assert_eq!(spec(&sink, name).value_type(), u64::static_type()); assert_eq!(sink.property::(name), value); } + assert_eq!(spec(&sink, "linger").default_value().get::(), Ok(30_000)); for name in [ "status", "connected", diff --git a/rs/moq-gst/tests/element.rs b/rs/moq-gst/tests/element.rs index 57d34434d2a..ea8197d8ab4 100644 --- a/rs/moq-gst/tests/element.rs +++ b/rs/moq-gst/tests/element.rs @@ -37,10 +37,12 @@ fn track_error_of(sink: &gst::Element, pad: &str) -> Option { /// A publisher pointed at a relay that cannot answer. Connect runs in the background, so the element /// still reaches PAUSED and creates its producers, which is all these tests need. +/// A sink that closes its broadcast at the end of stream, for tests not about the linger. fn publisher() -> gst::Element { gst::ElementFactory::make("moqsink") .property("url", "https://127.0.0.1:1") .property("broadcast", "test") + .property("linger", 0u64) .build() .expect("create moqsink") } @@ -226,10 +228,10 @@ fn play_one_frame(sink: &gst::Element) -> gst::Pad { pad } -// The end of stream closes the broadcast and posts EOS at once, even with media a late subscriber -// could still have drained. +// With no linger, the end of stream closes the broadcast and posts EOS at once, even with media a +// late subscriber could still have drained. #[test] -fn published_media_posts_the_eos_at_once() { +fn published_media_posts_the_eos_at_once_without_a_linger() { init(); let sink = publisher(); let seen = recording_bus(&sink); @@ -240,6 +242,25 @@ fn published_media_posts_the_eos_at_once() { let _ = sink.set_state(gst::State::Null); } +// By default, published media is served for the linger, so the EOS that ends `gst-launch` waits for +// it. Stopping the pipeline is the interrupt: it closes the broadcast at once, and the run it ended +// owes no EOS. +#[test] +fn a_lingering_sink_holds_the_eos_until_the_pipeline_stops() { + init(); + let sink = publisher(); + sink.set_property("linger", 30_000u64); + let seen = recording_bus(&sink); + let pad = play_one_frame(&sink); + + assert!(pad.send_event(gst::event::Eos::new())); + assert_eq!(status_of(&sink, "sink_0"), "ended"); + assert_eq!(posted_eos(&seen), 0, "the finished broadcast is still serving its tail"); + + assert_eq!(sink.set_state(gst::State::Null), Ok(gst::StateChangeSuccess::Success)); + assert_eq!(posted_eos(&seen), 0, "the stopped run owes no EOS"); +} + // After the publication ends, a buffer that got past the pad's own EOS flag (a flush clears it) must // report the end rather than be dropped as if it had been published. #[test] diff --git a/rs/moq-hls/MIGRATE.md b/rs/moq-hls/MIGRATE.md index 6b9ce1f99cd..7f5f8bb5225 100644 --- a/rs/moq-hls/MIGRATE.md +++ b/rs/moq-hls/MIGRATE.md @@ -4,6 +4,16 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `Import::finish` borrows the importer + +`Import::finish(self) -> Result` is now `Import::finish(&mut self) -> Result<()>`. +Keep the importer, which owns the finished tracks, and hold it with `moq_mux::Linger` for as long as the broadcast should stay up: + +```rust +import.finish()?; +moq_mux::Linger::new([broadcast], import, moq_mux::Linger::DEFAULT).wait().await; +``` + ### Denied broadcast references report authorization failure A rendition whose referenced broadcast is outside the source origin's scope now diff --git a/rs/moq-hls/src/import.rs b/rs/moq-hls/src/import.rs index 4d5653076a7..a3c0c9c219d 100644 --- a/rs/moq-hls/src/import.rs +++ b/rs/moq-hls/src/import.rs @@ -804,19 +804,18 @@ impl Import { } } - /// Finish every rendition's tracks and the catalog once [`run`](Self::run) returned `Ok`, - /// returning the finished broadcast, which stays announced until it is dropped. Keep it - /// until its media is stale, with [`Finish::wait`](moq_net::broadcast::Finish::wait), so - /// subscribers still catching up drain the tail. - pub fn finish(mut self) -> Result { + /// Finish every rendition's tracks and the catalog once [`run`](Self::run) returned `Ok`. + /// + /// The finished tracks stay servable while the import is held, so hand it and the + /// broadcast to a [`moq_mux::Linger`] to keep them up for late subscribers. + pub fn finish(&mut self) -> Result<()> { for track in self.video.iter_mut().chain(self.audio.iter_mut()) { if let Some(importer) = &mut track.importer { importer.finish()?; } } self.sink.catalog.finish()?; - // Taken while the importers are alive, so it holds the tracks just finished. - Ok(self.sink.broadcast.clone().finish()) + Ok(()) } /// Abort every rendition's importer and the catalog with `err` so subscribers see the real @@ -1791,8 +1790,8 @@ mod tests { } /// A followed playlist that gains `EXT-X-ENDLIST` ends the import once its last segment is in, - /// including one published alongside the end, and the finished broadcast keeps serving it, - /// catalog included, for the retention. + /// including one published alongside the end, and the finished import keeps serving it, + /// catalog included, while it is held. #[tokio::test] async fn a_playlist_that_ends_later_finishes_the_import() { tokio::time::pause(); @@ -1801,11 +1800,9 @@ mod tests { std::fs::write(dir.join("media.mp4"), &resource).unwrap(); std::fs::write(dir.join("media.m3u8"), playlist).unwrap(); - let max_age = Duration::from_secs(3); let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); - let config = moq_mux::catalog::Config::default().with_max_age(max_age); - let catalog = CatalogProducer::new(&mut broadcast, config).unwrap(); + let catalog = CatalogProducer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); let playlist = Url::from_file_path(dir.join("media.m3u8")).unwrap(); let mut import = Import::new(broadcast, catalog, Config::new(playlist)).unwrap(); @@ -1824,8 +1821,7 @@ mod tests { .expect("an ended playlist ends the import") .unwrap(); assert_eq!(import.video[0].next_sequence, Some(4), "the final segment was imported"); - let finish = import.finish().unwrap(); - assert_eq!(finish.linger(), max_age); + import.finish().unwrap(); let mut listed = consumer.track("catalog.json").unwrap().subscribe(None).await.unwrap(); assert!( listed.recv_group().await.unwrap().is_some(), diff --git a/rs/moq-mux/MIGRATE.md b/rs/moq-mux/MIGRATE.md index b1d217694db..5c333890950 100644 --- a/rs/moq-mux/MIGRATE.md +++ b/rs/moq-mux/MIGRATE.md @@ -4,6 +4,16 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `Programs::finish` returns the program broadcasts + +`Programs::finish` now returns each program's `broadcast::Producer` instead of a `broadcast::Finish`. +Hand them, with the `Programs` that owns their tracks, to the new `moq_mux::Linger` to keep the finished programs up for a fixed time from the end: + +```rust +let broadcasts = programs.finish()?; +moq_mux::Linger::new(broadcasts, programs, moq_mux::Linger::DEFAULT).wait().await; +``` + ### TS service information expires on the program clock TS import retracts SI sub-tables and incomplete generations that stop repeating within their table's maximum interval (30 seconds for unknown tables). diff --git a/rs/moq-mux/src/container/ts/programs.rs b/rs/moq-mux/src/container/ts/programs.rs index 994b71cd9aa..2ff047b78fc 100644 --- a/rs/moq-mux/src/container/ts/programs.rs +++ b/rs/moq-mux/src/container/ts/programs.rs @@ -220,22 +220,21 @@ impl Programs { /// Finish each program's tracks, then its catalog while it still lists them. /// - /// Returns each program's [`Finish`](moq_net::broadcast::Finish), which keeps it servable - /// until its media is stale; dropping one closes that program. - /// Fails when the input ended before its PAT listed a program or any program declared a track. - pub fn finish(&mut self) -> anyhow::Result> { + /// Returns each program's broadcast. The finished tracks stay servable while this importer + /// is held, so hand both to a [`Linger`](crate::Linger) to keep them up for late + /// subscribers. Fails when the input ended before its PAT listed a program or any program + /// declared a track. + pub fn finish(&mut self) -> anyhow::Result> { self.demux.finish(); anyhow::ensure!(!self.programs.is_empty(), "input ended before its PAT listed a program"); let mut announced = false; - let mut finished = Vec::with_capacity(self.programs.len()); for program in &mut self.programs { program.import.finish()?; announced |= program.announce_if_listed()?; program.catalog.finish()?; - finished.push(program.broadcast.clone().finish()); } anyhow::ensure!(announced, "input ended before any program declared a track"); - Ok(finished) + Ok(self.programs.iter().map(|program| program.broadcast.clone()).collect()) } /// Abort every program's catalog and media tracks with `err`. Consumes the importer. @@ -329,7 +328,7 @@ mod test { for chunk in input.chunks(100) { programs.decode(chunk).unwrap(); } - let _finished = programs.finish().unwrap(); + programs.finish().unwrap(); assert_eq!(programs.programs.len(), 50); let consumer = origin.consume().request_broadcast("event/50.hang", None).await.unwrap(); @@ -386,7 +385,7 @@ mod test { let origin = crate::source::produce_origin(); let mut programs = Programs::new(origin.clone(), "event.hang", catalog::Config::default()); programs.decode(&two_section_pat()).unwrap(); - let _finished = programs.finish().unwrap(); + programs.finish().unwrap(); assert_eq!(programs.programs.len(), 2); assert_eq!(renditions(&origin, "event/1.hang").await, 1); assert_eq!(renditions(&origin, "event/2.hang").await, 1); @@ -399,7 +398,7 @@ mod test { let origin = crate::source::produce_origin(); let mut programs = Programs::new(origin, "event.hang", catalog::Config::default()); programs.decode(&two_section_pat()).unwrap(); - let _finished = programs.finish().unwrap(); + programs.finish().unwrap(); let epochs: Vec<_> = programs .programs .iter() @@ -433,7 +432,7 @@ mod test { for packet in packets { programs.decode(packet).unwrap(); } - let _finished = programs.finish().unwrap(); + programs.finish().unwrap(); assert!( programs .programs @@ -554,7 +553,7 @@ mod test { "no program is known before the PAT is whole" ); programs.decode(&input[100..]).unwrap(); - let _finished = programs.finish().unwrap(); + programs.finish().unwrap(); let after = moq_clock::wall(); for (path, fills) in [("event/1.hang", [0xAA, 0xBB]), ("event/2.hang", [0xCC, 0xDD])] { diff --git a/rs/moq-mux/src/lib.rs b/rs/moq-mux/src/lib.rs index d23d7cff67c..0d973d185b6 100644 --- a/rs/moq-mux/src/lib.rs +++ b/rs/moq-mux/src/lib.rs @@ -33,6 +33,8 @@ //! HLS/DASH playlists without downloading media. //! - [`follow`] follows a broadcast across the publisher instances that //! serve it, for a reader that restarts on each one. +//! - [`Linger`] keeps a cleanly ended broadcast up for a fixed time, so a +//! subscriber that arrives after the end still gets its tail. pub mod binary; pub mod catalog; @@ -44,6 +46,7 @@ pub mod follow; pub mod import; mod jitter; pub mod json; +mod linger; mod pace; pub mod rate; pub mod select; @@ -53,6 +56,7 @@ pub mod timeline; pub use clock::Clock; pub use error::*; +pub use linger::Linger; pub use pace::Pacer; pub use source::{Binding, Source}; diff --git a/rs/moq-mux/src/linger.rs b/rs/moq-mux/src/linger.rs new file mode 100644 index 00000000000..022577edeaa --- /dev/null +++ b/rs/moq-mux/src/linger.rs @@ -0,0 +1,148 @@ +//! Keeping a cleanly ended broadcast up for subscribers that arrive late. + +use std::time::Duration; + +/// Keeps cleanly ended broadcasts announced, serving their finished tracks, for a fixed +/// duration, then closes them. +/// +/// Subscribers already reading a track get its tail when the broadcast closes; this serves +/// the ones that arrive after the end, such as a relay or player that read the final catalog +/// late. Finish the catalog and media tracks first, then hand over the broadcasts and +/// `owner`, whatever holds their track producers: a finished track stops being served once +/// its producers drop, so the owner drops only after the broadcasts close. +/// +/// The deadline is fixed at [`new`](Self::new), however late [`wait`](Self::wait) is polled. +/// Dropping the handle closes the broadcasts at once, which is how an interrupt ends it. A +/// newer publisher announced at the same path does not end it: that one wins the path while +/// it is up, and readers that come back here find the finished catalog. +#[must_use = "dropping a Linger closes its broadcasts at once"] +pub struct Linger { + broadcasts: Vec, + // Dropped after `Drop::drop` closes the broadcasts. + _owner: T, + // `None` when the duration overflows the clock, so only an interrupt ends it. + deadline: Option, +} + +impl Linger<()> { + /// How long a publisher keeps a cleanly ended broadcast up unless configured otherwise. + /// + /// Matches the default media max-age and a relay's idle track copy. + pub const DEFAULT: Duration = Duration::from_secs(30); +} + +impl Linger { + /// Keep `broadcasts` up for `duration` from now, holding `owner` until they close. + /// + /// A zero `duration` closes them on the first poll of [`wait`](Self::wait). + pub fn new( + broadcasts: impl IntoIterator, + owner: T, + duration: Duration, + ) -> Self { + Self { + broadcasts: broadcasts.into_iter().collect(), + _owner: owner, + deadline: moq_clock::now().checked_add(duration), + } + } + + /// Wait for the deadline on the ambient clock, then close the broadcasts. + pub async fn wait(self) { + match self.deadline { + Some(deadline) => moq_clock::sleep_until(deadline).await, + None => std::future::pending().await, + } + } +} + +impl Drop for Linger { + fn drop(&mut self) { + // Closed before the owner drops, so no lookup finds the broadcast without its tracks. + for broadcast in &self.broadcasts { + broadcast.close(); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const PATH: &str = "live"; + + /// A broadcast at [`PATH`] whose one track, `video`, has three finished groups. + fn finished(origin: &moq_net::origin::Producer) -> (moq_net::broadcast::Producer, moq_net::track::Producer) { + let broadcast = origin.publish(PATH, moq_net::origin::Route::default()).unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + for _ in 0..3 { + let mut group = track.append_group().unwrap(); + group.write_frame(None, bytes::Bytes::from_static(b"frame")).unwrap(); + group.finish().unwrap(); + } + track.finish().unwrap(); + (broadcast, track) + } + + /// Whether the origin has stopped routing [`PATH`]. + async fn unroutable(origin: &moq_net::origin::Producer) -> bool { + matches!( + origin.consume().request_broadcast(PATH, None).await, + Err(moq_net::Error::Unroutable) + ) + } + + /// A subscriber arriving just before the deadline gets every group, the latest by + /// subscribing and the earlier ones by fetching, and the path stops routing once it passes. + #[tokio::test(start_paused = true)] + async fn a_late_subscriber_drains_the_tail_until_the_deadline() { + let origin = crate::source::produce_origin(); + let (broadcast, track) = finished(&origin); + let start = moq_clock::now(); + let linger = tokio::spawn(Linger::new([broadcast], track, Duration::from_secs(30)).wait()); + + moq_clock::sleep(Duration::from_secs(29)).await; + let late = origin.consume().request_broadcast(PATH, None).await.unwrap(); + let video = late.track("video").unwrap(); + for sequence in 0..3 { + let mut group = video.fetch_group(sequence, None).await.unwrap(); + let frame = group.read_frame().await.unwrap().expect("a frame"); + assert_eq!(frame.payload.as_ref(), b"frame"); + } + let mut live = video.subscribe(None).await.unwrap(); + assert!(live.recv_group().await.unwrap().is_some(), "the latest group"); + assert!( + live.recv_group().await.unwrap().is_none(), + "the track ends after its tail" + ); + + linger.await.unwrap(); + assert_eq!(moq_clock::now() - start, Duration::from_secs(30)); + assert!(unroutable(&origin).await); + } + + /// The deadline counts from the end of the content, not from when the wait is polled. + #[tokio::test(start_paused = true)] + async fn the_deadline_is_fixed_at_the_end() { + let origin = crate::source::produce_origin(); + let (broadcast, track) = finished(&origin); + let start = moq_clock::now(); + let linger = Linger::new([broadcast], track, Duration::from_secs(30)); + + moq_clock::sleep(Duration::from_secs(20)).await; + linger.wait().await; + assert_eq!(moq_clock::now() - start, Duration::from_secs(30)); + } + + /// Dropping the handle, as an interrupt does, closes the broadcast at once. + #[tokio::test(start_paused = true)] + async fn dropping_closes_at_once() { + let origin = crate::source::produce_origin(); + let (broadcast, track) = finished(&origin); + let linger = Linger::new([broadcast], track, Duration::from_secs(30)); + assert!(!unroutable(&origin).await); + + drop(linger); + assert!(unroutable(&origin).await); + } +} diff --git a/rs/moq-rtc/src/ingest.rs b/rs/moq-rtc/src/ingest.rs index ca50aa0f874..86d1b8805aa 100644 --- a/rs/moq-rtc/src/ingest.rs +++ b/rs/moq-rtc/src/ingest.rs @@ -72,11 +72,10 @@ impl session::MediaSink for IngestSink { self.bridges.abort(err); } - fn finish(&mut self) -> Result { + fn finish(&mut self) -> Result { self.bridges.finish()?; self.catalog.finish()?; - // Taken while the bridges are alive, so it holds the tracks just finished. - Ok(self.broadcast.clone().finish()) + Ok(self.broadcast.clone()) } } @@ -111,13 +110,12 @@ mod tests { } /// A clean end finishes every track, the catalog and a video track that never got a frame - /// included, so the finished broadcast serves them to a late subscriber for the retention. + /// included, so the held sink serves them to a late subscriber for the linger. #[tokio::test] - async fn finish_keeps_every_track_for_its_retention() { + async fn finish_keeps_every_track_for_the_linger() { let broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); - let max_age = Duration::from_secs(3); - let mut sink = IngestSink::new(broadcast, moq_mux::catalog::Config::default().with_max_age(max_age)).unwrap(); + let mut sink = IngestSink::new(broadcast, moq_mux::catalog::Config::default()).unwrap(); let audio = str0m::media::MediaKind::Audio; sink.on_track("0".into(), audio, str0m::format::Codec::Opus, Some((48_000, 2))) .unwrap(); @@ -134,16 +132,15 @@ mod tests { ) .unwrap(); - let finish = sink.finish().unwrap(); - assert_eq!(finish.linger(), max_age); - drop(sink); + let broadcast = sink.finish().unwrap(); + let linger = moq_mux::Linger::new([broadcast], sink, Duration::from_secs(3)); let mut catalog = consumer.track("catalog.json").unwrap().subscribe(None).await.unwrap(); assert!( catalog.recv_group().await.unwrap().is_some(), "the late subscriber reads the catalog" ); assert!(consumer.track("0.opus").is_ok()); - drop(finish); + drop(linger); assert!(consumer.is_closed()); } } diff --git a/rs/moq-rtc/src/lib.rs b/rs/moq-rtc/src/lib.rs index df6ba0dde5b..6675d9719ec 100644 --- a/rs/moq-rtc/src/lib.rs +++ b/rs/moq-rtc/src/lib.rs @@ -260,25 +260,18 @@ mod tests { .unwrap() .unwrap() .unwrap(); - // The DELETE ended the ingest cleanly, so it serves the finished broadcast until its - // media is stale; a newer publish at the path ends that linger at once. + // The DELETE ended the ingest cleanly, so it serves the finished broadcast for the + // linger; dropping the session, as an interrupt does, ends that at once. let ingested = server_origin .consume() .request_broadcast("ingested", None) .await .unwrap(); assert!(!ingested.is_closed(), "the finished ingest lingers"); - let _replacement = server_origin - .publish( - "ingested", - moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()), - ) - .unwrap(); - tokio::time::timeout(TIMEOUT, whip_task) - .await - .unwrap() - .unwrap() - .unwrap(); + whip_task.abort(); + let stopped = tokio::time::timeout(TIMEOUT, whip_task).await.unwrap(); + assert!(stopped.unwrap_err().is_cancelled()); + assert!(ingested.is_closed(), "the interrupt closes the finished ingest"); let final_whip = whip_server_wire.stats(); let final_whep = whep_server_wire.stats(); assert!(!server.terminate(&whip_id)); diff --git a/rs/moq-rtc/src/server/mod.rs b/rs/moq-rtc/src/server/mod.rs index 1adc598316d..4e4961cd421 100644 --- a/rs/moq-rtc/src/server/mod.rs +++ b/rs/moq-rtc/src/server/mod.rs @@ -93,13 +93,14 @@ impl AcceptedSession { } // A DELETE is the peer's clean end after its final media. An ingest serves the - // finished broadcast until its media is stale, so subscribers still catching up - // drain the tail; dropping this future ends that early, as does a newer publish - // at the path. A peer that vanishes instead closes the broadcast at once. + // finished broadcast for the linger, so a subscriber that arrives after the end still + // gets the tail; dropping this future ends that early. A peer that vanishes instead + // closes the broadcast at once. tracing::debug!(role = self.role, "webrtc session terminated by DELETE"); - if let Some(finish) = session.finish()? { - tracing::info!(role = self.role, linger = ?finish.linger(), "serving the finished broadcast until its media is stale"); - finish.wait().await; + let linger = self.server.config().linger; + if let Some(finished) = session.finish(linger)? { + tracing::info!(role = self.role, ?linger, "serving the finished broadcast"); + finished.wait().await; } Ok(()) } @@ -168,6 +169,11 @@ pub struct Config { /// Connection allocator each ingested track claims its peak-hold bitrate on. /// Ingest only (`server publish` / WHIP); WHEP egress ignores this. pub bandwidth: moq_net::bandwidth::Allocator, + + /// How long an ingest the peer ended with a `DELETE` stays up, so a subscriber that + /// arrives after the end still gets the tail. Defaults to [`moq_mux::Linger::DEFAULT`]; + /// zero closes it at once. Ingest only (`server publish` / WHIP). + pub linger: Duration, } impl Default for Config { @@ -177,6 +183,7 @@ impl Default for Config { udp_bind: SocketAddr::from(([0, 0, 0, 0], 0)), max_age: None, bandwidth: moq_net::bandwidth::Allocator::unlimited(), + linger: moq_mux::Linger::DEFAULT, } } } diff --git a/rs/moq-rtc/src/session.rs b/rs/moq-rtc/src/session.rs index ab15548af30..825a543e9b2 100644 --- a/rs/moq-rtc/src/session.rs +++ b/rs/moq-rtc/src/session.rs @@ -50,8 +50,9 @@ pub trait MediaSink: Send { fn abort(&mut self, err: moq_net::Error); /// Called once when the peer ends the session cleanly after its final media: finish - /// every track and return the finished broadcast, which stays up until it drops. - fn finish(&mut self) -> Result; + /// every track and return the broadcast. The finished tracks stay servable while the sink + /// is held. + fn finish(&mut self) -> Result; } /// What the session does with the negotiated media stream. @@ -137,11 +138,14 @@ impl Session { self.handle_media(mid) } - /// End an ingest cleanly after its final media: finish the sink's tracks and return the - /// finished broadcast to linger on. `None` for an egress session, which publishes nothing. - pub fn finish(self) -> Result> { + /// End an ingest cleanly after its final media: finish the sink's tracks and keep the + /// finished broadcast up for `linger`. `None` for an egress session, which publishes nothing. + pub fn finish(self, linger: std::time::Duration) -> Result>>> { match self.role { - MediaRole::Ingest(mut sink) => sink.finish().map(Some), + MediaRole::Ingest(mut sink) => { + let broadcast = sink.finish()?; + Ok(Some(moq_mux::Linger::new([broadcast], sink, linger))) + } MediaRole::Egress(_) => Ok(None), } } diff --git a/rs/moq-rtmp/src/dial.rs b/rs/moq-rtmp/src/dial.rs index 722d4f2d0ad..026665798a7 100644 --- a/rs/moq-rtmp/src/dial.rs +++ b/rs/moq-rtmp/src/dial.rs @@ -75,6 +75,8 @@ pub struct Client { max_age: Option, /// Connection allocator each track [`pull`](Self::pull) ingests claims its peak-hold bitrate on. bandwidth: moq_net::bandwidth::Allocator, + /// How long a broadcast [`pull`](Self::pull) finished stays up. + linger: Duration, /// Script data [`pull`](Self::pull) discards, including any sent before Play.Start. metadata: crate::metadata::Dropped, } @@ -158,6 +160,7 @@ impl Client { max_delay: crate::DEFAULT_MAX_DELAY, max_age: None, bandwidth: moq_net::bandwidth::Allocator::unlimited(), + linger: moq_mux::Linger::DEFAULT, metadata: Default::default(), }) } @@ -201,6 +204,16 @@ impl Client { self } + /// Keep the broadcast [`pull`](Self::pull) publishes up for `linger` after the remote + /// completes the play, so a subscriber that arrives after the end still gets the tail. + /// Defaults to [`moq_mux::Linger::DEFAULT`]; zero closes it at once. + /// + /// The pull (ingest) direction only. + pub fn with_linger(mut self, linger: Duration) -> Self { + self.linger = linger; + self + } + /// Push a MoQ broadcast out to the remote: request `publish` on `stream_key`, /// mux the broadcast to FLV, and send each tag as an RTMP audio/video message /// until the broadcast ends or the connection drops. @@ -297,10 +310,9 @@ impl Client { /// on its own task. /// /// A remote that completes the play (`NetStream.Play.Complete` or `NetStream.Play.Stop`) - /// ends cleanly: the finished broadcast stays up until its media is stale, so subscribers - /// still catching up drain the tail, and the future resolves after that. Dropping the - /// future ends the linger early, as does a newer publish at `path`. A connection that - /// drops first closes the broadcast at once. + /// ends cleanly: the finished broadcast stays up for the [linger](Self::with_linger), and + /// the future resolves after that. Dropping the future ends the linger early. A + /// connection that drops first closes the broadcast at once. pub async fn pull(mut self, stream_key: &str, origin: &origin::Producer, path: impl moq_net::AsPath) -> Result<()> { let path = path.as_path(); let request = self @@ -322,14 +334,15 @@ impl Client { let result = self.pull_media(&mut publisher).await; match result { - // A clean end: serve the finished broadcast until its media is stale, so - // subscribers still catching up drain the tail. The remote is done, so hang up first. + // A clean end: serve the finished broadcast for the linger, so a subscriber that + // arrives after the end still gets the tail. The remote is done, so hang up first. Ok(true) => { + let linger = self.linger; drop(self); - match publisher.finish() { - Ok(finish) => { - tracing::info!(%path, linger = ?finish.linger(), "rtmp play completed, serving the finished broadcast until its media is stale"); - finish.wait().await; + match publisher.finish(linger) { + Ok(publisher) => { + tracing::info!(%path, ?linger, "rtmp play completed, serving the finished broadcast"); + publisher.wait().await; } Err(err) => tracing::debug!(%err, "error finishing RTMP pull"), } @@ -582,13 +595,12 @@ impl Publisher { Ok(self.importer.decode(&flv::tag(tag_type, timestamp, body))?) } - /// Flush any buffered media and finish every track, the catalog included, returning - /// the finished broadcast, which stays announced until it is dropped. - fn finish(mut self) -> anyhow::Result { + /// Flush any buffered media and finish every track, the catalog included, keeping the + /// finished broadcast up for `linger`. + fn finish(mut self, linger: Duration) -> anyhow::Result> { self.importer.finish()?; self.catalog.finish()?; - // Taken while the importer is alive, so it holds the tracks just finished. - Ok(self.broadcast.finish()) + Ok(moq_mux::Linger::new([self.broadcast.clone()], self, linger)) } /// Flush any buffered media, then end the broadcast so the origin unannounces it at once. @@ -642,6 +654,7 @@ mod tests { max_delay: crate::DEFAULT_MAX_DELAY, max_age: None, bandwidth: Default::default(), + linger: Duration::ZERO, metadata: crate::metadata::Dropped::new("metadata"), }; let origin = moq_tokio::origin::spawn(); @@ -650,7 +663,7 @@ mod tests { let queued = std::mem::take(&mut client.work); client.pull_results(queued, &mut publisher).await.unwrap(); assert_eq!(client.metadata.count, 2); - drop(publisher.finish().unwrap()); + drop(publisher.finish(Duration::ZERO).unwrap()); drop(client); logs_assert(|lines| { let warnings = lines @@ -694,11 +707,11 @@ mod tests { /// A clean end crosses an RTMP hop in both directions: a finished export unpublishes, /// so the server it pushes to lingers, and a server whose play completes lets a pull - /// linger. Each lingering side serves its finished copy for the declared max-age. + /// linger. Each lingering side serves its finished copy for its linger. #[tokio::test(start_paused = true)] async fn a_finished_broadcast_lingers_across_rtmp() { let peer = "127.0.0.1:1935".parse().unwrap(); - let max_age = Duration::from_secs(10); + let linger = Duration::from_secs(10); for push in [true, false] { let upstream = moq_tokio::origin::spawn(); let downstream = moq_tokio::origin::spawn(); @@ -710,11 +723,9 @@ mod tests { let ingest = downstream.clone(); let server = tokio::spawn(async move { match accept_stream(server_io, peer).await.unwrap().unwrap() { - Request::Publish(publish) => publish - .with_max_age(Some(max_age)) - .accept(&ingest, "live/cam0") - .await - .unwrap(), + Request::Publish(publish) => { + publish.with_linger(linger).accept(&ingest, "live/cam0").await.unwrap() + } Request::Play(play) => play.accept(&served, "live/cam0").await.unwrap(), } }); @@ -724,7 +735,7 @@ mod tests { match push { true => client.publish("cam0", exported, "live/cam0").await.unwrap(), false => client - .with_max_age(max_age) + .with_linger(linger) .pull("cam0", &pulled, "live/cam0") .await .unwrap(), @@ -749,7 +760,7 @@ mod tests { done.await.unwrap(); assert!(!copy.is_closed(), "push={push}: the copy lingers"); lingering.await.unwrap(); - assert_eq!(start.elapsed(), max_age, "push={push}"); + assert_eq!(start.elapsed(), linger, "push={push}"); assert!(copy.is_closed()); } } diff --git a/rs/moq-rtmp/src/server.rs b/rs/moq-rtmp/src/server.rs index 6458a6ee976..4cb03e20d82 100644 --- a/rs/moq-rtmp/src/server.rs +++ b/rs/moq-rtmp/src/server.rs @@ -557,6 +557,8 @@ pub struct Publish { /// Connection allocator each passthrough track claims its peak-hold bitrate on. /// Override with [`with_bandwidth`](Self::with_bandwidth). bandwidth: moq_net::bandwidth::Allocator, + /// How long an unpublished broadcast stays up. Override with [`with_linger`](Self::with_linger). + linger: Duration, } impl Publish { @@ -608,6 +610,14 @@ impl Publish { self } + /// Keep the broadcast up for `linger` after the client unpublishes, so a subscriber that + /// arrives after the end still gets the tail. Defaults to [`moq_mux::Linger::DEFAULT`]; + /// zero closes it at once. + pub fn with_linger(mut self, linger: Duration) -> Self { + self.linger = linger; + self + } + /// Accept the publish: announce a broadcast at `path` in `origin` and pump the /// RTMP media into it until the client disconnects. /// @@ -617,10 +627,9 @@ impl Publish { /// run it on its own task. /// /// A client that unpublishes (`deleteStream`) ends cleanly: the finished broadcast - /// stays up until its media is stale, so subscribers still catching up drain the tail, - /// and the future resolves after that. Dropping the future ends the linger early, as - /// does a newer publish at `path`. A connection that drops without unpublishing - /// closes the broadcast at once. + /// stays up for the [linger](Self::with_linger), and the future resolves after that. + /// Dropping the future ends the linger early. A connection that drops without + /// unpublishing closes the broadcast at once. pub async fn accept(mut self, origin: &origin::Producer, path: impl moq_net::AsPath) -> Result<()> { let path = path.as_path(); // Reserve the broadcast path before telling the client the publish succeeded: @@ -665,14 +674,14 @@ impl Publish { let peer = self.peer; match result { - // A clean end: serve the finished broadcast until its media is stale, so subscribers - // still catching up drain the tail. The encoder is done, so hang up on it first. + // A clean end: serve the finished broadcast for the linger, so a subscriber that + // arrives after the end still gets the tail. The encoder is done, so hang up first. Ok(End::Unpublished) => { drop(self.stream); - match publisher.finish() { - Ok(finish) => { - tracing::info!(%peer, %path, linger = ?finish.linger(), "rtmp publish ended, serving the finished broadcast until its media is stale"); - finish.wait().await; + match publisher.finish(self.linger) { + Ok(linger) => { + tracing::info!(%peer, %path, linger = ?self.linger, "rtmp publish ended, serving the finished broadcast"); + linger.wait().await; } Err(err) => tracing::debug!(%peer, %err, "error finishing RTMP publish"), } @@ -1163,6 +1172,7 @@ async fn accept_until_request(mut stream: S, peer: SocketAddr) -> any peer, max_age: None, bandwidth: moq_net::bandwidth::Allocator::unlimited(), + linger: moq_mux::Linger::DEFAULT, }))); } // The client wants to play: hand control back to the caller. @@ -1518,13 +1528,12 @@ impl Publisher { Ok(self.importer.decode(&flv::tag(tag_type, timestamp, body))?) } - /// Flush any buffered media and finish every track, the catalog included, returning - /// the finished broadcast, which stays announced until it is dropped. - fn finish(mut self) -> anyhow::Result { + /// Flush any buffered media and finish every track, the catalog included, keeping the + /// finished broadcast up for `linger`. + fn finish(mut self, linger: Duration) -> anyhow::Result> { self.importer.finish()?; self.catalog.finish()?; - // Taken while the importer is alive, so it holds the tracks just finished. - Ok(self.broadcast.finish()) + Ok(moq_mux::Linger::new([self.broadcast.clone()], self, linger)) } /// Flush any buffered media, close out the broadcast's open groups, and end @@ -1586,7 +1595,7 @@ mod tests { .await .unwrap(); assert_eq!(publisher.metadata.count, 2); - drop(publisher.finish().unwrap()); + drop(publisher.finish(Duration::ZERO).unwrap()); logs_assert(|lines| { let warnings = lines .iter() @@ -1724,21 +1733,17 @@ mod tests { } } - /// Accept the publish arriving on `stream` into `origin` at `live/cam0`, declaring `max_age`. - async fn accept_publish(stream: tokio::io::DuplexStream, origin: origin::Producer, max_age: Duration) { + /// Accept the publish arriving on `stream` into `origin` at `live/cam0`, lingering for `linger`. + async fn accept_publish(stream: tokio::io::DuplexStream, origin: origin::Producer, linger: Duration) { let peer = "127.0.0.1:1935".parse().unwrap(); let Some(Request::Publish(publish)) = accept_stream(stream, peer).await.unwrap() else { panic!("expected a publish request"); }; - publish - .with_max_age(Some(max_age)) - .accept(&origin, "live/cam0") - .await - .unwrap(); + publish.with_linger(linger).accept(&origin, "live/cam0").await.unwrap(); } - /// An unpublish is a clean end: the finished broadcast keeps serving a late subscriber - /// until its media is stale, then closes. A hangup without one closes at once. + /// An unpublish is a clean end: the finished broadcast stays up for the linger, then + /// closes. A hangup without one closes at once. #[tokio::test(start_paused = true)] async fn an_unpublish_lingers_and_a_hangup_does_not() { for (leave, linger) in [ @@ -1763,9 +1768,9 @@ mod tests { } /// A late subscriber still reads the finished media during the linger, and an encoder - /// reconnecting under the same key ends the linger at once. + /// reconnecting under the same key takes the path without ending it. #[tokio::test(start_paused = true)] - async fn a_reconnect_ends_the_linger() { + async fn a_reconnect_leaves_the_linger_running() { let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); let (client, server) = tokio::io::duplex(64 * 1024); @@ -1781,12 +1786,16 @@ mod tests { ); assert!(!accept.is_finished()); - let start = tokio::time::Instant::now(); let (client, server) = tokio::io::duplex(64 * 1024); let _reconnect = tokio::spawn(accept_publish(server, origin.clone(), Duration::from_secs(10))); let _encoder = tokio::spawn(publish_one_keyframe(client, Leave::Hangup)); + tokio::time::sleep(Duration::from_secs(1)).await; + assert!(!accept.is_finished(), "the newer publisher leaves the linger running"); + assert!(!finished.is_closed()); + + let start = tokio::time::Instant::now(); accept.await.unwrap(); - assert_eq!(start.elapsed(), Duration::ZERO); + assert_eq!(start.elapsed(), Duration::from_secs(4)); assert!(finished.is_closed()); } diff --git a/rs/moq-srt/MIGRATE.md b/rs/moq-srt/MIGRATE.md index 7b4443697ee..23788f8a2ec 100644 --- a/rs/moq-srt/MIGRATE.md +++ b/rs/moq-srt/MIGRATE.md @@ -4,6 +4,15 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `with_linger` keeps an ingest up; `with_follow` follows an egress + +| Before | After | +| --- | --- | +| `Subscribe::with_linger` | `Subscribe::with_follow` | +| `Client::with_linger` (export) | `Client::with_follow` | + +`Publish::with_linger` and `Client::with_linger` now set how long a cleanly ended ingest stays up, counted from its end (default 30 s, zero closes it at once), instead of its tracks' max age. + ### `run` and `Config` are removed `run`, `Config`, and `Error::ListenerClosed` are gone. diff --git a/rs/moq-srt/src/dial.rs b/rs/moq-srt/src/dial.rs index c109ae5f655..f0fc8e0008a 100644 --- a/rs/moq-srt/src/dial.rs +++ b/rs/moq-srt/src/dial.rs @@ -61,9 +61,12 @@ pub struct Client { /// The programs of a multi-program remote [`pull`] publishes, or `None` to refuse one. program: Option, + /// How long a broadcast [`pull`] finished stays up. [`pull`] only. + linger: Duration, + /// How long [`publish`] waits for the same publisher instance to come back once its /// broadcast ends. [`publish`] only. - linger: Duration, + follow: Duration, /// Whether [`publish`] follows another instance replacing its broadcast, as a program /// switch on the same connection. [`publish`] only. @@ -81,7 +84,8 @@ impl Client { max_age: None, bandwidth: moq_net::bandwidth::Allocator::unlimited(), program: None, - linger: Duration::ZERO, + linger: moq_mux::Linger::DEFAULT, + follow: Duration::ZERO, stitch: false, } } @@ -120,12 +124,22 @@ impl Client { self } - /// Wait up to `linger` for the same publisher instance to come back once the broadcast + /// Keep the broadcast [`pull`](Self::pull) publishes up for `linger` after the remote + /// stream ends, so a subscriber that arrives after the end still gets the tail. Defaults to + /// [`moq_mux::Linger::DEFAULT`]; zero closes it at once. + /// + /// The pull (ingest) direction only. + pub fn with_linger(mut self, linger: Duration) -> Self { + self.linger = linger; + self + } + + /// Wait up to `follow` for the same publisher instance to come back once the broadcast /// [`publish`](Self::publish) reads ends, carrying on with the same stream. Zero (the /// default) ends the push at the broadcast's end. /// A stitched replacement must also supply its first catalog within this bound. - pub fn with_linger(mut self, linger: Duration) -> Self { - self.linger = linger; + pub fn with_follow(mut self, follow: Duration) -> Self { + self.follow = follow; self } @@ -143,7 +157,7 @@ impl Client { let socket = self.call(Mode::Publish).await?; let options = crate::ts::Options { latency: self.max_delay, - linger: self.linger, + follow: self.follow, stitch: self.stitch, }; serve_subscribe(origin, path.as_str(), socket, options).await @@ -151,15 +165,19 @@ impl Client { /// Pull a remote MPEG-TS stream into `origin` at `path` until the remote ends. /// - /// The finished broadcast then stays up until its media is stale, as + /// The finished broadcast then stays up for the [linger](Self::with_linger), as /// [`Publish::accept`](crate::Publish::accept) describes, and the future resolves after that. pub async fn pull(&self, origin: &origin::Producer, path: impl moq_net::AsPath) -> Result<()> { let path = path.as_path(); let socket = self.call(Mode::Request).await?; - let catalog = moq_mux::catalog::Config::default() - .with_max_age(self.max_age) - .with_bandwidth(self.bandwidth.clone()); - serve_publish(origin, path.as_str(), socket, catalog, self.program).await + let ingest = crate::server::Ingest { + config: moq_mux::catalog::Config::default() + .with_max_age(self.max_age) + .with_bandwidth(self.bandwidth.clone()), + program: self.program, + linger: self.linger, + }; + serve_publish(origin, path.as_str(), socket, ingest).await } async fn call(&self, mode: Mode) -> Result { @@ -245,8 +263,12 @@ mod tests { let resource = request.resource().to_string(); let is_publish = matches!(request, Request::Publish(_)); if let Request::Publish(publish) = request { - // Runs until the caller disconnects; the test aborts it. - publish.accept(&origin, "ingested/cam0").await.ok(); + // Runs until the caller disconnects, then closes at once. + publish + .with_linger(Duration::ZERO) + .accept(&origin, "ingested/cam0") + .await + .ok(); } (resource, is_publish) }); diff --git a/rs/moq-srt/src/server.rs b/rs/moq-srt/src/server.rs index 05f85c39b0b..af2c04e9570 100644 --- a/rs/moq-srt/src/server.rs +++ b/rs/moq-srt/src/server.rs @@ -250,7 +250,8 @@ impl Server { max_age: None, bandwidth: moq_net::bandwidth::Allocator::unlimited(), program: None, - linger: Duration::ZERO, + linger: moq_mux::Linger::DEFAULT, + follow: Duration::ZERO, stitch: false, }; @@ -286,9 +287,11 @@ struct Pending { bandwidth: moq_net::bandwidth::Allocator, /// The programs of a multiplex an ingest publishes. Override with [`Publish::with_program`]. program: Option, - /// How long an egress waits for the same instance to return. Override with - /// [`Subscribe::with_linger`]. + /// How long an ended ingest stays up. Override with [`Publish::with_linger`]. linger: Duration, + /// How long an egress waits for the same instance to return. Override with + /// [`Subscribe::with_follow`]. + follow: Duration, /// Whether an egress follows a replacement. Override with [`Subscribe::with_stitch`]. stitch: bool, } @@ -396,6 +399,14 @@ impl Publish { self } + /// Keep the broadcast up for `linger` after the stream ends, so a subscriber that arrives + /// after the end still gets the tail. Defaults to [`moq_mux::Linger::DEFAULT`]; zero closes + /// it at once. + pub fn with_linger(mut self, linger: Duration) -> Self { + self.0.linger = linger; + self + } + /// Accept the publish: announce a broadcast at `path` in `origin` and pump the /// connection's MPEG-TS into it until the client disconnects. /// @@ -404,10 +415,9 @@ impl Publish { /// future resolves when the connection ends, so callers usually run it on its /// own task. /// - /// Once the stream ends, the finished broadcast stays up until its media is stale, so - /// subscribers still catching up drain the tail, and the future resolves after that. - /// Dropping the future ends the linger early, as does a newer publish at `path`. A - /// socket or demux error closes the broadcast at once. + /// Once the stream ends, the finished broadcast stays up for the + /// [linger](Self::with_linger), and the future resolves after that. Dropping the future + /// ends the linger early. A socket or demux error closes the broadcast at once. pub async fn accept(self, origin: &origin::Producer, path: impl moq_net::AsPath) -> Result<()> { let path = path.as_path(); let socket = self.0.request.accept(None).await?; @@ -415,7 +425,12 @@ impl Publish { let config = moq_mux::catalog::Config::default() .with_max_age(self.0.max_age) .with_bandwidth(self.0.bandwidth); - serve_publish(origin, path.as_str(), socket, config, self.0.program).await + let ingest = Ingest { + config, + program: self.0.program, + linger: self.0.linger, + }; + serve_publish(origin, path.as_str(), socket, ingest).await } /// Reject the publish with a verdict the client can distinguish on the wire. @@ -456,12 +471,12 @@ impl Subscribe { self.0.peer } - /// Wait up to `linger` for the same publisher instance to come back once the broadcast + /// Wait up to `follow` for the same publisher instance to come back once the broadcast /// ends, carrying on with the same stream. Zero (the default) closes the SRT stream at /// the broadcast's end. /// A stitched replacement must also supply its first catalog within this bound. - pub fn with_linger(mut self, linger: Duration) -> Self { - self.0.linger = linger; + pub fn with_follow(mut self, follow: Duration) -> Self { + self.0.follow = follow; self } @@ -485,7 +500,7 @@ impl Subscribe { tracing::info!(peer = %self.0.peer, %path, "SRT subscribe accepted"); let options = crate::ts::Options { latency: self.0.latency, - linger: self.0.linger, + follow: self.0.follow, stitch: self.0.stitch, }; serve_subscribe(origin, path.as_str(), socket, options).await @@ -508,17 +523,23 @@ async fn reject_log(request: ConnectionRequest, reason: Reject, peer: SocketAddr } } +/// How an ingest publishes: its catalog, which programs, and how long a clean end stays up. +pub(crate) struct Ingest { + pub config: moq_mux::catalog::Config, + pub program: Option, + pub linger: Duration, +} + /// Pump one accepted SRT socket's MPEG-TS payload into the origin (`m=publish`). pub(crate) async fn serve_publish( origin: &origin::Producer, path: &str, mut socket: SrtSocket, - config: moq_mux::catalog::Config, - program: Option, + ingest: Ingest, ) -> Result<()> { use futures::TryStreamExt; - let mut publisher = crate::ts::Publisher::new(origin, path, config, program)?; + let mut publisher = crate::ts::Publisher::new(origin, path, ingest.config, ingest.program)?; // Run the read/feed loop so an error surfaces here instead of unwinding past // the publisher, which would drop it (and its tracks) with a bare Error::Dropped. @@ -531,17 +552,16 @@ pub(crate) async fn serve_publish( .await; match result { - // The stream ended: serve the finished broadcasts until their media is stale, so - // subscribers still catching up drain the tail. srt-tokio ends the stream the same + // The stream ended: serve the finished broadcasts for the linger, so a subscriber + // that arrives after the end still gets the tail. srt-tokio ends the stream the same // way for a shutdown and a peer gone silent, so both count as clean here; the - // media up to that point is complete either way, and a reconnect cuts the linger. + // media up to that point is complete either way. Ok(()) => { drop(socket); - let finished = publisher.finish()?; - let linger = finished.iter().map(moq_net::broadcast::Finish::linger).max(); - tracing::info!(%path, ?linger, "SRT ingest ended, serving the finished broadcast until its media is stale"); - // Polled together, so each program closes at its own expiry. - futures::future::join_all(finished.into_iter().map(moq_net::broadcast::Finish::wait)).await; + let linger = ingest.linger; + let publisher = publisher.finish(linger)?; + tracing::info!(%path, ?linger, "SRT ingest ended, serving the finished broadcast"); + publisher.wait().await; Ok(()) } // The socket or demux failed: abort with the real cause so subscribers see it. @@ -584,7 +604,7 @@ pub(crate) async fn serve_subscribe( let mut egress = Egress::default(); loop { - // Keep watching the socket: a linger can hold `next` for as long as it is set, and + // Keep watching the socket: a follow can hold `next` for as long as it is set, and // nothing is sent meanwhile, so a caller who hangs up would otherwise hold the task. let next = tokio::select! { biased; @@ -1057,11 +1077,11 @@ mod tests { } } - /// Accept one `m=request` caller for `live` with `linger` and `stitch`, returning the + /// Accept one `m=request` caller for `live` with `follow` and `stitch`, returning the /// receiving socket and the egress task. async fn egress( origin: &moq_net::origin::Producer, - linger: Duration, + follow: Duration, stitch: bool, ) -> (SrtSocket, tokio::task::JoinHandle>) { let mut server = Server::bind("127.0.0.1:0".parse().unwrap(), Duration::from_millis(300)) @@ -1081,7 +1101,7 @@ mod tests { let task = tokio::spawn(async move { let _server = server; subscribe - .with_linger(linger) + .with_follow(follow) .with_stitch(stitch) .accept(&consumer, "live") .await @@ -1143,15 +1163,15 @@ mod tests { egress.abort(); } - /// An end that nothing replaces closes the SRT stream once the linger runs out, so a + /// An end that nothing replaces closes the SRT stream once the follow runs out, so a /// finished broadcast never leaves a caller connected indefinitely. #[tokio::test] - async fn an_unreplaced_end_closes_the_srt_stream_at_the_linger() { - const LINGER: Duration = Duration::from_secs(1); + async fn an_unreplaced_end_closes_the_srt_stream_once_the_follow_ends() { + const FOLLOW: Duration = Duration::from_secs(1); let (origin, driver) = moq_net::origin::Producer::new(moq_net::origin::Config::default()); tokio::spawn(moq_net::time::run(driver)); let mut live = Instance::publish(&origin, "a.aac"); - let (mut receiver, egress) = egress(&origin, LINGER, false).await; + let (mut receiver, egress) = egress(&origin, FOLLOW, false).await; live.write(5, 0); tokio::time::timeout(Duration::from_secs(10), receiver.next()) @@ -1168,15 +1188,15 @@ mod tests { } }) .await - .expect("the SRT stream stayed open past the linger"); - assert!(ended.elapsed() >= LINGER, "closed after only {:?}", ended.elapsed()); + .expect("the SRT stream stayed open past the follow"); + assert!(ended.elapsed() >= FOLLOW, "closed after only {:?}", ended.elapsed()); egress.await.unwrap().unwrap(); } - /// A caller hanging up while the egress lingers ends it then, rather than holding the - /// task and its socket until the linger runs out. + /// A caller hanging up while the egress follows ends it then, rather than holding the + /// task and its socket until the follow runs out. #[tokio::test] - async fn a_caller_hanging_up_during_the_linger_ends_the_egress() { + async fn a_caller_hanging_up_during_the_follow_ends_the_egress() { let (origin, driver) = moq_net::origin::Producer::new(moq_net::origin::Config::default()); tokio::spawn(moq_net::time::run(driver)); let mut live = Instance::publish(&origin, "a.aac"); @@ -1189,7 +1209,7 @@ mod tests { .expect("the SRT egress closed early") .unwrap(); live.finish(); - // Take everything the broadcast sent, so the egress is lingering when the caller goes. + // Take everything the broadcast sent, so the egress is following when the caller goes. while let Ok(Some(payload)) = tokio::time::timeout(Duration::from_secs(2), receiver.next()).await { payload.unwrap(); } diff --git a/rs/moq-srt/src/ts.rs b/rs/moq-srt/src/ts.rs index bfcabf395ec..360dc535d93 100644 --- a/rs/moq-srt/src/ts.rs +++ b/rs/moq-srt/src/ts.rs @@ -34,7 +34,7 @@ pub enum Program { /// Each chunk is handed straight to the TS importer, which consumes whole /// transport packets and retains any partial trailing packet internally for the /// next call (the same pattern `moq-cli import ... stdin ts` uses against stdin). -/// [`Self::finish`] hands back the finished broadcasts to linger on; dropping the +/// [`Self::finish`] keeps the finished broadcasts up for a linger; dropping the /// publisher ends them and unannounces their paths. pub struct Publisher { importer: Importer, @@ -134,9 +134,9 @@ impl Publisher { } } - /// Flush any buffered media and finish every track, the catalogs included, returning - /// each finished broadcast, which stays announced until it is dropped. - pub fn finish(mut self) -> Result> { + /// Flush any buffered media and finish every track, the catalogs included, keeping each + /// finished broadcast up for `linger`. + pub fn finish(mut self, linger: Duration) -> Result> { let finished = match &mut self.importer { Importer::One { import, @@ -145,14 +145,13 @@ impl Publisher { } => { import.finish().map_err(moq_mux::Error::from)?; catalog.finish()?; - // Taken while the importer is alive, so it holds the tracks just finished. - vec![broadcast.clone().finish()] + vec![broadcast.clone()] } Importer::All(programs) => programs.finish().map_err(moq_mux::Error::from)?, }; // The drain at end of input can publish a frame nothing vouched for. self.span.in_scope(|| self.log.finish(&self.stats())); - Ok(finished) + Ok(moq_mux::Linger::new(finished, self, linger)) } /// Abort the published tracks with `err` so subscribers see the real cause @@ -180,7 +179,7 @@ pub(crate) struct Options { /// The muxer's delay, at least [`MIN_DELAY`] ([`Subscriber::new`]). pub latency: Duration, /// How long a return or stitched replacement has to supply its first catalog. - pub linger: Duration, + pub follow: Duration, /// Follow another instance replacing the broadcast, as a full program switch. pub stitch: bool, } @@ -193,7 +192,7 @@ pub(crate) struct Options { /// the TS bytes plus the media timestamp used to pace delivery. /// /// The path's announcements drive it through [`ts::Follower`], as they drive a player: the same -/// publisher instance returning within the linger continues the stream, and a replacement either +/// publisher instance returning within the follow window continues the stream, and a replacement either /// ends it with [`moq_mux::Error::Replaced`] or, with `stitch`, switches the program. pub struct Subscriber { follower: ts::Follower, @@ -225,7 +224,7 @@ impl Subscriber { let catalog = source.catalog::(moq_mux::catalog::CatalogFormat::Hang).await?; let export = ts::Export::new(source, catalog).with_delay(options.latency.max(MIN_DELAY)); let follower = ts::Follower::new(export)? - .with_linger(options.linger) + .with_linger(options.follow) .with_stitch(options.stitch); Ok(Some(Self { follower })) } @@ -365,7 +364,7 @@ mod tests { let bytes: Vec = payload.iter().flat_map(|(_, pkt)| pkt.iter().copied()).collect(); publisher.feed(bytes.into()).unwrap(); } - drop(publisher.finish().unwrap()); + drop(publisher.finish(Duration::ZERO).unwrap()); } /// One payload-only TS packet carrying a complete PSI section (PUSI + pointer_field @@ -562,33 +561,24 @@ mod tests { .is_err(), "nothing is published on the bare path" ); - drop(publisher.finish().unwrap()); + drop(publisher.finish(Duration::ZERO).unwrap()); } - /// A finished ingest stays servable, its catalog included, until its media is stale, so a + /// A finished ingest stays servable, its catalog included, for the linger, so a /// subscriber arriving after the end still learns what the tracks were. #[tokio::test(start_paused = true)] - async fn a_finished_ingest_serves_its_catalog_until_stale() { + async fn a_finished_ingest_serves_its_catalog_for_the_linger() { let origin = produce_origin(); - let max_age = Duration::from_secs(3); - let config = moq_mux::catalog::Config::default().with_max_age(max_age); - let mut publisher = Publisher::new(&origin, "live/cam0", config, None).unwrap(); + let linger = Duration::from_secs(3); + let mut publisher = Publisher::new(&origin, "live/cam0", Default::default(), None).unwrap(); publisher.feed(Bytes::from_static(BBB5S)).unwrap(); let consumer = origin.consume(); consumer.routed("live/cam0").await.unwrap(); let broadcast = consumer.request_broadcast("live/cam0", None).await.unwrap(); - let finished = publisher.finish().unwrap(); - assert_eq!( - finished.iter().map(|finish| finish.linger()).collect::>(), - [max_age] - ); - let start = tokio::time::Instant::now(); - let wait = tokio::spawn(futures::future::join_all( - finished.into_iter().map(moq_net::broadcast::Finish::wait), - )); - tokio::time::sleep(max_age / 2).await; + let wait = tokio::spawn(publisher.finish(linger).unwrap().wait()); + tokio::time::sleep(linger / 2).await; let mut catalog = broadcast.track("catalog.json").unwrap().subscribe(None).await.unwrap(); assert!( catalog.recv_group().await.unwrap().is_some(), @@ -596,7 +586,7 @@ mod tests { ); wait.await.unwrap(); - assert_eq!(start.elapsed(), max_age); + assert_eq!(start.elapsed(), linger); assert!(broadcast.is_closed()); } @@ -726,7 +716,7 @@ mod tests { Err(_) => break, } } - drop(publisher.finish().unwrap()); + drop(publisher.finish(Duration::ZERO).unwrap()); let mut roundtrip = moq_net::broadcast::Info::new().produce(); let roundtrip_consumer = roundtrip.consume(); From 9b6c7a27db0e32d2ce72f3b5b920caff19793e73 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 08:50:23 -0700 Subject: [PATCH 3/4] fix(mux): drive Linger through poll_wait Co-Authored-By: Claude Opus 5.5 --- rs/moq-mux/src/linger.rs | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/rs/moq-mux/src/linger.rs b/rs/moq-mux/src/linger.rs index 022577edeaa..c387650b7e0 100644 --- a/rs/moq-mux/src/linger.rs +++ b/rs/moq-mux/src/linger.rs @@ -1,6 +1,6 @@ //! Keeping a cleanly ended broadcast up for subscribers that arrive late. -use std::time::Duration; +use std::{pin::Pin, task::Poll, time::Duration}; /// Keeps cleanly ended broadcasts announced, serving their finished tracks, for a fixed /// duration, then closes them. @@ -11,10 +11,10 @@ use std::time::Duration; /// `owner`, whatever holds their track producers: a finished track stops being served once /// its producers drop, so the owner drops only after the broadcasts close. /// -/// The deadline is fixed at [`new`](Self::new), however late [`wait`](Self::wait) is polled. -/// Dropping the handle closes the broadcasts at once, which is how an interrupt ends it. A -/// newer publisher announced at the same path does not end it: that one wins the path while -/// it is up, and readers that come back here find the finished catalog. +/// The deadline is fixed at [`new`](Self::new), however late [`poll_wait`](Self::poll_wait) is +/// first polled. Dropping the handle closes the broadcasts at once, which is how an interrupt +/// ends it. A newer publisher announced at the same path does not end it: that one wins the +/// path while it is up, and readers that come back here find the finished catalog. #[must_use = "dropping a Linger closes its broadcasts at once"] pub struct Linger { broadcasts: Vec, @@ -22,6 +22,8 @@ pub struct Linger { _owner: T, // `None` when the duration overflows the clock, so only an interrupt ends it. deadline: Option, + // Armed at the deadline, created on first poll. + timer: Option, } impl Linger<()> { @@ -34,7 +36,7 @@ impl Linger<()> { impl Linger { /// Keep `broadcasts` up for `duration` from now, holding `owner` until they close. /// - /// A zero `duration` closes them on the first poll of [`wait`](Self::wait). + /// A zero `duration` is due on the first poll. pub fn new( broadcasts: impl IntoIterator, owner: T, @@ -44,15 +46,23 @@ impl Linger { broadcasts: broadcasts.into_iter().collect(), _owner: owner, deadline: moq_clock::now().checked_add(duration), + timer: None, } } + /// Poll until the deadline passes on the ambient clock; drop the handle then to close the + /// broadcasts. + pub fn poll_wait(&mut self, waiter: &kio::Waiter) -> Poll<()> { + let Some(deadline) = self.deadline else { + return Poll::Pending; + }; + let timer = self.timer.get_or_insert_with(|| moq_clock::sleep_until(deadline)); + waiter.poll_future(Pin::new(timer)) + } + /// Wait for the deadline on the ambient clock, then close the broadcasts. - pub async fn wait(self) { - match self.deadline { - Some(deadline) => moq_clock::sleep_until(deadline).await, - None => std::future::pending().await, - } + pub async fn wait(mut self) { + kio::wait(|waiter| self.poll_wait(waiter)).await } } From be63e45a83df47510744bb5049ee0530ab7604d5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 09:08:26 -0700 Subject: [PATCH 4/4] doc: describe the linger in the unreleased MIGRATE entries Co-Authored-By: Claude Opus 5.5 --- rs/moq-hls/MIGRATE.md | 15 +++------------ rs/moq-mux/MIGRATE.md | 8 +------- rs/moq-rtc/MIGRATE.md | 2 +- rs/moq-rtmp/MIGRATE.md | 2 +- rs/moq-srt/MIGRATE.md | 2 +- 5 files changed, 7 insertions(+), 22 deletions(-) diff --git a/rs/moq-hls/MIGRATE.md b/rs/moq-hls/MIGRATE.md index 7f5f8bb5225..d3f820dc604 100644 --- a/rs/moq-hls/MIGRATE.md +++ b/rs/moq-hls/MIGRATE.md @@ -4,16 +4,6 @@ How to update your code for each breaking change, newest first. ## Unreleased -### `Import::finish` borrows the importer - -`Import::finish(self) -> Result` is now `Import::finish(&mut self) -> Result<()>`. -Keep the importer, which owns the finished tracks, and hold it with `moq_mux::Linger` for as long as the broadcast should stay up: - -```rust -import.finish()?; -moq_mux::Linger::new([broadcast], import, moq_mux::Linger::DEFAULT).wait().await; -``` - ### Denied broadcast references report authorization failure A rendition whose referenced broadcast is outside the source origin's scope now @@ -47,11 +37,12 @@ Set `import::Config::passthrough` to keep publishing CMAF. `import::Import::run` returns `Ok(())` once every rendition's playlist carries `EXT-X-ENDLIST` and its last segment is imported, instead of polling forever. A playlist that has already ended (`EXT-X-ENDLIST`) when a rendition first sees it is refused with the new `Error::PlaylistEnded`, from `init` and `run` alike, instead of importing its last three segments. -Call `Import::finish` then, and keep the returned `broadcast::Finish` (or await `Finish::wait`) so subscribers still catching up drain the tail. +Call `Import::finish` then, and hold the importer, which owns the finished tracks, with `moq_mux::Linger` so subscribers that arrive late still get the tail. ```rust import.run().await?; -import.finish()?.wait().await; +import.finish()?; +moq_mux::Linger::new([broadcast], import, moq_mux::Linger::DEFAULT).wait().await; ``` ### A timestamp rewind ends the import (#6426) diff --git a/rs/moq-mux/MIGRATE.md b/rs/moq-mux/MIGRATE.md index 5c333890950..5840560002e 100644 --- a/rs/moq-mux/MIGRATE.md +++ b/rs/moq-mux/MIGRATE.md @@ -6,7 +6,7 @@ How to update your code for each breaking change, newest first. ### `Programs::finish` returns the program broadcasts -`Programs::finish` now returns each program's `broadcast::Producer` instead of a `broadcast::Finish`. +`Programs::finish` now returns each program's `broadcast::Producer` instead of `()`. Hand them, with the `Programs` that owns their tracks, to the new `moq_mux::Linger` to keep the finished programs up for a fixed time from the end: ```rust @@ -102,12 +102,6 @@ Recordings published with `pes` framing no longer export; import their source ag `container::fmp4::Wire::new` returns `Result`, as `Wire::from_init` already did, and refuses any other handler instead of treating it as data. Handle the error from `Wire::new`. -### `ts::Programs::finish` returns each program's `Finish` (#5795) - -`Programs::finish` returns `anyhow::Result>` instead of `anyhow::Result<()>`. -Each handle keeps its program's broadcast servable until its media is stale, and dropping it closes that program. -Await `Finish::wait` on each handle to linger, or drop the handles to close every program now. - ### Timeline durations are hints (#5715) `timeline::Config::duration_max`, `with_duration_max`, and `DEFAULT_DURATION_MAX` are gone; set `duration_hint` with `with_duration_hint`, the expected record duration such as the GOP, and the split ceiling follows from it. diff --git a/rs/moq-rtc/MIGRATE.md b/rs/moq-rtc/MIGRATE.md index b688af26aa0..4b77d028a21 100644 --- a/rs/moq-rtc/MIGRATE.md +++ b/rs/moq-rtc/MIGRATE.md @@ -11,7 +11,7 @@ Call `server.publish_router(publisher)` and `server.subscribe_router(subscriber) ### A WHIP `DELETE` lingers -After a `DELETE`, a WHIP server session's `Response::run` finishes the ingest and resolves only once the finished broadcast goes stale. +After a `DELETE`, a WHIP server session's `Response::run` finishes the ingest and resolves only once the finished broadcast has stayed up for `server::Config::linger` (default 30 s). Drop the future to close the broadcast early. A peer that disconnects without one still closes it at once. ### Sessions return their UDP wire source (#5632) diff --git a/rs/moq-rtmp/MIGRATE.md b/rs/moq-rtmp/MIGRATE.md index 2b2f6a08ec5..26b5073d7b6 100644 --- a/rs/moq-rtmp/MIGRATE.md +++ b/rs/moq-rtmp/MIGRATE.md @@ -24,7 +24,7 @@ The default FLV release delay drops from 2 s to 500 ms, matching the `moq` CLI; ### A clean end lingers -`Publish::accept` and `Client::pull` now resolve only after a clean end's finished broadcast goes stale: an unpublish (`deleteStream`), or a remote's `NetStream.Play.Complete` or `NetStream.Play.Stop`. +`Publish::accept` and `Client::pull` now resolve only after a clean end's finished broadcast has stayed up for `with_linger` (default 30 s): an unpublish (`deleteStream`), or a remote's `NetStream.Play.Complete` or `NetStream.Play.Stop`. Drop the future to close the broadcast early. A connection that drops without one still closes at once. `Client::publish` sends `deleteStream` when its broadcast finishes, so the server ingesting it sees a clean end. diff --git a/rs/moq-srt/MIGRATE.md b/rs/moq-srt/MIGRATE.md index 23788f8a2ec..ff3f70c6dfb 100644 --- a/rs/moq-srt/MIGRATE.md +++ b/rs/moq-srt/MIGRATE.md @@ -30,7 +30,7 @@ To keep the first publisher on a path, track the paths you accepted and reject a ### A finished ingest lingers -`Publish::accept` and `Client::pull` now resolve only after the ended stream's finished broadcast goes stale. +`Publish::accept` and `Client::pull` now resolve only after the ended stream's finished broadcast has stayed up for `with_linger`. Drop the future to close the broadcast early. A socket or demux error still closes it at once. ### Egress runs at least one second behind (#5553)