diff --git a/Cargo.lock b/Cargo.lock index 05b9e2c38af..a56fa605131 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4477,6 +4477,7 @@ dependencies = [ "gstreamer", "hang", "moq-audio", + "moq-json", "moq-mux", "moq-net", "moq-tokio", diff --git a/doc/bin/gstreamer.md b/doc/bin/gstreamer.md index 02bf4803d7d..b1a762deb7a 100644 --- a/doc/bin/gstreamer.md +++ b/doc/bin/gstreamer.md @@ -104,8 +104,14 @@ rewound timestamps forward. Pads are named by kind and appear as the catalog announces renditions: `video_0`, `video_1`, `audio_0`. Link the pad you want by name; the terse `moqsrc ! decodebin3` form links only the first pad offered, which may be -audio. Each pad sends EOS when its rendition ends. A rendition that leaves the -catalog keeps its pad until its track ends, while one whose format changes is -replaced by a new pad. Properties: `url`, `broadcast`, `tls-disable-verify`. +audio. Each pad stays with its rendition for the whole session. When the +publisher restarts, or another takes over the broadcast, the pipeline switches +to the new one on the same pads, starting a new stream on each at the current +running time. A rendition whose format changes keeps its pad too, and new caps +that downstream refuses fail as not negotiated. A rendition that leaves the +catalog sends EOS once its track ends, and one the catalog adds gets a new pad. +When the broadcast ends, or its publisher goes away, its pads hold without EOS +until the broadcast is announced again. Properties: `url`, `broadcast`, +`tls-disable-verify`. Debug with `GST_DEBUG=*:4` for GStreamer and `RUST_LOG=debug` for the plugin. diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index 8652959cd75..7f9cac1e475 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -100,6 +100,5 @@ maintainer's pre-release merge of `release` into `main` keeps `main`'s - [Export ts](/quest/m0/broadcast-epoch/export-ts.md) - `export ts` and SRT egress linger only for the same epoch, and switch to a replacement only with `--stitch`, as a full program switch - [Publish catalog restart](/quest/m0/broadcast-epoch/publish-catalog-restart.md) - `@moq/publish` never reuses catalog group numbers under one name and epoch after a re-announce - [Bindings](/quest/m0/broadcast-epoch/bindings.md) - moq-ffi and every wrapper expose the epoch and let a publisher announce one -- [moqsrc](/quest/m0/broadcast-epoch/moqsrc.md) - moqsrc switches to the new broadcast on a `Restart`, keeping its pads by rendition name - [Follow gap](/quest/m0/broadcast-epoch/follow-gap.md) - a followed path reports a gap that ended its request, even onto a same-epoch covering prefix - [Stats totals and prefix tracks](/quest/m0/broadcast-epoch/stats-split.md) - the same release retires the per-path stats maps for totals and on-demand prefix tracks (decided 2026-10-05) diff --git a/quest/m0/broadcast-epoch/moqsrc.md b/quest/m0/broadcast-epoch/moqsrc.md deleted file mode 100644 index 4a2759e99eb..00000000000 --- a/quest/m0/broadcast-epoch/moqsrc.md +++ /dev/null @@ -1,82 +0,0 @@ -# [M] moqsrc follows a restart - -## Goal - -When the source behind `moqsrc`'s path restarts (an announce `Restart`), the -pipeline keeps running and switches to the new broadcast: a fresh catalog on -the same pads, with no error on the bus. Once subscriptions are sticky, -`moqsrc` would otherwise keep playing the replaced broadcast until its route -goes, then end or error. A broadcast that ends holds its pads until the path -is announced again, like a player going offline and back online. - -## Plan - -Decided in planning (2026-10-06, after #4955 found that no quest covered -`moqsrc`), re-scoped onto Restart -(2026-10-07): - -- **Follow announcements.** `moqsrc` follows the path's announcements - (`origin::Consumer::announced`) rather than resolving once with - `routed_broadcast`. It plays on `Start`, switches on `Restart` by - re-requesting the path (which resolves the new route), and ends on `End`. - An `Update` (a re-price or same-epoch failover) changes nothing, so it - stays a seamless mid-group resume. Restart owns deciding what is a source - change, so `moqsrc` never compares epochs itself. -- **Route-level restarts (maintainer, 2026-10-08).** A `Restart` names a - route, which can be a prefix covering the path (such as `pool`) rather - than the path itself. `moqsrc` switches on a `Restart` of the route its - path resolves through; the re-request re-resolves upstream, and if it - lands on the same source the run still restarts on the same pads. -- **Errors keep today's handling, except source loss.** A replacement no - longer ends the old subscriptions, so a malformed catalog or track posts - the session error or ends that pad, as today. Losing the source (the - publisher's session closing, or the route going away) is not fatal: it - enters the held state below even when it arrives before the announce - `End`, since `catalog.next()` and session completion - (`rs/moq-gst/src/source/imp.rs`) see it first today. The earlier rule treating `Unroutable` as - a switch in flight is dropped: it could not tell a switch from a dead route - and could hang. -- **Cutover.** Cancel the old pumps as soon as the `Restart` is seen, without - waiting for their subscriptions to end. -- **Pads (decided 2026-10-07, reconfirmed after the Restart re-scope).** - Keep each pad by rendition name across a switch, so a `gst-launch` - pipeline linked by name (`s.video_0 ! queue`) keeps playing. Fresh pads - were rejected: pad ids come from a process-wide - counter, so a new run's pads get new names, stay unlinked, and the pipeline - goes idle silently. Pad state outlives a pump: a pump hands its pad back to - the session instead of removing it. On the switch, push stream-start, the - new caps (even when they differ, so a codec change fails loudly as - not-negotiated rather than going quiet), and a segment whose `base` is the - current running time, so a synced sink does not drop the new run (PTS - restarts at 0) as late. An in-run caps or container change keeps its pad - the same way. A rendition the new catalog drops ends with EOS; a new or - renamed one gets a new pad. -- **Hold pads from `End` to `Start` (decided 2026-10-08, replacing the - `linger` timer).** This covers a lite-06 or moq-transport restart whose END - and START are not coalesced into a `Restart`, and a withdrawn source. A - track that ends or loses its source hands its pad back without EOS, since - tracks, the catalog, and `End` arrive on different streams in either order; - the next `Start` resumes on the same pads. The regression covers both - orders: source loss before `End`, and `End` before source loss. A rendition the catalog retired - still drains to EOS. No timer and no property: announcements already say - when the source is back, and a timer cannot tell a slow restart from a - dead source. -- **Where.** The logic stays in `rs/moq-gst/src/source`, following the path - through moq-net's `origin::Consumer::follow`, as `moq play` does. -- **Test.** An in-process test republishes the path under a newer epoch while - the old publisher stays up: the new run's buffers render through a synced - sink, a pad linked by name keeps flowing, and the bus carries no error. - Cover an epochless source change that also switches, a `Restart` of a - prefix route covering the path that switches too, a same-epoch - re-announce (`Update`) that does not, a switch with new caps that keeps the - pad, an `End` then `Start` that resumes on the same pads (media FIN, a - delay, catalog FIN, `End`, then `Start`), and an old publisher's session - closed (not finished) over a relay that resumes on the next `Start`. - -Update the `moqsrc` section of `doc/bin/gstreamer.md`: a restart keeps each -pad by rendition name, a format change keeps its pad, a dropped rendition gets -EOS, and an ended broadcast holds its pads until the next `Start`. - -Public API: no new property, but an ended broadcast no longer sends EOS on -its pads. Wire: none. - diff --git a/rs/moq-gst/Cargo.toml b/rs/moq-gst/Cargo.toml index d575ae77b4e..a7cb840bec2 100644 --- a/rs/moq-gst/Cargo.toml +++ b/rs/moq-gst/Cargo.toml @@ -20,6 +20,7 @@ anyhow = { workspace = true, features = ["backtrace"] } bytes = { workspace = true } gst = { package = "gstreamer", version = "0.23" } hang = { workspace = true } +moq-json = { workspace = true } moq-mux = { workspace = true } moq-net = { workspace = true } moq-tokio = { workspace = true, default-features = true } diff --git a/rs/moq-gst/src/source/imp.rs b/rs/moq-gst/src/source/imp.rs index fff1b9c75f1..baa1d3ecebe 100644 --- a/rs/moq-gst/src/source/imp.rs +++ b/rs/moq-gst/src/source/imp.rs @@ -1,4 +1,5 @@ -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; +use std::pin::Pin; use std::sync::atomic::{AtomicU8, AtomicU64, Ordering}; use std::sync::{Arc, LazyLock, Mutex}; use std::time::Duration; @@ -10,6 +11,7 @@ use gst::subclass::prelude::*; use tokio::sync::watch; use hang::moq_net; +use moq_net::announce::Event as AnnounceEvent; static CAT: LazyLock = LazyLock::new(|| gst::DebugCategory::new("moq-src", gst::DebugColorFlags::empty(), Some("MoQ Source Element"))); @@ -27,8 +29,8 @@ static RUNTIME: LazyLock = LazyLock::new(|| { /// first audio pad `audio_0`. That predictability matters because `gst-launch` links a /// source's sometimes-pads by name (`moqsrc name=s s.video_0 ! ...`); a single shared counter /// made the first pad's number depend on catalog arrival order (audio could claim `0`), -/// silently breaking those pipelines. Counters only ever increment, so a mid-stream reshape -/// still gets a fresh, collision-free id. +/// silently breaking those pipelines. Counters only ever increment, so a rendition that +/// appears later still gets a fresh, collision-free id. /// /// An id is claimed where the pad is created, not where its pump is spawned. A rendition whose /// subscription never resolves therefore reserves nothing, so it can't leave `video_0` pointing @@ -87,10 +89,10 @@ tokio::task_local! { static STREAMING: glib::WeakRef; } -/// The session task drives everything: it connects, follows the catalog, and -/// runs one [`Pump`] per active rendition. The element just starts and -/// stops it. No control-plane channel is needed because pumps push to their pads -/// directly from their own task (a source pad's push *is* its streaming thread), +/// The session task drives everything: it connects, follows the path's announcements, and +/// follows the catalog of the broadcast serving it with one [`Pump`] per active rendition. The +/// element just starts and stops it. No control-plane channel is needed because pumps push to +/// their pads directly from their own task (a source pad's push *is* its streaming thread), /// so there's nothing to marshal back onto the element. struct SessionController { shutdown: watch::Sender, @@ -104,12 +106,13 @@ impl SessionController { let (connection, origin) = connect(&settings)?; let task_connection = connection.clone(); let task_element = element.clone(); - Ok(Self::spawn(connection, element, move |mut shutdown| async move { + Ok(Self::spawn(connection, element, move |mut shutdown, pads| async move { run_session( &task_connection, origin, settings.broadcast, task_element, + &pads, &mut shutdown, ) .await @@ -117,17 +120,21 @@ impl SessionController { } /// Run `session` as this element's session task, reporting its error on the bus. + /// + /// The session's pads outlive every broadcast it plays, so they are removed here, once it ends. fn spawn(connection: moq_tokio::Connection, element: glib::WeakRef, session: F) -> Self where - F: FnOnce(watch::Receiver) -> Fut, + F: FnOnce(watch::Receiver, Pads) -> Fut, Fut: Future> + Send + 'static, { let (shutdown_tx, shutdown_rx) = watch::channel(false); - let task = session(shutdown_rx.clone()); + let pads = Pads::default(); + let task = session(shutdown_rx.clone(), pads.clone()); let task_connection = connection.clone(); let join = RUNTIME.spawn(STREAMING.scope(element.clone(), async move { let result = task.await; - // A broadcast that ends on its own releases the transport now rather than at stop. + pads.clear(&element); + // A session that fails on its own releases the transport now rather than at stop. task_connection.abort(moq_net::Error::Cancel); // Stopping cuts the session short, which is not a failure. if let Err(err) = result @@ -145,7 +152,7 @@ impl SessionController { } } - /// Stop the session, returning once its pumps have removed their pads and the connection closed. + /// Stop the session, returning once its pads are removed and the connection closed. /// /// A dial still running once the element reached NULL can outlive `main` (`gst-launch` exits right /// after), and aws-lc aborts the process when a thread asks it for randomness after its exit @@ -195,8 +202,8 @@ fn connect(settings: &ResolvedSettings) -> Result<(moq_tokio::Connection, moq_ne let _rt = RUNTIME.enter(); let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); - // One-shot: the catalog subscription dies with the session anyway, so a background redial could - // not resurrect this run. A drop surfaces as the catalog closing and the session winding down. + // One-shot: every subscription dies with the session anyway, so a background redial could not + // resurrect this run. A drop ends the session with the connection's error. let connection = config .init(Default::default())? .with_subscriber(origin) @@ -366,9 +373,9 @@ impl MoqSrc { } } -/// The identity we reconcile a rendition on: a change to either field tears the pad down and -/// recreates it. Caps cover codec/resolution; the container descriptor covers the wire framing -/// (e.g. legacy -> cmaf). +/// The identity we reconcile a rendition on: a change to either field restarts its pump, which +/// starts a new stream on the same pad. Caps cover codec/resolution; the container descriptor +/// covers the wire framing (e.g. legacy -> cmaf). #[derive(Clone, PartialEq)] struct Shape { caps: gst::Caps, @@ -377,9 +384,9 @@ struct Shape { /// A pump's progress, shared with its [`ActiveTrack`] so teardown and pad creation can't both /// win. A pump is torn down two different ways depending on how far it got: before it owns a pad, -/// stopping it means it must never create one; after, it owns a pad and has to drop it. One +/// stopping it means it must never take one; after, it owns a pad and has to hand it back. One /// compare-exchange settles which of the two happened, so a pad can't slip out between a -/// teardown's check and the pump's creation. +/// teardown's check and the pump taking it. struct PumpState(AtomicU8); impl PumpState { @@ -409,10 +416,10 @@ impl PumpState { /// A rendition we're currently serving, keyed in the session by moq track name. struct ActiveTrack { - /// Identity we diff against on each catalog update; a change recreates the pad. + /// Identity we diff against on each catalog update; a change restarts the pump. shape: Shape, - /// Tells the pump to drop its pad and exit (set on shutdown or when reconcile - /// replaces the rendition). + /// Tells the pump to hand its pad back and exit (set when the broadcast is stopped or + /// replaced, or when reconcile restarts the rendition). cancel: watch::Sender, /// Handle to the pump task in the session's `JoinSet`. We only read /// `is_finished()` to prune this entry once the pump ends (the `JoinSet` owns @@ -424,7 +431,7 @@ struct ActiveTrack { impl ActiveTrack { /// Tear the pump down whatever stage it reached: one still subscribing never takes a pad, - /// one that has drops it when it sees the watch. + /// one that has hands it back when it sees the watch. /// /// Terminal, so it consumes the handle: a cancelled rendition is removed from the session's /// active set, and respawns as a fresh pump if the catalog names it again. @@ -439,6 +446,7 @@ async fn run_session( origin: moq_net::origin::Consumer, broadcast: String, element: glib::WeakRef, + pads: &Pads, shutdown: &mut watch::Receiver, ) -> Result<()> { // Stop closes the connection only after the session ends, so the dial races shutdown here. @@ -447,83 +455,246 @@ async fn run_session( _ = shutdown.changed() => return Ok(()), } - // Wait for a route to cover the broadcast. Synchronous lookup would race the gossip - // of announcements that happens after the session is established. - tracing::info!(%broadcast, "waiting for broadcast to be announced"); - let broadcast = tokio::select! { - routed = origin.routed_broadcast(&broadcast) => { - routed.context("broadcast unavailable")? + // The dial is one-shot, so once it closes nothing can announce the path again. + let closed = async { + connection.closed().await.context("connection closed")?; + bail!("connection closed") + }; + follow_path(origin, &broadcast, &element, pads, shutdown, closed).await +} + +/// Follow the path's announcements for the whole session: a start plays the broadcast serving +/// it, a restart cuts over to the instance that replaced it, and an end leaves its pads held +/// until the next start. An update is the same instance, which its subscriptions ride out. +/// +/// Returns once `shutdown` fires, or with the error that ended the session: `closed`'s, a +/// malformed catalog's, or the origin closing. +async fn follow_path( + origin: moq_net::origin::Consumer, + path: &str, + element: &glib::WeakRef, + pads: &Pads, + shutdown: &mut watch::Receiver, + closed: impl Future>, +) -> Result<()> { + let mut follow = origin.follow(path)?; + tokio::pin!(closed); + tracing::info!(broadcast = %path, "waiting for broadcast to be announced"); + + let mut run: Option = None; + let result = loop { + tokio::select! { + _ = shutdown.changed() => break Ok(()), + result = &mut closed => break result, + // The broadcast ended on its own, holding its pads for the next start. + result = async { run.as_mut().expect("guarded").task.as_mut().await }, if run.is_some() => { + run = None; + if let Err(err) = result { + break Err(err); + } + } + event = follow.next() => { + let Some(event) = event else { break Err(anyhow::anyhow!("origin closed")) }; + let announce = match event { + AnnounceEvent::Start(announce) | AnnounceEvent::Restart(announce) => announce, + // With nothing playing, it is the path served again after a gap the + // announcements folded away, so play it. + AnnounceEvent::Update(announce) if run.is_none() => announce, + AnnounceEvent::Update(_) => continue, + AnnounceEvent::End(_) => { + tracing::info!(broadcast = %path, "offline, waiting for it to return"); + continue; + } + }; + tracing::info!( + broadcast = %path, + epoch = announce.route.epoch.as_ref().map(tracing::field::display), + "online" + ); + + // The old instance's pumps hand their pads back before the new one takes them. + if let Some(old) = run.take() { + old.stop().await; + } + run = Some(Run::start(&origin, path, element, pads, shutdown)); + } } - _ = shutdown.changed() => return Ok(()), }; - follow_catalog(broadcast, element, shutdown).await + if let Some(run) = run { + run.stop().await; + } + result } -/// Follow the broadcast's catalog for the whole session, keeping one [`Pump`] per -/// announced rendition in sync with it. Returns once the catalog closes and the last pump -/// drains, or `shutdown` fires. +/// The broadcast serving the path, followed until it ends or is replaced. +struct Run { + stop: watch::Sender, + task: Pin> + Send>>, +} + +impl Run { + fn start( + origin: &moq_net::origin::Consumer, + path: &str, + element: &glib::WeakRef, + pads: &Pads, + shutdown: &watch::Receiver, + ) -> Self { + let (stop, mut stopped) = watch::channel(false); + let (origin, path, element, pads, shutdown) = ( + origin.clone(), + path.to_string(), + element.clone(), + pads.clone(), + shutdown.clone(), + ); + let task = Box::pin(async move { + // Rides out the route moving between the announcement and the request. + let broadcast = tokio::select! { + routed = origin.routed_broadcast(&path) => match routed { + Ok(broadcast) => broadcast, + Err(err) => { + gst::warning!(CAT, "broadcast {path} unavailable: {err:?}"); + return Ok(()); + } + }, + _ = stopped.changed() => return Ok(()), + }; + follow_catalog(broadcast, &element, &pads, &mut stopped, &shutdown).await + }); + Self { stop, task } + } + + /// Stop following the broadcast, returning once its pumps have handed their pads back. + async fn stop(self) { + let _ = self.stop.send(true); + if let Err(err) = self.task.await { + gst::debug!(CAT, "replaced broadcast ended with error: {err:?}"); + } + } +} + +/// Whether `err`, from a track that was streaming, is its source going away (its session +/// closing, its route going) rather than something wrong with what it sent. A producer going +/// away mid-stream surfaces as almost any error, a group that can never be served included, so +/// only malformed content counts against the source. +fn source_lost(err: &moq_mux::Error) -> bool { + use moq_net::{Error, StreamError}; + match err { + moq_mux::Error::Moq(err) | moq_mux::Error::Json(moq_json::Error::Net(err)) => !matches!( + err, + Error::MalformedTrack | Error::Decode(_) | Error::Stream(StreamError::MalformedTrack) + ), + _ => false, + } +} + +/// Whether a subscribe that failed with `err` was refused by a source that is still up, or that +/// lacks the track, rather than losing its source. Little more survives a relay, where a dropped +/// producer arrives as an internal error, so anything else is a loss. +fn refused(err: &moq_net::Error) -> bool { + use moq_net::{Error, StreamError}; + matches!( + err, + Error::NotFound | Error::Unauthorized | Error::Stream(StreamError::NotFound) + ) +} + +/// Follow one broadcast's catalog, keeping one [`Pump`] per announced rendition in sync with +/// it. Returns once the catalog closes and the last pump drains, or `stop` fires. Each pump +/// hands its pad back to `pads` as it ends, so the next broadcast at the path resumes on it. +/// +/// Losing the source is not an error: the catalog and the tracks end, holding their pads. async fn follow_catalog( broadcast: moq_net::broadcast::Consumer, - element: glib::WeakRef, - shutdown: &mut watch::Receiver, + element: &glib::WeakRef, + pads: &Pads, + stop: &mut watch::Receiver, + shutdown: &watch::Receiver, ) -> Result<()> { - let catalog_track = broadcast.track(hang::catalog::Catalog::DEFAULT_NAME)?; // A publisher that never answers would otherwise hold stop until the connection closes. let catalog_track = tokio::select! { - track = catalog_track.subscribe(hang::catalog::Catalog::default_subscription()) => track?, - _ = shutdown.changed() => return Ok(()), + track = async { + broadcast + .track(hang::catalog::Catalog::DEFAULT_NAME)? + .subscribe(hang::catalog::Catalog::default_subscription()) + .await + } => match track { + Ok(track) => track, + Err(err) if refused(&err) => return Err(err).context("failed to subscribe to the catalog"), + Err(err) => { + gst::info!(CAT, "catalog lost its source before subscribing: {err}"); + return Ok(()); + } + }, + _ = stop.changed() => return Ok(()), }; let mut catalog_consumer = moq_mux::catalog::hang::Consumer::new(catalog_track); - // Follow the catalog for the whole session and reconcile our pumps against every update, + // Follow the catalog for the whole broadcast and reconcile our pumps against every update, // rather than building them once from the first frame. This covers reactive publishers // (the browser via @moq/hang) that announce an empty catalog before their encoder // configures, then add renditions a beat later, as well as renditions appearing, // disappearing, or changing codec/resolution mid-stream. let mut active: HashMap = HashMap::new(); let mut pumps: tokio::task::JoinSet<()> = tokio::task::JoinSet::new(); + // The renditions the newest catalog lists, once one has arrived. + let mut listed: Option> = None; let mut catalog_closed = false; - loop { + let result = loop { // Prune metadata for pumps that have ended (the JoinSet has already reaped the // tasks). Once the catalog is closed and the last pump drains we're done: each - // emitted EOS (or a pad drop on error) downstream via its own end path. + // handed its pad back, or ended it on error. active.retain(|_, track| !track.task.is_finished()); if catalog_closed && pumps.is_empty() { - break; + break Ok(()); } tokio::select! { - // Full session shutdown: break to the drain below. - _ = shutdown.changed() => break, - // A pump finished; loop back so the `retain` above prunes its entry and the - // break condition sees the drained set. - _ = pumps.join_next(), if !pumps.is_empty() => {} + // Stopped or replaced: break to the drain below. + _ = stop.changed() => break Ok(()), + // A pump finished and handed its pad back, which ends there if the catalog no longer + // lists it. Loop back so the `retain` above prunes its entry and the break condition + // sees the drained set. + _ = pumps.join_next(), if !pumps.is_empty() => { + if let Some(listed) = &listed { + pads.retire(listed, element); + } + } // The guard stops us polling a closed catalog track (which would spin the loop // returning None) while we wait for the remaining pumps to drain. - next = catalog_consumer.next(), if !catalog_closed => { - match next? { - Some(catalog) => reconcile(&catalog, &mut active, &mut pumps, &broadcast, &element, shutdown), - // Catalog track closed. Don't cancel the pumps: let each reach its - // natural Ok(None) -> EOS end so downstream sees a clean EOS rather than a - // bare pad drop. We just stop reconciling and wait for them to drain. - // - // That includes a pump still resolving its subscription, which is - // indistinguishable here from one that has simply not been polled yet: - // a final snapshot naming a track the publisher does serve arrives this - // way, and cancelling on "not live yet" would drop it. Whether a - // rendition will ever be served is the publisher's to answer, and - // ending the broadcast is where it does: every name it never served - // resolves with an error, which fails that pump's subscribe and ends - // it here. - None => catalog_closed = true, + next = catalog_consumer.next(), if !catalog_closed => match next { + Ok(Some(catalog)) => { + let names = reconcile(&catalog, &mut active, &mut pumps, &broadcast, element, pads, shutdown); + pads.retire(&names, element); + listed = Some(names); + } + // Catalog track closed. Don't cancel the pumps: let each reach its natural end, + // which holds its pad for the next broadcast. We just stop reconciling and wait + // for them to drain. + // + // That includes a pump still resolving its subscription, which is + // indistinguishable here from one that has simply not been polled yet: + // a final snapshot naming a track the publisher does serve arrives this + // way, and cancelling on "not live yet" would drop it. Whether a + // rendition will ever be served is the publisher's to answer, and + // ending the broadcast is where it does: every name it never served + // resolves with an error, which fails that pump's subscribe and ends + // it here. + Ok(None) => catalog_closed = true, + // The same for a source that went away: its tracks end the same way. + Err(err) if source_lost(&err) => { + gst::info!(CAT, "catalog lost its source: {err}"); + catalog_closed = true; } + Err(err) => break Err(err.into()), } } - } + }; - // Shutdown: cancel every pump, then wait for them all to drop their pads. Cancel all up front + // Cancel every pump, then wait for them all to hand their pads back. Cancel all up front // (pumps only exit on their own `cancel`), or the not-yet-cancelled ones would keep streaming // while we await the rest. // On the clean catalog-closed exit `active`/`pumps` are already drained, so this is a no-op. @@ -532,12 +703,64 @@ async fn follow_catalog( } while pumps.join_next().await.is_some() {} - Ok(()) + result +} + +/// A rendition's pad between the pumps that serve it, so a restarted broadcast, a format +/// change, or a source coming back resumes on the pad downstream linked by name. The pump +/// streaming to the pad holds the lock, so the next one waits for it to hand the pad back. +type Slot = Arc>>; + +/// The session's pads by rendition name, outliving every broadcast it plays. +#[derive(Clone, Default)] +struct Pads(Arc>>); + +impl Pads { + fn slot(&self, name: &str) -> Slot { + self.0.lock().unwrap().entry(name.to_string()).or_default().clone() + } + + /// End the held pads of the renditions `listed` leaves out, with EOS. A pad still streaming + /// is left to its pump and ends here once handed back. + /// + /// The slots stay: a pump may hold one it has yet to lock, and must hand its pad back where + /// this finds it. + fn retire(&self, listed: &HashSet, element: &glib::WeakRef) { + let retired: Vec = self + .0 + .lock() + .unwrap() + .iter() + .filter(|(name, _)| !listed.contains(*name)) + .filter_map(|(_, slot)| slot.try_lock().ok()?.take()) + .collect(); + for pad in retired { + let _ = tokio::task::block_in_place(|| pad.push_event(gst::event::Eos::builder().build())); + remove_pad(element, &pad); + } + } + + /// Remove every pad, once no pump is left to hold one. + fn clear(&self, element: &glib::WeakRef) { + let slots: Vec = self.0.lock().unwrap().drain().map(|(_, slot)| slot).collect(); + for slot in slots { + if let Some(pad) = slot.try_lock().ok().and_then(|mut held| held.take()) { + remove_pad(element, &pad); + } + } + } +} + +fn remove_pad(element: &glib::WeakRef, pad: &gst::Pad) { + let _ = pad.set_active(false); + if let Some(obj) = element.upgrade() { + let _ = obj.remove_pad(pad); + } } /// Bring the live set of pumps in line with `catalog`: spawn pumps for newly announced -/// renditions, recreate any whose caps or container changed, and leave ones that vanished to end -/// with their track. +/// renditions, restart any whose caps or container changed, and leave ones that vanished to end +/// with their track. Returns the renditions the catalog lists. /// /// Infallible by design: every way a single rendition can be unusable (unsupported codec, /// malformed init, a name the broadcast refuses) skips just that rendition, so one bad entry in @@ -548,8 +771,9 @@ fn reconcile( pumps: &mut tokio::task::JoinSet<()>, broadcast: &moq_net::broadcast::Consumer, element: &glib::WeakRef, + pads: &Pads, shutdown: &watch::Receiver, -) { +) -> HashSet { struct Desired { kind: TrackKind, shape: Shape, @@ -588,13 +812,13 @@ fn reconcile( &active.iter().map(|(name, t)| (name.clone(), t.shape.clone())).collect(), ); - // Drop anything that changed shape; each cancelled pump drops its own pad. Changed renditions - // also land in `plan.add`, so they respawn below under a fresh pad id. + // Stop anything that changed shape; each cancelled pump hands its pad back. Changed renditions + // also land in `plan.add`, so they respawn below and take the same pad. // // A rendition that vanished is only no longer selectable: a publisher retires one by // delisting it and finishing its track, and the two arrive in either order. A pump that owns - // a pad stays in `active` and ends with its track (EOS, or a pad drop on error), where - // `follow_catalog` prunes it. One that never took a pad is stopped. + // a pad stays in `active` and ends with its track, where `follow_catalog` prunes it and ends + // the pad it hands back. One that never took a pad is stopped. for name in plan.remove { if !desired.contains_key(&name) && active.get(&name).is_some_and(|track| !track.state.cancel_before_live()) { continue; @@ -648,6 +872,7 @@ fn reconcile( caps: d.shape.caps.clone(), track, container, + slot: pads.slot(&name), state: state.clone(), cancel: cancel_rx, shutdown: shutdown.clone(), @@ -667,6 +892,8 @@ fn reconcile( }, ); } + + desired.into_keys().collect() } /// Tear-down / spawn decisions for one catalog update, computed purely from the desired and @@ -691,13 +918,12 @@ fn plan_reconcile(desired: &HashMap, active: &HashMap` / `audio_` from a +/// Identifies a rendition's pad. Pads are named `video_` / `audio_` from a /// per-kind, process-unique counter (matching the `%u` templates) rather than after -/// the track name, so a rendition can be torn down and recreated (when its -/// codec/resolution changes mid-stream) without two pads ever sharing a name. The -/// first *live* pad of each kind is `video_0` / `audio_0`, so `gst-launch` can link them by -/// name regardless of which rendition the catalog announces first, or how many it announces -/// that never arrive. +/// the track name, so a rendition that ends and is later announced again never shares a +/// name with a pad still being removed. The first *live* pad of each kind is +/// `video_0` / `audio_0`, so `gst-launch` can link them by name regardless of which rendition +/// the catalog announces first, or how many it announces that never arrive. struct TrackDescriptor { kind: TrackKind, name: String, @@ -734,6 +960,8 @@ struct Pump { caps: gst::Caps, track: moq_net::track::Consumer, container: moq_mux::catalog::hang::Container, + /// The rendition's pad, held between the pumps that serve it. + slot: Slot, /// Shared with this rendition's [`ActiveTrack::state`]. state: Arc, cancel: watch::Receiver, @@ -742,9 +970,13 @@ struct Pump { } impl Pump { - /// Subscribe to the track, then read its frames and push them to a pad owned for the rest of - /// this pump's lifetime: create the pad, stream buffers, and remove the pad on exit. Runs - /// until the track ends (EOS), errors, or `cancel` fires. + /// Subscribe to the track, then read its frames and push them to the rendition's pad: take + /// it from the slot (or create it), start a new stream on it, and stream buffers. Runs until + /// the track ends, errors, or `cancel` fires. + /// + /// The pad goes back to the slot when the track ends or loses its source, or the pump is + /// cancelled, so the next pump resumes on it. A broken pad (a flow error, a malformed track) + /// is removed instead. async fn run(self) { let Pump { element, @@ -753,6 +985,7 @@ impl Pump { caps, track, container, + slot, state, mut cancel, shutdown, @@ -783,12 +1016,25 @@ impl Pump { return; } + // Wait for the pump that last served this rendition to hand its pad back. + let mut held = tokio::select! { + biased; + _ = cancel.changed() => return, + held = slot.lock_owned() => held, + }; // The pad appears only once the track is live, so a pad downstream can link to is a promise // that the rendition is actually flowing, and only a rendition that gets this far claims a - // pad id. - let descriptor = TrackDescriptor::claim(kind, name); - let Some(pad) = create_pad(&element, &descriptor, &caps) else { - return; + // pad id. A pad held from an earlier pump starts a new stream instead, with caps that may + // differ: downstream that can't take them fails the first push as not negotiated. + let pad = match held.take() { + Some(pad) => { + tokio::task::block_in_place(|| start_stream(&pad, &name, &caps)); + pad + } + None => match create_pad(&element, &TrackDescriptor::claim(kind, name.clone()), &caps) { + Some(pad) => pad, + None => return, + }, }; // Stop flushes only the pads it finds, and this one may have been added just after, while // this pump's cancel is still on its way. Flushing it here keeps a push from blocking stop. @@ -797,42 +1043,71 @@ impl Pump { } let mut reference_ts = None; - loop { + let hold = loop { tokio::select! { - // This rendition is being torn down (shutdown, or replaced by a catalog update). - _ = cancel.changed() => break, + // This rendition is being torn down (stopped, replaced, or reshaped). + _ = cancel.changed() => break true, frame = track.read() => match frame { Ok(Some(frame)) => { - let buffer = build_buffer(frame, &mut reference_ts, descriptor.kind); + // The segment waits for the first buffer so the stream starts at the + // running time it reaches downstream, not at the one it had when the + // pad was taken, and isn't late at a synced sink. + let segment = reference_ts.is_none().then(|| live_segment(&element)); + let buffer = build_buffer(frame, &mut reference_ts, kind); // pad.push() blocks until downstream accepts the buffer (full queues, a // clock-synced sink). block_in_place hands our sibling tasks to another // worker so a stalled downstream can't pin a runtime thread and starve // the session loop or other pumps. - if tokio::task::block_in_place(|| pad.push(buffer)).is_err() { - break; + let pushed = tokio::task::block_in_place(|| { + if let Some(segment) = segment { + pad.push_event(segment); + } + pad.push(buffer) + }); + match pushed { + Ok(_) => {} + // Stop is flushing the pad, which it removes once the session ends. + Err(gst::FlowError::Flushing) => break true, + // Nothing downstream wants this rendition. + Err(gst::FlowError::NotLinked | gst::FlowError::Eos) => break false, + Err(err) => { + if let Some(obj) = element.upgrade() { + gst::element_error!( + obj, + gst::StreamError::Failed, + ("Internal data flow error."), + ["pad {} stopped: {err:?}", pad.name()] + ); + } + break false; + } } } - Ok(None) => { - let _ = tokio::task::block_in_place(|| pad.push_event(gst::event::Eos::builder().build())); - break; + // The track ended, which is not the rendition ending: the catalog says + // whether it returns, so the pad is held without EOS until it does. + Ok(None) => break true, + Err(err) if source_lost(&err) => { + gst::info!(CAT, "track {name} lost its source: {err}"); + break true; } Err(err) => { - gst::warning!(CAT, "track {} failed: {err:?}", descriptor.name); - break; + gst::warning!(CAT, "track {name} failed: {err:?}"); + break false; } } } - } + }; - let _ = pad.set_active(false); - if let Some(obj) = element.upgrade() { - let _ = obj.remove_pad(&pad); + if hold { + *held = Some(pad); + } else { + remove_pad(&element, &pad); } } } -/// Create, activate, and add a src pad for the track, seeding it with the sticky -/// stream-start/caps/segment events. Returns `None` if the element is already gone. +/// Create, activate, and add a src pad for the track, starting its stream. Returns `None` if the +/// element is already gone. fn create_pad( element: &glib::WeakRef, descriptor: &TrackDescriptor, @@ -846,16 +1121,35 @@ fn create_pad( .build(); pad.set_active(true).ok()?; + start_stream(&pad, &descriptor.name, caps); + + obj.add_pad(&pad).ok()?; + Some(pad) +} + +/// Start a new stream on `pad` with the sticky stream-start and caps events. Its segment follows +/// with the first buffer. +fn start_stream(pad: &gst::Pad, stream_id: &str, caps: &gst::Caps) { pad.push_event( - gst::event::StreamStart::builder(&descriptor.name) + gst::event::StreamStart::builder(stream_id) .group_id(gst::GroupId::next()) .build(), ); pad.push_event(gst::event::Caps::new(caps)); - pad.push_event(gst::event::Segment::new(&gst::FormattedSegment::::new())); +} - obj.add_pad(&pad).ok()?; - Some(pad) +/// A segment starting at the element's current running time, so a stream that starts mid-run +/// lines up with the clock rather than replaying from zero. +fn live_segment(element: &glib::WeakRef) -> gst::Event { + // Running time only advances while playing; before that, the stream starts at zero. + let base = element + .upgrade() + .filter(|obj| obj.current_state() == gst::State::Playing) + .and_then(|obj| obj.current_running_time()) + .unwrap_or(gst::ClockTime::ZERO); + let mut segment = gst::FormattedSegment::::new(); + segment.set_base(base); + gst::event::Segment::new(&segment) } /// Wrap a decoded frame in a gst buffer, assigning a pts relative to the track's first frame. @@ -1094,7 +1388,7 @@ mod session_tests { use hang::catalog::{AudioCodec, AudioConfig, Container, H264, VideoConfig}; use tokio::sync::watch; - use super::{NEXT_VIDEO_PAD_ID, ResolvedSettings, SessionController, follow_catalog}; + use super::{NEXT_VIDEO_PAD_ID, Pads, ResolvedSettings, SessionController, follow_catalog}; /// The pad-id counters are process-global, so a test reading one has to be the only test /// allocating while it runs. `cargo test` shares a process across tests (nextest doesn't), @@ -1138,6 +1432,23 @@ mod session_tests { .collect() } + /// Follow `broadcast`'s catalog until the returned sender stops it, removing its pads once it + /// ends, as the session does. + fn follow( + element: &super::super::MoqSrc, + broadcast: moq_net::broadcast::Consumer, + ) -> (watch::Sender, tokio::task::JoinHandle>) { + let (stop, stopped) = watch::channel(false); + let weak = element.downgrade(); + let session = super::RUNTIME.spawn(async move { + let pads = Pads::default(); + let result = follow_catalog(broadcast, &weak, &pads, &mut stopped.clone(), &stopped).await; + pads.clear(&weak); + result + }); + (stop, session) + } + /// Block until a consumer asks the broadcast for a track, returning the request unanswered so /// its subscriber stays parked. Bounded so a session that never subscribes fails the test. fn await_request(dynamic: &mut moq_net::broadcast::Dynamic) -> moq_net::track::Request { @@ -1180,10 +1491,7 @@ mod session_tests { guard.audio.renditions = BTreeMap::from([("audio".to_string(), audio_rendition())]); } - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); // Wait for the audio subscription before announcing video, and hold the request // unanswered. A catalog consumer skips to the newest snapshot, so without this the @@ -1228,10 +1536,7 @@ mod session_tests { guard.video.renditions = BTreeMap::from([("video".to_string(), video_rendition())]); } - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); await_pad(&element, "video_"); @@ -1257,10 +1562,7 @@ mod session_tests { guard.video.renditions = BTreeMap::from([("stalled".to_string(), video_rendition())]); } - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); // Hold the subscription pending, then drop the rendition from the catalog so its pump // is cancelled while it is still waiting. @@ -1313,10 +1615,7 @@ mod session_tests { } catalog.finish().unwrap(); - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); await_pad(&element, "video_"); @@ -1326,8 +1625,9 @@ mod session_tests { /// The flip side of the test above: a rendition the publisher reserved and never served is /// answerable once the publisher ends the broadcast, which resolves every name it never - /// filled. That pump must end on its own so the session terminates on the media it was - /// serving, rather than holding the connection open until the element is stopped. + /// filled. That pump must end on its own so the broadcast ends on the media it was serving, + /// rather than waiting for one that replaces it. The served rendition's pad is held for that + /// one, without EOS. #[test] fn a_rendition_nobody_served_ends_with_the_broadcast() { let _pad_ids = pad_ids(); @@ -1347,12 +1647,13 @@ mod session_tests { guard.audio.renditions = BTreeMap::from([("audio".to_string(), audio_rendition())]); } - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (stop, stopped) = watch::channel(false); + let pads = Pads::default(); + let (consumer, weak, held) = (broadcast.consume(), element.downgrade(), pads.clone()); + let session = super::RUNTIME + .spawn(async move { follow_catalog(consumer, &weak, &held, &mut stopped.clone(), &stopped).await }); - // Watch for the EOS on the served rendition, before anything can end its track. + // Watch for an EOS on the served rendition, before anything can end its track. let pad = await_pad(&element, "video_"); let eos = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); let seen = eos.clone(); @@ -1370,14 +1671,35 @@ mod session_tests { video.finish().unwrap(); broadcast.close(); - // No shutdown is sent: the session has to end on the media draining alone. + // No stop is sent: the broadcast has to end on the media draining alone. super::RUNTIME .block_on(async { tokio::time::timeout(Duration::from_secs(10), session).await }) - .expect("the session outlived the renditions it was serving") + .expect("the broadcast outlived the renditions it was serving") .unwrap() .unwrap(); - assert!(eos.load(Ordering::Relaxed), "the served rendition never emitted EOS"); - drop(shutdown); + assert!(pad.parent().is_some(), "the served rendition's pad was not held"); + assert!(!eos.load(Ordering::Relaxed), "the held pad emitted EOS"); + pads.clear(&element.downgrade()); + drop(stop); + } + + /// A broadcast that is up without a catalog has not lost its source: the session fails, + /// rather than holding for a catalog that never comes. + #[test] + fn a_broadcast_without_a_catalog_fails_the_session() { + let element = element(); + let broadcast = moq_net::broadcast::Info::new().produce(); + let (_stop, session) = follow(&element, broadcast.consume()); + + let err = super::RUNTIME + .block_on(async { tokio::time::timeout(Duration::from_secs(10), session).await }) + .expect("the session held for a catalog that never comes") + .unwrap() + .expect_err("a missing catalog was taken for a lost source"); + assert!( + matches!(err.downcast_ref::(), Some(moq_net::Error::NotFound)), + "{err:?}" + ); } /// A publisher that ends one rendition finishes its track and retires it from the catalog, @@ -1408,10 +1730,7 @@ mod session_tests { guard.video.renditions = BTreeMap::from([("video".to_string(), video_rendition())]); } - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); // Count what reaches the pad. The pad has no peer, so the probe swallows each buffer, which // reports the push as OK. @@ -1475,8 +1794,8 @@ mod session_tests { } await_pad(&element, "audio_"); - // A pump the update cancelled still has to notice. It takes its pad with it when it does, - // so the wait is only ever served in full by one that is still reading. + // A pump the update cancelled still has to notice. Its pad ends once it hands it back, so + // the wait is only ever served in full by one that is still reading. for _ in 0..10 { if pads(&element, "video_").is_empty() { break; @@ -1490,7 +1809,7 @@ mod session_tests { } producer.finish().unwrap(); - // The pump removes its pad on the way out, whichever way it goes. + // The pad ends once the pump hands it back, since the catalog no longer lists it. for _ in 0..100 { if pads(&element, "video_").is_empty() { break; @@ -1542,29 +1861,15 @@ mod session_tests { } }); - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); let pad = await_pad(&element, "video_"); let buffers = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)); - let eos = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); - let (counted, seen) = (buffers.clone(), eos.clone()); - pad.add_probe( - gst::PadProbeType::BUFFER | gst::PadProbeType::EVENT_DOWNSTREAM, - move |_, info| match &info.data { - Some(gst::PadProbeData::Buffer(_)) => { - counted.fetch_add(1, Ordering::Relaxed); - gst::PadProbeReturn::Drop - } - Some(gst::PadProbeData::Event(event)) if event.type_() == gst::EventType::Eos => { - seen.store(true, Ordering::Relaxed); - gst::PadProbeReturn::Ok - } - _ => gst::PadProbeReturn::Ok, - }, - ); + let counted = buffers.clone(); + pad.add_probe(gst::PadProbeType::BUFFER, move |_, _| { + counted.fetch_add(1, Ordering::Relaxed); + gst::PadProbeReturn::Drop + }); let mut producer = moq_mux::container::Producer::new( video, @@ -1610,18 +1915,11 @@ mod session_tests { guard.video.renditions = BTreeMap::from([("video".to_string(), video_rendition())]); } - // The same track resumes with a new group, then ends. + // The same track resumes with a new group. for i in HEAD..HEAD + TAIL { write(i); } wait_for(HEAD + TAIL); - producer.finish().unwrap(); - for _ in 0..100 { - if pads(&element, "video_").is_empty() { - break; - } - std::thread::sleep(Duration::from_millis(50)); - } let _ = shutdown.send(true); super::RUNTIME.block_on(session).unwrap().unwrap(); @@ -1636,7 +1934,156 @@ mod session_tests { HEAD + TAIL, "the first pad lost frames" ); - assert!(eos.load(Ordering::Relaxed), "the track's end never reached the pad"); + } + + /// Poll until `ready`, failing the test with `what` if it never is. + fn eventually(what: &str, ready: impl Fn() -> bool) { + for _ in 0..100 { + if ready() { + return; + } + std::thread::sleep(Duration::from_millis(50)); + } + panic!("{what}"); + } + + /// Announce `live` on `origin` with one video rendition, returning what keeps it up. + fn announce( + origin: &moq_net::origin::Producer, + ) -> ( + moq_net::broadcast::Producer, + moq_mux::catalog::Producer, + moq_net::track::Producer, + ) { + let mut broadcast = origin.create_broadcast("live").unwrap(); + let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let video = broadcast + .create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + { + let mut guard = catalog.modify().unwrap(); + guard.video.renditions = BTreeMap::from([("video".to_string(), video_rendition())]); + } + broadcast + .announce(moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint())) + .unwrap(); + (broadcast, catalog, video) + } + + /// A source that goes away (its session closing, not finishing) holds its pads for the next + /// one to start. Its tracks failing and its announcement ending travel separately, so the + /// hold can't depend on which reaches the session first. + #[test] + fn a_lost_source_holds_its_pads_for_the_next_start() { + for end_first in [false, true] { + let _pad_ids = pad_ids(); + let element = element(); + let origin = { + let _runtime = super::RUNTIME.enter(); + moq_tokio::origin::spawn() + }; + let slots = Pads::default(); + let (shutdown, mut shutdown_rx) = watch::channel(false); + let (consumer, weak, session_slots) = (origin.consume(), element.downgrade(), slots.clone()); + let session = super::RUNTIME.spawn(async move { + let closed = std::future::pending::>(); + super::follow_path(consumer, "live", &weak, &session_slots, &mut shutdown_rx, closed).await + }); + + let (old, catalog, video) = announce(&origin); + let pad = await_pad(&element, "video_"); + let mut follow = origin.consume().follow("live").unwrap(); + let mut announced = || { + super::RUNTIME + .block_on(async { tokio::time::timeout(Duration::from_secs(10), follow.next()).await }) + .expect("the path never changed") + .unwrap() + }; + assert!(matches!(announced(), moq_net::announce::Event::Start(_))); + + let lose = move || { + video.abort(moq_net::Error::Cancel).unwrap(); + drop(catalog); + }; + let held = || slots.slot("video").try_lock().is_ok_and(|held| held.is_some()); + if end_first { + old.unannounce(); + assert!(matches!(announced(), moq_net::announce::Event::End(_))); + lose(); + eventually("the lost source's pad was never held", held); + } else { + lose(); + eventually("the lost source's pad was never held", held); + old.unannounce(); + assert!(matches!(announced(), moq_net::announce::Event::End(_))); + } + assert_eq!(pads(&element, "video_"), vec![pad.clone()], "end_first={end_first}"); + + // The next source to start takes the held pad. + let _new = announce(&origin); + eventually("the next source never took the held pad", || { + slots.slot("video").try_lock().is_err() + }); + assert_eq!(pads(&element, "video_"), vec![pad], "end_first={end_first}"); + + let _ = shutdown.send(true); + super::RUNTIME.block_on(session).unwrap().unwrap(); + slots.clear(&element.downgrade()); + } + } + + /// A rendition whose caps change mid-run starts a new stream with them on the pad downstream + /// already linked, rather than replacing it. + #[test] + fn a_reshaped_rendition_keeps_its_pad() { + let _pad_ids = pad_ids(); + let element = element(); + let (broadcast, mut catalog, _producer) = video_broadcast(); + + let added = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)); + let count = added.clone(); + element.connect_pad_added(move |_, _| { + count.fetch_add(1, Ordering::Relaxed); + }); + + let (shutdown, session) = follow(&element, broadcast.consume()); + let pad = await_pad(&element, "video_"); + let caps = std::sync::Arc::new(Mutex::new(Vec::new())); + let seen = caps.clone(); + pad.add_probe(gst::PadProbeType::EVENT_DOWNSTREAM, move |_, info| { + if let Some(gst::PadProbeData::Event(event)) = &info.data + && let gst::EventView::Caps(event) = event.view() + { + seen.lock().unwrap().push(event.caps_owned()); + } + gst::PadProbeReturn::Ok + }); + + // The same rendition, now carrying its parameter sets out of band. + { + let mut rendition = video_rendition(); + rendition.description = Some(bytes::Bytes::from_static(&[1, 0x42, 0x00, 0x1f, 0xff, 0xe0, 0x00])); + let mut guard = catalog.modify().unwrap(); + guard.video.renditions = BTreeMap::from([("video".to_string(), rendition)]); + } + let reshaped = || { + caps.lock().unwrap().iter().any(|caps| { + caps.structure(0) + .is_some_and(|s| s.get::<&str>("stream-format") == Ok("avc")) + }) + }; + for _ in 0..100 { + if reshaped() { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + assert!(reshaped(), "the new caps never reached the pad"); + assert_eq!(pads(&element, "video_"), vec![pad]); + assert_eq!(added.load(Ordering::Relaxed), 1, "the reshape added a pad"); + + let _ = shutdown.send(true); + super::RUNTIME.block_on(session).unwrap().unwrap(); } /// Pipelines link `moqsrc`'s pads by name, so the first video rendition that actually @@ -1662,10 +2109,7 @@ mod session_tests { } let first = NEXT_VIDEO_PAD_ID.load(Ordering::Relaxed); - let (shutdown, mut shutdown_rx) = watch::channel(false); - let consumer = broadcast.consume(); - let weak = element.downgrade(); - let session = super::RUNTIME.spawn(async move { follow_catalog(consumer, weak, &mut shutdown_rx).await }); + let (shutdown, session) = follow(&element, broadcast.consume()); await_pad(&element, "audio_"); @@ -1711,7 +2155,9 @@ mod session_tests { let session = SessionController::spawn( connection.clone(), element.downgrade(), - move |mut shutdown| async move { follow_catalog(broadcast, weak, &mut shutdown).await }, + move |shutdown, pads| async move { + follow_catalog(broadcast, &weak, &pads, &mut shutdown.clone(), &shutdown).await + }, ); (session, connection) } @@ -1833,7 +2279,8 @@ mod session_tests { } /// A pump whose subscription resolves while stop flushes the pads adds its own just after, with - /// its cancel still on the way. A push on that pad must not block stop either. + /// its cancel still on the way. A push on that pad must not block stop either, and the pad + /// goes back to its slot for the session to remove. #[test] fn a_pad_added_after_stop_flushed_does_not_block() { let _pad_ids = pad_ids(); @@ -1848,6 +2295,7 @@ mod session_tests { let (_cancel, cancel) = watch::channel(false); let (_shutdown, shutdown) = watch::channel(true); + let slot = super::Slot::default(); let pump = super::Pump { element: element.downgrade(), kind: super::TrackKind::Video, @@ -1855,6 +2303,7 @@ mod session_tests { caps: gst::Caps::new_empty_simple("video/x-h264"), track: broadcast.consume().track("video").unwrap(), container: moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), + slot: slot.clone(), state: std::sync::Arc::new(super::PumpState::new()), cancel, shutdown, @@ -1863,7 +2312,13 @@ mod session_tests { .block_on(async { tokio::time::timeout(Duration::from_secs(10), super::RUNTIME.spawn(pump.run())).await }) .expect("a push on a pad added after the flush blocked") .unwrap(); - assert!(pads(&element, "video_").is_empty(), "the pad outlived its pump"); + let held = slot + .try_lock() + .unwrap() + .take() + .expect("the pump never handed its pad back"); + assert_eq!(pads(&element, "video_"), vec![held.clone()]); + super::remove_pad(&element.downgrade(), &held); } /// A bus sync or pad handler can stop the element from inside a pump's push. Waiting for the diff --git a/rs/moq-gst/tests/source.rs b/rs/moq-gst/tests/source.rs new file mode 100644 index 00000000000..fbdaf3ddea7 --- /dev/null +++ b/rs/moq-gst/tests/source.rs @@ -0,0 +1,584 @@ +//! `moqsrc` following a broadcast through a relay, end to end: a pipeline playing a path over a +//! real QUIC session while its publishers come, go, and replace each other. +//! +//! The relay is a server sharing one origin between every session, on its own runtime. Publishers +//! publish into that origin directly, or over a session of their own where the test is about the +//! session. The pipeline renders through a synced sink with a video sink's lateness budget, so a +//! stream that starts behind the clock drops instead of rendering. + +use std::collections::BTreeMap; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, Once, mpsc}; +use std::time::{Duration, Instant}; + +use gst::prelude::*; +use hang::catalog::{Container, H264, VideoConfig}; +use moq_net::Epoch; +use moq_net::origin::Route; + +const TIMEOUT: Duration = Duration::from_secs(10); +const FRAME: Duration = Duration::from_millis(33); +/// Enough rendered buffers to call a stream flowing. +const FLOWING: usize = 10; +/// Room to connect or switch before a stream renders at its frame rate. A sink dropping late +/// buffers still renders one a second, so a stream behind the clock can't keep up. +const FLOW_TIMEOUT: Duration = Duration::from_secs(5); + +fn init() { + static INIT: Once = Once::new(); + INIT.call_once(|| { + gst::init().unwrap(); + gstmoq::plugin_register_static().expect("register moq plugin"); + }); +} + +/// A new publisher instance. +fn route() -> Route { + Route::default().with_epoch(Epoch::mint()) +} + +fn h264() -> VideoConfig { + let mut config = VideoConfig::new(H264 { + profile: 0x42, + constraints: 0x00, + level: 0x1f, + inline: false, + }); + config.container = Container::Legacy; + config +} + +/// [`h264`] with its parameter sets out of band, which changes its caps. +fn avc() -> VideoConfig { + let mut config = h264(); + config.description = Some(bytes::Bytes::from_static(&[1, 0x42, 0x00, 0x1f, 0xff, 0xe0, 0x00])); + config +} + +/// A relay: one origin shared by every session it accepts. +struct Relay { + runtime: Option, + origin: moq_net::origin::Producer, + port: u16, +} + +impl Relay { + fn start() -> Self { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .expect("build relay runtime"); + let (origin, port) = runtime.block_on(async { + let origin = moq_tokio::origin::spawn(); + let mut config = moq_tokio::listen::Config::default(); + config.bind = Some("127.0.0.1:0".parse().unwrap()); + config.tls.generate = vec!["localhost".into()]; + let mut server = config + .init(Default::default()) + .expect("server init") + .listen() + .await + .expect("listen"); + let port = server.local_addr().expect("local addr").port(); + + let shared = origin.clone(); + tokio::spawn(async move { + while let Some(request) = server.accept().await { + let origin = shared.clone(); + tokio::spawn(async move { + if let Ok(session) = request + .with_publisher(&origin) + .with_subscriber(origin.clone()) + .ok() + .await + { + session.closed().await; + } + }); + } + }); + (origin, port) + }); + + Self { + runtime: Some(runtime), + origin, + port, + } + } + + fn url(&self) -> String { + format!("https://127.0.0.1:{}/", self.port) + } + + fn runtime(&self) -> &tokio::runtime::Runtime { + self.runtime.as_ref().expect("relay running") + } + + /// Publish `path` straight into the relay. + fn publish(&self, path: &str, route: Route, config: VideoConfig, tag: u8) -> Publisher { + Publisher::announce(&self.origin, path, route, config, tag) + } + + /// Serve every path under `prefix` with one broadcast, as a dynamic route. + fn serve(&self, prefix: &str, route: Route, config: VideoConfig, tag: u8) -> Served { + let publisher = Publisher::new(moq_net::broadcast::Info::new().produce(), config, tag); + let dynamic = self.origin.dynamic(prefix, route).expect("serve the prefix"); + let broadcast = publisher.broadcast.consume(); + let handler = self.runtime().spawn(async move { + while let Ok(request) = dynamic.requested_broadcast().await { + request.accept(&broadcast); + } + }); + Served { + _publisher: publisher, + handler, + } + } + + /// Publish `path` from another origin, over a session of its own. + fn connect(&self, path: &str, route: Route, config: VideoConfig, tag: u8) -> Remote { + let _runtime = self.runtime().enter(); + let origin = moq_tokio::origin::spawn(); + let publisher = Publisher::announce(&origin, path, route, config, tag); + let mut client = moq_tokio::connect::Config::default(); + client.tls.insecure = Some(true); + let client = client.init(Default::default()).expect("client init"); + let url: url::Url = self.url().parse().unwrap(); + let connection = self.runtime().block_on(async { + tokio::time::timeout( + TIMEOUT, + client + .with_publisher(&origin) + .with_reconnect(false) + .connect(url) + .established(), + ) + .await + .expect("the publisher never connected") + .expect("publisher connects") + }); + Remote { + connection, + _publisher: publisher, + _origin: origin, + } + } + + /// Run `end`, then wait for nothing to serve `path` at the relay. + fn offline(&self, path: &str, end: impl FnOnce()) { + let mut follow = self.origin.consume().follow(path).expect("follow the path"); + let mut next = || { + self.runtime() + .block_on(async { tokio::time::timeout(TIMEOUT, follow.next()).await }) + .expect("the path never changed") + .expect("relay origin closed") + }; + assert!( + matches!(next(), moq_net::announce::Event::Start(_)), + "the path was not served" + ); + end(); + while !matches!(next(), moq_net::announce::Event::End(_)) {} + } +} + +impl Drop for Relay { + fn drop(&mut self) { + if let Some(runtime) = self.runtime.take() { + runtime.shutdown_background(); + } + } +} + +/// Frames written in real time on their own thread, each payload filled with its publisher's tag. +struct Feed { + stop: Arc, + thread: Option>>, +} + +impl Feed { + fn start(mut producer: moq_mux::container::Producer, tag: u8) -> Self { + let stop = Arc::new(AtomicBool::new(false)); + let stopped = stop.clone(); + let thread = std::thread::spawn(move || { + let start = Instant::now(); + for i in 0u32.. { + if stopped.load(Ordering::Relaxed) { + break; + } + let frame = moq_mux::container::Frame { + timestamp: moq_net::Timestamp::from_micros((FRAME * i).as_micros() as u64).unwrap(), + payload: bytes::Bytes::from(vec![tag; 64]), + keyframe: i % 30 == 0, + duration: None, + }; + // A track its session dropped refuses the rest, which is the session's to report. + if producer.write(frame).is_err() { + break; + } + std::thread::sleep((start + FRAME * (i + 1)).saturating_duration_since(Instant::now())); + } + producer + }); + Self { + stop, + thread: Some(thread), + } + } + + /// Stop writing, returning the producer to end the track with. + fn stop(mut self) -> moq_mux::container::Producer { + self.stop.store(true, Ordering::Relaxed); + self.thread.take().unwrap().join().expect("feed panicked") + } +} + +impl Drop for Feed { + fn drop(&mut self) { + self.stop.store(true, Ordering::Relaxed); + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } +} + +/// A broadcast with one video rendition, fed in real time. +struct Publisher { + broadcast: moq_net::broadcast::Producer, + catalog: moq_mux::catalog::Producer, + feed: Option, +} + +impl Publisher { + fn new(mut broadcast: moq_net::broadcast::Producer, config: VideoConfig, tag: u8) -> Self { + let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).expect("catalog"); + let video = broadcast + .create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video)) + .expect("video track"); + { + let mut guard = catalog.modify().unwrap(); + guard.video.renditions = BTreeMap::from([("video".to_string(), config)]); + } + let producer = moq_mux::container::Producer::new( + video, + moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), + ); + Self { + broadcast, + catalog, + feed: Some(Feed::start(producer, tag)), + } + } + + fn announce(origin: &moq_net::origin::Producer, path: &str, route: Route, config: VideoConfig, tag: u8) -> Self { + let publisher = Self::new(origin.create_broadcast(path).expect("create broadcast"), config, tag); + publisher.broadcast.announce(route).expect("announce"); + publisher + } + + /// End the broadcast the way `moqsink` does: the media, then the catalog, then the + /// announcement. + fn end(mut self) { + let mut media = self.feed.take().unwrap().stop(); + media.finish().expect("finish the media"); + std::thread::sleep(Duration::from_millis(100)); + self.catalog.finish().expect("finish the catalog"); + self.broadcast.unannounce(); + } +} + +/// A broadcast served under a prefix until dropped. +struct Served { + _publisher: Publisher, + handler: tokio::task::JoinHandle<()>, +} + +impl Drop for Served { + fn drop(&mut self) { + self.handler.abort(); + } +} + +/// A publisher on a session of its own. +struct Remote { + connection: moq_tokio::Connection, + _publisher: Publisher, + _origin: moq_net::origin::Producer, +} + +impl Drop for Remote { + fn drop(&mut self) { + self.connection.abort(moq_net::Error::Cancel); + } +} + +/// `moqsrc` playing a path into a synced sink. +struct Player { + pipeline: gst::Pipeline, + /// How many video pads `moqsrc` added. + added: Arc, + /// The stream-start, caps, and EOS events that reached the sink, in order. + events: Arc>>, + /// The publisher tag of every buffer the sink rendered. + rendered: mpsc::Receiver, +} + +impl Player { + fn start(url: &str, broadcast: &str) -> Self { + init(); + let pipeline = gst::Pipeline::new(); + let src = gst::ElementFactory::make("moqsrc") + .property("url", url) + .property("broadcast", broadcast) + .property("tls-disable-verify", true) + .build() + .expect("create moqsrc"); + let sink = gst::ElementFactory::make("fakesink") + .property("sync", true) + .property("max-lateness", gst::ClockTime::from_mseconds(20).nseconds() as i64) + .property("qos", true) + .property("signal-handoffs", true) + .build() + .expect("create fakesink"); + pipeline.add_many([&src, &sink]).unwrap(); + + // Linked the way a pipeline links by name: the first video pad, once. + let added = Arc::new(AtomicUsize::new(0)); + let sink_pad = sink.static_pad("sink").unwrap(); + let (count, link) = (added.clone(), sink_pad.clone()); + src.connect_pad_added(move |_, pad| { + if !pad.name().starts_with("video_") { + return; + } + count.fetch_add(1, Ordering::SeqCst); + if !link.is_linked() { + pad.link(&link).expect("link the video pad"); + } + }); + + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + sink_pad.add_probe(gst::PadProbeType::EVENT_DOWNSTREAM, move |_, info| { + if let Some(gst::PadProbeData::Event(event)) = &info.data { + let event = match event.view() { + gst::EventView::StreamStart(_) => "stream-start".to_string(), + gst::EventView::Caps(caps) => caps.caps().to_string(), + gst::EventView::Eos(_) => "eos".to_string(), + _ => return gst::PadProbeReturn::Ok, + }; + seen.lock().unwrap().push(event); + } + gst::PadProbeReturn::Ok + }); + + let (tx, rendered) = mpsc::channel(); + sink.connect("handoff", false, move |values| { + let buffer = values[1].get::().expect("handoff buffer"); + let tag = buffer.map_readable().expect("readable buffer")[0]; + let _ = tx.send(tag); + None + }); + + pipeline.set_state(gst::State::Playing).expect("play"); + Self { + pipeline, + added, + events, + rendered, + } + } + + /// Wait for the sink to render `count` buffers from the publisher tagged `tag`, at the rate a + /// stream on time renders. + fn rendered(&self, tag: u8, count: usize) { + let deadline = Instant::now() + FLOW_TIMEOUT + FRAME * count as u32; + let mut seen = 0; + while seen < count { + match self + .rendered + .recv_timeout(deadline.saturating_duration_since(Instant::now())) + { + Ok(rendered) if rendered == tag => seen += 1, + Ok(_) => {} + Err(_) => panic!( + "rendered {seen}/{count} buffers from publisher {tag}; errors: {:?}", + self.errors() + ), + } + } + } + + fn events(&self) -> Vec { + self.events.lock().unwrap().clone() + } + + fn stream_starts(&self) -> usize { + self.events().iter().filter(|event| *event == "stream-start").count() + } + + fn errors(&self) -> Vec { + let bus = self.pipeline.bus().unwrap(); + std::iter::from_fn(|| bus.pop_filtered(&[gst::MessageType::Error])) + .map(|message| match message.view() { + gst::MessageView::Error(err) => format!("{} ({:?})", err.error(), err.debug()), + _ => unreachable!(), + }) + .collect() + } + + /// Every stream played on the one pad, with no EOS and no error. + fn assert_one_pad(&self) { + assert_eq!(self.added.load(Ordering::SeqCst), 1, "moqsrc added another pad"); + let events = self.events(); + assert!(!events.iter().any(|event| event == "eos"), "a pad ended: {events:?}"); + assert_eq!(self.errors(), Vec::::new()); + } +} + +impl Drop for Player { + fn drop(&mut self) { + let _ = self.pipeline.set_state(gst::State::Null); + } +} + +/// A newer publisher instance takes the path while the old one is still publishing. The pipeline +/// cuts over to it on the pad it linked, starting the new stream at the clock's running time so a +/// synced sink renders it. +#[test] +fn a_newer_instance_takes_over_while_the_old_one_stays_up() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "live"); + + let _old = relay.publish("live", route(), h264(), 1); + player.rendered(1, FLOWING); + + let _new = relay.publish("live", route(), h264(), 2); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + assert_eq!(player.stream_starts(), 2); +} + +/// Without epochs, the path changing hands is still another instance. +#[test] +fn an_epochless_republish_switches() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "live"); + + let old = relay.publish("live", Route::default(), h264(), 1); + player.rendered(1, FLOWING); + + let _new = relay.publish("live", Route::default(), h264(), 2); + drop(old); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + assert_eq!(player.stream_starts(), 2); +} + +/// A route over a prefix serves the path as much as one at the path does, so another instance of +/// it is a switch too. +#[test] +fn a_restarted_prefix_switches() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "pool/live"); + + let _old = relay.serve("pool", route(), h264(), 1); + player.rendered(1, FLOWING); + + let _new = relay.serve("pool", route(), h264(), 2); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + assert_eq!(player.stream_starts(), 2); +} + +/// Re-pricing the same instance changes nothing about what plays. Updates are delivered in order, +/// so a later instance's switch is counted after any the update could have caused. +#[test] +fn an_update_does_not_switch() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "live"); + + let old = relay.publish("live", route(), h264(), 1); + player.rendered(1, FLOWING); + + old.broadcast + .announce(old.broadcast.route().expect("announced").with_cost(5)) + .expect("re-price"); + // Past the origin's update hold, so the update is not folded into the instance after it. + player.rendered(1, 3 * FLOWING); + + let _new = relay.publish("live", route(), h264(), 2); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + assert_eq!(player.stream_starts(), 2, "the update switched: {:?}", player.events()); +} + +/// The instance that takes over can carry other caps: they go out on the same pad, for +/// downstream to accept or refuse. +#[test] +fn a_switch_with_new_caps_keeps_the_pad() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "live"); + + let _old = relay.publish("live", route(), h264(), 1); + player.rendered(1, FLOWING); + + let _new = relay.publish("live", route(), avc(), 2); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + let caps: Vec = player + .events() + .into_iter() + .filter(|event| event.starts_with("video/")) + .collect(); + assert_eq!(caps.len(), 2, "{caps:?}"); + assert!(caps[0].contains("annexb"), "{caps:?}"); + assert!(caps[1].contains("avc"), "{caps:?}"); +} + +/// A broadcast that ends holds its pads without EOS, and the next one to start resumes on them. +#[test] +fn an_end_then_a_start_resumes_on_the_same_pad() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "live"); + + let old = relay.publish("live", route(), h264(), 1); + player.rendered(1, FLOWING); + + relay.offline("live", || old.end()); + // Nothing serves the path for a while. + std::thread::sleep(Duration::from_millis(300)); + + let _new = relay.publish("live", route(), h264(), 2); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + assert_eq!(player.stream_starts(), 2); +} + +/// A publisher whose session closes without finishing anything is the source going away, not a +/// broken stream: the pads hold for the next publisher. +#[test] +fn a_closed_publisher_session_resumes_on_the_next_start() { + let relay = Relay::start(); + let player = Player::start(&relay.url(), "live"); + + let old = relay.connect("live", route(), h264(), 1); + player.rendered(1, FLOWING); + + relay.offline("live", || old.connection.abort(moq_net::Error::Cancel)); + // Nothing serves the path for a while. + std::thread::sleep(Duration::from_millis(300)); + + let _new = relay.connect("live", route(), h264(), 2); + player.rendered(2, FLOWING); + + player.assert_one_pad(); + assert_eq!(player.stream_starts(), 2); +} diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index e3492106379..eefaede1431 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -2658,18 +2658,21 @@ async fn serve_front(task: FrontTask) -> Vec { let Some(instance) = instances.get(&source).cloned() else { continue; }; - let prefix = { + let (prefix, winner) = { let table = shared.read(); - table + let prefix = table .routes .covering(&path.as_path()) .find(|entry| entry.id == route) - .map(|entry| entry.prefix.clone()) + .map(|entry| entry.prefix.clone()); + let winner = prefix.as_ref().and_then(|prefix| table.presented(prefix, horizon)); + (prefix, winner) }; resolved = Some(Resolution { instance: instance.clone(), route, prefix, + winner, renewed: false, }); if let Ok(mut pending) = request.write() @@ -3678,6 +3681,16 @@ impl OriginState { } best } + + /// The instance announce cursors behind `horizon` present at `prefix`: its best + /// route there, ranked as [`Self::sync_route`] ranks them. + fn presented(&self, prefix: &Path, horizon: Horizon) -> Option { + self.routes + .at(prefix) + .filter(|entry| entry.live() && horizon.admits(entry)) + .min_by_key(|entry| route_order(prefix, entry)) + .map(RouteEntry::instance) + } } /// One-shot result of a dynamic broadcast request. @@ -3707,6 +3720,8 @@ struct Resolution { route: u64, /// That route's prefix, when it was still in the table as the front resolved. prefix: Option, + /// The instance the prefix's announcements presented as the front resolved. + winner: Option, /// Whether the front already renewed the prefix; see [`Self::moved`]. renewed: bool, } @@ -3715,7 +3730,9 @@ impl Resolution { /// The prefix to [renew](OriginState::renew), once per front: on a route without /// an epoch, a request for the path, beneath the prefix, now resolves through /// another route there. The prefix's own winner may be unchanged, so nothing else - /// tells its announce cursors that what they resolved under it moved. + /// tells its announce cursors that what they resolved under it moved. Once another + /// instance wins the prefix, they restarted it already, and a renewal would restart + /// it again. fn moved(&mut self, table: &OriginState, path: &Path, horizon: Horizon) -> Option { if self.renewed || !matches!(self.instance, Instance::Route(_)) { return None; @@ -3729,6 +3746,9 @@ impl Resolution { if best.id == self.route || best.prefix != *prefix || best.epoch.is_some() { return None; } + if table.presented(prefix, horizon) != self.winner { + return None; + } self.renewed = true; Some(prefix.clone()) } diff --git a/rs/moq-net/tests/restart.rs b/rs/moq-net/tests/restart.rs index ad9d8085a8c..f5563210d3b 100644 --- a/rs/moq-net/tests/restart.rs +++ b/rs/moq-net/tests/restart.rs @@ -60,6 +60,13 @@ async fn expect_restart(version: Version, announced: &mut announce::Consumer, pr } } +/// Nothing more arrives, past the update hold. +async fn quiet(version: Version, announced: &mut announce::Consumer) { + if let Ok(event) = moq_net_sim::timeout(Duration::from_secs(1), announced.next()).await { + panic!("{version}: expected nothing more, got {event:?}"); + } +} + /// Write one group of one frame tagged `tag`. fn write(track: &track::Producer, tag: &str) { let mut group = track.append_group().unwrap(); @@ -324,6 +331,10 @@ async fn dynamic_epoch(version: &str, from: Option, to: Option) { } claim.write(1, "new:0"); assert_eq!(read(&mut subscription).await, "new:0", "{version}"); + + // The sticky subscription's path resolving through the new route is the restart + // already delivered, not another. + quiet(version, &mut announced).await; } /// The epoch the claim starts at, and a newer one it moves to.