diff --git a/doc/bin/gstreamer.md b/doc/bin/gstreamer.md index 3b053b30a2d..b77e58889fb 100644 --- a/doc/bin/gstreamer.md +++ b/doc/bin/gstreamer.md @@ -124,7 +124,7 @@ rewound timestamps forward. Pads are named by kind and appear as the catalog announces renditions: `video_0`, `video_1`, `audio_0`. A rendition referencing another broadcast reads -that broadcast on the same connection, waiting for it to be announced. Link the pad you want by name; the terse +that broadcast on the same connection and follows its announcements the same way. Link the pad you want by name; the terse `moqsrc ! decodebin3` form links only the first pad offered, which may be audio. `moqsrc` follows the broadcast's announcements: when the publisher restarts, it switches to the new broadcast on the same pads, kept by rendition diff --git a/doc/concept/hang.md b/doc/concept/hang.md index 714ce485f61..841f74d2b0b 100644 --- a/doc/concept/hang.md +++ b/doc/concept/hang.md @@ -63,7 +63,7 @@ decoder once per track. A few things the catalog can express beyond decoder config: - **Labels.** Any rendition may carry a human-readable `label` for a track picker. The map key stays the rendition's ID, so labels need not be unique and renaming one doesn't rename the track. -- **Renditions in another broadcast.** A rendition may point at a relative broadcast path, so a transcoder can publish a ladder that adds low rungs and references the source's original rendition without re-publishing its bytes. The path resolves against where the consumer found the catalog, so a reference that escapes above the root names nothing and the catalog is rejected. `@moq/watch` follows a referenced path's announcements like its own, restarting the rendition on each instance that serves it; every other reader, such as `moqsrc`, `moq play`, the exporters, and the bindings, waits for the first one announced and keeps it, and an exporter ends the rendition with it. An optional `epoch` beside the path names the [publisher instance](/concept/moq-lite#publisher-epochs) it expects: a route proving another instance ends the rendition instead of falling back to it, while a route without an epoch (an older version) cannot prove one, so there it is best effort. +- **Renditions in another broadcast.** A rendition may point at a relative broadcast path, so a transcoder can publish a ladder that adds low rungs and references the source's original rendition without re-publishing its bytes. The path resolves against where the consumer found the catalog, so a reference that escapes above the root names nothing and the catalog is rejected. `@moq/watch` and `moqsrc` follow a referenced path's announcements like their own, restarting the rendition on each instance that serves it; every other reader, such as `moq play`, the exporters, and the bindings, waits for the first one announced and keeps it, and an exporter ends the rendition with it. An optional `epoch` beside the path names the [publisher instance](/concept/moq-lite#publisher-epochs) it expects: a route proving another instance ends the rendition instead of falling back to it, while a route without an epoch (an older version) cannot prove one, so there it is best effort. - **Track names.** The map key is a rendition's ID and, by default, its track name. A rendition sets `track` only when the two must differ, such as two renditions of same-named tracks in different broadcasts, or a rendition whose media changed and so needs a new name; readers subscribe to `track` when present. An ID can't name both a video and an audio rendition. - **Jitter.** A rendition can say how far its frames fell behind the media clock before the publisher flushed them, in whole milliseconds rounded up. Encoders report the spread of lateness above each rendition's own recent minimum, so a constant encoder delay is not jitter; container imports estimate batch spans without counting ingest delay. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent. - **Delay.** A rendition can also say how far its frames reach the transport behind the broadcast's earliest rendition, measured the same way from each rendition's minimum lateness, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's. diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index e3e052afc07..ff1b59f49dc 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -1,4 +1,4 @@ -# Broadcast epochs +# [L] Broadcast epochs ## Goal @@ -73,7 +73,7 @@ Decided: resolution on epoch-aware transports and remain best effort on epochless transports. Following never turns a replacement into seamless resume. The schemas, shared resolution, compatibility documentation, and `moq_mux::follow` - landed; `moqsrc` following siblings with it remains. + (which `moqsrc` follows siblings with) landed. - Every first-party publisher that can restart mints its own: the apps, moq-boy, the ingest gateways, moqsink, OBS, and the bindings. Players (`moq play`, `@moq/watch`, demo/web) follow the announce `Restart`. moq-stats mints one per @@ -122,7 +122,3 @@ restarts that sibling while preserving the catalog's explicit epoch constraints; it never mixes instances through an old handle. Check exact and covering routes, matching/different/absent epochs, and legacy End+Start. The child quests own their focused regressions. - -## Required - -- [moqsrc sibling follow](/quest/m0/broadcast-epoch/moqsrc-sibling-follow.md) - moqsrc restarts a referenced sibling's rendition on each instance its announcements hand out, never past a named epoch diff --git a/quest/m0/broadcast-epoch/moqsrc-sibling-follow.md b/quest/m0/broadcast-epoch/moqsrc-sibling-follow.md deleted file mode 100644 index f14cd2dd940..00000000000 --- a/quest/m0/broadcast-epoch/moqsrc-sibling-follow.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] moqsrc follows referenced broadcasts' announcements - -## Goal - -`moqsrc` follows a catalog-referenced sibling's announcements the way -`@moq/watch` does: each rendition reads the instance `moq_mux::Source::follow` -hands out, idles its pad (no EOS) while the sibling is offline, and restarts on -the same pad when another instance serves it. A reference that names an -`epoch` never plays another instance. Until this lands, `moqsrc` waits for a -sibling's first announcement and keeps that instance, like every other -non-following reader. - -## Plan - -Split from the reference announcements work (maintainer, 2026-10-22), which -landed `moq_mux::follow` and the waiting `Source::bind`. The moqsrc half drew a -new state-machine finding every review round, so it gets its own design pass. - -- Start binds the rendition; End idles its pad; another instance restarts the - stream on the same pad with a fresh stream-start, caps, and a segment. -- Renditions reading one sibling instance share that instance's timeline, - whenever each starts on it, including one still subscribing when a - replacement takes over. -- A sibling track that finishes while its instance stays up idles the pad and - keeps following; a finished record names the instance that served it, so a - reshape (disable and enable, an epoch handoff) still plays the next one. -- A rendition the catalog delists drains the instance it reads, then ends its - pad and follows no later instance. -- A pump that has returned but is still in `active` must not swallow a relist: - a relist arriving before the pump is joined matches its shape and starts - nothing, for an idle delisted sibling and for a delisted catalog rendition - whose track ends just before the relist. Cover both cases. - -Weigh a simpler state model (one task per sibling instance owning its -renditions, or reconciling after each pump exits) before patching each case. - -Public API: none (moqsrc behavior). Wire: none. - -## Related - -- [Sink restart follow](/quest/m1/sink-restart-follow.md) - moves moqsrc's own broadcast follow onto `follow::Instances` diff --git a/quest/m1/sink-restart-follow.md b/quest/m1/sink-restart-follow.md index 5ff849bc074..b76f09ecbfe 100644 --- a/quest/m1/sink-restart-follow.md +++ b/quest/m1/sink-restart-follow.md @@ -16,8 +16,7 @@ route dies, since its subscriptions then end with an error. Decided with the maintainer on 2026-10-20: widened from the server and push sinks to every moq-cli and moq-gst reader. Following a catalog's references -to other broadcasts is [moqsrc sibling follow](/quest/m0/broadcast-epoch/moqsrc-sibling-follow.md) -for `moqsrc`, through `moq_mux::Source::follow`; +to other broadcasts landed for `moqsrc` through `moq_mux::Source::follow`; `moq play` resolves a reference once with `Source::resolve`, which waits for a sibling's first announcement and keeps it. diff --git a/rs/moq-gst/src/source/imp.rs b/rs/moq-gst/src/source/imp.rs index 9316007c684..9184e1e13e3 100644 --- a/rs/moq-gst/src/source/imp.rs +++ b/rs/moq-gst/src/source/imp.rs @@ -1,5 +1,5 @@ use std::collections::{HashMap, HashSet}; -use std::sync::atomic::{AtomicU8, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, LazyLock, Mutex, OnceLock}; use std::time::Duration; @@ -30,7 +30,7 @@ static RUNTIME: LazyLock = LazyLock::new(|| { /// silently breaking those pipelines. A session keeps each pad by rendition across restarts /// and format changes, so only a new or renamed rendition takes another id. /// -/// An id is claimed where the pad is created, not where its pump is spawned. A rendition whose +/// An id is claimed where the pad is created, not where its reader is spawned. A rendition whose /// subscription never resolves therefore reserves nothing, so it can't leave `video_0` pointing /// at a pad that will never exist while the rendition that does arrive lands on `video_1`. static NEXT_VIDEO_PAD_ID: AtomicU64 = AtomicU64::new(0); @@ -104,14 +104,14 @@ impl TrackKind { } tokio::task_local! { - /// The element whose session task or pump is running, so [`SessionController::stop`] can tell + /// The element whose session task or reader is running, so [`SessionController::stop`] can tell /// it was reached from that element's own streaming context. static STREAMING: glib::WeakRef; } /// The session task drives everything: it connects, follows the catalog, and -/// runs one [`Pump`] per active rendition. The element just starts and -/// stops it. No control-plane channel is needed because pumps push to their pads +/// runs one [`Reader`] per rendition and instance. The element just starts and +/// stops it. No control-plane channel is needed because readers push to their pads /// directly from their own task (a source pad's push *is* its streaming thread), /// so there's nothing to marshal back onto the element. struct SessionController { @@ -169,7 +169,7 @@ impl SessionController { } } - /// Stop the session, returning once its pumps have removed their pads and the connection closed. + /// Stop the session, returning once its readers have removed their pads and the connection closed. /// /// A dial still running once the element reached NULL can outlive `main` (`gst-launch` exits right /// after), and aws-lc aborts the process when a thread asks it for randomness after its exit @@ -177,14 +177,14 @@ impl SessionController { fn stop(self, element: &super::MoqSrc) { let _ = self.shutdown.send(true); - // A pump blocked in a push returns once its pad flushes, the same unlock a GStreamer source + // A reader blocked in a push returns once its pad flushes, the same unlock a GStreamer source // gives its streaming thread on stop. A push held up inside a downstream element is released // by that element leaving PAUSED, which a pipeline does before it reaches its source. for pad in element.src_pads() { let _ = pad.set_active(false); } - // Reached from this element's own session task or pump (a bus sync or pad handler), waiting + // Reached from this element's own session task or reader (a bus sync or pad handler), waiting // for the session would wait on the caller's own stack. GStreamer refuses the same for its // sources; the connection still closes before this returns. let own = STREAMING @@ -420,9 +420,9 @@ impl MoqSrc { } } -/// The identity we reconcile a rendition on: a change to any field tears the pad down and -/// recreates it. Caps cover codec/resolution; the container descriptor covers the wire framing -/// (e.g. legacy -> cmaf); the broadcast and track identify the media, apart from its catalog ID. +/// The identity a rendition plays under: a change to any field restarts its stream on the same +/// pad. Caps cover codec/resolution; the container descriptor covers the wire framing (e.g. legacy +/// -> cmaf); the broadcast and track identify the media, apart from its catalog ID. #[derive(Clone, PartialEq)] struct Shape { caps: gst::Caps, @@ -432,94 +432,38 @@ struct Shape { epoch: Option, } -/// A pump's progress, shared with its [`ActiveTrack`] so teardown and pad creation can't both -/// win. A pump is torn down two different ways depending on how far it got: before it owns a pad, -/// stopping it means it must never create one; after, it owns a pad and has to drop it. One -/// compare-exchange settles which of the two happened, so a pad can't slip out between a -/// teardown's check and the pump's creation. -struct PumpState(AtomicU8); - -impl PumpState { - const SUBSCRIBING: u8 = 0; - const LIVE: u8 = 1; - const CANCELLED: u8 = 2; - - fn new() -> Self { - Self(AtomicU8::new(Self::SUBSCRIBING)) - } - - /// Claim the right to create a pad, false once a teardown got here first. - fn go_live(&self) -> bool { - self.0 - .compare_exchange(Self::SUBSCRIBING, Self::LIVE, Ordering::AcqRel, Ordering::Acquire) - .is_ok() - } - - /// Stop a pump that hasn't taken a pad, false if it already has one and must be torn down - /// through its cancel watch instead. - fn cancel_before_live(&self) -> bool { - self.0 - .compare_exchange(Self::SUBSCRIBING, Self::CANCELLED, Ordering::AcqRel, Ordering::Acquire) - .is_ok() - } -} - /// A rendition by kind and moq track name: what a pad is kept by. type Rendition = (TrackKind, String); -/// The media a rendition plays: its resolved broadcast and track name, whatever the catalog calls -/// it or however it spells the reference. -#[derive(Clone, PartialEq, Eq, Hash)] -struct Track { - broadcast: Option, - name: String, -} - -/// A rendition a run is serving, keyed in the run by moq track name. -struct ActiveTrack { - kind: TrackKind, - /// Identity we diff against on each catalog update; a change hands the pad to a new pump. - shape: Shape, - /// Tells the pump to hand its pad back and exit (set when the run ends or reconcile - /// replaces the rendition). - cancel: watch::Sender, - /// Handle to the pump task in the run's `JoinSet`. We only read - /// `is_finished()` to prune this entry once the pump ends (the `JoinSet` owns - /// the task and reaps it); teardown goes through `cancel`, never `abort()`. - task: tokio::task::AbortHandle, - /// Shared with the pump, so teardown and pad creation agree on which of them happened. - state: Arc, -} +/// The session's pads by rendition, and the renditions the latest catalog lists. The pads outlive +/// the readers and runs that stream to them, so a pipeline linked by name (`s.video_0 ! ...`) keeps +/// flowing across a restart. +#[derive(Clone, Default)] +struct Pads(Arc>); -impl ActiveTrack { - /// Tear the pump down whatever stage it reached: one still subscribing never takes a pad, - /// one that has hands it back when it sees the watch. - /// - /// Terminal, so it consumes the handle: a cancelled rendition is removed from the run's - /// active set, and respawns as a fresh pump if the catalog names it again. - fn cancel(self) { - self.state.cancel_before_live(); - let _ = self.cancel.send(true); - } +#[derive(Default)] +struct PadTable { + slots: HashMap, + /// Only a listed rendition takes a pad, or keeps one once its reader lets go. + listed: HashSet, } -/// The session's pads by rendition. They outlive the pumps and runs that stream to them, so a -/// pipeline linked by name (`s.video_0 ! ...`) keeps flowing across a restart. -#[derive(Clone, Default)] -struct Pads(Arc>>); - struct Slot { pad: gst::Pad, - /// Held by the pump streaming to the pad, so its replacement waits for it to let go. + /// Held by the reader streaming to the pad, so the next one waits for it to let go. owner: Arc>, + /// A retirement waits for the reader to let go. + retiring: bool, } -/// A pad for a pump to stream to, from [`Pads::claim`]. +/// A pad for a reader to stream to, from [`Pads::claim`]. enum Claim { - /// A new pad, already owned, for the pump to add to the element. + /// A new pad, already owned, for the reader to add to the element. New(gst::Pad, tokio::sync::OwnedMutexGuard<()>), - /// A kept pad, owned once its previous pump lets go. + /// A kept pad, owned once its previous reader lets go. Kept(gst::Pad, Arc>), + /// No pad: the latest catalog does not list the rendition. + Unlisted, } impl Pads { @@ -528,17 +472,20 @@ impl Pads { fn claim( &self, element: &glib::WeakRef, - rendition: Rendition, + rendition: &Rendition, cancel: &watch::Receiver, ) -> Option { let obj = element.upgrade()?; - let mut slots = self.0.lock().unwrap(); - // Checked under the lock: a run cancels a pump before it retires the pump's pad, so a - // cancelled pump never claims one behind the retirement. + let mut table = self.0.lock().unwrap(); + // Checked under the lock [`Self::list`] takes, so a reader that an update cancels or delists + // never claims a pad behind the update's retirements. if *cancel.borrow() { return None; } - if let Some(slot) = slots.get(&rendition) { + if !table.listed.contains(rendition) { + return Some(Claim::Unlisted); + } + if let Some(slot) = table.slots.get(rendition) { return Some(Claim::Kept(slot.pad.clone(), slot.owner.clone())); } @@ -549,36 +496,80 @@ impl Pads { .build(); let owner = Arc::new(tokio::sync::Mutex::new(())); let owned = owner.clone().try_lock_owned().expect("a new lock is free"); - slots.insert( - rendition, + table.slots.insert( + rendition.clone(), Slot { pad: pad.clone(), owner, + retiring: false, }, ); Some(Claim::New(pad, owned)) } - /// Take the rendition's pad out of the session, returning the task that ends it with EOS once - /// its pump lets go. - fn retire(&self, rendition: &Rendition) -> Option> + Send + 'static> { - let slot = self.0.lock().unwrap().remove(rendition)?; - Some(async move { - let _owned = slot.owner.clone().lock_owned().await; - slot.end(true); - None - }) + /// Whether the latest catalog lists the rendition. + fn is_listed(&self, rendition: &Rendition) -> bool { + self.0.lock().unwrap().listed.contains(rendition) + } + + /// List what the latest catalog names, returning a retirement for each pad it no longer lists. + /// Each ends its pad with EOS once its reader lets go, unless the rendition is listed again by + /// then, so a delisted rendition still playing out its track keeps its tail. + /// Nothing once `cancel` fired, since a later run lists the pads then. + fn list( + &self, + listed: HashSet, + cancel: &watch::Receiver, + ) -> Vec + Send + 'static> { + let mut table = self.0.lock().unwrap(); + if *cancel.borrow() { + return Vec::new(); + } + table.listed = listed; + let PadTable { slots, listed } = &mut *table; + slots + .iter_mut() + .filter(|(rendition, slot)| !slot.retiring && !listed.contains(*rendition)) + .map(|(rendition, slot)| { + slot.retiring = true; + let (pads, rendition, owner) = (self.clone(), rendition.clone(), slot.owner.clone()); + async move { + let _owned = owner.clone().lock_owned().await; + let slot = { + let mut table = pads.0.lock().unwrap(); + let PadTable { slots, listed } = &mut *table; + match slots.get_mut(&rendition) { + Some(slot) if Arc::ptr_eq(&slot.owner, &owner) => match listed.contains(&rendition) { + true => { + slot.retiring = false; + None + } + false => slots.remove(&rendition), + }, + _ => None, + } + }; + if let Some(slot) = slot { + slot.end(true); + } + } + }) + .collect() } - /// The renditions with a pad that `keep` rejects. - fn unwanted(&self, keep: impl Fn(&Rendition) -> bool) -> Vec { - let slots = self.0.lock().unwrap(); - slots.keys().filter(|rendition| !keep(rendition)).cloned().collect() + /// Whether `owner` still guards the rendition's pad, rather than a retirement having ended it. + fn holds(&self, rendition: &Rendition, owner: &Arc>) -> bool { + self.0 + .lock() + .unwrap() + .slots + .get(rendition) + .is_some_and(|slot| Arc::ptr_eq(&slot.owner, owner)) } /// Remove every pad once nothing streams to them any more, after an EOS if `eos`. fn release(&self, eos: bool) { - let slots = std::mem::take(&mut *self.0.lock().unwrap()); + let slots = std::mem::take(&mut self.0.lock().unwrap().slots); for slot in slots.into_values() { slot.end(eos); } @@ -666,7 +657,7 @@ async fn follow_path( }; if start { - // Cut over at once: the old run's pumps hand their pads back without waiting for their + // Cut over at once: the old run's readers hand their pads back without waiting for their // subscriptions to end. if let Some(cancel) = latest.take() { let _ = cancel.send(true); @@ -752,9 +743,9 @@ fn catalog_lost(err: &moq_mux::Error) -> bool { } } -/// Follow one broadcast's catalog, keeping one [`Pump`] per announced rendition in sync with it. -/// Returns once the catalog closes and the last pump drains, or `cancel` fires. The pads stay with -/// the session, except those of renditions the catalog retired. +/// Follow one broadcast's catalog, with one [`Follower`] per broadcast its renditions reference. +/// Returns once the catalog closes and the last follower drains, or `cancel` fires. The pads stay +/// with the session, except those of renditions the catalog retired. async fn follow_catalog( source: moq_mux::Source, pads: &Pads, @@ -778,28 +769,31 @@ async fn follow_catalog( }; let mut catalog_consumer = moq_mux::catalog::hang::Consumer::new(catalog_track); - // Follow the catalog for the whole run and reconcile our pumps against every update, - // rather than building them once from the first frame. This covers reactive publishers - // (the browser via @moq/hang) that announce an empty catalog before their encoder - // configures, then add renditions a beat later, as well as renditions appearing, - // disappearing, or changing codec/resolution mid-stream. + // Follow the catalog for the whole run and reconcile against every update, rather than + // building the renditions once from the first frame. This covers reactive publishers (the + // browser via @moq/hang) that announce an empty catalog before their encoder configures, then + // add renditions a beat later, as well as renditions appearing, disappearing, or changing + // codec/resolution mid-stream. + let (stop, stopped) = watch::channel(false); let playback = Playback { + element, + broadcast: broadcast.clone(), pads: pads.clone(), timeline: Arc::default(), - finished: Arc::default(), + shutdown: shutdown.clone(), + stop: stopped, }; - let mut active: HashMap = HashMap::new(); - let mut pumps: tokio::task::JoinSet> = tokio::task::JoinSet::new(); - // The renditions the latest catalog lists. - let mut listed: HashSet = HashSet::new(); + let mut followers: HashMap = HashMap::new(); + // Woken when a follower runs out of renditions, so the run can let it go without waiting for + // another catalog update. + let emptied = Arc::new(tokio::sync::Notify::new()); + // The followers, and the retirements of pads the catalog no longer lists. + let mut tasks = tokio::task::JoinSet::new(); let mut catalog_closed = false; let mut result = Ok(()); loop { - // Prune metadata for pumps that have ended (the JoinSet has already reaped the - // tasks). Once the catalog is closed and the last pump drains, the run is done. - active.retain(|_, track| !track.task.is_finished()); - if catalog_closed && pumps.is_empty() { + if catalog_closed && tasks.is_empty() { break; } @@ -807,94 +801,70 @@ async fn follow_catalog( biased; // Ended by a restart, or the session ending. _ = cancel.changed() => break, - // A pump finished; loop back so the `retain` above prunes its entry and the - // break condition sees the drained set. - joined = pumps.join_next(), if !pumps.is_empty() => { - // A rendition the catalog retired ends with EOS once its track does. One still - // listed keeps its pad, since the track, the catalog, and the announcements arrive - // on different streams in either order. - if let Some(Ok(Some(rendition))) = joined - && !listed.contains(&rendition) - && let Some(retire) = pads.retire(&rendition) - { - pumps.spawn_on(STREAMING.scope(element.clone(), retire), RUNTIME.handle()); - } - } + _ = tasks.join_next(), if !tasks.is_empty() => {} + _ = emptied.notified() => followers.retain(|_, follower| !follower.plan.is_closed() && !follower.idle()), // The guard stops us polling a closed catalog track (which would spin the loop - // returning None) while we wait for the remaining pumps to drain. - next = catalog_consumer.next(), if !catalog_closed => { - match next { - Ok(Some(catalog)) => { - listed = reconcile(&catalog, &mut active, &mut pumps, &source, &playback, &element, shutdown); - } - // Catalog track closed. Don't cancel the pumps: let each reach its natural end, - // and just stop reconciling while they drain. - // - // That includes a pump still resolving its subscription, which is - // indistinguishable here from one that has simply not been polled yet: - // a final snapshot naming a track the publisher does serve arrives this - // way, and cancelling on "not live yet" would drop it. Whether a - // rendition will ever be served is the publisher's to answer, and - // ending the broadcast is where it does: every name it never served - // resolves with an error, which fails that pump's subscribe and ends - // it here. - Ok(None) => catalog_closed = true, - Err(err) if catalog_lost(&err) => { - tracing::warn!(%err, "catalog lost, holding pads"); - catalog_closed = true; - } - Err(err) => { - result = Err(err.into()); - break; - } + // returning None) while the followers drain. + next = catalog_consumer.next(), if !catalog_closed => match next { + Ok(Some(catalog)) => reconcile(&catalog, &source, &playback, &mut followers, &mut tasks, &emptied, cancel), + // Catalog track closed. Don't stop the followers: let each reader reach its natural + // end, and just stop reconciling while they drain. + // + // That includes a reader still resolving its subscription, which is + // indistinguishable here from one that has simply not been polled yet: a final + // snapshot naming a track the publisher does serve arrives this way, and stopping on + // "not live yet" would drop it. Whether a rendition will ever be served is the + // publisher's to answer, and ending the broadcast is where it does: every name it + // never served resolves with an error, which fails that reader's subscribe and ends + // it. + Ok(None) => catalog_closed = true, + Err(err) if catalog_lost(&err) => { + tracing::warn!(%err, "catalog lost, holding pads"); + catalog_closed = true; + } + Err(err) => { + result = Err(err.into()); + break; } + }, + } + + if catalog_closed { + for (_, follower) in followers.drain() { + follower.plan.send_modify(|plan| plan.closed = true); } } } - // Cancel every pump, then wait for them all to hand their pads back. Cancel all up front - // (pumps only exit on their own `cancel`), or the not-yet-cancelled ones would keep streaming - // while we await the rest. On the clean catalog-closed exit `active`/`pumps` are already - // drained, so this is a no-op. - for (_, track) in active.drain() { - track.cancel(); - } - while pumps.join_next().await.is_some() {} + // Stop every follower, then wait for each to hand its pads back. On the clean catalog-closed + // exit the tasks already drained, so this is a no-op. + let _ = stop.send(true); + while tasks.join_next().await.is_some() {} result } -/// Bring the run's pumps in line with `catalog`: spawn pumps for newly announced renditions, -/// hand the pad of any whose [`Shape`] changed to a new pump, leave ones that vanished to -/// end with their track, stop disabled ones, and end the kept pads of renditions it no longer -/// lists. A rendition that already played its track to the end keeps its pad but gets no pump. -/// Returns the renditions it lists. +/// Bring the run in line with `catalog`: list its renditions on the pads, retiring the pads of +/// those it dropped, and hand each referenced broadcast's follower the renditions that read it. /// /// Infallible by design: every way a single rendition can be unusable (unsupported codec, -/// malformed init, a name the broadcast refuses) skips just that rendition, so one bad entry in -/// the catalog can never tear down the ones already streaming. +/// malformed init, a reference that escapes the root) skips just that rendition, so one bad entry +/// in the catalog can never tear down the ones already streaming. fn reconcile( catalog: &moq_mux::catalog::hang::Catalog, - active: &mut HashMap, - pumps: &mut tokio::task::JoinSet>, source: &moq_mux::Source, playback: &Playback, - element: &glib::WeakRef, - shutdown: &watch::Receiver, -) -> HashSet { - let pads = &playback.pads; - struct Desired { - kind: TrackKind, - shape: Shape, - } - + followers: &mut HashMap, + tasks: &mut tokio::task::JoinSet<()>, + emptied: &Arc, + cancel: &watch::Receiver, +) { // Build the desired shape for each rendition. This is deliberately cheap: caps come from the - // catalog config and the container is just the hang descriptor. We defer parsing the wire - // container (which re-parses the CMAF init) to spawn time below, so an unchanged rendition - // costs nothing here. A rendition whose caps we can't build (unsupported codec) is logged and - // skipped rather than failing the whole run, so one bad rendition can't tear down the - // others we're already serving. - let mut desired: HashMap = HashMap::new(); + // catalog config and the container is just the hang descriptor. The reader parses the wire + // container (which re-parses the CMAF init) when it starts, so an unchanged rendition costs + // nothing here. A rendition whose caps we can't build (unsupported codec) is logged and + // skipped. + let mut desired: HashMap = HashMap::new(); let mut insert = |name: &String, kind, caps: Result, @@ -908,7 +878,7 @@ fn reconcile( broadcast: reference.broadcast().cloned(), epoch: reference.epoch().cloned(), }; - desired.insert(name.clone(), Desired { kind, shape }); + desired.insert((kind, name.clone()), shape); } Err(err) => gst::warning!(CAT, "ignoring {kind:?} rendition {name}: {err:?}"), }; @@ -919,306 +889,642 @@ fn reconcile( insert(name, TrackKind::Audio, audio_caps(config), &config.container, config); } - // A disabled rendition stays listed: its pump stops, but its pad waits idle for a re-enable, + // A disabled rendition stays listed: its reader stops, but its pad waits idle for a re-enable, // since a pipeline links pads by name (`s.video_0 ! ...`). - let disabled: HashSet = catalog + let disabled: Vec<(Rendition, &dyn hang::catalog::Reference)> = catalog .video .renditions .iter() .filter(|(_, config)| !config.enabled) - .map(|(name, _)| (TrackKind::Video, name.clone())) + .map(|(name, config)| { + ( + (TrackKind::Video, name.clone()), + config as &dyn hang::catalog::Reference, + ) + }) .chain( catalog .audio .renditions .iter() .filter(|(_, config)| !config.enabled) - .map(|(name, _)| (TrackKind::Audio, name.clone())), + .map(|(name, config)| { + ( + (TrackKind::Audio, name.clone()), + config as &dyn hang::catalog::Reference, + ) + }), ) .collect(); - - // Pure set math: which pumps to tear down, which renditions to spawn. - let plan = plan_reconcile( - &desired - .iter() - .map(|(name, d)| (name.clone(), (d.kind, d.shape.clone()))) - .collect(), - &active - .iter() - .map(|(name, t)| (name.clone(), (t.kind, t.shape.clone()))) - .collect(), - ); - - // Cancel anything that changed shape; each cancelled pump hands its pad back. Changed - // renditions also land in `plan.add`, so they respawn below on the same pad. - // - // A rendition that vanished is only no longer selectable: a publisher retires one by - // delisting it and finishing its track, and the two arrive in either order. A pump that owns - // a pad stays in `active` and ends with its track, where `follow_catalog` ends its pad. One - // that never took a pad is stopped. - for name in plan.remove { - let Some(track) = active.get(&name) else { continue }; - // An explicit disable stops a live pump too; a delist still drains its tail. - let drains = !desired.contains_key(&name) - && !disabled.contains(&(track.kind, name.clone())) - && !track.state.cancel_before_live(); - if !drains && let Some(track) = active.remove(&name) { - track.cancel(); - } - } - - // A kept pad the catalog no longer lists ends with EOS, unless a pump still drains its track. - let unwanted = pads.unwanted(|rendition| { - let (kind, name) = rendition; - desired.get(name).is_some_and(|d| d.kind == *kind) - || disabled.contains(rendition) - || active.get(name).is_some_and(|t| t.kind == *kind) - }); - for rendition in unwanted { - if let Some(retire) = pads.retire(&rendition) { - pumps.spawn_on(STREAMING.scope(element.clone(), retire), RUNTIME.handle()); - } + // Listed before any follower hears of the update, so a reader it stops can't claim a pad. + let listed = desired + .keys() + .cloned() + .chain(disabled.iter().map(|(rendition, _)| rendition.clone())) + .collect(); + for retire in playback.pads.list(listed, cancel) { + tasks.spawn_on(STREAMING.scope(playback.element.clone(), retire), RUNTIME.handle()); } - // Spawn pumps for new or changed renditions. The wire container is parsed here, lazily and - // only for renditions we're actually starting, since parsing a CMAF init is wasted work for - // renditions that didn't change. A parse failure (malformed init) skips just this rendition. - for name in plan.add { - let d = &desired[&name]; - let track = Track { - broadcast: source.resolve_reference(d.shape.broadcast.as_ref()), - name: d.shape.track.clone(), - }; - let played = ((d.kind, name.clone()), track.clone()); - if playback.finished.lock().unwrap().contains(&played) { + let mut plans: HashMap = HashMap::new(); + for (rendition, shape) in desired { + let Some(path) = source.resolve_reference(shape.broadcast.as_ref()) else { + gst::warning!( + CAT, + "ignoring rendition {}: its broadcast escapes the root", + rendition.1 + ); continue; - } - let container = match moq_mux::catalog::hang::Container::new( - &d.shape.container, - match d.kind { - TrackKind::Video => moq_mux::container::Kind::Video, - TrackKind::Audio => moq_mux::container::Kind::Audio, - }, - ) { - Ok(container) => container, - Err(err) => { - gst::warning!(CAT, "ignoring rendition {name}: {err:?}"); - continue; - } }; + plans.entry(path).or_default().renditions.insert(rendition, shape); + } - // Bind without waiting: an unavailable sibling or track must not park the catalog loop - // and hold up every other rendition. The pump resolves both under its cancellation scope. - let binding = match source.bind(d.shape.broadcast.as_ref(), d.shape.epoch.as_ref()) { - Ok(binding) => binding, + // A broadcast no enabled rendition reads any more stops its readers, apart from a delisted + // one playing out its tail. Its follower stays reachable, with its finished records, while a + // disabled rendition still names it or a delisted one drains, so a re-enable or relist reaches + // it rather than a new follower. Once it plays nothing, the run lets it go. + let held: HashSet = disabled + .iter() + .filter_map(|(_, reference)| source.resolve_reference(reference.broadcast())) + .collect(); + followers.retain(|path, follower| { + // A follower whose broadcast failed exited, and this update follows it again. + if follower.plan.is_closed() { + return false; + } + follower.held = held.contains(path); + follower.plan.send_replace(plans.remove(path).unwrap_or_default()); + !follower.idle() + }); + for (path, plan) in plans { + // Follow without waiting: an unavailable sibling must not park the catalog loop and hold + // up every other rendition. The follower resolves its instances under the run. Every + // instance is followed, and each rendition checks the epoch it names. + let shape = plan.renditions.values().next().expect("a plan names a rendition"); + let instances = match source.follow(shape.broadcast.as_ref(), None) { + Ok(instances) => instances, Err(err) => { - gst::warning!(CAT, "ignoring rendition {name}: {err:?}"); + gst::warning!(CAT, "ignoring the renditions of {path}: {err:?}"); continue; } }; - - let (cancel_tx, cancel_rx) = watch::channel(false); - let state = Arc::new(PumpState::new()); - let task = pumps.spawn_on( - STREAMING.scope( - element.clone(), - Pump { - element: element.clone(), - kind: d.kind, - name: name.clone(), - caps: d.shape.caps.clone(), - binding, - track, - container, - pads: pads.clone(), - timeline: playback.timeline.clone(), - finished: playback.finished.clone(), - state: state.clone(), - cancel: cancel_rx, - shutdown: shutdown.clone(), - } - .run(), - ), + let (plan_tx, plan) = watch::channel(plan); + let (empty_tx, empty) = watch::channel(false); + let held = held.contains(&path); + let follower_task = Follower { + playback: playback.clone(), + path: path.clone(), + renditions: HashMap::new(), + readers: tokio::task::JoinSet::new(), + current: None, + resumable: HashMap::new(), + finished: Finished::default(), + started: 0, + next_id: 0, + empty: empty_tx, + emptied: emptied.clone(), + } + .run(instances, plan); + tasks.spawn_on( + STREAMING.scope(playback.element.clone(), follower_task), RUNTIME.handle(), ); - - active.insert( - name, - ActiveTrack { - kind: d.kind, - shape: d.shape.clone(), - cancel: cancel_tx, - task, - state, + followers.insert( + path, + Followed { + plan: plan_tx, + empty, + held, }, ); } - - desired - .into_iter() - .map(|(name, d)| (d.kind, name)) - .chain(disabled) - .collect() } -/// Tear-down / spawn decisions for one catalog update, computed purely from the desired and -/// active rendition sets. A name present in both with an equal shape is left untouched; a name -/// whose shape changed lands in both lists (cancel the old pump, spawn a fresh one). -struct ReconcilePlan { - remove: Vec, - add: Vec, +/// A [`Follower`] as its run reaches it. +struct Followed { + plan: watch::Sender, + /// Set while it plays no rendition. + empty: watch::Receiver, + /// A disabled rendition names its path, so a re-enable needs its finished records. + held: bool, } -fn plan_reconcile(desired: &HashMap, active: &HashMap) -> ReconcilePlan { - let remove = active - .iter() - .filter(|(name, shape)| desired.get(*name) != Some(*shape)) - .map(|(name, _)| name.clone()) - .collect(); - let add = desired - .iter() - .filter(|(name, shape)| active.get(*name) != Some(*shape)) - .map(|(name, _)| name.clone()) - .collect(); - ReconcilePlan { remove, add } +impl Followed { + /// Whether nothing reaches it again: it plays nothing, was asked for nothing, and keeps no + /// record a re-enable needs. Judged against the plan sent last, so a plan it has yet to take + /// keeps it. + fn idle(&self) -> bool { + !self.held && *self.empty.borrow() && self.plan.borrow().renditions.is_empty() + } } -/// Pads survive a broadcast switch; its timing and finished tracks do not. +/// What one broadcast run shares with every follower and reader it starts. Pads survive a +/// broadcast switch; its timing does not. +#[derive(Clone)] struct Playback { + element: glib::WeakRef, + /// The catalog's broadcast. A sibling outlives its closed catalog until it ends. + broadcast: moq_net::broadcast::Consumer, pads: Pads, + /// Shared by the catalog's own renditions and the first instance of each sibling. timeline: Arc, + /// The session's shutdown, which [`SessionController::stop`] sets before it flushes the pads. + shutdown: watch::Receiver, + /// Set once the run ends, stopping every follower. + stop: watch::Receiver, +} + +/// The renditions a run asks one [`Follower`] to play. +#[derive(Clone, Default)] +struct Plan { + renditions: HashMap, + /// The catalog closed: the renditions no longer change. The catalog's own follower ends once + /// its readers drain, and a sibling's keeps following its instances until the catalog's + /// broadcast ends. + closed: bool, +} + +/// Follows the instances at one path and plays the renditions that read it on each in turn: a +/// start reads them on that instance, an end idles their pads, and another instance taking over +/// restarts each stream on the same pad. The same instance returning resumes its state. The +/// catalog's own broadcast is a single instance that never ends. +/// +/// It owns its renditions' readers and the state of the instance they read, so whether a +/// rendition is reading, idle, or playing out its tail is its own state rather than a task's. +struct Follower { + playback: Playback, + path: moq_net::PathOwned, + renditions: HashMap, + readers: tokio::task::JoinSet<(Rendition, u64, Exit)>, + /// The instance last started. + current: Option, + /// The number and timeline of each instance a return would resume: the current one and each + /// a rendition pins, by epoch. An epochless instance cannot prove it is the one returning. + resumable: HashMap)>, + /// Shared with the readers. finished: Finished, + /// How many instances started, numbering each. + started: u64, + next_id: u64, + /// The run's [`Followed::empty`]. + empty: watch::Sender, + /// Woken as [`Self::empty`] sets, so the run checks whether to let it go. + emptied: Arc, } -/// Each pad's rendition with a track it played to the end. A new subscription to a finished -/// track replays its last group, so a catalog update that still lists the rendition (a -/// publisher's final snapshot lands after its tracks end) must not start it on that pad again. A -/// pad that never played the track joins it like any late subscriber. Shared with the pumps, -/// since a pump that replaces another on its pad learns this only once the other lets go. -type Finished = Arc>>; +/// The instance a [`Follower`] reads. +struct Current { + /// Released at its end, so an offline sibling holds none of its state. + instance: Option, + /// Its route's, which [`Follower::resumable`] keeps it by. + epoch: Option, + /// Shared by every rendition reading the instance, so their first frames fix one mapping for + /// all of them. + timeline: Arc, + /// Its number among the follower's instances. + serial: u64, +} -/// The first decoded frame fixes one mapping for every rendition of this broadcast run. -#[derive(Default)] -struct Timeline(OnceLock>); +/// A rendition a [`Follower`] plays. +struct Entry { + shape: Shape, + /// Reading the current instance, if it is. + reader: Option, + /// Delisted while reading: plays out the instance it reads, then leaves. + draining: bool, +} -impl Timeline { - fn segment(&self, pts: gst::ClockTime, base: gst::ClockTime) -> &gst::FormattedSegment { - self.0.get_or_init(|| { - let mut segment = gst::FormattedSegment::new(); - segment.set_start(pts); - segment.set_position(pts); - segment.set_base(base); - segment - }) +/// A running [`Reader`], cancelled when dropped. +struct ReaderHandle { + id: u64, + /// Set once the reader owns the rendition's pad. + live: Arc, + cancel: watch::Sender, +} + +impl Drop for ReaderHandle { + fn drop(&mut self) { + let _ = self.cancel.send(true); } } -/// One rendition's pump: everything [`reconcile`] hands a task it spawns. -struct Pump { - element: glib::WeakRef, - kind: TrackKind, - /// The rendition ID, which is also the pad's stream id. The pad's own name comes from a - /// per-kind counter instead, and isn't known until the subscription resolves. - name: String, - caps: gst::Caps, - binding: moq_mux::Binding, - track: Track, - container: moq_mux::catalog::hang::Container, - pads: Pads, +impl Follower { + /// Play the renditions `plan` names on each instance until the plan closes and the last reader + /// drains, the run stops, or the broadcast fails. + async fn run(mut self, mut instances: moq_mux::follow::Instances, mut plan: watch::Receiver) { + let mut stop = self.playback.stop.clone(); + // The catalog's own broadcast is ready at once, so its renditions play even when the + // catalog closed before this follower ran. + match instances.poll_next(&moq_net::kio::Waiter::noop()) { + std::task::Poll::Ready(Ok(moq_mux::follow::Event::Start(instance))) => self.start(instance), + std::task::Poll::Ready(Err(err)) => { + gst::warning!(CAT, "lost {}: {err:?}", self.path); + return; + } + _ => {} + } + let catalog = self.playback.broadcast.clone(); + let sibling = self.path != catalog.info().path; + let first = plan.borrow_and_update().clone(); + // Taking plan updates, until the catalog closes or stops referencing this path. + let mut listening = !first.closed; + // Following instances, which a sibling does past its catalog's close. + let mut following = listening || sibling; + self.apply(first.renditions); + self.play(); + + loop { + // Done once nothing it reads remains, or nothing is left to read and no plan can add + // any: a closed catalog stops renditions changing. + if (!following || !listening && self.renditions.is_empty()) && self.readers.is_empty() { + break; + } + let empty = self.renditions.is_empty(); + if self + .empty + .send_if_modified(|was| std::mem::replace(was, empty) != empty) + && empty + { + self.emptied.notify_one(); + } + + tokio::select! { + biased; + _ = stop.changed() => break, + changed = plan.changed(), if listening => match changed { + Ok(()) => { + let next = plan.borrow_and_update().clone(); + listening = !next.closed; + following = listening || sibling; + let fresh = self.apply(next.renditions); + // The final plan starts what it adds or reshapes, but restarts no reader + // that already ended. + match listening { + true => self.play(), + false => self.play_where(|rendition| fresh.contains(rendition)), + } + } + // No rendition reads this path any more. + Err(_) => { + (listening, following) = (false, false); + self.apply(HashMap::new()); + } + }, + Some(joined) = self.readers.join_next() => { + if let Ok((rendition, id, exit)) = joined { + self.exited(rendition, id, exit); + } + } + event = instances.next(), if following => match event { + Ok(moq_mux::follow::Event::Start(instance)) => { + gst::info!(CAT, "{} is online, starting its renditions", self.path); + self.start(instance); + } + Ok(moq_mux::follow::Event::End) => { + gst::info!(CAT, "{} is offline, holding its pads", self.path); + self.offline(); + } + Err(err) => { + gst::warning!(CAT, "lost {}: {err:?}", self.path); + break; + } + }, + // The catalog's broadcast ended after its catalog: the run is offline, so its + // sibling renditions idle with it. + _ = catalog.closed(), if following && !listening => { + following = false; + self.offline(); + } + } + } + + self.renditions.clear(); + while self.readers.join_next().await.is_some() {} + } + + /// Bring the renditions in line with `wanted`, stopping each one reshaped or gone, and return + /// those added or reshaped. [`Self::play`] starts the rest. + fn apply(&mut self, mut wanted: HashMap) -> HashSet { + let pads = &self.playback.pads; + let mut fresh = HashSet::new(); + self.renditions + .retain(|rendition, entry| match wanted.remove(rendition) { + Some(shape) => { + // A new shape hands the pad to a new reader. + if shape != entry.shape { + entry.shape = shape; + entry.reader = None; + fresh.insert(rendition.clone()); + } + entry.draining = false; + true + } + // A delisted rendition plays out the instance it reads, then leaves. One that never + // took its pad, or that is disabled or now reads another broadcast, stops now. + None => { + entry.draining = !pads.is_listed(rendition) + && entry + .reader + .as_ref() + .is_some_and(|reader| reader.live.load(Ordering::Acquire)); + entry.draining + } + }); + for (rendition, shape) in wanted { + fresh.insert(rendition.clone()); + self.renditions.insert( + rendition, + Entry { + shape, + reader: None, + draining: false, + }, + ); + } + self.forget(); + fresh + } + + /// Read `instance`, which took over from any before it. + fn start(&mut self, instance: moq_net::broadcast::Consumer) { + // Announced again without an end, its readers carry on. + if let Some(Current { + instance: Some(current), + .. + }) = &self.current + && current.is_clone(&instance) + { + self.play(); + return; + } + + // An instance served again keeps its timeline and the tracks it finished. + let epoch = instance.info().epoch.clone(); + let (serial, timeline) = match epoch.as_ref().and_then(|epoch| self.resumable.get(epoch)) { + Some((serial, timeline)) => (*serial, timeline.clone()), + None => { + self.started += 1; + // The first instance is the one the run started alongside. A later one's + // timestamps share no origin with the run's. + let timeline = match self.started { + 1 => self.playback.timeline.clone(), + _ => Arc::default(), + }; + (self.started, timeline) + } + }; + if let Some(epoch) = &epoch { + self.resumable.insert(epoch.clone(), (serial, timeline.clone())); + } + self.current = Some(Current { + instance: Some(instance), + epoch, + timeline, + serial, + }); + self.forget(); + self.idle(); + self.play(); + } + + /// Drop the state no return can resume: an instance neither current nor pinned by a rendition, + /// and the records of a retired pad. A disabled rendition keeps its pad, and its records. + fn forget(&mut self) { + let pinned: HashSet<&moq_net::Epoch> = self + .renditions + .values() + .filter_map(|entry| entry.shape.epoch.as_ref()) + .chain(self.current.as_ref().and_then(|current| current.epoch.as_ref())) + .collect(); + self.resumable.retain(|epoch, _| pinned.contains(epoch)); + let serials: HashSet = self + .resumable + .values() + .map(|(serial, _)| *serial) + .chain(self.current.as_ref().map(|current| current.serial)) + .collect(); + let pads = &self.playback.pads; + self.finished + .retain(|rendition, serial| pads.is_listed(rendition) && serials.contains(&serial)); + } + + /// The instance ended: release it and idle until the next one. + fn offline(&mut self) { + if let Some(current) = &mut self.current { + current.instance = None; + } + self.idle(); + } + + /// Stop reading the instance. A draining rendition leaves with it. + fn idle(&mut self) { + self.renditions.retain(|_, entry| !entry.draining); + for entry in self.renditions.values_mut() { + entry.reader = None; + } + } + + /// Start a reader on the current instance for each rendition without one that it may serve. + fn play(&mut self) { + self.play_where(|_| true); + } + + /// [`Self::play`] for the renditions `only` accepts. + fn play_where(&mut self, only: impl Fn(&Rendition) -> bool) { + let Some(current) = &self.current else { + return; + }; + let Some(instance) = ¤t.instance else { + return; + }; + for (rendition, entry) in &mut self.renditions { + if entry.reader.is_none() && only(rendition) && serves(instance, entry.shape.epoch.as_ref()) { + self.next_id += 1; + let (cancel, cancelled) = watch::channel(false); + let live = Arc::new(AtomicBool::new(false)); + let reader = Reader { + playback: self.playback.clone(), + rendition: rendition.clone(), + shape: entry.shape.clone(), + instance: instance.clone(), + serial: current.serial, + timeline: current.timeline.clone(), + finished: self.finished.clone(), + live: live.clone(), + cancel: cancelled, + }; + let (rendition, id) = (rendition.clone(), self.next_id); + self.readers.spawn_on( + STREAMING.scope(self.playback.element.clone(), async move { + let exit = reader.run().await; + (rendition, id, exit) + }), + RUNTIME.handle(), + ); + entry.reader = Some(ReaderHandle { id, live, cancel }); + } + } + } + + /// The rendition's reader `id` exited on its own. + fn exited(&mut self, rendition: Rendition, id: u64, exit: Exit) { + let Some(entry) = self.renditions.get_mut(&rendition) else { + return; + }; + if entry.reader.as_ref().is_none_or(|reader| reader.id != id) { + return; + } + entry.reader = None; + if entry.draining { + self.renditions.remove(&rendition); + } else if let Exit::Unlisted = exit + && self.playback.pads.is_listed(&rendition) + { + // Delisted and listed again while it went to claim its pad, which the plan coalesced. + self.play(); + } + } +} + +/// Whether `instance` may serve a rendition naming `epoch`. An epochless route cannot prove its +/// instance, so only a known mismatch refuses it. +fn serves(instance: &moq_net::broadcast::Consumer, epoch: Option<&moq_net::Epoch>) -> bool { + match (epoch, &instance.info().epoch) { + (Some(expected), Some(actual)) => expected == actual, + _ => true, + } +} + +/// Why a [`Reader`] exited. +enum Exit { + /// It is done with the instance. + Done, + /// The catalog did not list the rendition when it went to claim its pad. + Unlisted, +} + +/// One rendition reading one instance: everything [`Follower::play`] hands a task it spawns. +struct Reader { + playback: Playback, + rendition: Rendition, + shape: Shape, + instance: moq_net::broadcast::Consumer, + /// The instance's [`Current::serial`]. + serial: u64, timeline: Arc, - /// The run's [`Playback::finished`]. + /// The follower's [`Follower::finished`]. finished: Finished, - /// Shared with this rendition's [`ActiveTrack::state`]. - state: Arc, + /// Shared with this reader's [`ReaderHandle::live`]. + live: Arc, cancel: watch::Receiver, - /// The session's shutdown, which [`SessionController::stop`] sets before it flushes the pads. - shutdown: watch::Receiver, } -impl Pump { +impl Reader { /// Subscribe to the track, then stream its frames to the rendition's pad, owning it until the - /// track ends, errors, or `cancel` fires. The pad goes back to the session without an EOS - /// either way. Returns the rendition if its track ended rather than being cancelled, so the - /// run can end the pad of one the catalog retired. - async fn run(self) -> Option { - let Pump { - element, - kind, - name, - caps, - binding, - track: media, - container, - pads, + /// track ends, errors, its instance stops serving, or `cancel` fires. The pad stays with the + /// session either way. + async fn run(self) -> Exit { + let Reader { + playback, + rendition, + shape, + instance, + serial, timeline, finished, - state, + live, mut cancel, - shutdown, } = self; - // Resolves once the publisher answers, with the track info or with an error (which is - // what ending a broadcast produces for a name nobody served). A publisher that answers - // neither leaves this waiting, so racing `cancel` keeps such a pump reapable, and - // holding the wait here rather than in `reconcile` keeps it off every other - // rendition. + let Playback { + element, + pads, + shutdown, + .. + } = playback; + let (kind, name) = (rendition.0, &rendition.1); + + // An instance that already played this track to its end on the pad would replay its last + // group. + if finished.contains(&rendition, &shape.track, serial) { + return Exit::Done; + } + + // Resolves once the publisher answers, with the track info or with an error (which is what + // ending a broadcast produces for a name nobody served). A publisher that answers neither + // leaves this waiting, so racing `cancel` keeps such a reader reapable, and holding the + // wait here keeps it off every other rendition. let subscriber = tokio::select! { - _ = cancel.changed() => return None, + _ = cancel.changed() => return Exit::Done, subscriber = async { - let broadcast = binding.broadcast().await?; - let track = broadcast.track(&media.name)?; - Ok::<_, moq_mux::Error>(track.subscribe(moq_net::track::Subscription::default().with_max_delay(Duration::from_secs(1))).await?) + instance + .track(&shape.track)? + .subscribe(moq_net::track::Subscription::default().with_max_delay(Duration::from_secs(1))) + .await } => match subscriber { Ok(subscriber) => subscriber, + // The instance went while subscribing: that is its end, not this rendition's. + Err(_) if instance.is_closed() => return Exit::Done, Err(err) => { gst::warning!(CAT, "track {name} failed to subscribe: {err:?}"); - return Some((kind, name)); + return Exit::Done; } } }; + let container = match moq_mux::catalog::hang::Container::new( + &shape.container, + match kind { + TrackKind::Video => moq_mux::container::Kind::Video, + TrackKind::Audio => moq_mux::container::Kind::Audio, + }, + ) { + Ok(container) => container, + Err(err) => { + gst::warning!(CAT, "ignoring rendition {name}: {err:?}"); + return Exit::Done; + } + }; let mut track = moq_mux::container::Consumer::new(subscriber, container); - // Winning this is what earns a pad. Losing means a teardown got here while the - // subscription was still resolving (this rendition was removed, reshaped, or outlived by - // a closing catalog), and it must not publish a pad at all: the watch alone can't say - // that, since a cancel landing just after we read it would leave a pad exposed and then - // yanked without an EOS. - if !state.go_live() { - return None; - } - // A pad appears only once its track is live, so a pad downstream can link to is a promise // that the rendition is actually flowing, and only a rendition that gets this far claims a // pad id. A kept pad starts a new stream on it, with caps pushed even when they differ, so // a format downstream can't take fails loudly as not-negotiated rather than going quiet. - let (pad, _owned) = match pads.claim(&element, (kind, name.clone()), &cancel)? { - Claim::New(pad, owned) => { - let obj = element.upgrade()?; - pad.set_active(true).ok()?; - begin(&pad, &name, &caps); - obj.add_pad(&pad).ok()?; + let (pad, _owned) = match pads.claim(&element, &rendition, &cancel) { + None => return Exit::Done, + Some(Claim::Unlisted) => return Exit::Unlisted, + Some(Claim::New(pad, owned)) => { + let Some(obj) = element.upgrade() else { + return Exit::Done; + }; + if pad.set_active(true).is_err() { + return Exit::Done; + } + begin(&pad, name, &shape.caps); + if obj.add_pad(&pad).is_err() { + return Exit::Done; + } (pad, owned) } - Claim::Kept(pad, owner) => { + Some(Claim::Kept(pad, owner)) => { let owned = tokio::select! { biased; - _ = cancel.changed() => return None, - owned = owner.lock_owned() => owned, + _ = cancel.changed() => return Exit::Done, + owned = owner.clone().lock_owned() => owned, }; - // The pump this one replaces records its track as finished before letting go, so a - // replacement started while it was reading the end (a reshape) can't replay it. - if finished - .lock() - .unwrap() - .contains(&((kind, name.clone()), media.clone())) - { - return Some((kind, name)); + // The reader this one replaces records its track as finished before letting go, so + // one started while it was reading the end (a reshape) can't replay it. A + // retirement that got the pad first ended it. + if !pads.holds(&rendition, &owner) || finished.contains(&rendition, &shape.track, serial) { + return Exit::Done; } - tokio::task::block_in_place(|| begin(&pad, &name, &caps)); + tokio::task::block_in_place(|| begin(&pad, name, &shape.caps)); (pad, owned) } }; + live.store(true, Ordering::Release); // Stop flushes only the pads it finds, and this one may have been added just after, while - // this pump's cancel is still on its way. Flushing it here keeps a push from blocking stop. + // this reader's cancel is still on its way. Flushing it here keeps a push from blocking + // stop. if *shutdown.borrow() { let _ = pad.set_active(false); } @@ -1226,14 +1532,14 @@ impl Pump { let mut sent_segment = false; loop { tokio::select! { - // This rendition is being torn down (its run ended, or a catalog update replaced it). An - // end that arrived alongside still finishes the track, or the replacement waiting for - // this pad would replay it. + // The follower stopped this reader (the instance changed, the rendition was + // reshaped or delisted, or the run ended). An end that arrived alongside still + // finishes the track, or a reader waiting for this pad would replay it. _ = cancel.changed() => { if ended(&mut track) { - finished.lock().unwrap().insert(((kind, name.clone()), media)); + finished.insert(&rendition, &shape.track, serial); } - return None; + return Exit::Done; } frame = track.read() => match frame { Ok(Some(moq_mux::container::Read::Gap { .. } | moq_mux::container::Read::End { .. })) => {} @@ -1244,7 +1550,7 @@ impl Pump { if let Some(obj) = element.upgrade() { gst::element_error!(obj, gst::StreamError::Format, ("track {name} has an invalid timestamp"), ["{err:#}"]); } - return Some((kind, name)); + return Exit::Done; } }; let segment = (!sent_segment).then(|| { @@ -1255,7 +1561,7 @@ impl Pump { // pad.push() blocks until downstream accepts the buffer (full queues, a // clock-synced sink). block_in_place hands our sibling tasks to another // worker so a stalled downstream can't pin a runtime thread and starve - // the session loop or other pumps. + // the session loop or other readers. let pushed = tokio::task::block_in_place(|| { if let Some(segment) = segment { pad.push_event(segment); @@ -1266,21 +1572,25 @@ impl Pump { Ok(_) => {} Err(gst::FlowError::NotNegotiated) => { if let Some(obj) = element.upgrade() { - gst::element_error!(obj, gst::StreamError::Format, ("track {name} was not negotiated"), ["caps {caps}"]); + gst::element_error!(obj, gst::StreamError::Format, ("track {name} was not negotiated"), ["caps {}", shape.caps]); } - return Some((kind, name)); + return Exit::Done; } - Err(_) => return Some((kind, name)), + Err(_) => return Exit::Done, } } + // The track stopping with its instance is that instance's end, which idles the + // pad for the next one rather than ending the rendition. + Ok(None) | Err(_) if instance.is_closed() => return Exit::Done, + // Recorded while this reader still owns the pad, before any other can claim it, + // so a reshape on this instance does not replay it. Ok(None) => { - // Recorded while this pump still owns the pad, before any replacement can claim it. - finished.lock().unwrap().insert(((kind, name.clone()), media)); - return Some((kind, name)); + finished.insert(&rendition, &shape.track, serial); + return Exit::Done; } Err(err) => { gst::warning!(CAT, "track {name} failed: {err:?}"); - return Some((kind, name)); + return Exit::Done; } } } @@ -1288,8 +1598,62 @@ impl Pump { } } -/// Whether the track's end is the next media item, without waiting for it. Markers ready before it -/// are passed over; a frame or anything not ready yet means it has not ended. +/// The track each of a follower's renditions last played to the end on its pad, per instance the +/// follower keeps, by serial. A new subscription to a finished track replays its last group, so a +/// catalog update that still lists the rendition (a publisher's final snapshot lands after its +/// tracks end, or a disable and an enable) must not start it on that pad again. Another instance +/// carries a new track, and a pad that never played the track joins it like any late subscriber. +/// Only the last track is kept, so the records stay bounded by the catalog: a rendition switched +/// back to a track it finished before another replays its last group. Shared with the readers, +/// since a reader that replaces another on its pad learns this only once the other lets go. +#[derive(Clone, Default)] +struct Finished(Arc>>); + +impl Finished { + /// Record that instance `serial` served the rendition's `track` to its end. + fn insert(&self, rendition: &Rendition, track: &str, serial: u64) { + self.0 + .lock() + .unwrap() + .insert((rendition.clone(), serial), track.to_string()); + } + + /// Whether instance `serial` already served the rendition's `track` to its end on its pad. + fn contains(&self, rendition: &Rendition, track: &str, serial: u64) -> bool { + self.0 + .lock() + .unwrap() + .get(&(rendition.clone(), serial)) + .is_some_and(|finished| finished == track) + } + + /// Keep only the records `keep` accepts. + fn retain(&self, keep: impl Fn(&Rendition, u64) -> bool) { + self.0 + .lock() + .unwrap() + .retain(|(rendition, serial), _| keep(rendition, *serial)); + } +} + +/// The first decoded frame fixes one mapping for every rendition reading an instance. +#[derive(Default)] +struct Timeline(OnceLock>); + +impl Timeline { + fn segment(&self, pts: gst::ClockTime, base: gst::ClockTime) -> &gst::FormattedSegment { + self.0.get_or_init(|| { + let mut segment = gst::FormattedSegment::new(); + segment.set_start(pts); + segment.set_position(pts); + segment.set_base(base); + segment + }) + } +} + +/// Whether the track's end is the next media item, without waiting for it. Markers ready before it +/// are passed over; a frame or anything not ready yet means it has not ended. fn ended(track: &mut moq_mux::container::Consumer) -> bool { let mut cx = std::task::Context::from_waker(std::task::Waker::noop()); loop { @@ -1470,11 +1834,10 @@ fn opus_caps(config: &hang::catalog::AudioConfig, rate: i32, channels: i32) -> R #[cfg(test)] mod tests { - use super::{PumpState, Timeline, ended, plan_reconcile}; + use super::{Timeline, ended}; use moq_net::Timestamp; - use std::collections::HashMap; - /// A pump cancelled as its track ends records the end only when no media is left to read, + /// A reader cancelled as its track ends records the end only when no media is left to read, /// passing over the markers a publisher closes with. #[test] fn ended_sees_only_an_end_with_no_media_before_it() { @@ -1557,53 +1920,6 @@ mod tests { } } - // The shape type is generic, so the set math can be exercised with a plain integer standing - // in for (caps, container): equal value == unchanged rendition, different value == reshape. - fn renditions(pairs: &[(&str, u32)]) -> HashMap { - pairs.iter().map(|(name, shape)| (name.to_string(), *shape)).collect() - } - - fn sorted(mut names: Vec) -> Vec { - names.sort(); - names - } - - #[test] - fn plan_reconcile_diffs_by_name_and_shape() { - // keep: same shape (untouched). gone: removed. added: new. changed: same name, new - // shape, so it must be both torn down and respawned. - let active = renditions(&[("keep", 1), ("gone", 1), ("changed", 1)]); - let desired = renditions(&[("keep", 1), ("changed", 2), ("added", 9)]); - - let plan = plan_reconcile(&desired, &active); - assert_eq!(sorted(plan.remove), vec!["changed", "gone"]); - assert_eq!(sorted(plan.add), vec!["added", "changed"]); - } - - #[test] - fn plan_reconcile_noops_on_identical_sets() { - let set = renditions(&[("a", 1), ("b", 2)]); - let plan = plan_reconcile(&set, &set); - assert!(plan.remove.is_empty()); - assert!(plan.add.is_empty()); - } - - #[test] - fn plan_reconcile_empty_desired_removes_all() { - let active = renditions(&[("a", 1), ("b", 2)]); - let plan = plan_reconcile(&HashMap::new(), &active); - assert_eq!(sorted(plan.remove), vec!["a", "b"]); - assert!(plan.add.is_empty()); - } - - #[test] - fn plan_reconcile_empty_active_adds_all() { - let desired = renditions(&[("a", 1), ("b", 2)]); - let plan = plan_reconcile(&desired, &HashMap::new()); - assert!(plan.remove.is_empty()); - assert_eq!(sorted(plan.add), vec!["a", "b"]); - } - #[test] fn the_first_frame_fixes_the_segment_start_and_base() { gst::init().unwrap(); @@ -1640,37 +1956,14 @@ mod tests { }; assert!(super::build_buffer(frame, super::TrackKind::Video).is_err()); } - - /// The pad claim and the teardown race for every pump, and only one may win: a subscription - /// resolving after a teardown must not publish a rendition the session has finished with and - /// then yank it without an EOS. The cancel watch alone cannot say which happened, so this is - /// the state that does. - #[test] - fn a_pump_either_goes_live_or_is_cancelled() { - let cancelled = PumpState::new(); - assert!(cancelled.cancel_before_live()); - assert!(!cancelled.go_live(), "a cancelled pump still claimed a pad"); - assert!( - !cancelled.cancel_before_live(), - "a second teardown claimed the same transition" - ); - - let live = PumpState::new(); - assert!(live.go_live()); - assert!( - !live.cancel_before_live(), - "a live pump was dropped without its cancel watch" - ); - assert!(!live.go_live(), "a live pump claimed a second pad"); - } } #[cfg(test)] #[allow(clippy::disallowed_methods, clippy::disallowed_types)] mod session_tests { - use std::collections::BTreeMap; - use std::sync::Mutex; + use std::collections::{BTreeMap, HashSet}; use std::sync::atomic::Ordering; + use std::sync::{Arc, Mutex}; use std::time::Duration; use gst::glib; @@ -1692,7 +1985,7 @@ mod session_tests { PAD_IDS.lock().unwrap_or_else(|err| err.into_inner()) } - /// The pumps push from their own tasks, so the element only has to exist and own pads. + /// The readers push from their own tasks, so the element only has to exist and own pads. fn element() -> super::super::MoqSrc { gst::init().unwrap(); glib::Object::new() @@ -1749,7 +2042,7 @@ mod session_tests { } /// Real wire consumers and source pads, with each write acknowledged by a pad probe. - /// No arrival-time guesses: the first acknowledgement fixes which pump won the run. + /// No arrival-time guesses: the first acknowledgement fixes which reader won the run. #[test] fn audio_and_video_share_a_run_timeline_across_scales_and_restarts() { let _pad_ids = pad_ids(); @@ -1854,7 +2147,7 @@ mod session_tests { }) .unwrap(); rx.recv_timeout(Duration::from_secs(10)) - .expect("pump did not deliver frame") + .expect("reader did not deliver frame") }; let (name, pts, segment) = write(first, reference); assert!(name.starts_with(if audio_first { "audio" } else { "video" })); @@ -1898,7 +2191,7 @@ mod session_tests { .expect("broadcast closed") } - /// Poll for a pad rather than sleeping a fixed beat: the pumps run on another runtime, so + /// Poll for a pad rather than sleeping a fixed beat: the readers run on another runtime, so /// the only ordering we have is "eventually". Fails the test if it never shows up. fn await_pad(element: &super::super::MoqSrc, kind: &str) -> gst::Pad { for _ in 0..100 { @@ -1960,7 +2253,7 @@ mod session_tests { stopping.send(true).unwrap(); super::RUNTIME.block_on(following).unwrap().unwrap(); assert_eq!( - next.expect("the broadcast change did not replace its pump") + next.expect("the broadcast change did not replace its reader") .unwrap() .name(), "encoded" @@ -2046,12 +2339,11 @@ mod session_tests { super::RUNTIME.block_on(session).unwrap().unwrap(); } - /// A rendition delisted while its pump is still subscribing ends that pump, and answering - /// the subscription afterwards must not resurrect it into a pad. Which of the two the pump - /// sees first is the runtime's to decide, so the state machine that refuses the losing side - /// is covered by `a_pump_either_goes_live_or_is_cancelled` instead. + /// A rendition delisted while its reader is still subscribing ends that reader, and answering + /// the subscription afterwards must not resurrect it into a pad. Listing it again starts it + /// anew, since the reader that left is no longer the rendition's. #[test] - fn a_rendition_delisted_while_subscribing_takes_no_pad() { + fn a_rendition_delisted_while_subscribing_takes_no_pad_until_relisted() { let _pad_ids = pad_ids(); let element = element(); @@ -2069,7 +2361,7 @@ mod session_tests { let weak = element.downgrade(); let session = run(consumer, weak, shutdown_rx); - // Hold the subscription pending, then drop the rendition from the catalog so its pump + // Hold the subscription pending, then drop the rendition from the catalog so its reader // is cancelled while it is still waiting. let request = await_request(&mut dynamic); { @@ -2077,9 +2369,9 @@ mod session_tests { guard.video.renditions.clear(); } - // A cancelled pump returns out of its subscribe, dropping the only consumer this request + // A cancelled reader returns out of its subscribe, dropping the only consumer this request // has: that edge says the session reconciled the removal, which a fixed beat can only - // guess at. Answering before it lands is a pump that legitimately goes live, so the + // guess at. Answering before it lands is a reader that legitimately goes live, so the // wait is what the assertion below is about. let mut demand = request.demand(); let unwanted = async { @@ -2088,13 +2380,18 @@ mod session_tests { }; super::RUNTIME .block_on(async { tokio::time::timeout(Duration::from_secs(10), unwanted).await }) - .expect("the cancelled pump never dropped its subscription") + .expect("the cancelled reader never dropped its subscription") .expect("the pending request is still open"); - // Only now answer it. The pump is gone and its state is terminal, so no later scheduling - // can produce a pad. + // Only now answer it. The reader is gone, so no later scheduling can produce a pad. let _serving = request.accept(moq_net::track::Info::default()); - assert!(pads(&element, "video_").is_empty(), "a cancelled pump still took a pad"); + assert!( + pads(&element, "video_").is_empty(), + "a cancelled reader still took a pad" + ); + + catalog.modify().unwrap().video.renditions = BTreeMap::from([("stalled".to_string(), video_rendition())]); + await_pad(&element, "video_"); let _ = shutdown.send(true); super::RUNTIME.block_on(session).unwrap().unwrap(); @@ -2130,9 +2427,40 @@ mod session_tests { super::RUNTIME.block_on(session).unwrap().unwrap(); } + /// A rendition the catalog's final snapshot adds plays, even when the snapshot and the close + /// reach the follower together. + #[test] + fn a_closing_catalog_plays_the_rendition_its_final_snapshot_adds() { + let _pad_ids = pad_ids(); + let element = element(); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let _video = broadcast.create_track("video", None).unwrap(); + let _audio = broadcast.create_track("audio", None).unwrap(); + { + let mut guard = catalog.modify().unwrap(); + guard.video.renditions = BTreeMap::from([("video".to_string(), video_rendition())]); + } + + let (shutdown, shutdown_rx) = watch::channel(false); + let session = run(broadcast.consume(), element.downgrade(), shutdown_rx); + await_pad(&element, "video_"); + + { + let mut guard = catalog.modify().unwrap(); + guard.audio.renditions = BTreeMap::from([("audio".to_string(), audio_rendition())]); + } + catalog.finish().unwrap(); + await_pad(&element, "audio_"); + + let _ = shutdown.send(true); + super::RUNTIME.block_on(session).unwrap().unwrap(); + } + /// The flip side of the test above: a rendition the publisher reserved and never served is /// answerable once the publisher ends the broadcast, which resolves every name it never - /// filled. That pump must end on its own so the run ends on the media it was serving, and the + /// filled. That reader must end on its own so the run ends on the media it was serving, and the /// session can follow the path to its next start. The pad it served is held, not ended: the /// broadcast is offline, not retired. #[test] @@ -2190,7 +2518,7 @@ mod session_tests { /// A publisher that ends one rendition finishes its track and retires it from the catalog, /// which is what `moqsink` does on EOS. The two travel on different tracks, so the catalog - /// update can reach the subscriber before the tail of the media does. It must not cut the pump + /// update can reach the subscriber before the tail of the media does. It must not cut the reader /// short: the pad owes downstream every frame of the track and an EOS. /// /// The subscriber's view of that arrival order is reproduced exactly: the head of the track, @@ -2274,7 +2602,7 @@ mod session_tests { ); // The catalog update retiring the rendition arrives next. The catalog and the broadcast - // stay open, so nothing else can end the pump. The same update lists an audio rendition: + // stay open, so nothing else can end the reader. The same update lists an audio rendition: // its pad appearing proves the session acted on the update before the tail is written. { let mut guard = catalog.modify().unwrap(); @@ -2283,7 +2611,7 @@ mod session_tests { } await_pad(&element, "audio_"); - // A pump the update cancelled still has to notice. It takes its pad with it when it does, + // A reader the update cancelled still has to notice. It takes its pad with it when it does, // so the wait is only ever served in full by one that is still reading. for _ in 0..10 { if pads(&element, "video_").is_empty() { @@ -2298,14 +2626,14 @@ mod session_tests { } producer.finish().unwrap(); - // The pump removes its pad on the way out, whichever way it goes. + // The reader removes its pad on the way out, whichever way it goes. for _ in 0..100 { if pads(&element, "video_").is_empty() { break; } std::thread::sleep(Duration::from_millis(50)); } - assert!(pads(&element, "video_").is_empty(), "the pump never ended"); + assert!(pads(&element, "video_").is_empty(), "the reader never ended"); let _ = shutdown.send(true); super::RUNTIME.block_on(session).unwrap().unwrap(); @@ -2319,7 +2647,7 @@ mod session_tests { } /// A publisher that delists a rendition without finishing its track and lists it again later - /// (the browser toggling a source) is resuming the same track. The pump that is still reading + /// (the browser toggling a source) is resuming the same track. The reader that is still reading /// it carries on under the same pad, and a track that ends while listed keeps its pad without /// an EOS, since the announcements say whether the broadcast comes back. #[test] @@ -2471,7 +2799,7 @@ mod session_tests { .unwrap(); catalog.modify().unwrap().video.renditions = BTreeMap::from([("video".to_string(), video_rendition())]); - // Count what reaches the first video pad once it is added. A pump adds a new pad from the + // Count what reaches the first video pad once it is added. A reader adds a new pad from the // signal, before it pushes anything, so every buffer and every later stream is seen. let buffers = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)); let restarts = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)); @@ -2527,15 +2855,15 @@ mod session_tests { } producer.finish().unwrap(); - // The pump owns its pad until it has pushed the whole track and read its end. - let owner = pads.0.lock().unwrap()[&(TrackKind::Video, "video".to_string())] + // The reader owns its pad until it has pushed the whole track and read its end. + let owner = pads.0.lock().unwrap().slots[&(TrackKind::Video, "video".to_string())] .owner .clone(); drop(super::RUNTIME.block_on(owner.lock_owned())); assert_eq!(buffers.load(Ordering::Relaxed), FRAMES, "the track never played"); // The update reshapes the finished rendition over the same track, which hands its pad to a - // new pump, and lists a second rendition of that track. Its audio pad proves the session + // new reader, and lists a second rendition of that track. Its audio pad proves the session // acted on it. { let mut guard = catalog.modify().unwrap(); @@ -2553,7 +2881,7 @@ mod session_tests { "the update added {added:?}" ); - // The run ends only once every pump drains, a replaying one included. + // The run ends only once every reader drains, a replaying one included. audio.finish().unwrap(); catalog.finish().unwrap(); super::RUNTIME.block_on(session).unwrap().unwrap(); @@ -2708,6 +3036,13 @@ mod session_tests { publisher } + /// The `cam` broadcast a rendition of `room` references as `./cam`, under `route`. + fn sibling(origin: &moq_net::origin::Producer, route: moq_net::origin::Route) -> Self { + let publisher = Self::build(origin.create_broadcast("cam").unwrap(), video_rendition()); + publisher.broadcast.announce(route).unwrap(); + publisher + } + /// A broadcast at no path, for a covering route to serve. fn standalone(config: VideoConfig) -> Self { Self::build(moq_net::broadcast::Info::new().produce(), config) @@ -2748,6 +3083,11 @@ mod session_tests { }) .unwrap(); } + + /// Move the timestamps of later frames `by` ahead. + fn skip(&mut self, by: Duration) { + self.start -= by; + } } // A dial still running once the element reached NULL can outlive `main`, and aws-lc aborts the @@ -2810,7 +3150,7 @@ mod session_tests { assert!(is_closed(&connection)); } - /// Stop waits for every pump to remove its pad, so a pump held in a push has to be let go + /// Stop waits for every reader to remove its pad, so a reader held in a push has to be let go /// first, or stop never returns. #[test] fn stop_releases_a_blocked_push() { @@ -2836,8 +3176,8 @@ mod session_tests { assert!(is_closed(&connection)); } - /// A pump whose subscription resolves while stop flushes the pads adds its own just after, with - /// its cancel still on the way. A push on that pad must not block stop either. + /// A reader whose subscription resolves while stop flushes the pads adds its own just after, + /// with its cancel still on the way. A push on that pad must not block stop either. #[test] fn a_pad_added_after_stop_flushed_does_not_block() { let _pad_ids = pad_ids(); @@ -2850,36 +3190,47 @@ mod session_tests { }); publisher.write(0); + let rendition = (TrackKind::Video, "video".to_string()); let (_cancel, cancel) = watch::channel(false); let (_shutdown, shutdown) = watch::channel(true); - let pump = super::Pump { - element: element.downgrade(), - kind: super::TrackKind::Video, - name: "video".into(), - caps: gst::Caps::new_empty_simple("video/x-h264"), - binding: moq_mux::Binding::new(publisher.broadcast.consume()), - track: super::Track { + let (_stop, stop) = watch::channel(false); + let pads = Pads::default(); + assert!(pads.list(HashSet::from([rendition.clone()]), &cancel).is_empty()); + let instance = publisher.broadcast.consume(); + let reader = super::Reader { + playback: super::Playback { + element: element.downgrade(), + broadcast: instance.clone(), + pads, + timeline: Arc::default(), + shutdown, + stop, + }, + rendition, + shape: super::Shape { + caps: gst::Caps::new_empty_simple("video/x-h264"), + container: Container::Legacy, + track: "video".into(), broadcast: None, - name: "video".into(), + epoch: None, }, - container: moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), - pads: Pads::default(), - timeline: std::sync::Arc::default(), - finished: std::sync::Arc::default(), - state: std::sync::Arc::new(super::PumpState::new()), + instance, + serial: 1, + timeline: Arc::default(), + finished: super::Finished::default(), + live: Arc::default(), cancel, - shutdown, }; super::RUNTIME - .block_on(async { tokio::time::timeout(Duration::from_secs(10), super::RUNTIME.spawn(pump.run())).await }) + .block_on(async { tokio::time::timeout(Duration::from_secs(10), super::RUNTIME.spawn(reader.run())).await }) .expect("a push on a pad added after the flush blocked") .unwrap(); } - /// A bus sync or pad handler can stop the element from inside a pump's push. Waiting for the - /// session there would wait on that pump, so stop settles for the connection closing. + /// A bus sync or pad handler can stop the element from inside a reader's push. Waiting for the + /// session there would wait on that reader, so stop settles for the connection closing. #[test] - fn stop_from_its_own_pump_does_not_deadlock() { + fn stop_from_its_own_reader_does_not_deadlock() { let _pad_ids = pad_ids(); let element = element(); let origin = origin(); @@ -2902,7 +3253,7 @@ mod session_tests { publisher.write(0); returned .recv_timeout(Duration::from_secs(10)) - .expect("stop from the pump never returned"); + .expect("stop from the reader never returned"); assert!(is_closed(&connection)); } @@ -2917,6 +3268,8 @@ mod session_tests { /// A buffer, by its tag. Buffer(String, u8), Eos(String), + /// A segment, by its start. + Segment(String, i64), } /// Records what reaches the element's video pads. A new pad's first stream-start and caps @@ -2950,6 +3303,9 @@ mod session_tests { seen.push(Seen::Caps(name.clone(), media)); } gst::EventView::Eos(_) => seen.push(Seen::Eos(name.clone())), + gst::EventView::Segment(segment) => { + seen.push(Seen::Segment(name.clone(), segment.segment().start().value())) + } _ => {} }, _ => {} @@ -3117,6 +3473,677 @@ mod session_tests { recorder.assert_one_pad_held(); } + /// A `room` whose video rendition references the `cam` sibling, naming `epoch` when given. + fn referencing(origin: &moq_net::origin::Producer, epoch: Option) -> Publisher { + let mut config = video_rendition(); + config.broadcast = Some("./cam".into()); + config.epoch = epoch; + Publisher::new(origin, Default::default(), config) + } + + /// A sibling announced after the catalog starts its rendition, its end holds the pad, and its + /// return starts a new stream on that pad. + #[test] + fn a_late_sibling_starts_its_rendition_and_returns_after_an_end() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let _room = referencing(&origin, None); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + let mut follow = origin.consume().follow("cam").unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + drop(first); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::End(_))); + + let mut second = Publisher::sibling(&origin, Default::default()); + assert_eq!(recorder.written(&mut second, 2), pad, "the return moved to another pad"); + stop(session); + + assert!( + recorder.seen().contains(&Seen::Start(pad)), + "no new stream on the kept pad" + ); + recorder.assert_one_pad_held(); + } + + /// A rendition that joins a sibling instance after another started on it shares that instance's + /// timeline, so both map the instance's timestamps to one running time. + #[test] + fn a_rendition_joining_a_sibling_instance_shares_its_timeline() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let session = follow(&element, &origin); + + // A replacement starts a timeline of its own, which its first frame on `first` fixes. + let mut old = Publisher::sibling(&origin, Default::default()); + let first = recorder.written(&mut old, 1); + let mut new = Publisher::sibling(&origin, Default::default()); + assert_eq!(recorder.written(&mut new, 2), first); + // Media past the subscription's one second of delay makes the groups before it stale, so + // the joining rendition starts on a later frame than `first` did. + recorder.written(&mut new, 3); + new.skip(Duration::from_secs(2)); + recorder.written(&mut new, 4); + + let mut again = room.catalog.modify().unwrap().video.renditions["video"].clone(); + again.track = Some("video".into()); + room.catalog + .modify() + .unwrap() + .video + .renditions + .insert("again".into(), again); + let mut second = None; + for _ in 0..200 { + new.write(5); + std::thread::sleep(Duration::from_millis(50)); + second = recorder.seen().into_iter().find_map(|seen| match seen { + Seen::Buffer(pad, 5) if pad != first => Some(pad), + _ => None, + }); + if second.is_some() { + break; + } + } + let second = second.expect("the joining rendition never played"); + stop(session); + + let segment = |pad: &String| { + recorder.seen().into_iter().rev().find_map(|seen| match seen { + Seen::Segment(segmented, start) if segmented == *pad => Some(start), + _ => None, + }) + }; + assert_eq!( + segment(&second), + segment(&first), + "the joining rendition mapped the instance on its own: {:?}", + recorder.seen() + ); + drop(old); + } + + /// The same sibling instance announced again resumes its state, whether or not its end reached + /// the follower first: a track it finished does not replay on its pad. + #[test] + fn a_reannounced_sibling_instance_does_not_replay_its_finish() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let _room = referencing(&origin, None); + let session = follow(&element, &origin); + let mut follow = origin.consume().follow("cam").unwrap(); + + let route = epoch(); + let mut first = Publisher::sibling(&origin, route.clone()); + let pad = recorder.written(&mut first, 1); + first.video.finish().unwrap(); + await_finished(&first); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + first.broadcast.unannounce(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::End(_))); + first.broadcast.announce(route).unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + + let mut second = Publisher::sibling(&origin, epoch()); + assert_eq!( + recorder.written(&mut second, 2), + pad, + "the next instance moved to another pad" + ); + stop(session); + + // Only the next instance starts a stream on the pad. + let seen = recorder.seen(); + assert_eq!(recorder.count(|seen| *seen == Seen::Start(pad.clone())), 1, "{seen:?}"); + recorder.assert_one_pad_held(); + drop(first); + } + + /// A sibling the catalog stopped reading is let go: once the catalog closes, the run ends while + /// both broadcasts stay up. + #[test] + fn a_sibling_no_rendition_reads_does_not_outlive_its_catalog() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let (_shutdown, shutdown) = watch::channel(false); + let (_cancel, mut cancelled) = watch::channel(false); + let run = { + let weak = element.downgrade(); + let source = moq_mux::Source::new(origin.consume(), room.broadcast.consume()); + super::RUNTIME + .spawn(async move { follow_catalog(source, &Pads::default(), weak, &shutdown, &mut cancelled).await }) + }; + + let mut cam = Publisher::sibling(&origin, Default::default()); + recorder.written(&mut cam, 1); + // The delisted rendition plays out its track before its pad retires. + room.catalog.modify().unwrap().video.renditions.clear(); + cam.video.finish().unwrap(); + for _ in 0..200 { + if pads(&element, "video_").is_empty() { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + assert!(pads(&element, "video_").is_empty(), "the removal never landed"); + + room.catalog.finish().unwrap(); + super::RUNTIME + .block_on(async { tokio::time::timeout(Duration::from_secs(10), run).await }) + .expect("the sibling's follower outlived the catalog") + .unwrap() + .unwrap(); + drop((room, cam)); + } + + /// A catalog finished while its broadcast stays up still follows its siblings: the next + /// instance plays on the same pad, until the catalog's broadcast ends. + #[test] + fn a_sibling_is_followed_past_its_catalogs_close() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + room.catalog.finish().unwrap(); + let mut follow = origin.consume().follow("cam").unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + drop(first); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::End(_))); + + let mut second = Publisher::sibling(&origin, Default::default()); + assert_eq!(recorder.written(&mut second, 2), pad, "the return moved to another pad"); + + // The catalog's broadcast ending lets go of the sibling it read. + let mut demand = second.video.demand(); + drop(room); + let unwanted = async { + while demand.wanted_changed().await?.is_some() {} + moq_net::Result::Ok(()) + }; + super::RUNTIME + .block_on(async { tokio::time::timeout(Duration::from_secs(10), unwanted).await }) + .expect("the sibling outlived its catalog's broadcast") + .expect("the sibling track closed"); + stop(session); + recorder.assert_one_pad_held(); + } + + /// A sibling track finishing while its instance stays up idles the pad, and the instance that + /// replaces it starts a new stream there. + #[test] + fn a_finished_sibling_track_returns_on_the_next_instance() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let _room = referencing(&origin, None); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + first.video.finish().unwrap(); + let mut second = Publisher::sibling(&origin, Default::default()); + assert_eq!( + recorder.written(&mut second, 2), + pad, + "the next instance moved to another pad" + ); + stop(session); + + assert!(!recorder.seen().contains(&Seen::Eos(pad)), "the finish ended the pad"); + recorder.assert_one_pad_held(); + drop(first); + } + + /// Block until `publisher`'s video track reads to its end, as a reader reading it does. + fn await_finished(publisher: &Publisher) { + let broadcast = publisher.broadcast.consume(); + super::RUNTIME.block_on(async { + let finished = async { + let subscriber = broadcast + .track("video") + .unwrap() + .subscribe(moq_net::track::Subscription::default()) + .await + .unwrap(); + let mut track = moq_mux::container::Consumer::new( + subscriber, + moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), + ); + while track.read().await.unwrap().is_some() {} + }; + tokio::time::timeout(Duration::from_secs(10), finished) + .await + .expect("the track never finished"); + }); + } + + /// A sibling track that finished stays finished for its instance only: a disable and an enable + /// replays nothing on that instance, and the next instance still plays. + #[test] + fn a_reenabled_sibling_rendition_plays_the_next_instance_after_a_finish() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let _audio = ["audio", "again"].map(|name| { + room.broadcast + .create_track(name, hang::container::track_info(hang::catalog::PRIORITY.audio)) + .unwrap() + }); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + first.video.finish().unwrap(); + await_finished(&first); + + // The catalog consumer only yields the newest snapshot, so each update lists another audio + // rendition: its pad appearing proves the run acted on it. + { + let mut guard = room.catalog.modify().unwrap(); + guard.video.renditions.get_mut("video").unwrap().enabled = false; + guard.audio.renditions = BTreeMap::from([("audio".to_string(), audio_rendition())]); + } + await_pad(&element, "audio_"); + { + let mut guard = room.catalog.modify().unwrap(); + guard.video.renditions.get_mut("video").unwrap().enabled = true; + let mut again = audio_rendition(); + again.track = Some("again".into()); + guard.audio.renditions.insert("again".to_string(), again); + } + for _ in 0..200 { + if pads(&element, "audio_").len() == 2 { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + assert_eq!(pads(&element, "audio_").len(), 2, "the re-enable never landed"); + let replayed = recorder.count(|seen| *seen == Seen::Buffer(pad.clone(), 1)); + + let mut second = Publisher::sibling(&origin, Default::default()); + assert_eq!( + recorder.written(&mut second, 2), + pad, + "the next instance moved to another pad" + ); + stop(session); + + // Only the next instance starts a stream: the finished one stays finished on its pad. + let seen = recorder.seen(); + assert_eq!(recorder.count(|seen| *seen == Seen::Start(pad.clone())), 1, "{seen:?}"); + assert_eq!( + recorder.count(|seen| *seen == Seen::Buffer(pad.clone(), 1)), + replayed, + "{seen:?}" + ); + recorder.assert_one_pad_held(); + drop(first); + } + + /// A sibling rendition delisted while it plays and listed again before its track finishes stays + /// with the reader that drained it: the finished track does not replay on its pad, and the + /// next instance plays there. + #[test] + fn a_sibling_rendition_relisted_while_draining_does_not_replay_its_finish() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let _audio = ["audio", "again"].map(|name| { + room.broadcast + .create_track(name, hang::container::track_info(hang::catalog::PRIORITY.audio)) + .unwrap() + }); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + + // The catalog consumer only yields the newest snapshot, so each update lists another audio + // rendition: its pad appearing proves the run acted on it. + let listed = room.catalog.modify().unwrap().video.renditions.clone(); + { + let mut guard = room.catalog.modify().unwrap(); + guard.video.renditions.clear(); + guard.audio.renditions = BTreeMap::from([("audio".to_string(), audio_rendition())]); + } + await_pad(&element, "audio_"); + { + let mut guard = room.catalog.modify().unwrap(); + guard.video.renditions = listed; + let mut again = audio_rendition(); + again.track = Some("again".into()); + guard.audio.renditions.insert("again".to_string(), again); + } + for _ in 0..200 { + if pads(&element, "audio_").len() == 2 { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + assert_eq!(pads(&element, "audio_").len(), 2, "the relist never landed"); + + first.video.finish().unwrap(); + await_finished(&first); + let mut second = Publisher::sibling(&origin, Default::default()); + assert_eq!( + recorder.written(&mut second, 2), + pad, + "the next instance moved to another pad" + ); + stop(session); + + let seen = recorder.seen(); + assert_eq!(recorder.count(|seen| *seen == Seen::Start(pad.clone())), 1, "{seen:?}"); + recorder.assert_one_pad_held(); + drop(first); + } + + /// A publisher handoff after a sibling track finished, the catalog naming the new instance's + /// epoch, plays that instance on the same pad. + #[test] + fn an_epoch_handoff_after_a_sibling_finish_plays_the_new_instance() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let (old, new) = (moq_net::Epoch::mint(), moq_net::Epoch::mint()); + let mut room = referencing(&origin, Some(old.clone())); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, moq_net::origin::Route::default().with_epoch(old)); + let pad = recorder.written(&mut first, 1); + first.video.finish().unwrap(); + await_finished(&first); + + let mut second = Publisher::sibling(&origin, moq_net::origin::Route::default().with_epoch(new.clone())); + room.catalog + .modify() + .unwrap() + .video + .renditions + .get_mut("video") + .unwrap() + .epoch = Some(new); + assert_eq!( + recorder.written(&mut second, 2), + pad, + "the new instance moved to another pad" + ); + stop(session); + + recorder.assert_one_pad_held(); + drop(first); + } + + /// An instance a rendition pins that wins its path back from another resumes its state: the + /// track it finished replays on no pad, pinned or not. + #[test] + fn a_pinned_instance_winning_its_path_back_does_not_replay_its_finish() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let (pinned, other, next) = (moq_net::Epoch::mint(), moq_net::Epoch::mint(), moq_net::Epoch::mint()); + let mut room = referencing(&origin, Some(pinned.clone())); + { + let mut guard = room.catalog.modify().unwrap(); + let mut again = guard.video.renditions["video"].clone(); + again.track = Some("video".into()); + again.epoch = None; + guard.video.renditions.insert("again".into(), again); + } + let session = follow(&element, &origin); + + // Each rendition plays the first instance on a pad of its own. + let reached = |tag: u8| -> HashSet { + recorder + .seen() + .into_iter() + .filter_map(|seen| match seen { + Seen::Buffer(pad, seen) if seen == tag => Some(pad), + _ => None, + }) + .collect() + }; + let mut first = Publisher::sibling(&origin, moq_net::origin::Route::default().with_epoch(pinned)); + for _ in 0..200 { + if reached(1).len() == 2 { + break; + } + first.write(1); + std::thread::sleep(Duration::from_millis(50)); + } + assert_eq!(reached(1).len(), 2, "{:?}", recorder.seen()); + first.video.finish().unwrap(); + await_finished(&first); + + // Another instance wins the path, which only the unpinned rendition plays, then leaves it + // to the pinned one. + let mut second = Publisher::sibling(&origin, moq_net::origin::Route::default().with_epoch(other)); + let unpinned = recorder.written(&mut second, 2); + let pad = reached(1).into_iter().find(|pad| *pad != unpinned).unwrap(); + let mut follow = origin.consume().follow("cam").unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + drop(second); + await_event(&mut follow, |event| { + matches!( + event, + moq_net::announce::Event::Restart(_) | moq_net::announce::Event::Start(_) + ) + }); + + let mut third = Publisher::sibling(&origin, moq_net::origin::Route::default().with_epoch(next.clone())); + room.catalog + .modify() + .unwrap() + .video + .renditions + .get_mut("video") + .unwrap() + .epoch = Some(next); + for _ in 0..200 { + if reached(3).contains(&pad) { + break; + } + third.write(3); + std::thread::sleep(Duration::from_millis(50)); + } + assert!(reached(3).contains(&pad), "the next instance moved to another pad"); + stop(session); + + // Neither pad replays the first instance's track on its return. + let seen = recorder.seen(); + for pad in [&pad, &unpinned] { + assert_eq!( + recorder.count(|seen| *seen == Seen::Buffer(pad.clone(), 1)), + 1, + "{seen:?}" + ); + } + drop(first); + } + + /// A rendition delisted while its sibling is offline ends its idle pad, and the sibling's next + /// instance plays nothing on it. + #[test] + fn a_delisted_sibling_rendition_follows_no_later_instance() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + let mut follow = origin.consume().follow("cam").unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + drop(first); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::End(_))); + + room.catalog.modify().unwrap().video.renditions.clear(); + for _ in 0..200 { + if recorder.seen().contains(&Seen::Eos(pad.clone())) { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + assert!( + recorder.seen().contains(&Seen::Eos(pad.clone())), + "the idle pad never ended" + ); + + let mut second = Publisher::sibling(&origin, Default::default()); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + for _ in 0..10 { + second.write(2); + std::thread::sleep(Duration::from_millis(50)); + } + stop(session); + + let seen = recorder.seen(); + assert!(!seen.iter().any(|seen| matches!(seen, Seen::Buffer(_, 2))), "{seen:?}"); + } + + /// A rendition delisted while its sibling is offline and listed again follows the sibling's next + /// instance, even though the reader that left took its pad with it. + #[test] + fn a_relisted_sibling_rendition_follows_the_next_instance() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let mut room = referencing(&origin, None); + let session = follow(&element, &origin); + + let mut first = Publisher::sibling(&origin, Default::default()); + let pad = recorder.written(&mut first, 1); + let mut follow = origin.consume().follow("cam").unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + drop(first); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::End(_))); + + let listed = room.catalog.modify().unwrap().video.renditions.clone(); + room.catalog.modify().unwrap().video.renditions.clear(); + for _ in 0..200 { + if recorder.seen().contains(&Seen::Eos(pad.clone())) { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + assert!( + recorder.seen().contains(&Seen::Eos(pad.clone())), + "the idle pad never ended" + ); + room.catalog.modify().unwrap().video.renditions = listed; + + let mut second = Publisher::sibling(&origin, Default::default()); + assert_ne!(recorder.written(&mut second, 2), pad, "the ended pad came back"); + stop(session); + } + + /// A sibling replaced while the old one stays up restarts a reference that names no epoch, on + /// the same pad, and nothing more from the old instance gets through. + fn sibling_restart(route: impl Fn() -> moq_net::origin::Route) { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let _room = referencing(&origin, None); + let mut old = Publisher::sibling(&origin, route()); + let session = follow(&element, &origin); + let pad = recorder.written(&mut old, 1); + + let mut new = Publisher::sibling(&origin, route()); + assert_eq!(recorder.written(&mut new, 2), pad, "the restart moved to another pad"); + old.write(1); + recorder.written(&mut new, 3); + stop(session); + + let seen = recorder.seen(); + let switched = seen + .iter() + .position(|seen| *seen == Seen::Buffer(pad.clone(), 2)) + .unwrap(); + assert!( + !seen[switched..].contains(&Seen::Buffer(pad.clone(), 1)), + "the replaced sibling kept playing: {seen:?}" + ); + recorder.assert_one_pad_held(); + } + + #[test] + fn a_sibling_restart_switches_on_the_same_pad() { + sibling_restart(epoch); + } + + #[test] + fn an_epochless_sibling_restart_switches_on_the_same_pad() { + sibling_restart(moq_net::origin::Route::default); + } + + /// A reference naming its sibling's epoch never reads the instance replacing it: the pad + /// idles, and plays again once the named instance serves the path again. + #[test] + fn a_sibling_restart_idles_a_reference_naming_the_old_epoch() { + let _pad_ids = pad_ids(); + let element = element(); + let recorder = Recorder::attach(&element); + let origin = origin(); + let named = moq_net::Epoch::mint(); + let _room = referencing(&origin, Some(named.clone())); + let mut old = Publisher::sibling(&origin, moq_net::origin::Route::default().with_epoch(named)); + let session = follow(&element, &origin); + let pad = recorder.written(&mut old, 1); + + let mut follow = origin.consume().follow("cam").unwrap(); + await_event(&mut follow, |event| matches!(event, moq_net::announce::Event::Start(_))); + let mut new = Publisher::sibling(&origin, epoch()); + await_event(&mut follow, |event| { + matches!(event, moq_net::announce::Event::Restart(_)) + }); + new.write(2); + drop(new); + await_event(&mut follow, |event| { + matches!(event, moq_net::announce::Event::Restart(_)) + }); + + assert_eq!( + recorder.written(&mut old, 3), + pad, + "the named instance moved to another pad" + ); + stop(session); + assert!( + !recorder.seen().contains(&Seen::Buffer(pad, 2)), + "another epoch played: {:?}", + recorder.seen() + ); + recorder.assert_one_pad_held(); + } + /// Re-pricing the same instance is an update, which the run rides out without a new stream. #[test] fn an_update_keeps_the_run() { @@ -3171,7 +4198,7 @@ mod session_tests { recorder.assert_one_pad_held(); } - /// Disabling a rendition stops its pump but holds its pad idle, without an EOS, so a pipeline + /// Disabling a rendition stops its reader but holds its pad idle, without an EOS, so a pipeline /// linked by name resumes on it once the rendition is enabled again. #[test] fn a_reenabled_rendition_resumes_on_the_same_pad() { @@ -3216,7 +4243,7 @@ mod session_tests { let seen = recorder.seen(); assert!( seen.contains(&Seen::Start(pad)), - "the disabled pump kept running: {seen:?}" + "the disabled reader kept running: {seen:?}" ); recorder.assert_one_pad_held(); }