From daed2a2ce851f9e17ee1e9d0f53bd5786a4c8b68 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 09:17:26 -0700 Subject: [PATCH 1/7] chore(quest): claim quest/m1/sink-restart-follow.md Quest-Claim: 61749f7fadb3eec5 From 5b38f27ab8b7087f8accd29ec1ea086a5b66b2ef Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 09:43:02 -0700 Subject: [PATCH 2/7] fix(hls)!: give each publisher instance its own playlist URLs The HLS server follows a broadcast's instances through moq_mux::follow::Instances. The master playlist and DASH manifest list the instance serving the path now, and every URL they list sits under @{instance} (the epoch, or a minted id). A replaced instance keeps serving its URLs until its broadcast closes. Co-Authored-By: Claude Opus 5.5 --- doc/bin/hls.md | 2 +- rs/moq-hls/MIGRATE.md | 11 ++ rs/moq-hls/src/export/master.rs | 31 +-- rs/moq-hls/src/export/mod.rs | 36 +++- rs/moq-hls/src/export/mpd.rs | 48 +++-- rs/moq-hls/src/server/mod.rs | 299 ++++++++++++++++++++--------- rs/moq-hls/src/server/routes.rs | 321 ++++++++++++++++++++++---------- 7 files changed, 525 insertions(+), 223 deletions(-) diff --git a/doc/bin/hls.md b/doc/bin/hls.md index fc2a6d729b3..88c3c33ec3f 100644 --- a/doc/bin/hls.md +++ b/doc/bin/hls.md @@ -47,7 +47,7 @@ What that means for you: - **Publish timelines.** A broadcast whose catalog advertises no timelines is skipped, and a rendition is listed only once its own timeline is advertised. Without a duration hint, a publisher declares that timeline after the rendition's first segment. - **Align GOPs across video renditions.** Segment boundaries come from one reference rendition. Other video renditions snap to their nearest keyframe, and a segment with none nearby becomes an `EXT-X-GAP`. - **Disabled renditions are hidden.** A rendition the catalog marks `enabled: false` leaves the multivariant playlist and DASH manifest until it re-enables. Its media playlist keeps serving, so a player already on it is not ended with `EXT-X-ENDLIST`. If it supplied segment boundaries, an enabled sibling takes over at its first record numbered past the last advertised one, and DASH starts a new period. The media sequence is the record number, so every edge agrees: a sibling numbered ahead restarts the playlist with a discontinuity, and one numbered behind (such as a re-enabled rung) holds every playlist until its numbering catches up. `EXT-X-DISCONTINUITY-SEQUENCE` and DASH periods are counted per edge, so edges agree on them only when no discontinuity or handoff happened while each was running. -- **Segment URLs are stable.** Every edge and every reload derive the same URLs, so a CDN in front of export caches them. +- **URLs name the publisher instance.** Every URL the multivariant playlist and DASH manifest list sits under `@`, the publisher's epoch, so every edge and every reload derive the same URLs and a CDN in front of export caches them. A restarted publisher gets URLs of its own, whose numbering starts over. The multivariant playlist and manifest switch to it at once, while a player already on the old instance keeps playing until that publisher leaves; the player then has to reload the multivariant playlist. An instance without an epoch (one that reached export over moq-lite-06 or older) gets an id each export mints, so only that export serves its URLs. - **Only a clean finish ends a playlist.** `EXT-X-ENDLIST` and a `static` DASH manifest mean the timeline finished. If the segment-boundary rendition's timeline fails instead, such as a publisher dropping its session, every media playlist and the DASH manifest answer HTTP 500 with `Cache-Control: no-store`, so a truncated recording never looks complete. A rendition whose own timeline fails answers the error alone and leaves the multivariant playlist and DASH manifest, so players switch to a sibling. A [`moq-archive`](https://docs.rs/moq-archive) recording replayed through its diff --git a/rs/moq-hls/MIGRATE.md b/rs/moq-hls/MIGRATE.md index 6b9ce1f99cd..141329d1ecd 100644 --- a/rs/moq-hls/MIGRATE.md +++ b/rs/moq-hls/MIGRATE.md @@ -4,6 +4,17 @@ How to update your code for each breaking change, newest first. ## Unreleased +### Rendition URLs name the publisher instance + +`Server` follows each instance that takes over a broadcast, and every URL the +master playlist and DASH manifest list now sits under that instance: +`/{broadcast}/@{instance}/{kind}/{rendition}/...` instead of +`/{broadcast}/{kind}/{rendition}/...`. `Route::parse` refuses the old layout, +and `Resource::{Media, Init, Segment, SegmentAt}` carry the `instance` they +name. `Broadcaster::master_playlist` and `Broadcaster::manifest` render the new +layout. An embedder that answers through `Server::respond` needs no change; one +that routes rendition URLs itself must pass the `@{instance}` segment through. + ### Denied broadcast references report authorization failure A rendition whose referenced broadcast is outside the source origin's scope now diff --git a/rs/moq-hls/src/export/master.rs b/rs/moq-hls/src/export/master.rs index 79aca598fae..ff72e5d6982 100644 --- a/rs/moq-hls/src/export/master.rs +++ b/rs/moq-hls/src/export/master.rs @@ -3,7 +3,8 @@ //! Each variant carries its own URI, so a caller with a different layout (a VOD //! recorder writing `/media.m3u8`, say) reuses the grouping and attribute //! rules. [`Broadcaster::master_playlist`](super::Broadcaster::master_playlist) -//! renders the live layout, relative to `//master.m3u8`. +//! renders the live layout, relative to `//master.m3u8`: each rendition under the +//! publisher instance it lists (`@///media.m3u8`). use std::collections::BTreeMap; use std::fmt::Write; @@ -21,11 +22,11 @@ const AUDIO_GROUP: &str = "aud"; /// in both layouts. pub(crate) const PATH_SEGMENT: &AsciiSet = &NON_ALPHANUMERIC.remove(b'-').remove(b'.').remove(b'_').remove(b'~'); -/// The live layout's media-playlist URI for a rendition, relative to the master, with an -/// optional query (without the leading `?`) appended. -pub(crate) fn rendition_uri(kind: Kind, name: &str, query: Option<&str>) -> String { +/// The live layout's media-playlist URI for a rendition of the publisher `instance`, relative to +/// the master, with an optional query (without the leading `?`) appended. +pub(crate) fn rendition_uri(instance: &str, kind: Kind, name: &str, query: Option<&str>) -> String { let mut uri = format!( - "{}/{}/media.m3u8", + "@{instance}/{}/{}/media.m3u8", kind.as_str(), utf8_percent_encode(name, PATH_SEGMENT) ); @@ -197,7 +198,7 @@ mod tests { fn video(name: &str, width: Option, height: Option) -> VideoVariant { VideoVariant { - uri: rendition_uri(Kind::Video, name, None), + uri: rendition_uri("run", Kind::Video, name, None), bandwidth: 2_500_000, width, height, @@ -208,7 +209,7 @@ mod tests { fn audio(name: &str, bandwidth: u64, codec: &str) -> AudioVariant { AudioVariant { name: name.into(), - uri: rendition_uri(Kind::Audio, name, None), + uri: rendition_uri("run", Kind::Audio, name, None), bandwidth, codec: codec.into(), } @@ -222,12 +223,12 @@ mod tests { ); assert!(out.starts_with("#EXTM3U\n#EXT-X-VERSION:9\n")); assert!(out.contains( - "#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID=\"aud\",NAME=\"audio\",DEFAULT=YES,AUTOSELECT=YES,URI=\"audio/audio/media.m3u8\"\n" + "#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID=\"aud\",NAME=\"audio\",DEFAULT=YES,AUTOSELECT=YES,URI=\"@run/audio/audio/media.m3u8\"\n" )); assert!(out.contains( "#EXT-X-STREAM-INF:BANDWIDTH=2628000,RESOLUTION=1280x720,CODECS=\"avc1.42c01f,mp4a.40.2\",AUDIO=\"aud\"\n" )); - assert!(out.contains("\nvideo/video/media.m3u8\n")); + assert!(out.contains("\n@run/video/video/media.m3u8\n")); } #[test] @@ -257,25 +258,25 @@ mod tests { assert!(out.contains("GROUP-ID=\"aud-1\",NAME=\"opus\",DEFAULT=YES")); assert!(out.contains("BANDWIDTH=2628000,RESOLUTION=1280x720,CODECS=\"avc1.42c01f,mp4a.40.2\",AUDIO=\"aud-0\"")); assert!(out.contains("BANDWIDTH=2660000,RESOLUTION=1280x720,CODECS=\"avc1.42c01f,opus\",AUDIO=\"aud-1\"")); - assert_eq!(out.matches("\nvideo/video/media.m3u8\n").count(), 2); + assert_eq!(out.matches("\n@run/video/video/media.m3u8\n").count(), 2); } #[test] fn audio_only_is_playable() { let out = render(&[], &[audio("audio", 128_000, "opus")]); assert!(out.contains("#EXT-X-STREAM-INF:BANDWIDTH=128000,CODECS=\"opus\"\n")); - assert!(out.contains("\naudio/audio/media.m3u8\n")); + assert!(out.contains("\n@run/audio/audio/media.m3u8\n")); } #[test] fn rendition_names_are_percent_encoded_in_uris() { assert_eq!( - rendition_uri(Kind::Video, "cam#1/main?alt", Some("jwt=abc.def")), - "video/cam%231%2Fmain%3Falt/media.m3u8?jwt=abc.def" + rendition_uri("run", Kind::Video, "cam#1/main?alt", Some("jwt=abc.def")), + "@run/video/cam%231%2Fmain%3Falt/media.m3u8?jwt=abc.def" ); assert_eq!( - rendition_uri(Kind::Audio, "audio #1", None), - "audio/audio%20%231/media.m3u8" + rendition_uri("run", Kind::Audio, "audio #1", None), + "@run/audio/audio%20%231/media.m3u8" ); } diff --git a/rs/moq-hls/src/export/mod.rs b/rs/moq-hls/src/export/mod.rs index 9648e4cae04..c4f8e830f53 100644 --- a/rs/moq-hls/src/export/mod.rs +++ b/rs/moq-hls/src/export/mod.rs @@ -137,6 +137,9 @@ pub struct Broadcaster { ) )] broadcast: moq_net::broadcast::Consumer, + /// The publisher instance served, which every URL the master playlist and manifest list + /// starts with: its epoch, or an id minted for an epochless instance. + instance: Arc, /// The current rendition set, reconciled from the catalog by the watcher task. renditions: renditions::Producer, /// Aborts the catalog watcher when the broadcaster is dropped; `Drop` then retires the @@ -152,10 +155,13 @@ impl Broadcaster { /// Start tracking the renditions of `source`'s catalog broadcast. pub fn new(source: moq_mux::Source, config: Config) -> Arc { let broadcast = source.broadcast(); + // An epoch names the instance on every edge, so each serves it under the same URLs. + let instance = broadcast.info().epoch.clone().unwrap_or_else(moq_net::Epoch::mint); let renditions = renditions::Producer::new(config.window); let timeline_watchers = Arc::new(Mutex::new(std::collections::BTreeMap::new())); let broadcaster = Arc::new(Self { broadcast, + instance: Arc::from(instance.as_str()), renditions: renditions.clone(), watcher: Mutex::new(None), timeline_watchers: timeline_watchers.clone(), @@ -167,6 +173,15 @@ impl Broadcaster { broadcaster } + /// The publisher instance every listed URL is under (`@{instance}/{kind}/{rendition}/...`). + #[cfg_attr( + not(feature = "server"), + allow(dead_code, reason = "only the `server` feature routes by instance") + )] + pub(crate) fn instance(&self) -> &Arc { + &self.instance + } + /// Whether the source broadcast has closed (ended or dropped). pub fn is_closed(&self) -> bool { self.broadcast.is_closed() @@ -263,7 +278,7 @@ impl Broadcaster { } match rendition.kind { Kind::Video => video.push(master::VideoVariant { - uri: master::rendition_uri(Kind::Video, &rendition.name, query), + uri: master::rendition_uri(&self.instance, Kind::Video, &rendition.name, query), bandwidth: rendition.bandwidth(), width: rendition.width, height: rendition.height, @@ -271,7 +286,7 @@ impl Broadcaster { }), Kind::Audio => audio.push(master::AudioVariant { name: rendition.name.clone(), - uri: master::rendition_uri(Kind::Audio, &rendition.name, query), + uri: master::rendition_uri(&self.instance, Kind::Audio, &rendition.name, query), bandwidth: rendition.bandwidth(), codec: rendition.codec.clone(), }), @@ -343,6 +358,7 @@ impl Broadcaster { Ok(Some(mpd::render_manifest( &mpd::Manifest { + instance: self.instance.clone(), availability_start, publish: moq_clock::wall(), window: self.renditions.window(), @@ -2161,7 +2177,8 @@ mod tests { "the init needs no generation" ); let manifest = broadcaster.manifest(None).unwrap().expect("manifest"); - assert!(manifest.contains("media=\"video/video0/seg/run-1.656e3d1b.t$Time$.m4s\"")); + let run = broadcaster.instance(); + assert!(manifest.contains(&format!("media=\"@{run}/video/video0/seg/run-1.656e3d1b.t$Time$.m4s\""))); broadcaster.set_generation(Some("run-2")).unwrap(); assert!( @@ -2430,7 +2447,8 @@ mod tests { ); assert!(manifest.contains("id=\"video/video0\"")); assert!(manifest.contains("id=\"audio/audio0\"")); - assert!(manifest.contains("media=\"video/video0/seg/656e3d1b.t$Time$.m4s\"")); + let run = broadcaster.instance(); + assert!(manifest.contains(&format!("media=\"@{run}/video/video0/seg/656e3d1b.t$Time$.m4s\""))); // Both renditions list the same aligned segment spans (the live edge is not listed). assert_eq!(manifest.matches("").count(), 2); assert_eq!(manifest.matches("").count(), 2); @@ -2441,10 +2459,16 @@ mod tests { .manifest(Some("jwt=abc.def")) .unwrap() .expect("manifest renders"); - let hash = between(&signed, "initialization=\"video/video0/init.", ".mp4?jwt=abc.def\""); + let hash = between( + &signed, + &format!("initialization=\"@{run}/video/video0/init."), + ".mp4?jwt=abc.def\"", + ); let init = video_rendition.init().await.unwrap().expect("init segment"); assert_eq!(video_rendition.init_versioned(hash).await.unwrap(), Some(init)); - assert!(signed.contains("media=\"video/video0/seg/656e3d1b.t$Time$.m4s?jwt=abc.def\"")); + assert!(signed.contains(&format!( + "media=\"@{run}/video/video0/seg/656e3d1b.t$Time$.m4s?jwt=abc.def\"" + ))); // $Time$ resolves to the same bytes the aligned number does; unknown times miss. let by_time = video_rendition diff --git a/rs/moq-hls/src/export/mpd.rs b/rs/moq-hls/src/export/mpd.rs index d31e17f788c..eb006069d1a 100644 --- a/rs/moq-hls/src/export/mpd.rs +++ b/rs/moq-hls/src/export/mpd.rs @@ -7,7 +7,8 @@ //! that skips a span (a gap) or a client that joins mid-window never mis-addresses a segment //! the way positional numbering would. //! -//! URIs are relative to the manifest (`//manifest.mpd`). The rendition name is +//! URIs are relative to the manifest (`//manifest.mpd`), under the publisher instance +//! it lists (`@///...`). The rendition name is //! one percent-encoded path segment, using the HLS master's set, and the finished URL is //! then XML-escaped, so a name containing `/` still resolves under the broadcast. The //! `Representation` id keeps the raw name. The resources are shared byte-for-byte with the @@ -66,6 +67,8 @@ pub(crate) struct Representation { /// Everything a manifest render needs; built by `Broadcaster::manifest`. pub(crate) struct Manifest { + /// The publisher instance every representation's URLs are under. + pub instance: Arc, /// The wall-clock time of timeline `pts` 0 (`MPD@availabilityStartTime`). Required while /// live (`type="dynamic"`); ignored once [`finished`](Self::finished). pub availability_start: Option, @@ -151,7 +154,7 @@ fn max_segment_duration<'a>(representations: impl Iterator) { +fn render_representation(out: &mut String, instance: &str, rep: &Representation, query: Option<&str>) { let kind = rep.kind.as_str(); // The id is the raw name. The URL encodes it first, as one path segment: a slash left // raw would name a different broadcast, and XML-escaping before encoding would turn `&` @@ -164,9 +167,9 @@ fn render_representation(out: &mut String, rep: &Representation, query: Option<& .map(|generation| format!("{generation}.")) .unwrap_or_default(); let suffix = query.map(|query| format!("?{query}")).unwrap_or_default(); - let initialization = escape(&format!("{kind}/{segment}/init.{}.mp4{suffix}", rep.init)); + let initialization = escape(&format!("@{instance}/{kind}/{segment}/init.{}.mp4{suffix}", rep.init)); let media = escape(&format!( - "{kind}/{segment}/seg/{generation}{}.t$Time$.m4s{suffix}", + "@{instance}/{kind}/{segment}/seg/{generation}{}.t$Time$.m4s{suffix}", rep.tag )); @@ -218,6 +221,7 @@ fn render_representation(out: &mut String, rep: &Representation, query: Option<& fn render_adaptation_sets<'a>( out: &mut String, + instance: &str, kind: Kind, representations: impl Iterator, query: Option<&str>, @@ -246,7 +250,7 @@ fn render_adaptation_sets<'a>( " " ); for rep in representations { - render_representation(out, rep, query); + render_representation(out, instance, rep, query); } let _ = writeln!(out, " "); } @@ -321,12 +325,14 @@ pub(crate) fn render_manifest(manifest: &Manifest, query: Option<&str>) -> Strin ); render_adaptation_sets( &mut out, + &manifest.instance, Kind::Video, manifest.video.iter().filter(|rep| rep.period == period), query, ); render_adaptation_sets( &mut out, + &manifest.instance, Kind::Audio, manifest.audio.iter().filter(|rep| rep.period == period), query, @@ -369,6 +375,7 @@ mod tests { next.period = moq_net::Timestamp::from_secs(6).unwrap(); next.tag = "next".into(); let manifest = Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH), publish: Wall::UNIX_EPOCH, window: None, @@ -394,6 +401,7 @@ mod tests { next.timescale = 90_000; next.period = moq_net::Timestamp::new(1, scale).unwrap(); let manifest = Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH), publish: Wall::UNIX_EPOCH, window: None, @@ -429,6 +437,7 @@ mod tests { #[test] fn renders_dynamic_manifest() { let manifest = Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH + Duration::from_millis(1_751_846_400_123)), publish: Wall::UNIX_EPOCH + Duration::from_millis(1_751_846_410_000), window: Some(Duration::from_secs(16)), @@ -459,7 +468,7 @@ mod tests { "" )); assert!(out.contains( - "" + "" )); assert!(out.contains("")); assert!(out.contains("")); @@ -469,6 +478,7 @@ mod tests { #[test] fn renders_static_manifest_when_finished() { let manifest = Manifest { + instance: "run".into(), availability_start: None, publish: Wall::UNIX_EPOCH, window: Some(Duration::from_secs(16)), @@ -493,6 +503,7 @@ mod tests { #[test] fn query_rides_child_urls_escaped() { let manifest = Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH), publish: Wall::UNIX_EPOCH, window: Some(Duration::from_secs(16)), @@ -502,8 +513,8 @@ mod tests { }; let out = render_manifest(&manifest, Some("jwt=abc.def&x=1")); - assert!(out.contains("initialization=\"video/video0/init.0123abcd.mp4?jwt=abc.def&x=1\"")); - assert!(out.contains("media=\"video/video0/seg/ab12cd34.t$Time$.m4s?jwt=abc.def&x=1\"")); + assert!(out.contains("initialization=\"@run/video/video0/init.0123abcd.mp4?jwt=abc.def&x=1\"")); + assert!(out.contains("media=\"@run/video/video0/seg/ab12cd34.t$Time$.m4s?jwt=abc.def&x=1\"")); } #[test] @@ -511,6 +522,7 @@ mod tests { let mut rep = video(vec![(0, 2_000)], false); rep.generation = Some("run-7".into()); let manifest = Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH), publish: Wall::UNIX_EPOCH, window: Some(Duration::from_secs(16)), @@ -520,8 +532,8 @@ mod tests { }; let out = render_manifest(&manifest, None); - assert!(out.contains("initialization=\"video/video0/init.0123abcd.mp4\"")); - assert!(out.contains("media=\"video/video0/seg/run-7.ab12cd34.t$Time$.m4s\"")); + assert!(out.contains("initialization=\"@run/video/video0/init.0123abcd.mp4\"")); + assert!(out.contains("media=\"@run/video/video0/seg/run-7.ab12cd34.t$Time$.m4s\"")); } #[test] @@ -534,6 +546,7 @@ mod tests { #[test] fn empty_and_gap_only_representations_are_omitted() { let manifest = Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH), publish: Wall::UNIX_EPOCH, window: Some(Duration::from_secs(16)), @@ -552,6 +565,7 @@ mod tests { rep.name = name.into(); render_manifest( &Manifest { + instance: "run".into(), availability_start: Some(Wall::UNIX_EPOCH), publish: Wall::UNIX_EPOCH, window: Some(Duration::from_secs(16)), @@ -570,14 +584,14 @@ mod tests { let out = render_named("video/1080p", Some("jwt=abc.def&x=1")); assert!(out.contains("id=\"video/video/1080p\""), "{out}"); assert!( - out.contains("initialization=\"video/video%2F1080p/init.0123abcd.mp4?jwt=abc.def&x=1\""), + out.contains("initialization=\"@run/video/video%2F1080p/init.0123abcd.mp4?jwt=abc.def&x=1\""), "{out}" ); assert!( - out.contains("media=\"video/video%2F1080p/seg/ab12cd34.t$Time$.m4s?jwt=abc.def&x=1\""), + out.contains("media=\"@run/video/video%2F1080p/seg/ab12cd34.t$Time$.m4s?jwt=abc.def&x=1\""), "{out}" ); - assert!(!out.contains("initialization=\"video/video/1080p/"), "{out}"); + assert!(!out.contains("initialization=\"@run/video/video/1080p/"), "{out}"); let cases = [ ("cam?1", "cam%3F1"), @@ -592,11 +606,11 @@ mod tests { "id stays raw for {name}: {out}" ); assert!( - out.contains(&format!("initialization=\"video/{segment}/init.0123abcd.mp4\"")), + out.contains(&format!("initialization=\"@run/video/{segment}/init.0123abcd.mp4\"")), "init for {name}: {out}" ); assert!( - out.contains(&format!("media=\"video/{segment}/seg/ab12cd34.t$Time$.m4s\"")), + out.contains(&format!("media=\"@run/video/{segment}/seg/ab12cd34.t$Time$.m4s\"")), "media for {name}: {out}" ); } @@ -607,11 +621,11 @@ mod tests { let out = render_named("a&b\"c", Some("x=1&y=2")); assert!(out.contains("id=\"video/a&b"c\""), "{out}"); assert!( - out.contains("initialization=\"video/a%26b%22c/init.0123abcd.mp4?x=1&y=2\""), + out.contains("initialization=\"@run/video/a%26b%22c/init.0123abcd.mp4?x=1&y=2\""), "{out}" ); assert!( - out.contains("media=\"video/a%26b%22c/seg/ab12cd34.t$Time$.m4s?x=1&y=2\""), + out.contains("media=\"@run/video/a%26b%22c/seg/ab12cd34.t$Time$.m4s?x=1&y=2\""), "{out}" ); assert!(!out.contains("&amp;"), "{out}"); diff --git a/rs/moq-hls/src/server/mod.rs b/rs/moq-hls/src/server/mod.rs index 5aed7aa6dc1..03beb0763f3 100644 --- a/rs/moq-hls/src/server/mod.rs +++ b/rs/moq-hls/src/server/mod.rs @@ -5,12 +5,19 @@ //! ```text //! GET /{broadcast}/master.m3u8 //! GET /{broadcast}/manifest.mpd -//! GET /{broadcast}/{kind}/{rendition}/media.m3u8 -//! GET /{broadcast}/{kind}/{rendition}/init.{hash}.mp4 -//! GET /{broadcast}/{kind}/{rendition}/seg/{reference}.{segment}.m4s -//! GET /{broadcast}/{kind}/{rendition}/seg/{reference}.t{pts}.m4s +//! GET /{broadcast}/@{instance}/{kind}/{rendition}/media.m3u8 +//! GET /{broadcast}/@{instance}/{kind}/{rendition}/init.{hash}.mp4 +//! GET /{broadcast}/@{instance}/{kind}/{rendition}/seg/{reference}.{segment}.m4s +//! GET /{broadcast}/@{instance}/{kind}/{rendition}/seg/{reference}.t{pts}.m4s //! ``` //! +//! The master playlist and manifest list the publisher instance serving the broadcast now, +//! and every URL they list is under that `@{instance}`: its epoch, or an id the server mints +//! for an epochless instance. A publisher that restarts gets its own URLs, so a playlist URL +//! never resets its media sequence. A replaced instance keeps serving its URLs while its +//! broadcast stays open, so a player already on it keeps playing until its publisher goes, +//! then reloads the master to reach the replacement. +//! //! `{kind}` is `video` or `audio`, so a video and an audio rendition that share a //! name address distinct resources. The two `seg/` forms name the same bytes: HLS //! playlists address a segment by its aligned number, the DASH manifest by its @@ -23,9 +30,9 @@ //! Every request is served. To gate access, wrap [`Server::router`] in your own //! [`axum`] middleware. It runs before routing, so a rejected request never reaches //! the origin, but it also sees the raw request URI rather than the path segments -//! the server decodes. A broadcast occupies every segment before the HLS resource -//! suffix, still percent-encoded: `/project/li%76e/master.m3u8` serves the broadcast -//! `project/live`. Decode each segment before matching it against a policy, or a +//! the server decodes. A broadcast occupies every segment before the instance (or the +//! master and manifest), still percent-encoded: `/project/li%76e/master.m3u8` serves the +//! broadcast `project/live`. Decode each segment before matching it against a policy, or a //! name can be encoded past the check. //! //! An embedder that authorizes each request itself (a token scoping the broadcast, a @@ -62,6 +69,7 @@ use std::task::Poll; use std::time::Duration; use axum::Router; +use moq_mux::follow; use crate::export::{Broadcaster, Config}; @@ -81,10 +89,67 @@ struct Inner { /// client could otherwise name without bound. _watch: moq_net::announce::Consumer, config: Config, - broadcasters: Mutex>>, - /// Closes with the last server clone, ending every eviction task, which would otherwise keep - /// its broadcaster for as long as the publisher stays live. - alive: kio::Producer<()>, + /// Each requested broadcast, followed until nothing serves its path. + broadcasts: Mutex>>, +} + +/// The publisher instances serving one broadcast path. +struct Followed { + instances: Mutex, + /// Follows the path. Aborted with the last handle, so a dropped server stops following. + task: Mutex>>, +} + +struct Instances { + /// The instance the master playlist and manifest list: the latest to take over. + current: Arc, + /// Every instance a listed URL still names, the current one included. A replaced instance + /// stays until its broadcast closes, so a player already on it keeps playing. + by_id: HashMap, Arc>, +} + +impl Followed { + fn new(current: Arc) -> Self { + let by_id = HashMap::from([(current.instance().clone(), current.clone())]); + Self { + instances: Mutex::new(Instances { current, by_id }), + task: Mutex::new(None), + } + } + + fn current(&self) -> Arc { + self.instances.lock().unwrap().current.clone() + } + + fn instance(&self, id: &str) -> Option> { + self.instances.lock().unwrap().by_id.get(id).cloned() + } + + /// List `broadcaster` from now on. An instance returning after a gap keeps its epoch, and so + /// its id, which then names the new broadcaster. + fn start(&self, broadcaster: Arc) { + let mut instances = self.instances.lock().unwrap(); + instances + .by_id + .insert(broadcaster.instance().clone(), broadcaster.clone()); + instances.current = broadcaster; + } + + /// Forget every instance whose broadcast closed, registering `waiter` on the rest. + fn forget_closed(&self, waiter: &kio::Waiter) { + let mut instances = self.instances.lock().unwrap(); + instances + .by_id + .retain(|_, broadcaster| broadcaster.poll_closed(waiter).is_pending()); + } +} + +impl Drop for Followed { + fn drop(&mut self) { + if let Some(task) = self.task.lock().unwrap().take() { + task.abort(); + } + } } impl Server { @@ -97,8 +162,7 @@ impl Server { _watch: origin.clone().with_hidden(true).announced(), origin, config, - broadcasters: Mutex::new(HashMap::new()), - alive: kio::Producer::new(()), + broadcasts: Mutex::new(HashMap::new()), }), } } @@ -108,85 +172,132 @@ impl Server { routes::router(self.clone()) } - /// Get or create the [`Broadcaster`] for `name`, resolving the broadcast from - /// the relay. Returns `None` if the broadcast is not announced once the origin has - /// synced with the relay. + /// The [`Broadcaster`] for the publisher instance serving `name` now, following each + /// instance that takes over. Returns `None` if the broadcast is not announced once the + /// origin has synced with the relay. pub async fn broadcaster(&self, name: &str) -> Option> { - { - let mut broadcasters = self.inner.broadcasters.lock().unwrap(); - if let Some(existing) = broadcasters.get(name) { - if !existing.is_closed() { - return Some(existing.clone()); - } - broadcasters.remove(name); - } + Some(self.followed(name).await?.current()) + } + + /// The [`Broadcaster`] for the publisher instance `id` of `name`, until its broadcast + /// closes. + pub(crate) async fn instance(&self, name: &str, id: &str) -> Option> { + self.followed(name).await?.instance(id) + } + + /// Follow `name` from the instance serving it now, unless it is followed already. + async fn followed(&self, name: &str) -> Option> { + if let Some(followed) = self.cached(name) { + return Some(followed); } // Confirm a route covers the broadcast (and it is in scope) before resolving it. The // source resolves a rendition's sibling `broadcast` reference through the same origin. // Once the origin has synced, a path with no route is not announced, so answer now // rather than wait out a publisher that has gone. - let mut follow = self.inner.origin.follow(name).ok()?; - let routed = kio::wait(|waiter| match follow.poll_next(waiter) { + let mut routes = self.inner.origin.follow(name).ok()?; + let routed = kio::wait(|waiter| match routes.poll_next(waiter) { Poll::Ready(Some(moq_net::announce::Event::End(_)) | None) => Poll::Ready(false), Poll::Ready(Some(_)) => Poll::Ready(true), - Poll::Pending => follow.poll_synced(waiter).map(|()| false), + Poll::Pending => routes.poll_synced(waiter).map(|()| false), }); if !moq_clock::timeout(RESOLVE_TIMEOUT, routed).await.unwrap_or(false) { return None; } - let origin = self.inner.origin.consume(); - // A covering dynamic route may never answer, so the request is bounded too. - let broadcast = moq_clock::timeout(RESOLVE_TIMEOUT, origin.request_broadcast(name, None)) - .await - .map_err(|_| tracing::warn!(%name, "timed out resolving broadcast")) - .ok()? - .map_err(|err| tracing::warn!(%err, %name, "failed to resolve broadcast")) + let mut instances = follow::Instances::new(&self.inner.origin, name, None) + .map_err(|err| tracing::warn!(%err, %name, "failed to follow broadcast")) .ok()?; - let broadcaster = Broadcaster::new(moq_mux::Source::new(origin, broadcast), self.inner.config.clone()); + // A covering dynamic route may never answer, so the resolution is bounded too. + let broadcast = match moq_clock::timeout(RESOLVE_TIMEOUT, instances.next()).await { + Ok(Ok(follow::Event::Start(broadcast))) => broadcast, + // Nothing was handed out, so there is nothing to end. + Ok(Ok(follow::Event::End)) => return None, + Ok(Err(err)) => { + tracing::warn!(%err, %name, "failed to resolve broadcast"); + return None; + } + Err(_) => { + tracing::warn!(%name, "timed out resolving broadcast"); + return None; + } + }; + let followed = Arc::new(Followed::new(self.inner.serve(broadcast))); - let mut broadcasters = self.inner.broadcasters.lock().unwrap(); - if let Some(existing) = broadcasters.get(name) { - if !existing.is_closed() { + { + let mut broadcasts = self.inner.broadcasts.lock().unwrap(); + if let Some(existing) = broadcasts.get(name) + && !existing.current().is_closed() + { return Some(existing.clone()); } - broadcasters.remove(name); + broadcasts.insert(name.to_string(), followed.clone()); } - let name = name.to_string(); - broadcasters.insert(name.clone(), broadcaster.clone()); - let alive = self.inner.alive.consume(); - tokio::spawn(evict_closed( + let task = tokio::spawn(run_follow( Arc::downgrade(&self.inner), - alive, - name, - broadcaster.clone(), + name.to_string(), + Arc::downgrade(&followed), + instances, )); - Some(broadcaster) + *followed.task.lock().unwrap() = Some(task); + Some(followed) + } + + /// The followed `name`, unless its current instance closed. Its follower is about to forget + /// it then, so resolve whatever serves the path now instead. + fn cached(&self, name: &str) -> Option> { + let mut broadcasts = self.inner.broadcasts.lock().unwrap(); + let followed = broadcasts.get(name)?; + if followed.current().is_closed() { + broadcasts.remove(name); + return None; + } + Some(followed.clone()) } } -/// Forget `broadcaster` once its source closes, unless the server is dropped first. The task -/// holds the server weakly, so it never keeps the server's state alive. -async fn evict_closed(inner: Weak, alive: kio::Consumer<()>, name: String, broadcaster: Arc) { - let source_closed = kio::wait(|waiter| { - if broadcaster.poll_closed(waiter).is_ready() { - return Poll::Ready(true); +impl Inner { + /// A broadcaster serving the instance `broadcast`. + fn serve(&self, broadcast: moq_net::broadcast::Consumer) -> Arc { + let source = moq_mux::Source::new(self.origin.consume(), broadcast); + Broadcaster::new(source, self.config.clone()) + } +} + +/// Follow `name`'s instances into `followed` until nothing serves the path, then forget it, so the +/// next request follows the path afresh. Holds both weakly, so it never keeps the server's state +/// alive; dropping `followed` aborts it. +async fn run_follow(inner: Weak, name: String, followed: Weak, mut instances: follow::Instances) { + let (inner, followed) = loop { + let next = kio::wait(|waiter| { + let Some(followed) = followed.upgrade() else { + return Poll::Ready(None); + }; + followed.forget_closed(waiter); + instances.poll_next(waiter).map(Some) + }) + .await; + let (Some(inner), Some(followed)) = (inner.upgrade(), followed.upgrade()) else { + return; + }; + match next { + Some(Ok(follow::Event::Start(broadcast))) => followed.start(inner.serve(broadcast)), + Some(Ok(follow::Event::End)) => break (inner, followed), + Some(Err(err)) => { + tracing::warn!(%err, %name, "stopped following broadcast"); + break (inner, followed); + } + None => return, } - alive.poll_closed(waiter).map(|()| false) - }) - .await; - let Some(inner) = inner.upgrade().filter(|_| source_closed) else { - return; }; - let mut broadcasters = inner.broadcasters.lock().unwrap(); - if broadcasters + let mut broadcasts = inner.broadcasts.lock().unwrap(); + if broadcasts .get(&name) - .is_some_and(|current| Arc::ptr_eq(current, &broadcaster)) + .is_some_and(|current| Arc::ptr_eq(current, &followed)) { - broadcasters.remove(&name); + broadcasts.remove(&name); } } @@ -321,10 +432,10 @@ mod tests { server .inner - .broadcasters + .broadcasts .lock() .unwrap() - .insert("live".to_string(), stale.clone()); + .insert("live".to_string(), Arc::new(Followed::new(stale.clone()))); let _producer = origin.create_broadcast("live").expect("publish allowed"); _producer.announce(Default::default()).expect("publish allowed"); settle().await; @@ -332,7 +443,7 @@ mod tests { let fresh = server.broadcaster("live").await.expect("broadcast announced"); assert!(!Arc::ptr_eq(&stale, &fresh)); - assert!(server.inner.broadcasters.lock().unwrap().contains_key("live")); + assert!(server.inner.broadcasts.lock().unwrap().contains_key("live")); } /// Dropping the last server clone releases its broadcasters while their publisher stays @@ -364,28 +475,46 @@ mod tests { producer.close(); } - #[tokio::test] - async fn eviction_keeps_newer_cached_instance() { + /// Wait out the origin's update hold until `ready` holds. + async fn eventually(mut ready: impl AsyncFnMut() -> bool) { + for _ in 0..100 { + if ready().await { + return; + } + moq_clock::sleep(Duration::from_millis(100)).await; + } + panic!("never happened"); + } + + /// A replacement becomes the instance the entry points list, under its own id, while the old + /// instance keeps serving its URLs until its broadcast closes. Once nothing serves the path, + /// the server forgets it. + #[tokio::test(start_paused = true)] + async fn a_replaced_instance_serves_until_it_closes() { let origin = produce_origin(); let server = Server::new(origin.consume(), Config::default()); - let old = closed_broadcaster().await; - let new_producer = origin.create_broadcast("live").expect("publish allowed"); - new_producer.announce(Default::default()).expect("publish allowed"); - settle().await; - let new = Broadcaster::new(crate::resolved(&origin, "live").await, Config::default()); + let (old_epoch, new_epoch) = (moq_net::Epoch::mint(), moq_net::Epoch::mint()); + let route = |epoch: &moq_net::Epoch| moq_net::origin::Route::default().with_epoch(epoch.clone()); - server - .inner - .broadcasters - .lock() - .unwrap() - .insert("live".to_string(), new.clone()); - - let alive = server.inner.alive.consume(); - evict_closed(Arc::downgrade(&server.inner), alive, "live".to_string(), old).await; - - let cached = server.inner.broadcasters.lock().unwrap().get("live").cloned(); - assert!(cached.is_some_and(|cached| Arc::ptr_eq(&cached, &new))); - new_producer.close(); + let old = origin.publish("live", route(&old_epoch)).expect("publish allowed"); + settle().await; + let first = server.broadcaster("live").await.expect("broadcast announced"); + assert_eq!(first.instance().as_ref(), old_epoch.as_str()); + + let new = origin.publish("live", route(&new_epoch)).expect("publish allowed"); + eventually(async || { + let current = server.broadcaster("live").await.expect("broadcast announced"); + current.instance().as_ref() == new_epoch.as_str() + }) + .await; + let kept = server.instance("live", old_epoch.as_str()).await; + assert!(kept.is_some_and(|kept| Arc::ptr_eq(&kept, &first)), "the old instance serves"); + + drop(old); + eventually(async || server.instance("live", old_epoch.as_str()).await.is_none()).await; + assert!(server.instance("live", new_epoch.as_str()).await.is_some()); + + drop(new); + eventually(async || server.inner.broadcasts.lock().unwrap().is_empty()).await; } } diff --git a/rs/moq-hls/src/server/routes.rs b/rs/moq-hls/src/server/routes.rs index 85a8e83b741..708ea728275 100644 --- a/rs/moq-hls/src/server/routes.rs +++ b/rs/moq-hls/src/server/routes.rs @@ -41,21 +41,25 @@ pub struct Route { #[derive(Clone, Debug, PartialEq, Eq)] #[non_exhaustive] pub enum Resource { - /// `master.m3u8`: the HLS multivariant playlist. + /// `master.m3u8`: the HLS multivariant playlist of the instance serving the broadcast now. Master, - /// `manifest.mpd`: the DASH manifest. + /// `manifest.mpd`: the DASH manifest of the instance serving the broadcast now. Manifest, - /// `{kind}/{rendition}/media.m3u8`: a rendition's HLS media playlist. + /// `@{instance}/{kind}/{rendition}/media.m3u8`: a rendition's HLS media playlist. #[non_exhaustive] Media { + /// The publisher instance the URL names, without its `@`. + instance: String, /// The rendition's kind. kind: Kind, /// The rendition's name, percent-decoded. rendition: String, }, - /// `{kind}/{rendition}/init.{hash}.mp4`: a rendition's CMAF init segment. + /// `@{instance}/{kind}/{rendition}/init.{hash}.mp4`: a rendition's CMAF init segment. #[non_exhaustive] Init { + /// The publisher instance the URL names, without its `@`. + instance: String, /// The rendition's kind. kind: Kind, /// The rendition's name, percent-decoded. @@ -63,10 +67,12 @@ pub enum Resource { /// The hash of the init bytes the URL names. hash: String, }, - /// `{kind}/{rendition}/seg/[{generation}.]{reference}.{sequence}.m4s`: a segment by its HLS - /// number. + /// `@{instance}/{kind}/{rendition}/seg/[{generation}.]{reference}.{sequence}.m4s`: a segment + /// by its HLS number. #[non_exhaustive] Segment { + /// The publisher instance the URL names, without its `@`. + instance: String, /// The rendition's kind. kind: Kind, /// The rendition's name, percent-decoded. @@ -78,10 +84,12 @@ pub enum Resource { /// The segment's aligned number. sequence: u64, }, - /// `{kind}/{rendition}/seg/[{generation}.]{reference}.t{pts}.m4s`: the same bytes, addressed - /// by the DASH timeline pts (`$Time$`). + /// `@{instance}/{kind}/{rendition}/seg/[{generation}.]{reference}.t{pts}.m4s`: the same + /// bytes, addressed by the DASH timeline pts (`$Time$`). #[non_exhaustive] SegmentAt { + /// The publisher instance the URL names, without its `@`. + instance: String, /// The rendition's kind. kind: Kind, /// The rendition's name, percent-decoded. @@ -96,10 +104,11 @@ pub enum Resource { } impl Route { - /// Parse a request path such as `/project/live/video/hd/media.m3u8`, or `None` when it - /// addresses nothing. + /// Parse a request path such as `/project/live/@{instance}/video/hd/media.m3u8`, or `None` + /// when it addresses nothing. /// - /// The broadcast occupies every segment before the resource suffix. Each segment is + /// The broadcast occupies every segment before the resource suffix, which for every + /// resource but the master and manifest starts with the `@{instance}` segment. Each segment is /// percent-decoded, and one that decodes to a `/` is refused, so an encoded separator /// cannot smuggle a different broadcast past a policy check on [`broadcast`](Self::broadcast). pub fn parse(path: &str) -> Option { @@ -120,22 +129,25 @@ impl Route { let (broadcast, resource) = match parts.as_slice() { [broadcast @ .., file] if file == "master.m3u8" => (broadcast, Resource::Master), [broadcast @ .., file] if file == "manifest.mpd" => (broadcast, Resource::Manifest), - [broadcast @ .., kind, rendition, file] if file == "media.m3u8" => ( + [broadcast @ .., instance, kind, rendition, file] if file == "media.m3u8" => ( broadcast, Resource::Media { + instance: instance_id(instance)?, kind: Kind::parse(kind)?, rendition: rendition.clone(), }, ), - [broadcast @ .., kind, rendition, file] if init_hash(file).is_some() => ( + [broadcast @ .., instance, kind, rendition, file] if init_hash(file).is_some() => ( broadcast, Resource::Init { + instance: instance_id(instance)?, kind: Kind::parse(kind)?, rendition: rendition.clone(), hash: init_hash(file)?.to_string(), }, ), - [broadcast @ .., kind, rendition, directory, file] if directory == "seg" => { + [broadcast @ .., instance, kind, rendition, directory, file] if directory == "seg" => { + let instance = instance_id(instance)?; let kind = Kind::parse(kind)?; let rendition = rendition.clone(); let stem = file.strip_suffix(".m4s")?; @@ -148,6 +160,7 @@ impl Route { }; let resource = match stem.strip_prefix('t') { Some(pts) => Resource::SegmentAt { + instance, kind, rendition, generation, @@ -155,6 +168,7 @@ impl Route { pts: pts.parse().ok()?, }, None => Resource::Segment { + instance, kind, rendition, generation, @@ -177,6 +191,13 @@ impl Route { } } +/// The publisher instance an `@{instance}` path segment names. +fn instance_id(part: &str) -> Option { + part.strip_prefix('@') + .filter(|instance| !instance.is_empty()) + .map(str::to_string) +} + /// The content hash in an `init.{hash}.mp4` file name. fn init_hash(file: &str) -> Option<&str> { file.strip_prefix("init.")? @@ -204,27 +225,40 @@ impl Server { match &route.resource { Resource::Master => master(self, broadcast, query).await, Resource::Manifest => manifest(self, broadcast, query).await, - Resource::Media { kind, rendition } => media(self, broadcast, *kind, rendition, query).await, - Resource::Init { kind, rendition, hash } => init(self, broadcast, *kind, rendition, hash).await, + Resource::Media { + instance, + kind, + rendition, + } => media(self, At::new(broadcast, instance, *kind, rendition), query).await, + Resource::Init { + instance, + kind, + rendition, + hash, + } => init(self, At::new(broadcast, instance, *kind, rendition), hash).await, Resource::Segment { + instance, kind, rendition, generation, reference, sequence, } => { - let at = SegmentAt::Sequence(reference, *sequence); - segment(self, broadcast, *kind, rendition, generation.as_deref(), at).await + let at = At::new(broadcast, instance, *kind, rendition); + let by = SegmentAt::Sequence(reference, *sequence); + segment(self, at, generation.as_deref(), by).await } Resource::SegmentAt { + instance, kind, rendition, generation, reference, pts, } => { - let at = SegmentAt::Pts(reference, *pts); - segment(self, broadcast, *kind, rendition, generation.as_deref(), at).await + let at = At::new(broadcast, instance, *kind, rendition); + let by = SegmentAt::Pts(reference, *pts); + segment(self, at, generation.as_deref(), by).await } } } @@ -264,12 +298,12 @@ async fn manifest(server: &Server, broadcast: &str, query: Option<&str>) -> Resp } } -async fn media(server: &Server, broadcast: &str, kind: Kind, rendition: &str, query: Option<&str>) -> Response { +async fn media(server: &Server, at: At<'_>, query: Option<&str>) -> Response { let (range, child_query) = match playlist_range(query) { Ok(parsed) => parsed, Err(()) => return bad_request(), }; - let Some(rendition) = rendition_for(server, broadcast, kind, rendition).await else { + let Some(rendition) = rendition_for(server, at).await else { return not_found(); }; match moq_clock::timeout(READY_TIMEOUT, rendition.playlist(range, child_query.as_deref())).await { @@ -318,8 +352,8 @@ fn media_time(value: &str) -> Result { Duration::try_from_secs_f64(seconds).map_err(|_| ()) } -async fn init(server: &Server, broadcast: &str, kind: Kind, rendition: &str, hash: &str) -> Response { - let Some(rendition) = rendition_for(server, broadcast, kind, rendition).await else { +async fn init(server: &Server, at: At<'_>, hash: &str) -> Response { + let Some(rendition) = rendition_for(server, at).await else { return not_found(); }; media_result(rendition.init_versioned(hash).await, server) @@ -333,15 +367,8 @@ enum SegmentAt<'a> { Pts(&'a str, u64), } -async fn segment( - server: &Server, - broadcast: &str, - kind: Kind, - rendition: &str, - generation: Option<&str>, - at: SegmentAt<'_>, -) -> Response { - let Some(rendition) = rendition_for(server, broadcast, kind, rendition).await else { +async fn segment(server: &Server, at: At<'_>, generation: Option<&str>, by: SegmentAt<'_>) -> Response { + let Some(rendition) = rendition_for(server, at).await else { return not_found(); }; // Only the current run's URLs are served: a restart reuses segment numbers, so another @@ -350,7 +377,7 @@ async fn segment( if !current() { return not_found(); } - let result = match at { + let result = match by { SegmentAt::Sequence(reference, sequence) => rendition.segment(reference, sequence).await, SegmentAt::Pts(reference, pts) => rendition.segment_at(reference, pts).await, }; @@ -361,11 +388,31 @@ async fn segment( media_result(result, server) } +/// A rendition of one publisher instance, as an instance-scoped URL names it. +#[derive(Clone, Copy)] +struct At<'a> { + broadcast: &'a str, + instance: &'a str, + kind: Kind, + rendition: &'a str, +} + +impl<'a> At<'a> { + fn new(broadcast: &'a str, instance: &'a str, kind: Kind, rendition: &'a str) -> Self { + Self { + broadcast, + instance, + kind, + rendition, + } + } +} + /// Resolve a rendition, waiting for the catalog to populate. -async fn rendition_for(server: &Server, broadcast: &str, kind: Kind, rendition: &str) -> Option> { - let broadcaster = server.broadcaster(broadcast).await?; +async fn rendition_for(server: &Server, at: At<'_>) -> Option> { + let broadcaster = server.instance(at.broadcast, at.instance).await?; let _ = moq_clock::timeout(READY_TIMEOUT, broadcaster.ready()).await; - broadcaster.rendition(kind, rendition) + broadcaster.rendition(at.kind, at.rendition) } fn m3u8(body: String) -> Response { @@ -391,11 +438,11 @@ fn media_result(result: crate::Result>, server: &Server) -> Respon } fn media_bytes(body: Bytes, server: &Server) -> Response { - // Init/segment bytes never change while their URL is listed, but a segment URL without a - // generation is not globally unique: a restarted publisher starts a new timeline whose - // segment numbers and pts can repeat the old ones with different media. Cap shared caching - // at the playlist window - every concurrent viewer of the live window still hits the cache, - // while a stale run's bytes age out as fast as the window that stopped listing them. + // Init/segment bytes never change while their URL is listed, but a run spliced into one + // instance without a generation (see `Broadcaster::set_generation`) can repeat the old run's + // segment numbers and pts with different media. Cap shared caching at the playlist window - + // every concurrent viewer of the live window still hits the cache, while a stale run's bytes + // age out as fast as the window that stopped listing them. let max_age = server.inner.config.window.as_secs().max(1); ( [ @@ -473,10 +520,11 @@ mod tests { #[test] fn parses_multisegment_broadcast_and_encoded_rendition() { assert_eq!( - Route::parse("/project/live/video/cam%231%2Fmain%3Falt/media.m3u8"), + Route::parse("/project/live/@run/video/cam%231%2Fmain%3Falt/media.m3u8"), Some(Route { broadcast: "project/live".to_string(), resource: Resource::Media { + instance: "run".to_string(), kind: Kind::Video, rendition: "cam#1/main?alt".to_string(), }, @@ -499,22 +547,19 @@ mod tests { }; // The manifest for broadcast `live` and rendition `video/1080p` emits these paths. - // Unencoded, the same name is broadcast `live/video`, rendition `1080p`. + // Unencoded, the same name names no instance. let cases = [ - ("/live/video/video%2F1080p/init.0123abcd.mp4", "video/1080p"), - ("/live/video/video%2F1080p/seg/ab12cd34.t2000.m4s", "video/1080p"), - ("/live/video/cam%3F1/init.0123abcd.mp4", "cam?1"), - ("/live/video/cam%231/media.m3u8", "cam#1"), - ("/live/audio/100%25/init.0123abcd.mp4", "100%"), - ("/live/video/price%249/seg/ab12cd34.t0.m4s", "price$9"), + ("/live/@run/video/video%2F1080p/init.0123abcd.mp4", "video/1080p"), + ("/live/@run/video/video%2F1080p/seg/ab12cd34.t2000.m4s", "video/1080p"), + ("/live/@run/video/cam%3F1/init.0123abcd.mp4", "cam?1"), + ("/live/@run/video/cam%231/media.m3u8", "cam#1"), + ("/live/@run/audio/100%25/init.0123abcd.mp4", "100%"), + ("/live/@run/video/price%249/seg/ab12cd34.t0.m4s", "price$9"), ]; for (path, rendition) in cases { assert_eq!(parsed(path), Some(("live".into(), rendition.into())), "{path}"); } - assert_eq!( - parsed("/live/video/video/1080p/init.0123abcd.mp4"), - Some(("live/video".into(), "1080p".into())) - ); + assert_eq!(parsed("/live/@run/video/video/1080p/init.0123abcd.mp4"), None); } #[test] @@ -523,16 +568,18 @@ mod tests { assert_eq!(resource("/project/live/master.m3u8"), Some(Resource::Master)); assert_eq!(resource("/project/live/manifest.mpd"), Some(Resource::Manifest)); assert_eq!( - resource("/project/live/audio/main/init.0123abcd.mp4"), + resource("/project/live/@run/audio/main/init.0123abcd.mp4"), Some(Resource::Init { + instance: "run".to_string(), kind: Kind::Audio, rendition: "main".to_string(), hash: "0123abcd".to_string(), }) ); assert_eq!( - resource("/project/live/video/main/seg/ab12cd34.42.m4s"), + resource("/project/live/@run/video/main/seg/ab12cd34.42.m4s"), Some(Resource::Segment { + instance: "run".to_string(), kind: Kind::Video, rendition: "main".to_string(), generation: None, @@ -541,8 +588,9 @@ mod tests { }) ); assert_eq!( - resource("/project/live/video/main/seg/run-1.ab12cd34.42.m4s"), + resource("/project/live/@run/video/main/seg/run-1.ab12cd34.42.m4s"), Some(Resource::Segment { + instance: "run".to_string(), kind: Kind::Video, rendition: "main".to_string(), generation: Some("run-1".to_string()), @@ -551,8 +599,9 @@ mod tests { }) ); assert_eq!( - resource("/project/live/video/main/seg/ab12cd34.t2000.m4s"), + resource("/project/live/@run/video/main/seg/ab12cd34.t2000.m4s"), Some(Resource::SegmentAt { + instance: "run".to_string(), kind: Kind::Video, rendition: "main".to_string(), generation: None, @@ -561,8 +610,9 @@ mod tests { }) ); assert_eq!( - resource("/project/live/video/main/seg/init.ab12cd34.t2000.m4s"), + resource("/project/live/@run/video/main/seg/init.ab12cd34.t2000.m4s"), Some(Resource::SegmentAt { + instance: "run".to_string(), kind: Kind::Video, rendition: "main".to_string(), generation: Some("init".to_string()), @@ -571,16 +621,21 @@ mod tests { }) ); assert!(Route::parse("/master.m3u8").is_none()); - assert!(Route::parse("/project/live/video/main/unknown").is_none()); - assert!(Route::parse("/project/live/data/main/media.m3u8").is_none()); - assert!(Route::parse("/project/live/video/main/init.mp4").is_none()); - assert!(Route::parse("/project/live/video/main/init..mp4").is_none()); - assert!(Route::parse("/project/live/video/main/seg/ab12cd34.tx.m4s").is_none()); assert!( - Route::parse("/project/live/video/main/seg/42.m4s").is_none(), + Route::parse("/project/live/video/main/media.m3u8").is_none(), + "a rendition URL names its instance" + ); + assert!(Route::parse("/project/live/@/video/main/media.m3u8").is_none()); + assert!(Route::parse("/project/live/@run/video/main/unknown").is_none()); + assert!(Route::parse("/project/live/@run/data/main/media.m3u8").is_none()); + assert!(Route::parse("/project/live/@run/video/main/init.mp4").is_none()); + assert!(Route::parse("/project/live/@run/video/main/init..mp4").is_none()); + assert!(Route::parse("/project/live/@run/video/main/seg/ab12cd34.tx.m4s").is_none()); + assert!( + Route::parse("/project/live/@run/video/main/seg/42.m4s").is_none(), "a segment URL names its reference" ); - assert!(Route::parse("/project/live/video/main/seg/1.ts").is_none()); + assert!(Route::parse("/project/live/@run/video/main/seg/1.ts").is_none()); } #[test] @@ -596,6 +651,23 @@ mod tests { const TIMEOUT: Duration = Duration::from_secs(10); + /// The epoch each test's catalog broadcast announces, and so the instance its URLs name. + const RUN: &str = "01920000-0000-7000-8000-000000000000"; + + fn run() -> moq_net::origin::Route { + moq_net::origin::Route::default().with_epoch(RUN.parse().unwrap()) + } + + /// The instance `server` lists for `broadcast`, which it mints for a session older than + /// lite-07, since those carry no epoch. + async fn listed(server: &Server, broadcast: &str) -> String { + let broadcaster = tokio::time::timeout(TIMEOUT, server.broadcaster(broadcast)) + .await + .expect("resolve timed out") + .expect("announced"); + broadcaster.instance().to_string() + } + fn vp8_frame(micros: u64, keyframe: bool) -> moq_mux::container::Frame { let payload = if keyframe { &[0x10, 0x00, 0x00, 0x9d, 0x01, 0x2a, 0x40, 0x01, 0xf0, 0x00][..] @@ -759,7 +831,8 @@ mod tests { write_three_gops(&mut media); let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); - let mut route = Route::parse("/tenant/live/video/video0/media.m3u8").expect("media route"); + let run = listed(&server, "live").await; + let mut route = Route::parse(&format!("/tenant/live/@{run}/video/video0/media.m3u8")).expect("media route"); route.broadcast = route.broadcast.strip_prefix("tenant/").expect("scoped").to_string(); let deadline = tokio::time::Instant::now() + TIMEOUT; @@ -873,7 +946,7 @@ mod tests { async fn master_answers_404_until_a_rendition_can_start() { let origin = moq_tokio::origin::spawn(); let mut broadcast = origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); + broadcast.announce(run()).expect("announce"); let (_catalog, _registration, _track, mut media) = publish_video(&mut broadcast, video_config(), timed()); let app = Server::new(origin.consume(), crate::export::Config::default()).router(); @@ -883,13 +956,57 @@ mod tests { wait_listed(&app, "/live/master.m3u8", "video/video0/media.m3u8").await; } + /// A restarted publisher gets URLs of its own, whose numbering starts over: the master lists + /// the replacement, while a player already on the old instance keeps its playlist until that + /// instance's broadcast closes. + #[tokio::test(start_paused = true)] + async fn a_restart_lists_the_replacement_under_its_own_urls() { + let origin = moq_tokio::origin::spawn(); + let mut old = origin.create_broadcast("live").expect("publish"); + old.announce(run()).expect("announce"); + let (old_catalog, _, old_track, mut old_media) = publish_video(&mut old, video_config(), timed()); + write_three_gops(&mut old_media); + + let app = Server::new(origin.consume(), crate::export::Config::default()).router(); + let old_playlist = format!("/live/@{RUN}/video/video0/media.m3u8"); + let old_segment = format!("/live/@{RUN}/video/video0/seg/656e3d1b.0.m4s"); + wait_listed(&app, "/live/master.m3u8", &format!("@{RUN}/video/video0/media.m3u8")).await; + wait_listed(&app, &old_playlist, "seg/656e3d1b.0.m4s").await; + + let next = moq_net::Epoch::mint(); + let mut new = origin.create_broadcast("live").expect("publish"); + new.announce(moq_net::origin::Route::default().with_epoch(next.clone())) + .expect("announce"); + let (_new_catalog, _, _new_track, mut new_media) = publish_video(&mut new, video_config(), timed()); + write_three_gops(&mut new_media); + let new_playlist = format!("/live/@{}/video/video0/media.m3u8", next.as_str()); + wait_listed( + &app, + "/live/master.m3u8", + &format!("@{}/video/video0/media.m3u8", next.as_str()), + ) + .await; + wait_listed(&app, &new_playlist, "seg/656e3d1b.0.m4s").await; + assert_eq!(status(&app, &old_playlist).await, StatusCode::OK); + assert_eq!(status(&app, &old_segment).await, StatusCode::OK); + + drop((old_catalog, old_track, old_media, old)); + let deadline = tokio::time::Instant::now() + TIMEOUT; + while status(&app, &old_playlist).await != StatusCode::NOT_FOUND { + assert!(tokio::time::Instant::now() < deadline, "the closed instance kept serving"); + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert_eq!(status(&app, &old_segment).await, StatusCode::NOT_FOUND); + assert_eq!(status(&app, &new_playlist).await, StatusCode::OK); + } + /// A failed timeline answers every listing, a clip included, with an uncacheable error /// rather than ending it as a complete recording or refusing the range. #[tokio::test(start_paused = true)] async fn a_failed_timeline_answers_an_uncacheable_error() { let origin = moq_tokio::origin::spawn(); let mut broadcast = origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); + broadcast.announce(run()).expect("announce"); let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); let name = hang::timeline::default_name("video0"); catalog @@ -919,7 +1036,7 @@ mod tests { ..Default::default() }; let app = Server::new(origin.consume(), config).router(); - wait_listed(&app, "/live/video/video0/media.m3u8", ".2.m4s").await; + wait_listed(&app, &format!("/live/@{RUN}/video/video0/media.m3u8"), ".2.m4s").await; assert_eq!(status(&app, "/live/manifest.mpd").await, StatusCode::OK); drop(timeline); @@ -927,8 +1044,8 @@ mod tests { .abort(moq_net::Error::Transport("connection lost".into())) .unwrap(); for uri in [ - "/live/video/video0/media.m3u8", - "/live/video/video0/media.m3u8?start=0&end=2", + &format!("/live/@{RUN}/video/video0/media.m3u8"), + &format!("/live/@{RUN}/video/video0/media.m3u8?start=0&end=2"), "/live/manifest.mpd", ] { let deadline = tokio::time::Instant::now() + TIMEOUT; @@ -951,17 +1068,17 @@ mod tests { async fn serves_only_versioned_media_urls() { let origin = moq_tokio::origin::spawn(); let mut broadcast = origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); + broadcast.announce(run()).expect("announce"); let (_catalog, _registration, _track, mut media) = publish_video(&mut broadcast, video_config(), timed()); write_three_gops(&mut media); let server = Server::new(origin.consume(), crate::export::Config::default()); let app = server.router(); - wait_listed(&app, "/live/video/video0/media.m3u8", "seg/656e3d1b.0.m4s").await; + wait_listed(&app, &format!("/live/@{RUN}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; let broadcaster = server.broadcaster("live").await.expect("broadcaster"); broadcaster.set_generation(Some("run-1")).unwrap(); - let response = oneshot(app.clone(), "/live/video/video0/media.m3u8").await; + let response = oneshot(app.clone(), &format!("/live/@{RUN}/video/video0/media.m3u8")).await; let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap(); let playlist = String::from_utf8(body.to_vec()).unwrap(); assert!(playlist.contains("\nseg/run-1.656e3d1b.0.m4s\n"), "{playlist}"); @@ -973,36 +1090,36 @@ mod tests { .unwrap(); assert_eq!( - status(&app, &format!("/live/video/video0/{init}")).await, + status(&app, &format!("/live/@{RUN}/video/video0/{init}")).await, StatusCode::OK ); - assert_eq!(status(&app, "/live/video/video0/init.mp4").await, StatusCode::NOT_FOUND); + assert_eq!(status(&app, &format!("/live/@{RUN}/video/video0/init.mp4")).await, StatusCode::NOT_FOUND); assert_eq!( - status(&app, "/live/video/video0/init.0000000000000000.mp4").await, + status(&app, &format!("/live/@{RUN}/video/video0/init.0000000000000000.mp4")).await, StatusCode::NOT_FOUND ); assert_eq!( - status(&app, "/live/video/video0/seg/run-1.656e3d1b.0.m4s").await, + status(&app, &format!("/live/@{RUN}/video/video0/seg/run-1.656e3d1b.0.m4s")).await, StatusCode::OK ); assert_eq!( - status(&app, "/live/video/video0/seg/run-1.656e3d1b.t0.m4s").await, + status(&app, &format!("/live/@{RUN}/video/video0/seg/run-1.656e3d1b.t0.m4s")).await, StatusCode::OK ); assert_eq!( - status(&app, "/live/video/video0/seg/656e3d1b.0.m4s").await, + status(&app, &format!("/live/@{RUN}/video/video0/seg/656e3d1b.0.m4s")).await, StatusCode::NOT_FOUND ); assert_eq!( - status(&app, "/live/video/video0/seg/run-0.656e3d1b.0.m4s").await, + status(&app, &format!("/live/@{RUN}/video/video0/seg/run-0.656e3d1b.0.m4s")).await, StatusCode::NOT_FOUND ); // A new run: the previous generation's URLs are refused even for numbers it reuses. broadcaster.set_generation(Some("run-2")).unwrap(); assert_eq!( - status(&app, "/live/video/video0/seg/run-1.656e3d1b.0.m4s").await, + status(&app, &format!("/live/@{RUN}/video/video0/seg/run-1.656e3d1b.0.m4s")).await, StatusCode::NOT_FOUND ); } @@ -1022,10 +1139,12 @@ mod tests { let (_catalog, _registration, _track, mut media) = publish_video(&mut broadcast, video_config(), timed()); write_three_gops(&mut media); - let app = Server::new(pair.sub_origin.consume(), crate::export::Config::default()).router(); - wait_listed(&app, "/live/video/video0/media.m3u8", "seg/656e3d1b.0.m4s").await; + let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); + let run = listed(&server, "live").await; + let app = server.router(); + wait_listed(&app, &format!("/live/@{run}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; - let response = oneshot(app, "/live/video/video0/seg/656e3d1b.0.m4s").await; + let response = oneshot(app, &format!("/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; assert_eq!(response.status(), StatusCode::NOT_FOUND); assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); @@ -1040,8 +1159,10 @@ mod tests { let (catalog, registration, track, mut media) = publish_video(&mut broadcast, video_config(), timed()); write_three_gops(&mut media); - let app = Server::new(pair.sub_origin.consume(), crate::export::Config::default()).router(); - wait_listed(&app, "/live/video/video0/media.m3u8", "seg/656e3d1b.0.m4s").await; + let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); + let run = listed(&server, "live").await; + let app = server.router(); + wait_listed(&app, &format!("/live/@{run}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; let remote = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().request_broadcast("live", None)) .await @@ -1052,7 +1173,7 @@ mod tests { .await .expect("publisher close timed out"); - let response = oneshot(app, "/live/video/video0/seg/656e3d1b.0.m4s").await; + let response = oneshot(app, &format!("/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; assert_eq!(response.status(), StatusCode::NOT_FOUND); assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); @@ -1128,7 +1249,7 @@ mod tests { async fn denied_sibling_media_answers_403_without_poisoning_allowed_siblings() { let origin = moq_tokio::origin::spawn(); let mut broadcast = origin.create_broadcast("room/live").unwrap(); - broadcast.announce(Default::default()).unwrap(); + broadcast.announce(run()).unwrap(); let source = origin.create_broadcast("room/source").unwrap(); source.announce(Default::default()).unwrap(); let secret = origin.create_broadcast("room/secret").unwrap(); @@ -1165,15 +1286,15 @@ mod tests { let server = Server::new(restricted, Default::default()); let broadcaster = server.broadcaster("room/live").await.unwrap(); broadcaster.ready().await; - let allowed_uri = "/room/live/video/video0/seg/656e3d1b.0.m4s"; - let denied_uri = "/room/live/video/video1/seg/656e3d1b.0.m4s"; + let allowed_uri = &format!("/room/live/@{RUN}/video/video0/seg/656e3d1b.0.m4s"); + let denied_uri = &format!("/room/live/@{RUN}/video/video1/seg/656e3d1b.0.m4s"); let app = server.router(); assert_eq!(oneshot(app.clone(), allowed_uri).await.status(), StatusCode::OK); for uri in [ denied_uri, - "/room/live/video/video1/media.m3u8", - "/room/live/video/video1/media.m3u8?start=0&end=2", - "/room/live/video/video1/init.unknown.mp4", + &format!("/room/live/@{RUN}/video/video1/media.m3u8"), + &format!("/room/live/@{RUN}/video/video1/media.m3u8?start=0&end=2"), + &format!("/room/live/@{RUN}/video/video1/init.unknown.mp4"), ] { let response = oneshot(app.clone(), uri).await; assert_eq!(response.status(), StatusCode::FORBIDDEN, "{uri}"); @@ -1208,7 +1329,7 @@ mod tests { assert_eq!(oneshot(app.clone(), denied_uri).await.status(), StatusCode::FORBIDDEN); assert_eq!(oneshot(app.clone(), allowed_uri).await.status(), StatusCode::OK); assert_eq!( - oneshot(app, "/room/live/video/video0/seg/656e3d1b.9999.m4s") + oneshot(app, &format!("/room/live/@{RUN}/video/video0/seg/656e3d1b.9999.m4s")) .await .status(), StatusCode::NOT_FOUND @@ -1264,8 +1385,10 @@ mod tests { drop(reserved); write_three_gops(&mut media); - let app = Server::new(pair.sub_origin.consume(), crate::export::Config::default()).router(); - wait_listed(&app, "/room/live/video/video0/media.m3u8", "seg/656e3d1b.0.m4s").await; + let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); + let run = listed(&server, "room/live").await; + let app = server.router(); + wait_listed(&app, &format!("/room/live/@{run}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; // Waited on rather than requested once: a client only asks about what it watches. let remote_media = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().routed_broadcast("room/source")) @@ -1279,7 +1402,7 @@ mod tests { .await .expect("sibling close timed out"); - let response = oneshot(app, "/room/live/video/video0/seg/656e3d1b.0.m4s").await; + let response = oneshot(app, &format!("/room/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; assert_eq!(response.status(), StatusCode::NOT_FOUND); assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); From 5a2697a8d7ee226517d5b3ca2ee25bbab0802b97 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 10:04:01 -0700 Subject: [PATCH 3/7] fix(rtc)!: keep exports on their instance and fail once its route dies An egress session swallowed its catalog's failure, so a WHEP server session or WHIP client idled forever once its source publisher left. It now fails. Client::publish and Client::subscribe detached their session, so moq import and export rtc --connect exited right after negotiation. They now resolve when the session ends. Tests cover both RTC exports and both RTMP exports staying on their instance while a replacement takes over, then failing once the old route dies. Co-Authored-By: Claude Opus 5.5 --- doc/bin/rtc.md | 6 + rs/moq-rtc/MIGRATE.md | 6 + rs/moq-rtc/src/client/mod.rs | 8 +- rs/moq-rtc/src/client/whep.rs | 7 +- rs/moq-rtc/src/client/whip.rs | 7 +- rs/moq-rtc/src/egress.rs | 23 ++- rs/moq-rtc/src/lib.rs | 256 +++++++++++++++++++++++++++++----- rs/moq-rtc/src/server/mod.rs | 19 +-- rs/moq-rtc/src/session.rs | 36 ++++- rs/moq-rtmp/src/dial.rs | 90 ++++++++++++ rs/moq-rtmp/src/server.rs | 128 +++++++++++++++-- 11 files changed, 490 insertions(+), 96 deletions(-) diff --git a/doc/bin/rtc.md b/doc/bin/rtc.md index d024188c442..e9c8b81375b 100644 --- a/doc/bin/rtc.md +++ b/doc/bin/rtc.md @@ -37,5 +37,11 @@ resource URL tears a session down per the RFC. Each WHIP session publishes under a fresh [epoch](/concept/moq-lite#publisher-epochs), so a publisher that reconnects while its stale session lingers replaces it at once. +An export stays on the publisher it started with. A replacement publisher never +reaches its session, so timestamps and codec state never reset mid-session. +Once the original publisher leaves, the session ends with an error, and the +player or `export rtc --connect` must reconnect to reach the replacement. +`--connect` runs until its session ends. + Built on [str0m](https://crates.io/crates/str0m); the library is [`moq-rtc`](https://docs.rs/moq-rtc). diff --git a/rs/moq-rtc/MIGRATE.md b/rs/moq-rtc/MIGRATE.md index b688af26aa0..eba66748c3f 100644 --- a/rs/moq-rtc/MIGRATE.md +++ b/rs/moq-rtc/MIGRATE.md @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first. ## Unreleased +### Client calls run the session + +`Client::publish` and `Client::subscribe` resolve once the session ends, instead of once it negotiates. +Spawn or select on the future to run other work alongside it, and drop it to end the session. +A peer disconnect resolves `Ok(())`. A `publish` whose source publisher leaves fails, as does a WHEP server session's `Response::run`. + ### One router path `whip::router(server, publisher)` and `whep::router(server, subscriber)` are private. diff --git a/rs/moq-rtc/src/client/mod.rs b/rs/moq-rtc/src/client/mod.rs index 69d8f67ed44..66addbbd070 100644 --- a/rs/moq-rtc/src/client/mod.rs +++ b/rs/moq-rtc/src/client/mod.rs @@ -66,14 +66,16 @@ impl Client { } /// `client subscribe`: pull a remote WHEP feed and publish it as - /// `broadcast` on the local origin. Returns once the session is - /// running in the background. + /// `broadcast` on the local origin, until the session ends. pub async fn subscribe(&self, url: Url, broadcast: moq_net::broadcast::Producer) -> crate::Result<()> { whep::dial(self, url, broadcast).await } /// `client publish`: pull the broadcast at `path` from `origin` and push it to a - /// remote WHIP endpoint. + /// remote WHIP endpoint, until the session ends. + /// + /// It stays on the publisher instance it resolved, and fails once that instance's + /// route dies. /// /// Taking the origin plus path (rather than a resolved [`moq_net::broadcast::Consumer`]) /// lets the egress resolve a rendition whose catalog `broadcast` field references a diff --git a/rs/moq-rtc/src/client/whep.rs b/rs/moq-rtc/src/client/whep.rs index 26b448eb783..444232eebc3 100644 --- a/rs/moq-rtc/src/client/whep.rs +++ b/rs/moq-rtc/src/client/whep.rs @@ -61,10 +61,5 @@ pub(crate) async fn dial(client: &Client, url: Url, broadcast: moq_net::broadcas for mid in mids { session.open_media(mid)?; } - tokio::spawn(async move { - let result = session.run().await; - session::log_session_end("whep client", &result); - }); - - Ok(()) + session::end("whep client", session.run().await) } diff --git a/rs/moq-rtc/src/client/whip.rs b/rs/moq-rtc/src/client/whip.rs index 703dd817ee6..021e672515c 100644 --- a/rs/moq-rtc/src/client/whip.rs +++ b/rs/moq-rtc/src/client/whip.rs @@ -79,10 +79,5 @@ pub(crate) async fn dial( for mid in mids { session.open_media(mid)?; } - tokio::spawn(async move { - let result = session.run().await; - session::log_session_end("whip client", &result); - }); - - Ok(()) + session::end("whip client", session.run().await) } diff --git a/rs/moq-rtc/src/egress.rs b/rs/moq-rtc/src/egress.rs index 6cfb43c632f..2b946329317 100644 --- a/rs/moq-rtc/src/egress.rs +++ b/rs/moq-rtc/src/egress.rs @@ -93,6 +93,8 @@ pub struct EgressSource { /// The latest catalog snapshot, followed for the session's life so each pump serves the /// best enabled rendition. Its sender drops when the catalog ends. catalog: watch::Receiver, + /// Why the catalog stopped, until the session takes it. + catalog_end: Option>>, writes_tx: mpsc::Sender, writes_rx: Option>, } @@ -111,12 +113,13 @@ impl EgressSource { let mut consumer = source.catalog::<()>(moq_mux::catalog::CatalogFormat::Hang).await?; let catalog = consumer.next().await?.ok_or(Error::CatalogClosed)?; let (catalog_tx, catalog) = watch::channel(catalog); - tokio::spawn(follow_catalog(consumer, catalog_tx)); + let catalog_end = tokio::spawn(follow_catalog(consumer, catalog_tx)); let (tx, rx) = mpsc::channel(64); Ok(Self { source, catalog, + catalog_end: Some(catalog_end), writes_tx: tx, writes_rx: Some(rx), }) @@ -128,6 +131,12 @@ impl EgressSource { self.writes_rx.take().expect("EgressSource writes_rx already taken") } + /// One-shot extractor for why the catalog stopped. A failure, such as the source's route + /// dying, ends the session, since nothing more will flow. + pub(crate) fn take_catalog_end(&mut self) -> tokio::task::JoinHandle> { + self.catalog_end.take().expect("EgressSource catalog_end already taken") + } + /// Spawn a pump task for a newly added (sendonly) media line. /// /// `mid` and `formats` come from str0m's negotiated state, `formats` in the peer's order. @@ -184,21 +193,22 @@ impl EgressSource { } } -/// Publish each catalog snapshot until the catalog ends or every pump is gone. -async fn follow_catalog(mut consumer: moq_mux::catalog::Consumer<()>, tx: watch::Sender) { +/// Publish each catalog snapshot until the catalog ends or every pump is gone, failing with the +/// catalog's failure. +async fn follow_catalog(mut consumer: moq_mux::catalog::Consumer<()>, tx: watch::Sender) -> Result<()> { loop { tokio::select! { next = consumer.next() => match next { Ok(Some(catalog)) => { tx.send_replace(catalog); } - Ok(None) => return, + Ok(None) => return Ok(()), Err(err) => { tracing::warn!(%err, "egress catalog failed"); - return; + return Err(err.into()); } }, - _ = tx.closed() => return, + _ = tx.closed() => return Ok(()), } } } @@ -606,6 +616,7 @@ mod tests { let egress = EgressSource { source, catalog: watch::channel(catalog).1, + catalog_end: None, writes_tx, writes_rx: Some(writes_rx), }; diff --git a/rs/moq-rtc/src/lib.rs b/rs/moq-rtc/src/lib.rs index cf590d1a745..eca7f97feb1 100644 --- a/rs/moq-rtc/src/lib.rs +++ b/rs/moq-rtc/src/lib.rs @@ -92,6 +92,208 @@ mod tests { const TIMEOUT: Duration = Duration::from_secs(10); const OPUS_PACKET: &[u8] = &[0xfc, 0xff, 0xfe]; + const OLD_PACKET: &[u8] = &[0xfc, 0x01, 0x02]; + const NEW_PACKET: &[u8] = &[0xfc, 0x99, 0x99]; + + /// Serve `app` on a loopback port. + async fn serve(app: Router) -> (std::net::SocketAddr, tokio::task::JoinHandle<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind HTTP listener"); + let address = listener.local_addr().expect("HTTP listener address"); + let http = tokio::spawn(async move { axum::serve(listener, app).await.expect("serve HTTP") }); + (address, http) + } + + /// Subscribe to the first Opus rendition in `broadcast`'s catalog. + async fn opus_output(broadcast: &moq_net::broadcast::Consumer) -> Track { + let catalog_track = broadcast + .track(hang::Catalog::DEFAULT_NAME) + .expect("output catalog track") + .subscribe(hang::Catalog::default_subscription()) + .await + .expect("subscribe to output catalog"); + let mut catalogs = moq_mux::catalog::hang::Consumer::<()>::new(catalog_track); + let catalog = tokio::time::timeout(TIMEOUT, catalogs.next()) + .await + .expect("output catalog timed out") + .expect("read output catalog") + .expect("output catalog ended"); + let (track_name, config) = catalog.audio.renditions.iter().next().expect("output Opus rendition"); + let track = broadcast + .track(track_name) + .expect("output Opus track") + .subscribe(None) + .await + .expect("subscribe to output Opus track"); + Track::opus(track, config).expect("output Opus container") + } + + async fn next_payload(track: &mut Track) -> Vec { + let frame = tokio::time::timeout(TIMEOUT, track.next()) + .await + .expect("output Opus packet timed out") + .expect("read output Opus packet") + .expect("output Opus track ended"); + frame.payload.to_vec() + } + + fn push(bridge: &mut dyn Bridge, timestamp_us: u64, payload: &'static [u8]) { + bridge + .push(Frame { + timestamp_us, + payload: Bytes::from_static(payload), + }) + .expect("publish source packet"); + } + + /// Publish an Opus instance of `source` under `epoch`, keeping its catalog so the test can + /// kill the instance. + fn publish_opus( + origin: &moq_net::origin::Producer, + epoch: moq_net::Epoch, + ) -> (Box, moq_mux::catalog::Producer) { + let mut broadcast = origin.create_broadcast("source").expect("create source broadcast"); + broadcast + .announce(moq_net::origin::Route::default().with_epoch(epoch)) + .expect("announce source broadcast"); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()) + .expect("create source catalog"); + let bridge = crate::codec::opus::Bridge::new(broadcast, catalog.clone(), 48_000, 2).expect("create Opus bridge"); + (Box::new(bridge), catalog) + } + + /// Publish a replacement instance of `source`, returning once the origin serves it. + async fn take_over(origin: &moq_net::origin::Producer) -> (Box, moq_mux::catalog::Producer) { + let epoch = moq_net::Epoch::mint(); + let mut instances = moq_mux::follow::Instances::new(&origin.consume(), "source", Some(&epoch)).unwrap(); + let (mut new, catalog) = publish_opus(origin, epoch); + push(&mut *new, 20_000, NEW_PACKET); + let started = tokio::time::timeout(TIMEOUT, instances.next()) + .await + .expect("the replacement never took over") + .unwrap(); + assert!(matches!(started, moq_mux::follow::Event::Start(_))); + (new, catalog) + } + + /// Kill an instance the way a dropped publisher session does. + fn kill(bridge: Box, catalog: moq_mux::catalog::Producer) { + let err = moq_net::Error::Transport("connection lost".into()); + bridge.abort(err.clone()); + catalog.abort(err); + } + + /// A WHEP viewer stays on the instance its session resolved: a publisher taking over the + /// path never reaches it, so the session never resets its timestamps or codec state, and it + /// fails once that instance's route dies. + #[tokio::test] + async fn a_whep_viewer_stays_on_its_instance_until_the_route_dies() { + let origin = moq_tokio::origin::spawn(); + let (mut old, old_catalog) = publish_opus(&origin, moq_net::Epoch::mint()); + push(&mut *old, 20_000, OPUS_PACKET); + + let server = Server::new(server::Config::default()); + let whep_origin = origin.consume(); + let (sessions_tx, mut sessions) = tokio::sync::mpsc::channel(1); + let app = Router::new().route( + "/whep/source", + axum::routing::post(move |offer: String| { + let server = server.clone(); + let origin = whep_origin.clone(); + let sessions = sessions_tx.clone(); + async move { + let response = server::whep::accept(&server, &origin, "source", &offer).await.unwrap(); + let answer = response.answer.clone(); + sessions.send(tokio::spawn(response.run())).await.unwrap(); + answer + } + }), + ); + let (address, http) = serve(app).await; + + let output_origin = moq_tokio::origin::spawn(); + let output = output_origin + .create_broadcast("output") + .expect("create output broadcast"); + let output_consumer = output.consume(); + let client = Client::new(client::Config::default()); + let url = format!("http://{address}/whep/source").parse().expect("WHEP URL"); + let viewer = tokio::spawn(async move { client.subscribe(url, output).await }); + let session = tokio::time::timeout(TIMEOUT, sessions.recv()) + .await + .expect("WHEP negotiation timed out") + .unwrap(); + let mut played = opus_output(&output_consumer).await; + assert_eq!(next_payload(&mut played).await, OPUS_PACKET); + + let _new = take_over(&origin).await; + push(&mut *old, 40_000, OLD_PACKET); + assert_eq!(next_payload(&mut played).await, OLD_PACKET, "the viewer left its instance"); + + kill(old, old_catalog); + let ended = tokio::time::timeout(TIMEOUT, session) + .await + .expect("the session outlived its route") + .unwrap(); + assert!(ended.is_err(), "the session ended cleanly"); + viewer.abort(); + http.abort(); + } + + /// A WHIP push stays on the instance it resolved the same way, and + /// [`Client::publish`] fails once that instance's route dies. + #[tokio::test] + async fn a_whip_push_stays_on_its_instance_until_the_route_dies() { + let origin = moq_tokio::origin::spawn(); + let (mut old, old_catalog) = publish_opus(&origin, moq_net::Epoch::mint()); + push(&mut *old, 20_000, OPUS_PACKET); + + let server = Server::new(server::Config::default()); + let ingest_origin = moq_tokio::origin::spawn(); + let whip_origin = ingest_origin.clone(); + let app = Router::new().route( + "/whip/ingested", + axum::routing::post(move |offer: String| { + let server = server.clone(); + let origin = whip_origin.clone(); + async move { + let response = server::whip::accept(&server, &origin, "ingested", &offer) + .await + .unwrap(); + let answer = response.answer.clone(); + tokio::spawn(response.run()); + answer + } + }), + ); + let (address, http) = serve(app).await; + + let client = Client::new(client::Config::default()); + let url = format!("http://{address}/whip/ingested").parse().expect("WHIP URL"); + let source = origin.consume(); + let pushed = tokio::spawn(async move { client.publish(url, source, "source").await }); + let ingested = ingest_origin.consume(); + tokio::time::timeout(TIMEOUT, ingested.routed("ingested")) + .await + .expect("WHIP negotiation timed out") + .expect("the ingest origin closed"); + let ingested = ingested.request_broadcast("ingested", None).await.unwrap(); + let mut played = opus_output(&ingested).await; + assert_eq!(next_payload(&mut played).await, OPUS_PACKET); + + let _new = take_over(&origin).await; + push(&mut *old, 40_000, OLD_PACKET); + assert_eq!(next_payload(&mut played).await, OLD_PACKET, "the push left its instance"); + + kill(old, old_catalog); + let pushed = tokio::time::timeout(TIMEOUT, pushed) + .await + .expect("the push outlived its route") + .unwrap(); + assert!(pushed.is_err(), "the push ended cleanly"); + http.abort(); + } #[tokio::test] #[tracing_test::traced_test] @@ -169,21 +371,22 @@ mod tests { } }), ); - let listener = tokio::net::TcpListener::bind("127.0.0.1:0") - .await - .expect("bind HTTP listener"); - let address = listener.local_addr().expect("HTTP listener address"); - let http = tokio::spawn(async move { axum::serve(listener, app).await.expect("serve HTTP") }); + let (address, http) = serve(app).await; let client = Client::new(client::Config::default()); // Credentials in the dialed URLs must never reach the client's connect logs. let whip = format!("http://user:pass@{address}/whip/ingested?jwt=secret") .parse() .expect("WHIP URL"); - tokio::time::timeout(TIMEOUT, client.publish(whip, source_consumer, "source")) + // Each client call resolves once its session ends, so the test drives them aside. + let whip_client = tokio::spawn({ + let client = client.clone(); + async move { client.publish(whip, source_consumer, "source").await } + }); + let (whip_id, whip_task) = tokio::time::timeout(TIMEOUT, sources_rx.recv()) .await .expect("WHIP negotiation timed out") - .expect("WHIP negotiation failed"); + .unwrap(); let output_origin = moq_tokio::origin::spawn(); let output = output_origin @@ -193,45 +396,20 @@ mod tests { let whep = format!("http://user:pass@{address}/whep/ingested?jwt=secret") .parse() .expect("WHEP URL"); - tokio::time::timeout(TIMEOUT, client.subscribe(whep, output)) + let whep_client = tokio::spawn(async move { client.subscribe(whep, output).await }); + let (whep_id, whep_task) = tokio::time::timeout(TIMEOUT, sources_rx.recv()) .await .expect("WHEP negotiation timed out") - .expect("WHEP negotiation failed"); + .unwrap(); + + let mut opus = opus_output(&output_consumer).await; + assert_eq!(next_payload(&mut opus).await, OPUS_PACKET); assert!(logs_contain("whip client connected")); assert!(logs_contain("whep client connected")); for secret in ["jwt", "secret", "user:pass"] { assert!(!logs_contain(secret), "client logs leaked {secret}"); } - - let catalog_track = output_consumer - .track(hang::Catalog::DEFAULT_NAME) - .expect("output catalog track") - .subscribe(hang::Catalog::default_subscription()) - .await - .expect("subscribe to output catalog"); - let mut catalogs = moq_mux::catalog::hang::Consumer::<()>::new(catalog_track); - let catalog = tokio::time::timeout(TIMEOUT, catalogs.next()) - .await - .expect("output catalog timed out") - .expect("read output catalog") - .expect("output catalog ended"); - let (track_name, config) = catalog.audio.renditions.iter().next().expect("output Opus rendition"); - let track = output_consumer - .track(track_name) - .expect("output Opus track") - .subscribe(None) - .await - .expect("subscribe to output Opus track"); - let mut opus = Track::opus(track, config).expect("output Opus container"); - let frame = tokio::time::timeout(TIMEOUT, opus.next()) - .await - .expect("output Opus packet timed out") - .expect("read output Opus packet") - .expect("output Opus track ended"); - assert_eq!(frame.payload.as_ref(), OPUS_PACKET); - let (whip_id, whip_task) = sources_rx.recv().await.unwrap(); - let (whep_id, whep_task) = sources_rx.recv().await.unwrap(); assert!(server.terminate(&whep_id)); assert!(server.terminate(&whip_id)); tokio::time::timeout(TIMEOUT, whep_task) @@ -260,6 +438,8 @@ mod tests { .unwrap(); assert!(!server.terminate(&whip_id)); assert!(!server.terminate(&whep_id)); + whip_client.abort(); + whep_client.abort(); http.abort(); http.await.unwrap_err(); } diff --git a/rs/moq-rtc/src/server/mod.rs b/rs/moq-rtc/src/server/mod.rs index 470b436a52c..bc94d05d67f 100644 --- a/rs/moq-rtc/src/server/mod.rs +++ b/rs/moq-rtc/src/server/mod.rs @@ -22,7 +22,7 @@ use axum::Router; use axum::http::{HeaderValue, StatusCode, Uri}; use tokio::sync::{OnceCell, oneshot}; -use crate::{Error, Result}; +use crate::Result; use mux::Mux; /// The result of a WHIP/WHEP [`whip::accept`] / [`whep::accept`]: the SDP answer @@ -86,8 +86,7 @@ impl AcceptedSession { } }; if let Some(res) = ended { - crate::session::log_session_end(self.role, &res); - return normalize_session_result(res); + return crate::session::end(self.role, res); } // A DELETE is the peer's clean end after its final media. An ingest serves the @@ -109,15 +108,6 @@ impl Drop for AcceptedSession { } } -/// Fold an ordinary peer disconnect into `Ok` so [`Response::run`] only errors -/// on genuine failures. -fn normalize_session_result(result: Result<()>) -> Result<()> { - match result { - Ok(()) | Err(Error::SessionClosed) => Ok(()), - Err(err) => Err(err), - } -} - /// Build the `Location` header for a negotiated session by appending the /// resource id to the request path, preserving whatever prefix the router is /// mounted under. @@ -316,11 +306,6 @@ mod tests { assert!(!server.terminate(id), "unregistered session can't be terminated"); } - #[test] - fn peer_close_is_a_successful_session_result() { - assert!(normalize_session_result(Err(Error::SessionClosed)).is_ok()); - } - #[cfg(feature = "server")] #[test] fn session_location_preserves_mount_path() { diff --git a/rs/moq-rtc/src/session.rs b/rs/moq-rtc/src/session.rs index 26c5a109ada..affb5458ecc 100644 --- a/rs/moq-rtc/src/session.rs +++ b/rs/moq-rtc/src/session.rs @@ -92,6 +92,9 @@ pub struct Session { /// sessions; pumps send frames here, the main loop forwards them into /// str0m's [`Writer`](str0m::media::Writer). writes_rx: Option>, + /// Why the egress source's catalog stopped. `Some` only for [`MediaRole::Egress`] sessions, + /// until it does. + catalog_end: Option>>, /// Rebases each ingested track's raw RTP timestamps onto one session /// timeline so audio and video stay in sync. Unused by egress sessions. ingest_clock: IngestClock, @@ -109,6 +112,7 @@ impl Session { role: MediaRole::Ingest(sink), opened_media: HashSet::new(), writes_rx: None, + catalog_end: None, ingest_clock: IngestClock::default(), egress_clock: EgressClock::default(), }) @@ -118,11 +122,13 @@ impl Session { /// `inbound` comes from the server mux, or [`moq_ice::reader`] for a 1:1 client socket. pub fn egress(rtc: Rtc, socket: moq_ice::Socket, mut source: EgressSource) -> Result { let writes_rx = source.take_writes(); + let catalog_end = source.take_catalog_end(); Ok(Self { agent: moq_ice::Agent::new(rtc, socket)?, role: MediaRole::Egress(Box::new(source)), opened_media: HashSet::new(), writes_rx: Some(writes_rx), + catalog_end: Some(catalog_end), ingest_clock: IngestClock::default(), egress_clock: EgressClock::default(), }) @@ -172,6 +178,21 @@ impl Session { crate::egress::dispatch(&mut self.agent.rtc_mut(), req, wallclock); } + // A failed catalog, as every subscription fails once the source's route dies, + // means nothing more will flow, so fail rather than idle. A clean end keeps + // serving what the pumps still hold. + ended = async { + match self.catalog_end.as_mut() { + Some(end) => end.await, + None => std::future::pending().await, + } + } => { + self.catalog_end = None; + if let Ok(Err(err)) = ended { + return Err(err); + } + } + event = self.agent.next() => match event { Ok(moq_ice::Event::Rtc(event)) => self.handle_event(event)?, // No STUN servers are configured, so there is nothing to trickle. @@ -422,18 +443,21 @@ fn to_u64(value: i128) -> u64 { value.clamp(0, u64::MAX as i128) as u64 } -/// Log a finished session at the right level: an ordinary peer disconnect -/// ([`Error::SessionClosed`]) is debug, a genuine failure is a warning. Keeps -/// normal WebRTC churn out of the warning stream. `role` labels the path -/// (e.g. `"whip server"`). -pub(crate) fn log_session_end(role: &str, result: &Result<()>) { - match result { +/// Log a finished session at the right level, then fold an ordinary peer disconnect +/// ([`Error::SessionClosed`]) into `Ok` so only genuine failures error. Keeps normal +/// WebRTC churn out of the warning stream. `role` labels the path (e.g. `"whip server"`). +pub(crate) fn end(role: &str, result: Result<()>) -> Result<()> { + match &result { Ok(()) | Err(Error::SessionClosed) => tracing::debug!(role, "session ended"), // An abandoned offer (peer answered but never connected) is normal churn, not // a failure: keep it out of the warning stream. Err(Error::IceTimeout) => tracing::debug!(role, "session ended: ICE never connected"), Err(err) => tracing::warn!(%err, role, "session ended"), } + match result { + Ok(()) | Err(Error::SessionClosed) => Ok(()), + Err(err) => Err(err), + } } /// Convert a str0m [`MediaTime`](str0m::media::MediaTime) to microseconds. diff --git a/rs/moq-rtmp/src/dial.rs b/rs/moq-rtmp/src/dial.rs index 722d4f2d0ad..8af0f12b859 100644 --- a/rs/moq-rtmp/src/dial.rs +++ b/rs/moq-rtmp/src/dial.rs @@ -754,6 +754,96 @@ mod tests { } } + /// Contains `needle` somewhere in `haystack`. + fn contains(haystack: &[u8], needle: &[u8]) -> bool { + haystack.windows(needle.len()).any(|window| window == needle) + } + + /// A push stays on the instance it resolved: a publisher taking over the path never reaches + /// the remote, so the push never splices in another timeline or codec state, and it fails + /// once its instance's route dies. + #[tokio::test(start_paused = true)] + async fn a_push_stays_on_its_instance_until_the_route_dies() { + const OLD_IDR: &[u8] = &[0x65, 0x88, 0x84, 0x21, 0x00]; + const OLD_DELTA: &[u8] = &[0x41, 0x9a, 0x02, 0x03]; + const NEW_IDR: &[u8] = &[0x65, 0x99, 0x99, 0x99, 0x99]; + let publish = |origin: &origin::Producer, epoch: &moq_net::Epoch, idr: &[u8]| { + let mut vseq = vec![0x17, 0x00, 0x00, 0x00, 0x00]; + vseq.extend_from_slice(&[0x01, 0x42, 0xc0, 0x1f, 0xff, 0xe1, 0x00, 0x04, 0x67, 0x42, 0xc0, 0x1f]); + vseq.extend_from_slice(&[0x01, 0x00, 0x04, 0x68, 0xce, 0x3c, 0x80]); + let mut vframe = vec![0x17, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, idr.len() as u8]; + vframe.extend_from_slice(idr); + let mut broadcast = origin.create_broadcast("live/cam0").unwrap(); + broadcast + .announce(origin::Route::default().with_epoch(epoch.clone())) + .unwrap(); + let catalog = + moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let mut importer = FlvImport::new(broadcast, catalog.reserve()); + importer.decode(&flv::file_header()).unwrap(); + importer.decode(&flv::tag(flv::TAG_VIDEO, 0, &vseq)).unwrap(); + importer.decode(&flv::tag(flv::TAG_VIDEO, 0, &vframe)).unwrap(); + (importer, catalog) + }; + + let upstream = moq_tokio::origin::spawn(); + let downstream = moq_tokio::origin::spawn(); + let (mut old, old_catalog) = publish(&upstream, &moq_net::Epoch::mint(), OLD_IDR); + let (client_io, server_io) = tokio::io::duplex(64 * 1024); + let ingest = downstream.clone(); + let server = tokio::spawn(async move { + let Some(Request::Publish(publish)) = accept_stream(server_io, "127.0.0.1:1935".parse().unwrap()) + .await + .unwrap() + else { + panic!("expected a publish request"); + }; + let _ = publish.accept(&ingest, "live/cam0").await; + }); + let exported = upstream.consume(); + let pushed = tokio::spawn(async move { + let client = Client::with_stream(client_io, "live").await.unwrap(); + client.publish("cam0", exported, "live/cam0").await + }); + + // Read what reaches the remote back out as FLV. + let consumer = downstream.consume(); + consumer.routed("live/cam0").await.unwrap(); + let copy = consumer.request_broadcast("live/cam0", None).await.unwrap(); + let source = moq_mux::Source::new(consumer, copy); + let catalog = source + .catalog::<()>(moq_mux::catalog::CatalogFormat::default()) + .await + .unwrap(); + let mut remote = FlvExport::new(source, catalog); + let mut received = Vec::new(); + let mut until = async |needle: &[u8]| { + while !contains(&received, needle) { + let chunk = remote.next().await.unwrap().expect("the remote copy ended"); + received.extend_from_slice(&chunk); + } + }; + until(OLD_IDR).await; + + let epoch = moq_net::Epoch::mint(); + let (_new, _new_catalog) = publish(&upstream, &epoch, NEW_IDR); + let local = upstream.consume(); + while local.request_broadcast("live/cam0", None).await.unwrap().info().epoch.as_ref() != Some(&epoch) { + tokio::time::sleep(Duration::from_millis(10)).await; + } + + let mut delta = vec![0x27, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, OLD_DELTA.len() as u8]; + delta.extend_from_slice(OLD_DELTA); + old.decode(&flv::tag(flv::TAG_VIDEO, 40, &delta)).unwrap(); + until(OLD_DELTA).await; + assert!(!contains(&received, NEW_IDR), "the push left its instance"); + + old.abort(moq_net::Error::Transport("connection lost".into())); + drop(old_catalog); + assert!(pushed.await.unwrap().is_err(), "the push ended cleanly"); + server.abort(); + } + /// Loopback: dial the crate's own server and pull a broadcast the server plays /// out of a populated origin. Exercises the client handshake, connect, play /// request, and the receive -> FLV-demux -> republish path end to end. diff --git a/rs/moq-rtmp/src/server.rs b/rs/moq-rtmp/src/server.rs index 6458a6ee976..ef34e83a573 100644 --- a/rs/moq-rtmp/src/server.rs +++ b/rs/moq-rtmp/src/server.rs @@ -1843,14 +1843,35 @@ mod tests { } } + /// Drive an RTMP play client over `stream`, collecting media messages until `want` have + /// arrived or the server hangs up. + async fn play_client_collect(stream: S, want: usize, caps_ex: u32, fourccs: &[&[u8; 4]]) -> Vec { + let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel(); + let client = play_client(stream, caps_ex, fourccs, tx); + tokio::pin!(client); + let mut media = Vec::new(); + while media.len() < want { + tokio::select! { + Some(next) = rx.recv() => media.push(next), + () = &mut client => break, + } + } + while media.len() < want + && let Ok(next) = rx.try_recv() + { + media.push(next); + } + media + } + /// Drive an RTMP play client over `stream` through handshake -> connect(`live`) - /// -> play(`cam0`), collecting media messages until `want` have arrived. - async fn play_client_collect( + /// -> play(`cam0`), sending each media message to `media` until the server hangs up. + async fn play_client( mut stream: S, - want: usize, caps_ex: u32, fourccs: &[&[u8; 4]], - ) -> Vec { + media: tokio::sync::mpsc::UnboundedSender, + ) { // Handshake. let mut handshake = Handshake::new(PeerType::Client); stream @@ -1886,7 +1907,6 @@ mod tests { set_connect_capabilities(&mut session, caps_ex, fourccs); work.push_back(session.request_connection("live".to_string()).unwrap()); - let mut media = Vec::new(); loop { while let Some(result) = work.pop_front() { match result { @@ -1897,21 +1917,18 @@ mod tests { work.push_back(session.request_playback("cam0".to_string()).unwrap()); } ClientSessionResult::RaisedEvent(ClientSessionEvent::VideoDataReceived { data, .. }) => { - media.push((true, data)); + let _ = media.send((true, data)); } ClientSessionResult::RaisedEvent(ClientSessionEvent::AudioDataReceived { data, .. }) => { - media.push((false, data)); + let _ = media.send((false, data)); } _ => {} } } - if media.len() >= want { - return media; - } - let n = stream.read(&mut buffer).await.unwrap(); - if n == 0 { - return media; - } + let n = match stream.read(&mut buffer).await { + Ok(0) | Err(_) => return, + Ok(n) => n, + }; work.extend(session.handle_input(&buffer[..n]).unwrap()); } } @@ -2104,6 +2121,89 @@ mod tests { server_task.abort(); } + /// Publish an AVC broadcast at `live/cam0` under `epoch`, starting with a keyframe carrying + /// `idr`, and return its importer for more tags. + fn publish_avc( + origin: &origin::Producer, + epoch: moq_net::Epoch, + idr: &[u8], + ) -> (FlvImport, moq_mux::catalog::Producer) { + let mut vseq = vec![0x17, 0x00, 0x00, 0x00, 0x00]; + vseq.extend_from_slice(&[0x01, 0x42, 0xc0, 0x1f, 0xff, 0xe1, 0x00, 0x04, 0x67, 0x42, 0xc0, 0x1f]); + vseq.extend_from_slice(&[0x01, 0x00, 0x04, 0x68, 0xce, 0x3c, 0x80]); + let mut vframe = vec![0x17, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, idr.len() as u8]; + vframe.extend_from_slice(idr); + + let mut broadcast = origin.create_broadcast("live/cam0").unwrap(); + broadcast + .announce(origin::Route::default().with_epoch(epoch)) + .unwrap(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let mut importer = FlvImport::new(broadcast, catalog.reserve()); + importer.decode(&flv::file_header()).unwrap(); + importer.decode(&flv::tag(flv::TAG_VIDEO, 0, &vseq)).unwrap(); + importer.decode(&flv::tag(flv::TAG_VIDEO, 0, &vframe)).unwrap(); + (importer, catalog) + } + + /// A play stays on the instance it resolved: a publisher taking over the path never reaches + /// it, so the session never splices in another timeline or codec state, and the play fails + /// once its instance's route dies. + #[tokio::test] + async fn a_play_stays_on_its_instance_until_the_route_dies() { + const OLD_DELTA: &[u8] = &[0x41, 0x9a, 0x02, 0x03]; + let origin = moq_tokio::origin::spawn(); + let consumer = origin.consume(); + let (mut old, old_catalog) = publish_avc(&origin, moq_net::Epoch::mint(), &[0x65, 0x88, 0x84, 0x21, 0x00]); + + let mut server = Server::bind("127.0.0.1:0".parse().unwrap()).await.unwrap(); + let addr = server.local_addr().unwrap(); + let play_origin = consumer.clone(); + let played = tokio::spawn(async move { + let Some(Request::Play(play)) = server.accept().await else { + panic!("expected a play request"); + }; + play.accept(&play_origin, "live/cam0").await + }); + let (tx, mut media) = tokio::sync::mpsc::unbounded_channel(); + let client = tokio::spawn(play_client(TcpStream::connect(addr).await.unwrap(), 0, &[], tx)); + let mut next = async || { + tokio::time::timeout(Duration::from_secs(5), media.recv()) + .await + .expect("no media") + .expect("the play ended") + }; + // The sequence header, then the keyframe. + assert_eq!(next().await.1[..2], [0x17, 0x00]); + assert_eq!(next().await.1[..2], [0x17, 0x01]); + + let epoch = moq_net::Epoch::mint(); + let (_new, _new_catalog) = publish_avc(&origin, epoch.clone(), &[0x65, 0x99, 0x99, 0x99, 0x99]); + tokio::time::timeout(Duration::from_secs(5), async { + while consumer.request_broadcast("live/cam0", None).await.unwrap().info().epoch.as_ref() != Some(&epoch) { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("the new publisher takes over"); + + let mut delta = vec![0x27, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, OLD_DELTA.len() as u8]; + delta.extend_from_slice(OLD_DELTA); + old.decode(&flv::tag(flv::TAG_VIDEO, 40, &delta)).unwrap(); + let (video, body) = next().await; + assert!(video && body.starts_with(&[0x27, 0x01]), "{body:?}"); + assert!(body.ends_with(OLD_DELTA), "the play left its instance: {body:?}"); + + old.abort(moq_net::Error::Transport("connection lost".into())); + drop(old_catalog); + let played = tokio::time::timeout(Duration::from_secs(5), played) + .await + .expect("the play outlived its route") + .unwrap(); + assert!(played.is_err(), "the play ended cleanly"); + client.abort(); + } + #[tokio::test] async fn play_enhanced_codec_rejects_legacy_client() { const VP9_KEYFRAME_320X240: &[u8] = &[0x82, 0x49, 0x83, 0x42, 0x20, 0x13, 0xf0, 0x0e, 0xf0, 0x00]; From b90d4c4ee6385f2e3062c54238e9a5c7151d2e0f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 10:04:50 -0700 Subject: [PATCH 4/7] quest: finish readers follow restarts Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 - quest/m1/api-audit/README.md | 1 - quest/m1/api-audit/resolve.md | 1 - quest/m1/broadcast-lifecycle/ts-follow.md | 5 +- quest/m1/rendition-move-follow.md | 4 +- quest/m1/sink-restart-follow.md | 64 ----------------------- 6 files changed, 4 insertions(+), 72 deletions(-) delete mode 100644 quest/m1/sink-restart-follow.md diff --git a/quest/m1/README.md b/quest/m1/README.md index b94fb4ecc84..ac4aef0a4d9 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -92,7 +92,6 @@ audit's receive-memory, priority, and lease bugs rank ahead of that reshape. - [JS qmux stream credit](/quest/m1/js-qmux-credit.md) - credit returns when both halves close, so `@moq/net` drops its caps - [Every close records its cause](/quest/m1/close-cause.md) - protocol-violation closes and abandoned SETUP replies record why - [Empty segments refused](/quest/m1/empty-segments.md) - `room//cam` fails loud instead of naming `room/cam` -- [Readers follow restarts](/quest/m1/sink-restart-follow.md) - every moq-cli and moq-gst reader, moqsrc and `moq play` included, follows instances through `announced_broadcast` and an epochless `Restart` where its output can mark it; otherwise it stays on the old instance and fails once that route dies - [Readers follow a moved rendition](/quest/m1/rendition-move-follow.md) - moq-rtc, moq-ffi, moq-gst and `moq play` follow a rendition the catalog moves to a new track - [In-band auth](/quest/m1/auth/README.md) - a session tells its peer what it may publish and subscribe to, unions tokens presented in band, and fails loud on an out-of-scope publish - [Cluster routing](/quest/m1/cluster-routing/README.md) - prove shared routing contracts, implement per-origin routes, and connect outbound local clusters diff --git a/quest/m1/api-audit/README.md b/quest/m1/api-audit/README.md index 0efc4194a40..991b796d02a 100644 --- a/quest/m1/api-audit/README.md +++ b/quest/m1/api-audit/README.md @@ -89,6 +89,5 @@ Rejected in planning: ## Related - [Claim request type](/quest/m1/api-audit/claim-request-type.md) - removes `accept` from application requests; [the origin announces](/quest/m1/api-audit/announce-owner.md) narrows `Prepared` beside it -- [Readers follow restarts](/quest/m1/sink-restart-follow.md) - moves readers onto the follow operation that [two broadcast resolutions](/quest/m1/api-audit/resolve.md) names - [API-diff check on release PRs](/quest/m1/api-diff.md) - the backstop that catches breaks this epic misses - [Main is no slower than release](/quest/m1/release-perf/README.md) - the other release-readiness check diff --git a/quest/m1/api-audit/resolve.md b/quest/m1/api-audit/resolve.md index b9daee282da..7cce21a65ce 100644 --- a/quest/m1/api-audit/resolve.md +++ b/quest/m1/api-audit/resolve.md @@ -72,5 +72,4 @@ Wire: none. ## Related -- [Readers follow restarts](/quest/m1/sink-restart-follow.md) - moves readers to `follow`; whichever lands second adopts the other's names - [Origin fronts park](/quest/m2/origin-front-parks.md) - touches the same retry diff --git a/quest/m1/broadcast-lifecycle/ts-follow.md b/quest/m1/broadcast-lifecycle/ts-follow.md index 7df7f4e6179..421e557fa21 100644 --- a/quest/m1/broadcast-lifecycle/ts-follow.md +++ b/quest/m1/broadcast-lifecycle/ts-follow.md @@ -28,11 +28,10 @@ Decided with the maintainer on 2026-10-23: (`rs/moq-cli/src/args.rs`) keeps refusing; point its message at `--follow` only where following applies. - Move the follower onto `moq_mux::follow::Instances` here (decided - 2026-10-24), as [readers follow restarts](/quest/m1/sink-restart-follow.md) - did for `moq play`, `moqsrc`, and `transcode`. `with_linger` waits from the + 2026-10-24), as `moq play`, `moqsrc`, and `transcode` already do. `with_linger` waits from the announcement through resolution and the first catalog, but `Instances` yields `Start` only after resolution, so settle what `--follow` bounds - before moving the timer onto it. Coordinate the name with that quest. + before moving the timer onto it. - Docs: `doc/bin/cli.md`, plus a migration note for moq-cli, moq-mux, and moq-srt in the Impact bullet. diff --git a/quest/m1/rendition-move-follow.md b/quest/m1/rendition-move-follow.md index 77723f728f7..7bf0afb465b 100644 --- a/quest/m1/rendition-move-follow.md +++ b/quest/m1/rendition-move-follow.md @@ -14,8 +14,8 @@ consumers, moq-gst's source, and `moq play` are candidates to check. Audit each, make the ones that resolve once follow catalog updates, and test a move mid-playback for each. -[Readers follow restarts](/quest/m1/sink-restart-follow.md) covers broadcast -restarts, not track moves inside one broadcast. +Broadcast restarts are out of scope: readers already follow those through +`moq_mux::follow::Instances`, or stay on the old instance until its route dies. Public API: none expected. Wire: none. diff --git a/quest/m1/sink-restart-follow.md b/quest/m1/sink-restart-follow.md deleted file mode 100644 index 27e1e97828d..00000000000 --- a/quest/m1/sink-restart-follow.md +++ /dev/null @@ -1,64 +0,0 @@ -# [L] moq-cli and moq-gst readers follow a broadcast's restarts - -## Goal - -Every moq-cli and moq-gst reader that resolves a broadcast and keeps reading -it follows publisher instances through moq-net's -`origin::Consumer::announced_broadcast` -([#5925](https://github.com/moq-dev/moq/pull/5925)) instead of a hand-rolled -`origin.follow` loop. Each handles an epochless -`Restart` on purpose: it follows the new instance where its output can mark -the break. A route change alone is never fatal: a reader that cannot follow -stays on the old instance while its publisher serves it, and fails once that -route dies, since its subscriptions then end with an error. - -## Plan - -Decided with the maintainer on 2026-10-20: widened from the server and push -sinks to every moq-cli and moq-gst reader. Following a catalog's references -to other broadcasts landed for `moqsrc` through `moq_mux::Source::follow`; -`moq play` resolves a reference once with `Source::resolve`, which waits for -a sibling's first announcement and keeps it. - -`moq play`, `moqsrc`, and `transcode` follow through -`moq_mux::follow::Instances::new(origin, path, None)` (decided 2026-10-20, -maintainer: no reader hand-rolls its own follow loop). Each -`follow::Event::Start` builds a `moq_mux::Source` from the instance it hands -out, and a selected handler's refusal is terminal. `transcode` retires the old -ladder and publishes a fresh derivative broadcast with its own epoch for each -source instance; a retired ladder aborts its catalog and rungs with `Cancel`. - -A finite read such as `fetch`, every stdout export except TS with `--stitch`, -and TS without `--stitch` stay pinned to the instance they resolved until its -route dies and its tracks error. Never concatenate independent instances into -one stream. - -Decided with the maintainer on 2026-10-24, when the play, moqsrc, and -transcode moves landed: - -- The TS follower moves onto `Instances` in - [TS export follows by cause](/quest/m1/broadcast-lifecycle/ts-follow.md), - which reworks the same wait, so its timers change once. -- `export hls` gives each instance its own playlist URL. While the old - publisher stays live, `Server::broadcaster` (`rs/moq-hls/src/server/mod.rs`) - answers new requests from its cached broadcaster, so an overlapping - replacement never reaches them; swapping the cache alone would reset the - source-derived `EXT-X-MEDIA-SEQUENCE` at a stable URL. Rejected: one - continuous playlist with monotonic media and discontinuity sequences. A - failed timeline keeps answering its playlists with a no-store 5xx. -- `export rtmp` and `export rtc`, both `--listen` and `--connect`, stay on the - old instance until its route dies, never silently resetting timestamps or - codec state. Rejected for now: reconnecting the output per instance, and - marking the break in-session. - -Remaining, each with a test that replaces the publisher while the reader runs: - -- `export hls`: a playlist URL per instance. -- `export rtmp` and `export rtc`: confirm both sides stay pinned and fail once - the old route dies. - -Public API: none. Wire: none. - -## Related - -- [Finite drain for a replaced run's HLS recording](/quest/m2/hls-restart-drain-cutoff.md) - the recording side of an HLS restart From fdab42d96535819c7487bada7bdbef95ad0b4bb7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 10:16:27 -0700 Subject: [PATCH 5/7] test(rtmp): wait for a takeover through Instances, and format Co-Authored-By: Claude Opus 5.5 --- rs/moq-hls/src/server/mod.rs | 5 ++++- rs/moq-hls/src/server/routes.rs | 38 +++++++++++++++++++++++++++------ rs/moq-rtc/src/lib.rs | 15 ++++++++++--- rs/moq-rtmp/src/dial.rs | 12 +++++------ rs/moq-rtmp/src/server.rs | 18 ++++++---------- 5 files changed, 61 insertions(+), 27 deletions(-) diff --git a/rs/moq-hls/src/server/mod.rs b/rs/moq-hls/src/server/mod.rs index 03beb0763f3..c504e3195b9 100644 --- a/rs/moq-hls/src/server/mod.rs +++ b/rs/moq-hls/src/server/mod.rs @@ -508,7 +508,10 @@ mod tests { }) .await; let kept = server.instance("live", old_epoch.as_str()).await; - assert!(kept.is_some_and(|kept| Arc::ptr_eq(&kept, &first)), "the old instance serves"); + assert!( + kept.is_some_and(|kept| Arc::ptr_eq(&kept, &first)), + "the old instance serves" + ); drop(old); eventually(async || server.instance("live", old_epoch.as_str()).await.is_none()).await; diff --git a/rs/moq-hls/src/server/routes.rs b/rs/moq-hls/src/server/routes.rs index 708ea728275..16697e3dd9e 100644 --- a/rs/moq-hls/src/server/routes.rs +++ b/rs/moq-hls/src/server/routes.rs @@ -993,7 +993,10 @@ mod tests { drop((old_catalog, old_track, old_media, old)); let deadline = tokio::time::Instant::now() + TIMEOUT; while status(&app, &old_playlist).await != StatusCode::NOT_FOUND { - assert!(tokio::time::Instant::now() < deadline, "the closed instance kept serving"); + assert!( + tokio::time::Instant::now() < deadline, + "the closed instance kept serving" + ); tokio::time::sleep(Duration::from_millis(50)).await; } assert_eq!(status(&app, &old_segment).await, StatusCode::NOT_FOUND); @@ -1074,7 +1077,12 @@ mod tests { let server = Server::new(origin.consume(), crate::export::Config::default()); let app = server.router(); - wait_listed(&app, &format!("/live/@{RUN}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; + wait_listed( + &app, + &format!("/live/@{RUN}/video/video0/media.m3u8"), + "seg/656e3d1b.0.m4s", + ) + .await; let broadcaster = server.broadcaster("live").await.expect("broadcaster"); broadcaster.set_generation(Some("run-1")).unwrap(); @@ -1093,7 +1101,10 @@ mod tests { status(&app, &format!("/live/@{RUN}/video/video0/{init}")).await, StatusCode::OK ); - assert_eq!(status(&app, &format!("/live/@{RUN}/video/video0/init.mp4")).await, StatusCode::NOT_FOUND); + assert_eq!( + status(&app, &format!("/live/@{RUN}/video/video0/init.mp4")).await, + StatusCode::NOT_FOUND + ); assert_eq!( status(&app, &format!("/live/@{RUN}/video/video0/init.0000000000000000.mp4")).await, StatusCode::NOT_FOUND @@ -1142,7 +1153,12 @@ mod tests { let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); let run = listed(&server, "live").await; let app = server.router(); - wait_listed(&app, &format!("/live/@{run}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; + wait_listed( + &app, + &format!("/live/@{run}/video/video0/media.m3u8"), + "seg/656e3d1b.0.m4s", + ) + .await; let response = oneshot(app, &format!("/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; assert_eq!(response.status(), StatusCode::NOT_FOUND); @@ -1162,7 +1178,12 @@ mod tests { let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); let run = listed(&server, "live").await; let app = server.router(); - wait_listed(&app, &format!("/live/@{run}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; + wait_listed( + &app, + &format!("/live/@{run}/video/video0/media.m3u8"), + "seg/656e3d1b.0.m4s", + ) + .await; let remote = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().request_broadcast("live", None)) .await @@ -1388,7 +1409,12 @@ mod tests { let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); let run = listed(&server, "room/live").await; let app = server.router(); - wait_listed(&app, &format!("/room/live/@{run}/video/video0/media.m3u8"), "seg/656e3d1b.0.m4s").await; + wait_listed( + &app, + &format!("/room/live/@{run}/video/video0/media.m3u8"), + "seg/656e3d1b.0.m4s", + ) + .await; // Waited on rather than requested once: a client only asks about what it watches. let remote_media = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().routed_broadcast("room/source")) diff --git a/rs/moq-rtc/src/lib.rs b/rs/moq-rtc/src/lib.rs index eca7f97feb1..d0553e5f6ae 100644 --- a/rs/moq-rtc/src/lib.rs +++ b/rs/moq-rtc/src/lib.rs @@ -159,7 +159,8 @@ mod tests { .expect("announce source broadcast"); let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()) .expect("create source catalog"); - let bridge = crate::codec::opus::Bridge::new(broadcast, catalog.clone(), 48_000, 2).expect("create Opus bridge"); + let bridge = + crate::codec::opus::Bridge::new(broadcast, catalog.clone(), 48_000, 2).expect("create Opus bridge"); (Box::new(bridge), catalog) } @@ -229,7 +230,11 @@ mod tests { let _new = take_over(&origin).await; push(&mut *old, 40_000, OLD_PACKET); - assert_eq!(next_payload(&mut played).await, OLD_PACKET, "the viewer left its instance"); + assert_eq!( + next_payload(&mut played).await, + OLD_PACKET, + "the viewer left its instance" + ); kill(old, old_catalog); let ended = tokio::time::timeout(TIMEOUT, session) @@ -284,7 +289,11 @@ mod tests { let _new = take_over(&origin).await; push(&mut *old, 40_000, OLD_PACKET); - assert_eq!(next_payload(&mut played).await, OLD_PACKET, "the push left its instance"); + assert_eq!( + next_payload(&mut played).await, + OLD_PACKET, + "the push left its instance" + ); kill(old, old_catalog); let pushed = tokio::time::timeout(TIMEOUT, pushed) diff --git a/rs/moq-rtmp/src/dial.rs b/rs/moq-rtmp/src/dial.rs index 8af0f12b859..8533c2a3a94 100644 --- a/rs/moq-rtmp/src/dial.rs +++ b/rs/moq-rtmp/src/dial.rs @@ -777,8 +777,7 @@ mod tests { broadcast .announce(origin::Route::default().with_epoch(epoch.clone())) .unwrap(); - let catalog = - moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); let mut importer = FlvImport::new(broadcast, catalog.reserve()); importer.decode(&flv::file_header()).unwrap(); importer.decode(&flv::tag(flv::TAG_VIDEO, 0, &vseq)).unwrap(); @@ -826,11 +825,12 @@ mod tests { until(OLD_IDR).await; let epoch = moq_net::Epoch::mint(); + let mut instances = moq_mux::follow::Instances::new(&upstream.consume(), "live/cam0", Some(&epoch)).unwrap(); let (_new, _new_catalog) = publish(&upstream, &epoch, NEW_IDR); - let local = upstream.consume(); - while local.request_broadcast("live/cam0", None).await.unwrap().info().epoch.as_ref() != Some(&epoch) { - tokio::time::sleep(Duration::from_millis(10)).await; - } + tokio::time::timeout(Duration::from_secs(5), instances.next()) + .await + .expect("the new publisher takes over") + .unwrap(); let mut delta = vec![0x27, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, OLD_DELTA.len() as u8]; delta.extend_from_slice(OLD_DELTA); diff --git a/rs/moq-rtmp/src/server.rs b/rs/moq-rtmp/src/server.rs index ef34e83a573..381a21920a7 100644 --- a/rs/moq-rtmp/src/server.rs +++ b/rs/moq-rtmp/src/server.rs @@ -2135,9 +2135,7 @@ mod tests { vframe.extend_from_slice(idr); let mut broadcast = origin.create_broadcast("live/cam0").unwrap(); - broadcast - .announce(origin::Route::default().with_epoch(epoch)) - .unwrap(); + broadcast.announce(origin::Route::default().with_epoch(epoch)).unwrap(); let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); let mut importer = FlvImport::new(broadcast, catalog.reserve()); importer.decode(&flv::file_header()).unwrap(); @@ -2178,14 +2176,12 @@ mod tests { assert_eq!(next().await.1[..2], [0x17, 0x01]); let epoch = moq_net::Epoch::mint(); - let (_new, _new_catalog) = publish_avc(&origin, epoch.clone(), &[0x65, 0x99, 0x99, 0x99, 0x99]); - tokio::time::timeout(Duration::from_secs(5), async { - while consumer.request_broadcast("live/cam0", None).await.unwrap().info().epoch.as_ref() != Some(&epoch) { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .expect("the new publisher takes over"); + let mut instances = moq_mux::follow::Instances::new(&consumer, "live/cam0", Some(&epoch)).unwrap(); + let (_new, _new_catalog) = publish_avc(&origin, epoch, &[0x65, 0x99, 0x99, 0x99, 0x99]); + tokio::time::timeout(Duration::from_secs(5), instances.next()) + .await + .expect("the new publisher takes over") + .unwrap(); let mut delta = vec![0x27, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, OLD_DELTA.len() as u8]; delta.extend_from_slice(OLD_DELTA); From aa4dc820e0d1386288cc5243be6fa36b6d4b51b9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 11:42:15 -0700 Subject: [PATCH 6/7] fix(hls): keep open replaced instances when the listed one closes A request that finds the listed instance closed before the follower does resolved the path afresh and dropped every replaced instance still open. An epochless one gets a new id when resolved again, so a player on it lost its URLs for good. The fresh entry now adopts them. Co-Authored-By: Claude Opus 5.5 --- rs/moq-hls/src/server/mod.rs | 74 +++++++++++++++++++++++++++--------- 1 file changed, 56 insertions(+), 18 deletions(-) diff --git a/rs/moq-hls/src/server/mod.rs b/rs/moq-hls/src/server/mod.rs index c504e3195b9..4293e7b2de1 100644 --- a/rs/moq-hls/src/server/mod.rs +++ b/rs/moq-hls/src/server/mod.rs @@ -135,6 +135,17 @@ impl Followed { instances.current = broadcaster; } + /// Keep serving every instance of `replaced` still open, unless this one lists its id. + fn adopt(&self, replaced: &Followed) { + let replaced = replaced.instances.lock().unwrap(); + let mut instances = self.instances.lock().unwrap(); + for (id, broadcaster) in &replaced.by_id { + if !broadcaster.is_closed() { + instances.by_id.entry(id.clone()).or_insert_with(|| broadcaster.clone()); + } + } + } + /// Forget every instance whose broadcast closed, registering `waiter` on the rest. fn forget_closed(&self, waiter: &kio::Waiter) { let mut instances = self.instances.lock().unwrap(); @@ -187,8 +198,12 @@ impl Server { /// Follow `name` from the instance serving it now, unless it is followed already. async fn followed(&self, name: &str) -> Option> { - if let Some(followed) = self.cached(name) { - return Some(followed); + // Once the listed instance closes, its follower is about to move on or forget the path, so + // resolve whatever serves the path now instead. + if let Some(followed) = self.inner.broadcasts.lock().unwrap().get(name) + && !followed.current().is_closed() + { + return Some(followed.clone()); } // Confirm a route covers the broadcast (and it is in scope) before resolving it. The @@ -226,10 +241,13 @@ impl Server { { let mut broadcasts = self.inner.broadcasts.lock().unwrap(); - if let Some(existing) = broadcasts.get(name) - && !existing.current().is_closed() - { - return Some(existing.clone()); + if let Some(existing) = broadcasts.get(name) { + if !existing.current().is_closed() { + return Some(existing.clone()); + } + // A player may still be on a replaced instance, and an epochless one would not get + // its id back. + followed.adopt(existing); } broadcasts.insert(name.to_string(), followed.clone()); } @@ -243,18 +261,6 @@ impl Server { *followed.task.lock().unwrap() = Some(task); Some(followed) } - - /// The followed `name`, unless its current instance closed. Its follower is about to forget - /// it then, so resolve whatever serves the path now instead. - fn cached(&self, name: &str) -> Option> { - let mut broadcasts = self.inner.broadcasts.lock().unwrap(); - let followed = broadcasts.get(name)?; - if followed.current().is_closed() { - broadcasts.remove(name); - return None; - } - Some(followed.clone()) - } } impl Inner { @@ -520,4 +526,36 @@ mod tests { drop(new); eventually(async || server.inner.broadcasts.lock().unwrap().is_empty()).await; } + + /// A request that finds the listed instance closed before the follower does resolves the path + /// afresh, yet a replaced instance still open keeps its URLs. An epochless one gets a new id + /// when resolved again, so its old URLs would never serve otherwise. + #[tokio::test] + async fn a_closed_current_instance_keeps_the_open_ones() { + let origin = produce_origin(); + let server = Server::new(origin.consume(), Config::default()); + let _old = origin + .publish("live", moq_net::origin::Route::default()) + .expect("publish allowed"); + settle().await; + let open = Broadcaster::new(crate::resolved(&origin, "live").await, Config::default()); + + // The follower has not yet seen the takeover instance close. + let followed = Followed::new(open.clone()); + followed.start(closed_broadcaster().await); + server + .inner + .broadcasts + .lock() + .unwrap() + .insert("live".to_string(), Arc::new(followed)); + + let current = server.broadcaster("live").await.expect("broadcast announced"); + assert!(!current.is_closed()); + let kept = server.instance("live", open.instance()).await; + assert!( + kept.is_some_and(|kept| Arc::ptr_eq(&kept, &open)), + "the open instance lost its URLs" + ); + } } From 22ae15d4ac769c4d096f056f1717bbc74250cd36 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 11:54:43 -0700 Subject: [PATCH 7/7] fix(rtc): poll the catalog end before egress writes The biased select polled writes first, so a write channel that stays ready could starve the catalog failure and keep the session alive. Co-Authored-By: Claude Opus 5.5 --- rs/moq-rtc/src/session.rs | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/rs/moq-rtc/src/session.rs b/rs/moq-rtc/src/session.rs index 72c7156a0c2..5788c109031 100644 --- a/rs/moq-rtc/src/session.rs +++ b/rs/moq-rtc/src/session.rs @@ -172,19 +172,10 @@ impl Session { // Egress writes get drained promptly; `Agent::next` is cancel safe. biased; - Some(req) = async { - match self.writes_rx.as_mut() { - Some(rx) => rx.recv().await, - None => std::future::pending::>().await, - } - } => { - let wallclock = self.egress_clock.wallclock(req.time, Instant::now()); - crate::egress::dispatch(&mut self.agent.rtc_mut(), req, wallclock); - } - // A failed catalog, as every subscription fails once the source's route dies, // means nothing more will flow, so fail rather than idle. A clean end keeps - // serving what the pumps still hold. + // serving what the pumps still hold. Polled before the writes, which could + // otherwise stay ready and starve it; it completes once. ended = async { match self.catalog_end.as_mut() { Some(end) => end.await, @@ -197,6 +188,16 @@ impl Session { } } + Some(req) = async { + match self.writes_rx.as_mut() { + Some(rx) => rx.recv().await, + None => std::future::pending::>().await, + } + } => { + let wallclock = self.egress_clock.wallclock(req.time, Instant::now()); + crate::egress::dispatch(&mut self.agent.rtc_mut(), req, wallclock); + } + event = self.agent.next() => match event { Ok(moq_ice::Event::Rtc(event)) => self.handle_event(event)?, // No STUN servers are configured, so there is nothing to trickle.