diff --git a/doc/bin/hls.md b/doc/bin/hls.md index bb564de8f58..95f8f01d194 100644 --- a/doc/bin/hls.md +++ b/doc/bin/hls.md @@ -55,6 +55,7 @@ What that means for you: - **Live joins cost the window.** The exporter reads the record checkpoint and lists at most 256 segments. - **Segments are timed by their records.** A segment is listed once the record after it arrives, and its duration, its DASH `t` and `d`, and its last sample's duration all come from its own record and the next, so a segment's bytes never change with a catalog estimate. A stop record (a break or the end of the content) is listed as a zero-length gap, and what follows it starts a discontinuity. - **The target duration is declared.** Every media playlist keeps one `EXT-X-TARGETDURATION` while its rendition stays in the catalog, even across a reconfigure: the longest segment duration the broadcast's first renditions declare, rounded up. A rendition added later that declares longer segments gets a larger target of its own, while playlists already served keep theirs. A rendition removed and re-added gets the target its declared duration gives, so keep a rendition's declared duration fixed for the run. A GOP longer than the target, once rounded, is still listed rather than dropped or split, against RFC 8216's rule: hls.js plays it, but Apple's `mediastreamvalidator` flags it, so keep keyframe intervals within the declared duration. +- **A reconfigure keeps the playlist.** A rendition whose decoder config changes (its resolution, say) keeps its media playlist: the segments already listed stay, under the init they were muxed for, and the new config's segments follow behind an `EXT-X-DISCONTINUITY` and a new `EXT-X-MAP`, as RFC 8216 requires. The DASH manifest lists only the new config's segments. - **Disabled renditions are hidden.** A rendition the catalog marks `enabled: false` leaves the multivariant playlist and DASH manifest until it re-enables. Its media playlist keeps serving, so a player already on it is not ended with `EXT-X-ENDLIST`. - **URLs name the stream.** Every URL the multivariant playlist and DASH manifest list sits under `@`, the stream's instance, so every edge and every reload derive the same URLs and a CDN in front of export caches them. A restarted publisher gets URLs of its own, whose numbering starts over. The multivariant playlist and manifest switch to it at once, while the old stream's playlists end at the segments they listed before the switch, even if its publisher keeps sending; a player already on it plays those out and then has to reload the multivariant playlist. A stream that reached export over moq-lite-06 or older, or moq-transport, has an instance its session minted, so only that export serves its URLs. - **Only a clean finish ends a playlist.** `EXT-X-ENDLIST` and a `static` DASH manifest mean the records finished, or, on a replaced stream's media playlists, that another publisher took the broadcast over. A rendition whose records fail instead, such as a publisher dropping its session, answers HTTP 500 with `Cache-Control: no-store`, so a truncated recording never looks complete, and leaves the multivariant playlist and DASH manifest so players switch to a sibling. With none left, the DASH manifest answers the error too. diff --git a/quest/m1/README.md b/quest/m1/README.md index db829b3e3d6..62ba47d5c71 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -175,7 +175,7 @@ per-frame CPU growth since #5926. - [A follower is told when the serving instance's route leaves](/quest/m1/instance-leave-restart.md) - wire-level tests pin the restart a follower gets when the serving instance's route is withdrawn or dies while another instance's remains - [Poll-first async](/quest/m1/poll-first/README.md) - library crates compile no coroutines by default: async wrappers are generated from `poll_*` and gated behind an `async` feature - [Authored tracks default to a 30 s max-age](/quest/m1/max-age-defaults.md) - raw and media tracks advertise 30 s unless their author chooses otherwise, with explicit unlimited in every binding -- [HLS export follows a resolution change](/quest/m1/hls-reinit-same-codec.md) - a decoder-identity change under the same codec string starts a new init and segment tag +- [HLS export lists a same-codec reconfigure's first group under the new init](/quest/m1/hls-reinit-same-codec.md) - the hand-over pick, whichever of the catalog and the records arrives first, after the catalog `hls` removal is planned - [moq-quic deletes its remaining 0-RTT code](/quest/m1/quic-zero-rtt-leftovers.md) - only the code that drops a 0-RTT packet remains, with a test - [Gate the no-em-dash rule](/quest/m1/em-dash-gate.md) - `just check` fails on an em dash in any tracked text file - [Every package shares one minor version](/quest/m1/shared-version.md) - a recipe sets every published package to one minor at the cut (next: 0.22.0), and `just check` enforces it and the `0.0.x` rule for experiments diff --git a/quest/m1/hls-conformance.md b/quest/m1/hls-conformance.md index 35bad6177bd..2a5dfdb296e 100644 --- a/quest/m1/hls-conformance.md +++ b/quest/m1/hls-conformance.md @@ -44,7 +44,7 @@ own quest below. ## Required -- [HLS export follows a resolution change](/quest/m1/hls-reinit-same-codec.md) - the conformance test's same-codec switch fails until it lands +- [HLS export lists a same-codec reconfigure's first group under the new init](/quest/m1/hls-reinit-same-codec.md) - the conformance test's same-codec switch fails until it lands ## Related diff --git a/quest/m1/hls-reinit-same-codec.md b/quest/m1/hls-reinit-same-codec.md index d00d94ff06b..84e3873137b 100644 --- a/quest/m1/hls-reinit-same-codec.md +++ b/quest/m1/hls-reinit-same-codec.md @@ -1,16 +1,20 @@ -# [M] HLS export follows a resolution change under the same codec string +# [M] HLS export lists a same-codec reconfigure's first group under the new init ## Goal -When a rendition changes resolution (or any other decoder-affecting field) while keeping its codec string, the HLS and DASH export starts a new init segment and segment tag for it, so every served segment decodes under the init it is listed with, the rendition's first segments are listed, and its recording cursor keeps up. +When a rendition changes resolution (or any other decoder-affecting field) while keeping its codec string, the first group after the change is listed under the new init and segment tag, whichever of the catalog and the HLS records reaches the export first. + +The conformance test in [moq-dev/moq#7387](https://github.com/moq-dev/moq/pull/7387) is the end-to-end regression test: its `video1` same-codec switch must pass. ## Plan -Found by the conformance test in [moq-dev/moq#7387](https://github.com/moq-dev/moq/pull/7387): after `video1` switches from 320x180 to 480x270 with the same codec string, the export keeps the old segment tag and init. The 480x270 segments are served under the 320x180 init with no in-band parameter sets, so they don't decode, that rendition's first segments are never listed, and its recording cursor fails with `RecordingLagged`. When the codec string also changes, the importer makes a new rendition name and everything passes. +Found by the conformance test in [moq-dev/moq#7387](https://github.com/moq-dev/moq/pull/7387). The export already replaces the instance on a decoder-identity change (the same fields hang's `same_identity` compares, plus geometry), and since [moq-dev/moq#7436](https://github.com/moq-dev/moq/pull/7436) the playlist keeps the retired instance's segments, under their own init, ahead of the replacement's behind a discontinuity (RFC 8216 6.2.2). + +What remains is the hand-over race. The catalog and the records are separate tracks, so the new config's first group can reach the old instance's records watcher before the catalog update. The retiring instance claims it, so it is muxed under the old init and the replacement starts one group late. -Decided 2026-10-30 (maintainer, follow-up planning after the /quest-afk run). Treat a change of the rendition's decoder identity (the same fields hang's `same_identity` compares) as a new instance in the export, as a codec-string change already is. Reuse the existing instance-replacement path rather than adding a second one. The conformance test's `video1` case is the regression test; it must pass with the same-codec switch restored. +Waiting on a plan: the maintainer decided on 2026-11-02 to remove the catalog's `hls` field (`rs/hang/src/catalog/hls.rs` and the JS zod schema), planned separately. That changes how the export finds a rendition's records, so the hand-over pick is designed after that plan, not with a catalog `start` field, a record track per config, or a marker in each record (all three dropped the same day). Once the removal is planned, add it under `Required` here or fold the pick into it. -Public API: none. Wire: none. +Public API: none expected. Wire: decided by the removal plan. ## Related diff --git a/rs/moq-hls/src/export/mod.rs b/rs/moq-hls/src/export/mod.rs index f9be6e15c7f..c6fe3def8c3 100644 --- a/rs/moq-hls/src/export/mod.rs +++ b/rs/moq-hls/src/export/mod.rs @@ -422,17 +422,17 @@ impl Broadcaster { self.renditions.video_init().await; } - /// Resolve once the broadcast has something listable (its first complete segment, or - /// already-ended records). Renditions start their segments together, so the first - /// rendition's readiness stands in for the broadcast's. Bounding the wait is the caller's - /// policy. + /// Resolve once the broadcast has something the DASH manifest lists (its first complete + /// segment, or already-ended records); a reconfigured rendition's predecessor's segments + /// don't count. Renditions start their segments together, so the first rendition's readiness + /// stands in for the broadcast's. Bounding the wait is the caller's policy. #[cfg_attr( not(feature = "server"), allow(dead_code, reason = "only the `server` feature uses it") )] - pub(crate) async fn playable(&self) { + pub(crate) async fn represented(&self) { if let Some(rendition) = self.renditions.snapshot().into_iter().next() { - rendition.playable().await; + rendition.represented().await; } } @@ -1659,8 +1659,9 @@ mod tests { listed_in_manifest(&broadcaster).await; } - // A reconfigure that changes the init bytes changes the init URL, and the old URL stops - // resolving, so a cache can never hand a player the previous init under the new one. + // A reconfigure that changes the init bytes changes the init URL, so a cache can never hand a + // player the previous init under the new one. The old URL keeps naming the old bytes while the + // playlist still lists the segments muxed under them. #[tokio::test] async fn a_reconfigure_versions_the_init_url() { let origin = produce_origin(); @@ -1702,10 +1703,20 @@ mod tests { let new_init = new.latest_init().await.unwrap().expect("init segment"); let _ = kio::clock::timeout(Duration::from_secs(5), new.playable()).await; let new_playlist = new.media_playlist(None).unwrap().expect("playable"); - let new_hash = init_hash(&new_playlist); + let maps: Vec<_> = new_playlist + .split("#EXT-X-MAP:URI=\"init.") + .skip(1) + .map(|rest| rest.split_once(".mp4").unwrap().0) + .collect(); + let new_hash = *maps.last().unwrap(); assert_ne!(new_hash, old_hash, "different init bytes get a different URL"); + assert_eq!(maps, [old_hash.as_str(), new_hash], "{new_playlist}"); assert_eq!(new.init(new_hash).await.unwrap(), Some(new_init)); - assert!(new.init(&old_hash).await.unwrap().is_none()); + assert_eq!( + new.init(&old_hash).await.unwrap(), + old.latest_init().await.unwrap(), + "the predecessor's segments still listed map its init" + ); } // Background work ends with its owners, not with the publisher: once a reconfigure replaced @@ -2407,7 +2418,7 @@ mod tests { let broadcaster = Broadcaster::spawned(crate::resolved(&origin, "live").await, Config::default()); let _ = kio::clock::timeout(Duration::from_secs(5), broadcaster.ready()).await; - let _ = kio::clock::timeout(Duration::from_secs(5), broadcaster.playable()).await; + let _ = kio::clock::timeout(Duration::from_secs(5), broadcaster.represented()).await; // A clean finish stops the open segment and ends the records. media.finish().unwrap(); @@ -3940,14 +3951,248 @@ mod tests { panic!("the replacement is reported"); }; // The replacement muxes the same records differently, so it lists them under its - // own URLs, and only those its predecessor didn't. + // own URLs, and only those its predecessor didn't. The playlist keeps the + // predecessor's behind a discontinuity, each under its own init. assert_ne!(new.tag(), rendition.tag(), "one URL never names two muxes"); records.push(&gop(3)).unwrap(); - assert_eq!(numbers(&test.listed("video0", 2).await), [2]); + let snapshot = test.listed("video0", 2).await; + assert_eq!(numbers(&snapshot), [0, 1, 2]); + let tags: Vec<_> = snapshot.segments.iter().map(|s| s.tag.as_str()).collect(); + assert_eq!(tags, [rendition.tag(), rendition.tag(), new.tag()]); + let breaks: Vec<_> = snapshot.segments.iter().map(|s| s.discontinuity).collect(); + assert_eq!(breaks, [false, false, true]); + let maps: Vec<_> = snapshot.segments.iter().map(|s| s.map.is_some()).collect(); + assert_eq!(maps, [true, true, false], "only the predecessor's map their own init"); } } } + // A reconfigured rendition's playlist URI keeps serving its predecessor's segments, under their + // own URLs, until the window slides past them (RFC 8216 6.2.2), then releases the predecessor. + #[tokio::test(start_paused = true)] + async fn a_reconfigure_keeps_serving_its_predecessors_segments() { + let mut test = Records::with(&["video0"], &[]); + let (_, mut records) = test.publish("video0"); + let _media = test.media("video0", 24); + for sequence in 0..3 { + records.push(&gop(sequence)).unwrap(); + } + test.listed("video0", 1).await; + let old = test.rendition("video0"); + test.catalog.video.renditions.get_mut("video0").unwrap().coded_width = Some(1280); + test.sync(); + let new = test.rendition("video0"); + assert!(!Arc::ptr_eq(&old, &new), "the reconfigure replaces the rendition"); + assert_eq!( + numbers(&new.snapshot()), + [0, 1], + "nothing the predecessor listed is dropped" + ); + let id = |tag: &str, segment| segments::Id { + generation: None, + tag: tag.to_string(), + at: segments::At::Sequence(segment), + }; + assert!(new.segment(&id(old.tag(), 0)).await.unwrap().is_some()); + + // The predecessor lists the segment it was waiting on, then the replacement its own. + records.push(&gop(3)).unwrap(); + records.push(&gop(4)).unwrap(); + let snapshot = test.listed("video0", 3).await; + assert_eq!(numbers(&snapshot), [0, 1, 2, 3]); + let breaks: Vec<_> = snapshot.segments.iter().map(|s| s.discontinuity).collect(); + assert_eq!(breaks, [false, false, false, true]); + assert!(new.segment(&id(old.tag(), 2)).await.unwrap().is_some()); + assert!(new.segment(&id(new.tag(), 3)).await.unwrap().is_some()); + assert!( + new.segment(&id(new.tag(), 2)).await.unwrap().is_none(), + "one URL never names two muxes" + ); + + // Once the window slides past the predecessor's segments, nothing holds it. + for sequence in 5..24 { + records.push(&gop(sequence)).unwrap(); + } + let snapshot = test.listed("video0", 22).await; + assert!(snapshot.segments.iter().all(|s| s.segment > 3 && s.map.is_none())); + assert_eq!( + snapshot.discontinuity_sequence, 1, + "the slid-out break is still counted" + ); + assert_eq!(Arc::strong_count(&old), 1, "the replacement released its predecessor"); + assert!(new.segment(&id(old.tag(), 2)).await.unwrap().is_none()); + } + + // An inline-codec replacement builds its init only from a segment of its own, so until it lists + // one, its playlist is the predecessor's segments under the predecessor's init. + #[tokio::test(start_paused = true)] + async fn an_inherited_playlist_needs_no_init_of_its_own() { + let mut test = Records::with(&["video0"], &[]); + let (_, mut records) = test.publish("video0"); + let _media = test.media("video0", 4); + for sequence in 0..3 { + records.push(&gop(sequence)).unwrap(); + } + test.listed("video0", 1).await; + let old = test.rendition("video0"); + let rendered = kio::clock::timeout(Duration::from_secs(5), old.playlist(None)) + .await + .expect("the playlist renders") + .unwrap() + .expect("playable"); + let map = rendered + .lines() + .find(|line| line.starts_with("#EXT-X-MAP")) + .expect("a map") + .to_string(); + // The new config leaves its geometry to the keyframes, so only its media builds its init. + let config = test.catalog.video.renditions.get_mut("video0").unwrap(); + config.coded_width = None; + config.coded_height = None; + test.sync(); + let new = test.rendition("video0"); + assert!(!Arc::ptr_eq(&old, &new), "the reconfigure replaces the rendition"); + + let rendered = kio::clock::timeout(Duration::from_secs(5), new.playlist(None)) + .await + .expect("the playlist renders without the replacement's media") + .unwrap() + .expect("the predecessor's segments are playable"); + assert_eq!(new.media_playlist(None).unwrap().as_deref(), Some(rendered.as_str())); + let maps: Vec<_> = rendered.lines().filter(|line| line.starts_with("#EXT-X-MAP")).collect(); + assert_eq!(maps, [map.as_str()], "{rendered}"); + assert!(rendered.contains(&format!("seg/{}.1.m4s", old.tag())), "{rendered}"); + + // Its own first segment needs its own init, which the playlist builds before listing it. + records.push(&gop(3)).unwrap(); + records.push(&gop(4)).unwrap(); + test.listed("video0", 3).await; + let rendered = kio::clock::timeout(Duration::from_secs(5), new.playlist(None)) + .await + .expect("the playlist renders") + .unwrap() + .expect("playable"); + assert!(rendered.contains(&format!("seg/{}.3.m4s", new.tag())), "{rendered}"); + assert_eq!( + new.media_playlist(None).unwrap().as_deref(), + Some(rendered.as_str()), + "its own init is built" + ); + } + + // The DASH manifest lists only the current instance's segments, so its readiness waits for the + // replacement's own first segment, even while the HLS playlist serves the predecessor's. + #[tokio::test(start_paused = true)] + async fn dash_readiness_waits_for_the_replacements_own_segment() { + let mut test = Records::with(&["video0"], &[]); + let (_, mut records) = test.publish("video0"); + let _media = test.media("video0", 5); + for sequence in 0..3 { + records.push(&gop(sequence)).unwrap(); + } + test.listed("video0", 1).await; + resize(&mut test, 1280); + let new = test.rendition("video0"); + kio::clock::timeout(Duration::from_secs(1), new.playable()) + .await + .expect("the playlist lists the predecessor's segments"); + assert!( + kio::clock::timeout(Duration::from_secs(1), new.represented()) + .await + .is_err(), + "the replacement lists nothing of its own yet" + ); + assert!( + kio::clock::timeout(Duration::from_secs(1), test.renditions.video_init()) + .await + .is_err(), + "no manifest lists the replacement yet" + ); + + records.push(&gop(3)).unwrap(); + records.push(&gop(4)).unwrap(); + test.listed("video0", 3).await; + kio::clock::timeout(Duration::from_secs(1), new.represented()) + .await + .expect("the replacement lists its own segment"); + kio::clock::timeout(Duration::from_secs(1), test.renditions.video_init()) + .await + .expect("a manifest lists the replacement"); + } + + /// Reconfigure `video0` to `width`. + fn resize(test: &mut Records, width: u32) { + test.catalog.video.renditions.get_mut("video0").unwrap().coded_width = Some(width); + test.sync(); + } + + // A predecessor still waiting on its first segment keeps it, whenever the playlist renders. + #[tokio::test(start_paused = true)] + async fn a_predecessors_pending_first_segment_is_kept() { + let mut test = Records::with(&["video0"], &[]); + let (_, mut records) = test.publish("video0"); + let _media = test.media("video0", 4); + records.push(&gop(0)).unwrap(); + kio::clock::sleep(Duration::from_millis(100)).await; + resize(&mut test, 1280); + assert!(test.rendition("video0").snapshot().segments.is_empty()); + records.push(&gop(1)).unwrap(); + records.push(&gop(2)).unwrap(); + assert_eq!(numbers(&test.listed("video0", 1).await), [0, 1]); + } + + // Reconfigures that list nothing keep only the instance that does, however many there are. + #[tokio::test(start_paused = true)] + async fn reconfigures_listing_nothing_hold_nothing() { + let mut test = Records::with(&["video0"], &[]); + let (_, mut records) = test.publish("video0"); + let _media = test.media("video0", 2); + records.push(&gop(0)).unwrap(); + records.push(&gop(1)).unwrap(); + test.listed("video0", 0).await; + for width in 1..=8 { + resize(&mut test, width); + kio::clock::sleep(Duration::from_millis(10)).await; + } + assert_eq!(numbers(&test.rendition("video0").snapshot()), [0]); + assert!(test.renditions.targets() <= 3, "{} targets", test.renditions.targets()); + } + + // A config that returns while its earlier instance's segments are still listed gets its tag + // back, and each segment URL still reaches the instance that lists it. + #[tokio::test(start_paused = true)] + async fn a_returning_config_serves_both_instances_segments() { + let mut test = Records::with(&["video0"], &[]); + let (_, mut records) = test.publish("video0"); + let _media = test.media("video0", 8); + for sequence in 0..3 { + records.push(&gop(sequence)).unwrap(); + } + test.listed("video0", 1).await; + let first = test.rendition("video0"); + resize(&mut test, 1280); + records.push(&gop(3)).unwrap(); + records.push(&gop(4)).unwrap(); + test.listed("video0", 3).await; + resize(&mut test, 320); + let returned = test.rendition("video0"); + assert_eq!(returned.tag(), first.tag(), "the same config muxes the same way"); + records.push(&gop(5)).unwrap(); + records.push(&gop(6)).unwrap(); + assert_eq!(numbers(&test.listed("video0", 5).await), [0, 1, 2, 3, 4, 5]); + let id = |segment| segments::Id { + generation: None, + tag: first.tag().to_string(), + at: segments::At::Sequence(segment), + }; + for segment in [0, 2, 5] { + assert!( + returned.segment(&id(segment)).await.unwrap().is_some(), + "segment {segment}" + ); + } + } + // A replacement reading another record track shares no numbering with its predecessor, so it // lists that track's records from the first. #[tokio::test(start_paused = true)] diff --git a/rs/moq-hls/src/export/playlist.rs b/rs/moq-hls/src/export/playlist.rs index 1495586b051..694d7f8b8b6 100644 --- a/rs/moq-hls/src/export/playlist.rs +++ b/rs/moq-hls/src/export/playlist.rs @@ -40,6 +40,10 @@ pub(crate) struct Segment { pub segment: u64, /// The [`tag`](super::segments::tag) of the record track numbering the segment. pub tag: String, + /// The hash of the init segment it decodes under, when that is a predecessor's rather than + /// the rendition's own: a reconfigure keeps its predecessor's segments listed (RFC 8216 + /// 6.2.2), each under the `EXT-X-MAP` it was muxed for. + pub map: Option, /// `EXTINF` duration. pub duration: Duration, /// The rendition has no content for this span (`EXT-X-GAP`): the segment keeps its slot in @@ -53,7 +57,8 @@ pub(crate) struct Segment { pub program_date_time: Option, } -/// Render a media playlist for one rendition from a [`Snapshot`], mapping `init.{init}.mp4`. +/// Render a media playlist for one rendition from a [`Snapshot`], mapping `init.{init}.mp4` +/// except where a segment names another [`map`](Segment::map). /// /// `query` is an optional query string (without the leading `?`, e.g. `jwt=`) /// appended to every child URL (the init map and each segment), so a stock player that @@ -81,12 +86,21 @@ pub(crate) fn render_media(snapshot: &Snapshot, init: &str, query: Option<&str>) if snapshot.discontinuity_sequence > 0 { let _ = writeln!(out, "#EXT-X-DISCONTINUITY-SEQUENCE:{}", snapshot.discontinuity_sequence); } - let _ = writeln!(out, "#EXT-X-MAP:URI=\"init.{init}.mp4{suffix}\""); + fn map<'a>(segment: &'a Segment, init: &'a str) -> &'a str { + segment.map.as_deref().unwrap_or(init) + } + let mut current = snapshot.segments.first().map_or(init, |segment| map(segment, init)); + let _ = writeln!(out, "#EXT-X-MAP:URI=\"init.{current}.mp4{suffix}\""); for (index, segment) in snapshot.segments.iter().enumerate() { if segment.discontinuity { let _ = writeln!(out, "#EXT-X-DISCONTINUITY"); } + // A new init follows the discontinuity its reconfigure starts (RFC 8216 4.3.2.5). + if map(segment, init) != current { + current = map(segment, init); + let _ = writeln!(out, "#EXT-X-MAP:URI=\"init.{current}.mp4{suffix}\""); + } if (index == 0 || segment.discontinuity) && let Some(pdt) = segment.program_date_time { @@ -126,6 +140,7 @@ mod tests { Segment { segment: 10, tag: "ab12cd34".into(), + map: None, duration: Duration::from_secs(2), gap: false, discontinuity: false, @@ -134,6 +149,7 @@ mod tests { Segment { segment: 11, tag: "ab12cd34".into(), + map: None, duration: Duration::from_millis(1960), gap: false, discontinuity: false, @@ -170,6 +186,7 @@ mod tests { let segment = |segment, duration, gap, discontinuity, program_date_time| Segment { segment, tag: "ab12cd34".into(), + map: None, duration: Duration::from_millis(duration), gap, discontinuity, @@ -208,6 +225,7 @@ mod tests { segments: vec![Segment { segment: 0, tag: "ab12cd34".into(), + map: None, duration: Duration::from_secs(2), gap: false, discontinuity: false, @@ -231,6 +249,7 @@ mod tests { segments: vec![Segment { segment: 0, tag: "ab12cd34".into(), + map: None, duration: Duration::from_secs(4), gap: false, discontinuity: false, @@ -244,4 +263,40 @@ mod tests { assert!(out.contains("#EXT-X-ENDLIST\n")); assert!(!out.contains("PROGRAM-DATE-TIME")); } + + // A reconfigured rendition keeps its predecessor's segments, each under its own init, and + // maps the new init after the discontinuity that starts it. + #[test] + fn a_predecessor_keeps_its_own_map() { + let segment = |segment, map: Option<&str>, discontinuity| Segment { + segment, + tag: "ab12cd34".into(), + map: map.map(str::to_string), + duration: Duration::from_secs(2), + gap: false, + discontinuity, + program_date_time: None, + }; + let snapshot = Snapshot { + target_duration: 2, + media_sequence: 3, + discontinuity_sequence: 0, + segments: vec![ + segment(3, Some("00000000"), false), + segment(4, Some("00000000"), false), + segment(5, None, true), + ], + finished: false, + generation: None, + }; + + let out = render_media(&snapshot, "11111111", None); + assert!( + out.contains( + "#EXT-X-MAP:URI=\"init.00000000.mp4\"\n#EXTINF:2.00000,\nseg/ab12cd34.3.m4s\n#EXTINF:2.00000,\nseg/ab12cd34.4.m4s\n#EXT-X-DISCONTINUITY\n#EXT-X-MAP:URI=\"init.11111111.mp4\"\n#EXTINF:2.00000,\nseg/ab12cd34.5.m4s\n" + ), + "{out}" + ); + assert_eq!(out.matches("EXT-X-MAP").count(), 2, "{out}"); + } } diff --git a/rs/moq-hls/src/export/rendition.rs b/rs/moq-hls/src/export/rendition.rs index 5e423df1c72..f8cacd9029c 100644 --- a/rs/moq-hls/src/export/rendition.rs +++ b/rs/moq-hls/src/export/rendition.rs @@ -284,6 +284,9 @@ pub(crate) struct Handoff { next: u64, /// Its `EXT-X-TARGETDURATION`, which a replacement serving the same playlist URI keeps. target_duration: u64, + /// The retired instance, while it lists or still waits to list a segment: the replacement + /// keeps listing those (see [`Rendition::after`]). + previous: Option>, } /// Which records a rendition lists, shared by its records watcher and [`Rendition::retire`] so a @@ -297,6 +300,8 @@ struct Claim { next: u64, /// Set by [`Rendition::retire`]: nothing at or past `next` is listed. retired: bool, + /// A record waits on its successor to be listed. + pending: bool, } /// A single HLS rendition: master-playlist metadata, its window over its own HLS records (which @@ -350,6 +355,10 @@ pub struct Rendition { retiring: tokio::sync::watch::Sender, /// Which records this instance lists. claim: Mutex, + /// The retired instance this one replaced on the same records, whose segments its playlist + /// still lists ahead of its own until the window slides past them (see [`after`](Self::after)). + /// Released once none is listed, or by a restart. + previous: Mutex>>, /// The broadcast serving this rendition's media. A sibling is bound when the rendition is /// created, and ends the rendition when it ends. Shared with a background init build. media: Arc, @@ -461,6 +470,7 @@ impl Rendition { watcher: Mutex::new(None), retiring: tokio::sync::watch::Sender::new(false), claim: Mutex::default(), + previous: Mutex::new(None), media: Arc::new(media), run: kio::Producer::default(), building: Arc::new(tokio::sync::Mutex::new(())), @@ -500,6 +510,7 @@ impl Rendition { watcher: Mutex::new(None), retiring: tokio::sync::watch::Sender::new(false), claim: Mutex::default(), + previous: Mutex::new(None), media: Arc::new(media), run: kio::Producer::default(), building: Arc::new(tokio::sync::Mutex::new(())), @@ -521,6 +532,12 @@ impl Rendition { /// Take over from a retired predecessor (see [`retire`](Self::retire)): keep its target /// duration and list only the records it won't. One reading another record track shares no /// numbering with it, so it lists all. + /// + /// On the same records, the playlist URI keeps listing the predecessor's segments, under the + /// init they were muxed for, until the window slides past them: a server never removes a + /// segment from the middle or end of a live playlist (RFC 8216 6.2.2). This instance lists its + /// own behind a discontinuity, once the predecessor has listed all it will, so the two never + /// interleave. pub(crate) fn after(mut self, handoff: Option<&Handoff>) -> Self { let Some(handoff) = handoff else { return self; @@ -535,8 +552,10 @@ impl Rendition { start, next: start, retired: false, + pending: false, }); self.live = Arc::new(segments::Producer::new(start)); + self.previous = Mutex::new(handoff.previous.clone()); self } @@ -546,18 +565,98 @@ impl Rendition { /// Its records go on (a replacement follows them too, and the publisher never finishes a /// dropped track's), so they can't end it. Ending it at once would lose the last segment of /// every recording, since it is listed only when the record after it arrives. - pub(crate) fn retire(&self) -> Handoff { - let next = { + pub(crate) fn retire(self: &Arc) -> Handoff { + let (next, pending) = { let mut claim = self.claim.lock().expect("claim lock poisoned"); claim.retired = true; - claim.next + (claim.next, claim.pending) }; self.retiring.send_replace(true); + // Once retired, one that neither lists nor waits to list a segment never will, so the + // replacement keeps only what this one was keeping: churn can't stack empty instances. + let lists = pending || !self.live.window().segments.is_empty(); Handoff { track: self.hls.track.clone(), next, target_duration: self.target_duration, + previous: match lists { + true => Some(self.clone()), + false => self.previous(), + }, + } + } + + fn previous(&self) -> Option> { + self.previous.lock().expect("previous lock poisoned").clone() + } + + /// The retired instances whose segments this one's playlist still lists, oldest first. + fn lineage(&self) -> Vec> { + let mut lineage = Vec::new(); + let mut next = self.previous(); + while let Some(rendition) = next { + next = rendition.previous(); + lineage.push(rendition); + } + lineage.reverse(); + lineage + } + + /// The predecessors' rows listed ahead of `own`, oldest first, each with its instance. + /// + /// They are trimmed as one window with `own`, as if one instance had listed them all. Rows + /// before a jump in the numbering are dropped, as a skip clears one instance's window. A + /// predecessor none of whose rows is left is released, along with every older one, once none + /// of them can list another. + fn inherited(&self, own: &[segments::Row]) -> Vec<(Arc, segments::Row)> { + let lineage = self.lineage(); + if lineage.is_empty() { + return Vec::new(); + } + // Each read whole, so its rows are all it will ever list once it reads closed. + let windows: Vec<_> = lineage.iter().map(|rendition| rendition.live.window()).collect(); + let closed = windows.iter().all(|window| window.closed); + let mut rows: std::collections::VecDeque<_> = lineage + .iter() + .zip(windows) + .flat_map(|(rendition, window)| { + window + .segments + .into_iter() + .map(move |row| (Some(rendition.clone()), row)) + }) + .collect(); + let inherited = rows.len(); + rows.extend(own.iter().map(|row| (None, row.clone()))); + // Records skipped across the hand-over leave a jump the inherited rows can't bridge. + let numbers: Vec<_> = rows.iter().map(|(_, row)| row.segment).collect(); + let mut trimmed = broken(&numbers, inherited); + rows.drain(..trimmed); + let window = self.fanout.window(); + while trimmed < inherited { + let rest = match rows.get(1) { + Some((_, second)) => rows.back().expect("two rows").1.end.saturating_sub(second.pts.into()), + None => Duration::ZERO, + }; + if !segments::evicts(window, rows.len(), rest) { + break; + } + rows.pop_front(); + trimmed += 1; + } + let rows: Vec<_> = rows + .into_iter() + .take(inherited - trimmed) + .map(|(rendition, row)| (rendition.expect("an inherited row"), row)) + .collect(); + // Nothing lists anything older than the oldest instance left, whose rows followed every + // older one's last. + match rows.first() { + Some((oldest, _)) => *oldest.previous.lock().expect("previous lock poisoned") = None, + None if closed => *self.previous.lock().expect("previous lock poisoned") = None, + None => {} } + rows } /// Advertise `target_duration` seconds as `EXT-X-TARGETDURATION`: the one this rendition's @@ -610,6 +709,10 @@ impl Rendition { fn push(&self, row: segments::Row) { if self.media.admits(&self.live) { self.live.push(row, self.fanout.window()); + // Release the predecessors this push slid out, even with nobody rendering. + if self.previous().is_some() { + self.inherited(&self.live.window().segments); + } } } @@ -627,6 +730,8 @@ impl Rendition { // never start its own. self.build.lock().expect("build lock poisoned").take(); self.live.clear(); + // Its predecessors' rows are the previous run's too. + *self.previous.lock().expect("previous lock poisoned") = None; let mut run = self.run_mut(); *run = Run { generation, @@ -701,6 +806,32 @@ impl Rendition { /// Poll until this rendition is playable (it lists a segment, or its records ended), its /// media is denied, or its window failed. pub(crate) fn poll_playable(&self, waiter: &kio::Waiter) -> Poll<()> { + self.poll_media(waiter); + if self.media.refusal().is_some() { + return Poll::Ready(()); + } + // A replacement's playlist lists its predecessors' segments until it has its own. + if self.live.poll_listed(waiter).is_ready() + || self + .lineage() + .iter() + .any(|rendition| rendition.live.poll_rows(waiter).is_ready()) + { + return Poll::Ready(()); + } + Poll::Pending + } + + /// Resolve once the DASH [representation](Self::representation) lists a segment or this + /// rendition fails, as [`playable`](Self::playable) does for the media playlist. + pub(crate) async fn represented(&self) { + kio::wait(|waiter| self.poll_represented(waiter)).await; + } + + /// Poll until this instance lists a segment of its own (or its records ended), its media is + /// denied, or its window failed. Unlike [`poll_playable`](Self::poll_playable), a + /// predecessor's segments don't count: the DASH manifest lists none of them. + pub(crate) fn poll_represented(&self, waiter: &kio::Waiter) -> Poll<()> { self.poll_media(waiter); if self.media.refusal().is_some() { return Poll::Ready(()); @@ -727,8 +858,8 @@ impl Rendition { /// A denied referenced broadcast propagates the underlying authorization error; it is /// not missing media. The HTTP server answers it with 403 and no-store. /// - /// The playlist names its init segment by a hash of the bytes, so it is also `None` until - /// the init is known. An inline-parameter-set codec (no catalog `description`) only learns + /// The playlist names each init segment by a hash of the bytes, so it is also `None` until + /// the init of every listed segment is known. An inline-parameter-set codec (no catalog `description`) only learns /// it from media: await [`init`](Self::init) first. /// /// `query` is an optional query string (without the leading `?`, e.g. `jwt=`) @@ -740,11 +871,24 @@ impl Rendition { // Read the run before the rows (see `restart`). let run = self.run(); let snapshot = self.snapshot_as(run.generation.clone())?; - let Some(init) = self.built_init(&run) else { + if snapshot.segments.is_empty() && !snapshot.finished { return Ok(None); + } + // A predecessor's segment maps its own init, `""` while that is not built. + if snapshot.segments.iter().any(|s| s.map.as_deref() == Some("")) { + return Ok(None); + } + // Nothing maps this instance's init until it lists a segment of its own, and an inline + // codec builds it only from that segment. + let mapped = snapshot.segments.is_empty() || snapshot.segments.iter().any(|s| s.map.is_none()); + let init = match mapped { + true => match self.built_init(&run) { + Some(init) => init.hash.clone(), + None => return Ok(None), + }, + false => String::new(), }; - let playable = !snapshot.segments.is_empty() || snapshot.finished; - Ok(playable.then(|| super::render_media(&snapshot, &init.hash, query))) + Ok(Some(super::render_media(&snapshot, &init, query))) } /// Wait until the media playlist is servable, then render it: at least one segment is @@ -761,8 +905,26 @@ impl Rendition { self.playable().await; // Failed records may never supply the keyframe an inline init waits for. self.check()?; - // An inline-codec init needs a keyframe group fetched first. Building it caches, so the - // player's follow-up GET of the init is free. + // Build the inits the listed segments map. An inline-codec init needs a keyframe group + // fetched first. Building it caches, so the player's follow-up GET of the init is free. + let own = self.live.window().segments; + let mut inherited = self.inherited(&own); + inherited.dedup_by(|(a, _), (b, _)| Arc::ptr_eq(a, b)); + for (rendition, _) in &inherited { + if rendition.load_init().await?.is_none() { + return Ok(None); + } + } + // This instance's own init is mapped only once it lists a segment (see `media_playlist`). + let mapped = !own.is_empty() || inherited.is_empty(); + if mapped && self.load_init().await?.is_none() { + return Ok(None); + } + let rendered = self.media_playlist(query)?; + if rendered.is_some() || mapped { + return Ok(rendered); + } + // It listed its first segment meanwhile. if self.load_init().await?.is_none() { return Ok(None); } @@ -789,7 +951,8 @@ impl Rendition { return Err(err); } let finished = window.ended; - Ok(self.snapshot_from_window(window, finished, generation)) + let inherited = self.inherited(&window.segments); + Ok(self.snapshot_from_window(inherited, window, finished, generation)) } /// Why listings refuse: the referenced media is denied, or the window failed. @@ -836,16 +999,37 @@ impl Rendition { } } - fn snapshot_from_window(&self, window: segments::Window, finished: bool, generation: Option>) -> Snapshot { - let mut previous = None; - let segments = window - .segments + /// The snapshot listing `inherited` (see [`inherited`](Self::inherited)), then `window`. + /// + /// An inherited segment maps its instance's init, or `""` while that is not built, which + /// [`media_playlist`](Self::media_playlist) refuses to render. + fn snapshot_from_window( + &self, + inherited: Vec<(Arc, segments::Row)>, + window: segments::Window, + finished: bool, + generation: Option>, + ) -> Snapshot { + let media_sequence = inherited.first().map_or(window.sequence, |(_, row)| row.segment); + let discontinuity_sequence = inherited + .first() + .map_or(window.discontinuity, |(_, row)| row.discontinuity); + let rows = inherited .iter() - .map(|s| { + .map(|(rendition, row)| { + // Cached once built, so this is a read for every row after the first. + let map = rendition.built_init(&rendition.run()); + (Some(map.map_or_else(String::new, |init| init.hash.clone())), row) + }) + .chain(window.segments.iter().map(|row| (None, row))); + let mut previous = None; + let segments = rows + .map(|(map, s)| { let discontinuity = previous.replace(s.discontinuity).is_some_and(|p| p != s.discontinuity); Segment { segment: s.segment, tag: s.tag.to_string(), + map, duration: s.duration, gap: s.stop, discontinuity, @@ -855,8 +1039,8 @@ impl Rendition { .collect(); Snapshot { target_duration: self.target_duration, - media_sequence: window.sequence, - discontinuity_sequence: window.discontinuity, + media_sequence, + discontinuity_sequence, segments, finished, generation, @@ -993,6 +1177,17 @@ impl Rendition { /// media remains usable after its broadcast ends or is replaced; replacement media is never /// read. pub async fn init(&self, hash: &str) -> Result> { + // A predecessor's segments still listed map its init, which was built before any playlist + // listed them (see `media_playlist`), so it is served without building this instance's. + // Only the instance supplying the bytes is authorized: a denied one leaves them to this + // instance, which may have the same config. + for rendition in self.lineage() { + if let Some(init) = rendition.built_init(&rendition.run()).filter(|init| init.hash == hash) + && rendition.authorize().is_ok() + { + return Ok(Some(init.bytes.clone())); + } + } let init = self.load_init().await?; Ok(init.filter(|init| init.hash == hash).map(|init| init.bytes.clone())) } @@ -1125,6 +1320,21 @@ impl Rendition { /// [generation](super::Broadcaster::set_generation), since a restart reuses segment numbers /// and another run's URL would name different bytes. pub async fn segment(&self, id: &segments::Id) -> Result> { + // A predecessor's segments still listed are muxed under its config. A config that returns + // gets its tag back, but the instances' segment numbers never overlap. + if let Some(rendition) = self + .lineage() + .into_iter() + .find(|rendition| *rendition.tag == *id.tag && rendition.live.holds(&id.tag, id.at)) + { + return rendition.own_segment(id).await; + } + self.own_segment(id).await + } + + /// Fetch and transmux the segment `id` names among this instance's own rows; see + /// [`segment`](Self::segment). + async fn own_segment(&self, id: &segments::Id) -> Result> { self.authorize()?; let current = || self.generation().as_deref() == id.generation.as_deref(); if !current() { @@ -1229,6 +1439,18 @@ impl Rendition { /// /// A record is listed once the next one arrives, which bounds its groups and, unless it states /// its own, its duration. A stop is listed at once, as a gap. +/// How many leading rows of `segments` a jump in their numbering cuts off, looking only at the +/// first `inherited` rows and the one after them. +/// +/// A media playlist numbers its rows from the first one's segment, so rows listed before a jump +/// would shift every later row's media sequence number once they slide out. +fn broken(segments: &[u64], inherited: usize) -> usize { + (1..segments.len().min(inherited + 1)) + .rev() + .find(|&i| segments[i - 1].checked_add(1) != Some(segments[i])) + .unwrap_or(0) +} + async fn watch_records( broadcast: moq_net::broadcast::Consumer, hls: Hls, @@ -1236,14 +1458,23 @@ async fn watch_records( mut retiring: tokio::sync::watch::Receiver, ) { let result: Result<()> = async { + // A replacement lists nothing until its predecessor has listed all it will, so their + // segments never interleave and its own start a new discontinuity: they decode under + // another init. + let previous = rendition.upgrade().and_then(|rendition| rendition.previous()); + let mut broken = previous.is_some(); + if let Some(previous) = previous { + tokio::select! { + _ = previous.live.closed() => {} + _ = retiring.wait_for(|retiring| *retiring) => return Ok(()), + } + } let mut records = tokio::select! { records = moq_mux::hls::Consumer::subscribe(&broadcast, &hls) => records?, _ = retiring.wait_for(|retiring| *retiring) => return Ok(()), }; // The newest record, waiting for its successor. let mut pending: Option = None; - // The next listed segment starts a new discontinuity. - let mut broken = false; loop { let event = tokio::select! { event = records.next() => match event? { @@ -1289,6 +1520,7 @@ async fn watch_records( if !entry.is_stop() { pending = Some(entry); } + claim.pending = pending.is_some(); } moq_mux::hls::Event::Pop(range) => { // HLS assigns consecutive positions within a playlist. Removing an interior @@ -1303,6 +1535,7 @@ async fn watch_records( return Ok(()); } pending = None; + claim.pending = false; broken = true; claim.next = claim.next.max(range.end); rendition.live.clear(); @@ -1723,4 +1956,17 @@ mod tests { .expect("the failure wakes the wait") .unwrap(); } + + // A jump in the numbering, from records skipped across a hand-over, drops the inherited rows + // before it, but never the rendition's own. + #[test] + fn a_numbering_jump_cuts_the_inherited_rows() { + assert_eq!(super::broken(&[0, 1, 2, 3], 2), 0); + assert_eq!(super::broken(&[0, 1, 5, 6], 2), 2, "between inherited and own"); + assert_eq!(super::broken(&[0, 3, 4, 5], 2), 1, "between two predecessors"); + assert_eq!(super::broken(&[0, 3, 7], 2), 2, "the last jump counts"); + assert_eq!(super::broken(&[0, 1, 2, 9], 2), 0, "own rows are never cut"); + assert_eq!(super::broken(&[0, 1], 2), 0); + assert_eq!(super::broken(&[], 0), 0); + } } diff --git a/rs/moq-hls/src/export/renditions.rs b/rs/moq-hls/src/export/renditions.rs index 66229971a5f..5dea7915b41 100644 --- a/rs/moq-hls/src/export/renditions.rs +++ b/rs/moq-hls/src/export/renditions.rs @@ -429,7 +429,7 @@ impl Producer { } /// Start every enabled video rendition's init build, then resolve once one of them has its - /// init and is [advertised](Rendition::is_advertised), so a manifest lists it, or none is left + /// init and lists a segment of its own, so a manifest lists it, or none is left /// that listings do not refuse: a denied or failed one can never be listed. A build that has /// nothing to read from yet starts on the wake that indexes its media. #[cfg_attr( @@ -453,7 +453,7 @@ impl Producer { // Rerun on every wake: `poll_refused` registered `waiter` on its records, so // the rows a build needs wake this wait to start it. rendition.start_init(); - rendition.poll_init(waiter).is_ready() && rendition.poll_advertised(waiter).is_ready() + rendition.poll_init(waiter).is_ready() && rendition.poll_represented(waiter).is_ready() }) { Poll::Ready(()) } else { diff --git a/rs/moq-hls/src/export/segments.rs b/rs/moq-hls/src/export/segments.rs index 38a8b16562e..0ff817ee0ca 100644 --- a/rs/moq-hls/src/export/segments.rs +++ b/rs/moq-hls/src/export/segments.rs @@ -164,6 +164,9 @@ pub(crate) struct Window { pub ended: bool, /// Why the window failed, if it did. pub failed: Option, + /// Whether the window is closed, so these are all the rows it will ever list (see + /// [`Producer::closed`]). + pub closed: bool, } /// The next segment a [`Consumer`] should emit, resolved from the window. @@ -189,6 +192,7 @@ impl State { segments: self.rows.iter().cloned().collect(), ended: self.ended, failed: self.failed.clone(), + closed: false, } } @@ -329,7 +333,11 @@ impl Producer { allow(dead_code, reason = "only the `server` feature uses it") )] pub fn window(&self) -> Window { - self.state.read().window() + let state = self.state.read(); + Window { + closed: state.is_closed(), + ..state.window() + } } /// The row of segment `segment` numbered by the record track tagged `tag`, or `None` if it @@ -387,6 +395,32 @@ impl Producer { } } + /// Poll until the window lists a row, even after it closed: a predecessor whose rows its + /// replacement still lists (see [`Rendition::after`]). + pub fn poll_rows(&self, waiter: &kio::Waiter) -> Poll<()> { + match self.state.poll_ref(waiter, |state| match state.rows.is_empty() { + false => Poll::Ready(()), + true => Poll::Pending, + }) { + Poll::Ready(Ok(())) => Poll::Ready(()), + Poll::Ready(Err(state)) if !state.rows.is_empty() => Poll::Ready(()), + _ => Poll::Pending, + } + } + + /// Resolve once the window is closed: its records watcher lists no more rows. + pub async fn closed(&self) { + self.state.closed().await; + } + + /// Whether the window lists segment `at` of the record track tagged `tag`. + pub fn holds(&self, tag: &str, at: At) -> bool { + match at { + At::Sequence(segment) => self.row(tag, segment).is_some(), + At::Pts(time) => self.segment_number_at(tag, time).is_some(), + } + } + /// Poll until the window fails. #[cfg_attr( not(feature = "server"), diff --git a/rs/moq-hls/src/server/routes.rs b/rs/moq-hls/src/server/routes.rs index 604fd2c1889..f70cb587217 100644 --- a/rs/moq-hls/src/server/routes.rs +++ b/rs/moq-hls/src/server/routes.rs @@ -90,7 +90,7 @@ async fn manifest(server: &Server, broadcast: &str, query: Option<&str>) -> Resp } // A manifest whose renditions are all empty confuses players; give the broadcast a moment // to index its first complete segment before answering. - let _ = kio::clock::timeout(READY_TIMEOUT, broadcaster.playable()).await; + let _ = kio::clock::timeout(READY_TIMEOUT, broadcaster.represented()).await; // Each representation names its init by hash, which an inline codec only learns from media. // A player keeps the adaptation sets of the first manifest it loads, so wait for one video // init rather than every init: a stuck rendition never holds a sibling that can play.