diff --git a/quest/m1/README.md b/quest/m1/README.md index 37cd44f93f7..d7a6f2cfa84 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -149,7 +149,7 @@ Catalog HLS field. - [DASH init readiness](/quest/m1/dash-init-ready.md) - a DASH manifest lists the renditions whose init exists instead of waiting on one that is stuck - [Binding peer version](/quest/m1/binding-peer-version.md) - every binding names itself ahead of the Rust core in the SETUP Implementation parameter - [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - A/A and deliberately slowed Nightly runs set the paired comparison's threshold, so its regression alerts can be trusted -- [HLS export tests over a session](/quest/m1/hls-session-tests.md) - an integration suite records and serves through a real moq-net session, so a publisher disconnect, finish, or takeover ends the cursor and playlists as specified +- [HLS takeover leak](/quest/m1/hls-takeover-leak.md) - a replaced instance's recording cursor never yields the replacement's media, and ends or fails instead of stalling - [HLS and DASH segment conformance](/quest/m1/hls-conformance.md) - nightly ffprobe checks that advertised timing, sync flags, and bytes match the media served - [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; the cache sheds parts of a group under pressure and refills the holes in place - [Metric-safe tiers](/quest/m1/metric-tier.md) - tiers are `[A-Za-z0-9._-]` paths refused where they are set, and every metric label value is escaped diff --git a/quest/m1/hls-session-tests.md b/quest/m1/hls-session-tests.md deleted file mode 100644 index d1dd1c119fa..00000000000 --- a/quest/m1/hls-session-tests.md +++ /dev/null @@ -1,57 +0,0 @@ -# [M] HLS export tests over a session - -## Goal - -A `rs/moq-hls/tests` integration suite records and serves a broadcast whose -publisher sits on the far side of a real moq-net session, and fails when the -export's end-of-recording contract breaks. The suite uses only moq-hls's public -API. - -Each scenario checks both the recording cursor (`segments::Consumer::next`) and -the served playlists: - -- **Publisher disconnect**: the publisher's session drops mid-segment or - mid-catalog. The cursor drains the completed tail, then fails every call with - a typed `TimelineFailed`, `CatalogFailed`, or `SegmentUnavailable`, whichever - failure lands first. It never returns a raw transport error such as - `Unroutable` or `Session(Cancel)`. The media playlist refuses rather than - ending with `EXT-X-ENDLIST`. -- **Clean finish**: the catalog and tracks finish, with an announce End both - before and after catalog EOF. The cursor reads to `None`, and the playlist - carries `EXT-X-ENDLIST`. -- **Restart and takeover**: a new instance takes over the path. The old run's - cursor ends, the new run serves under its own `@{instance}` and generation, - and an old run's URL never serves the new run's bytes. - -Out of scope: - -- HLS import. -- A relay hop between the publisher and the export. The CLI recorder covers - that path end to end. -- Wall-clock or socket transports. - -## Plan - -Decided 2026-10-25 (maintainer interview): - -- **Why**: today's cursor failure tests inject errors into local producers - (`cut()`, `finish(Err(..))`). Only `live_fmp4_segments` and a few - `server/routes.rs` tests cross a session at all. A publisher that really - disconnected, failing the catalog before the timeline, returned the raw - error until [#6694](https://github.com/moq-dev/moq/pull/6694). Only a downstream embedder's end-to-end run caught it. -- **Shape**: integration tests in `rs/moq-hls/tests`, so an API shape an - embedder cannot use fails here too. Promote what the suite needs from the - `LitePair` helper in `server/routes.rs` rather than duplicating it. -- **Transport**: `moq_net::transport::memory::pair` with a `moq_net` `Client` - and `Server`, under `moq_clock`'s simulated clock, as `live_fmp4_segments` - does. A disconnect is dropping a side. No sleeps, no wall clock. -- **Versions**: the default moq-lite version only. -- **Unit tests stay**: the local-producer tests in `export/mod.rs` remain. They - pin the cursor's logic, and this suite pins the session behavior. -- Each scenario must fail on a tree that reintroduces the bug it guards. For - the disconnect race, revert [#6694](https://github.com/moq-dev/moq/pull/6694)'s wrapping and confirm the suite fails. - -## Related - -- [HLS restart drain cutoff](/quest/m2/hls-restart-drain-cutoff.md) - changes what a replaced run's cursor drains; update the takeover scenario with it -- [Test coverage audit](/quest/m2/audit-test-coverage.md) - the recurring audit this gap slipped past diff --git a/quest/m1/hls-takeover-leak.md b/quest/m1/hls-takeover-leak.md new file mode 100644 index 00000000000..34e728ab1b7 --- /dev/null +++ b/quest/m1/hls-takeover-leak.md @@ -0,0 +1,211 @@ +# [M] HLS takeover leaks the replacement's media + +## Goal + +After a new publisher instance takes over a broadcast path, a recording cursor on the +replaced instance never yields the replacement's media. It drains the old run's own +segments, then ends cleanly or fails with a typed error. It never stalls. + +The `takeover` scenario below lands in `rs/moq-hls/tests/integration/restart.rs` with +the fix, as part of the session suite. + +## Plan + +Found 2026-10-25 by the session suite. The takeover scenario fails on `main`. + +**Symptom.** The old run publishes three complete segments and leaves a GOP open at 12s. +A new instance takes over `live` and publishes its own groups from zero. The old run +finishes. The old cursor reads segments 1 and 2 with the old bytes. Segment 3, the old +run's final GOP (group 6), comes back with the new run's bytes. + +- If the new run has not completed its own group 6, the old cursor stalls on segment 3 + instead. +- The leak also happens when the old run finishes before the new one announces. +- It reproduces on moq-lite-06 (the default) and moq-lite-07-wip, with an epochless old + route and a new route with an epoch. +- Over HTTP, the old instance's URLs answer 404 once it is replaced, so only the cursor + leaks. + +**Where the fetch goes.** Temporary logging in `Rendition::fetch` +(`rs/moq-hls/src/export/rendition.rs`) showed the segment-3 fetch going through the +replaced broadcast: its epoch was `None` and `is_closed()` was true. The fetch reads +through the track that `Handle::capture` pinned with a `track.query()` before the +takeover, yet `fetch_group(6)` returned the replacement's group with the same sequence. + +**A bare moq-net reproduction did not leak.** The setup used lite-07, an epochless old +broadcast, and a held `track.query()` taken before the replacement, with: + +- a subscription on the old track, with its in-flight group either read or dropped unread; +- the replacement announced with an epoch and writing groups 0 to 2; +- the old group 1 finished after the takeover; +- in one variant, a concurrent subscription to the replacement. + +`fetch_group(1)` on the old query returned the old bytes. Groups the old subscription +never delivered answered `Unroutable`. So the leak needs something the export adds. + +**Where I looked:** + +- `rs/moq-hls/src/export/rendition.rs`: `Media`, `Handle::capture`, `poll_sync`, + `poll_track`, `Rendition::fetch`, and the free `fetch`. Pinning and window-ending on + close apply only to siblings (`self.sibling`). The catalog's own broadcast relies on + the source pin. +- `rs/moq-hls/src/export/segments.rs`: `Consumer::poll_feed`. The cursor's feed + subscribes through the binding (`poll_subscribe`, `FEED_MAX_DELAY`) and drops each + group unread. Its doc says a resumed subscription "only pulls while it is read". +- `rs/moq-mux/src/source.rs`: `Source::new` pins the catalog path to the served + instance; `Binding::poll_broadcast` returns the pin. +- `rs/moq-net/tests/integration/restart.rs`: `retired_fetch` covers a retired handle + whose query is taken after the restart, not one held before it. +- `rs/moq-net/src/lite/subscriber.rs`: `SubscriptionCleanup` tears subscriptions down in + `HashMap` order. That explains why the disconnect failure order varies, not this leak. + +**Leading hypothesis, unconfirmed:** the feed subscription, or a resubscribe from the +pinned query, resumes across the route change onto the replacement and fills the +retired track's cache for a group the old run had not delivered yet. Confirm by logging +which subscription delivers group 6 into the old track, then decide whether moq-net +refuses the resume or the export stops reading a replaced instance's tail. + +The takeover scenario uses the suite's `support` harness and needs a `contains` helper, +which was dropped from `support.rs` when the scenario moved here: + +```rust +//! A new publisher instance taking over a path is served under its own URLs, while the run it +//! replaced finishes its own recording and never serves the new run's bytes. + +use std::time::Duration; + +use axum::http::StatusCode; + +use crate::support::*; + +const OLD: u8 = 1; +const NEW: u8 = 2; + +/// The old run keeps publishing through the takeover and ends after it. Until a replaced run's +/// drain has a cutoff of its own, its cursor ends only once the old publisher does. +#[tokio::test(start_paused = true)] +async fn takeover() { + let pair = Pair::connect().await; + let mut old = Publisher::new(&pair.publisher, "live", Default::default()); + let server = pair.server(); + let app = server.router(); + let rendition = rendition(&server, "live").await; + let mut cursor = rendition.segments(); + + old.write(0, 13, OLD); + let old_media = media_uri(&app, "live").await; + listed(&app, &old_media, ".2.m4s").await; + let replaced = server.broadcaster("live").await.expect("announced"); + replaced.set_generation(Some("run-1")).expect("generation"); + let playlist = listed(&app, &old_media, "seg/run-1.").await; + let old_segments: Vec<_> = (0..3).map(|index| segment_uri(&old_media, &playlist, index)).collect(); + for uri in &old_segments { + assert_old(&app, uri).await; + } + // Reading, as a recorder does, so the cursor's subscription carries every group across. + let first = moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the first segment") + .expect("records") + .expect("a segment"); + assert_eq!(first.segment, 0); + assert!(contains(&first.media, &[OLD; 8])); + + // The new instance numbers its groups and segments from zero again. + let route = moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()); + let mut new = Publisher::new(&pair.publisher, "live", route); + let deadline = moq_clock::now() + TIMEOUT; + let current = loop { + let current = server.broadcaster("live").await.expect("announced"); + if !std::sync::Arc::ptr_eq(¤t, &replaced) { + break current; + } + assert!(moq_clock::now() < deadline, "the new instance never took over"); + moq_clock::sleep(Duration::from_millis(50)).await; + }; + current.set_generation(Some("run-2")).expect("generation"); + new.write(0, 13, NEW); + + // The master lists the new instance, whose playlist lists its own generation. + let new_media = media_uri(&app, "live").await; + assert_ne!(new_media, old_media, "a new instance gets its own URLs"); + let playlist = listed(&app, &new_media, ".2.m4s").await; + assert!(!playlist.contains("run-1."), "{playlist}"); + let new_segment = segment_uri(&new_media, &playlist, 0); + assert!(new_segment.contains("/seg/run-2."), "{new_segment}"); + let response = get(&app, &new_segment).await; + assert_eq!(response.status(), StatusCode::OK); + assert!(contains(response.body(), &[NEW; 8]) && !contains(response.body(), &[OLD; 8])); + // Nor does the new instance answer the old run's generation. + let crossed = new_segment.replace("/seg/run-2.", "/seg/run-1."); + assert_eq!(get(&app, &crossed).await.status(), StatusCode::NOT_FOUND); + + // The old run's URLs, under either generation, never answer with the new run's bytes. + for uri in &old_segments { + assert_old(&app, uri).await; + assert_old(&app, &uri.replace("/seg/run-1.", "/seg/run-2.")).await; + } + + // The old run ends while the new one carries on past the old run's last group. + old.finish(); + new.write(13, 21, NEW); + drop(old); + + // The old recording drains every segment of its own, its final GOP included, and nothing else. + let mut read = Vec::new(); + loop { + let next = moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the replaced run's cursor never ended"); + let Some(segment) = next.expect("the replaced run finished cleanly") else { + break; + }; + assert!( + contains(&segment.media, &[OLD; 8]) && !contains(&segment.media, &[NEW; 8]), + "the replaced run's segment {} carries the new run's media", + segment.segment + ); + read.push(segment.segment); + } + assert_eq!(read, [1, 2, 3]); + let playlist = rendition.media_playlist(None).expect("finished").expect("listed"); + assert!(playlist.ends_with("#EXT-X-ENDLIST\n"), "{playlist}"); + + for uri in &old_segments { + assert_old(&app, uri).await; + } + assert_eq!(get(&app, &new_segment).await.status(), StatusCode::OK); + drop(new); +} + +/// `uri` answers the old run's bytes or nothing at all. +async fn assert_old(app: &axum::Router, uri: &str) { + let response = get(app, uri).await; + assert!(!contains(response.body(), &[NEW; 8]), "{uri} served the new run"); + match response.status() { + StatusCode::OK => assert!(contains(response.body(), &[OLD; 8]), "{uri}"), + _ => assert_missing(&response), + } +} + +/// The URI of the `index`th segment `playlist` lists, beside the media playlist at `media`. +fn segment_uri(media: &str, playlist: &str, index: usize) -> String { + let line = playlist + .lines() + .filter(|line| line.starts_with("seg/")) + .nth(index) + .unwrap_or_else(|| panic!("no segment {index} in {playlist}")); + media.replace("media.m3u8", line) +} + +/// Whether `haystack` contains `needle`. +fn contains(haystack: &[u8], needle: &[u8]) -> bool { + haystack.windows(needle.len()).any(|window| window == needle) +} +``` + +## Related + +- [HLS restart drain cutoff](/quest/m2/hls-restart-drain-cutoff.md) - changes what a replaced run's cursor drains; update the takeover scenario with it +- `export hls --output` (`rs/moq-cli/src/hls/record.rs`) - the recorder that would write the leaked segment +- moq-hls's `moq-tokio` dev-dependency features (`tcp`, `quic`, `aws-lc-rs`) now serve only `origin::spawn`. Trimming them to `net` first needs moq-tokio's `Backoff::validate` fixed, since it is dead code under `net` alone. diff --git a/rs/moq-hls/src/server/routes.rs b/rs/moq-hls/src/server/routes.rs index de2ae00f2a1..d5109355688 100644 --- a/rs/moq-hls/src/server/routes.rs +++ b/rs/moq-hls/src/server/routes.rs @@ -235,16 +235,6 @@ mod tests { moq_net::origin::Route::default().with_epoch(RUN.parse().unwrap()) } - /// The instance `server` lists for `broadcast`, which it mints for a session older than - /// lite-07, since those carry no epoch. - async fn listed(server: &Server, broadcast: &str) -> String { - let broadcaster = tokio::time::timeout(TIMEOUT, server.broadcaster(broadcast)) - .await - .expect("resolve timed out") - .expect("announced"); - broadcaster.instance().to_string() - } - fn vp8_frame(micros: u64, keyframe: bool) -> moq_mux::container::Frame { let payload = if keyframe { &[0x10, 0x00, 0x00, 0x9d, 0x01, 0x2a, 0x40, 0x01, 0xf0, 0x00][..] @@ -267,71 +257,6 @@ mod tests { config } - /// A moq-lite publisher↔subscriber pair over loopback TCP, so a FETCH actually - /// crosses a session. - struct LitePair { - pub_origin: moq_net::origin::Producer, - sub_origin: moq_net::origin::Producer, - _connection: moq_tokio::Connection, - accept: tokio::task::JoinHandle<()>, - } - - async fn lite_pair() -> LitePair { - lite_pair_pub(moq_net::Hop::random()).await - } - - async fn lite_pair_pub(config: impl Into) -> LitePair { - lite_pair_on("moq-lite-05", config).await - } - - async fn lite_pair_on(version: &str, config: impl Into) -> LitePair { - let lite: moq_net::Version = version.parse().expect("lite version"); - let pub_origin = moq_tokio::origin::spawn_config(config.into()); - let sub_origin = moq_tokio::origin::spawn(); - - let mut listen = moq_tokio::listen::Config::default(); - listen.tcp.bind = Some("127.0.0.1:0".parse().expect("addr")); - listen.version = vec![lite]; - let mut server = listen - .init(Default::default()) - .expect("server") - .listen() - .await - .expect("listen"); - let addr = server.tcp_local_addr().expect("tcp listener bound"); - - let pub_for_accept = pub_origin.clone(); - let accept = tokio::spawn(async move { - let request = server.accept().await.expect("accept"); - let session = request - .with_publisher(&pub_for_accept) - .ok() - .await - .expect("publisher session"); - let _ = session.closed().await; - }); - - let mut connect = moq_tokio::connect::Config::default(); - connect.version = vec![lite]; - let client = connect - .init(Default::default()) - .expect("client") - .with_subscriber(sub_origin.clone()) - .with_reconnect(false); - let url: url::Url = format!("tcp://{addr}/").parse().expect("url"); - let connection = tokio::time::timeout(TIMEOUT, client.connect(url).established()) - .await - .expect("connect timed out") - .expect("connect"); - - LitePair { - pub_origin, - sub_origin, - _connection: connection, - accept, - } - } - async fn oneshot(app: axum::Router, uri: &str) -> axum::http::Response { use tower::ServiceExt; app.oneshot( @@ -396,123 +321,6 @@ mod tests { media.write(vp8_frame(4_000_000, true)).unwrap(); } - /// An embedder parses the path, rewrites the broadcast into its own scope, and answers - /// through `respond` with the query propagated to every child URL. - #[tokio::test] - async fn respond_serves_a_route_rewritten_into_scope() { - let pair = lite_pair().await; - let broadcast = pair.pub_origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); - let (_catalog, _registration, _track, mut media) = - publish_video(&mut broadcast.broadcast().clone(), video_config(), timed()); - write_three_gops(&mut media); - - let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); - let run = listed(&server, "live").await; - let mut route = Route::parse(&format!("/tenant/live/@{run}/video/video0/media.m3u8")).expect("media route"); - route.broadcast = route.broadcast.strip_prefix("tenant/").expect("scoped").to_string(); - - let deadline = tokio::time::Instant::now() + TIMEOUT; - let body = loop { - let response = server.respond(&route, Some("jwt=abc")).await; - if response.status() == StatusCode::OK { - let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap(); - let body = String::from_utf8(body.to_vec()).unwrap(); - if body.contains("seg/656e3d1b.0.m4s") { - break body; - } - } - assert!( - tokio::time::Instant::now() < deadline, - "media playlist never listed a segment" - ); - tokio::time::sleep(Duration::from_millis(50)).await; - }; - assert!(body.contains(".mp4?jwt=abc\""), "{body}"); - assert!(body.contains("seg/656e3d1b.0.m4s?jwt=abc"), "{body}"); - - pair.accept.abort(); - } - - /// A recorder's cursor gets the media of a publisher whose moq-lite has no FETCH: the - /// cursor's own subscription carries every group to the subscriber's cache. - #[tokio::test] - async fn a_cursor_records_a_publisher_without_fetch() { - for version in [ - "moq-lite-01", - "moq-lite-02", - "moq-lite-03", - "moq-lite-04", - "moq-lite-05", - ] { - let pair = lite_pair_on(version, moq_net::Hop::random()).await; - let broadcast = pair.pub_origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); - // A hinted timeline is declared before any media, so the rendition is listed early. - let hinted = moq_mux::timeline::Config::default().with_duration_hint(Duration::from_secs(2)); - let catalog = moq_mux::catalog::Producer::new( - &mut broadcast.broadcast().clone(), - moq_mux::catalog::Config::default().with_timeline(hinted), - ) - .unwrap(); - let reserved = catalog.reserve(); - let track = broadcast.create_track("video0", timed()).unwrap(); - let mut media = reserved - .video( - track.clone(), - moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), - video_config(), - ) - .unwrap(); - drop(reserved); - - let origin = pair.sub_origin.consume(); - tokio::time::timeout(TIMEOUT, origin.routed("live")) - .await - .expect("announce timed out") - .expect("routed"); - let remote = tokio::time::timeout(TIMEOUT, origin.request_broadcast("live", None)) - .await - .expect("request timed out") - .expect("remote broadcast"); - let source = moq_mux::Source::new(origin, remote); - let broadcaster = crate::export::Broadcaster::new(source, crate::export::Config::default()); - // Recorded as soon as the catalog lists it with its timeline, as a recorder does. - let mut renditions = broadcaster.renditions(); - let rendition = match tokio::time::timeout(TIMEOUT, renditions.next()).await { - Ok(Ok(Some(crate::export::renditions::Event::Added(rendition)))) => rendition, - _ => panic!("{version}: the catalog never listed the rendition"), - }; - let mut cursor = rendition.segments(); - let mut next = std::pin::pin!(cursor.next()); - // Waiting for its first segment, the cursor subscribes before any media exists. - let mut demand = track.demand(); - let wanted = async { - while demand.wanted_changed().await?.is_none() {} - moq_net::Result::Ok(()) - }; - tokio::time::timeout(TIMEOUT, async { - tokio::select! { - _ = next.as_mut() => panic!("{version}: a segment before any media"), - wanted = wanted => wanted.expect("track open"), - } - }) - .await - .unwrap_or_else(|_| panic!("{version}: the cursor never subscribed")); - - write_three_gops(&mut media); - let segment = tokio::time::timeout(TIMEOUT, next) - .await - .unwrap_or_else(|_| panic!("{version}: no segment recorded")) - .unwrap_or_else(|err| panic!("{version}: {err}")) - .expect("a segment"); - assert_eq!(segment.segment, 0, "{version}"); - assert!(!segment.media.expect("media, not a gap").is_empty(), "{version}"); - - pair.accept.abort(); - } - } - async fn status(app: &axum::Router, uri: &str) -> StatusCode { oneshot(app.clone(), uri).await.status() } @@ -708,139 +516,6 @@ mod tests { ); } - /// A miss that crossed a moq-lite session answers 404, not 500. - #[tokio::test] - async fn a_session_crossed_cache_miss_answers_404() { - let pool = moq_net::cache::Pool::new(moq_net::cache::Config::default().with_capacity(1)); - let pair = lite_pair_pub({ - let mut config = moq_net::origin::Config::default(); - config.pool = pool; - config - }) - .await; - let broadcast = pair.pub_origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); - let (_catalog, _registration, _track, mut media) = - publish_video(&mut broadcast.broadcast().clone(), video_config(), timed()); - write_three_gops(&mut media); - - let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); - let run = listed(&server, "live").await; - let app = server.router(); - wait_listed( - &app, - &format!("/live/@{run}/video/video0/media.m3u8"), - "seg/656e3d1b.0.m4s", - ) - .await; - - let response = oneshot(app, &format!("/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; - assert_eq!(response.status(), StatusCode::NOT_FOUND); - assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); - - pair.accept.abort(); - } - - #[tokio::test] - async fn a_disconnected_publisher_on_the_catalog_broadcast_answers_404() { - let pair = lite_pair().await; - let broadcast = pair.pub_origin.create_broadcast("live").expect("publish"); - broadcast.announce(Default::default()).expect("announce"); - let (catalog, registration, track, mut media) = - publish_video(&mut broadcast.broadcast().clone(), video_config(), timed()); - write_three_gops(&mut media); - - let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); - let run = listed(&server, "live").await; - let app = server.router(); - wait_listed( - &app, - &format!("/live/@{run}/video/video0/media.m3u8"), - "seg/656e3d1b.0.m4s", - ) - .await; - - let remote = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().request_broadcast("live", None)) - .await - .expect("remote resolve timed out") - .expect("remote broadcast"); - drop((catalog, registration, track, media, broadcast)); - tokio::time::timeout(TIMEOUT, remote.closed()) - .await - .expect("publisher close timed out"); - - let response = oneshot(app, &format!("/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; - assert_eq!(response.status(), StatusCode::NOT_FOUND); - assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); - - pair.accept.abort(); - } - - /// The server watches its whole scope, so a lookup rides one request to the peer - /// rather than opening its own for every path an HTTP client names. - #[tokio::test] - async fn lookups_ride_the_servers_scope_watch() { - let pair = lite_pair().await; - let other = pair.pub_origin.create_broadcast("room/other").expect("other"); - other.announce(Default::default()).expect("announce other"); - - let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); - let missing = tokio::time::timeout(TIMEOUT, server.broadcaster("room/missing")) - .await - .expect("lookup timed out"); - assert!(missing.is_none()); - - // Nothing watches `room/other` on its own, yet the synced lookup already discovered it. - pair.sub_origin - .consume() - .request_broadcast("room/other", None) - .await - .expect("discovered through the scope watch"); - - drop(other); - pair.accept.abort(); - } - - /// A client asks its peer only about watched paths, so a sibling no listing covers is - /// asked about before it is requested, on every lite version, including those whose - /// announce stream marks no initial set. - #[tokio::test] - async fn a_sibling_resolves_through_a_client_session() { - for version in [ - "moq-lite-01", - "moq-lite-02", - "moq-lite-03", - "moq-lite-04", - "moq-lite-05", - "moq-lite-06", - "moq-lite-07-wip", - ] { - let pair = lite_pair_on(version, moq_net::origin::Config::default()).await; - let catalog = pair.pub_origin.create_broadcast("room/live").expect("catalog"); - catalog.announce(Default::default()).expect("announce catalog"); - let media = pair.pub_origin.create_broadcast("room/source").expect("media"); - media.announce(Default::default()).expect("announce media"); - - let origin = pair.sub_origin.consume(); - let broadcast = tokio::time::timeout(TIMEOUT, origin.routed_broadcast("room/live")) - .await - .expect("catalog resolve timed out") - .expect("catalog broadcast"); - let source = moq_mux::Source::new(origin, broadcast); - let sibling = tokio::time::timeout( - TIMEOUT, - source.resolve(Some(&moq_net::path::Relative::new("./source")), None), - ) - .await - .expect("sibling resolve timed out") - .unwrap_or_else(|err| panic!("{version}: sibling broadcast: {err}")); - assert!(!sibling.is_closed(), "{version}"); - - drop((catalog, media)); - pair.accept.abort(); - } - } - #[tokio::test(start_paused = true)] async fn denied_sibling_media_answers_403_without_poisoning_allowed_siblings() { let origin = moq_tokio::origin::spawn(); @@ -958,63 +633,4 @@ mod tests { assert!(retained.latest_init().await.unwrap().is_some()); assert!(retained.segment(&first(&retained)).await.unwrap().is_some()); } - - #[tokio::test] - async fn a_disconnected_sibling_publisher_answers_404() { - let pair = lite_pair().await; - let catalog_broadcast = pair.pub_origin.create_broadcast("room/live").expect("catalog"); - catalog_broadcast - .announce(Default::default()) - .expect("announce catalog"); - let media_broadcast = pair.pub_origin.create_broadcast("room/source").expect("media"); - media_broadcast.announce(Default::default()).expect("announce media"); - - let catalog = moq_mux::catalog::Producer::new( - &mut catalog_broadcast.broadcast().clone(), - moq_mux::catalog::Config::default(), - ) - .unwrap(); - let reserved = catalog.reserve(); - let mut config = video_config(); - config.broadcast = Some(moq_net::path::Relative::new("./source").to_owned()); - let track = media_broadcast.create_track("video0", timed()).unwrap(); - let mut media = reserved - .video( - track, - moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), - config, - ) - .unwrap(); - drop(reserved); - write_three_gops(&mut media); - - let server = Server::new(pair.sub_origin.consume(), crate::export::Config::default()); - let run = listed(&server, "room/live").await; - let app = server.router(); - wait_listed( - &app, - &format!("/room/live/@{run}/video/video0/media.m3u8"), - "seg/656e3d1b.0.m4s", - ) - .await; - - // Waited on rather than requested once: a client only asks about what it watches. - let remote_media = tokio::time::timeout(TIMEOUT, pair.sub_origin.consume().routed_broadcast("room/source")) - .await - .expect("sibling resolve timed out") - .expect("sibling broadcast"); - // Only the sibling goes: the catalog keeps listing the rendition, so the request - // reaches the media fetch rather than missing the rendition. - media_broadcast.close(); - tokio::time::timeout(TIMEOUT, remote_media.closed()) - .await - .expect("sibling close timed out"); - - let response = oneshot(app, &format!("/room/live/@{run}/video/video0/seg/656e3d1b.0.m4s")).await; - assert_eq!(response.status(), StatusCode::NOT_FOUND); - assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); - - drop((catalog, catalog_broadcast, media)); - pair.accept.abort(); - } } diff --git a/rs/moq-hls/tests/integration/disconnect.rs b/rs/moq-hls/tests/integration/disconnect.rs new file mode 100644 index 00000000000..d69adb36617 --- /dev/null +++ b/rs/moq-hls/tests/integration/disconnect.rs @@ -0,0 +1,108 @@ +//! A publisher whose session drops leaves an incomplete recording: the cursor drains what +//! already arrived, then fails with a typed error, and the playlist refuses. + +use axum::http::StatusCode; + +use crate::support::*; + +/// Where in the publisher's output the session drops. +#[derive(Clone, Copy, Debug)] +enum At { + /// With a GOP open past the last completed segment. + Segment, + /// Right after a catalog update, before it reaches the export. + Catalog, +} + +/// Drop the publisher's session at `at` and check the recording ends as incomplete. +/// +/// The catalog's and the timeline's subscriptions both fail with the session. Whichever reaches +/// the cursor first must end it with a typed error, so the order is left to the session. +async fn disconnect(at: At) { + let mut pair = Pair::connect().await; + let mut publisher = Publisher::new(&pair.publisher, "live", Default::default()); + let server = pair.server(); + let app = server.router(); + let rendition = rendition(&server, "live").await; + // Recording from the start, as a recorder does, so every group reaches the export. + let mut cursor = rendition.segments(); + + // Three completed segments, then a GOP still open at 12s. + publisher.write(0, 13, 1); + let media = media_uri(&app, "live").await; + let playlist = listed(&app, &media, ".2.m4s").await; + assert_eq!(numbers(&playlist), [0, 1, 2], "{playlist}"); + let first = moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the first segment") + .expect("records") + .expect("a segment"); + assert_eq!(first.segment, 0); + + if let At::Catalog = at { + publisher + .catalog + .mutate(|catalog| { + let config = catalog.video.renditions.get_mut(TRACK).expect("listed"); + config.coded_width = Some(640); + config.coded_height = Some(480); + }) + .expect("catalog update"); + } + pair.disconnect(); + + // The completed tail drains, and the open GOP is never read as a segment. + let (read, end) = drain(&mut cursor).await; + assert_eq!(read, [1, 2], "{at:?}"); + let err = end.expect_err("an incomplete recording never reads as complete"); + // A raw transport error would read as retryable, so a recorder would never finalize. + assert!( + matches!( + err, + moq_hls::Error::CatalogFailed { .. } + | moq_hls::Error::TimelineFailed { .. } + | moq_hls::Error::SegmentUnavailable { .. } + ), + "{at:?}: a disconnect ended the recording with {err:?}" + ); + // The failure is final: every later call answers it again. + for _ in 0..2 { + let again = moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the cursor stalled"); + let Err(again) = again else { + panic!("{at:?}: a failed cursor resumed"); + }; + assert_eq!(again.to_string(), err.to_string(), "{at:?}"); + } + + // The playlist refuses rather than ending with `EXT-X-ENDLIST`. + let refused = rendition + .media_playlist(None) + .expect_err("the playlist refuses rather than ending"); + assert!( + matches!( + refused, + moq_hls::Error::CatalogFailed { .. } | moq_hls::Error::TimelineFailed { .. } + ), + "{at:?}: {refused:?}" + ); + let response = get(&app, &media).await; + assert_ne!( + response.status(), + StatusCode::OK, + "{at:?}: {}", + String::from_utf8_lossy(response.body()) + ); + drop(publisher); +} + +#[tokio::test(start_paused = true)] +async fn mid_segment() { + disconnect(At::Segment).await; +} + +#[tokio::test(start_paused = true)] +async fn mid_catalog() { + disconnect(At::Catalog).await; +} diff --git a/rs/moq-hls/tests/integration/finish.rs b/rs/moq-hls/tests/integration/finish.rs new file mode 100644 index 00000000000..02dd28aa357 --- /dev/null +++ b/rs/moq-hls/tests/integration/finish.rs @@ -0,0 +1,84 @@ +//! A publisher that finishes cleanly leaves a complete recording, whichever of its announce +//! `End` and its catalog's EOF reaches the export first. + +use std::time::Duration; + +use axum::http::StatusCode; + +use crate::support::*; + +/// Which reaches the export first. +#[derive(Clone, Copy, Debug)] +enum Order { + /// The catalog's EOF, then the announce `End`. + EndAfterEof, + /// The announce `End`, then the catalog's EOF. + EndBeforeEof, +} + +async fn finish(order: Order) { + let pair = Pair::connect().await; + let mut publisher = Publisher::new(&pair.publisher, "live", Default::default()); + let server = pair.server(); + let app = server.router(); + let rendition = rendition(&server, "live").await; + let mut cursor = rendition.segments(); + + publisher.write(0, 13, 1); + let media = media_uri(&app, "live").await; + listed(&app, &media, ".2.m4s").await; + // Reading, as a recorder does, so the cursor's subscription carries every group across. + let first = moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the first segment") + .expect("records") + .expect("a segment"); + assert_eq!(first.segment, 0); + + match order { + Order::EndAfterEof => { + publisher.finish(); + // The served playlist ends while the broadcast is still announced. + let playlist = listed(&app, &media, "#EXT-X-ENDLIST").await; + assert_eq!(numbers(&playlist), [0, 1, 2, 3], "{playlist}"); + publisher.broadcast.close(); + } + Order::EndBeforeEof => { + publisher.media.finish().expect("finish media"); + publisher.broadcast.close(); + // The server stops serving the instance once `End` reaches it. + let deadline = moq_clock::now() + TIMEOUT; + while get(&app, &media).await.status() != StatusCode::NOT_FOUND { + assert!(moq_clock::now() < deadline, "End never reached the export"); + moq_clock::sleep(Duration::from_millis(50)).await; + } + // `End` alone confirms nothing: the recording stays open until the catalog's EOF. + let playlist = rendition.media_playlist(None).expect("live").expect("listed"); + assert!(!playlist.contains("#EXT-X-ENDLIST"), "{playlist}"); + publisher.catalog.finish().expect("finish catalog"); + } + } + + // The final GOP is promoted to the last segment, and the recording reads to its end. + let (read, end) = drain(&mut cursor).await; + end.unwrap_or_else(|err| panic!("{order:?}: {err}")); + assert_eq!(read, [1, 2, 3], "{order:?}"); + let again = moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the cursor stalled"); + assert!(matches!(again, Ok(None)), "{order:?}: a finished cursor stays finished"); + + let playlist = rendition.media_playlist(None).expect("finished").expect("listed"); + assert!(playlist.ends_with("#EXT-X-ENDLIST\n"), "{order:?}: {playlist}"); + assert_eq!(numbers(&playlist), [0, 1, 2, 3], "{order:?}"); +} + +#[tokio::test(start_paused = true)] +async fn end_after_catalog_eof() { + finish(Order::EndAfterEof).await; +} + +#[tokio::test(start_paused = true)] +async fn end_before_catalog_eof() { + finish(Order::EndBeforeEof).await; +} diff --git a/rs/moq-hls/tests/integration/main.rs b/rs/moq-hls/tests/integration/main.rs new file mode 100644 index 00000000000..4016ac1a3b9 --- /dev/null +++ b/rs/moq-hls/tests/integration/main.rs @@ -0,0 +1,9 @@ +//! The export over a real moq-net session: a recording cursor and the served playlists for a +//! publisher on the far side of one, through moq-hls's public API only. + +#![cfg(feature = "server")] + +mod disconnect; +mod finish; +mod server; +mod support; diff --git a/rs/moq-hls/tests/integration/server.rs b/rs/moq-hls/tests/integration/server.rs new file mode 100644 index 00000000000..1cc94e6c780 --- /dev/null +++ b/rs/moq-hls/tests/integration/server.rs @@ -0,0 +1,239 @@ +//! The HTTP serve path and the export's lookups, for broadcasts that cross a session. + +use axum::http::StatusCode; +use moq_hls::export::{Broadcaster, Route, renditions}; + +use crate::support::*; + +/// An embedder parses the path, rewrites the broadcast into its own scope, and answers through +/// `respond` with the query propagated to every child URL. +#[tokio::test(start_paused = true)] +async fn respond_serves_a_route_rewritten_into_scope() { + let pair = Pair::connect().await; + let mut publisher = Publisher::new(&pair.publisher, "live", Default::default()); + publisher.write(0, 5, 1); + + let server = pair.server(); + let media = media_uri(&server.router(), "live").await; + let mut route = Route::parse(&format!("/tenant{media}")).expect("media route"); + route.broadcast = route.broadcast.strip_prefix("tenant/").expect("scoped").to_string(); + + let deadline = moq_clock::now() + TIMEOUT; + let body = loop { + let response = server.respond(&route, Some("jwt=abc")).await; + if response.status() == StatusCode::OK { + let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let body = String::from_utf8(body.to_vec()).unwrap(); + if body.contains("seg/656e3d1b.0.m4s") { + break body; + } + } + assert!(moq_clock::now() < deadline, "media playlist never listed a segment"); + moq_clock::sleep(std::time::Duration::from_millis(50)).await; + }; + assert!(body.contains(".mp4?jwt=abc\""), "{body}"); + assert!(body.contains("seg/656e3d1b.0.m4s?jwt=abc"), "{body}"); +} + +/// A recorder's cursor gets the media of a publisher whose moq-lite has no FETCH: the cursor's +/// own subscription carries every group to the subscriber's cache. +#[tokio::test(start_paused = true)] +async fn a_cursor_records_a_publisher_without_fetch() { + for version in [ + "moq-lite-01", + "moq-lite-02", + "moq-lite-03", + "moq-lite-04", + "moq-lite-05", + ] { + let pair = Pair::new(version.parse().unwrap(), Default::default()).await; + let mut publisher = Publisher::new(&pair.publisher, "live", Default::default()); + + let origin = pair.subscriber.consume(); + moq_clock::timeout(TIMEOUT, origin.routed("live")) + .await + .expect("announce timed out") + .expect("routed"); + let remote = moq_clock::timeout(TIMEOUT, origin.request_broadcast("live", None)) + .await + .expect("request timed out") + .expect("remote broadcast"); + let broadcaster = Broadcaster::new(moq_mux::Source::new(origin, remote), Default::default()); + // Recorded as soon as the catalog lists it with its timeline, as a recorder does. + let mut renditions = broadcaster.renditions(); + let rendition = match moq_clock::timeout(TIMEOUT, renditions.next()).await { + Ok(Ok(Some(renditions::Event::Added(rendition)))) => rendition, + _ => panic!("{version}: the catalog never listed the rendition"), + }; + let mut cursor = rendition.segments(); + let mut next = std::pin::pin!(cursor.next()); + // Waiting for its first segment, the cursor subscribes before any media exists. + let mut demand = publisher.media.demand(); + let wanted = async { + while demand.wanted_changed().await?.is_none() {} + moq_net::Result::Ok(()) + }; + moq_clock::timeout(TIMEOUT, async { + tokio::select! { + _ = next.as_mut() => panic!("{version}: a segment before any media"), + wanted = wanted => wanted.expect("track open"), + } + }) + .await + .unwrap_or_else(|_| panic!("{version}: the cursor never subscribed")); + + publisher.write(0, 5, 1); + let segment = moq_clock::timeout(TIMEOUT, next) + .await + .unwrap_or_else(|_| panic!("{version}: no segment recorded")) + .unwrap_or_else(|err| panic!("{version}: {err}")) + .expect("a segment"); + assert_eq!(segment.segment, 0, "{version}"); + assert!(!segment.media.expect("media, not a gap").is_empty(), "{version}"); + } +} + +/// A miss that crossed a moq-lite session answers 404, not 500. +#[tokio::test(start_paused = true)] +async fn a_session_crossed_cache_miss_answers_404() { + let mut config = moq_net::origin::Config::default(); + config.pool = moq_net::cache::Pool::new(moq_net::cache::Config::default().with_capacity(1)); + let version = *moq_net::Versions::default().iter().next().unwrap(); + let pair = Pair::new(version, config).await; + let mut publisher = Publisher::new(&pair.publisher, "live", Default::default()); + publisher.write(0, 5, 1); + + let app = pair.server().router(); + let media = media_uri(&app, "live").await; + listed(&app, &media, "seg/656e3d1b.0.m4s").await; + assert_missing(&get(&app, &media.replace("media.m3u8", "seg/656e3d1b.0.m4s")).await); +} + +#[tokio::test(start_paused = true)] +async fn a_disconnected_publisher_on_the_catalog_broadcast_answers_404() { + let pair = Pair::connect().await; + let mut publisher = Publisher::new(&pair.publisher, "live", Default::default()); + publisher.write(0, 5, 1); + + let app = pair.server().router(); + let media = media_uri(&app, "live").await; + listed(&app, &media, "seg/656e3d1b.0.m4s").await; + + let remote = moq_clock::timeout(TIMEOUT, pair.subscriber.consume().request_broadcast("live", None)) + .await + .expect("remote resolve timed out") + .expect("remote broadcast"); + drop(publisher); + moq_clock::timeout(TIMEOUT, remote.closed()) + .await + .expect("publisher close timed out"); + + assert_missing(&get(&app, &media.replace("media.m3u8", "seg/656e3d1b.0.m4s")).await); +} + +/// The server watches its whole scope, so a lookup rides one request to the peer rather than +/// opening its own for every path an HTTP client names. +#[tokio::test(start_paused = true)] +async fn lookups_ride_the_servers_scope_watch() { + let pair = Pair::connect().await; + let other = pair.publisher.create_broadcast("room/other").expect("other"); + other.announce(Default::default()).expect("announce other"); + + let server = pair.server(); + let missing = moq_clock::timeout(TIMEOUT, server.broadcaster("room/missing")) + .await + .expect("lookup timed out"); + assert!(missing.is_none()); + + // Nothing watches `room/other` on its own, yet the synced lookup already discovered it. + pair.subscriber + .consume() + .request_broadcast("room/other", None) + .await + .expect("discovered through the scope watch"); +} + +/// A client asks its peer only about watched paths, so a sibling no listing covers is asked about +/// before it is requested, on every lite version, including those whose announce stream marks no +/// initial set. +#[tokio::test(start_paused = true)] +async fn a_sibling_resolves_through_a_client_session() { + for version in [ + "moq-lite-01", + "moq-lite-02", + "moq-lite-03", + "moq-lite-04", + "moq-lite-05", + "moq-lite-06", + "moq-lite-07-wip", + ] { + let pair = Pair::new(version.parse().unwrap(), Default::default()).await; + let catalog = pair.publisher.create_broadcast("room/live").expect("catalog"); + catalog.announce(Default::default()).expect("announce catalog"); + let media = pair.publisher.create_broadcast("room/source").expect("media"); + media.announce(Default::default()).expect("announce media"); + + let origin = pair.subscriber.consume(); + let broadcast = moq_clock::timeout(TIMEOUT, origin.routed_broadcast("room/live")) + .await + .expect("catalog resolve timed out") + .expect("catalog broadcast"); + let source = moq_mux::Source::new(origin, broadcast); + let sibling = moq_clock::timeout( + TIMEOUT, + source.resolve(Some(&moq_net::path::Relative::new("./source")), None), + ) + .await + .expect("sibling resolve timed out") + .unwrap_or_else(|err| panic!("{version}: sibling broadcast: {err}")); + assert!(!sibling.is_closed(), "{version}"); + } +} + +#[tokio::test(start_paused = true)] +async fn a_disconnected_sibling_publisher_answers_404() { + let pair = Pair::connect().await; + let catalog_broadcast = pair.publisher.create_broadcast("room/live").expect("catalog"); + catalog_broadcast + .announce(Default::default()) + .expect("announce catalog"); + let media_broadcast = pair.publisher.create_broadcast("room/source").expect("media"); + media_broadcast.announce(Default::default()).expect("announce media"); + + let catalog = + moq_mux::catalog::Producer::new(&mut catalog_broadcast.broadcast().clone(), Default::default()).unwrap(); + let reserved = catalog.reserve(); + let mut config = video_config(); + config.broadcast = Some(moq_net::path::Relative::new("./source").to_owned()); + let track = media_broadcast.create_track(TRACK, timed()).unwrap(); + let mut media = reserved + .video( + track, + moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), + config, + ) + .unwrap(); + drop(reserved); + for second in 0..5 { + media.write(frame(second, 1)).unwrap(); + } + + let app = pair.server().router(); + let uri = media_uri(&app, "room/live").await; + listed(&app, &uri, "seg/656e3d1b.0.m4s").await; + + // Waited on rather than requested once: a client only asks about what it watches. + let remote_media = moq_clock::timeout(TIMEOUT, pair.subscriber.consume().routed_broadcast("room/source")) + .await + .expect("sibling resolve timed out") + .expect("sibling broadcast"); + // Only the sibling goes: the catalog keeps listing the rendition, so the request reaches the + // media fetch rather than missing the rendition. + media_broadcast.close(); + moq_clock::timeout(TIMEOUT, remote_media.closed()) + .await + .expect("sibling close timed out"); + + assert_missing(&get(&app, &uri.replace("media.m3u8", "seg/656e3d1b.0.m4s")).await); + drop((catalog, catalog_broadcast, media)); +} diff --git a/rs/moq-hls/tests/integration/support.rs b/rs/moq-hls/tests/integration/support.rs new file mode 100644 index 00000000000..c232fadd154 --- /dev/null +++ b/rs/moq-hls/tests/integration/support.rs @@ -0,0 +1,261 @@ +//! A publisher and an HLS export on either side of a moq-net session over the in-memory +//! transport. + +use std::time::Duration; + +use axum::http::{Response, StatusCode, header}; +use bytes::Bytes; +use moq_hls::Server; +use moq_hls::export::{Kind, Rendition, segments}; + +/// How long a step may take on the paused clock before the test fails. +pub const TIMEOUT: Duration = Duration::from_secs(10); + +/// The media track every test publishes, which is also its rendition name. +pub const TRACK: &str = "video0"; + +/// A subscriber connected to a publisher over one session, as an export connects to its relay. +pub struct Pair { + /// The publisher's origin, which the server session publishes. + pub publisher: moq_net::origin::Producer, + /// The subscriber's origin, which the client session inserts remote broadcasts into. + pub subscriber: moq_net::origin::Producer, + _client: moq_net::Session, + server: Option, +} + +impl Pair { + /// Connect on the default moq-lite version. + pub async fn connect() -> Self { + let version = *moq_net::Versions::default().iter().next().expect("a default version"); + Self::new(version, Default::default()).await + } + + /// Connect on `version`, with the publisher's origin built from `config`. + pub async fn new(version: moq_net::Version, config: moq_net::origin::Config) -> Self { + let publisher = origin(config); + let subscriber = origin(Default::default()); + let mut transport = moq_net::transport::memory::Config::default(); + transport.protocol = Some(version.alpn().to_string()); + let (client_transport, server_transport) = moq_net::transport::memory::pair(transport); + let client = moq_net::Client::new() + .with_versions(version.into()) + .with_subscriber(subscriber.clone()); + let server = moq_net::Server::new() + .with_versions(version.into()) + .with_publisher(publisher.consume()); + let (client, server) = tokio::join!( + async { + let (session, driver) = client + .connect(moq_clock::now(), client_transport) + .await + .expect("client session"); + tokio::spawn(moq_net::time::run(driver)); + session + }, + async { + let (session, driver) = server + .accept(moq_clock::now(), server_transport) + .await + .expect("server session"); + tokio::spawn(moq_net::time::run(driver)); + session + } + ); + Self { + publisher, + subscriber, + _client: client, + server: Some(server), + } + } + + /// An HLS server exporting everything the subscriber receives. + pub fn server(&self) -> Server { + Server::new(self.subscriber.consume(), Default::default()) + } + + /// Drop the publisher's session, as a publisher that lost its connection would. + pub fn disconnect(&mut self) { + drop(self.server.take().expect("connected")); + } +} + +fn origin(config: moq_net::origin::Config) -> moq_net::origin::Producer { + let (origin, driver) = moq_net::origin::Producer::new(config); + tokio::spawn(moq_net::time::run(driver)); + origin +} + +/// One publisher run of a broadcast: a catalog and a single VP8 rendition whose timeline the +/// catalog advertises. +pub struct Publisher { + /// The broadcast, which announces its path while it lives. + pub broadcast: moq_net::origin::Publication, + /// The catalog. + pub catalog: moq_mux::catalog::Producer, + /// The media track, recording its timeline as it is written. + pub media: moq_mux::container::Producer, +} + +impl Publisher { + /// Announce `path` on `origin` under `route` and publish its catalog. + pub fn new(origin: &moq_net::origin::Producer, path: &str, route: moq_net::origin::Route) -> Self { + let broadcast = origin.create_broadcast(path).expect("publish"); + broadcast.announce(route).expect("announce"); + // A hinted timeline is advertised before any media, so a recorder can start first. + let timeline = moq_mux::timeline::Config::default().with_duration_hint(Duration::from_secs(2)); + let config = moq_mux::catalog::Config::default().with_timeline(timeline); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast.broadcast().clone(), config).expect("catalog"); + let reserved = catalog.reserve(); + let track = broadcast.create_track(TRACK, timed()).expect("track"); + let media = reserved + .video( + track, + moq_mux::catalog::hang::Container::Legacy(moq_mux::container::Kind::Video), + video_config(), + ) + .expect("rendition"); + drop(reserved); + Self { + broadcast, + catalog, + media, + } + } + + /// Write one-second frames for each second in `start..end`, a keyframe on every even second. + /// Segments are cut every four seconds, so writing up to 13 completes segments 0 to 2 and + /// leaves the GOP at 12s open. + pub fn write(&mut self, start: u64, end: u64, payload: u8) { + for second in start..end { + self.media.write(frame(second, payload)).expect("write"); + } + } + + /// Finish the media and then the catalog, as a publisher ending cleanly does. + pub fn finish(&mut self) { + self.media.finish().expect("finish media"); + self.catalog.finish().expect("finish catalog"); + } +} + +/// A millisecond track: the frames written here carry timestamps. +pub fn timed() -> moq_net::track::Info { + moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI) +} + +/// A 320x240 VP8 rendition, whose init its catalog config alone describes. +pub fn video_config() -> hang::catalog::VideoConfig { + let mut config = hang::catalog::VideoConfig::new(hang::catalog::VideoCodec::VP8); + config.coded_width = Some(320); + config.coded_height = Some(240); + config.framerate = Some(30.0); + config +} + +/// A VP8 frame at `second`, a keyframe on even seconds, tagged with `payload` so each run's +/// media differs byte for byte. +pub fn frame(second: u64, payload: u8) -> moq_mux::container::Frame { + let keyframe = second.is_multiple_of(2); + let mut bytes = if keyframe { + vec![0x10, 0x00, 0x00, 0x9d, 0x01, 0x2a, 0x40, 0x01, 0xf0, 0x00] + } else { + vec![0x31, 0x00, 0x00] + }; + bytes.extend_from_slice(&[payload; 8]); + moq_mux::container::Frame { + timestamp: moq_net::Timestamp::from_secs(second).expect("timestamp"), + payload: bytes.into(), + keyframe, + duration: None, + } +} + +/// The rendition `server` exports for `path` once its catalog lists it. +pub async fn rendition(server: &Server, path: &str) -> std::sync::Arc { + let broadcaster = moq_clock::timeout(TIMEOUT, server.broadcaster(path)) + .await + .expect("resolve timed out") + .expect("announced"); + let mut renditions = broadcaster.renditions(); + loop { + match moq_clock::timeout(TIMEOUT, renditions.next()).await { + Ok(Ok(Some(moq_hls::export::renditions::Event::Added(rendition)))) if rendition.name == TRACK => { + return rendition; + } + Ok(Ok(Some(_))) => continue, + result => panic!("the catalog never listed {TRACK}: {:?}", result.map(|r| r.map(|_| ()))), + } + } +} + +/// Read `cursor` until it ends, returning the segment numbers read and how it ended. +pub async fn drain(cursor: &mut segments::Consumer) -> (Vec, moq_hls::Result<()>) { + let mut read = Vec::new(); + loop { + match moq_clock::timeout(TIMEOUT, cursor.next()) + .await + .expect("the cursor stalled") + { + Ok(Some(segment)) => read.push(segment.segment), + Ok(None) => return (read, Ok(())), + Err(err) => return (read, Err(err)), + } + } +} + +/// GET `uri` from `app`, with the whole body read. +pub async fn get(app: &axum::Router, uri: &str) -> Response { + use tower::ServiceExt; + let request = axum::extract::Request::builder() + .uri(uri) + .body(axum::body::Body::empty()) + .expect("request"); + let (parts, body) = app.clone().oneshot(request).await.expect("infallible").into_parts(); + let body = axum::body::to_bytes(body, usize::MAX).await.expect("body"); + Response::from_parts(parts, body) +} + +/// Poll `uri` until it answers 200 with a body containing `needle`, returning the body. +pub async fn listed(app: &axum::Router, uri: &str, needle: &str) -> String { + let deadline = moq_clock::now() + TIMEOUT; + loop { + let response = get(app, uri).await; + let body = String::from_utf8_lossy(response.body()).into_owned(); + if response.status() == StatusCode::OK && body.contains(needle) { + return body; + } + assert!( + moq_clock::now() < deadline, + "{uri} never listed {needle}: {} {body}", + response.status() + ); + moq_clock::sleep(Duration::from_millis(50)).await; + } +} + +/// Assert `response` is a 404 that a CDN won't pin, since the resource may appear later. +pub fn assert_missing(response: &Response) { + assert_eq!(response.status(), StatusCode::NOT_FOUND); + assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); +} + +/// The segment numbers a media playlist lists, in order. +pub fn numbers(playlist: &str) -> Vec { + playlist + .lines() + .filter_map(|line| line.strip_prefix("seg/")?.strip_suffix(".m4s")) + .map(|name| name.rsplit('.').next().expect("a number").parse().expect("numbered")) + .collect() +} + +/// The media playlist URI the master playlist of `path` lists for [`TRACK`], under its instance. +pub async fn media_uri(app: &axum::Router, path: &str) -> String { + let master = listed(app, &format!("/{path}/master.m3u8"), TRACK).await; + let line = master + .lines() + .find(|line| line.ends_with(&format!("/{}/{TRACK}/media.m3u8", Kind::Video.as_str()))) + .unwrap_or_else(|| panic!("no {TRACK} variant in {master}")); + format!("/{path}/{line}") +}