diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 930cc5cf546..0b84d308897 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -73,7 +73,9 @@ authorization and hidden names work exactly as without one. worker closed a path and serves it again. Such a worker keeps the path's group sequence going, even when its own input starts over at 0. Otherwise a viewer returning within a relay's cache window gets the old output's latest group, - then nothing until the new sequence passes it. + then nothing until the new sequence passes it. On moq-lite 07 and + moq-transport, the relay next to the worker sees the sequence go back and + gives its own viewers an error instead, so they request the path again. - **Newest wins.** Among routes at the same prefix, the newest epoch ranks first, ahead of cost, and a route without an epoch ranks last. Newest means the latest UUIDv7 timestamp, so hosts with skewed clocks can lose a restart diff --git a/quest/m1/README.md b/quest/m1/README.md index ab371e2b1ff..834e8aba9ba 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -48,7 +48,7 @@ session outcomes, and embedder refusals inside session outcomes. - [Interop close logs](/quest/m1/interop-close-log.md) - logging fix landed; delete once the required Interop run is green - [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall - [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical readers obey publisher max-age and an optional tighter reader budget, including while holding a group -- [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache +- [Refused idle copy](/quest/m1/refused-copy.md) - behind any chain of relays a returning reader never gets a restarted publisher's old group: a refused idle copy withholds its cache and no hop answers from it - [Untimed decisions](/quest/m1/untimed-decisions.md) - the maintainer decides whether moq-archive keeps refusing untimed tracks, and whether a malformed FETCH object ends its track - [FFI shape](/quest/m1/ffi-shape/README.md) - the bindings mirror Rust's layers: net at the root, then media, json, flate, audio, and video namespaces built from the handle below - [Publishing never invents a timestamp](/quest/m1/publish-timestamp.md) - no Rust or binding publish API fills in a timestamp; an untimed payload goes out untimed diff --git a/quest/m1/largest-regression.md b/quest/m1/largest-regression.md deleted file mode 100644 index f645ecce759..00000000000 --- a/quest/m1/largest-regression.md +++ /dev/null @@ -1,65 +0,0 @@ -# [S] A relay copy fails loud when upstream's position goes backwards - -## Goal - -A relay copy that subscribes upstream again and hears that upstream's largest -group is below the newest group it has cached treats it as a reused name: the -copy ends with an error, its source closes so the front ends, and the readers -it was serving re-request. A new request arriving as the front ends gets a -fresh front without an error. Today the copy keeps its stale live floor (`set_live` in -`rs/moq-net/src/model/track.rs`), so a viewer returning within the linger to a -publisher that restarted at group 0 gets the old instance's cached group and -then nothing until the new sequence passes it. - -Covers lite-07 and moq-transport, whose answers carry the largest position. -lite-05 and 06 stay uncovered on purpose: their only hint is the first -group's sequence, and they keep waiting on the floor. - -## Plan - -Found while scoping idle fronts (2026-10-07): -under a prefix claim the relay never sees the worker close a broadcast, so a -restart at the same path reaches a lingering copy unannounced. A publisher -that restarts its group sequence under the same name is buggy, and this makes -that bug visible instead of a silent stall. A change of route can't cause -it: every route change, including a per-path winner change under a prefix -pool, reaches downstream as a `Restart` of the route, which unsets the cached -copy of every broadcast nested under it -(maintainer, 2026-10-08, in Restart). -Only a restart behind an unchanged route reaches a copy unannounced. - -Decided 2026-10-08: shrunk to the two answer-carrying wires and moved to m1. -The heuristic is unproven and risks a false failover, so it builds on the -front and route shapes Restart and idle fronts leave rather than ahead of -them. - -Where it lands: since #4914, a rejoining copy goes idle (`set_idle`) and the -route's answer reaches `set_live`, which holds readers in `Feed::Answered` -until the cache shows the answered position. An answer below what the cache -already shows goes straight to `Feed::Live` and keeps the old floor; that is -the case to end instead, when it qualifies below. - -A lower largest group alone doesn't prove a restart: a same-epoch resume onto -a replica that has fallen behind reports one legitimately, and must keep -waiting as today. The copy ends only when the answer comes from the route it -was last served from without an epoch, where no cross-route resume happens, -or names a different epoch than the copy holds. Check for any other -legitimate case before relying on that split. - -Whatever ends a front, it leaves the origin's front table before its broadcast -closes or anything fails, and `request()` treats a front whose broadcast closed -as gone. So a reader re-requesting after seeing the front's end mints a fresh -front. The copy's error must reach readers through that end, not an `Abort` -ahead of it, or a re-request can still join the old front. A request that -joined before the front decided to end is in flight, like a subscription. - -Verification: mocked time, a publisher restarting a path at group 0 within -the linger. The returning reader gets an error and then the new instance from -group 0, never the old group. A same-epoch standby behind the cached copy -keeps the copy and resumes it. - -Public API: none. Wire: none. - -## Related - -- [Announcement takeover](/quest/m1/announce-takeover.md) - proves continuity when a lazy job gains an independent announcement; a replacement never takes over its predecessor's content diff --git a/quest/m1/refused-copy.md b/quest/m1/refused-copy.md new file mode 100644 index 00000000000..ef2e2a31974 --- /dev/null +++ b/quest/m1/refused-copy.md @@ -0,0 +1,77 @@ +# [M] A refused idle copy withholds its cache at every hop + +## Goal + +Behind any chain of relays, a returning reader never gets a restarted +publisher's old group. An idle copy that its upstream refuses before +answering, or answers below its cache, withholds its cache and ends, and no +hop answers from a withheld cache. Covers lite-07 and moq-transport; lite-05 +and 06 carry no largest position and keep today's behavior. + +Today only the relay next to the worker catches the restart (#5057). With an +edge relay in between, the returning viewer still gets the old group on every +version: the edge's idle copy is refused, aborts, and an aborted copy lets its +readers drain the old cache. + +## Plan + +Facts as of #5057: + +- lite: on lite-05+ the upstream's reset lands in `ServeLoop::poll` + (`lite/subscriber.rs`) as `ServeEnd::GiveBack(err)`, and `TrackServeRun` + calls `serving.abort(err)`. An aborted copy is readable + (`TrackState::readable`, `model/track.rs`), so readers drain the idle + cache. `Unroutable` is not in `route_failed` (`model/resume.rs`), and since + #5137 a front on a route without an epoch passes the abort on rather than + re-splicing. +- moq-transport: the middle relay waits on `track.poll_live` + (`ietf/publisher.rs`), but its own copy's abort unblocks that wait, and + `live_edge` reads `latest()` and `peek_latest`, which ignore the + `live_floor` that `withhold_cache` raises. Its SUBSCRIBE_OK carries the + hidden group as Largest, so the edge sees no regression, goes live on its + old group, and then gets PUBLISH_DONE `InternalError`, which aborts its + copy without closing the source (`ietf/subscriber.rs`, `subscribe_once`). + The error-answer branch there also aborts without withholding. +- Every source of a refusal toward a rejoining copy today: no covering route + (`model/origin.rs`), a lite-07 epoch mismatch (`model/origin.rs`), and + #5057's regression. On the wire they can't be told apart: lite resets with + 0x36, and moq-transport sends DOES_NOT_EXIST, read back as `NotFound` + (`ietf/error.rs`). +- lite-07: a group header or a datagram (`lite/subscriber.rs`) makes the copy + live before START, after which `idle_cached()` returns `None` and the + regression check is skipped. A lite-07 publisher always writes START before + the first group. + +Decided 2026-10-08: + +- An idle copy refused before it was answered can't judge its cache, so it + withholds it, whatever the error. A verdict error closes the source so the + front ends (#5054's guarantee); a `route_failed` error fails over as today. + This is right for every refusal source above, and needs no wire change. + Rejected: matching UNROUTABLE only, which moq-transport can't carry + distinctly, and a new regression error code, a wire change that doesn't + help moq-transport. +- No hop answers from a withheld cache: `live_edge` honors the withheld + floor, and a moq-transport relay whose copy ended refuses the downstream + SUBSCRIBE instead of sending SUBSCRIBE_OK with the hidden group. +- lite-07 defers going live on a group until START. A datagram-only track + never gets a START, so it keeps the gap, which needs a reader starting + below the cache; document it as a known limit. +- Decided 2026-10-08: m1, beside the regression check it extends. A relay + that under-reports its upstream's largest (a cold copy answering from + older groups) can trip a false regression; that costs a spurious error and + a re-request, never stale data, and is accepted: cold-relay Largest stays + INVALID_RANGE (`quest/m0/README.md`). + +Verification: mocked time, a viewer behind an edge relay and the relay next +to a claim worker that restarts at group 0 within the linger, on lite-07 and +moq-transport 14, 16 and 22: the returning viewer gets an error and then the +new output from group 0, never the old group. A lite-07 case where a group +stream arrives before START. A same-epoch standby still resumes, and a +`route_failed` refusal still fails over. + +Public API: none expected. Wire: none new. + +## Related + +- [Announcement takeover](/quest/m1/announce-takeover.md) - a worker that wants readers kept across a restart proves continuity instead of relying on an epochless claim diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 793f349f855..c71aa34307a 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -1905,9 +1905,9 @@ where let mut this = self.clone(); let path = path.to_owned(); - let broadcast = broadcast.clone(); + let source = (*source).clone(); subscribes.push(async move { - this.run_subscribe(path, broadcast, request).await; + this.run_subscribe(path, source, request).await; }); } @@ -1917,9 +1917,10 @@ where async fn run_subscribe( &mut self, broadcast_path: Path<'_>, - // Held for the subscription's lifetime but never watched: the broadcast ending - // is a retraction, which does not disturb a subscription already in flight. - _broadcast: broadcast::Dynamic, + // The minted source, never watched: its end is a retraction, which does not disturb a + // subscription already in flight. Closed once the route shows it serves new content + // under the name. + source: broadcast::Producer, request: track::Request, ) { // Data streams wait on the alias bound by SUBSCRIBE_OK, so leave the model request @@ -1964,7 +1965,14 @@ where }; loop { let Some(idle) = self - .subscribe_once(&broadcast_path, &track_name, target, &mut group_fetches, &mut gate) + .subscribe_once( + &broadcast_path, + &track_name, + &source, + target, + &mut group_fetches, + &mut gate, + ) .await else { return; @@ -1986,6 +1994,7 @@ where &mut self, broadcast_path: &Path<'_>, track_name: &str, + source: &broadcast::Producer, target: Target, group_fetches: &mut TaskSet, gate: &mut crate::auth::Gate, @@ -2250,6 +2259,20 @@ where // Aborted with the session while the answer was in hand. return None; } + // moq-transport names no publisher instance, so a Largest below what the copy + // cached is a publisher reusing the name for new content. The source closes + // first: its front ends before any reader sees the copy fail, so a reader + // re-requesting gets a fresh one. + if idle.track.regresses(resuming.flatten()) { + tracing::warn!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, ?largest, "upstream went back in its group sequence"); + let mut track = idle.track; + track.withhold_cache(); + source.close(); + self.remove_subscribe(request_id); + let _ = track.abort(Error::Unroutable); + self.cancel_subscribe(stream, request_id).await; + return None; + } (idle.track, idle.dynamic) } None => { @@ -5352,8 +5375,9 @@ mod tests { drop(track); drop(consumer); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); moq_net_sim::timeout(std::time::Duration::from_secs(1), serving) @@ -5402,8 +5426,9 @@ mod tests { let request = dynamic.requested_track().await.expect("no track requested"); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); // Let the SUBSCRIBE go out. No SUBSCRIBE_OK is coming, so the subscription never @@ -5466,8 +5491,9 @@ mod tests { let request = dynamic.requested_track().await.expect("no track requested"); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); // Let the SUBSCRIBE go out, then retract the broadcast before any response. @@ -5523,9 +5549,10 @@ mod tests { let request = dynamic.requested_track().await.expect("no track requested"); let mut fetch = None; + let source = producer.clone(); let serving = match kind { Allocated::Subscribe => moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }), Allocated::TrackStatus => moq_net_sim::spawn(async move { assert!( @@ -5650,8 +5677,9 @@ mod tests { let request = dynamic.requested_track().await.expect("no track requested"); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); settle().await; @@ -5793,8 +5821,9 @@ mod tests { // subscription reached Established rather than assume it. let probe = subscriber.clone(); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); // Let the SUBSCRIBE go out and the SUBSCRIBE_OK come back, so the subscription is @@ -5872,8 +5901,9 @@ mod tests { let request = dynamic.requested_track().await.expect("no track requested"); let probe = subscriber.clone(); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); settle().await; @@ -5971,8 +6001,9 @@ mod tests { let request = dynamic.requested_track().await.expect("no track requested"); let probe = subscriber.clone(); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; + subscriber.run_subscribe(Path::new("broadcast"), source, request).await; }); settle().await; let _subscription = track.subscribe(None); @@ -9662,9 +9693,10 @@ mod joining_fetch_tests { let request = dynamic.requested_track().await.expect("no track requested"); let mut serving_subscriber = subscriber.clone(); + let source = producer.clone(); let serving = moq_net_sim::spawn(async move { serving_subscriber - .run_subscribe(Path::new("broadcast"), dynamic, request) + .run_subscribe(Path::new("broadcast"), source, request) .await; }); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index d0bc17833ed..ec3cce57cdd 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -1556,6 +1556,7 @@ impl kio::Task for SourceServe { subscriber: self.subscriber.clone(), path: self.path.clone(), epoch: self.epoch.clone(), + source: (*self.source).clone(), name: request.name().to_string(), }; // One machine per track serves its lone subscription and any number @@ -1997,6 +1998,7 @@ mod tests { subscriber, path: Path::new("room").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "video".to_string(), }; let broadcast = crate::broadcast::Info::new().produce(); @@ -2054,6 +2056,7 @@ mod tests { subscriber, path: Path::new("room").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "video".to_string(), }; let broadcast = crate::broadcast::Info::new().produce(); @@ -2230,6 +2233,7 @@ mod tests { subscriber: subscriber.clone(), path: Path::new("room").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "video".to_string(), }, request, @@ -2289,6 +2293,7 @@ mod tests { subscriber, path: Path::new("room").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "video".to_string(), }, request, @@ -2340,6 +2345,7 @@ mod tests { subscriber, path: Path::new("room/host").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "catalog.json".to_string(), }; @@ -2407,6 +2413,7 @@ mod tests { subscriber, path: Path::new("room/host").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "audio".to_string(), }; @@ -2474,6 +2481,7 @@ mod tests { subscriber, path: Path::new("room/host").to_owned(), epoch: None, + source: crate::broadcast::Info::new().produce(), name: "catalog.json".to_string(), }, session, @@ -4182,6 +4190,9 @@ struct TrackServe { path: PathOwned, /// The publisher instance to ask for; the peer refuses a request for another. epoch: Option, + /// The minted source the track belongs to, closed once the route shows it serves new + /// content under the name. + source: crate::broadcast::Producer, name: String, } @@ -5171,6 +5182,16 @@ impl ServeLoop { lite::SubscribeResponse::Start(start) => { // Where the live feed is, on versions whose answer says. if serve.subscriber.version.has_largest() { + // A route without an epoch serves one instance, so a feed below what + // the copy cached is a publisher reusing the name for new content. + // The source closes first: its front ends before any reader sees the + // copy fail, so a reader re-requesting gets a fresh one. + if serve.epoch.is_none() && self.serving.regresses(start.largest) { + tracing::warn!(broadcast = %serve.subscriber.log_path(&serve.path), track = %serve.name, largest = ?start.largest, "upstream went back in its group sequence"); + self.serving.withhold_cache(); + serve.source.close(); + return Poll::Ready(ServeEnd::GiveBack(Error::Unroutable)); + } self.serving.set_live(start.largest); } // A START describes the demand the SUBSCRIBE carried. diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 9483097dacb..6cbc24b843b 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1540,15 +1540,8 @@ impl TrackState { if self.feed != Feed::Idle { return; } - // Judged by what survived: the cancel that idled the copy resets the group in flight, - // and an answer past it leaves a gap after the newest group still cached. - let cached = self.idle_newest.take().and_then(|newest| { - self.lookup - .range(..=newest) - .rev() - .find(|(_, slot)| slot.is_shown()) - .map(|(sequence, _)| *sequence) - }); + let cached = self.idle_cached(); + self.idle_newest = None; self.live_floor = match (largest, cached) { (Some(largest), Some(cached)) if largest.group > cached.saturating_add(1) => Some(cached + 1), // The route has nothing, so whatever is cached is not its feed. @@ -1561,6 +1554,18 @@ impl TrackState { }; } + /// While idle, the newest group cached before the track went idle that readers can + /// still see. Judged by what survived: the cancel that idled the copy resets the group + /// in flight, and an answer past it leaves a gap after the newest group still cached. + fn idle_cached(&self) -> Option { + let newest = self.idle_newest.filter(|_| self.feed == Feed::Idle)?; + self.lookup + .range(..=newest) + .rev() + .find(|(_, slot)| slot.is_shown()) + .map(|(sequence, _)| *sequence) + } + /// Whether readers can already see content at or past `sequence`. fn shows(&self, sequence: u64) -> bool { self.lookup.range(sequence..).any(|(_, slot)| slot.is_shown()) @@ -2053,6 +2058,26 @@ impl Producer { } } + /// While idle, whether the route's answer, its largest position, is below the newest + /// group the copy still holds. A route with one publisher instance behind it went back + /// in its group sequence; one with replicas may only be serving a lagging one. + pub(crate) fn regresses(&self, largest: Option) -> bool { + let state = self.state.read(); + largest + .zip(state.idle_cached()) + .is_some_and(|(largest, cached)| largest.group < cached) + } + + /// Readers get nothing cached so far, however the track ends: it is another publisher + /// instance's content. An abort alone would let them drain it first. + pub(crate) fn withhold_cache(&mut self) { + if let Ok(mut state) = self.modify() + && let Some(max) = state.max_sequence + { + state.live_floor = Some(max.saturating_add(1)); + } + } + /// Whether readers may take from the cache; see [`Self::set_idle`]. pub(crate) fn is_live(&self) -> bool { self.state.read().feed == Feed::Live @@ -6087,6 +6112,38 @@ mod test { assert_eq!(drain(&mut subscriber), vec![2]); } + /// An idle copy judges the route's answer by the groups that survived the leave, and + /// once one went back below them, its readers get the copy's error and none of the old + /// groups an abort would otherwise let them drain. + #[test] + fn an_answer_below_the_cache_regresses() { + let mut producer = track_producer("test", Info::timed()); + append_at(&mut producer, 0); + append_at(&mut producer, 100); + let mut open = producer.append_group().unwrap(); + open.write_frame(Timestamp::from_millis(200).unwrap(), bytes::Bytes::from_static(b"x")) + .unwrap(); + assert!( + !producer.regresses(Some(Position::group(0))), + "a live copy judges nothing" + ); + + producer.set_idle(); + open.abort(Error::Cancel).unwrap(); + // The leave reset group 2, so group 1 is the newest the copy still holds. + assert!(!producer.regresses(Some(Position::group(1)))); + assert!(!producer.regresses(None), "a route with nothing yet"); + assert!(producer.regresses(Some(Position::group(0)))); + + let mut subscriber = producer.subscribe(replay()); + producer.withhold_cache(); + producer.clone().abort(Error::Unroutable).unwrap(); + assert!(matches!( + subscriber.recv_group().now_or_never(), + Some(Err(Error::Unroutable)) + )); + } + #[test] fn real_time_skips_a_backlog_to_the_live_edge() { let mut producer = track_producer("test", Info::timed()); diff --git a/rs/moq-net/tests/largest_regression.rs b/rs/moq-net/tests/largest_regression.rs new file mode 100644 index 00000000000..8fd484c7f7e --- /dev/null +++ b/rs/moq-net/tests/largest_regression.rs @@ -0,0 +1,209 @@ +//! A relay's lingering copy of a track judges the route's answer when a reader returns. +//! +//! A worker claims the `pool` prefix with `origin::Producer::dynamic`, so relay `R` +//! never sees it close an output. When it serves `pool/p` again, the new output starts +//! over at group 0, below what `R` still caches for a returning reader. Through a route +//! without an epoch that is new content under the old name: the reader gets an error +//! and re-requests, rather than the old output's group and then nothing. +//! +//! Through a route with an epoch the answer may come from a replica that lags behind +//! the copy, so `R` keeps the copy and waits for the replica to catch up. + +mod support; + +use std::{cell::RefCell, rc::Rc, time::Duration}; + +use moq_net::{Error, Hop, Timestamp, Version, broadcast, origin, track}; +use support::harness::{MockConnectOptions, MockPair, connect_mock}; + +/// A millisecond track: the frames written here carry timestamps. +fn timed() -> moq_net::track::Info { + moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI) +} + +const TEST_TIMEOUT: Duration = Duration::from_secs(60); + +/// The versions whose answer carries the route's largest position. +const VERSIONS: &[&str] = &[ + "moq-lite-07-wip", + "moq-transport-14", + "moq-transport-16", + "moq-transport-22", +]; + +fn produce_origin(hop: u64) -> origin::Producer { + let (producer, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + support::harness::spawn(driver); + producer +} + +/// Have `to` pull everything `from` publishes. +async fn link(version: Version, from: &origin::Producer, to: &origin::Producer) -> MockPair { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(from.consume()); + options.client_subscribe = Some(to.clone()); + connect_mock(options).await +} + +/// Let every task settle. Time is paused, so this returns once the runtime is idle. +async fn settle() { + moq_net_sim::sleep(Duration::from_millis(500)).await; +} + +/// Write one finished group carrying `payload`. +fn append(track: &track::Producer, payload: String) { + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, payload.into_bytes()).unwrap(); + group.finish().unwrap(); +} + +/// Subscribe to `video` at `path` through `relay` and read the first group's frame. +async fn watch(relay: &origin::Producer, path: &str) -> Result<(u64, String), Error> { + let broadcast = relay.consume().request_broadcast(path, None).await?; + let mut subscription = broadcast.track("video")?.subscribe(None).await?; + let mut group = subscription.recv_group().await?.ok_or(Error::Dropped)?; + let frame = group.read_frame().await?.ok_or(Error::Dropped)?; + Ok((group.sequence, String::from_utf8(frame.payload.to_vec()).unwrap())) +} + +/// A worker claiming `pool`: each request gets a new output whose `video` track starts +/// at group 0. The first output writes groups 0 to 2, later ones group 0 only. +struct Worker { + _claim: Rc, + outputs: Rc>>, + origin: origin::Producer, +} + +impl Worker { + fn new(hop: u64) -> Self { + let origin = produce_origin(hop); + let claim = Rc::new(origin.dynamic("pool", origin::Route::default()).unwrap()); + let outputs = Rc::new(RefCell::new(Vec::new())); + let (handler, served) = (claim.clone(), outputs.clone()); + drop(moq_net_sim::spawn(async move { + while let Ok(request) = handler.requested_broadcast().await { + let output = broadcast::Info::new().produce(); + let track = output.create_track("video", timed()).unwrap(); + let generation = served.borrow().len(); + let groups = if generation == 0 { 3 } else { 1 }; + for sequence in 0..groups { + append(&track, format!("{generation}:{sequence}")); + } + request.accept(&output); + served.borrow_mut().push((output, track)); + } + })); + Self { + _claim: claim, + outputs, + origin, + } + } +} + +async fn restart_within_the_linger(version: Version) -> Result<(), String> { + let relay = produce_origin(2); + let worker = Worker::new(1); + let _link = link(version, &worker.origin, &relay).await; + settle().await; + + // A viewer reads the first output's newest group and leaves; the relay keeps the copy. + match watch(&relay, "pool/p").await { + Ok((2, payload)) if payload == "0:2" => {} + other => return Err(format!("the first viewer got {other:?}")), + } + settle().await; + + // The worker closes the output, so the next request it gets starts a new one at group 0. + worker.outputs.borrow().iter().for_each(|(output, _)| output.close()); + settle().await; + + // Back within the linger: the copy still holds the old output's group 2. + match watch(&relay, "pool/p").await { + Err(Error::Unroutable) => {} + other => return Err(format!("the returning viewer got {other:?}")), + } + + // The re-request reaches the new output from its first group. + match watch(&relay, "pool/p").await { + Ok((0, payload)) if payload == "1:0" => {} + other => return Err(format!("the re-request got {other:?}")), + } + match worker.outputs.borrow().len() { + 2 => Ok(()), + outputs => Err(format!("the worker served {outputs} outputs")), + } +} + +#[moq_net_sim::test] +async fn a_restart_ends_the_lingering_copy() { + let mut failures = Vec::new(); + for version in VERSIONS { + let version: Version = version.parse().unwrap(); + match moq_net_sim::timeout(TEST_TIMEOUT, restart_within_the_linger(version)).await { + Ok(Ok(())) => {} + Ok(Err(err)) => failures.push(format!("{version}: {err}")), + Err(_) => failures.push(format!("{version}: timed out")), + } + } + assert!(failures.is_empty(), "\n{}", failures.join("\n")); +} + +/// `P1` and `P2` publish `live` under one epoch, and relay `X` pulls both for relay `R`. +/// `R` caches group 3 from `P1`, its reader leaves, and `P1` goes, so `X` serves `P2`, +/// which is only at group 1. A returning reader on `R` hears group 1 as the largest, +/// below the copy, and still gets the cached group and then `P2`'s next ones. +#[moq_net_sim::test] +async fn a_lagging_replica_with_the_epoch_resumes_the_copy() { + moq_net_sim::timeout(TEST_TIMEOUT, async { + let version: Version = "moq-lite-07-wip".parse().unwrap(); + let epoch = moq_net::Epoch::mint(); + let relay_x = produce_origin(3); + let relay_r = produce_origin(4); + + let mut replicas = Vec::new(); + for (name, hop, cost, groups) in [("P1", 1, 1, 4), ("P2", 2, 5, 2)] { + let publisher = produce_origin(hop); + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast.create_track("video", timed()).unwrap(); + for sequence in 0..groups { + append(&track, format!("{name}:{sequence}")); + } + broadcast + .announce(origin::Route::default().with_epoch(epoch.clone()).with_cost(cost)) + .unwrap(); + let link = link(version, &publisher, &relay_x).await; + replicas.push((publisher, broadcast, track, link)); + } + let _x_r = link(version, &relay_x, &relay_r).await; + settle().await; + + assert_eq!(watch(&relay_r, "live").await.unwrap(), (3, "P1:3".into())); + settle().await; + + // `P1` goes, so `X` serves `P2`, and a reader on `X` brings its group 1 in. + let (_publisher, _broadcast, _track, p1) = replicas.remove(0); + p1.server.abort(Error::Cancel); + p1.client.abort(Error::Cancel); + settle().await; + let local = relay_x.consume().request_broadcast("live", None).await.unwrap(); + let mut local = local.track("video").unwrap().subscribe(None).await.unwrap(); + assert_eq!(local.recv_group().await.unwrap().unwrap().sequence, 1); + + let broadcast = relay_r.consume().request_broadcast("live", None).await.unwrap(); + let mut subscription = broadcast.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = subscription.recv_group().await.unwrap().unwrap(); + let frame = group.read_frame().await.unwrap().unwrap(); + assert_eq!((group.sequence, frame.payload.as_ref()), (3, b"P1:3".as_ref())); + + let (_, _, p2, _) = &replicas[0]; + for sequence in 2..5 { + append(p2, format!("P2:{sequence}")); + } + let mut group = subscription.recv_group().await.unwrap().unwrap(); + let frame = group.read_frame().await.unwrap().unwrap(); + assert_eq!((group.sequence, frame.payload.as_ref()), (4, b"P2:4".as_ref())); + }) + .await + .expect("timed out"); +}