diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index e773589c9b..7ccc76cea1 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -33,10 +33,12 @@ Its proof is `rs/moq-net/verify/route-algebra/`, and the quests written against the old wire were retargeted or deleted. [moq-dev/moq#9111](https://github.com/moq-dev/moq/pull/9111) ran the cases no -test covered on real sessions. The lite-07 cut still waits on this epic's -children, the two flaws it found, -[a reader resuming on a detour learned after its session closed](/quest/m2/multicast/detour-resume.md) -and [a route moving behind one next hop resending a group](/quest/m2/multicast/resend.md), +test covered on real sessions, and +[moq-dev/moq#9154](https://github.com/moq-dev/moq/pull/9154) fixed the first +flaw it found: a front now rides through a lost claim's debounce, so a reader +resumes on a detour learned after its session closed. The lite-07 cut still +waits on this epic's children, the other flaw, +[a route moving behind one next hop resending a group](/quest/m2/multicast/resend.md), whose fix changes the wire, and two flaws the mesh replay found, [feeding an old relay's subscription mid-roll](/quest/m2/multicast/roll-echo-stall.md) diff --git a/quest/m2/multicast/detour-resume.md b/quest/m2/multicast/detour-resume.md deleted file mode 100644 index 4c9a76c6d1..0000000000 --- a/quest/m2/multicast/detour-resume.md +++ /dev/null @@ -1,73 +0,0 @@ -# [M] A reader resumes on a detour learned after its session closed - -## Goal - -A relay whose serving session closes before a detour to the same Instance -reaches it keeps its front, and its readers resume on the detour once it -arrives, every group once. Today the front ends at the close: the reader -restarts onto another Instance serving the path, or, when none does, reads -nothing more for good, even though the same Instance is back on the detour a -round trip later. - -`route_audit::failures::a_relay_resumes_on_a_detour_learned_after_the_cut` -and the readers `detour_lost` pins in `rs/moq-net/tests/integration/route_audit/` -read on seamlessly, and `detour_lost` is gone. - -## Plan - -Decided 2026-11-19 (coordinator, under the maintainer's pick that day to -merge on the coordinator's recommendation): ✅ kept, under this epic, so the -lite-07 cut waits on it (rejected: dropping it; m3). A reader that reads -nothing more for good after one link cut is the worst failure the audit -found, and it needs only a ring of four. - -Found by [the uncovered cases](https://github.com/moq-dev/moq/pull/9111). It -happens wherever every neighbour's route to the announcing relay ran through -the relay that lost its session, so none was offered back to it: a ring of -four or more, or any mesh where the relay's neighbours reach the publisher -through it. - -Root cause: `source_closed` excludes the closed source's route id -(`rs/moq-net/src/model/front.rs`, `self.excluded.insert(candidate.route)`). -A lite-07 claim keeps one table entry while the route layer moves it to -another next hop, so this excludes the very entry the detour revives. With no -other route at that instant, `pick` finds no candidate, and the front ends -for good (`select` in `rs/moq-net/src/model/origin.rs`, then -`end(Error::Unroutable)` in `front.rs`). A following application is then -stuck: `Following` hands out a closed broadcast again only once its Instance -stops serving the path (`rs/moq-net/src/model/follow.rs`, `moved`), and the -Instance never stops: it is back on the detour. - -Recommended: a front rides through its claim's debounce. A closed source no -longer excludes a route-layer entry (`entry.node.is_some()`), or `select` -waits rather than ends while an excluded entry of the resolved Instance is -still shown unavailable. A track that nothing resumes still ends with the -session's error, not a later `Unroutable`. Rerun the `verify-failover` model, -since this changes the front lifecycle it covers. - -An experiment with the first form (about 8 lines in `pick`) made every audit -case seamless and changed 7 existing tests: two read on where they expected -an end (`route_cluster::a_partitioned_lan_reaches_itself_through_the_cdn_and_heals` -and `catalog_handover::lite_07::reconnect_with_viewer`), and five ended a -debounce later with `Unroutable` instead of at once with the session's error -(`subscription_end_integrity::a_session_death_ends_the_track_with_its_error`, -`subscription_end_integrity::a_session_close_cancels_the_track_from_either_side`, -`track_tail::a_session_death_before_the_tail_aborts_the_track`, and both -`multi_cdn::transport_*_secondary_restarts`). The first two are the fix -working. The other five keep their error once the session's error is -carried to the end. - -Rejected: letting `Following` hand out the same Instance again once its -broadcast ended while the Instance still serves the path, which removes the -permanent stall but restarts the reader at the live edge, skipping groups; -and accepting it, which leaves a reader stuck on a one-round-trip gap, -against requirement 2 of [the epic](/quest/m2/multicast/README.md). - -[The mesh replay](/test/mesh-replay/README.md) hits it on real relays too, -mid-roll: in the ring of six at 32 broadcasts per relay, rolling `c`, `d`, and -`e` onto the new build each left about 32 readers ended (`unroutable` or -`internal error`) with no `offline` or `online` after, for good. The replay's -script names this quest for a `stuck` reader, so it is reported without -failing the run. - -Not a wire change. diff --git a/quest/m2/multicast/reopen-starve.md b/quest/m2/multicast/reopen-starve.md index f080e0ec12..cf943f5484 100644 --- a/quest/m2/multicast/reopen-starve.md +++ b/quest/m2/multicast/reopen-starve.md @@ -58,4 +58,3 @@ Not a wire change. ## Related - [roll-echo-stall](/quest/m2/multicast/roll-echo-stall.md) - the mid-roll starvation, through a lite-06 neighbour -- [detour-resume](/quest/m2/multicast/detour-resume.md) - a front that ends, rather than waits, when its session closes before the detour arrives diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs index 788b7176ff..25c07ea91c 100644 --- a/rs/moq-net/src/model/front.rs +++ b/rs/moq-net/src/model/front.rs @@ -45,6 +45,9 @@ pub(super) struct Candidate { /// a carried instance says the answer is the same stream. A minted one names the /// announcement, and the route could answer with whatever stream now wins upstream. pub carried: bool, + /// Whether the route is a lite-07 claim, which the route layer keeps in the table + /// through its debounce once its session closes; see [`Front::waiting`]. + pub claim: bool, } /// Why an upstream request through a route did not produce a source. @@ -197,6 +200,10 @@ struct Track { /// re-splice is awaited. A refusal of that re-splice, or the front ending first, reports /// this instead: the source closing is what ended the track. lost: Option, + /// What the copy died of when its source closed under it with nothing serving the + /// track. A front ending before another source splices it reports this, but unlike + /// [`Self::lost`] a refusal does not: the next source's answer is its own verdict. + closed: Option, /// Whether anyone reads the track: a copy is only spliced in for a reader. wanted: bool, /// A detached source whose copy still feeds the track until the serving source's @@ -236,6 +243,8 @@ pub(super) struct Front { upstream: Option, /// Routes excluded from selection: their source ended while still advertised. excluded: HashSet, + /// The excluded routes that are claims, while the table still shows them. + orphans: HashSet, /// Whether the parked requesters were resolved (the first source attached). resolved: bool, tracks: BTreeMap, Track>, @@ -264,6 +273,7 @@ impl Front { serving_closing: false, upstream: None, excluded: HashSet::new(), + orphans: HashSet::new(), resolved: false, tracks: BTreeMap::new(), parked: BTreeSet::new(), @@ -284,6 +294,16 @@ impl Front { /// Forget excluded routes that left the table (a reconnect is a fresh entry). pub(super) fn retain_routes(&mut self, standing: impl Fn(u64) -> bool) { self.excluded.retain(|route| standing(*route)); + self.orphans.retain(|route| self.excluded.contains(route)); + } + + /// Whether a selection that finds no route leaves the front waiting rather than ending. + /// A claim whose source closed with its session stays in the table, unavailable, until + /// another session serves it, a detour that may be learned only a round trip later and + /// always arrives as another route, or until its debounce runs out. Its own route is + /// never asked again: it would re-request the broadcast that just ended. + pub(super) fn waiting(&self) -> bool { + !self.orphans.is_empty() } /// The route the attached source came through, if any. @@ -299,7 +319,8 @@ impl Front { /// Why `track`'s serving copy died, while its re-splice is still awaited. A track the /// front ends meanwhile ends with this rather than the front's end. pub(super) fn lost(&self, track: &str) -> Option { - self.tracks.get(track)?.lost.clone() + let track = self.tracks.get(track)?; + track.lost.clone().or_else(|| track.closed.clone()) } /// Whether the front is over. @@ -342,6 +363,7 @@ impl Front { refused: HashSet::new(), refusal: None, lost: None, + closed: None, wanted: false, ended: None, draining: None, @@ -433,6 +455,8 @@ impl Front { self.upstream = Some(candidate); actions.push(Action::Request { route: candidate.route }); } + // Nothing serves the path yet, but a claim it lost may still come back on a detour. + None if self.waiting() => self.upstream = None, // Nothing serves the path: the front is over, even with a live source. Its // route left the table and nothing replaced it, so it serves nobody new; // a later announcement gets a fresh front. @@ -466,8 +490,12 @@ impl Front { // The winning route's answer is final, even while another source serves: // asking a sibling or a shorter prefix instead would turn one refusal into // a request per candidate. + // A closed source's error gives way to it too: the route's answer is its own verdict. Err(Refusal { err, standing: true }) => { self.upstream = None; + for track in self.tracks.values_mut() { + track.closed = None; + } self.end(err, actions); } } @@ -520,6 +548,9 @@ impl Front { // would re-request the broadcast that just ended; another route may still // resume it. self.excluded.insert(candidate.route); + if candidate.claim { + self.orphans.insert(candidate.route); + } self.serving = None; self.serving_closing = false; actions.push(Action::Detach { source, drain: true }); @@ -606,6 +637,7 @@ impl Front { Ok(()) => { track.draining = None; track.lost = None; + track.closed = None; track.stranded = None; track.state = TrackState::Spliced { source }; actions.push(Action::Splice { @@ -628,10 +660,14 @@ impl Front { ) { let strands = self.strands(); let serving = self.serving.map(|(serving, _)| serving); + // Nothing serves or is being asked, but a claim the front lost may still resume it. + let waiting = serving.is_none() && self.upstream.is_none() && self.waiting(); let Some(track) = self.tracks.get_mut(&name) else { return; }; let spliced = track.state == (TrackState::Spliced { source }); + // A copy the closed source answered after its close: neither served nor drained. + let late = serving != Some(source) && track.draining != Some(source); // A track finishing on a copy other than the serving source's asks that source for // one anew, since finishing lets go of a query in flight: the group reads still // holding the track continue from it, as after a later source change. @@ -654,6 +690,14 @@ impl Front { return; } if !spliced { + // The source closed under the copy and nothing serves yet: what the copy died + // of is what ends the track, unless a route the front waits on resumes it. + if let Err(err) = result + && serving.is_none() + && track.ended.is_none() + { + track.closed.get_or_insert(err); + } return; } } @@ -673,7 +717,8 @@ impl Front { // route a switch already under way moves to, whatever the error, since the // route it leaves may be what failed. A copy still read from a source that // closed during the switch has no serving source to judge the instance by, so - // the route the switch asks speaks for it. + // the route the switch asks speaks for it. What that copy died of is no + // serving copy's loss: the route's answer is its own verdict, as while waiting. Err(err) if delivered && (err.transient() || self.upstream.is_some()) @@ -682,9 +727,20 @@ impl Front { .is_some_and(|candidate| candidate.carried) => { track.state = TrackState::Idle; - track.lost = Some(err); + if late { + track.closed.get_or_insert(err); + } else { + track.lost = Some(err); + } self.fail_over(name, actions); } + // A copy the closed source answered after its close died while the front waits on + // the claim it served: as for a draining copy, what it died of ends the track + // unless a session that serves the claim next resumes it. + Err(err) if delivered && waiting => { + track.state = TrackState::Idle; + track.closed.get_or_insert(err); + } // The route lost the content, which another route of the instance continues: // wait for one rather than ask this route again, which can only refuse. A copy // drained from a replaced source leaves the track to the serving one. @@ -901,6 +957,7 @@ impl Front { TrackState::Idle | TrackState::Querying { .. } => { track.state = TrackState::Parked { since: now }; track.lost = None; + track.closed = None; // An ended track has nothing to drop. if track.ended.is_none() { actions.push(Action::Park { track: name }); @@ -956,6 +1013,7 @@ mod tests { route, local: false, carried: true, + claim: false, } } @@ -964,6 +1022,7 @@ mod tests { route, local: true, carried: true, + claim: false, } } @@ -1226,6 +1285,288 @@ mod tests { ); } + fn claim(route: u64) -> Candidate { + Candidate { + claim: true, + ..remote(route) + } + } + + /// A front whose claim's session closed under `video` before its copy failed, with no + /// other route yet: it waits on the claim the table still shows, never asking it again. + fn orphaned() -> Front { + let mut front = serving(claim(1), 100); + assert_actions( + front.step(Event::SourceClosed { source: 100 }), + &[ + Action::Detach { + source: 100, + drain: true, + }, + Action::Reselect, + ], + ); + assert_eq!(front.excluded_routes(), &HashSet::from([1])); + assert!(front.waiting()); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[], + ); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::Session(crate::SessionError::Cancel)), + delivered: true, + }), + &[], + ); + front + } + + /// The claim comes back on another session, a detour learned after the close: the + /// front resumes on it rather than end. + #[test] + fn a_closed_claim_resumes_on_the_session_that_serves_it_next() { + let mut front = orphaned(); + assert_actions( + front.step(Event::Selected { + best: Some(claim(2)), + serving_closing: false, + }), + &[Action::Request { route: 2 }], + ); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[Action::Query { + track: name("video"), + source: 200, + }], + ); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Ok(info()), + retry: None, + }), + &[Action::Splice { + track: name("video"), + source: 200, + }], + ); + assert!(front.lost("video").is_none()); + } + + /// The session that serves the claim next frames the track differently: its verdict + /// ends the track, not the closed session's error. + #[test] + fn a_closed_claims_next_session_refusing_ends_with_its_verdict() { + let mut front = orphaned(); + front.step(Event::Selected { + best: Some(claim(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Ok(info().with_compression(track::Compression::Deflate)), + retry: None, + }), + &[Action::Abort { + track: name("video"), + err: Error::Unsupported, + retry: None, + }], + ); + } + + /// Nothing serves the claim again: the front ends, and its track with what ended the + /// copy, the session's error, not the later `Unroutable`. + #[test] + fn a_closed_claim_nothing_resumes_ends_with_the_sessions_error() { + let mut front = orphaned(); + // The claim's debounce ran out: it left the table. + front.retain_routes(|_| false); + assert!(!front.waiting()); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[Action::End { err: Error::Unroutable }], + ); + assert_eq!( + front.lost("video").map(|err| err.to_string()), + Some(Error::Session(crate::SessionError::Cancel).to_string()) + ); + } + + /// The route that serves the claim next refuses the broadcast: every track ends with + /// that refusal, as its requesters do, not the closed session's error. + #[test] + fn a_closed_claims_next_route_refusing_ends_with_its_verdict() { + let mut front = orphaned(); + front.step(Event::Selected { + best: Some(claim(2)), + serving_closing: false, + }); + assert_actions(refuse(&mut front, 2), &[Action::End { err: Error::NotFound }]); + assert!(front.lost("video").is_none()); + } + + /// The closed source answers a query it was asked before its close, and the copy it + /// splices then dies while the front waits on its claim: the track waits too, and the + /// session that serves the claim next resumes it. + #[test] + fn a_copy_spliced_after_its_claim_closed_dying_waits_for_the_detour() { + let mut front = querying(claim(1), 100); + front.step(Event::SourceClosed { source: 100 }); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[], + ); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 100, + closing: true, + result: Ok(info()), + retry: None, + }), + &[Action::Splice { + track: name("video"), + source: 100, + }], + ); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: true, + result: Err(Error::Session(crate::SessionError::Cancel)), + delivered: true, + }), + &[], + ); + assert_eq!( + front.lost("video").map(|err| err.to_string()), + Some(Error::Session(crate::SessionError::Cancel).to_string()) + ); + assert_actions( + front.step(Event::Selected { + best: Some(claim(2)), + serving_closing: false, + }), + &[Action::Request { route: 2 }], + ); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[Action::Query { + track: name("video"), + source: 200, + }], + ); + } + + /// A front whose claim closed, asking the route that serves it next, with a copy the + /// closed source answered after its close spliced in and then dead. + fn late_copy_died_while_asking() -> Front { + let mut front = querying(claim(1), 100); + front.step(Event::SourceClosed { source: 100 }); + assert_actions( + front.step(Event::Selected { + best: Some(claim(2)), + serving_closing: false, + }), + &[Action::Request { route: 2 }], + ); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 100, + closing: true, + result: Ok(info()), + retry: None, + }), + &[Action::Splice { + track: name("video"), + source: 100, + }], + ); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: true, + result: Err(Error::Session(crate::SessionError::Cancel)), + delivered: true, + }), + &[], + ); + front + } + + /// The late copy dies while the next route is asked, and that route refuses the + /// broadcast: the track ends with the refusal, not the closed session's error. + #[test] + fn a_late_copy_dying_while_asking_yields_to_a_broadcast_refusal() { + let mut front = late_copy_died_while_asking(); + assert_actions(refuse(&mut front, 2), &[Action::End { err: Error::NotFound }]); + assert!(front.lost("video").is_none()); + } + + /// The late copy dies while the next route is asked, whose source then refuses the + /// track: the track aborts with that refusal, not the closed session's error. + #[test] + fn a_late_copy_dying_while_asking_yields_to_a_track_refusal() { + let mut front = late_copy_died_while_asking(); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[Action::Query { + track: name("video"), + source: 200, + }], + ); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(Error::Unsupported), + retry: None, + }), + &[Action::Abort { + track: name("video"), + err: Error::Unsupported, + retry: None, + }], + ); + } + fn refuse(front: &mut Front, route: u64) -> Vec { front.step(Event::Resolved { route, @@ -1594,6 +1935,7 @@ mod tests { route: 1, local, carried, + claim: false, }; let abort = [Action::Abort { track: name("video"), @@ -1846,6 +2188,7 @@ mod tests { route: 1, local, carried: false, + claim: false, }, 100, ); @@ -2725,8 +3068,9 @@ mod tests { best: Some(remote(1)), serving_closing: false, }, + // A claim, which its source closing leaves excluded but waited on. Event::Selected { - best: Some(remote(2)), + best: Some(claim(2)), serving_closing: false, }, Event::Selected { @@ -2749,6 +3093,7 @@ mod tests { }), }, Event::SourceClosed { source: 100 }, + Event::SourceClosed { source: 200 }, Event::TrackAssigned { track: name("v"), now: t0, diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index d092c4133c..47091077cf 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -4793,11 +4793,13 @@ impl FrontServe { route: entry.id, local: entry.local, carried: !entry.instance.is_minted(), + claim: entry.node.is_some(), }); - // No candidate irreversibly ends this front. Leave the cache under the same lock - // a lookup takes, before a route can return and a newcomer join the broadcast the - // end actions are about to close. Existing requesters keep their original verdict. - if best.is_none() { + // No candidate irreversibly ends this front, unless it waits on a claim. Leave the + // cache under the same lock a lookup takes, before a route can return and a newcomer + // join the broadcast the end actions are about to close. Existing requesters keep + // their original verdict. + if best.is_none() && !self.front.waiting() { table .fronts .remove_if(&(self.path.clone(), self.horizon, self.pin.clone()), |front| { diff --git a/rs/moq-net/tests/integration/catalog_handover.rs b/rs/moq-net/tests/integration/catalog_handover.rs index 2147dfd016..82ab11c973 100644 --- a/rs/moq-net/tests/integration/catalog_handover.rs +++ b/rs/moq-net/tests/integration/catalog_handover.rs @@ -68,8 +68,8 @@ enum Trigger { Unannounce, /// `R` gains a direct route to `P` that beats `A`. BetterRoute, - /// The session `A -> R` dies with no standby, then reconnects. The broadcast closes - /// with its last route, so a viewer's subscription ends rather than resuming. + /// The session `A -> R` dies with no standby, then reconnects within the claim's + /// debounce: the claim comes back on the new session, and a viewer resumes on it. Reconnect, } @@ -249,15 +249,6 @@ impl Reader { assert_eq!(self.take(expected.len(), who).await, expected, "{who}"); } - /// The subscription fails or ends, rather than hanging or delivering more. - async fn expect_end(&mut self, who: &str) { - match kio::clock::timeout(READ_TIMEOUT, self.rx.next()).await { - Ok(None | Some(Err(_))) => {} - Ok(Some(Ok(frame))) => panic!("{who}: delivered {frame:?} past its route"), - Err(_) => panic!("{who}: hung instead of ending"), - } - } - /// Nothing more has arrived: no duplicate frame or group trails. fn expect_quiet(&mut self, who: &str) { if let Ok(extra) = self.rx.try_recv() { @@ -301,9 +292,6 @@ async fn handover(version: &str, trigger: Trigger, holder: Holder) { }; topology.trigger(trigger).await; - if let (Trigger::Reconnect, Some(mut ended)) = (trigger, viewer.take()) { - ended.expect_end(&format!("{tag}: viewer")).await; - } topology.write("delta2"); if let Some(viewer) = &mut viewer { diff --git a/rs/moq-net/tests/integration/route_audit/contest.rs b/rs/moq-net/tests/integration/route_audit/contest.rs index fc44d30f45..c40c7da872 100644 --- a/rs/moq-net/tests/integration/route_audit/contest.rs +++ b/rs/moq-net/tests/integration/route_audit/contest.rs @@ -3,7 +3,7 @@ //! one Instance to the other mid-stream. moq-transport and lite-06 publishers carry no //! Instance, so each relay that takes their announcement mints one, and the same holds. -use super::{Feed, Join, Mesh, Source, detour_lost, readers, reports, ring, shown}; +use super::{Feed, Join, Mesh, Source, readers, reports, ring, shown}; use crate::path_routes::{lite07, quiesce}; use moq_net::Version; @@ -91,11 +91,6 @@ async fn opposite_claims_each_serve_their_nearest() { let reports = reports(&mut readers, since, "opposite claims, R0-R1 cut"); for (i, reader) in readers.iter_mut().enumerate() { let report = &reports[i]; - if i == 1 { - // The session `R1` read over closed, and its detour arrives a round trip later. - assert!(detour_lost(report, true), "R1: {report:?}"); - continue; - } assert!(report.seamless(), "R{i}: {report:?}"); if let (Some(before), Some(now)) = (near[i], after[i]) && before != now @@ -142,11 +137,6 @@ async fn unversioned(versions: [Version; 2]) { let cut = reports(&mut readers, since, &format!("{versions:?}, R0-R1 cut")); for (i, reader) in readers.iter_mut().enumerate() { let report = &cut[i]; - if i == 1 { - // The session `R1` read over closed, and its detour arrives a round trip later. - assert!(detour_lost(report, true), "{versions:?}: R1: {report:?}"); - continue; - } assert!(report.seamless(), "{versions:?}: R{i}: {report:?}"); assert_eq!(reader.tag().as_deref(), Some(nearest[i]), "{versions:?}: R{i} reads on"); } @@ -161,8 +151,7 @@ async fn unversioned(versions: [Version; 2]) { let report = &lost[i]; assert!(report.clean(), "{versions:?}: R{i}: {report:?}"); assert_eq!(reader.tag().as_deref(), Some("B"), "{versions:?}: R{i} ends on B"); - // `R1` already restarted onto `B` when its detour was lost. - let restarts = usize::from(nearest[i] == "A" && i != 1); + let restarts = usize::from(nearest[i] == "A"); assert_eq!(report.runs.len(), 1 + restarts, "{versions:?}: R{i}: {report:?}"); } } diff --git a/rs/moq-net/tests/integration/route_audit/failures.rs b/rs/moq-net/tests/integration/route_audit/failures.rs index 34fe4f7a11..2b5f3e5249 100644 --- a/rs/moq-net/tests/integration/route_audit/failures.rs +++ b/rs/moq-net/tests/integration/route_audit/failures.rs @@ -2,7 +2,7 @@ //! its neighbours notice seconds apart, failures at one instant on a mesh larger than any //! exhaustive check, and no group crossing a link twice for one subscription. -use super::{Feed, Join, Mesh, Source, detour_lost, measured, readers, reports, ring, shown}; +use super::{Feed, Join, Mesh, Source, measured, readers, reports, ring, shown}; use crate::path_routes::{cut, lite07, quiesce}; use crate::support::Rng; @@ -10,10 +10,9 @@ use std::collections::BTreeSet; use std::time::Duration; /// A ring of `count` relays with a publisher on `R0`, the link from `R1` to `R2` priced -/// `r1_r2`. The link from `R0` to `R1` closes, and `R1` resumes the way round the ring. -/// Unless `held`, every other route to `R0` ran through `R1`, so `R1` held none at the cut: -/// its detour arrives a round trip later, once `R2` hears the closed session named. -async fn detour(count: usize, r1_r2: u64, held: bool) { +/// `r1_r2`. The link from `R0` to `R1` closes, and every reader, `R1`'s included, reads on +/// the way round the ring, every group once. +async fn detour(count: usize, r1_r2: u64) { let mut mesh = Mesh::new(count); for i in 0..count { let cost = if i == 1 { r1_r2 } else { 1 }; @@ -30,10 +29,6 @@ async fn detour(count: usize, r1_r2: u64, held: bool) { quiesce().await; let case = format!("ring of {count}, R1-R2 at {r1_r2}, R0-R1 cut"); for (i, report) in reports(&mut readers, since, &case).iter().enumerate() { - if i == 1 && !held { - assert!(detour_lost(report, false), "{case}: R1: {report:?}"); - continue; - } assert!(report.seamless(), "{case}: R{i}: {report:?}"); } assert_eq!(mesh.doubled(), [], "ring of {count}, R1-R2 at {r1_r2}"); @@ -44,15 +39,17 @@ async fn detour(count: usize, r1_r2: u64, held: bool) { #[kio::test] async fn a_relay_resumes_on_a_detour_it_already_holds() { for (count, r1_r2) in [(3, 1), (4, 3), (5, 4)] { - detour(count, r1_r2, true).await; + detour(count, r1_r2).await; } } -/// `R1` holds no other route to `R0` when its session closes. +/// `R1` holds no other route to `R0` when its session closes: every other one ran through +/// `R1`, so its detour arrives a round trip later, once `R2` hears the closed session named. +/// `R1`'s front rides through the claim's debounce until it does. #[kio::test] async fn a_relay_resumes_on_a_detour_learned_after_the_cut() { for count in [4, 5, 6] { - detour(count, 1, false).await; + detour(count, 1).await; } } diff --git a/rs/moq-net/tests/integration/route_audit/mod.rs b/rs/moq-net/tests/integration/route_audit/mod.rs index d7cd920b51..9137b1dcd2 100644 --- a/rs/moq-net/tests/integration/route_audit/mod.rs +++ b/rs/moq-net/tests/integration/route_audit/mod.rs @@ -439,18 +439,6 @@ impl Reader { } } -/// Pinned, not endorsed, until [the detour fix](/quest/m2/multicast/detour-resume.md): a relay -/// whose serving session closes before the detour to the same Instance reaches it, one -/// round trip later, ends its front. Its reader restarts cleanly onto another Instance -/// when one serves the path (`other`), and otherwise reads nothing more. The fix makes -/// each such reader seamless, and each case calling this changes with it. -pub(super) fn detour_lost(report: &Report, other: bool) -> bool { - match other { - true => report.clean() && report.runs.len() == 2, - false => report.after == 0, - } -} - /// What a reader saw across a failure. #[derive(Clone, Debug, Default)] pub(super) struct Report { diff --git a/rs/moq-net/tests/integration/route_cluster.rs b/rs/moq-net/tests/integration/route_cluster.rs index 228362a5f1..3ade39a7b5 100644 --- a/rs/moq-net/tests/integration/route_cluster.rs +++ b/rs/moq-net/tests/integration/route_cluster.rs @@ -463,8 +463,8 @@ impl Lan { /// publisher only over the CDN's own expensive link, never through the LAN. /// /// At the cut `lan[0]` moves onto the route through the CDN at once: its reader's -/// announcement stays, available. The closed session still ends the subscription it -/// carried, so a fresh one reads on through the CDN. +/// announcement stays, available, and the subscription the closed session carried reads on +/// through the CDN. #[kio::test] async fn a_partitioned_lan_reaches_itself_through_the_cdn_and_heals() { let mut lan = Lan::new().await; @@ -509,15 +509,6 @@ async fn a_partitioned_lan_reaches_itself_through_the_cdn_and_heals() { && moved.at.saturating_duration_since(cut_at) < DEBOUNCE, "through the CDN at once: {partitioned:?}" ); - // The closed session ends the subscription it carried, and a fresh one joins at the - // latest group, the one already read. - let ended = kio::clock::timeout(Duration::from_secs(1), rx.next()).await; - assert!( - matches!(ended, Ok(Some((_, Err(_))))), - "the closed session ended the track: {ended:?}" - ); - let (_broadcast, mut rx) = subscribe(&lan.lan[0], "local").await; - read_group(&mut rx, "O", 0).await; local.group(1); read_group(&mut rx, "O", 1).await; diff --git a/test/mesh-replay/script.toml b/test/mesh-replay/script.toml index 00132c8947..c1cb3e938a 100644 --- a/test/mesh-replay/script.toml +++ b/test/mesh-replay/script.toml @@ -36,12 +36,14 @@ kind = "end-source" kind = "roll" to = "old" -# A follow left holding a broadcast whose track ended, which it never replaces while the same -# Instance serves the path again: the front ended at its session's close. +# Mid-roll, a relay still on the release ends a front at its session's close rather than wait +# for the detour, leaving a follow holding a track that ended. It lasts until the release cut +# ships the fix. [[known]] check = "unfed" kind = "stuck" -quest = "/quest/m2/multicast/detour-resume.md" +event = "roll" +quest = "/quest/m3/release/cut.md" # Mid-roll, a new relay accepts an old neighbour's subscribe that arrives as the neighbour # withdraws its echo of the broadcast, then forwards nothing. `half-roll.toml` reproduces it