diff --git a/.claude/skills/verify-failover/SKILL.md b/.claude/skills/verify-failover/SKILL.md index 4d93b2a9eae..fa0baf0e6c5 100644 --- a/.claude/skills/verify-failover/SKILL.md +++ b/.claude/skills/verify-failover/SKILL.md @@ -24,18 +24,10 @@ Each rule is an INVARIANT, except `Settles`, which is TLC's deadlock check: ever ```sh quest gate just rs tla moq-net/verify/failover/FailoverChain.tla quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Leave.cfg -quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/EndedRelay.cfg +quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Mesh.cfg -simulate num=200000 -depth 100 ``` -The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.2M states. Each takes seconds and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop: it fails at the sync commit on the open finding below at depth 31, after about 50M distinct states and minutes. - -## Open finding: an ended relay cannot move a fetch off a dying route - -`EndedRelay.cfg` breaks `NoSkipWhileHeld`. `E` reads group 0 through `A` and hands out group 1, then switches to `B`, asking it from group 1, and fetches the rest of group 0 from `B`. `B` moves its route to `A`, its copy from group 1 completes, and its track ends (`Action::Finish`); then `A` dies. `B`'s front no longer switches, so its fetch for group 0 fails although `B` still reaches `O` directly, and `E`, whose serving route gave up on the group, cuts it short. - -`residual_a_fetch_cannot_move_off_a_dying_route_after_the_end` in `rs/moq-net/tests/integration/failover_chain.rs` replays it, and the real stack does worse: the read waits out the subscriber's budget (`Settles` too). `B`'s logical track closes once it finished, so its front reads the track as unread and splices no copy from the route to `O` that replaces `A`'s; the fetch's `Recover` waits on a concluded route whose serving copy completed cleanly before its FETCH failed. A fix keeps an ended track routable for a fetch, here and in the model's `Switch` and `Abort`; then flip the test, merge `EndedRelay.cfg`'s `Mesh` into `FailoverChain.cfg` if the bounds allow, and delete `EndedRelay.cfg`. - -The model lets a fetch reach the serving route whenever that route lives, after the track ended too. Without the mesh the code now does: a relay's finished copy keeps answering fetches while it is read (`ending_the_track_keeps_a_fetch_in_flight`, `a_fetch_from_a_completed_copy_goes_on_the_wire`). +The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.2M states. Each takes under a minute and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes. ## Reading a failure @@ -53,8 +45,8 @@ Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e | `rs/moq-net/src/model/resume.rs` `Recover::wants`, `poll`, `held`, `poll_serving` | `Recover`, `Holders`, `Exhausted`, `Abort` | | `rs/moq-net/src/model/resume.rs` `Recover::poll_fetch`, `Fetching`; `rs/moq-net/src/model/track.rs` `fetch_group` | `Ask` | | `rs/moq-net/src/model/front.rs` `track_ended`, `Action::Finish`; `rs/moq-net/src/model/origin.rs` `TrackIo::end` | `FrontFinish`, `Whole` | -| `rs/moq-net/src/model/front.rs` failover and its unroutable end | `Switch`, `FrontFail` | +| `rs/moq-net/src/model/front.rs` failover and its unroutable end, `Action::Reroute` | `Switch`, `FrontFail` | ## Not covered -Routing is abstracted to any live upstream winning, so feasibility, ROUTE_END, and announce order are `verify-route`'s; how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. A pass covers two groups of two frames, one death, one switch per relay, and one client. +Routing is abstracted to any live upstream winning, so feasibility, ROUTE_END, and announce order are `verify-route`'s; how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. A pass covers two groups of two frames, one death, one switch per relay, and one client. diff --git a/quest/m1/failover-fetch-residuals.md b/quest/m1/failover-fetch-residuals.md deleted file mode 100644 index 792afb7c939..00000000000 --- a/quest/m1/failover-fetch-residuals.md +++ /dev/null @@ -1,29 +0,0 @@ -# [M] A fetch survives the track ending on a relay chain - -## Goal - -A relay whose track ended still moves a fetch off a route that dies: -`residual_a_fetch_cannot_move_off_a_dying_route_after_the_end` in -`rs/moq-net/tests/integration/failover_chain.rs` flips to assert delivery, and -`EndedRelay.cfg`'s open finding is gone (its `Mesh` merged into -`FailoverChain.cfg` if the bounds allow). - -## Plan - -Found 2026-11-14 by the [feature review](/quest/m3/release/feature-review.md). -The first two residuals (a fetch in flight when the track ends, and a fetch -asked of a completed copy) are fixed: a relay's finished copy keeps serving -fetches while it is read. - -What remains is the mesh case. B's front ends the track (`Action::Finish`), -so its logical track closes and reads as unread; when A dies, nothing splices -a copy from B's direct route to O, and the fetch's `Recover` waits on a -concluded route until the subscriber's budget runs out. - -- Keep an ended track routable for a fetch in flight in `model/front.rs`, - minding the parked index and `forget`. -- In `FailoverChain.tla`, allow `Switch` for a track with `end = "ok"` whose - serving copy died, and adjust the `Abort` guard to match. -- Run `/verify-failover` and `/verify-resume` on the head. - -Public API: none. Wire: none. diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs index 122db890cc5..214624a1ed8 100644 --- a/rs/moq-net/src/model/front.rs +++ b/rs/moq-net/src/model/front.rs @@ -141,6 +141,13 @@ pub(super) enum Action { /// arrived meanwhile: it went unread for the linger, or the source is local. /// Feed back [`Event::Forgotten`] once it is gone. Forget { track: Arc }, + /// Ask `source` for its copy of the finished `track`, for the reads still holding it: + /// a group read on it continues from that copy if its own route dies. Nothing to do + /// when nothing holds the track; else feed the answer back as [`Event::TrackInfo`]. + Reroute { track: Arc, source: u64 }, + /// No source will replace the copy of the finished `track` any more: a group read on it + /// fails once its own route does, rather than wait for one. + Conclude { track: Arc }, /// The logical track completed. Finish { track: Arc }, /// The logical track failed for good, with the Retry Interval of the upstream refusal it @@ -199,9 +206,20 @@ struct Track { /// table to move, rather than ask that source again at once or end; a draining copy /// keeps feeding it meanwhile. stranded: Option, - /// The track finished or aborted: nothing is spliced again, and it stays only - /// until it goes unread for the linger. - ended: bool, + /// How the track ended, once it did. It stays only until it goes unread for the linger. + ended: Option, +} + +/// How a logical track ended. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Ended { + /// Completed: its readers have all it holds, but a group still read on it moves with + /// the serving source, from a copy only that group's reads use. + Finished, + /// Finished, and the serving source refused it since: nothing is spliced again. + Concluded, + /// Failed for good: nothing is spliced again. + Aborted, } /// One front's state; see the module docs. @@ -323,7 +341,7 @@ impl Front { refusal: None, lost: None, wanted: false, - ended: false, + ended: None, draining: None, stranded: None, }, @@ -464,14 +482,22 @@ impl Front { self.resolved = true; actions.push(Action::Resolve); } - // A read track is never parked, so idle is the only state to re-query. + // A read track is never parked, so idle is the only state to re-query. A finished + // one keeps its state, lingering as before, while a read on it follows the source. for (name, track) in &mut self.tracks { - if track.wanted && !track.ended && track.state == TrackState::Idle { - track.state = TrackState::Querying { source }; - actions.push(Action::Query { + match track.ended { + None if track.wanted && track.state == TrackState::Idle => { + track.state = TrackState::Querying { source }; + actions.push(Action::Query { + track: name.clone(), + source, + }); + } + Some(Ended::Finished) => actions.push(Action::Reroute { track: name.clone(), source, - }); + }), + _ => {} } } } @@ -519,9 +545,40 @@ impl Front { result: Result)>, actions: &mut Vec, ) { + let serving = self.serving.is_some_and(|(serving, _)| serving == source); + let strands = self.strands(); let Some(track) = self.tracks.get_mut(&name) else { return; }; + // A rerouted copy of a finished track joins its reads only from the serving source, + // and only with the same metadata. The serving source refusing it is its verdict, as + // for an open track, so a read whose route dies fails rather than wait for another. + // A closing source's answer or a lost route's is none: its replacement is asked. + if track.ended == Some(Ended::Finished) { + if !serving { + return; + } + let verdict = match result { + Ok(info) if self.info.get(&name) == Some(&info) => Ok(()), + Ok(_) => Err(Error::Unsupported), + Err((err, _)) => Err(err), + }; + match verdict { + Ok(()) => { + track.stranded = None; + actions.push(Action::Splice { track: name, source }); + } + Err(_) if closing => {} + // The route lost it: ask again once the table moves, as for an open track. A + // local or minted route's word is the verdict, as it is there. + Err(err) if err.unroutable() && strands => track.stranded = Some(source), + Err(_) => { + track.ended = Some(Ended::Concluded); + actions.push(Action::Conclude { track: name }); + } + } + return; + } if track.state != (TrackState::Querying { source }) { return; } @@ -563,19 +620,30 @@ impl Front { actions: &mut Vec, ) { let strands = self.strands(); + let serving = self.serving.map(|(serving, _)| serving); let Some(track) = self.tracks.get_mut(&name) else { return; }; let spliced = track.state == (TrackState::Spliced { source }); + // A track finishing on a copy other than the serving source's asks that source for + // one anew, since finishing lets go of a query in flight: the group reads still + // holding the track continue from it, as after a later source change. + let reroute = serving.filter(|serving| track.state != (TrackState::Spliced { source: *serving })); + let finish = |name: Arc, actions: &mut Vec| { + actions.push(Action::Finish { track: name.clone() }); + if let Some(source) = reroute { + actions.push(Action::Reroute { track: name, source }); + } + }; if track.draining == Some(source) { track.draining = None; // Every source of a front serves one broadcast, so a copy that completed ends // the track whoever serves it now. One that failed leaves the track to the // serving source. - if result.is_ok() && !track.ended { + if result.is_ok() && track.ended.is_none() { track.state = TrackState::Idle; - track.ended = true; - actions.push(Action::Finish { track: name }); + track.ended = Some(Ended::Finished); + finish(name, actions); return; } if !spliced { @@ -590,8 +658,8 @@ impl Front { // once unread for the linger, like any other track. Ok(()) => { track.state = TrackState::Idle; - track.ended = true; - actions.push(Action::Finish { track: name }); + track.ended = Some(Ended::Finished); + finish(name, actions); } // Died mid-serve after delivering from the same instance: normal // failover, re-splice from the serving source at the first missing group. @@ -618,7 +686,7 @@ impl Front { // failure cannot be retried against the same content. Err(err) if delivered => { track.state = TrackState::Idle; - track.ended = true; + track.ended = Some(Ended::Aborted); actions.push(Action::Abort { track: name, err, @@ -686,12 +754,12 @@ impl Front { return; }; let track = self.tracks.get_mut(&name).expect("dispatching a known track"); - if !track.wanted || track.ended || track.state != TrackState::Idle || track.stranded == Some(source) { + if !track.wanted || track.ended.is_some() || track.state != TrackState::Idle || track.stranded == Some(source) { return; } if track.refused.contains(&source) { let (err, retry) = track.refusal.clone().unwrap_or((Error::NotFound, None)); - track.ended = true; + track.ended = Some(Ended::Aborted); actions.push(Action::Abort { track: name, err, @@ -729,7 +797,9 @@ impl Front { track.stranded = None; match track.state { TrackState::Spliced { source: draining } - if track.draining == Some(draining) && track.wanted && !track.ended && !self.serving_closing => + if track.draining == Some(draining) + && track.wanted && track.ended.is_none() + && !self.serving_closing => { track.state = TrackState::Querying { source }; actions.push(Action::Query { @@ -737,6 +807,15 @@ impl Front { source, }); } + // A finished track asks for a rerouted copy anew; its state stays as it lingers. + _ if track.ended == Some(Ended::Finished) => { + if !self.serving_closing { + actions.push(Action::Reroute { + track: name.clone(), + source, + }); + } + } _ => idle.push(name.clone()), } } @@ -786,7 +865,7 @@ impl Front { track.state = TrackState::Parked { since: now }; track.lost = None; // An ended track has nothing to drop. - if !track.ended { + if track.ended.is_none() { actions.push(Action::Park { track: name }); } } @@ -1939,7 +2018,8 @@ mod tests { } /// A finished track stays for readers still draining it and for the linger - /// after the last one, is never spliced again, and is then forgotten. + /// after the last one, and is then forgotten. A replacement source reroutes it for the + /// reads still holding it, without touching the linger. #[test] fn a_finished_track_lingers_then_is_forgotten() { let mut front = serving(remote(1), 100); @@ -1968,7 +2048,8 @@ mod tests { front.step(Event::Wanted { track: name("video") }), &[Action::Arm { at: None }], ); - // A replacement source does not re-splice it either. + // A replacement source asks for its copy, and splices the answer, for a group still + // read on the track. Nothing ends or lingers again. front.step(Event::Selected { best: Some(remote(2)), serving_closing: false, @@ -1978,11 +2059,40 @@ mod tests { route: 2, result: Ok(200), }), - &[Action::Detach { - source: 100, - drain: false, + &[ + Action::Detach { + source: 100, + drain: false, + }, + Action::Reroute { + track: name("video"), + source: 200, + }, + ], + ); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Ok(info()), + retry: None, + }), + &[Action::Splice { + track: name("video"), + source: 200, }], ); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 200, + closing: false, + result: Ok(()), + delivered: false, + }), + &[], + ); let t1 = t0.saturating_add(LINGER); front.step(Event::Unwanted { track: name("video"), @@ -1998,6 +2108,128 @@ mod tests { assert!(!front.tracks.contains_key(&name("video"))); } + /// A rerouted copy of a finished track joins its reads only from the serving source and + /// with the track's metadata. A lost route is asked again once the table moves. That + /// source refusing it, other than closing or losing the route, concludes the track: + /// nothing is rerouted again. A parked one keeps its linger either way. + #[test] + fn a_finished_track_splices_only_a_matching_copy_from_the_serving_source() { + for refusal in [None, Some(Ok(track::Info::default())), Some(Err(Error::NotFound))] { + let mut front = serving(remote(1), 100); + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Ok(()), + delivered: true, + }); + let t0 = Instant::now(); + front.step(Event::Unwanted { + track: name("video"), + now: t0, + }); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + let answer = |source, closing, result| Event::TrackInfo { + track: name("video"), + source, + closing, + result, + retry: None, + }; + + // From a source the front no longer serves from, a closing one's refusal, or a + // lost route's. + assert_actions(front.step(answer(100, false, Ok(info()))), &[]); + assert_actions(front.step(answer(200, true, Err(Error::NotFound))), &[]); + assert_actions(front.step(answer(200, false, Err(Error::Unroutable))), &[]); + // The lost route is asked again once the table moves, and only then. + let moved = || Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }; + assert_actions( + front.step(moved()), + &[Action::Reroute { + track: name("video"), + source: 200, + }], + ); + assert_actions(front.step(moved()), &[]); + match refusal { + // Other metadata, or a refusal. + Some(result) => { + assert_actions( + front.step(answer(200, false, result)), + &[Action::Conclude { track: name("video") }], + ); + assert_actions(front.step(answer(200, false, Ok(info()))), &[]); + front.step(Event::Selected { + best: Some(remote(1)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 1, + result: Ok(300), + }), + &[Action::Detach { + source: 200, + drain: false, + }], + ); + } + None => assert_actions( + front.step(answer(200, false, Ok(info()))), + &[Action::Splice { + track: name("video"), + source: 200, + }], + ), + } + assert_index_matches_scan(&front); + assert_eq!(front.next_deadline(), Some(t0.saturating_add(LINGER))); + } + } + + /// A local route has no stale view to wait out: its unroutable answer for a finished + /// track's rerouted copy concludes the track, as any other refusal does. + #[test] + fn a_local_routes_unroutable_answer_concludes_a_finished_track() { + let mut front = serving(remote(1), 100); + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Ok(()), + delivered: true, + }); + front.step(Event::Selected { + best: Some(local(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(Error::Unroutable), + retry: None, + }), + &[Action::Conclude { track: name("video") }], + ); + } + /// An aborted track lingers the same way rather than staying for the life of /// the front. #[test] @@ -2532,4 +2764,43 @@ mod tests { &[Action::Finish { track: name("video") }], ); } + + /// A draining copy ending cleanly while the serving source's query is in flight finishes + /// the track, and asks that source again: finishing lets go of the query, and a group + /// read still holding the track continues from the serving source's copy. + #[test] + fn a_drained_clean_end_reroutes_to_the_serving_source() { + let mut front = serving(remote(1), 100); + front.step(Event::SourceClosed { source: 100 }); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[Action::Query { + track: name("video"), + source: 200, + }], + ); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Ok(()), + delivered: true, + }), + &[ + Action::Finish { track: name("video") }, + Action::Reroute { + track: name("video"), + source: 200, + }, + ], + ); + } } diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index eda22ae8a92..2848fa8fd9c 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -4845,8 +4845,9 @@ impl FrontServe { } } // An awaited copy follows the readers, or it may never send the group - // they now want. - if asked.info.is_some() { + // they now want. The readers of an ended track stay on the copy it ended + // on, so one rerouted for its group reads subscribes nothing. + if asked.info.is_some() && !matches!(io.logical, Logical::Ended) { follow_demand( &mut asked.sub, Some(&asked.copy), @@ -4858,12 +4859,15 @@ impl FrontServe { } // While another copy may still deliver, a matching replacement answers only // once it delivers or ends: one failing first refuses, and that copy keeps - // serving. Different metadata is refused at once. + // serving. Different metadata is refused at once. An ended track's readers + // stay on its last copy, so a copy rerouted for its group reads answers with + // its info alone: it may never deliver anything unasked. if let Some(info) = &asked.info - && (io - .copy - .as_ref() - .is_none_or(|(_, copy, _)| copy.poll_complete(waiter).is_ready()) + && (matches!(io.logical, Logical::Ended) + || io + .copy + .as_ref() + .is_none_or(|(_, copy, _)| copy.poll_complete(waiter).is_ready()) || matches!(&io.logical, Logical::Accepted(accepted) if accepted.info() != info) || asked.copy.poll_grouped(waiter).is_ready() || asked.copy.poll_complete(waiter).is_ready()) @@ -5126,6 +5130,50 @@ impl kio::Task for FrontServe { None => {} } } + Action::Reroute { track: name, source } => { + let Some(io) = self.tracks.get_mut(&name) else { continue }; + // Nothing reads it any more, so nothing would continue on the copy. + if !io.weak.is_held() { + continue; + } + io.ready.wake_by_ref(); + let closing = self.sources.get(&source).is_some_and(|s| s.is_closing()); + let lookup = match self.sources.get(&source).map(|s| s.track(&name)) { + Some(Ok(lookup)) => lookup, + // Refused before any lookup, as a query would be: the front judges + // it the same way, so a held fetch is not left waiting on it. + Some(Err(err)) => { + self.events.push_back(Event::TrackInfo { + track: name, + source, + closing, + result: Err(err), + retry: None, + }); + continue; + } + None => continue, + }; + // No subscription: the track is over, and a read that continues on the + // copy asks it for what it needs. + let query = lookup.info().into_inner(); + let copy = query.track(); + io.query = Some(Asked { + source, + staged: io.routes.stage(copy.clone()), + copy, + query, + info: None, + sub: None, + }); + } + Action::Conclude { track: name } => { + let Some(io) = self.tracks.get_mut(&name) else { continue }; + io.ready.wake_by_ref(); + io.routes.conclude(); + io.query = None; + io.staged = None; + } Action::Splice { track: name, source } => { let Some(io) = self.tracks.get_mut(&name) else { continue }; let Some(asked) = io.staged.take() else { @@ -5385,10 +5433,11 @@ impl kio::Task for FrontServe { }, Err(err) => Err(err), }; - // Staged only while the track has a reader: without one the machine - // will not splice, and a held copy would keep the source subscribed. + // Staged only while the track has a reader, or a read holds the finished + // track: without one the machine will not splice, and a held copy would + // keep the source subscribed. if let Ok(info) = &result - && io.wanted + && (io.wanted || io.routes.ended()) { io.staged = Some(Asked { info: Some(info.clone()), @@ -5424,9 +5473,11 @@ impl kio::Task for FrontServe { continue; } io.wanted = !io.wanted; - if !io.wanted { - // Nothing will be spliced now: let go of the copies a query - // holds, or the source stays subscribed with nobody reading. + // Nothing will be spliced now: let go of the copies a query holds, or the + // source stays subscribed with nobody reading. An ended track's query is a + // reroute for the reads still holding it, which subscribes nothing: finishing + // is what closes the track, so its edge may come only after the reroute. + if !io.wanted && !io.routes.ended() { io.query = None; io.staged = None; } @@ -15656,6 +15707,63 @@ mod tests { .expect("open"); } + /// A fetch holds a finished track whose group its route lost, and the replacement neither + /// has the track nor takes requests: refused at the lookup itself, that concludes the + /// track as a refused query would, so the fetch fails rather than wait out its budget. + #[kio::test] + #[cfg(feature = "async")] + async fn a_finished_tracks_fetch_fails_when_the_replacement_lacks_the_track() { + let producer = origin(1).produce(); + let server = producer + .forward( + "room", + RouteInfo::new().with_instance(instance(), 0).with_hops(hops(&[10])), + ) + .unwrap(); + let pending = producer.consume().request("room/alice", None); + let upstream = broadcast::Info::new().produce(); + let old = upstream.create_track("video", crate::track::Info::timed()).unwrap(); + let fills = old.requests(); + queued(&server).await.serve(&upstream); + let resolved = pending.await.expect("resolves"); + let video = resolved.track("video").unwrap(); + let mut open = old.append_next().unwrap(); + open.write_frame(open.key_time(), b"a".as_ref()).unwrap(); + let finished = old.finish().unwrap(); + let mut fetched = video.fetch_group(0, None).await.unwrap(); + assert_eq!(&fetched.read_frame().await.unwrap().unwrap().payload[..], b"a"); + + // The route loses the rest of the group and refuses to fill it. + open.abort(Error::Cancel).unwrap(); + assert!(fetched.read_frame().now_or_never().is_none()); + let request = kio::clock::timeout(Duration::from_secs(1), fills.requested_fill_group()) + .await + .expect("the fetch asked the route to fill the group") + .expect("request"); + request.reject(Error::Cancel); + drop((fills, finished)); + for _ in 0..10 { + kio::clock::sim::yield_now().await; + } + assert!( + fetched.read_frame().now_or_never().is_none(), + "the fetch waits for another route" + ); + + let newcomer = producer + .route("room/alice", RouteOptions::hidden().with_instance(instance(), 0)) + .unwrap(); + let _other = newcomer + .broadcast() + .create_track("audio", crate::track::Info::timed()) + .unwrap(); + newcomer.announce().unwrap(); + let res = kio::clock::timeout(Duration::from_secs(1), fetched.read_frame()) + .await + .expect("the fetch fails once the replacement refused the track"); + assert!(res.is_err(), "{res:?}"); + } + /// The newcomer answers "video" with matching metadata but aborts before delivering a /// group, all before the front polls the answer: that is a refusal, so the incumbent /// keeps serving rather than the track ending on the newcomer's error. diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 7c370d5a628..37b1608d80e 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -87,6 +87,19 @@ impl Route { false => Poll::Pending, } } + + /// The copy a read goes to first. Once the track finished, that is the copy it ended + /// on until that copy fails: one the front serves after the end caches nothing, so + /// asking it would send upstream what the last copy still answers. + fn answering(&self) -> Option<(u64, &track::Consumer)> { + match &self.phase { + Phase::Finished { + last: (generation, Some(copy)), + .. + } if !copy_failed(copy) => Some((*generation, copy)), + _ => self.copy.as_ref().map(|copy| (self.generation, copy)), + } + } } #[derive(Default)] @@ -95,8 +108,16 @@ enum Phase { Open, /// Readers follow the last copy to its end; no front will replace it. Concluded, - /// The front's final verdict on the logical track. - Ended(Result<()>), + /// The front ended the track cleanly. Its readers stay on the generation and copy it + /// ended on (`last`), which holds everything below the end. While `routed`, the front + /// still replaces the serving copy when its source changes, for the group reads and + /// fetches still holding the track, so they move off a route that dies. + Finished { + last: (u64, Option), + routed: bool, + }, + /// The front failed the track for good. + Failed(Error), } impl Phase { @@ -104,10 +125,17 @@ impl Phase { !matches!(self, Self::Open) } - fn end(&self) -> Option<&Result<()>> { + /// Whether a front may still replace the copy. + fn moves(&self) -> bool { + matches!(self, Self::Open | Self::Finished { routed: true, .. }) + } + + /// The front's verdict on the track, once it gave one. + fn end(&self) -> Option> { match self { - Self::Ended(end) => Some(end), - _ => None, + Self::Finished { .. } => Some(Ok(())), + Self::Failed(err) => Some(Err(err.clone())), + Self::Open | Self::Concluded => None, } } } @@ -248,22 +276,31 @@ impl Producer { self.state.read().phase.end().is_some() } - /// The track ended: cleanly with `Ok`, or failed for good. + /// The track ended: cleanly with `Ok`, or failed for good. A clean end still moves to + /// each copy the front serves from until it concludes. pub(super) fn end(&self, result: Result<()>) { if let Ok(mut route) = self.state.write() && route.phase.end().is_none() { - route.phase = Phase::Ended(result); + route.phase = match result { + Ok(()) => Phase::Finished { + last: (route.generation, route.copy.clone()), + routed: !matches!(route.phase, Phase::Concluded), + }, + Err(err) => Phase::Failed(err), + }; } } /// No front will replace the serving copy: readers follow it to its end, while this /// still holds it for the readers on their way. pub(super) fn conclude(&self) { - if let Ok(mut route) = self.state.write() - && !route.phase.terminal() - { - route.phase = Phase::Concluded; + if let Ok(mut route) = self.state.write() { + match &mut route.phase { + Phase::Open => route.phase = Phase::Concluded, + Phase::Finished { routed, .. } => *routed = false, + Phase::Concluded | Phase::Failed(_) => {} + } } } @@ -274,6 +311,9 @@ impl Producer { route.copy = None; route.parked = None; route.spliced = None; + if let Phase::Finished { last: (_, copy), .. } = &mut route.phase { + *copy = None; + } } } @@ -317,9 +357,10 @@ pub(super) struct Consumer { } impl Consumer { - /// The serving route's copy, if any. + /// The copy a read goes to first: the serving route's, or the one a finished track + /// ended on while it has not failed. pub(super) fn serving(&self) -> Option { - self.state.read().copy.clone() + self.state.read().answering().map(|(_, copy)| copy.clone()) } /// The copy the front parked, while it serves none and may still serve one again. @@ -345,8 +386,8 @@ impl Consumer { // One read, so a copy served just before the front concluded is the one polled. let (generation, copy, end) = { let route = self.state.read(); - let end = route.phase.terminal().then(|| route.phase.end().cloned()); - (route.generation, route.copy.clone(), end) + let end = route.phase.terminal().then(|| route.phase.end()); + (route.generation, route.answering().map(|(_, copy)| copy.clone()), end) }; // A replacement copy can start below what an earlier one reported. match copy.map(|copy| copy.poll_largest_after(seen, bounds, waiter)) { @@ -367,7 +408,11 @@ impl Consumer { // One read, so a copy served just before the front concluded is the one polled. let (generation, copy, terminal) = { let route = self.state.read(); - (route.generation, route.copy.clone(), route.phase.terminal()) + ( + route.generation, + route.answering().map(|(_, copy)| copy.clone()), + route.phase.terminal(), + ) }; if copy.is_some_and(|copy| copy.poll_grouped(waiter).is_ready()) { return Poll::Ready(()); @@ -440,6 +485,7 @@ impl Consumer { options, logical, pending: None, + left: None, } } @@ -467,7 +513,13 @@ impl Consumer { /// decision. fn end(&self) -> Option>> { let route = self.state.read(); - route.phase.terminal().then(|| route.phase.end().cloned()) + route.phase.terminal().then(|| route.phase.end()) + } + + /// Whether a front may still replace the copy: the track is open, or finished while + /// its front still serves it. + fn moves(&self) -> bool { + self.state.read().phase.moves() } } @@ -1244,7 +1296,13 @@ impl Reader { let (generation, serving) = { let route = self.route.state.read(); - (route.generation, route.copy.clone()) + match &route.phase { + Phase::Finished { + last: (generation, copy), + .. + } => (*generation, copy.clone()), + _ => (route.generation, route.copy.clone()), + } }; if self.generation == Some(generation) { return; @@ -2256,23 +2314,31 @@ impl Recover { } } loop { - let (generation, serving, end, terminal) = { + let (generation, serving, end, moves) = { let route = self.route.consumer().read(); ( route.generation, route.copy.clone(), - route.phase.end().cloned(), - route.phase.terminal(), + route.phase.end(), + route.phase.moves(), ) }; - // Registered before looking, so a route change from here on wakes the reader. Not - // through the watch: `generation` is the one just read, not `self.generation`. + // Once no front will replace the copy, nothing comes after it. + let done = !moves; + // Registered before looking, so a route change from here on wakes the reader, a + // finished track's too. Not through the watch: `generation` is the one just read, + // not `self.generation`. let moved = self .route .consumer() - .poll(waiter, |route| route.moved_past(generation)) + .poll(waiter, |route| { + match route.generation != generation || !route.phase.moves() { + true => Poll::Ready(()), + false => Poll::Pending, + } + }) .is_ready(); - if moved && !terminal { + if moved && moves { continue; } if let Some(Err(err)) = &end { @@ -2297,13 +2363,13 @@ impl Recover { // the withdrawal's cause with a refusal. None comes once the front is done. let withdrawn = failed.filter(|err| generation == self.generation && err.unroutable()); if let Some(err) = withdrawn - && terminal + && done { return Poll::Ready(Err(err.clone())); } let Some(serving) = serving else { // Nothing serves: a failed copy waits for a route, unless none will come. - return match (failed, terminal) { + return match (failed, done) { (Some(err), true) => Poll::Ready(Err(err.clone())), _ => Poll::Pending, }; @@ -2313,9 +2379,10 @@ impl Recover { return Poll::Pending; } let res = self.poll_serving(generation, &serving, index, failed, withdrawn.is_none(), waiter); - // The serving route failed too, and no front is left to replace it. + // The serving route failed too, or will not fetch the rest while its subscription + // will not deliver it either, and no front is left to replace it. if let (Poll::Pending, Some(err)) = (&res, failed) - && terminal && copy_failed(&serving) + && done && (copy_failed(&serving) || self.refused(generation, &serving)) { return Poll::Ready(Err(err.clone())); } @@ -2329,6 +2396,14 @@ impl Recover { } } + /// Whether `serving`, the copy of route `generation`, failed or refused the fetch of the + /// rest, and is the copy that failed or will not deliver the group otherwise, as + /// [`Self::poll_fetch`] judges a refusal. + fn refused(&self, generation: u64, serving: &track::Consumer) -> bool { + matches!(&self.fetch, Some((asked, Fetch::Refused(_))) if *asked == generation) + && (generation == self.generation || serving.poll_group(self.key, &kio::Waiter::noop()).is_ready()) + } + /// The group from the newest of the reader's other copies that still holds it unfailed, /// positioned at frame `index`. fn held(&self, index: u64) -> Option { @@ -2543,6 +2618,9 @@ pub(super) struct Fetching { logical: kio::Consumer, /// The fetch in flight, and the copy it went to. pending: Option, + /// The generation of a finished track's last copy that failed this fetch, which leaves + /// it to the copy served after the end. + left: Option, } /// Which copy a fetch went to. @@ -2579,11 +2657,19 @@ impl Fetching { } } - /// Poll for the serving copy or the staged one to change, or the track to end. + /// Whether the serving copy or the staged one changed since the snapshot of `generation` + /// and `staged`. + fn moved(&self, generation: u64, staged: Option) -> bool { + let route = self.route.state.read(); + route.generation != generation || route.staged.as_ref().map(|(id, _)| *id) != staged + } + + /// Poll for the serving copy or the staged one to change, or for no front to be left to + /// change it. fn poll_moved(&self, generation: u64, staged: Option, waiter: &kio::Waiter) -> Poll<()> { match self.route.state.poll(waiter, |route| { let moved = route.generation != generation || route.staged.as_ref().map(|(id, _)| *id) != staged; - match moved || route.phase.terminal() { + match moved || !route.phase.moves() { true => Poll::Ready(()), false => Poll::Pending, } @@ -2603,11 +2689,19 @@ impl kio::Task for Fetching { let route = self.route.state.read(); ( route.generation, - route.copy.clone(), + route.answering().map(|(generation, copy)| (generation, copy.clone())), route.staged.clone(), route.spliced, ) }; + // A finished track's last copy that failed this fetch leaves it to the copy served + // after the end. + let serving = match serving { + Some((last, _)) if last != generation && self.left == Some(last) => { + self.route.state.read().copy.clone().map(|copy| (generation, copy)) + } + serving => serving, + }; let staged_id = staged.as_ref().map(|(id, _)| *id); // A fetch sent to a staged copy carries on once that copy is spliced. if let Some(asked) = &mut self.pending @@ -2620,28 +2714,31 @@ impl kio::Task for Fetching { // The serving copy answers; with none, the copy whose info is in flight is asked // at once rather than a round trip later, and judged once it is spliced. let (target, copy) = match (serving, staged) { - (Some(serving), _) => (Target::Serving(generation), serving), - (None, Some((id, copy))) if self.route.end().is_none() => (Target::Staged(id), copy), + (Some((generation, serving)), _) => (Target::Serving(generation), serving), + (None, Some((id, copy))) if self.route.moves() => (Target::Staged(id), copy), (None, _) => { - let end = self.route.end(); + if self.route.moves() { + ready!(self.poll_moved(generation, staged_id, waiter)); + continue; + } + // The route moved since the snapshot, as the front left it. + if self.moved(generation, staged_id) { + continue; + } // A fetch whose staged copy failed reports that failure once no route // follows, as one the serving copy failed does. - if end.is_some() - && let Some(err) = self.staged_failure(waiter) - { + if let Some(err) = self.staged_failure(waiter) { return Poll::Ready(Err(err)); } - return match end { + return match self.route.end() { Some(Some(Err(err))) => Poll::Ready(Err(err)), - Some(_) => Poll::Ready(Err(Error::NotFound)), - None => { - ready!(self.poll_moved(generation, staged_id, waiter)); - continue; - } + _ => Poll::Ready(Err(Error::NotFound)), }; } }; - if self.poll_moved(generation, staged_id, waiter).is_ready() && self.route.end().is_none() { + // Re-read whenever the route moved past the snapshot, even if no front is left to + // move it again: the copy served last may answer what the snapshot's cannot. + if self.poll_moved(generation, staged_id, waiter).is_ready() && self.moved(generation, staged_id) { continue; } if self.pending.as_ref().is_none_or(|asked| asked.target != target) { @@ -2664,14 +2761,17 @@ impl kio::Task for Fetching { } }, }; - // Never handed out before its copy's info passes the front's check. - if let Target::Staged(_) = target { - asked.answer = Some(result); - return match self.route.end() { - Some(Some(Err(err))) => Poll::Ready(Err(err)), - _ => Poll::Pending, - }; - } + let served = match target { + Target::Serving(served) => served, + // Never handed out before its copy's info passes the front's check. + Target::Staged(_) => { + asked.answer = Some(result); + return match self.route.end() { + Some(Some(Err(err))) => Poll::Ready(Err(err)), + _ => Poll::Pending, + }; + } + }; let serving = copy; return match result { Ok(group) => { @@ -2685,7 +2785,7 @@ impl kio::Task for Fetching { reader: None, max_delay: self.options.options.max_delay, route: self.route.watch(), - generation, + generation: served, copy: serving.clone(), handoff, lease: None, @@ -2695,9 +2795,19 @@ impl kio::Task for Fetching { recovering: false, }))) } - // The route failed: ask the next one, unless none will come. Kept as the answer, - // since the same copy is never asked twice. - Err(err) if (err.retryable() || copy_failed(&serving)) && self.route.end().is_none() => { + // A finished track's last copy failed: the copy served after the end is already + // there, whether or not a front may still replace it. + Err(err) if (err.retryable() || copy_failed(&serving)) && served != generation => { + self.left = Some(served); + continue; + } + // The route failed: ask the next one, unless none will come and none came since + // the snapshot, which a front may have spliced just before concluding. Kept as + // the answer, since the same copy is never asked twice. + Err(err) + if (err.retryable() || copy_failed(&serving)) + && (self.route.moves() || self.moved(generation, staged_id)) => + { if let Some(asked) = &mut self.pending { asked.answer = Some(Err(err)); } @@ -4167,6 +4277,126 @@ mod test { assert_eq!(read(&mut fetched), None); } + /// A fetched group whose route dies after the front finished the track continues on the + /// copy the front serves next, and fails, rather than waiting, once the front concluded. + #[test] + #[cfg(feature = "async")] + fn a_finished_tracks_fetch_moves_off_a_dying_route() { + for rerouted in [true, false] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + let dynamic = a.requests(); + routes.serve(a.consume()); + let mut open = a.create_group(group::Info { start: 0 }).unwrap(); + open.write_frame(ts(0), b"a".as_ref()).unwrap(); + let _a = a.finish().unwrap(); + let mut fetched = logical.consume().fetch_group(0, None).now_or_never().unwrap().unwrap(); + assert_eq!(read(&mut fetched), Some(b"a".to_vec())); + routes.end(Ok(())); + + // The route dies under the open group, and fails the fetch of the rest too. + open.abort(Error::Cancel).unwrap(); + assert!(fetched.read_frame().now_or_never().is_none(), "rerouted={rerouted}"); + let request = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); + request.reject(Error::Cancel); + assert!(fetched.read_frame().now_or_never().is_none(), "rerouted={rerouted}"); + + if !rerouted { + drop(routes); + let res = fetched.read_frame().now_or_never(); + assert!(matches!(res, Some(Err(Error::Cancel))), "{res:?}"); + continue; + } + let b = copy(); + let mut whole = b.create_group(group::Info { start: 0 }).unwrap(); + whole.write_frame(ts(0), b"a".as_ref()).unwrap(); + whole.write_frame(ts(1), b"b".as_ref()).unwrap(); + whole.close().unwrap(); + routes.serve(b.consume()); + assert_eq!(read(&mut fetched), Some(b"b".to_vec())); + assert_eq!(read(&mut fetched), None); + } + } + + /// A fetch of a finished track goes to the copy it ended on, which caches what the + /// copy rerouted after the end does not, and to the rerouted copy only once that one + /// failed. + #[test] + #[cfg(feature = "async")] + fn a_finished_tracks_fetch_goes_to_its_last_copy_first() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + let dynamic = a.requests(); + routes.serve(a.consume()); + let mut group = a.create_group(group::Info { start: 0 }).unwrap(); + group.write_frame(ts(0), b"a".as_ref()).unwrap(); + group.close().unwrap(); + // Groups 1 and 2 are missing below the end: a fetch asks for them. + a.create_group(group::Info { start: 3 }).unwrap().close().unwrap(); + let last = a.consume(); + let a = a.finish().unwrap(); + routes.end(Ok(())); + + let b = copy(); + let rerouted = b.requests(); + routes.serve(b.consume()); + assert!(routes.consume().serving().is_some_and(|copy| copy.same_channel(&last))); + let mut fetched = logical.consume().fetch_group(0, None).now_or_never().unwrap().unwrap(); + assert_eq!(read(&mut fetched), Some(b"a".to_vec())); + assert!( + rerouted.requested_fill_group().now_or_never().is_none(), + "the rerouted copy is not asked" + ); + + // The last copy failing a fetch, or failing outright, leaves it to the rerouted one, + // even once no front is left to replace that. + routes.conclude(); + let mut fetch = logical.consume().fetch_group(1, None); + assert!((&mut fetch).now_or_never().is_none()); + let request = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); + request.reject(Error::Cancel); + assert!((&mut fetch).now_or_never().is_none()); + assert!( + rerouted + .requested_fill_group() + .now_or_never() + .is_some_and(|fill| fill.is_ok()) + ); + a.abort(Error::Cancel).unwrap(); + let mut fetch = logical.consume().fetch_group(2, None); + assert!((&mut fetch).now_or_never().is_none()); + assert!( + rerouted + .requested_fill_group() + .now_or_never() + .is_some_and(|fill| fill.is_ok()) + ); + } + + /// A copy the front serves after the clean end only continues group reads and fetches: a + /// reader still draining the copy the track ended on stays there, rather than losing what + /// only that copy holds (`ResumeReader.tla`'s `CleanEndIsComplete`). + #[test] + fn a_reader_stays_on_the_copy_its_track_finished_on() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 2); + let _a = a.finish().unwrap(); + routes.end(Ok(())); + + let b = copy(); + write(&b, 0); + routes.serve(b.consume()); + assert_eq!(drain(&mut sub, false), [2]); + let end = sub.poll_recv_group(&kio::Waiter::noop()); + assert!(matches!(end, Poll::Ready(Ok(None))), "the track ends cleanly"); + } + /// A session's copy whose info is still in flight, and its fetch handler. fn pending_copy() -> (track::Request, track::Requests) { let request = track::Request::new(Arc::new(broadcast::Info::default()), "video"); diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 7c47c639982..2977264af7c 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -5620,6 +5620,12 @@ impl TrackWeak { !self.state.is_closed() && self.state.wanted() } + /// Whether any handle still reads the track, closed or not, such as a fetch of a finished + /// track still reading its group. + pub(super) fn is_held(&self) -> bool { + self.state.wanted() + } + /// A [`Demand`] on the cached track, for a broadcast to watch. pub(super) fn demand(&self) -> Demand { Demand::new(self.name.clone(), self.state.clone()) diff --git a/rs/moq-net/tests/integration/failover_chain.rs b/rs/moq-net/tests/integration/failover_chain.rs index d5e9ecbcfe9..fe2030fdc22 100644 --- a/rs/moq-net/tests/integration/failover_chain.rs +++ b/rs/moq-net/tests/integration/failover_chain.rs @@ -209,21 +209,34 @@ async fn a_fetch_after_the_finished_copies_linger_goes_on_the_wire() { assert_eq!(frame.unwrap().unwrap(), Some(payload(0, 1))); } -/// Pins a hang that `verify/failover/EndedRelay.cfg` finds (`NoSkipWhileHeld`). `E`'s fetch -/// of the rest of group 0 goes through `B`, which serves it through `A`. The publisher ends -/// the track, then `A` dies: `B`'s fetch fails, and `B`'s front, whose track ended, splices -/// no copy from `P` for it though `B` still reaches `P`, so the read waits out the -/// subscriber's budget. A fix moves the fetch to `P`, so the frame arrives. +/// `E`'s fetch of the rest of group 0 goes through `B`, which serves it through `A`. The +/// publisher ends the track, then `A` dies: `B`'s fetch fails, so `B`'s front, whose track +/// ended, moves it to `P`, which `B` still reaches, and the rest arrives (`NoSkipWhileHeld` +/// in `verify/failover/FailoverChain.tla` with `Mesh`). #[kio::test] -async fn residual_a_fetch_cannot_move_off_a_dying_route_after_the_end() { +async fn a_fetch_moves_off_a_dying_route_after_the_end() { let mut chain = Chain::mesh().await; let mut rx = chain.read_on(); settle().await; chain.end().await; chain.kill_a().await; chain.finish_group0(); - assert!( - kio::clock::timeout(PATIENCE, rx.next()).await.is_err(), - "group 0 continued" - ); + let frame = kio::clock::timeout(PATIENCE, rx.next()).await.expect("group 0 hung"); + assert_eq!(frame.unwrap().unwrap(), Some(payload(0, 1))); +} + +/// [`a_fetch_moves_off_a_dying_route_after_the_end`] with `A` dying only past the linger: +/// the fetch in flight keeps `B`'s finished copy open, so `B`'s front ends the track, and +/// starts its linger, only as `A` dies, and the fetch still moves. +#[kio::test] +async fn a_fetch_held_past_the_linger_moves_off_a_dying_route() { + let mut chain = Chain::mesh().await; + let mut rx = chain.read_on(); + settle().await; + chain.end().await; + kio::clock::sleep(Duration::from_secs(40)).await; + chain.kill_a().await; + chain.finish_group0(); + let frame = kio::clock::timeout(PATIENCE, rx.next()).await.expect("group 0 hung"); + assert_eq!(frame.unwrap().unwrap(), Some(payload(0, 1))); } diff --git a/rs/moq-net/verify/failover/FailoverChain.cfg b/rs/moq-net/verify/failover/FailoverChain.cfg index dff6afd9946..11cc8451601 100644 --- a/rs/moq-net/verify/failover/FailoverChain.cfg +++ b/rs/moq-net/verify/failover/FailoverChain.cfg @@ -1,8 +1,8 @@ \* The client reads two groups of two frames through E, which fails over \* between A and B, each pulling from the origin. One relay may die, and each \* relay may move off a live route once. The client stays, so a deadlock is a -\* hang; Leave.cfg lets it go to check cleanup. EndedRelay.cfg adds B's -\* route through A while its finding is open. +\* hang; Leave.cfg lets it go to check cleanup. Mesh.cfg adds B's +\* route through A. CONSTANTS G = 2 F = 2 diff --git a/rs/moq-net/verify/failover/FailoverChain.tla b/rs/moq-net/verify/failover/FailoverChain.tla index 4d70aec1467..c63a2cbd581 100644 --- a/rs/moq-net/verify/failover/FailoverChain.tla +++ b/rs/moq-net/verify/failover/FailoverChain.tla @@ -34,8 +34,8 @@ (* every hop subscribes before the first group, and a copy's range is *) (* resolved from its first group (SUBSCRIBE_OK), so a relay knows which *) (* groups its copy owes. A fetch reaches the serving route while that *) -(* route lives, after the track ended too: the skill's open findings are *) -(* where the code does not yet. *) +(* route lives, and a relay whose track ended still moves to another route *) +(* for the reads that hold it. *) (***************************************************************************) EXTENDS Integers, FiniteSets @@ -225,7 +225,7 @@ Recover(e, g, s) == /\ front[u].end # "err" /\ s \in Holders(u, e, g) /\ \/ SrcDead(u, cp[e].src[g], g) - \/ cp[e].src[g] # front[u].cur /\ s = front[u].cur /\ front[u].end = "open" + \/ cp[e].src[g] # front[u].cur /\ s = front[u].cur /\ cp' = [cp EXCEPT ![e].src[g] = s] /\ UNCHANGED <> @@ -254,8 +254,9 @@ Ask(u, g) == \* A short group nothing can continue fails (`Recover::poll` gives up): its \* source failed, no other copy holds it, and no fetch can reach the rest, -\* because the front has no route left, ended with its serving copy gone, or -\* the live serving copy is the source that gave up on the group. +\* because the front has no route left, ended with its serving copy gone and +\* no route to move to, or the live serving copy is the source that gave up +\* on the group. Abort(e, g) == LET u == e[2] IN /\ u # "O" @@ -264,7 +265,7 @@ Abort(e, g) == /\ SrcDead(u, cp[e].src[g], g) /\ Holders(u, e, g) = {} /\ \/ front[u].end = "err" - \/ front[u].end = "ok" /\ ~Live(Serving(u)) + \/ front[u].end = "ok" /\ ~Live(Serving(u)) /\ \A w \in Up(u) : ~Routable(w) \/ cp[e].src[g] = front[u].cur /\ Live(Serving(u)) /\ cp' = [cp EXCEPT ![e].bad = @ \cup {g}] /\ UNCHANGED <> @@ -281,22 +282,26 @@ Subscribe(u) == \* The front moves to another route: forced off a dead one, or at will while \* the switches last (`Producer::splice`). A copy it still holds of that route -\* is asked again from where the readers got to; otherwise a new one is. +\* is asked again from where the readers got to; otherwise a new one is. A +\* track that ended moves too, for the group reads and fetches still holding +\* it (`Action::Reroute`): its readers stay on the copy it ended on, so a +\* copy asked for them subscribes nothing, and one still held is left as is. Switch(u, w) == LET usable == Live(Serving(u)) /\ Routable(front[u].cur) e1 == <> + open == front[u].end = "open" IN /\ u \in Relays - /\ front[u].end = "open" + /\ front[u].end # "err" /\ Wanted(u) /\ w \in Up(u) /\ w # front[u].cur /\ Routable(w) /\ usable => front[u].sw < MaxSwitches /\ front' = [front EXCEPT ![u].cur = w, ![u].sw = @ + (IF usable THEN 1 ELSE 0)] /\ cp' = [cp EXCEPT ![e1] = - IF Live(e1) /\ ~cp[e1].fin THEN [@ EXCEPT !.ask = Demand(u)] + IF Live(e1) /\ ~cp[e1].fin /\ open THEN [@ EXCEPT !.ask = Demand(u)] ELSE IF Live(e1) THEN @ - ELSE Fresh(Demand(u))] + ELSE Fresh(IF open THEN Demand(u) ELSE {})] /\ UNCHANGED <> \* The serving copy completed: the front ends the track (`Action::Finish`). diff --git a/rs/moq-net/verify/failover/EndedRelay.cfg b/rs/moq-net/verify/failover/Mesh.cfg similarity index 53% rename from rs/moq-net/verify/failover/EndedRelay.cfg rename to rs/moq-net/verify/failover/Mesh.cfg index 79aaeb21eeb..c12cafc6c68 100644 --- a/rs/moq-net/verify/failover/EndedRelay.cfg +++ b/rs/moq-net/verify/failover/Mesh.cfg @@ -1,8 +1,8 @@ -\* Open finding: NoSkipWhileHeld. A relay whose track ended cannot move a fetch -\* off a route that dies (see the verify-failover skill), pinned by -\* residual_a_fetch_cannot_move_off_a_dying_route_after_the_end. \* FailoverChain.cfg with B also routing through A, so the route can move at -\* the middle hop. +\* the middle hop, after B's track ended too. Too large to exhaust (past 139M +\* distinct states at depth 38 with its queue still growing), so Nightly +\* samples it; an exhaustive run passed every state through depth 37. +\* Simulate: -simulate num=200000 -depth 100 CONSTANTS G = 2 F = 2 diff --git a/rs/moq-net/verify/resume/ResumeReader.tla b/rs/moq-net/verify/resume/ResumeReader.tla index b26ec19746a..b6e33495759 100644 --- a/rs/moq-net/verify/resume/ResumeReader.tla +++ b/rs/moq-net/verify/resume/ResumeReader.tla @@ -391,7 +391,10 @@ Finish(t) == /\ UNCHANGED <> \* The front serves another live route (`Producer::splice`), and every reader -\* follows at once (`serve_from`). Leaving a failed route is not counted. +\* follows at once (`serve_from`). Leaving a failed route is not counted. A +\* copy the front serves after a clean end (`Action::Reroute`) is for group +\* reads and fetches, not this reader, which stays on the copy the track ended +\* on (`Route::finished`), so it is no step here. Serve(t) == /\ front.end = "open" /\ Live(t) /\ t # front.cur