From 0b78266561eecd10d61f7ab357099d85279b74ff Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 17:33:40 -0800 Subject: [PATCH 1/2] chore(quest): claim quest/m1/precut-fixes/failover-models-sync.md Quest-Claim: 2f864b501ccbbd15 From 02ccad10ce4c6ab1c4ce6993e9023339dccdcf22 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 19:12:48 -0800 Subject: [PATCH 2/2] fix(moq-net): a finished track a read holds outlives its linger, and the failover and resume models mirror the code The front's driver forgot a finished track once its linger ran out even while a fetch or group read still held it, since a closed track counted as unread and `abort_unwanted` reports a closed track gone. A held read then could no longer move off a dying route. A finished track whose reads move with the front is now read while anything holds it. FailoverChain.tla models the linger (`Forget`), where a finished track's fetch goes first (`Route::answering`), and that it keeps the copy it ended on; the pre-fix linger broke NoSkipWhileHeld in Mesh. ResumeReader.tla's comments name the current code. Both skills' sync commits move forward. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-failover/SKILL.md | 10 ++- .claude/skills/verify-resume/SKILL.md | 6 +- quest/m1/precut-fixes/failover-models-sync.md | 30 ------- rs/kio/src/consumer.rs | 20 ++++- rs/kio/src/tests.rs | 26 ++++++ rs/kio/src/weak.rs | 8 ++ rs/moq-net/src/model/front.rs | 3 +- rs/moq-net/src/model/origin.rs | 77 ++++++++++++++--- rs/moq-net/src/model/resume.rs | 6 ++ rs/moq-net/src/model/track.rs | 6 ++ rs/moq-net/verify/failover/FailoverChain.tla | 86 +++++++++++++------ rs/moq-net/verify/resume/ResumeReader.tla | 9 +- 12 files changed, 207 insertions(+), 80 deletions(-) delete mode 100644 quest/m1/precut-fixes/failover-models-sync.md diff --git a/.claude/skills/verify-failover/SKILL.md b/.claude/skills/verify-failover/SKILL.md index fa0baf0e6c5..6fed668338e 100644 --- a/.claude/skills/verify-failover/SKILL.md +++ b/.claude/skills/verify-failover/SKILL.md @@ -5,7 +5,7 @@ description: Model-check failover along a chain of relays (`rs/moq-net/verify/fa # Verify failover along a relay chain -`rs/moq-net/verify/failover/FailoverChain.tla` models one track served to a client `C` through relay `E`, which starts on relay `A` and may fail over to relay `B`; both pull from the origin `O`, and with `Mesh` `B` may also route through `A`. Each layer is abstracted to the steps its own model checked, so the state stays small: a route change is one `Switch` (`verify-route`), a switch asks the new copy from where the readers got to and each reader hands a group out once and ends once nothing it replaced may still deliver (`verify-resume`), and a group handed out crosses a hop unless its session dies (`verify-subscribe`, with an hour's budget). What the chain adds is frames: a group whose copy fails or stalls mid-read continues on another copy at the same frame (`Recover`), from a copy its relay holds, from the serving copy, or by fetching the rest from the serving route, which asks its own upstream in turn. +`rs/moq-net/verify/failover/FailoverChain.tla` models one track served to a client `C` through relay `E`, which starts on relay `A` and may fail over to relay `B`; both pull from the origin `O`, and with `Mesh` `B` may also route through `A`. Each layer is abstracted to the steps its own model checked, so the state stays small: a route change is one `Switch` (`verify-route`), a switch asks the new copy from where the readers got to and each reader hands a group out once and ends once nothing it replaced may still deliver (`verify-resume`), and a group handed out crosses a hop unless its session dies (`verify-subscribe`, with an hour's budget). What the chain adds is frames: a group whose copy fails or stalls mid-read continues on another copy at the same frame (`Recover`), from a copy its relay holds, from the serving copy, or by fetching the rest from the serving route, which asks its own upstream in turn. A relay whose track ended keeps moving the reads that still hold it (`Switch` after `FrontFinish`) and forgets the track only once nothing reads it (`Forget`, the linger). ## Rules @@ -27,7 +27,7 @@ quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq- quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Mesh.cfg -simulate num=200000 -depth 100 ``` -The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.2M states. Each takes under a minute and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes. +The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.6M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.8M states. Each takes about a minute and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes. ## Reading a failure @@ -35,7 +35,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `61f89568394638553a519d7d8cfc0019385dd3a0`, the base of [moq-dev/moq#9048](https://github.com/moq-dev/moq/pull/9048), which mirrored the code since 88572c583e and kept a finished track a read holds past its linger. Run `git log 61f8956839.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | @@ -46,7 +46,9 @@ Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e | `rs/moq-net/src/model/resume.rs` `Recover::poll_fetch`, `Fetching`; `rs/moq-net/src/model/track.rs` `fetch_group` | `Ask` | | `rs/moq-net/src/model/front.rs` `track_ended`, `Action::Finish`; `rs/moq-net/src/model/origin.rs` `TrackIo::end` | `FrontFinish`, `Whole` | | `rs/moq-net/src/model/front.rs` failover and its unroutable end, `Action::Reroute` | `Switch`, `FrontFail` | +| `rs/moq-net/src/model/resume.rs` `Route::answering`, `Phase::Finished` | `Answering`, `LetGo` | +| `rs/moq-net/src/model/front.rs` `Action::Forget`; `rs/moq-net/src/model/track.rs` `TrackWeak::is_held`, `TrackWeak::poll_held_changed` | `Forget`, `Reads` | ## Not covered -Routing is abstracted to any live upstream winning, so feasibility, ROUTE_END, and announce order are `verify-route`'s; how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. A pass covers two groups of two frames, one death, one switch per relay, and one client. +Routing is abstracted to any live upstream winning, so feasibility, ROUTE_END, and announce order are `verify-route`'s; how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). Time is abstracted, so the linger may run out at any point once nothing reads the track. The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. A pass covers two groups of two frames, one death, one switch per relay, and one client. diff --git a/.claude/skills/verify-resume/SKILL.md b/.claude/skills/verify-resume/SKILL.md index aaed284d979..ebe336bc4f0 100644 --- a/.claude/skills/verify-resume/SKILL.md +++ b/.claude/skills/verify-resume/SKILL.md @@ -40,7 +40,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path as a ## Coverage -Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `74e05ddc7039cc32f8968d9425e8224e82d4d886`. Run `git log 74e05ddc70.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | @@ -49,8 +49,8 @@ Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07 | `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe` | `Poll`, `Scan`, `Peek`, `Deliverable`, `Take` | | `rs/moq-net/src/model/resume.rs` `Reader::bound_at_end`, `let_go`, `Copy::retains` | `BoundAtEnd`, `LetGo`, `Retains` | | `rs/moq-net/src/model/resume.rs` `Reader::poll_end` | `PollEnd` | -| `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, the front's end | `Serve`, `FrontFinish`, `FrontFail` | -| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` | +| `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, `Producer::end`, `Phase::Finished` | `Serve`, `FrontFinish`, `FrontFail` | +| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `Consumer::latest_key`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` | ## Not covered diff --git a/quest/m1/precut-fixes/failover-models-sync.md b/quest/m1/precut-fixes/failover-models-sync.md deleted file mode 100644 index cd2dd929996..00000000000 --- a/quest/m1/precut-fixes/failover-models-sync.md +++ /dev/null @@ -1,30 +0,0 @@ -# [M] The failover and resume models mirror the code, and an ended track a reader holds keeps rerouting - -## Goal - -`FailoverChain.tla` and `ResumeReader.tla` describe the code that ships, their -skills' sync commits name a commit at or after the last covered change, and -an ended track that a reader still holds keeps moving a slow read to another -route after its 30 s linger. - -## Plan - -Found by the 2026-11-17 pre-cut correctness audit, from source. - -- `.claude/skills/verify-failover/SKILL.md` still names 88572c583e and - `verify-resume/SKILL.md` names 103052ed07, though - [moq-dev/moq#8896](https://github.com/moq-dev/moq/pull/8896) edited both - models and #8859, #8958, #8867, and #8638 changed covered code since. - `ResumeReader.tla` cites `Route::finished`, which no longer exists. Run both - skills on `main`, mirror what changed, and move the sync commits. -- The linger and `Action::Conclude` are not modelled, and `Mesh.cfg`, the only - config that reaches the reroute, is sampled rather than exhaustive. Model the - linger. -- `rs/moq-net/src/model/front.rs` and the `Forget` arm in `origin.rs`: - `abort_unwanted` returns true for a closed track a reader still holds, so - after the linger the track's I/O drops and a slow read of the last group - fails when its upstream route dies. The `Reroute` path checks `is_held()`; - `Forget` does not. Recommended: don't forget an ended track while it is - held. Reproduce with the model before fixing. From #8896. - -Public API: none. Wire: none. diff --git a/rs/kio/src/consumer.rs b/rs/kio/src/consumer.rs index 6c45c1d1de2..11d5cc06746 100644 --- a/rs/kio/src/consumer.rs +++ b/rs/kio/src/consumer.rs @@ -77,11 +77,29 @@ pub(crate) fn poll_wanted_changed( counts: &Counts, waiter: &Waiter, last: bool, +) -> Poll> { + poll_consumers(state, counts, waiter, last, true) +} + +/// [`poll_wanted_changed`], counting consumers past closure too: those still draining +/// the final value. +#[cfg_attr(feature = "sim", track_caller)] +pub(crate) fn poll_held_changed(state: &Channel, counts: &Counts, waiter: &Waiter, last: bool) -> Poll { + poll_consumers(state, counts, waiter, last, false).map(|res| res.unwrap_or(last)) +} + +#[cfg_attr(feature = "sim", track_caller)] +fn poll_consumers( + state: &Channel, + counts: &Counts, + waiter: &Waiter, + last: bool, + until_closed: bool, ) -> Poll> { // Closure is checked first, so a closed channel reports `Closed` from every handle // rather than a consumer count that only dropped because it's dead. let check = || { - if state.is_closed() { + if until_closed && state.is_closed() { return Some(Err(Closed)); } let wanted = counts.consumers.load(Ordering::Relaxed) > 0; diff --git a/rs/kio/src/tests.rs b/rs/kio/src/tests.rs index 3083ef17c70..cbedabec7b2 100644 --- a/rs/kio/src/tests.rs +++ b/rs/kio/src/tests.rs @@ -254,3 +254,29 @@ fn consumer_creation_with_existing_demand_does_not_relock() { .expect("consumer creation reacquired the held lock"); worker.join().unwrap(); } + +/// Unlike `wanted_changed`, a held change counts the consumers still draining a closed +/// channel, and wakes when the last of them leaves. +#[test] +fn held_changed_outlives_the_close() { + let producer = Producer::new(0u32); + let weak = producer.weak(); + let consumer = producer.consume(); + + let (waker, w) = counting(); + let waiter = crate::Waiter::new(w); + producer.close().ok().expect("open"); + assert!( + weak.poll_wanted_changed(&waiter, true).is_ready(), + "wanted_changed stops at the close" + ); + assert!( + weak.poll_held_changed(&waiter, true).is_pending(), + "a draining consumer holds it" + ); + + drop(consumer); + assert!(waker.count() >= 1, "the last consumer leaving did not wake it"); + assert_eq!(weak.poll_held_changed(&waiter, true), Poll::Ready(false)); + assert_eq!(weak.poll_held_changed(&waiter, false), Poll::Pending); +} diff --git a/rs/kio/src/weak.rs b/rs/kio/src/weak.rs index 6bda3eb0084..c7a50b1b86f 100644 --- a/rs/kio/src/weak.rs +++ b/rs/kio/src/weak.rs @@ -170,6 +170,14 @@ impl ProducerWeak { crate::consumer::poll_wanted_changed(&self.state, &self.counts, waiter, last) } + /// Poll for [`Self::wanted`] to differ from `last`, whether or not the channel closed: + /// a closed channel's consumers may still be draining its final value. `Ready` with the + /// new value once it does, else `Pending`. + #[cfg_attr(feature = "sim", track_caller)] + pub fn poll_held_changed(&self, waiter: &Waiter, last: bool) -> Poll { + crate::consumer::poll_held_changed(&self.state, &self.counts, waiter, last) + } + /// Returns `true` if both handles share the same underlying state. pub fn same_channel(&self, other: &Self) -> bool { Arc::ptr_eq(&self.state, &other.state) diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs index ac11d384dac..572f2d1b436 100644 --- a/rs/moq-net/src/model/front.rs +++ b/rs/moq-net/src/model/front.rs @@ -97,7 +97,8 @@ pub(super) enum Event { }, /// A reader arrived on the track. Wanted { track: Arc }, - /// The last reader left the track. + /// The last reader left the track. A finished track is read until nothing holds it, + /// such as a fetch still reading a group, which keeps moving off a dying route. Unwanted { track: Arc, now: Instant }, /// The armed deadline passed. Deadline { now: Instant }, diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 94174062b1a..5bfec3ae571 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -4288,6 +4288,16 @@ impl Logical { } impl TrackIo { + /// Whether anyone reads the track: a reader of the open track, or anything still holding + /// a finished one whose reads move with the front, such as a fetch of a group below its + /// end. A track that failed or concluded owes its holders nothing. + fn read(&self) -> bool { + match self.routes.moves() { + true => self.weak.is_held(), + false => self.weak.is_wanted(), + } + } + /// Serve the track from `asked`'s copy, accepting it with that copy's info first. fn splice(&mut self, asked: Asked) { self.ready.wake_by_ref(); @@ -4872,13 +4882,14 @@ impl FrontServe { let delivered = copy.latest_key().is_some(); return Some(FrontStep::Ended(name.clone(), *source, result, delivered)); } - // Watch the demand edge in whichever direction is unmet. Only `Pending` - // parks, so an answer steps and the handler looks again, polling anew when - // nothing changed. A closed track is unread: it owes one `Unwanted` if - // recorded read, and is no edge after. - let edge = match io.wanted { - true => io.weak.poll_unwanted(waiter).is_ready(), - false => matches!(io.weak.poll_wanted(waiter), Poll::Ready(Ok(()))), + // Watch the demand edge in whichever direction is unmet (`TrackIo::read`). + // Only `Pending` parks, so an answer steps and the handler looks again, + // polling anew when nothing changed. Any other closed track is unread: it owes + // one `Unwanted` if recorded read, and is no edge after. + let edge = match (io.routes.moves(), io.wanted) { + (true, wanted) => io.weak.poll_held_changed(waiter, wanted).is_ready(), + (false, true) => io.weak.poll_unwanted(waiter).is_ready(), + (false, false) => matches!(io.weak.poll_wanted(waiter), Poll::Ready(Ok(()))), }; if edge { return Some(FrontStep::Demand(name.clone())); @@ -5177,11 +5188,12 @@ impl kio::Task for FrontServe { } Action::Forget { track: name } => { // A reader that looked the track up since the machine decided keeps - // it. Feed its `Wanted` edge here: the demand poll only sees the - // current level, so a reader gone before the next poll would - // otherwise leave the track unread with no linger armed. + // it, as does one still holding it once it finished. Feed its `Wanted` + // edge here: the demand poll only sees the current level, so a reader + // gone before the next poll would otherwise leave the track unread + // with no linger armed. if let Some(io) = self.tracks.get_mut(&name) - && !io.weak.abort_unwanted(Error::Cancel) + && (!io.weak.abort_unwanted(Error::Cancel) || io.read()) { if !io.wanted { io.wanted = true; @@ -5456,7 +5468,7 @@ impl kio::Task for FrontServe { let Some(io) = self.tracks.get_mut(&name) else { continue }; // Read again: the edge may have flipped back since it fired, and the next // wait parks on it. - if io.weak.is_wanted() == io.wanted { + if io.read() == io.wanted { continue; } io.wanted = !io.wanted; @@ -15782,6 +15794,47 @@ mod tests { assert!(res.is_err(), "{res:?}"); } + /// A finished track that something still reads outlives its linger, so a replacement + /// route is still asked for it, for a read that has to move off a dying route. Once + /// nothing holds it, it lingers and is forgotten as before. + #[kio::test] + #[cfg(feature = "async")] + async fn a_held_finished_track_outlives_its_linger() { + let (producer, _server, _upstream, old, mut subscription) = remote_front().await; + drop(old.finish().unwrap()); + assert!(matches!(next_group(&mut subscription).await, Ok(None))); + + // Held past the linger: a replacement is asked for the track. + kio::clock::sleep(Duration::from_secs(60)).await; + let newcomer = producer + .route("room/alice", RouteOptions::hidden().with_instance(instance(), 0)) + .unwrap(); + let mut handler = newcomer.broadcast().requests(); + newcomer.announce().unwrap(); + let request = kio::clock::timeout(Duration::from_secs(1), handler.requested_track()) + .await + .expect("the front rerouted the held track") + .expect("request"); + assert_eq!(request.name(), "video"); + drop(request); + + // Unheld, it is forgotten after the linger: a later route is asked nothing. + drop(subscription); + kio::clock::sleep(Duration::from_secs(60)).await; + drop(newcomer); + let third = producer + .route("room/alice", RouteOptions::hidden().with_instance(instance(), 0)) + .unwrap(); + let mut handler = third.broadcast().requests(); + third.announce().unwrap(); + assert!( + kio::clock::timeout(Duration::from_secs(1), handler.requested_track()) + .await + .is_err(), + "an unheld finished track was rerouted after its linger" + ); + } + /// The newcomer answers "video" with matching metadata but aborts before delivering a /// group, all before the front polls the answer: that is a refusal, so the incumbent /// keeps serving rather than the track ending on the newcomer's error. diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 2dbdb619e1d..fd5431e97cd 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -276,6 +276,12 @@ impl Producer { self.state.read().phase.end().is_some() } + /// Whether a front may still replace the copy: the track is open, or finished with + /// reads still to move off a dying route. + pub(super) fn moves(&self) -> bool { + self.state.read().phase.moves() + } + /// The track ended: cleanly with `Ok`, or failed for good. A clean end still moves to /// each copy the front serves from until it concludes. pub(super) fn end(&self, result: Result<()>) { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 4a8f327b4cb..2a78ba1bccd 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -5786,6 +5786,12 @@ impl TrackWeak { self.state.poll_wanted_changed(waiter, false).map_ok(drop) } + /// `Ready` once [`Self::is_held`] differs from `last`, closed or not, and otherwise park + /// `waiter` for the change. + pub(super) fn poll_held_changed(&self, waiter: &kio::Waiter, last: bool) -> Poll { + self.state.poll_held_changed(waiter, last) + } + /// `Ready(Ok)` while nobody reads the track, `Ready(Err)` once it closed, and /// otherwise park `waiter` for the last reader leaving. Only `Pending` registers, /// as with [`Self::poll_wanted`]. diff --git a/rs/moq-net/verify/failover/FailoverChain.tla b/rs/moq-net/verify/failover/FailoverChain.tla index c63a2cbd581..fffeb1e9a95 100644 --- a/rs/moq-net/verify/failover/FailoverChain.tla +++ b/rs/moq-net/verify/failover/FailoverChain.tla @@ -35,7 +35,7 @@ (* resolved from its first group (SUBSCRIBE_OK), so a relay knows which *) (* groups its copy owes. A fetch reaches the serving route while that *) (* route lives, and a relay whose track ended still moves to another route *) -(* for the reads that hold it. *) +(* for the reads that hold it, forgetting the track only once none does. *) (***************************************************************************) EXTENDS Integers, FiniteSets @@ -72,7 +72,8 @@ VARIABLES alive, \* which relays are up deaths, \* relays that died cp, \* cp[e]: the copy at e[1] of the track e[2] serves, and e[2]'s reader for it - front, \* front[r]: the route r serves from, how its track ended, switches made + front, \* front[r]: the route r serves from, how its track ended, the route it + \* ended on, switches made left \* the client unsubscribed vars == <> @@ -126,11 +127,24 @@ ResumeAsk(e) == from == IF got = {} THEN 0 ELSE IF gaps # {} THEN Min(gaps) ELSE newest IN {g \in cp[e].ask : g >= from} -\* What a copy at u asks: the union of its readers' demand (`Producer::subscription`). -Demand(u) == UNION {ResumeAsk(e) : e \in {x \in Downs(u) : Live(x)}} +\* A downstream copy still reads the track at its upstream: its subscription is open, or +\* a group it was handed or fetched is not over (`TrackWeak::is_held`). +Reads(e) == + /\ Live(e) + /\ \/ ~cp[e].fin + \/ \E g \in Groups : Short(e, g) \/ (g \in cp[e].fetch /\ cp[e].fr[g] = -1) + +\* What a copy at u asks: the union of its open readers' demand (`Producer::subscription`). +Demand(u) == UNION {ResumeAsk(e) : e \in {x \in Downs(u) : Live(x) /\ ~cp[x].fin}} Wanted(u) == \E e \in Downs(u) : Live(e) Serving(u) == <> +\* Where a fetch at u goes (`Route::answering`): once the track finished, the copy it +\* ended on until that copy fails, since a copy served after the end caches nothing. +Answering(u) == + IF front[u].end = "ok" /\ Live(<>) + THEN <> + ELSE Serving(u) Routable(w) == IF w = "O" THEN TRUE ELSE alive[w] /\ front[w].end # "err" \* A complete copy (`TrackState::is_complete` once every group below the end @@ -179,7 +193,8 @@ Init == /\ cp = [e \in Edges |-> IF e \in {CE, <<"E", "A">>, <<"A", "O">>} THEN Fresh(Groups) ELSE Gone] /\ front = [r \in Relays |-> - [cur |-> IF r = "E" THEN "A" ELSE "O", end |-> "open", sw |-> 0]] + [cur |-> IF r = "E" THEN "A" ELSE "O", end |-> "open", last |-> None, + sw |-> 0]] /\ left = FALSE \* The origin writes a group whole, in order. Frames split only at the hops. @@ -229,27 +244,33 @@ Recover(e, g, s) == /\ cp' = [cp EXCEPT ![e].src[g] = s] /\ UNCHANGED <> -\* A relay asks its serving route for a group a reader needs that no copy it -\* holds has or may still land: a backfill for a widened or fetched range, or a -\* fetch of the rest of a short group (`Recover::poll_fetch`). +\* A relay asks upstream for a group a reader needs that no copy it holds has or +\* may still land: a backfill for a widened or fetched range, sent where its reads +\* go first (`Fetching`), or a fetch of the rest of a short group, sent to the +\* serving route (`Recover::poll_fetch`). Ask(u, g) == LET w == front[u].cur - e0 == Serving(u) + \* A missing group goes where the relay's reads go first; the rest of a short + \* one goes to the serving route. + missing == \E e \in Downs(u) : + /\ Lacks(e, g) + /\ \A s \in Srcs(u) : SrcFr(u, s, g) < 0 + /\ ~MayLand(u, g) + short == \E e \in Downs(u) : + /\ Short(e, g) + /\ Exhausted(e, g) + /\ cp[e].src[g] # w + /\ Holders(u, e, g) = {} IN /\ u \in Relays /\ front[u].end # "err" - /\ Live(e0) - /\ g \notin cp[e0].ask \cup cp[e0].fetch - /\ cp[e0].fr[g] = -1 - /\ \E e \in Downs(u) : - \/ /\ Lacks(e, g) - /\ \A s \in Srcs(u) : SrcFr(u, s, g) < 0 - /\ ~MayLand(u, g) - \/ /\ Short(e, g) - /\ Exhausted(e, g) - /\ cp[e].src[g] # w - /\ Holders(u, e, g) = {} - /\ cp' = [cp EXCEPT ![e0].fetch = @ \cup {g}] + /\ \E e0 \in {Answering(u), Serving(u)} : + /\ \/ e0 = Answering(u) /\ missing + \/ e0 = Serving(u) /\ short + /\ Live(e0) + /\ g \notin cp[e0].ask \cup cp[e0].fetch + /\ cp[e0].fr[g] = -1 + /\ cp' = [cp EXCEPT ![e0].fetch = @ \cup {g}] /\ UNCHANGED <> \* A short group nothing can continue fails (`Recover::poll` gives up): its @@ -274,7 +295,7 @@ Abort(e, g) == Subscribe(u) == /\ u \in Relays /\ front[u].end = "open" - /\ Wanted(u) + /\ \E e \in Downs(u) : Reads(e) /\ Routable(front[u].cur) /\ ~Held(Serving(u)) /\ cp' = [cp EXCEPT ![Serving(u)] = Fresh(Demand(u))] @@ -310,7 +331,7 @@ FrontFinish(u) == /\ front[u].end = "open" /\ Live(Serving(u)) /\ Whole(Serving(u)) - /\ front' = [front EXCEPT ![u].end = "ok"] + /\ front' = [front EXCEPT ![u].end = "ok", ![u].last = front[u].cur] /\ UNCHANGED <> \* No route is left: the front fails the track, and every reader with it. @@ -344,6 +365,8 @@ LetGo(u, s) == /\ s \in Up(u) /\ Held(e1) /\ s # front[u].cur \/ front[u].end = "err" + \* A finished track keeps the copy it ended on (`Phase::Finished`). + /\ ~(front[u].end = "ok" /\ s = front[u].last) /\ \A e \in Downs(u), g \in Groups : /\ Short(e, g) => cp[e].src[g] # s /\ cp[e1].fr[g] < 0 /\ ~MayLand(u, g) /\ Lacks(e, g) => cp[e1].fr[g] < 0 /\ ~(Live(e1) /\ g \in cp[e1].ask) @@ -358,7 +381,20 @@ Park(u) == /\ ~Wanted(u) /\ \E s \in Up(u) : Held(<>) /\ cp' = [e \in Edges |-> IF e[1] = u THEN Gone ELSE cp[e]] - /\ front' = [front EXCEPT ![u].end = IF @ = "ok" THEN "open" ELSE @] + /\ front' = [front EXCEPT ![u].end = IF @ = "ok" THEN "open" ELSE @, + ![u].last = None] + /\ UNCHANGED <> + +\* The linger runs out on a relay whose track ended and that nothing reads any +\* more (`Action::Forget`): it is forgotten, so the next reader gets a fresh front. +\* A read still holding the track keeps it, so the read still moves off a dying +\* route (`TrackWeak::is_held`). +Forget(u) == + /\ u \in Relays + /\ front[u].end = "ok" + /\ ~\E e \in Downs(u) : Reads(e) + /\ front' = [front EXCEPT ![u].end = "open", ![u].last = None] + /\ cp' = [e \in Edges |-> IF e[1] = u THEN Gone ELSE cp[e]] /\ UNCHANGED <> \* A relay dies: copies downstream of it fail, and its own vanish. @@ -400,7 +436,7 @@ Next == \/ \E s \in Srcs(e[2]) : HandOut(e, g, s) \/ Recover(e, g, s) \/ Frame(e, g) \/ Abort(e, g) \/ \E u \in Relays : - \/ Subscribe(u) \/ FrontFinish(u) \/ FrontFail(u) \/ Park(u) + \/ Subscribe(u) \/ FrontFinish(u) \/ FrontFail(u) \/ Park(u) \/ Forget(u) \/ \E w \in Up(u) : Switch(u, w) \/ LetGo(u, w) \/ \E g \in Groups : Ask(u, g) \/ \E e \in Edges : EndOk(e) diff --git a/rs/moq-net/verify/resume/ResumeReader.tla b/rs/moq-net/verify/resume/ResumeReader.tla index b6e33495759..fd9d8596c63 100644 --- a/rs/moq-net/verify/resume/ResumeReader.tla +++ b/rs/moq-net/verify/resume/ResumeReader.tla @@ -52,7 +52,7 @@ vars == <> Landed(t) == {trk[t].arr[i] : i \in 1 .. Len(trk[t].arr)} Live(t) == trk[t].made /\ ~trk[t].failed -\* `max_key`: the newest group the track holds. +\* `track::Consumer::latest_key` (`TrackState::max_key`): the newest group the track holds. Latest(t) == IF Landed(t) = {} THEN None ELSE Max(Landed(t)) \* `TrackState::is_complete`: the end is declared and the tail holds every group @@ -85,7 +85,8 @@ NewCopy(id, t, gen, floor, rep, until) == until |-> until, done |-> "no", since |-> rep, idx |-> 1, took |-> {}, owed |-> {}] -\* The wire asks for [floor, until] instead (`Copy::update`/`send`): the cursor +\* The wire asks for [floor, until] instead (`Copy::bound`, or `Copy::send` on a +\* reuse or a widening; neither reopens a drained copy): the cursor \* passes what its window leaves out, holding nothing, and a widening owes it \* the cached groups the window newly covers. Window(c, floor, until) == @@ -393,8 +394,8 @@ Finish(t) == \* The front serves another live route (`Producer::splice`), and every reader \* follows at once (`serve_from`). Leaving a failed route is not counted. A \* copy the front serves after a clean end (`Action::Reroute`) is for group -\* reads and fetches, not this reader, which stays on the copy the track ended -\* on (`Route::finished`), so it is no step here. +\* reads and fetches, not this reader: `Reader::sync` stays on the generation +\* and copy the track ended on (`Phase::Finished`), so it is no step here. Serve(t) == /\ front.end = "open" /\ Live(t) /\ t # front.cur