Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .claude/skills/verify-failover/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Model-check failover along a chain of relays (`rs/moq-net/verify/fa

# Verify failover along a relay chain

`rs/moq-net/verify/failover/FailoverChain.tla` models one track served to a client `C` through relay `E`, which starts on relay `A` and may fail over to relay `B`; both pull from the origin `O`, and with `Mesh` `B` may also route through `A`. Each layer is abstracted to the steps its own model checked, so the state stays small: a route change is one `Switch`, a switch asks the new copy from where the readers got to and each reader hands a group out once and ends once nothing it replaced may still deliver (`verify-resume`), and a group handed out crosses a hop unless its session dies (`verify-subscribe`, with an hour's budget). What the chain adds is frames: a group whose copy fails or stalls mid-read continues on another copy at the same frame (`Recover`), from a copy its relay holds, from the serving copy, or by fetching the rest from the serving route, which asks its own upstream in turn.
`rs/moq-net/verify/failover/FailoverChain.tla` models one track served to a client `C` through relay `E`, which starts on relay `A` and may fail over to relay `B`; both pull from the origin `O`, and with `Mesh` `B` may also route through `A`. Each layer is abstracted to the steps its own model checked, so the state stays small: a route change is one `Switch`, a switch asks the new copy from where the readers got to and each reader hands a group out once and ends once nothing it replaced may still deliver (`verify-resume`), and a group handed out crosses a hop unless its session dies (`verify-subscribe`, with an hour's budget). What the chain adds is frames: a group whose copy fails or stalls mid-read continues on another copy at the same frame (`Recover`), from a copy its relay holds, from the serving copy, or by fetching the rest from the serving route, which asks its own upstream in turn. A relay whose track ended keeps moving the reads that still hold it (`Switch` after `FrontFinish`) and forgets the track only once nothing reads it (`Forget`, the linger).

## Rules

Expand All @@ -27,15 +27,15 @@ quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Mesh.cfg -simulate num=200000 -depth 100
```

The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.2M states. Each takes under a minute and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes.
The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.6M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.8M states. Each takes about a minute and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop, after `B`'s track ended too (`Action::Reroute`). It is too large to exhaust (past 139M distinct states at depth 38 after an hour, its queue still growing), so its `Simulate:` line samples 800k random behaviors in about 7 minutes, and Nightly samples new ones each run. An exhaustive run passed every state through depth 37. Before the reroute, a sample that size missed the depth-31 hang that BFS found after about 48M states, so a passing sample is weak evidence: check a change at the middle hop by running it without the `-simulate` arguments for as long as it takes.

## Reading a failure

Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the real stack in `rs/moq-net/tests/integration/failover_chain.rs`: relays are `origin::Producer`s linked over the memory transport (`link`, `priced_link`, a cost to make a route win), a relay or hop dies by aborting its sessions, a group read past a failed copy is what makes `Recover` fetch, and time is simulated. Pin a path the code follows as a `residual_*` test with its own config so the others still pass.

## Coverage

Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `61f89568394638553a519d7d8cfc0019385dd3a0`, plus [moq-dev/moq#9048](https://github.com/moq-dev/moq/pull/9048), which mirrored the code since 88572c583e and kept a finished track a read holds past its linger. Run `git log 61f8956839.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
Expand All @@ -46,7 +46,9 @@ Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e
| `rs/moq-net/src/model/resume.rs` `Recover::poll_fetch`, `Fetching`; `rs/moq-net/src/model/track.rs` `fetch_group` | `Ask` |
| `rs/moq-net/src/model/front.rs` `track_ended`, `Action::Finish`; `rs/moq-net/src/model/origin.rs` `TrackIo::end` | `FrontFinish`, `Whole` |
| `rs/moq-net/src/model/front.rs` failover and its unroutable end, `Action::Reroute` | `Switch`, `FrontFail` |
| `rs/moq-net/src/model/resume.rs` `Route::answering`, `Phase::Finished` | `Answering`, `LetGo` |
| `rs/moq-net/src/model/front.rs` `Action::Forget`; `rs/moq-net/src/model/track.rs` `TrackWeak::is_held`, `TrackWeak::poll_held_changed` | `Forget`, `Reads` |

## Not covered

Routing is abstracted to any live upstream winning, so which route wins and announce order are the router's own tests' (`rs/moq-net/src/route/tests`); how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. A pass covers two groups of two frames, one death, one switch per relay, and one client.
Routing is abstracted to any live upstream winning, so which route wins and announce order are the router's own tests' (`rs/moq-net/src/route/tests`); how a reader picks among its copies, bounds replaced copies, and orders groups is `verify-resume`'s; drops, skips, and re-offers on a hop are `verify-subscribe`'s. The client's track fails only through `FrontFail`, so whether one dead relay can end playback is that step's guard, not a rule. Every hop has an hour's budget, so nothing goes stale or expires. No source refuses the track, so a finished track's reroute never concludes (`Action::Conclude`). Time is abstracted, so the linger may run out at any point once nothing reads the track. The origin writes each group whole and never fails, `E` never dies, a dead relay never returns, and a relay's cache is dropped when it parks. A fetched group holds its head from frame 0 (`lend_head`), and compressed groups are left out. A pass covers two groups of two frames, one death, one switch per relay, and one client.
6 changes: 3 additions & 3 deletions .claude/skills/verify-resume/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path as a

## Coverage

Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `74e05ddc7039cc32f8968d9425e8224e82d4d886`. Run `git log 74e05ddc70.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
Expand All @@ -49,8 +49,8 @@ Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07
| `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe` | `Poll`, `Scan`, `Peek`, `Deliverable`, `Take` |
| `rs/moq-net/src/model/resume.rs` `Reader::bound_at_end`, `let_go`, `Copy::retains` | `BoundAtEnd`, `LetGo`, `Retains` |
| `rs/moq-net/src/model/resume.rs` `Reader::poll_end` | `PollEnd` |
| `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, the front's end | `Serve`, `FrontFinish`, `FrontFail` |
| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` |
| `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, `Producer::end`, `Phase::Finished` | `Serve`, `FrontFinish`, `FrontFail` |
| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `Consumer::latest_key`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` |

## Not covered

Expand Down
30 changes: 0 additions & 30 deletions quest/m1/precut-fixes/failover-models-sync.md

This file was deleted.

20 changes: 19 additions & 1 deletion rs/kio/src/consumer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,29 @@ pub(crate) fn poll_wanted_changed<T>(
counts: &Counts,
waiter: &Waiter,
last: bool,
) -> Poll<Result<bool, Closed>> {
poll_consumers(state, counts, waiter, last, true)
}

/// [`poll_wanted_changed`], counting consumers past closure too: those still draining
/// the final value.
#[cfg_attr(feature = "sim", track_caller)]
pub(crate) fn poll_held_changed<T>(state: &Channel<T>, counts: &Counts, waiter: &Waiter, last: bool) -> Poll<bool> {
poll_consumers(state, counts, waiter, last, false).map(|res| res.unwrap_or(last))
}

#[cfg_attr(feature = "sim", track_caller)]
fn poll_consumers<T>(
state: &Channel<T>,
counts: &Counts,
waiter: &Waiter,
last: bool,
until_closed: bool,
) -> Poll<Result<bool, Closed>> {
// Closure is checked first, so a closed channel reports `Closed` from every handle
// rather than a consumer count that only dropped because it's dead.
let check = || {
if state.is_closed() {
if until_closed && state.is_closed() {
return Some(Err(Closed));
}
let wanted = counts.consumers.load(Ordering::Relaxed) > 0;
Expand Down
26 changes: 26 additions & 0 deletions rs/kio/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -254,3 +254,29 @@ fn consumer_creation_with_existing_demand_does_not_relock() {
.expect("consumer creation reacquired the held lock");
worker.join().unwrap();
}

/// Unlike `wanted_changed`, a held change counts the consumers still draining a closed
/// channel, and wakes when the last of them leaves.
#[test]
fn held_changed_outlives_the_close() {
let producer = Producer::new(0u32);
let weak = producer.weak();
let consumer = producer.consume();

let (waker, w) = counting();
let waiter = crate::Waiter::new(w);
producer.close().ok().expect("open");
assert!(
weak.poll_wanted_changed(&waiter, true).is_ready(),
"wanted_changed stops at the close"
);
assert!(
weak.poll_held_changed(&waiter, true).is_pending(),
"a draining consumer holds it"
);

drop(consumer);
assert!(waker.count() >= 1, "the last consumer leaving did not wake it");
assert_eq!(weak.poll_held_changed(&waiter, true), Poll::Ready(false));
assert_eq!(weak.poll_held_changed(&waiter, false), Poll::Pending);
}
8 changes: 8 additions & 0 deletions rs/kio/src/weak.rs
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,14 @@ impl<T> ProducerWeak<T> {
crate::consumer::poll_wanted_changed(&self.state, &self.counts, waiter, last)
}

/// Poll for [`Self::wanted`] to differ from `last`, whether or not the channel closed:
/// a closed channel's consumers may still be draining its final value. `Ready` with the
/// new value once it does, else `Pending`.
#[cfg_attr(feature = "sim", track_caller)]
pub fn poll_held_changed(&self, waiter: &Waiter, last: bool) -> Poll<bool> {
crate::consumer::poll_held_changed(&self.state, &self.counts, waiter, last)
}

/// Returns `true` if both handles share the same underlying state.
pub fn same_channel(&self, other: &Self) -> bool {
Arc::ptr_eq(&self.state, &other.state)
Expand Down
3 changes: 2 additions & 1 deletion rs/moq-net/src/model/front.rs
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,8 @@ pub(super) enum Event {
},
/// A reader arrived on the track.
Wanted { track: Arc<str> },
/// The last reader left the track.
/// The last reader left the track. A finished track is read until nothing holds it,
/// such as a fetch still reading a group, which keeps moving off a dying route.
Unwanted { track: Arc<str>, now: Instant },
/// The armed deadline passed.
Deadline { now: Instant },
Expand Down
77 changes: 65 additions & 12 deletions rs/moq-net/src/model/origin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4288,6 +4288,16 @@ impl Logical {
}

impl TrackIo {
/// Whether anyone reads the track: a reader of the open track, or anything still holding
/// a finished one whose reads move with the front, such as a fetch of a group below its
/// end. A track that failed or concluded owes its holders nothing.
fn read(&self) -> bool {
match self.routes.moves() {
true => self.weak.is_held(),
false => self.weak.is_wanted(),
}
}

/// Serve the track from `asked`'s copy, accepting it with that copy's info first.
fn splice(&mut self, asked: Asked) {
self.ready.wake_by_ref();
Expand Down Expand Up @@ -4873,13 +4883,14 @@ impl FrontServe {
let delivered = copy.latest_key().is_some();
return Some(FrontStep::Ended(name.clone(), *source, result, delivered));
}
// Watch the demand edge in whichever direction is unmet. Only `Pending`
// parks, so an answer steps and the handler looks again, polling anew when
// nothing changed. A closed track is unread: it owes one `Unwanted` if
// recorded read, and is no edge after.
let edge = match io.wanted {
true => io.weak.poll_unwanted(waiter).is_ready(),
false => matches!(io.weak.poll_wanted(waiter), Poll::Ready(Ok(()))),
// Watch the demand edge in whichever direction is unmet (`TrackIo::read`).
// Only `Pending` parks, so an answer steps and the handler looks again,
// polling anew when nothing changed. Any other closed track is unread: it owes
// one `Unwanted` if recorded read, and is no edge after.
let edge = match (io.routes.moves(), io.wanted) {
(true, wanted) => io.weak.poll_held_changed(waiter, wanted).is_ready(),
(false, true) => io.weak.poll_unwanted(waiter).is_ready(),
(false, false) => matches!(io.weak.poll_wanted(waiter), Poll::Ready(Ok(()))),
};
if edge {
return Some(FrontStep::Demand(name.clone()));
Expand Down Expand Up @@ -5178,11 +5189,12 @@ impl kio::Task for FrontServe {
}
Action::Forget { track: name } => {
// A reader that looked the track up since the machine decided keeps
// it. Feed its `Wanted` edge here: the demand poll only sees the
// current level, so a reader gone before the next poll would
// otherwise leave the track unread with no linger armed.
// it, as does one still holding it once it finished. Feed its `Wanted`
// edge here: the demand poll only sees the current level, so a reader
// gone before the next poll would otherwise leave the track unread
// with no linger armed.
if let Some(io) = self.tracks.get_mut(&name)
&& !io.weak.abort_unwanted(Error::Cancel)
&& (!io.weak.abort_unwanted(Error::Cancel) || io.read())
{
if !io.wanted {
io.wanted = true;
Expand Down Expand Up @@ -5457,7 +5469,7 @@ impl kio::Task for FrontServe {
let Some(io) = self.tracks.get_mut(&name) else { continue };
// Read again: the edge may have flipped back since it fired, and the next
// wait parks on it.
if io.weak.is_wanted() == io.wanted {
if io.read() == io.wanted {
continue;
}
io.wanted = !io.wanted;
Expand Down Expand Up @@ -15814,6 +15826,47 @@ mod tests {
assert!(res.is_err(), "{res:?}");
}

/// A finished track that something still reads outlives its linger, so a replacement
/// route is still asked for it, for a read that has to move off a dying route. Once
/// nothing holds it, it lingers and is forgotten as before.
#[kio::test]
#[cfg(feature = "async")]
async fn a_held_finished_track_outlives_its_linger() {
let (producer, _server, _upstream, old, mut subscription) = remote_front().await;
drop(old.finish().unwrap());
assert!(matches!(next_group(&mut subscription).await, Ok(None)));

// Held past the linger: a replacement is asked for the track.
kio::clock::sleep(Duration::from_secs(60)).await;
let newcomer = producer
.route("room/alice", RouteOptions::hidden().with_instance(instance(), 0))
.unwrap();
let mut handler = newcomer.broadcast().requests();
newcomer.announce().unwrap();
let request = kio::clock::timeout(Duration::from_secs(1), handler.requested_track())
.await
.expect("the front rerouted the held track")
.expect("request");
assert_eq!(request.name(), "video");
drop(request);

// Unheld, it is forgotten after the linger: a later route is asked nothing.
drop(subscription);
kio::clock::sleep(Duration::from_secs(60)).await;
drop(newcomer);
let third = producer
.route("room/alice", RouteOptions::hidden().with_instance(instance(), 0))
.unwrap();
let mut handler = third.broadcast().requests();
third.announce().unwrap();
assert!(
kio::clock::timeout(Duration::from_secs(1), handler.requested_track())
.await
.is_err(),
"an unheld finished track was rerouted after its linger"
);
}

/// The newcomer answers "video" with matching metadata but aborts before delivering a
/// group, all before the front polls the answer: that is a refusal, so the incumbent
/// keeps serving rather than the track ending on the newcomer's error.
Expand Down
Loading