Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions .claude/skills/verify-failover/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,11 @@ The first checks every rule with `E` failing over between `A` and `B`, one death

## Open finding: an ended relay cannot move a fetch off a dying route

`EndedRelay.cfg` breaks `NoSkipWhileHeld`. `E` reads group 0 through `A` and hands out group 1, then switches to `B`, asking it from group 1, and fetches the rest of group 0 from `B`. `B` moves its route to `A`, its copy from group 1 completes, and its track ends (`Action::Finish`); then `A` dies. `B`'s front no longer switches, so its fetch for group 0 fails although `B` still reaches `O` directly (`Fetching` returns the dead copy's error once `route.end()` is set), and `E`, whose serving route gave up on the group, cuts it short. The real stack breaks the same rule earlier, on the first code finding below, so this path has no replay of its own yet; once that is fixed, replay it in `failover_chain.rs` and pin or fix it. Once both are fixed, merge `EndedRelay.cfg`'s `Mesh` into `FailoverChain.cfg` if the bounds allow, and delete `EndedRelay.cfg`.
`EndedRelay.cfg` breaks `NoSkipWhileHeld`. `E` reads group 0 through `A` and hands out group 1, then switches to `B`, asking it from group 1, and fetches the rest of group 0 from `B`. `B` moves its route to `A`, its copy from group 1 completes, and its track ends (`Action::Finish`); then `A` dies. `B`'s front no longer switches, so its fetch for group 0 fails although `B` still reaches `O` directly, and `E`, whose serving route gave up on the group, cuts it short.

## Open findings: the code falls short of the model
`residual_a_fetch_cannot_move_off_a_dying_route_after_the_end` in `rs/moq-net/tests/integration/failover_chain.rs` replays it, and the real stack does worse: the read waits out the subscriber's budget (`Settles` too). `B`'s logical track closes once it finished, so its front reads the track as unread and splices no copy from the route to `O` that replaces `A`'s; the fetch's `Recover` waits on a concluded route whose serving copy completed cleanly before its FETCH failed. A fix keeps an ended track routable for a fetch, here and in the model's `Switch` and `Abort`; then flip the test, merge `EndedRelay.cfg`'s `Mesh` into `FailoverChain.cfg` if the bounds allow, and delete `EndedRelay.cfg`.

The model lets a fetch reach the serving route whenever that route lives, after the track ended too. Replaying its paths in `rs/moq-net/tests/integration/failover_chain.rs` found two places where the code does not, each pinned as a `residual_*` test that passes while the bug stands; `a_half_read_group_continues_through_the_new_route` is the same path without the end, which passes. Once one is fixed, flip its test.

- `residual_ending_the_track_fails_a_fetch_in_flight`: `E`'s fetch of the rest of a group is in flight through `B` when the publisher ends the track. `B`'s copy, asked from a later group, completes, `B`'s fetch from the publisher is cancelled, and `E`'s read fails with `NotFound` while the publisher still writes the rest (`NoSkipWhileHeld`).
- `residual_a_fetch_from_a_completed_copy_never_leaves_the_relay`: the same group read on only after `E`'s serving copy completed. `Recover::poll_fetch` calls `fetch_group` on that copy, but no FETCH goes on the wire, so the read waits out the subscriber's budget (`Settles`).
The model lets a fetch reach the serving route whenever that route lives, after the track ended too. Without the mesh the code now does: a relay's finished copy keeps answering fetches while it is read (`ending_the_track_keeps_a_fetch_in_flight`, `a_fetch_from_a_completed_copy_goes_on_the_wire`).

## Reading a failure

Expand Down
36 changes: 18 additions & 18 deletions quest/m1/failover-fetch-residuals.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,28 @@

## Goal

On a relay chain, a fetch already in flight when the publisher ends the track
still delivers the rest of its group, and a fetch asked of a copy that has
completed goes on the wire instead of waiting out the subscriber's budget.
Both `residual_*` tests in `rs/moq-net/tests/integration/failover_chain.rs`
flip to assert delivery, and `EndedRelay.cfg`'s open finding is gone.
A relay whose track ended still moves a fetch off a route that dies:
`residual_a_fetch_cannot_move_off_a_dying_route_after_the_end` in
`rs/moq-net/tests/integration/failover_chain.rs` flips to assert delivery, and
`EndedRelay.cfg`'s open finding is gone (its `Mesh` merged into
`FailoverChain.cfg` if the bounds allow).

## Plan

Found 2026-11-14 by the [feature review](/quest/m3/release/feature-review.md):
the tests pin two bugs `verify/failover/FailoverChain.tla` rules out, and no
quest or PR owns them.
Found 2026-11-14 by the [feature review](/quest/m3/release/feature-review.md).
The first two residuals (a fetch in flight when the track ends, and a fetch
asked of a completed copy) are fixed: a relay's finished copy keeps serving
fetches while it is read.

- `residual_ending_the_track_fails_a_fetch_in_flight` (`NoSkipWhileHeld`):
E's fetch of group 0 is in flight through B when the track ends; B's copy,
asked from group 1, completes, which cancels B's fetch from P, so E's read
fails though P still writes the rest.
- `residual_a_fetch_from_a_completed_copy_never_leaves_the_relay`
(`Settles`): reading group 0 on after E's track ended asks B's completed
copy, and that fetch never goes on the wire, a hang bounded only by the
subscriber's budget.
What remains is the mesh case. B's front ends the track (`Action::Finish`),
so its logical track closes and reads as unread; when A dies, nothing splices
a copy from B's direct route to O, and the fetch's `Recover` waits on a
concluded route until the subscriber's budget runs out.

Replay `EndedRelay.cfg` on the real stack, fix at the cause, and run
`/verify-failover` and `/verify-resume` on the head.
- Keep an ended track routable for a fetch in flight in `model/front.rs`,
minding the parked index and `forget`.
- In `FailoverChain.tla`, allow `Switch` for a track with `end = "ok"` whose
serving copy died, and adjust the `Abort` guard to match.
- Run `/verify-failover` and `/verify-resume` on the head.

Public API: none. Wire: none.
56 changes: 56 additions & 0 deletions rs/kio/src/task.rs
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,18 @@ impl<T> Tasks<T> {
}
}

/// Cancel every task `keep` rejects.
pub fn retain(&mut self, mut keep: impl FnMut(&T) -> bool) {
for (index, slot) in self.children.iter_mut().enumerate() {
if slot.as_ref().is_some_and(|occupant| !keep(&occupant.task)) {
// A wake bit left set costs the slot's next occupant one spurious poll at most.
*slot = None;
self.free.push(index);
self.len -= 1;
}
}
}

/// Take a task out of the set before it finishes, cancelling it there.
///
/// `None` when the task already finished or was removed.
Expand Down Expand Up @@ -641,6 +653,50 @@ mod tests {
assert_eq!(b_polls.load(Ordering::SeqCst), 1, "b was not");
}

/// A task named by its tag, finishing when its gate opens.
struct Tagged {
tag: usize,
gate: crate::Consumer<bool>,
}

impl Task for Tagged {
type Output = ();

fn poll(&mut self, waiter: &Waiter) -> Poll<()> {
self.gate
.poll(waiter, |open| if **open { Poll::Ready(()) } else { Poll::Pending })
.map(|_| ())
}
}

/// A rejected task is cancelled and its slot reused, while a kept one still runs.
#[test]
fn retain_cancels_only_the_rejected() {
let mut tasks = Tasks::new();
let gates: Vec<_> = (0..3).map(|_| crate::Producer::new(false)).collect();
for (tag, gate) in gates.iter().enumerate() {
tasks.push(Tagged {
tag,
gate: gate.consume(),
});
}
let waiter = Waiter::noop();
assert!(tasks.poll(&waiter).is_pending());

tasks.retain(|task| task.tag == 1);
assert_eq!(tasks.len(), 1);
assert_eq!(tasks.iter().map(|task| task.tag).collect::<Vec<_>>(), [1]);

tasks.push(Tagged {
tag: 3,
gate: gates[0].consume(),
});
assert_eq!(tasks.children.len(), 3, "a freed slot is reused");
open(&gates[0]);
open(&gates[1]);
assert!(tasks.poll(&waiter).is_ready(), "the kept task still finishes");
}

/// Finished tasks retire, their slots are reused, and an empty set is Ready.
#[test]
fn finished_tasks_retire_and_slots_recycle() {
Expand Down
Loading