Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re
| --- | --- | --- |
| `NoStaleSession` | The session a `Connection` reports as live is never one the policy aborted or started closing. | An app opens subscriptions on a session that is going away. |
| `StatusTruthful` | Connected is on the policy's live session, Migrating on its predecessor while nothing else is live, Closed exactly when the policy is done, and the predecessor `Connection::close` drains is the policy's. | Status or `close` acts on the wrong session, or a closed loop looks alive. |
| `OfflineMeansNothingServes` | Disconnected is reported only when no predecessor still serves (`Action::Offline`: "nothing serves anymore"). | An app waiting on `established` stalls, and the UI says disconnected, while a session still carries its media. |
| `OfflineMeansNothingServes` | Disconnected is reported only when no predecessor still serves (`Action::Offline`: "nothing serves anymore"); one closing gracefully carries nothing and takes no new work. | An app waiting on `established` stalls, and the UI says disconnected, while a session still carries its media. |
| `NoOrphan` | Until the policy is done, every open session is the live one or the predecessor. | A transport and the peer's state stay open with nothing able to reach them. |
| `PresenceBalanced` | `sessions_started - sessions_ended` is 1 while connected and 0 otherwise. | Presence counts drift on every migration. |
| `FirstWalkFailsFast` | Nothing backs off before a session has been established: a failed first walk ends the policy (a refusal of since-replaced tokens is the exception, and tokens are not modeled). | A setup mistake (an untrusted certificate, no version in common) retries forever instead of surfacing. |
Expand All @@ -27,21 +27,16 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re
quest gate just rs tla moq-net/verify/reconnect/Handover.tla
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg
```

The first checks every rule but the two with open findings, over four sessions: about 1.7M distinct states, about a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 290k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
The first checks every rule but the one with an open finding, over four sessions: about 1.7M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 290k states. Both pass at the sync commit. The last fails at the sync commit, on its finding below, within seconds. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.

## Open findings

### A redirect after a probe got through is lost

`RedirectLost.cfg` fails. On a fallback, `probed` moves the live session into the predecessor and drops its `goaway::Consumer` with the `Live` it came from, so nothing reads that session's GOAWAY anymore. A redirect its peer sends while the handshake over the preferred transport runs is never followed: when that handshake fails, the next walk dials the address the peer is leaving. `residual_a_redirect_after_a_probe_got_through_is_lost` in `reconnect/tests.rs` replays it. Once fixed, flip that test, move `RedirectFollowed` into `Handover.cfg`, and delete `RedirectLost.cfg`.

### A replacement dropping mid-handover reports Disconnected

`OfflineWhileServing.cfg` fails. When the replacement closes while the predecessor still drains, `closed` queues `Action::Offline`, whose doc promises nothing serves, and `Shared::offline` reports Disconnected and stops handing out a session, though the predecessor still carries its subscriptions until it retires. `residual_a_dropped_replacement_reports_offline_while_the_predecessor_serves` replays it on the policy. Once fixed, flip that test, move `OfflineMeansNothingServes` into `Handover.cfg`, and delete `OfflineWhileServing.cfg`.

## Reading a failure

Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the policy with the helpers in `reconnect/tests.rs` (`next`, `handshake`, `on_fallback`, `probe`, `connected`, `failed`, `probed`, a server's `drain().send(..)`), one call per step, on the simulated clock. A path through `Shared` needs `moq-tokio`'s connection tests instead.
Expand All @@ -58,7 +53,7 @@ Sync commit: `4486b7adc108649bb3e7924554394ee30fa0e5f2`. Run `git log 4486b7adc1
| `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` |
| `rs/moq-net/src/reconnect.rs` `Restart::Wait` | `LiveEnds` with `Restart = "wait"` (`RestartWait.cfg`) |
| `rs/moq-net/src/reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` |
| `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `predecessor`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` |
| `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `retired`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` |

## Not covered

Expand Down
38 changes: 38 additions & 0 deletions js/net/src/connection/migrate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,44 @@ test("an empty GOAWAY migrates the follower to the replacement session", async (
}
});

test("a replacement that closes mid-handover hands back to the draining session", async () => {
const clock = new Manual();
const installed = clock.install();
const { dials } = fleet();
const reload = new Reload({
url,
websocket: { enabled: false },
goaway: { handover },
delay: { initial: Duration.millis(1) },
});

try {
await waitUntil(() => reload.established.peek() !== undefined);
const first = reload.established.peek();
const drained = dials[0];
if (!drained) throw new Error("no first dial");
await goaway(drained.server, "");
await elapse(clock, () => reload.established.peek() !== first && reload.established.peek() !== undefined);
const replacement = dials[1];
if (!replacement) throw new Error("no replacement dial");

replacement.server.close();
await elapse(clock, () => replacement.closed);
await elapse(clock, () => reload.established.peek() === first || reload.established.peek() === undefined);
// Nothing is down: the drained session still serves until its handover ends.
expect(reload.established.peek()).toBe(first);
expect(reload.status.peek()).toBe("connected");
expect(drained.closed).toBe(false);

// Its handover still bounds it; then a redial serves.
await elapse(clock, () => drained.closed);
expect(reload.established.peek()).not.toBe(first);
} finally {
reload.close();
installed.close();
}
});

test("a GOAWAY without a timeout hands over at the configured cap", async () => {
const clock = new Manual();
const installed = clock.install();
Expand Down
8 changes: 7 additions & 1 deletion js/net/src/connection/reload.ts
Original file line number Diff line number Diff line change
Expand Up @@ -644,7 +644,8 @@ export class Reload {
console.warn(`connection closed, reconnecting: ${ended ? reason(ended) : "no error"}`);
this.#redirects.reset();
current = undefined;
if (this.established.peek() === connection) this.#serve(undefined);
// A predecessor still draining serves again, so this is no outage.
if (this.established.peek() === connection) this.#serve(this.#draining?.serving());
this.#retry(effect, connected, ended ?? undefined);
return;
}
Expand Down Expand Up @@ -1035,6 +1036,11 @@ class Draining {
});
}

/** The old session while it still serves, before it retires. */
serving(): Session | undefined {
return this.#retired ? undefined : this.#connection;
}

/** Stop serving from the old session, returning it the first time for the caller to end. */
retire(): Session | undefined {
if (this.#retired) return undefined;
Expand Down
32 changes: 0 additions & 32 deletions quest/m3/reconnect-predecessor-status.md

This file was deleted.

20 changes: 16 additions & 4 deletions rs/moq-net/src/reconnect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -328,15 +328,16 @@ pub enum Action<A, E> {
/// Drop the dial or probe in flight: it ran out its deadline, which already counted as
/// its failure.
Abandon,
/// The live session got a GOAWAY and now drains as the [`predecessor`](Reconnect::predecessor)
/// while a replacement is dialed.
/// The [`predecessor`](Reconnect::predecessor) serves while a replacement is dialed: the
/// live session got a GOAWAY and now drains as it, or the replacement closed while it
/// still drains.
Migrating,
/// The predecessor closed or overstayed its handover.
Retired,
/// Nothing serves anymore: the live session ended with this error, or the predecessor
/// retired between sessions. A redial follows.
Offline(Option<E>),
/// Why the policy is waiting to redial: a walk failed or a session ended too soon.
/// Why the policy is waiting to redial: a walk failed, or a session ended.
Retrying(E),
/// Poll again by this instant, or on a wake.
Wait(Option<Instant>),
Expand Down Expand Up @@ -1006,7 +1007,18 @@ impl<A: Target, E: Clone + std::fmt::Display + From<Error> + From<Failure>> Reco
return self.done(Err(err));
}

self.queue.push_back(Action::Offline(Some(err.clone())));
// A predecessor still draining serves again, so this is no outage: the caller moves
// back onto it. One closing gracefully already carries nothing and takes no new work.
if self
.predecessor
.as_ref()
.is_some_and(|predecessor| !predecessor.closing)
{
self.queue.push_back(Action::Migrating);
self.queue.push_back(Action::Retrying(err.clone()));
} else {
self.queue.push_back(Action::Offline(Some(err.clone())));
}
if healthy {
// The window was reset, so a one-off drop reconnects promptly.
tracing::warn!(peer = %Shown(url), "session closed, reconnecting");
Expand Down
68 changes: 59 additions & 9 deletions rs/moq-net/src/reconnect/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1002,11 +1002,11 @@ async fn busy_handshake() -> (Session, Session, Vec<Box<dyn std::any::Any>>) {
)
}

/// Pins an open finding (`verify-reconnect`, `OfflineMeansNothingServes`): a replacement
/// that drops while the predecessor still drains reports [`Action::Offline`], whose
/// promise is that nothing serves, though the predecessor still carries a subscription.
/// A replacement that drops while the predecessor still serves hands back to it as
/// [`Action::Migrating`], never [`Action::Offline`]: nothing is down, and the predecessor
/// still carries a subscription. Only its retirement then takes the connection offline.
#[kio::test]
async fn residual_a_dropped_replacement_reports_offline_while_the_predecessor_serves() {
async fn a_dropped_replacement_migrates_back_to_the_serving_predecessor() {
let a = url("tcp://peer:1/");
let b = url("tcp://peer:2/");
let mut machine: Reconnect<Url, TestError> = Reconnect::new(Config::default(), vec![a], kio::clock::now()).unwrap();
Expand All @@ -1026,16 +1026,66 @@ async fn residual_a_dropped_replacement_reports_offline_while_the_predecessor_se
drop(replacement_server);

let action = next(&mut machine).await;
assert!(
matches!(action, Action::Offline(Some(_))),
"the replacement's drop no longer reports offline: flip this test ({action:?})"
);
let (predecessor, _) = machine.predecessor().expect("the predecessor still drains");
assert!(matches!(action, Action::Migrating), "{action:?}");
let action = next(&mut machine).await;
assert!(matches!(action, Action::Retrying(TestError::Net(_))), "{action:?}");
assert!(machine.session().is_none(), "the replacement is gone");
let (predecessor, deadline) = machine.predecessor().expect("the predecessor still drains");
assert!(predecessor.poll_closed(&kio::Waiter::noop()).is_pending());
assert!(
predecessor.poll_idle(&kio::Waiter::noop()).is_pending(),
"it still serves"
);

// The redial waits out its backoff while the predecessor serves, and its handover still
// bounds it: once that ends, nothing serves.
assert!(matches!(next(&mut machine).await, Action::Dial(_)));
let action = next(&mut machine).await;
assert!(matches!(action, Action::Retired), "{action:?}");
assert!(kio::clock::now() >= deadline);
let action = next(&mut machine).await;
assert!(matches!(action, Action::Offline(None)), "{action:?}");
}

/// A replacement that drops while the predecessor closes gracefully, carrying nothing,
/// leaves nothing serving: it reports [`Action::Offline`].
#[kio::test]
async fn a_dropped_replacement_goes_offline_over_a_closing_predecessor() {
let a = url("tcp://peer:1/");
let mut machine: Reconnect<Url, TestError> =
Reconnect::new(Config::default(), vec![a.clone()], kio::clock::now()).unwrap();
let mut run = Run::default();
drive(&mut machine, &[(a, Peer::Serve)], &mut run, |run| {
!run.accepts.is_empty()
})
.await;

kio::clock::sleep(Duration::from_secs(2)).await;
run.servers[0].drain().send(goaway::Goaway::new()).unwrap();
assert!(matches!(next(&mut machine).await, Action::Migrating));
assert!(matches!(next(&mut machine).await, Action::Dial(_)));
let (replacement, replacement_server) = handshake().await;
machine.connected(kio::clock::now(), replacement.clone());
// The replacement is already closed when the next poll starts the idle predecessor's
// graceful close, so its close is seen over a predecessor that is closing, not retired.
drop(replacement_server);
replacement.closed().await;

let action = next(&mut machine).await;
assert!(matches!(action, Action::Offline(Some(_))), "{action:?}");
let (_, deadline) = machine.predecessor().expect("the predecessor is still closing");
// The graceful close then ends it well before its handover would.
let mut actions = Vec::new();
loop {
match next(&mut machine).await {
Action::Dial(_) => break,
Action::Retired => actions.push("retired"),
Action::Offline(None) => actions.push("offline"),
action => panic!("unexpected {action:?}"),
}
}
assert_eq!(actions, ["retired", "offline"]);
assert!(kio::clock::now() < deadline);
}

/// A target that never reaches the preferred transport keeps its fallback session and
Expand Down
9 changes: 9 additions & 0 deletions rs/moq-net/src/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,15 @@ pub struct Session {
datagrams_unrouted: Option<Arc<AtomicU64>>,
}

/// Two handles are equal when they are clones of the same session.
impl PartialEq for Session {
fn eq(&self, other: &Self) -> bool {
self.close.same_channel(&other.close)
}
}

impl Eq for Session {}

impl Session {
/// Returns the negotiated protocol version.
pub fn version(&self) -> Version {
Expand Down
4 changes: 2 additions & 2 deletions rs/moq-net/verify/reconnect/Handover.cfg
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
\* Two endpoints a GOAWAY may move between, one it may not, and four sessions.
\* RedirectFollowed and OfflineMeansNothingServes live in RedirectLost.cfg and
\* OfflineWhileServing.cfg while their findings are open.
\* RedirectFollowed lives in RedirectLost.cfg while its finding is open.
CONSTANTS
Endpoints = {"a", "b"}
Start = "a"
Expand All @@ -14,6 +13,7 @@ VIEW View
INVARIANTS
NoStaleSession
StatusTruthful
OfflineMeansNothingServes
NoOrphan
PresenceBalanced
FirstWalkFailsFast
29 changes: 20 additions & 9 deletions rs/moq-net/verify/reconnect/Handover.tla
Original file line number Diff line number Diff line change
Expand Up @@ -111,13 +111,21 @@ Abort(sx, s) == IF s # None /\ sx[s].open
\* `Shared::predecessor`.
Mirror(p) == spred' = IF lc \in {"live", "offline"} THEN p ELSE spred

\* `Action::Migrating`: publish the predecessor, then `Shared::migrating`.
\* Leaving the fallback drops the probe in flight.
\* `Action::Migrating`: `Shared::migrating` with the policy's predecessor. The
\* mirrored one means the replacement on top of it closed, and the handle
\* serves on it again; any other is the live session, which got the GOAWAY,
\* and is mirrored as it. Leaving the fallback drops the probe in flight.
OnMigrating(p) ==
/\ Mirror(p)
/\ migrating' = IF lc = "live" THEN TRUE ELSE migrating
/\ IF lc = "live" /\ p # None
THEN /\ migrating' = TRUE
/\ IF spred = p
THEN /\ lcSess' = p
/\ UNCHANGED spred
ELSE /\ spred' = p
/\ UNCHANGED lcSess
ELSE UNCHANGED <<migrating, spred, lcSess>>
/\ probing' = FALSE
/\ UNCHANGED <<lc, lcSess, started, ended>>
/\ UNCHANGED <<lc, started, ended>>

OnRetired ==
/\ Mirror(None)
Expand Down Expand Up @@ -279,13 +287,15 @@ EndGoaway(s, uri) ==
/\ Emit(<<"Migrating">>, s)
/\ UNCHANGED watch

\* `Reconnect::closed`: every close retries, whatever its error.
\* `Reconnect::closed`: every close retries, whatever its error. A predecessor
\* that is not closing gracefully serves again, so the policy migrates back
\* onto it; the `Retrying` that follows changes no status, so it is left out.
EndClosed ==
/\ live' = None
/\ phase' = "backoff"
/\ timer' = [timer EXCEPT !["backoff"] = FALSE]
/\ probe' = "none"
/\ Emit(<<"Offline">>, pred)
/\ Emit(IF pred # None /\ ~sess[pred].closing THEN <<"Migrating">> ELSE <<"Offline">>, pred)
/\ UNCHANGED <<pred, sess, target, watch>>

\* `Phase::Live`: a GOAWAY is checked before the close.
Expand Down Expand Up @@ -537,9 +547,10 @@ StatusTruthful ==
/\ (lc = "closed") = (phase = "done")
/\ lc # "closed" => spred = pred

\* `Action::Offline`: nothing serves anymore.
\* `Action::Offline`: nothing serves anymore. A predecessor closing gracefully
\* already carries nothing and takes no new work.
OfflineMeansNothingServes ==
Parked /\ lc = "offline" => ~Open(pred)
Parked /\ lc = "offline" => ~Open(pred) \/ sess[pred].closing

\* Every session the policy let go of is closed: one left open holds a
\* transport and the peer's state with nothing able to reach it.
Expand Down
15 changes: 0 additions & 15 deletions rs/moq-net/verify/reconnect/OfflineWhileServing.cfg

This file was deleted.

1 change: 1 addition & 0 deletions rs/moq-net/verify/reconnect/RestartWait.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ VIEW View
INVARIANTS
NoStaleSession
StatusTruthful
OfflineMeansNothingServes
NoOrphan
PresenceBalanced
FirstWalkFailsFast
Loading