Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,25 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re
```sh
quest gate just rs tla moq-net/verify/reconnect/Handover.tla
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg
```

The first checks every rule but the one with an open finding, over four sessions: about 1.7M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 290k states. Both pass at the sync commit. The last fails at the sync commit, on its finding below, within seconds. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
The first checks every rule over four sessions: about 1.4M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 300k states. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.

## Open findings

### A redirect after a probe got through is lost

`RedirectLost.cfg` fails. On a fallback, `probed` moves the live session into the predecessor and drops its `goaway::Consumer` with the `Live` it came from, so nothing reads that session's GOAWAY anymore. A redirect its peer sends while the handshake over the preferred transport runs is never followed: when that handshake fails, the next walk dials the address the peer is leaving. `residual_a_redirect_after_a_probe_got_through_is_lost` in `reconnect/tests.rs` replays it. Once fixed, flip that test, move `RedirectFollowed` into `Handover.cfg`, and delete `RedirectLost.cfg`.
None.

## Reading a failure

Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the policy with the helpers in `reconnect/tests.rs` (`next`, `handshake`, `on_fallback`, `probe`, `connected`, `failed`, `probed`, a server's `drain().send(..)`), one call per step, on the simulated clock. A path through `Shared` needs `moq-tokio`'s connection tests instead.

## Coverage

Sync commit: `4486b7adc108649bb3e7924554394ee30fa0e5f2`. Run `git log 4486b7adc1.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `bd617cc1eebc93e0c76a362cc83f2f3852c50244`. Run `git log bd617cc1ee.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
| `rs/moq-net/src/reconnect.rs` `Reconnect::poll` (queue, predecessor, phases) | `Pop`, `Retire`, `IdleClose`, `Dial`, `Abandon`, `BackoffEnds`, `LiveEnds`, `ProbeAsk`, `ProbeAbandon` |
| `rs/moq-net/src/reconnect.rs` `Reconnect::poll` (queue, predecessor, phases), `follow` | `Pop`, `Retire`, `IdleClose`, `FollowPred`, `Dial`, `Abandon`, `BackoffEnds`, `LiveEnds`, `ProbeAsk`, `ProbeAbandon` |
| `rs/moq-net/src/reconnect.rs` `established`, and `walk_failed` ending the first walk | `Established`, `Failed`, `Abandon` |
| `rs/moq-net/src/reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` |
| `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` |
Expand Down
141 changes: 141 additions & 0 deletions js/net/src/connection/reload.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1670,6 +1670,147 @@ test("a WebSocket fallback returns to WebTransport once it is reachable", async
}
});

// A redirect the fallback's peer sends while it drains, after a probe got through but its
// handshake failed, still decides where the replacement goes: the redial after the backoff goes
// where the peer named, not back to the node that is leaving.
test("a redirect from the draining WebSocket fallback is followed", async () => {
using clock = manual();
const originalWebTransport = globalThis.WebTransport;
const originalStream = (globalThis as { WebSocketStream?: unknown }).WebSocketStream;
const url = new URL("https://fallback-redirect.example.com/");
const elsewhere = new URL("https://fallback-redirect.example.com:4444/");

let blocked = true;
const dialed: string[] = [];
const servers: Promise<Session>[] = [];
let fallback: WebTransport | undefined;
globalThis.WebTransport = function StubWebTransport(to: string | URL) {
if (blocked) {
const refused = Promise.reject(new Error("UDP blocked"));
refused.catch(() => {});
return { ready: refused, closed: refused, close() {} };
}
dialed.push(String(to));
// The leaving node answers the probe, but no MoQ version is in common with it.
if (new URL(to).port !== elsewhere.port) return createMockTransportPair("h3").client;
const pair = createMockTransportPair(Lite.ALPN_06);
const server = accept(pair.server, new URL(to));
server.catch(() => {});
servers.push(server);
return pair.client;
} as unknown as typeof WebTransport;
(globalThis as { WebSocketStream?: unknown }).WebSocketStream = linkedWebSocketStream((socket, protocol) => {
const transport = Qmux.accept(socket, { protocol }) as unknown as WebTransport;
fallback = transport;
servers.push(transport.ready.then(() => accept(transport, url)));
});

const warned = spyOn(console, "warn").mockImplementation(() => {});
const reload = new Reload({ url, versions: [Lite.ALPN_06], delay: { initial: Duration.millis(5_000) } });
try {
for (let i = 0; i < 10 && reload.established.peek()?.transport !== "websocket"; i++) {
await clock.advance(Duration.millis(100));
}
const server = servers[0];
if (!server || !fallback) throw new Error("no fallback session");

blocked = false;
// The probe gets through, the fallback tells its peer it is leaving, and the handshake
// over the probe fails, so the loop backs off with the fallback still serving.
for (let i = 0; i < 20 && dialed.length === 0; i++) await clock.advance(Duration.millis(1_000));
expect(await (await server).draining).toEqual({ uri: "" });
await clock.advance(Duration.millis(10));
expect(reload.established.peek()?.transport).toBe("websocket");

const stream = await Stream.open(fallback, { version: Lite.Version.DRAFT_06 });
await stream.writer.u53(Lite.StreamId.Goaway);
await new Lite.Goaway(elsewhere.href).encode(stream.writer, Lite.Version.DRAFT_06);
for (let i = 0; i < 100 && reload.established.peek()?.transport !== "webtransport"; i++) {
await clock.advance(Duration.millis(100));
}
expect(reload.established.peek()?.transport).toBe("webtransport");
expect(dialed.map((to) => new URL(to).port)).toEqual(["", elsewhere.port]);
} finally {
reload.close();
warned.mockRestore();
for (const server of servers) (await server.catch(() => undefined))?.abort();
globalThis.WebTransport = originalWebTransport;
(globalThis as { WebSocketStream?: unknown }).WebSocketStream = originalStream;
}
});

// A redirect the draining fallback's peer sends while the probe's replacement serves is kept:
// once that replacement closes and the fallback serves again, the redial goes where the peer
// named.
test("a redirect from the draining WebSocket fallback outlives a replacement that closes", async () => {
using clock = manual();
const originalWebTransport = globalThis.WebTransport;
const originalStream = (globalThis as { WebSocketStream?: unknown }).WebSocketStream;
const url = new URL("https://fallback-held.example.com/");
const elsewhere = new URL("https://fallback-held.example.com:4444/");

let blocked = true;
const dialed: string[] = [];
const replacements: Promise<Session>[] = [];
const servers: Promise<Session>[] = [];
let fallback: WebTransport | undefined;
globalThis.WebTransport = function StubWebTransport(to: string | URL) {
if (blocked) {
const refused = Promise.reject(new Error("UDP blocked"));
refused.catch(() => {});
return { ready: refused, closed: refused, close() {} };
}
dialed.push(String(to));
const pair = createMockTransportPair(Lite.ALPN_06);
const server = accept(pair.server, new URL(to));
server.catch(() => {});
servers.push(server);
replacements.push(server);
return pair.client;
} as unknown as typeof WebTransport;
(globalThis as { WebSocketStream?: unknown }).WebSocketStream = linkedWebSocketStream((socket, protocol) => {
const transport = Qmux.accept(socket, { protocol }) as unknown as WebTransport;
fallback = transport;
servers.push(transport.ready.then(() => accept(transport, url)));
});

const warned = spyOn(console, "warn").mockImplementation(() => {});
const reload = new Reload({
url,
versions: [Lite.ALPN_06],
delay: { initial: Duration.millis(1_000) },
goaway: { handover: Duration.millis(60_000) },
});
try {
for (let i = 0; i < 10 && reload.established.peek()?.transport !== "websocket"; i++) {
await clock.advance(Duration.millis(100));
}
if (!fallback) throw new Error("no fallback session");

blocked = false;
for (let i = 0; i < 20 && reload.established.peek()?.transport !== "webtransport"; i++) {
await clock.advance(Duration.millis(1_000));
}
expect(reload.established.peek()?.transport).toBe("webtransport");

const stream = await Stream.open(fallback, { version: Lite.Version.DRAFT_06 });
await stream.writer.u53(Lite.StreamId.Goaway);
await new Lite.Goaway(elsewhere.href).encode(stream.writer, Lite.Version.DRAFT_06);
await clock.advance(Duration.millis(10));

// The replacement closes, so the fallback serves again until the redial.
(await replacements[0])?.abort();
for (let i = 0; i < 100 && dialed.length < 2; i++) await clock.advance(Duration.millis(100));
expect(dialed.map((to) => new URL(to).port)).toEqual(["", elsewhere.port]);
} finally {
reload.close();
warned.mockRestore();
for (const server of servers) (await server.catch(() => undefined))?.abort();
globalThis.WebTransport = originalWebTransport;
(globalThis as { WebSocketStream?: unknown }).WebSocketStream = originalStream;
}
});

// A WebSocket win is provisional: once the session it carried ends, the redial gives
// WebTransport its head start again, since the outage that lost the race may be over.
test("a WebSocket session ending restores WebTransport's head start", async () => {
Expand Down
55 changes: 52 additions & 3 deletions js/net/src/connection/reload.ts
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,8 @@ export class Reload {
// A session the peer sent GOAWAY on, serving its groups in flight while the replacement
// dials. Retired when it closes, at its handover cap, or when the sequence ends.
#draining: Draining | undefined;
// A redirect from the draining fallback's peer, held while a replacement serves.
#pending: { connection: Session; next: URL } | undefined;

// Sessions this loop ended that are still draining, each until its close settles.
#ending = new Map<Session, Promise<void>>();
Expand Down Expand Up @@ -646,6 +648,7 @@ export class Reload {
current = undefined;
// A predecessor still draining serves again, so this is no outage.
if (this.established.peek() === connection) this.#serve(this.#draining?.serving());
this.#applyPending();
this.#retry(effect, connected, ended ?? undefined);
return;
}
Expand All @@ -664,6 +667,7 @@ export class Reload {
// backoff already paces it, so its replacement dials at once. A fallback that
// served past the healthy bar still earns the healthy reset.
if (reached) {
this.#follow(connection, at, pinnedTransport(connection.transport, this.#pinned()));
if (now().since(connected).ge(this.#pacing().healthy)) {
this.#delay = undefined;
this.#deadline = undefined;
Expand Down Expand Up @@ -766,10 +770,8 @@ export class Reload {
* redirect is refused, which ends the sequence rather than redialing.
*/
#migrate(connection: Session, dialing: URL, drain: Drain): boolean {
const hashes = this.webtransport?.serverCertificateHashes?.length ?? 0;
const configured = hashes > 0 || this.webtransport?.serverCertificate !== undefined;
// The pin is a WebTransport option. A WebSocket that won the race never used it.
const pinned = pinnedTransport(connection.transport, configured);
const pinned = pinnedTransport(connection.transport, this.#pinned());

let next: URL | undefined;
try {
Expand Down Expand Up @@ -803,6 +805,53 @@ export class Reload {
return true;
}

/** Whether a certificate pin is configured for WebTransport. */
#pinned(): boolean {
const hashes = this.webtransport?.serverCertificateHashes?.length ?? 0;
return hashes > 0 || this.webtransport?.serverCertificate !== undefined;
}

/**
* Follow a redirect the peer of `connection`, a fallback session we left for a probe that
* got through, sends while it drains: whenever nothing else serves, the next dial goes where
* the peer named, and a dial still headed for the leaving peer starts over. A restart in
* place or a refused redirect is ignored, since we are already leaving. No loop scoring:
* only a probe that got through leads here, and the probe's backoff paces those.
*/
#follow(connection: Session, dialing: URL, pinned: boolean): void {
void connection.draining.then((drain) => {
let next: URL | undefined;
try {
next = target(this.goaway.redirect ?? "same-host", drain.uri, dialing, pinned);
} catch {
return;
}
if (!next) return;
this.#pending = { connection, next };
this.#applyPending();
});
}

/**
* Apply the pending redirect once its session is the one serving, which a replacement that
* closes hands back to; drop it once that session retires.
*/
#applyPending(): void {
const pending = this.#pending;
if (!pending) return;
if (this.#draining?.serving() !== pending.connection) {
this.#pending = undefined;
return;
}
if (this.established.peek() !== pending.connection) return;
this.#pending = undefined;
console.info("draining fallback peer redirected; following");
this.#redirect.set(pending.next);
this.#redirectHref = this.#sequenceHref;
// A backoff still paces the next dial, which now goes to the new target.
if (this.#dialing) this.#tick.update((prev) => prev + 1);
}

/**
* End a session this loop is done with. Its groups in flight finish within the close
* deadline, unless the page is going away: nothing would run to finish them, and an abort
Expand Down
33 changes: 0 additions & 33 deletions quest/m1/reconnect-probe-redirect.md

This file was deleted.

Loading