Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3cddafa
chore(quest): claim quest/m2/reconnect-no-error-kinds.md
kixelated Nov 13, 2026
3a08ed8
feat(reconnect)!: stop judging errors by kind
kixelated Nov 13, 2026
0b22109
test: integration tests assert retries over error kinds
kixelated Nov 13, 2026
82bc2a7
docs(verify): the reconnect model's closes all retry
kixelated Nov 13, 2026
2afb84b
test(js): a rejected session redials on the manual clock
kixelated Nov 13, 2026
3c3373d
fix: format, drill patch follows the policy
kixelated Nov 13, 2026
35b9647
test(moq-c): a rejected session redials once connected
kixelated Nov 13, 2026
177a46d
test(moq-c): the redialing client still reports the rejection's kind
kixelated Nov 13, 2026
db52dba
test(moq-gst): a refused redial keeps redialing after playback
kixelated Nov 13, 2026
d63133c
test(moq-relay): an in-band refusal keeps the native loop redialing
kixelated Nov 13, 2026
daa28b0
chore(quest): finish reconnect-no-error-kinds
kixelated Nov 13, 2026
ca025b7
docs(auth): a refusal retries with backoff
kixelated Nov 13, 2026
9b9a045
fix(review): reaches reads WebSocket's enabled flag, model's Failed r…
kixelated Nov 13, 2026
958cbd8
style: fmt
kixelated Nov 13, 2026
45a0d74
fix(auth): a session closed before answering its credential answers t…
kixelated Nov 13, 2026
c1c9c3e
Merge origin/main into quest/m2/reconnect-no-error-kinds
kixelated Nov 13, 2026
cf2931f
fix(review): reaches compiles without a transport, redial tests requi…
kixelated Nov 13, 2026
9e0b099
fix(auth): a session refused before the reconnect loop binds it still…
kixelated Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg
```

The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
The first checks every rule but the two with open findings, over four sessions: about 1.7M distinct states, about a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 290k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.

## Open findings

Expand All @@ -48,7 +48,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th

## Coverage

Sync commit: `c681b7a095bf2e1ce41bc405acdaa7993c5fe217`. Run `git log c681b7a095.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `4486b7adc108649bb3e7924554394ee30fa0e5f2`. Run `git log 4486b7adc1.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
Expand All @@ -62,4 +62,4 @@ Sync commit: `c681b7a095bf2e1ce41bc405acdaa7993c5fe217`. Run `git log c681b7a095

## Not covered

One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens and `Verdict::Refused`, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing.
One configured address, so the walk across several, `ATTEMPT`, and `Verdict::Settled` and `Unusable` are left out. So are the retry window and `Failure::Exhausted`, tokens, connecting once, fixed targets, and redirect-loop scoring (`Redirects`): a session's healthiness is chosen freely instead, which covers a looped one. Backoff delays and jitter are left out; every deadline can pass in any order. `Connection::close`, `Monitor`, bandwidth forwarding, and the QUIC fresh-port retry are left out, and a session's subscriptions reduce to whether the predecessor is idle. A pass covers four sessions under `Restart::Redial` and three under `Restart::Wait`, with two endpoints a GOAWAY may name and one it may not. The tests in `reconnect/tests.rs` and `moq-tokio`'s connection tests own the timing.
4 changes: 3 additions & 1 deletion doc/bin/relay/auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,9 @@ A native client can instead replace its configured tokens through its
connection's tokens (`Connection::tokens()`, `session.tokens()` in the
bindings): the next dial presents them, a live session presents them in band and
withdraws each old one once its replacement is answered, `answered()` reports
the relay's verdict, and a refusal redials only once the tokens have changed.
the relay's verdict, and once a session has connected, a refusal that closes
it is retried with backoff like any other failure, within the retry window,
presenting whatever tokens are current.
Older versions refuse in-band tokens.

Each accepted token adds its publish and subscribe patterns to the session's
Expand Down
51 changes: 24 additions & 27 deletions js/net/src/connection/reload.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ test("an ended auth handle gives up instead of retrying", async () => {
}
});

test("a page hide after give-up does not retry the refused URL", async () => {
test("a page hide after give-up does not retry the failed URL", async () => {
const original = globalThis.WebTransport;
const previousWindow = globalThis.window;
const previousDocument = globalThis.document;
Expand All @@ -345,10 +345,9 @@ test("a page hide after give-up does not retry the refused URL", async () => {
let attempts = 0;
const stub = function StubWebTransport() {
attempts++;
// Refused before any session, so the URL's first attempt fails and the loop gives up.
const pair = createMockTransportPair(Lite.ALPN_06);
void accept(pair.server, url).then(() => {
pair.server.close({ closeCode: SessionCode.Unauthorized, reason: "unauthorized" });
});
pair.server.close({ closeCode: SessionCode.Unauthorized, reason: "unauthorized" });
return pair.client;
};
globalThis.WebTransport = stub as unknown as typeof WebTransport;
Expand All @@ -366,7 +365,7 @@ test("a page hide after give-up does not retry the refused URL", async () => {
const err = reload.error.peek();

// Suspension shares sequence reset with give-up; resume must not look like a new
// sequence or it would clear `error` and redial the JWT the peer already refused.
// sequence or it would clear `error` and redial a URL that already gave up.
win.dispatchEvent(new Event("pagehide"));
await settle();
win.dispatchEvent(new Event("pageshow"));
Expand All @@ -385,12 +384,15 @@ test("a page hide after give-up does not retry the refused URL", async () => {
});

for (const code of [SessionCode.Unauthorized, SessionCode.MalformedAuthToken, SessionCode.Expired]) {
test(`a session rejected with ${code} surfaces the code and stops retrying`, async () => {
test(`a session rejected with ${code} surfaces the code and redials`, async () => {
using clock = manual();
const original = globalThis.WebTransport;
const url = new URL("https://example.com/");
const closes: (Error | null)[] = [];
let server: MockTransport | undefined;
const dials = new Signal(0);
const stub = function StubWebTransport() {
dials.update((n) => n + 1);
const pair = createMockTransportPair(Lite.ALPN_06);
server = pair.server;
void accept(pair.server, url);
Expand Down Expand Up @@ -418,18 +420,13 @@ for (const code of [SessionCode.Unauthorized, SessionCode.MalformedAuthToken, Se
});

try {
// The code reaches the app rather than being flattened away, and because
// the code is specified rather than guessed at, the loop stops this URL
// instead of retrying credentials that cannot work. `timeout: 0` means unlimited
// retries, so `error` settling at all is what proves it stopped on the rejection.
await waitUntil(() => reload.error.peek() !== undefined);
const err = reload.error.peek();
expect(err).toBeInstanceOf(SessionError);
expect((err as SessionError).code).toBe(code);
expect(reload.closed.peek()).toBeUndefined();

// It still surfaced through the established session before the loop gave up.
// The code reaches the app rather than being flattened away, and the loop redials
// like after any other close instead of judging the error's kind.
await clock.drive(waitFor(dials, (n) => n >= 2));
expect(closes.find((e) => e instanceof SessionError)).toBeInstanceOf(SessionError);
expect((closes.find((e) => e instanceof SessionError) as SessionError).code).toBe(code);
expect(reload.error.peek()).toBeUndefined();
expect(reload.closed.peek()).toBeUndefined();
} finally {
watch.close();
reload.close();
Expand Down Expand Up @@ -487,7 +484,7 @@ test("a refused credential replaced since its dial redials with the new one", as
}
});

test("new tokens redial a URL that stopped on refusing the old ones", async () => {
test("a refused credential keeps redialing until the tokens change", async () => {
using clock = manual();
const original = globalThis.WebTransport;
const { promise: release, resolve } = Promise.withResolvers<void>();
Expand All @@ -506,8 +503,9 @@ test("new tokens redial a URL that stopped on refusing the old ones", async () =
// Refused only once a session is up: a URL's first attempt failing stops it whatever the cause.
await clock.drive(waitFor(reload.established, (established) => established !== undefined));
resolve();
await clock.drive(waitFor(reload.error, (error) => error !== undefined));
expect(dialed.peek()).toEqual(["old"]);
await clock.drive(waitFor(dialed, (jwts) => jwts.length >= 3));
expect(dialed.peek().every((jwt) => jwt === "old")).toBe(true);
expect(reload.error.peek()).toBeUndefined();
tokens.set(["new"]);
await clock.drive(waitFor(dialed, (jwts) => jwts.includes("new")));
await clock.drive(waitFor(reload.established, (established) => established !== undefined));
Expand All @@ -518,7 +516,7 @@ test("new tokens redial a URL that stopped on refusing the old ones", async () =
}
});

test("an unauthorized session close during setup stops retrying", async () => {
test("an unauthorized session close during the first setup stops retrying", async () => {
const original = globalThis.WebTransport;
const url = new URL("https://example.com/");
let attempts = 0;
Expand Down Expand Up @@ -546,7 +544,7 @@ test("an unauthorized session close during setup stops retrying", async () => {
url,
websocket: { enabled: false },
tokens: ["old"],
// Unlimited retries prove that `closed` settles only because the rejection is terminal.
// Unlimited retries prove that `error` settles only because the first attempt failed.
delay: { initial: Duration.millis(1), multiplier: 1, max: Duration.millis(1), timeout: null },
});

Expand All @@ -558,15 +556,14 @@ test("an unauthorized session close during setup stops retrying", async () => {
expect(attempts).toBe(1);
expect(reload.closed.peek()).toBeUndefined();

// The first attempt failing stops the URL, yet new credentials still earn a dial.
// New credentials alone do not restart a URL that gave up.
reload.tokens.set(["new"]);
await waitUntil(() => attempts === 2);
await waitUntil(() => reload.error.peek() !== undefined);
for (let i = 0; i < 20; i++) await settle();
expect(attempts).toBe(1);

// A new URL starts another sequence on the same loop.
reload.url.set(new URL("https://example.com/changed"));
await waitUntil(() => attempts >= 3);
expect(attempts).toBeGreaterThanOrEqual(3);
await waitUntil(() => attempts >= 2);
} finally {
reload.close();
globalThis.WebTransport = original;
Expand Down
72 changes: 8 additions & 64 deletions js/net/src/connection/reload.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import type * as Announce from "../announced.ts";
import { Handle as AuthHandle } from "../auth.ts";
import { ended } from "../auth_session.ts";
import { Allocator, reserved, Viewers } from "../bandwidth.ts";
import { error, reason, SessionCode, SessionError } from "../error.ts";
import { error, reason } from "../error.ts";
import type { Consumer as OriginConsumer, Producer as OriginProducer } from "../origin.ts";
import * as Path from "../path.ts";
import { Follow } from "../util/observed.ts";
Expand Down Expand Up @@ -36,29 +36,15 @@ const PROBE_INITIAL = Duration.secs(10);
/** The longest wait between WebTransport probes, before jitter. */
const PROBE_MAX = Duration.secs(300);

function sameTokens(a: readonly string[], b: readonly string[]): boolean {
return a.length === b.length && a.every((token, i) => token === b[i]);
}

/** Whether `cause` is the peer refusing the credentials a session presented. */
function refusal(cause: unknown): cause is SessionError {
return (
cause instanceof SessionError &&
(cause.code === SessionCode.Unauthorized ||
cause.code === SessionCode.MalformedAuthToken ||
cause.code === SessionCode.Expired)
);
}

/**
* Exponential backoff settings for the reconnect loop.
*
* The delays carry jitter, so a fleet of tabs knocked offline together doesn't reconnect in
* lockstep. Nothing retries until a session to the current URL has been established: the
* first attempt failing stops that URL with its error. After that, every failure is retried,
* including an attempt that outlives {@link ConnectProps.timeout}, until
* {@link ReloadDelay.timeout} runs out; only an auth rejection or a refused GOAWAY redirect
* stops the current URL sooner. A new URL or a disable/re-enable starts another sequence.
* {@link ReloadDelay.timeout} runs out, whatever the error, an auth rejection included; only a
* refused GOAWAY redirect stops the current URL sooner. A new URL or a disable/re-enable starts another sequence.
*
* A GOAWAY off a session that outlived {@link ReloadDelay.healthy} migrates at once, unless it
* sends the connection back to an endpoint that already redirected it elsewhere and the
Expand Down Expand Up @@ -220,8 +206,7 @@ export class Reload {
/**
* Credentials every dial presents; see {@link ConnectProps.tokens}. Replacing them reaches
* the next dial and, on a live session with AUTH, presents the new tokens at once,
* withdrawing the replaced ones once every new one has its answer. A session refused as
* unauthorized redials only when these changed since its dial.
* withdrawing the replaced ones once every new one has its answer.
*/
tokens: Signal<readonly string[]>;

Expand All @@ -231,7 +216,7 @@ export class Reload {
/**
* The failure that stopped retrying the current URL, or undefined while the loop is live.
*
* Set when the first attempt for a URL fails, on an auth rejection, or when the retry window
* Set when the first attempt for a URL fails, or when the retry window
* expires. Cleared when a new URL or a
* disable/re-enable starts another sequence, not on a page hide/show. Transient drops
* that are still being retried leave this empty.
Expand Down Expand Up @@ -376,15 +361,10 @@ export class Reload {
#sequenceHref: string | undefined;

// The href that exhausted its retry sequence. Survives a page hide/show so resume does
// not redial credentials the peer already refused. Cleared when the URL changes or the
// does not restart a sequence that already gave up. Cleared when the URL changes or the
// loop is disabled.
#givenUpHref: string | undefined;

// The tokens the latest dial presented, so a refusal can tell whether they changed since.
#dialedTokens: readonly string[] = [];
// Whether the sequence stopped because the peer refused those tokens.
#refused = false;

// The configured href a session has been established for. Until it matches, a failed attempt
// gives up rather than retrying. Kept across a page hide/show and a disable/re-enable, which
// resume a URL that worked.
Expand Down Expand Up @@ -506,9 +486,6 @@ export class Reload {
if (connection) heldOf(connection)?.update(tokens);
});

// New credentials redial a URL that stopped on an auth refusal of the old ones.
this.#signals.run((effect) => this.#redialRefused(effect.get(this.tokens)));

this.#url = this.#signals.computed((effect) => effect.get(this.url)?.href);
// Create a reactive root so cleanup is easier.
this.#signals.run(this.#connect.bind(this));
Expand Down Expand Up @@ -593,15 +570,15 @@ export class Reload {
this.#deadline ??= this.#window(now());

this.#setDialing(true);
this.#dialedTokens = reached?.tokens ?? this.tokens.peek();
const tokens = reached?.tokens ?? this.tokens.peek();
let connection: Session;
const reusing = reached;
reached = undefined;
try {
connection = await establish(
{
url: dialing,
tokens: this.#dialedTokens,
tokens,
// A redirect names the relay; a fallback URL pinned for the old one does not follow.
websocket: this.#redirect.peek()
? { ...this.websocket, url: undefined }
Expand Down Expand Up @@ -742,10 +719,6 @@ export class Reload {
if (this.#establishedHref !== href) {
console.warn("connection failed, not retrying:", err);
this.#giveUp(error(err));
// A refused credential still redials once the tokens change, including a
// change that landed while this dial was in flight.
this.#refused = refusal(err);
this.#redialRefused(this.tokens.peek());
return;
}

Expand Down Expand Up @@ -918,25 +891,6 @@ export class Reload {
this.#deadline = undefined;
}

// An auth rejection stops this URL however long the session lived. UNAUTHORIZED,
// MALFORMED_AUTH_TOKEN, and EXPIRED_AUTH_TOKEN are specified codes rather than ones we guessed at, so this is the
// peer saying these credentials will never work again; retrying them just burns the
// window, and an expired token needs the app to refresh it via a new URL. Matches
// moq-tokio's reconnect loop, which stops on the same close. A new URL or a
// disable/re-enable starts another sequence; the handle itself is not disposed.
//
// Only a session close says that. The stream registry gives 2 to DELIVERY_TIMEOUT,
// so a stream reset during the SETUP exchange would otherwise suppress reconnect
// for good.
//
// Tokens replaced since that dial are new credentials, so those earn another dial.
if (refusal(cause) && sameTokens(this.tokens.peek(), this.#dialedTokens)) {
console.warn("session rejected as unauthorized, not retrying");
this.#giveUp(cause);
this.#refused = true;
return;
}

const at = now();
// A healthy session's end starts the window here.
this.#deadline ??= this.#window(at);
Expand Down Expand Up @@ -967,17 +921,7 @@ export class Reload {
}, wait);
}

// Redial a URL that stopped on an auth refusal, once `tokens` differ from the refused ones.
#redialRefused(tokens: readonly string[]): void {
if (!this.#refused || this.#givenUpHref === undefined || sameTokens(tokens, this.#dialedTokens)) return;
this.#refused = false;
this.#givenUpHref = undefined;
this.#error.set(undefined);
this.#tick.update((tick) => tick + 1);
}

#giveUp(cause: Error): void {
this.#refused = false;
this.#error.set(cause);
this.#givenUpHref = this.#sequenceHref;
this.#resetSequence();
Expand Down
Loading