Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,8 +347,8 @@ services.moq-relay = {
```

Each dialed session binds its own UDP socket on an ephemeral port of
`connect.bind`'s address. Once `connect.backoff.timeout` (10s by default; `0`
takes that default here) passes without a session, the redial leaves from a
`connect.bind`'s address. Once `connect.backoff.timeout` (10s when unset or `0`,
which would otherwise retry forever) passes without a session, the redial leaves from a
fresh port, so a network path that drops one UDP flow can't hold a link down
while other ports to the same peer would work. A fixed `connect.bind` port opts
out: static and API peers then share one socket on that port, and a redial
Expand Down
2 changes: 1 addition & 1 deletion doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ threads. `pin = true` also keeps that thread on one core, and needs `workers`.

```toml
[runtime]
workers = 8 # Single-threaded QUIC workers. Omit for the shared runtime.
workers = 8 # Single-threaded QUIC workers: "shared" (default), "auto" (one per core), or a count.
pin = false # Pin each worker to a core. Off by default.
io_uring = false # Drive them with io_uring instead of tokio.
```
Expand Down
2 changes: 1 addition & 1 deletion go/wrapper/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ func (c ClientConfig) ffi() (ffi.MoqClientConfig, error) {
// After Timeout of consecutive failures the connection gives up for good.
//
// Every field is optional: the zero value means the native default (1s, x2, 5s,
// and a 10s window), so a partial Backoff overrides only what it sets. Pass
// and no give-up), so a partial Backoff overrides only what it sets. Pass
// RetryForever as Timeout to keep retrying indefinitely.
type Backoff struct {
Initial time.Duration // delay before the first retry
Expand Down
4 changes: 2 additions & 2 deletions js/net/api/index.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,9 @@ export namespace Connection {
export type Backoff = {
healthy?: Milli;
initial?: Milli;
max?: Milli;
max?: Milli | null;
multiplier?: number;
timeout?: Milli;
timeout?: Milli | null;
};
export interface CertificateHash {
algorithm?: "sha-256";
Expand Down
4 changes: 2 additions & 2 deletions js/net/src/connection/pool.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ test("connection-specific options infer a private loop and origin", async () =>
const options = [
{ webtransport: { congestionControl: "throughput" as const } },
{ websocket: { enabled: false } },
{ delay: { timeout: Milli(0) } },
{ delay: { timeout: null } },
{ timeout: Milli(10_000) },
{ publish: origin.consume() },
{ consume: origin },
Expand Down Expand Up @@ -336,7 +336,7 @@ test("explicit share: true refuses connection-specific options", () => {
/share: false/,
);
expect(() => new Connection({ share: true, websocket: { enabled: false } })).toThrow(/share: false/);
expect(() => new Connection({ share: true, delay: { timeout: Milli(0) } })).toThrow(/share: false/);
expect(() => new Connection({ share: true, delay: { timeout: null } })).toThrow(/share: false/);
expect(() => new Connection({ share: true, timeout: Milli(10_000) })).toThrow(/share: false/);
} finally {
origin.close();
Expand Down
64 changes: 49 additions & 15 deletions js/net/src/connection/reload.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ test("an explicitly undefined delay field falls back to its default", async () =
enabled: true,
url,
websocket: { enabled: false },
delay: { initial: undefined, multiplier: undefined, max: undefined, timeout: Milli(0) },
delay: { initial: undefined, multiplier: undefined, max: undefined, timeout: null },
});
try {
await clock.advance(Nano.fromMilli(Milli(100)));
Expand Down Expand Up @@ -307,7 +307,7 @@ test("an ended auth handle gives up instead of retrying", async () => {
enabled: true,
url,
websocket: { enabled: false },
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: null },
});
const warned = spyOn(console, "warn").mockImplementation(() => {});
try {
Expand Down Expand Up @@ -351,7 +351,7 @@ test("a page hide after give-up does not retry the refused URL", async () => {
enabled: true,
url,
websocket: { enabled: false },
delay: { initial: Milli(1), multiplier: 1, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 1, max: Milli(1), timeout: null },
});

try {
Expand Down Expand Up @@ -396,7 +396,7 @@ for (const code of [SessionCode.Unauthorized, SessionCode.MalformedAuthToken, Se
enabled: true,
url,
websocket: { enabled: false },
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: null },
});
const watch = new Effect();
watch.run((effect) => {
Expand Down Expand Up @@ -465,7 +465,7 @@ test("a refused credential replaced since its dial redials with the new one", as
versions: ["moq-lite-06"],
websocket: { enabled: false },
tokens,
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: null },
});
try {
await clock.drive(waitFor(reload.established, (established) => established !== undefined));
Expand Down Expand Up @@ -493,7 +493,7 @@ test("new tokens redial a URL that stopped on refusing the old ones", async () =
versions: ["moq-lite-06"],
websocket: { enabled: false },
tokens,
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 2, max: Milli(1), timeout: null },
});
try {
await clock.drive(waitFor(reload.error, (error) => error !== undefined));
Expand Down Expand Up @@ -536,7 +536,7 @@ test("an unauthorized session close during setup stops retrying", async () => {
url,
websocket: { enabled: false },
// Unlimited retries prove that `closed` settles only because the rejection is terminal.
delay: { initial: Milli(1), multiplier: 1, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 1, max: Milli(1), timeout: null },
});

try {
Expand Down Expand Up @@ -587,7 +587,7 @@ test("a setup stream reset is not mistaken for an unauthorized session", async (
websocket: { enabled: false },
// Unlimited retries, so a second attempt can only happen because the first failure
// was treated as retryable rather than terminal.
delay: { initial: Milli(1), multiplier: 1, max: Milli(1), timeout: Milli(0) },
delay: { initial: Milli(1), multiplier: 1, max: Milli(1), timeout: null },
});

try {
Expand Down Expand Up @@ -1293,6 +1293,8 @@ const windows: [string, ReloadDelay, number, number][] = [
["an attempt under way finishes past the window", { initial: Milli(10), timeout: Milli(100) }, 110, 3],
// The first failure at 30 ms leaves 30 ms, less than the 50 ms wait, so it gives up at the deadline.
["a wait that reaches the deadline gives up there", { initial: Milli(50), timeout: Milli(60) }, 60, 1],
// Zero is a real bound, not "unlimited": the first failure at 30 ms gives up.
["a zero window gives up when the first attempt fails", { initial: Milli(10), timeout: Milli(0) }, 30, 1],
];
for (const [name, delay, gaveUp, dials] of windows) {
test(`retry window: ${name}`, async () => {
Expand Down Expand Up @@ -1325,6 +1327,40 @@ for (const [name, delay, gaveUp, dials] of windows) {
});
}

// The window is checked again when the backoff ends: a timer that fires late, as after the
// device slept, gives up instead of dialing past the window.
test("retry window: a backoff that fires past the window gives up", async () => {
const clock = new Manual();
const installed = clock.install();
const random = spyOn(Math, "random").mockReturnValue(1);
const warned = spyOn(console, "warn").mockImplementation(() => {});
const url = new URL("https://relay.example/room");
const plan = dialPlan(() => "hang", url);
const reload = new Reload({
url,
websocket: { enabled: false },
timeout: Milli(30),
delay: { initial: Milli(10), multiplier: 1, max: Milli(1000), timeout: Milli(100) },
});
try {
await waitUntil(() => plan.dials() === 1);
// The first attempt times out at 30 ms and schedules the next dial for 40 ms.
await clock.advance(Nano.fromMilli(Milli(30)));
for (let i = 0; i < 3; i++) await settle();
expect(reload.error.peek()).toBeUndefined();
clock.jump(Nano.fromMilli(Milli(200)));
for (let i = 0; i < 3; i++) await settle();
expect(reload.error.peek()).toBeInstanceOf(ConnectTimeout);
expect(plan.dials()).toBe(1);
} finally {
reload.close();
plan.restore();
warned.mockRestore();
random.mockRestore();
installed.close();
}
});

test("an invalid timeout is refused when the loop is built, not by every attempt", () => {
for (const timeout of [-1, Number.NaN, Number.POSITIVE_INFINITY]) {
expect(() => new Reload({ url: new URL("https://example.com/"), timeout: timeout as Milli })).toThrow(
Expand All @@ -1339,12 +1375,10 @@ const backoffs: [string, ReloadDelay, number[]][] = [
// Past `initial` but short of the default bar, so each end is a failed attempt.
["a session short of the default healthy bar escalates", {}, [10, 20, 40]],
["a session past the healthy bar resets", { healthy: Milli(50) }, [10, 10, 10]],
[
"a zero max leaves the delay uncapped",
{ initial: Milli(1000), multiplier: 4, max: Milli(0) },
[1000, 4000, 16000],
],
["a null max leaves the delay uncapped", { initial: Milli(1000), multiplier: 4, max: null }, [1000, 4000, 16000]],
["a max below initial caps the first delay too", { max: Milli(5) }, [5, 5, 5]],
// Zero is a real cap, not "uncapped".
["a zero max retries at once", { max: Milli(0) }, [0, 0, 0]],
["a zero multiplier keeps the delay constant", { multiplier: 0 }, [10, 10, 10]],
["a fractional multiplier below 1 keeps the delay constant", { multiplier: 0.5 }, [10, 10, 10]],
];
Expand Down Expand Up @@ -1406,7 +1440,7 @@ test("backoff: an uncapped delay saturates instead of overflowing", async () =>
const reload = new Reload({
url,
websocket: { enabled: false },
delay: { initial: Milli(10), multiplier: 1e300, max: Milli(0) },
delay: { initial: Milli(10), multiplier: 1e300, max: null },
});
try {
for (let session = 0; session < 2; session++) {
Expand All @@ -1432,7 +1466,7 @@ test("a delay field with no meaning is refused when the loop is built", () => {
const invalid: ReloadDelay[] = [
// Zero would count every session as healthy.
{ healthy: Milli(0) },
// Zero `max` is uncapped and zero `timeout` unlimited, so infinity has no spelling here.
// `null` spells an uncapped `max` and an unlimited `timeout`, so infinity has no meaning here.
...bad.flatMap((value) => [{ initial: value }, { max: value }, { healthy: value }, { timeout: value }]),
{ multiplier: Number.NaN },
{ multiplier: Number.POSITIVE_INFINITY },
Expand Down
47 changes: 29 additions & 18 deletions js/net/src/connection/reload.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,11 @@ export type ReloadDelay = {
/** The multiplier for the delay after each failure (default: 2; below 1 keeps it constant). */
multiplier?: number;

/** The maximum delay (default: 5000ms; zero leaves it uncapped). A cap below `initial` caps the first delay too. */
max?: Milli;
/**
* The maximum delay (default: 5000ms; `null` leaves it uncapped; zero retries at once). A cap
* below `initial` caps the first delay too.
*/
max?: Milli | null;

/**
* How long a session must stay up to count as healthy (default: 1000ms; must be positive).
Expand All @@ -64,13 +67,13 @@ export type ReloadDelay = {

/**
* Maximum total time to spend retrying the current URL before giving up
* (default: 0, unlimited). Counts from the first dial, and restarts after a healthy session
* (default: `null`, unlimited; zero gives up when the first attempt fails). Counts from the first dial, and restarts after a healthy session
* outside a redirect loop, a URL change, or a disable/re-enable. It decides whether another
* attempt starts and never cuts one that is running, so giving up can take up to one attempt
* longer than the window; {@link ConnectProps.timeout} bounds each attempt. A peer that
* restarts, draining each session with a GOAWAY naming no URI, restarts it too.
*/
timeout?: Milli;
timeout?: Milli | null;
Comment thread
kixelated marked this conversation as resolved.
Resolved
};

/**
Expand Down Expand Up @@ -126,7 +129,7 @@ const DEFAULT_DELAY: Required<ReloadDelay> = {
multiplier: 2,
max: Milli(5000),
healthy: Milli(1000),
timeout: Milli(0),
timeout: null,
};

/**
Expand All @@ -138,20 +141,22 @@ const DEFAULT_DELAY: Required<ReloadDelay> = {
*/
function pacing(delay: ReloadDelay) {
const initial = Milli(delay.initial ?? DEFAULT_DELAY.initial);
const max = Milli(delay.max ?? DEFAULT_DELAY.max);
const cap = max > 0 ? max : undefined;
// `null` means no bound, so only `undefined` takes the default.
const max = delay.max === undefined ? DEFAULT_DELAY.max : delay.max;
const cap = max === null ? null : Milli(max);
const multiplier = delay.multiplier ?? DEFAULT_DELAY.multiplier;
if (!Number.isFinite(multiplier)) throw new RangeError(`invalid delay.multiplier: ${multiplier}`);
// Zero would count every session as healthy, resetting the retry window forever under a
// peer that drops each one on sight.
const healthy = Milli(delay.healthy ?? DEFAULT_DELAY.healthy);
if (healthy === 0) throw new RangeError("delay.healthy must be positive");
const timeout = Milli(delay.timeout ?? DEFAULT_DELAY.timeout);
const window = delay.timeout === undefined ? DEFAULT_DELAY.timeout : delay.timeout;
const timeout = window === null ? null : Milli(window);
// A multiplier below 1 means no growth, not a delay that shrinks to nothing.
const growth = Math.max(multiplier, 1);

return {
initial: cap === undefined ? initial : Milli(Math.min(initial, cap)),
initial: cap === null ? initial : Milli(Math.min(initial, cap)),
healthy,
timeout,
// How long a session must last to start a new run of redirects. One that outlived the
Expand Down Expand Up @@ -754,7 +759,7 @@ export class Reload {
// When a retry window starting `at` expires.
#window(at: Instant): number {
const { timeout } = this.#pacing();
return timeout > 0 ? Instant.add(at, Nano.fromMilli(timeout)) : Number.POSITIVE_INFINITY;
return timeout === null ? Number.POSITIVE_INFINITY : Instant.add(at, Nano.fromMilli(timeout));
}

/**
Expand Down Expand Up @@ -830,21 +835,27 @@ export class Reload {

// No attempt starts once the window has passed: a wait that reaches the deadline gives up
// there instead of dialing.
const left = this.#deadline - at;
const deadline = this.#deadline;
const left = deadline - at;
// A graceful close has no error, so report the timeout itself.
const failure = cause === undefined ? new Error("reconnect timed out") : error(cause);
const giveUp = () => {
console.warn("reconnect timed out");
this.#giveUp(failure);
};
if (wait >= left) {
// A graceful close has no error, so report the timeout itself.
const failure = cause === undefined ? new Error("reconnect timed out") : error(cause);
const giveUp = () => {
console.warn("reconnect timed out");
this.#giveUp(failure);
};
if (left > 0) effect.timer(giveUp, Nano(left));
else giveUp();
return;
}

const tick = this.#tick.peek() + 1;
effect.timer(() => this.#tick.update((prev) => Math.max(prev, tick)), Nano(wait));
effect.timer(() => {
// Checked again when the wait ends: a timer that fires late, as after the device
// slept, must not dial past the window.
if (now() >= deadline) return giveUp();
this.#tick.update((prev) => Math.max(prev, tick));
}, Nano(wait));
}

#giveUp(cause: Error): void {
Expand Down
2 changes: 1 addition & 1 deletion py/moq-rs/moq/_records.py
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,7 @@ class Backoff:
The delay starts at ``initial``, multiplies by ``multiplier`` after each failed
attempt, and caps at ``max``. After ``timeout`` of consecutive failures the
connection gives up; ``timedelta(0)`` retries forever. ``None`` keeps each
default: 1s, x2, 5s, and a 10s window.
default: 1s, x2, 5s, and no give-up.
"""

initial: timedelta | None = None
Expand Down
7 changes: 5 additions & 2 deletions py/moq-rs/tests/clock.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class Clock:

def __init__(self) -> None:
self._advance = _symbol("moq_ffi_test_clock_advance", (ctypes.c_uint64,), ctypes.c_bool)
self._now = _symbol("moq_ffi_test_clock_now", (), ctypes.c_uint64)
self._now = _symbol("moq_ffi_test_clock_now", (ctypes.POINTER(ctypes.c_uint64),), ctypes.c_bool)
if not _symbol("moq_ffi_test_clock_start", (), ctypes.c_bool)():
raise RuntimeError("the native runtime already started on the host clock")

Expand All @@ -45,4 +45,7 @@ def advance(self, seconds: float) -> None:

def time(self) -> float:
"""Seconds elapsed since the clock started."""
return self._now() / 1_000_000_000
nanos = ctypes.c_uint64()
if not self._now(ctypes.byref(nanos)):
raise RuntimeError("the native test clock is not running")
return nanos.value / 1_000_000_000
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,6 @@ quests that require them.
- [Stream credit by priority](/quest/m1/stream-credit/README.md) - every stream opens through one priority entry point, and a higher-priority group takes over or resets a lower one when credit runs out (held in m1 by an open claim)
- [Finalize moq-lite-07](/quest/m2/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it
- [Unbounded groups](/quest/m2/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh
- [No sentinel values in public API](/quest/m2/no-sentinels.md) - receive windows, cache durations, reconnect timeouts, and the FFI test clock use `Option` instead of a huge value or zero
- [moq-video codecs split submission from receive](/quest/m2/video-codec-receive.md) - video encoders and decoders submit, then `receive` one output, the shape moq-audio took, so no call allocates a `Vec` of outputs
- [Connection state carries each phase's data](/quest/m2/connection-state-phases.md) - moq-tokio's `Observation`, moq-ffi's `MoqConnectionState`, and moq-c's `moq_connection_state` each hold one phase per value, so no field is readable in a phase where it means nothing
- [moq-video takes devices, not environment variables](/quest/m2/video-devices.md) - VA-API and V4L2 device paths are config fields, and `moq_video::ENV` never ships
Expand Down
35 changes: 0 additions & 35 deletions quest/m2/no-sentinels.md

This file was deleted.

Loading
Loading