From 586728195035eb7f2d2e2a3afcb135db6069bb25 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 13:09:26 -0700 Subject: [PATCH 1/3] quest: plan the #5446 and #5447 follow-ups Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/ffi-panic-supervision.md | 33 +++++++++++++++++++ quest/m1/test-flakes-2/README.md | 1 + quest/m1/test-flakes-2/auth-refresh-expiry.md | 23 +++++++++++++ quest/m2/README.md | 1 + quest/m2/task-supervision.md | 25 ++++++++++++++ 6 files changed, 84 insertions(+) create mode 100644 quest/m1/ffi-panic-supervision.md create mode 100644 quest/m1/test-flakes-2/auth-refresh-expiry.md create mode 100644 quest/m2/task-supervision.md diff --git a/quest/m1/README.md b/quest/m1/README.md index a9671b9a20d..01a985a77c7 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -86,6 +86,7 @@ above them. - [Slow source fails](/quest/m1/slow-source.md) - an export whose source runs further off real time than the output may follow fails promptly naming the rate, instead of dropping frames for hours - [Same-instance TS resume](/quest/m1/ts-same-instance-resume.md) - prove valid continuation and fix the unexpected final exporter failure without weakening identity - [Wire compat media](/quest/m1/wire-compat-media.md) - wire-compat always runs same-version media cells through the compat reader, so planned breaks can't hide its regressions +- [Binding panic supervision](/quest/m1/ffi-panic-supervision.md) - a panic in any moq-c or moq-ffi background task reaches the host as a panic, never a lost callback or Cancel - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away diff --git a/quest/m1/ffi-panic-supervision.md b/quest/m1/ffi-panic-supervision.md new file mode 100644 index 00000000000..1d2ae06a98c --- /dev/null +++ b/quest/m1/ffi-panic-supervision.md @@ -0,0 +1,33 @@ +# [M] A panic in a binding's background task reaches the host + +## Goal + +A panic in any task moq-c or moq-ffi spawns (consume, publish, origin, audio, +video, server, and `ffi::spawn`'s tasks, including the origin driver) +reaches the host as a panic error through that object's final callback or +the uniffi call, never as a lost callback or `Cancel`. A moq-tokio driver +panic under a binding reports `Panic`, not `Cancel`. + +## Plan + +Follow-up from [#5447](https://github.com/moq-dev/moq/pull/5447), which made +a panic close only its session and supervised moq-c's session tasks. The +other moq-c tasks still drop a panic, so the host's final callback never +fires and a C host waits forever; moq-ffi's `ffi::spawn` tasks do the same +for uniffi callers. + +Decided 2026-10-13 (maintainer): m1, one quest for both bindings. Supervise +each task's `JoinHandle` the way #5447 did for moq-c sessions, reusing +`moq_tokio::panic`. Carry a driver's panic through as a panic rather than +the `Cancel` its dropped channel produces today. Add a regression per +binding that injects a panic into one background task and asserts the +host sees `MOQ_ERROR_PANIC` (moq-c) or the binding's panic error (moq-ffi), +while another object keeps working. Also add the regression #5447 left out: +a moq-tokio accept loop that panics ends its listener. + +Public API: none expected, beyond the panic error reaching more callbacks; +report any new error variant through the Cross-Package Sync table. Wire: none. + +## Related + +- [Long-lived task supervision](/quest/m2/task-supervision.md) - the remaining process-wide tasks diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 4e8ad3bf150..3c732858bef 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -35,6 +35,7 @@ Public API: none. Wire: none. - [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session - [QUIC-wins race](/quest/m1/test-flakes-2/quic-winning-race.md) - moq-tokio's QUIC-wins test sets up QUIC's win instead of racing the WebSocket head start - [Handshake ACK](/quest/m1/test-flakes-2/handshake-ack.md) - the impaired cluster drill's connect survives a lost handshake ACK +- [Auth refresh expiry](/quest/m1/test-flakes-2/auth-refresh-expiry.md) - the auth refresh outage test measures the grant's life from its admission ## Related diff --git a/quest/m1/test-flakes-2/auth-refresh-expiry.md b/quest/m1/test-flakes-2/auth-refresh-expiry.md new file mode 100644 index 00000000000..7155363716a --- /dev/null +++ b/quest/m1/test-flakes-2/auth-refresh-expiry.md @@ -0,0 +1,23 @@ +# [XS] Auth refresh measures a grant's life from its admission + +## Goal + +moq-relay's `an_outage_refuses_transiently_and_never_extends_the_expiry` +(`rs/moq-relay/src/auth/refresh.rs`) passes under a loaded `just check`, +fixed at its cause. + +## Plan + +Seen 2026-10-13 on #5446's first `just check`: the session lived 28.862 s +against the test's 29 s floor, then the test passed alone and on rerun. + +The test runs on a paused clock but takes `start` after `admission()` and +`sessions()`. The grant's 30 s starts at admission, and the paused clock +auto-advances whenever the runtime idles in between, so under load `start` +lands late and the measured life comes up short. Measure from the instant +the grant is admitted (take `start` before `admission()`, or read the +grant's own start), keeping the same bounds. Find what idles the runtime in +`sessions()`; if a real socket is involved, apply the epic's rules rather +than widening the window. + +Public API: none. Wire: none. diff --git a/quest/m2/README.md b/quest/m2/README.md index 9f0f8f507ee..eb21da015c4 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -173,3 +173,4 @@ ranked just below deterministic TS legs, which it requires. - [Release execution](/quest/m2/release/README.md) - publication, installed-artifact proof, cleanup, and downstream adoption after m1 - [IETF driver double poll](/quest/m2/ietf-driver-double-poll.md) - a fired deadline costs a moq-transport session's driver one poll, as #5393 makes moq-lite's - [Play resume](/quest/m2/play-resume.md) - `moq-cli play` plays out pictures buffered before a declared end when new media arrives quickly +- [Task supervision](/quest/m2/task-supervision.md) - the auth decider loop, stats publisher, and moq-tokio's token and WebSocket internals are supervised like #5447's tasks diff --git a/quest/m2/task-supervision.md b/quest/m2/task-supervision.md new file mode 100644 index 00000000000..d9498b3dfa0 --- /dev/null +++ b/quest/m2/task-supervision.md @@ -0,0 +1,25 @@ +# [S] The remaining long-lived tasks are supervised + +## Goal + +A panic in the relay's auth decider loop, the stats publisher, or +moq-tokio's token and WebSocket internals either ends only what depends on +it or exits the process, by the rule [#5447](https://github.com/moq-dev/moq/pull/5447) +set: a per-connection task's panic is isolated and counted, a long-lived +task's panic propagates out of `Relay::run` or the CLI's main. None is +swallowed. + +## Plan + +Follow-up from #5447, which supervised the session drivers and the relay's +critical tasks but left these. Decided 2026-10-13 (maintainer): m2, low +priority, since each is a single process-wide task. Classify each task as +per-connection or long-lived, observe its `JoinHandle` with +`moq_tokio::panic`, and add a regression that injects a panic into one task +of each class. + +Public API: none expected. Wire: none. + +## Related + +- [Binding panic supervision](/quest/m1/ffi-panic-supervision.md) - the moq-c and moq-ffi tasks, ahead of this From efdad979e1b3fe9f4f8de7aca9a81cb576359238 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 13:14:24 -0700 Subject: [PATCH 2/3] quest: binding panic supervision audits every moq-ffi consumer Co-Authored-By: Claude Opus 5.5 --- quest/m1/ffi-panic-supervision.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/quest/m1/ffi-panic-supervision.md b/quest/m1/ffi-panic-supervision.md index 1d2ae06a98c..aa062ba1e80 100644 --- a/quest/m1/ffi-panic-supervision.md +++ b/quest/m1/ffi-panic-supervision.md @@ -25,8 +25,10 @@ host sees `MOQ_ERROR_PANIC` (moq-c) or the binding's panic error (moq-ffi), while another object keeps working. Also add the regression #5447 left out: a moq-tokio accept loop that panics ends its listener. -Public API: none expected, beyond the panic error reaching more callbacks; -report any new error variant through the Cross-Package Sync table. Wire: none. +Public API: none expected, beyond the panic error reaching more callbacks. +That is a host-visible change even without a new error variant, so walk the +`rs/moq-ffi` row of the Cross-Package Sync table: every wrapper and +`doc/lib` page surfaces the panic error. Wire: none. ## Related From 17e9b169357ca406220e903eca26982f66abd8aa Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 13:19:21 -0700 Subject: [PATCH 3/3] quest: auth refresh flake is a wall-to-Tokio clock mismatch Co-Authored-By: Claude Opus 5.5 --- quest/m1/test-flakes-2/README.md | 2 +- quest/m1/test-flakes-2/auth-refresh-expiry.md | 22 +++++++++++-------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 3c732858bef..5f8ee18eeb5 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -35,7 +35,7 @@ Public API: none. Wire: none. - [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session - [QUIC-wins race](/quest/m1/test-flakes-2/quic-winning-race.md) - moq-tokio's QUIC-wins test sets up QUIC's win instead of racing the WebSocket head start - [Handshake ACK](/quest/m1/test-flakes-2/handshake-ack.md) - the impaired cluster drill's connect survives a lost handshake ACK -- [Auth refresh expiry](/quest/m1/test-flakes-2/auth-refresh-expiry.md) - the auth refresh outage test measures the grant's life from its admission +- [Auth refresh expiry](/quest/m1/test-flakes-2/auth-refresh-expiry.md) - the auth refresh outage test measures the grant's life on one clock ## Related diff --git a/quest/m1/test-flakes-2/auth-refresh-expiry.md b/quest/m1/test-flakes-2/auth-refresh-expiry.md index 7155363716a..4f73db43611 100644 --- a/quest/m1/test-flakes-2/auth-refresh-expiry.md +++ b/quest/m1/test-flakes-2/auth-refresh-expiry.md @@ -1,4 +1,4 @@ -# [XS] Auth refresh measures a grant's life from its admission +# [XS] Auth refresh measures a grant's life on one clock ## Goal @@ -11,13 +11,17 @@ fixed at its cause. Seen 2026-10-13 on #5446's first `just check`: the session lived 28.862 s against the test's 29 s floor, then the test passed alone and on rerun. -The test runs on a paused clock but takes `start` after `admission()` and -`sessions()`. The grant's 30 s starts at admission, and the paused clock -auto-advances whenever the runtime idles in between, so under load `start` -lands late and the measured life comes up short. Measure from the instant -the grant is admitted (take `start` before `admission()`, or read the -grant's own start), keeping the same bounds. Find what idles the runtime in -`sessions()`; if a real socket is involved, apply the epic's rules rather -than widening the window. +The test's `grant()` stamps the expiry with `SystemTime::now() + 30s`, but +`Grant::deadline()` (`rs/moq-auth/src/grant.rs`) converts the remaining wall +time onto Tokio's paused clock only when the lease snapshots it. Wall time +keeps running under load while the paused clock does not, so a 1.14 s +scheduling delay before the snapshot leaves 28.86 s on the Tokio clock. +Moving `start` earlier does not fix that; `sessions()` uses an in-memory +duplex, so no real socket is involved. + +Measure against one clock: assert the session ends at the deadline the lease +captured, or build the grant's expiry from the moment that deadline is +taken, keeping the bounds. Reproduce first by delaying wall time between +`grant()` and admission. Public API: none. Wire: none.