diff --git a/quest/m0/README.md b/quest/m0/README.md index e45ab629105..0ac02930d59 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -62,6 +62,11 @@ released wire-compat step aborts on planned breaks the last release made stale. Both stay in m0 (maintainer, 2026-10-13) and rank with TS duration fidelity: a red Interop hides real regressions. +Crashes (2026-10-13, #5407): a publisher on a host whose wall clock reads +before 2020 panics, and `panic = "abort"` turns any library panic into a +dead host app. Both joined m0 (maintainer): fix the unset clock at its +source, then unwind so a panic closes only its connection. + Nightly on main (triaged 2026-10-13): two build breaks from 2026-10-11 get quests here. The nixpkgs bump broke the Linux zigbuild release builds, and fat LTO in the release profile broke the fuzz build. Audio quality has been @@ -78,6 +83,8 @@ have fixes on `main`: macOS tests (#5335), drill-sensitivity (#5338), loom - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.5, before Seattle, then draft-20+ FETCH - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - Interop's TS compliance captures the whole round-tripped stream again, fixing a regression that turns Interop red on main - [Upgrade restarts epochless broadcasts](/quest/m0/upgrade-unroutable.md) - a WebSocket-to-QUIC upgrade retires epochless handles only once their `Restart` is delivered, and first-party consumers follow it, fixing Interop's `unroutable` rounds +- [Unset host clock](/quest/m0/wall-clock-omit.md) - a publisher whose wall clock reads before 2020 publishes without a catalog clock instead of panicking +- [Panic isolation](/quest/m0/panic-isolation.md) - unwind, ignore lock poison, and supervise every task, so a panic closes only its connection or exits on a long-lived task - [Wire compat on main](/quest/m0/wire-compat-green.md) - the nightly released-wire step runs again: planned breaks re-measured against the new releases, and a released-API client for the released JS lanes - [Release builds link](/quest/m0/release-link.md) - the Linux zigbuild release binaries and moq-gst packages link again after the 2026-10-11 nixpkgs bump, so the next release cut can publish - [Fuzz builds under LTO](/quest/m0/fuzz-lto.md) - the nightly fuzz targets build again despite the release profile's fat LTO diff --git a/quest/m0/panic-isolation.md b/quest/m0/panic-isolation.md new file mode 100644 index 00000000000..37530c25e67 --- /dev/null +++ b/quest/m0/panic-isolation.md @@ -0,0 +1,70 @@ +# [L] A panic closes only its connection + +## Goal + +A panic inside one connection or session closes that connection and leaves +the process running: the relay, the CLI, and every FFI host (a library panic +reaches the host as `MOQ_ERROR_PANIC` or a binding exception). A panic in a +long-lived process task still exits the process, so nothing becomes a silent +zombie. + +## Plan + +Found from [#5407](https://github.com/moq-dev/moq/issues/5407): the +workspace's `panic = "abort"` (dev and release profiles, added in #804) makes +moq-c's `catch_unwind` (`rs/moq-c/src/ffi.rs`) and uniffi's `rust_call` +dead in shipped builds, so any library panic kills the host app. + +#804 chose abort because a rustls panic in a detached HTTPS task left the +relay up but serving nothing. That bug is gone (the provider is installed at +startup and the HTTPS config is built before binding), but the failure mode +returns for any detached long-lived task under unwind. + +Decided 2026-10-13 (maintainer): + +- Unwind workspace-wide: delete `panic = "abort"` from both profiles. Rejected: + a separate `ffi` profile for binding builds, since relay connections need + isolation too. wasm needs no override (stable std aborts there anyway). + Measure the moq-ffi staticlib against the Go-mirror size ceiling noted in + `Cargo.toml`. +- Ignore poison. kio's `Lock::lock` (`rs/kio/src/lock.rs`) panics on a + poisoned mutex, so one panic under a shared origin/broadcast/track lock + would cascade to every session, or double-panic into an abort in + `Producer::drop`. Use `unwrap_or_else(PoisonError::into_inner)` there + (loom-compatible) and at the cross-session std locks (moq-net cache, stats, + resume, frame, time; the relay; moq-c `STATE`; moq-ffi polls). Per-session + locks may keep panicking. Rejected: poison closing the channel (more kio + code), and parking_lot (no loom). +- Supervise every task. Every spawned task's `JoinHandle` is observed: + - a per-connection task's panic is logged (`tracing::error!` with the + connection span, a panic counter on `/metrics`) and ends only that + connection; dropping its driver already sends CONNECTION_CLOSE and + retracts its broadcasts; + - a long-lived task's panic propagates out of `Relay::run` (or the CLI's + main) and exits the process: the origin driver + (`moq_tokio::origin::spawn_config`), cache governor, HTTPS cert reload, + and the noq endpoint driver; + - a cluster dial task is per peer, so it counts as a connection, and its + peer is re-dialed instead of swallowed by `join_next`. + Rejected: a panic hook that aborts unless a task-local "isolated" flag is + set (catches tasks nobody listed, but hides policy in a hook), and + log-only (reintroduces #804). +- moq-c: await each session task's inner handle and report `MOQ_ERROR_PANIC` + through its terminal callback, so the entry doesn't leak. +- Regression test: a fake transport whose poll panics. Only that session + closes, its broadcasts leave the origin, and a second session keeps + working. +- Regression test for the fatal side: inject a deterministic panic into one + named long-lived task (such as the cache governor) and assert + `Relay::run` returns with the panic instead of logging and continuing. +- Update comments and docs that assume abort inline, including + `doc/lib/cpp/index.md`, and document the panic counter's name and meaning + in `doc/bin/relay/http.md`. + +Public API: none in Rust. Operational: a new relay `/metrics` panic counter. +Wire: none. + +## Related + +- [Unset host clock](/quest/m0/wall-clock-omit.md) - removes the panic that surfaced this +- [Fuzz builds under LTO](/quest/m0/fuzz-lto.md) - also edits the release profile diff --git a/quest/m0/release-22/README.md b/quest/m0/release-22/README.md index e6cf8827ce5..aec05f4475e 100644 --- a/quest/m0/release-22/README.md +++ b/quest/m0/release-22/README.md @@ -7,6 +7,8 @@ on 2026-10-12 with draft-22 media working both ways: the draft-22 LOCATION_FILTER form (#4847), a clear FIRST_OBJECT at object 0 accepted (#5027), and moq-noq 1.3.5. A draft-22 peer (imquic main 925e3d83) can subscribe with any of the six filters and publish through the relay. +The next 0.17 patch also fixes #5407's unset-clock crash and #5413's lost +quiet edge. ## Plan @@ -29,10 +31,16 @@ Decided 2026-10-08: The backports shipped in moq-relay 0.17.3 (#5119) and `@moq/net` 0.4.3 (refreshed 2026-10-12). Draft-22 interop on 2026-10-12 then found FETCH still -refused on draft-20+ sessions. This README's remaining work: once the child -lands, cut the next 0.17 patch and `@moq/net` with it (`/bump`, then the -release-plz PR), and tell the testers. +refused on draft-20+ sessions. This README's remaining work: once the children +land, cut the next 0.17 patch, `@moq/net`, and `@moq/publish` with them (`/bump`, then the +release-plz PR), and tell the testers and the reporters of #5407 and #5413. + +Decided 2026-10-13: the same patch also carries two crash and availability +fixes reported against the release line, an unset host clock and a quiet +track's latest group lost after rejoin. ## Required - [Draft-20+ FETCH](/quest/m0/release-22/fetch.md) - backport #4971 so a draft-22 FETCH is served, not refused +- [Unset host clock](/quest/m0/release-22/wall-clock-omit.md) - a publisher whose wall clock reads before 2020 omits the catalog clock instead of aborting the host +- [Quiet latest group](/quest/m0/release-22/quiet-edge.md) - a late subscriber to a quiet track gets its latest group after rejoin, cache expiry, or takeover diff --git a/quest/m0/release-22/quiet-edge.md b/quest/m0/release-22/quiet-edge.md new file mode 100644 index 00000000000..d294772a2f9 --- /dev/null +++ b/quest/m0/release-22/quiet-edge.md @@ -0,0 +1,42 @@ +# [S] A quiet track's latest group survives rejoin on the 0.17 release line + +## Goal + +On a 0.17.x moq-relay, a late subscriber to a live but quiet track (such as +a catalog published once) receives its latest group after earlier readers +left and rejoined and the cache expiry passed, including across a publisher +takeover. + +## Plan + +Reported in [#5413](https://github.com/moq-dev/moq/issues/5413) against +moq-relay 0.17.3 / moq-net 0.3.11. Verified 2026-10-13: the reporter's +`rejoined_quiet_catalog_survives_cache_expiry` test fails on the +`moq-net-v0.3.11` tag and its port passes on `main`, where #4741 deleted the +warm-copy code (`WarmGroup`, `WarmCopy`, `TrackIo`) and readers read the +serving route's copy directly. + +Root cause on `release` (`rs/moq-net/src/model/origin.rs`): the cache exempts +only a live track's latest group from expiry. On re-splice, dropping the +`WarmCopy` finishes its track, so its latest group is demoted while +`io.head` still holds it as the live edge. The new upstream copy resumes +after it, so nothing re-sends it, and the pool expires it. `TrackIo::retire` +returns early on a finished held group, so takeover loses it the same way. + +Decided 2026-10-13 (maintainer): + +- Backport to `release` only; `main` gets [a regression test](/quest/m1/quiet-edge-test.md). +- Keep the `WarmCopy` alive and unfinished until the new copy delivers a + group newer than its edge, so the one existing rule (a live track's + latest is protected) covers it. Give `retire` the same treatment for a + finished held group. Preferred over the reporter's patch (`WarmGroup` + holding a `cache::Protection` released by `Step::Held`), which adds a + second protection path to keep in sync. Fall back to it only if keeping + the copy alive proves invasive. +- Land the reporter's test (Lite05/06, superseded group still expires) and a + takeover variant. +- Ships with the next 0.17 patch alongside [draft-20+ FETCH](/quest/m0/release-22/fetch.md). + +## Closes + +- [#5413](https://github.com/moq-dev/moq/issues/5413) - quiet catalog lost after reader rejoin and cache expiry diff --git a/quest/m0/release-22/wall-clock-omit.md b/quest/m0/release-22/wall-clock-omit.md new file mode 100644 index 00000000000..647859c4ad0 --- /dev/null +++ b/quest/m0/release-22/wall-clock-omit.md @@ -0,0 +1,34 @@ +# [XS] An unset host clock publishes on the 0.17 release line + +## Goal + +The next 0.17 patch (libmoq 0.6.x, moq-ffi 0.4.x, `@moq/publish`) carries +the unset-clock fix, so a publisher on a host whose wall clock reads before +2020 publishes without a root `clock` instead of aborting the host. + +## Plan + +Decided 2026-10-13 (maintainer): a crash qualifies for backport +(CONTRIBUTING). Port the `main` fix onto `release` as its own PR, including +its tests and the `@moq/publish` mirror. No wire change, so the release line +can take it in a patch. + +Decided 2026-10-13 (maintainer, from review on #5418): keep the public Rust +API. `main` makes `Clock::wall()` return `Option`, which a patch can't +(release is moq-mux 0.10.x). On `release`, store the mapping as a private +`Option`, which `Producer::new` uses internally, and port `main`'s +whole-clock comparison in `Listing::stamp` with its withheld-catalog test, +since two wall-less clocks compare equal by mapping alone. Keep +`wall() -> hang::catalog::Clock`, which panics only when a caller asks an +unset host for its mapping; that host already crashed at construction +before. Rejected: a breaking moq-mux 0.11.0 cut from `release`, which takes +the number `main`'s next breaking release needs, and an additive public +accessor, the shim pattern the repo avoids. + +## Required + +- [Unset host clock](/quest/m0/wall-clock-omit.md) - lands the fix on `main` first + +## Closes + +- [#5407](https://github.com/moq-dev/moq/issues/5407) - moq-mux panics when the system clock is before 2020 diff --git a/quest/m0/wall-clock-omit.md b/quest/m0/wall-clock-omit.md new file mode 100644 index 00000000000..bf213118f63 --- /dev/null +++ b/quest/m0/wall-clock-omit.md @@ -0,0 +1,65 @@ +# [S] An unset host clock publishes without a wall mapping + +## Goal + +A `moq_mux::catalog::Producer` on a host whose wall clock reads before 2020, +such as an embedded or Android device booted before NTP sync, publishes +normally with no root `clock` in its catalog, instead of panicking in +`Clock::new()` and aborting the host process. Rust and `@moq/publish` alike. + +Non-goals: moving the epoch, and panic isolation, which +[its own quest](/quest/m0/panic-isolation.md) owns. + +## Plan + +Reported in [#5407](https://github.com/moq-dev/moq/issues/5407) from an +Android device (libmoq 0.6.13). Verified on `main` 2026-10-13: +`Producer::new` (`rs/moq-mux/src/catalog/producer.rs`) falls back to +`Clock::default()`, whose `Clock::new()` `expect`s `arrival(LEAD)`, and +`wall_clock()` refuses a time before the 2020 epoch. Every production +publisher without `Config::with_clock` reaches it: moq-ffi (every binding), +moq-c, moq-cli, moq-gst, moq-rtmp, moq-srt, moq-rtc, moq-hls import, +moq-transcode. `js/publish/src/catalog.ts` `pageClock()` throws the same way +through `u53`. + +Decided 2026-10-13 (maintainer): + +- Keep the 2020 epoch. The root `clock` is optional on the wire (hang draft, + Rust `Option`, zod `optional`), and only feeds HLS `PROGRAM-DATE-TIME` and + DASH `availabilityStartTime`. A reading before 2020 reliably means the + clock is unset. No clock is better for consumers than a false one: DASH + falls back to the relay's anchor, while a 1970 AST is unplayable. + Rejected: moving to a 1970 epoch under a renamed root key (hang-04), which + buys consistency at the cost of a wire rename and a mapping consumers + can't use. Rejected: failing `Producer::new`, which stops an offline + device from ever publishing, though media delivery needs no wall. +- `moq_mux::Clock`'s wall mapping becomes optional and `Clock::wall()` + returns `Option`. `Clock::new()` stays infallible: the monotonic part + needs no wall. `Producer::new` keeps its signature and sets + `catalog.clock = clock.wall()`. +- Fail loud where a wall is actually needed: `reading(wall)`, and an + importer placing a source wall on a wall-less clock, return an error. +- Clock identity can't be the wall mapping. `Listing::stamp` + (`rs/moq-mux/src/catalog/data.rs`) detects an importer re-anchoring the + clock by comparing `wall()`, which is `None` for both clocks on an unset + host, so it would measure jitter, delay, and bitrate across unrelated + timelines. Compare whole clocks (monotonic anchor, reading, and wall). + Cover a withheld catalog plus a data producer and an importer in the test. + Found in review on #5418. +- Mirror in `pageClock()`: omit `clock` instead of throwing. +- Interplay with [catalog clock adopt](/quest/m1/catalog-clock-adopt.md), + which assumes `wall()` is always present: whichever lands second + reconciles. An absent wall matches an absent catalog clock. +- Test through a private `now` seam (until [Controlled time](/quest/m1/time/README.md) + makes the wall mockable): a pre-2020 host yields a clock-less catalog and + media still publishes. +- Fix the `Clock::TIMESCALE` doc's "until the year 2255", the 1970 figure + (2020 lasts until 2305). +- Then backport ([release line](/quest/m0/release-22/wall-clock-omit.md)). + +Public API: `moq_mux::Clock::wall()` returns `Option`. Wire: none. + +## Related + +- [Release backport](/quest/m0/release-22/wall-clock-omit.md) - cherry-picks this onto the 0.17 line +- [Panic isolation](/quest/m0/panic-isolation.md) - a library panic no longer aborts the host diff --git a/quest/m1/README.md b/quest/m1/README.md index 38a87c57bec..a6605828f02 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -67,6 +67,7 @@ disabled rung strands WHEP, HLS, and moq-mux export viewers. - [Interop close logs](/quest/m1/interop-close-log.md) - logging fix landed; delete once the required Interop run is green - [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall +- [Quiet latest group test](/quest/m1/quiet-edge-test.md) - pins on `main` that a quiet track's latest group survives rejoin, expiry, and takeover - [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical readers obey publisher max-age and an optional tighter reader budget, including while holding a group - [Claim answer epochs](/quest/m1/claim-answer-epochs.md) - a claim answers each output by publishing it at its exact path under its own epoch, so a restarted output never reaches a returning reader as the old one's cache - [Untimed decisions](/quest/m1/untimed-decisions.md) - the maintainer decides whether moq-archive keeps refusing untimed tracks, and whether a malformed FETCH object ends its track diff --git a/quest/m1/quiet-edge-test.md b/quest/m1/quiet-edge-test.md new file mode 100644 index 00000000000..65efa7f3be1 --- /dev/null +++ b/quest/m1/quiet-edge-test.md @@ -0,0 +1,22 @@ +# [XS] Pin a quiet track's latest group across rejoin and takeover + +## Goal + +`rs/moq-net/tests/rejoin.rs` proves that a late subscriber to a live, quiet +track receives its latest group after readers left and rejoined and the cache +expiry passed, and after a publisher takeover, on every version, while a +superseded group still expires. + +## Plan + +`main` already behaves (verified 2026-10-13), but the 0.17 line broke it +([#5413](https://github.com/moq-dev/moq/issues/5413)); the test keeps a +future origin rework from regressing. Decided 2026-10-13 (maintainer): land +the port of the reporter's test plus a takeover variant, on lite-05, 06 and +07 and transport-19 and 22, using `moq_net_sim::test` time. Mention #5413 +without a closing keyword; the [release backport](/quest/m0/release-22/quiet-edge.md) +closes it. + +## Related + +- [Release backport](/quest/m0/release-22/quiet-edge.md) - the 0.17 fix this pins on `main`