diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 11695bdbd19..be643b0c2f1 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -88,7 +88,8 @@ authorization and hidden names work exactly as without one. on the old instance until dropped or its route goes, while new requests get the new one, so a player follows the restart by requesting the path again. Held handles can still read their old cached content, but cannot open new - epochless requests after retirement. Request the path again for new content. + epochless requests once the restart is delivered. Request the path again for + new content. That is how a restarted encoder, whose group numbers start over, takes the name instead of stalling viewers on the old sequence. A relay drops its copy of the old instance, so a new subscriber never sees its cached groups. diff --git a/quest/m0/README.md b/quest/m0/README.md index fa0d6b06f9b..688b38fa062 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -55,10 +55,11 @@ Audio playout: the jitter target is default in both languages (#4162); the remain and the release only requires the production clock and playback contracts that depend on them. -Interop on main (2026-10-13): the cross-client matrix fails with `unroutable` -whenever WebSocket wins the connect race: since #5294, #4180's upgrade -retires epochless handles before its held `Restart` reaches anyone. It -stays in m0 (maintainer, 2026-10-13): a red Interop hides real regressions. +Interop on main (2026-10-13): the cross-client matrix failed with `unroutable` +whenever WebSocket won the connect race: since #5294, #4180's upgrade +retired epochless handles before its held `Restart` reached anyone. Handles +now retire only once it is deliverable. First-party consumers following it +moved to m1 (maintainer, 2026-10-13), since the upgrade is being removed. Crashes (2026-10-13, #5407): a publisher on a host whose wall clock reads before 2020 panics, and `panic = "abort"` turns any library panic into a @@ -78,7 +79,6 @@ have fixes on `main`: macOS tests (#5335), drill-sensitivity (#5338), loom - [CI cost and latency](/quest/m0/ci/README.md) - PRs stop queueing behind hosted runner caps: macOS and Windows move to the merge queue, caches move to R2, and setup is skipped when no work is selected - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.5, then draft-20+ FETCH -- [Upgrade restarts epochless broadcasts](/quest/m0/upgrade-unroutable.md) - a WebSocket-to-QUIC upgrade retires epochless handles only once their `Restart` is delivered, and first-party consumers follow it, fixing Interop's `unroutable` rounds - [Panic isolation](/quest/m0/panic-isolation.md) - unwind, ignore lock poison, and supervise every task, so a panic closes only its connection or exits on a long-lived task - [nixpkgs ships the lld fix](/quest/m0/nixpkgs-lld-fix.md) - waiting on NixOS/nixpkgs#568680 to reach `nixpkgs-unstable`, which the release build fix bumps to - [Release builds link](/quest/m0/release-link.md) - the Linux zigbuild release binaries and moq-gst packages link again after the 2026-10-11 nixpkgs bump, so the next release cut can publish diff --git a/quest/m0/upgrade-unroutable.md b/quest/m0/upgrade-unroutable.md deleted file mode 100644 index 89a3f2ec830..00000000000 --- a/quest/m0/upgrade-unroutable.md +++ /dev/null @@ -1,144 +0,0 @@ -# [L] A WebSocket-to-QUIC upgrade restarts epochless broadcasts explicitly - -## Goal - -When a client's session upgrades from the WebSocket fallback to QUIC, every -epochless broadcast it had resolved gets an explicit `Restart` announce -event, and every first-party consumer follows it by resolving the path -again. Interop's full matrix on `main` stops failing with `unroutable`, -and its idle-out failures go away with it. - -Non-goals: carrying an epochless route's identity across the upgrade, and -treating `Unroutable` as a cue to retry. Epoch routes are unchanged: a -matching epoch already continues across the upgrade. - -## Plan - -Seen 2026-10-12 and 2026-10-13: Interop fails on the nightly schedule run of -`main` (run 38284317921) and on every PR that touches `test/interop` -(#5381's run 38337824317, among others). Only the cross-client rounds fail; -every `tail` case passes. The same cells pass on one run and fail on the next: - -- Rust (`moq export fmp4`), Go, and C++ subscribers exit with `unroutable` - (`catalog consumer: unroutable`, `container consumer: unroutable`), from - any publisher. -- The C subscriber reports `session status: 1`, then `2`, then `timed out - waiting for data`. -- The round then fails with `relay connection N (subscriber) idled out - instead of closing`: a client that failed exited without closing its - session. - -Cause, from the logs. In the failing js -> rust cell, the Rust subscriber -logs `connected version=moq-lite-07-wip transport=websocket`, then -`upgraded from WebSocket ... transport=webtransport` 30 ms later, then -`track info failed ... track=catalog.json err=unroutable`. On the CI runner -the debug-build QUIC dial (it fetches the certificate fingerprint over HTTP -first) often loses the race to the WebSocket head start, so the -upgrade from #4180 (2026-10-06) runs right after connect. Since #5294 -(f0fcbefd6, merged 2026-10-11 19:23 UTC), an epochless route's requests -carry a `RequestValidity` (`rs/moq-net/src/model/broadcast.rs`). It reports -`Unroutable` once the best route from the same `via` is a different -`Instance`, and an epochless instance is the route entry's generation -(`RouteEntry::instance` in `model/origin.rs`). The upgraded session -re-announces the same paths from the same peer under new generations. Every -handle resolved before the upgrade is then treated as retired, and its next -request fails. Epochless is the default: lite-06 carries no epoch, and the -browser and binding publishers don't set one. - -Onset matches. The first Interop run with `unroutable` started at 20:09 UTC on -2026-10-11. The last red runs before it failed for other reasons (`media`, -`ts`). Runs before #5294 have no `unroutable`. It still passes sometimes, -whenever QUIC wins the race. Users hit it too: `moq export` on a default -connection exits whenever WebSocket wins briefly. - -Reproduced while planning, on `main` at 079d7d8d4, by extending -`websocket_upgrades_to_quic` (`rs/moq-tokio/src/connection.rs`) with an -epochless `audio` track that nothing subscribes until after the upgrade: - -- A subscribe through the pre-upgrade `cam` handle ends with `Unroutable`. - With `RequestValidity::check` forced to pass, it succeeds. -- An announce consumer opened on the subscriber origin before the upgrade - does get `Restart` for `cam`, but only after the origin's `update_hold` - (300 ms by default; `TableCursor::update` holds epochless restarts, and - `identical_reannounce_restarts_without_an_epoch` covers the event). A - first check within about 300 ms saw nothing; a 2 s wait saw the `Restart`. - Codex caught this on #5388's review. -- A fresh `routed_broadcast("cam")` after the upgrade subscribes `audio` - fine. - -So the signal exists, but it arrives late: `RequestValidity` retires the old -handle as soon as the new route wins, while its `Restart` waits out the hold. -A consumer that asks for a track inside that window, which every interop -subscriber does right after connecting, gets `Unroutable` before any -`Restart` could reach it. And no interop consumer follows `Restart` anyway. - -Decided by the maintainer 2026-10-13 on #5388: an upgrade on an epochless -route is a Restart. "No re-requests after Unroutable please, explicitly -Restart only." This matches the -[transport upgrade](/quest/m1/transport-upgrade/README.md) plan ("epochless -routes instead signal Restart and need a fresh follower resolution"). The -other options, carrying the route identity across the upgrade and not -upgrading epochless sessions, were rejected. - -1. Retirement and its Restart agree. A holder must never see `Unroutable` - from a retired handle before the `Restart` it could have followed has been - delivered. The origin already emits the `Restart`; align the two in the - origin, not in moq-tokio, so a relay hop behaves the same when a publisher - upgrades. The relay's lite subscriber builds the same validity for a - publisher's session; check that hop too. Recommended: a handle's - `RequestValidity` retires only once its held `Restart` becomes - deliverable, so the old route keeps serving new requests through the - hold. Queueing is not enough: today `OriginState::sync_route` queues the - `PendingUpdate::Restart` (via `TableCursor::update`) before - `routes.poke_below` invalidates the `Selection`, and the hold is only - applied later in `AnnounceConsumer::poll_next`. The old session is - still draining, so it can. The alternative is to skip the hold for a - replacement that retires handles, but the hold exists to absorb path - hunting after a withdrawal, so check what that would cost first. Don't - change same-`via` handling: an identical epochless re-announcement already - restarts. Subscriptions already open stay sticky on the old session until - it drains, as today. -2. First-party consumers follow it. Each holds the announce stream for the - path it resolved, and on `Restart` drops its handles and resolves afresh. - On `Unroutable` it fails; it never retries a request. Audit and fix: - - Rust: `moq export` and `moq subscribe` (the interop subscriber), plus - any other moq-cli command that resolves a broadcast and keeps reading. - `moq play` already follows `Restart` (see - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md)); confirm it. - - Bindings: moq-ffi's catalog and container consumers, as Go, C++, and - Python reach them, and moq-c's (`moq_origin_announced_broadcast`, - `moq_consume_catalog`). Prefer following once in moq-ffi and moq-c over - duplicating it in every language wrapper. If following needs new API, - report it per the Cross-Package Sync table. - - JS: `js/net` doesn't upgrade yet (the - [JavaScript half](/quest/m1/transport-upgrade/js.md) of the upgrade - line), but a Rust relay's Restart reaches JS consumers. Confirm - `@moq/watch` and the js-native interop subscriber follow it. - - The interop clients under `test/interop/clients`, where they resolve a - broadcast themselves. -3. Regression: land the reproduction above in `websocket_upgrades_to_quic` - or a sibling, with the clock under control rather than a wall-clock wait - for the hold. It asserts that a request through the old handle made - right after the upgrade, before the `Restart` is delivered, does not fail - with `Unroutable`. It also asserts that the announce consumer then sees - `Restart` for `cam`, and that following it (a fresh `routed_broadcast`) - serves the new track. Add the same case on an epoch route, where the old handle - keeps working and no `Restart` is sent. Then run `just test interop --all`. - The race can't be forced locally (a fast machine lets QUIC win), so the - unit test carries the proof. - -If a failing client still leaves its connection to idle out, fix that -client's error path to close its session, as a separate commit. The C -client's `fail()` and the C++ client on a consumer error both look suspect. - -Public API: the upgrade's `Restart` already exists. What changes is when -an old handle retires relative to it, and that first-party consumers follow -it. Consumer-follow API in moq-ffi and moq-c only if the audit needs it; -report it in the PR. Wire: none. - -## Related - -- [Transport upgrade](/quest/m1/transport-upgrade/README.md) - its "epochless routes signal Restart" decision, which this quest implements for the Rust upgrade -- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - #5294's retired-request rule and the players that already follow `Restart` -- [#5294](https://github.com/moq-dev/moq/pull/5294) - refuse new requests from retired sources, the commit that exposed this -- [#4180](https://github.com/moq-dev/moq/pull/4180) - the WebSocket-to-QUIC upgrade diff --git a/quest/m1/README.md b/quest/m1/README.md index 01a985a77c7..df22c091740 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -54,10 +54,10 @@ above them. ## Required - [Epoch correctness](/quest/m0/broadcast-epoch/README.md) - publisher identity, duplicate suppression, and catalog pinning gate the breaking cut -- [Upgrade recovery](/quest/m0/upgrade-unroutable.md) - browser transport upgrades deliver Restart before retiring handles - [Release builds](/quest/m0/release-link.md) - package artifacts can be built - [qmux credit](/quest/m0/qmux-credit.md) - dropped/stopped streams return credit and close is delivered +- [Restart followers](/quest/m1/restart-followers.md) - first-party consumers follow an epochless `Restart`, from a GOAWAY drain or a publisher replacement, by resolving the path again - [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive baseline and residual stale-revival counterexamples before the route rewrite - [Cluster routing](/quest/m1/cluster-routing/README.md) - prove shared routing contracts, implement per-origin routes, and connect outbound local clusters - [P2P](/quest/m1/p2p/README.md) - application-selected peer sessions use shared priorities, routing, and same-epoch handover diff --git a/quest/m1/restart-followers.md b/quest/m1/restart-followers.md new file mode 100644 index 00000000000..a3d6803b766 --- /dev/null +++ b/quest/m1/restart-followers.md @@ -0,0 +1,63 @@ +# [L] First-party consumers follow an epochless Restart + +## Goal + +Every first-party consumer that resolves a broadcast and keeps reading holds +the announce stream for its path, and on `Restart` drops its handles and +resolves the path again. On `Unroutable` it fails; it never retries a +request. A GOAWAY drain or a publisher replacement on an epochless route +then costs a long-lived consumer nothing but the restart. + +## Plan + +Moved to m1 on 2026-10-13 (maintainer): the WebSocket-to-QUIC upgrade is +being removed (separate m0 quest), so an epochless Restart no longer follows +every connect; it remains for GOAWAY drains and publisher replacement. + +The origin side is done: a handle resolved through a peer keeps serving new +requests while the peer's epochless replacement is held, and refuses them +with `Unroutable` only once the `Restart` is deliverable (`Release` in `rs/moq-net/src/model/origin.rs`, +`rs/moq-net/tests/upgrade.rs`). So a consumer that never follows `Restart` +works through the hold (300 ms by default), and its open subscriptions stay +sticky on the old session until it closes, where they end cleanly. A +consumer that opens a request after the hold, or reads past the old +session's close, needs this quest. + +Audit and fix: + +- Rust: `moq export` and `moq subscribe` (the interop subscriber), and any + other moq-cli command that resolves a broadcast and keeps reading. The TS + export's `Follower` (`rs/moq-mux/src/container/ts/follower.rs`) already + watches the path, but treats any epochless return as a replacement: + sticky until the old instance ends, then `Error::Replaced`, unless + stitching. The fmp4, mkv, h264, h265, and flv exports don't follow at all. + Decide whether an epochless `Restart` restarts the output (a new init + segment mid-stream), stitches like TS, or ends the export. + `moq play` already follows `Restart` (`rs/moq-cli/src/play/media.rs`). +- Bindings: moq-ffi's catalog and container consumers, as Go, C++, and + Python reach them, and moq-c's (`moq_origin_announced_broadcast`, + `moq_consume_catalog`). Prefer following once in moq-ffi and moq-c over + duplicating it in every language wrapper. New API goes through the + Cross-Package Sync table. +- JS: `js/net`'s origin owns restart following (`js/watch/src/broadcast.ts` + mirrors its current handle). Confirm the js-native interop subscriber, + which requests with `announced: true`, follows a Rust relay's `Restart`. +- The interop clients under `test/interop/clients`, where they resolve a + broadcast themselves. + +Regression: extend `rs/moq-net/tests/upgrade.rs` (or the consumer's own +tests) so each follower reads past the old session's close after an +epochless upgrade. Then run `just test interop --all`. + +If a failing client still leaves its connection to idle out, fix that +client's error path to close its session, as a separate commit. The C +client's `fail()` (`_exit` without `moq_session_close`) and the C++ client +on a consumer error both look suspect. + +Public API: consumer-follow API in moq-ffi and moq-c, if the audit needs it. +Wire: none. + +## Related + +- [Transport upgrade](/quest/m1/transport-upgrade/README.md) - its "epochless routes signal Restart and need a fresh follower resolution" decision +- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - the players that already follow `Restart` diff --git a/quest/m1/transport-upgrade/README.md b/quest/m1/transport-upgrade/README.md index 8865ba953c7..d691001f8d5 100644 --- a/quest/m1/transport-upgrade/README.md +++ b/quest/m1/transport-upgrade/README.md @@ -82,4 +82,4 @@ and no cached groups splice across instances. Keep JS's matrix consistent. ## Related - [Closed fallback](https://github.com/moq-dev/moq/pull/5284) - a WebSocket session that closes right after connecting falls back to the pending QUIC dial instead of redialing -- [Upgrade restarts epochless broadcasts](/quest/m0/upgrade-unroutable.md) - the Rust upgrade retires epochless handles before their `Restart` arrives, and the consumers that follow it +- [Restart followers](/quest/m1/restart-followers.md) - the first-party consumers that follow the `Restart` an epochless upgrade delivers diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 61708b02db9..46e31e30df0 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -1201,7 +1201,7 @@ impl RequestValidity { /// Capture the lifetime of one pending request. pub fn start(&self) -> kio::Consumer<()> { match self.0.start() { - crate::model::broadcast::RequestValidity::Route(route) => route, + crate::model::broadcast::RequestValidity::Period(route, _) => route, _ => unreachable!("a source watch always captures a route lease"), } } diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index 196f2667b22..1486de7607c 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -534,6 +534,8 @@ pub(crate) enum RequestValidity { Unrestricted, Route(kio::Consumer<()>), Selected(std::sync::Arc), + /// One period of a selection, captured by [`Self::start`]. + Period(kio::Consumer<()>, std::sync::Arc), } impl RequestValidity { @@ -551,7 +553,7 @@ impl RequestValidity { /// Capture one selection period so a pending request never revives after a replacement. pub(crate) fn start(&self) -> Self { match self { - Self::Selected(selected) => Self::Route(selected.current()), + Self::Selected(selected) => Self::Period(selected.current(), selected.clone()), other => other.clone(), } } @@ -559,6 +561,12 @@ impl RequestValidity { pub(crate) fn check(&self, waiter: &kio::Waiter) -> Result<(), crate::Error> { match self { Self::Route(route) if route.poll_closed(waiter).is_ready() => Err(crate::Error::Unroutable), + // A replacement retires the period only once its held `Restart` is released. + Self::Period(route, selected) + if route.poll_closed(waiter).is_ready() || selected.poll_retired(waiter).is_ready() => + { + Err(crate::Error::Unroutable) + } Self::Selected(_) => self.start().check(waiter), _ => Ok(()), } diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index a43273c52c7..8a3730d9dea 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -126,6 +126,11 @@ pub struct Config { /// the wave's spread across the mesh. Defaults to [`DEFAULT_UPDATE_HOLD`]; /// zero delivers updates at once. /// + /// A broadcast resolved through a peer's route keeps serving new requests while + /// the peer's replacement of it is held, and refuses them with + /// [`Error::Unroutable`] once the restart is delivered, so a holder following + /// restarts never sees that refusal first. + /// /// A relay further out can still pass a stale path to a cursor that already /// retracted the prefix, which would announce it anew. So for two seconds after /// a retraction, an announcement from the same publisher (first hop and epoch) @@ -659,6 +664,96 @@ enum Held { Until(Instant), } +/// When one table change's held restart ends: the announce cursors deliver its +/// `Restart`, and the request handles it replaced retire. Shared by both, so a +/// handle never retires before its holder could see the `Restart` to follow. +/// +/// Anchored like [`Held`], on the first clock reading after the change. +#[derive(Clone)] +struct Release(Arc>); + +struct ReleaseState { + held: Held, + hold: Duration, + clock: Clock, + deadline: crate::time::Deadline, +} + +impl Release { + /// A release `hold` after a change made now, or `None` when nothing is held: the + /// hold is zero, or the origin's driver never read the clock. + fn new(clock: &Clock, hold: Duration) -> Option { + let now = clock.try_now().filter(|_| !hold.is_zero())?; + Some(Self(Arc::new(std::sync::Mutex::new(ReleaseState { + held: Held::Anchoring(now), + hold, + clock: clock.clone(), + deadline: crate::time::Deadline::new(clock), + })))) + } + + /// When this releases, as far as `now` can tell: past a stale anchor, it is + /// still waiting for a fresh reading to count from. + fn until(&self, now: Instant) -> Instant { + self.0.lock().unwrap().until(now) + } + + /// Whether the clock has reached the release. + fn released(&self) -> bool { + let mut state = self.0.lock().unwrap(); + let Some(now) = state.clock.try_now() else { + return false; + }; + state.until(now) <= now + } + + /// `Ready` once released, registering `waiter` until then. + fn poll(&self, waiter: &kio::Waiter) -> Poll<()> { + let mut state = self.0.lock().unwrap(); + let Some(now) = state.clock.try_now() else { + return Poll::Pending; + }; + let until = state.until(now); + state.deadline.set(Some(until)); + state.deadline.poll(waiter) + } +} + +impl ReleaseState { + fn until(&mut self, now: Instant) -> Instant { + match self.held { + Held::Anchoring(seen) if now <= seen => seen + Duration::from_nanos(1), + Held::Anchoring(_) => { + let until = now + self.hold; + self.held = Held::Until(until); + until + } + Held::Until(until) => until, + } + } +} + +/// The [`Release`] one table change's held restarts share, made on first use. +struct Hold<'a> { + clock: &'a Clock, + hold: Duration, + release: std::cell::OnceCell>, +} + +impl<'a> Hold<'a> { + fn new(clock: &'a Clock, hold: Duration) -> Self { + Self { + clock, + hold, + release: std::cell::OnceCell::new(), + } + } + + fn release(&self) -> Option { + self.release.get_or_init(|| Release::new(self.clock, self.hold)).clone() + } +} + /// One coalesced update queued for an `AnnounceConsumer`. /// /// At most one entry exists per prefix, so a slow consumer's pending set is @@ -670,9 +765,13 @@ type AnnounceMeta = (RouteMeta, Option>); enum PendingUpdate { Announce(AnnounceMeta), - Restart(AnnounceMeta), + /// Held until its [`Release`], if any. + Restart(AnnounceMeta, Option), Unannounce(AnnounceMeta), - UnannounceAnnounce { old: AnnounceMeta, new: AnnounceMeta }, + UnannounceAnnounce { + old: AnnounceMeta, + new: AnnounceMeta, + }, } /// Pending updates keyed by prefix. `BTreeMap` keeps memory strictly bounded by @@ -699,7 +798,7 @@ impl OriginConsumerState { // First announce, a stale announce being replaced, or a metadata update. None | Some(PendingUpdate::Announce(_)) => PendingUpdate::Announce(meta), // The consumer still owes the restart; this only moves its metadata. - Some(PendingUpdate::Restart(_)) => PendingUpdate::Restart(meta), + Some(PendingUpdate::Restart(_, release)) => PendingUpdate::Restart(meta, release), // Consumer needs to observe the retraction before this announce. Some(PendingUpdate::Unannounce(old) | PendingUpdate::UnannounceAnnounce { old, .. }) => { PendingUpdate::UnannounceAnnounce { old, new: meta } @@ -709,14 +808,24 @@ impl OriginConsumerState { } /// Another instance replaces the route at `prefix`, or announces it afresh when - /// the consumer never saw the old one. - fn apply_restart(&mut self, prefix: PathOwned, meta: RouteMeta, captures: Option>) { + /// the consumer never saw the old one. A restart owed already keeps its release, + /// unless this one goes at once. + fn apply_restart( + &mut self, + prefix: PathOwned, + meta: RouteMeta, + captures: Option>, + release: Option, + ) { let meta = (meta, captures); let new = match self.pending.remove(&prefix) { Some(PendingUpdate::Unannounce(old) | PendingUpdate::UnannounceAnnounce { old, .. }) => { PendingUpdate::UnannounceAnnounce { old, new: meta } } - _ if self.delivered.contains(&prefix) => PendingUpdate::Restart(meta), + Some(PendingUpdate::Restart(_, owed)) => { + PendingUpdate::Restart(meta, release.map(|new| owed.unwrap_or(new))) + } + _ if self.delivered.contains(&prefix) => PendingUpdate::Restart(meta, release), // The consumer never saw the old instance, so this is its first announce. _ => PendingUpdate::Announce(meta), }; @@ -731,7 +840,7 @@ impl OriginConsumerState { Some(PendingUpdate::Announce(_)) if !self.delivered.contains(&prefix) => {} // Either nothing is pending or the pending announce was a metadata // update on a delivered route; the consumer still owes a retraction. - None | Some(PendingUpdate::Announce(_) | PendingUpdate::Restart(_) | PendingUpdate::Unannounce(_)) => { + None | Some(PendingUpdate::Announce(_) | PendingUpdate::Restart(..) | PendingUpdate::Unannounce(_)) => { self.pending.insert(prefix, PendingUpdate::Unannounce(last)); } // The embedded announce cancels with this retraction; the consumer still @@ -742,18 +851,13 @@ impl OriginConsumerState { } } - /// Whether the pending entry at `prefix` changes a route the consumer holds - /// to another route, rather than adding or removing one, and so waits out the - /// hold. A restart to a route without an epoch waits too, since a withdrawal - /// wave's stale path looks just like one; a newer epoch is a publisher's own - /// restart and goes at once. + /// Whether the pending entry at `prefix` changes the metadata of a route the + /// consumer holds, and so waits out the hold. A restart to a route without an + /// epoch waits too, since a withdrawal wave's stale path looks just like one, + /// but on its [`Release`]: the handles it replaces retire on that one. A newer + /// epoch is a publisher's own restart and goes at once. fn is_update(&self, prefix: &PathOwned, update: &PendingUpdate) -> bool { - let held = match update { - PendingUpdate::Announce(_) => true, - PendingUpdate::Restart((meta, _)) => meta.3.is_none(), - _ => false, - }; - held && self.delivered.contains(prefix) + matches!(update, PendingUpdate::Announce(_)) && self.delivered.contains(prefix) } /// The first pending prefix ready to deliver, or when the next held one is. @@ -771,6 +875,14 @@ impl OriginConsumerState { let Some(now) = now.filter(|_| !hold.is_zero() && !self.ended) else { return Ok(prefix.clone()); }; + if let PendingUpdate::Restart(_, Some(release)) = update { + let until = release.until(now); + if until <= now { + return Ok(prefix.clone()); + } + wake = Some(wake.map_or(until, |w: Instant| w.min(until))); + continue; + } if !self.is_update(prefix, update) { let quarantine = match update { PendingUpdate::Announce((meta, _)) => retracted.get(prefix).filter(|q| q.holds(meta)), @@ -815,7 +927,7 @@ impl OriginConsumerState { }; (meta, kind) } - PendingUpdate::Restart(meta) => { + PendingUpdate::Restart(meta, _) => { self.delivered.insert(prefix.clone()); (meta, AnnounceEvent::Restart) } @@ -1174,7 +1286,7 @@ impl TableCursor { /// Deliver a changed winner at this presented prefix, as of `now`, the /// [`OriginState::next_route`]. - fn update(&mut self, presented: &PathOwned, best: Option<&RouteEntry>, now: u64) { + fn update(&mut self, presented: &PathOwned, best: Option<&RouteEntry>, now: u64, hold: &Hold) { match best { Some(entry) => { let meta = (entry.hops.clone(), entry.cost, entry.entered(), entry.epoch.clone()); @@ -1202,8 +1314,9 @@ impl TableCursor { } // Another instance: whatever was resolved under the prefix is stale. Some((prev, ..)) if prev != instance => { + let release = meta.3.is_none().then(|| hold.release()).flatten(); if let Ok(mut state) = self.state.write() { - state.apply_restart(self.under.join(presented), meta, captures); + state.apply_restart(self.under.join(presented), meta, captures, release); } } // Unchanged metadata and servability: nothing the consumer could @@ -1529,8 +1642,8 @@ impl Producer { pub fn new(config: Config) -> (Self, Driver) { let (tasks, set) = TaskSet::new(); let scope = OriginScope::default(); - let shared = kio::Shared::new(OriginState::new(config.update_hold)); let timers = Clock::default(); + let shared = kio::Shared::new(OriginState::new(config.update_hold, timers.clone())); let pool = config.pool.clone(); let producer = Self { hop: config.hop, @@ -1638,7 +1751,7 @@ impl Producer { hop, scope: OriginScope::empty(), root: PathOwned::default(), - shared: kio::Shared::new(OriginState::new(DEFAULT_UPDATE_HOLD)), + shared: kio::Shared::new(OriginState::new(DEFAULT_UPDATE_HOLD, Clock::default())), pool: cache::Pool::default(), cache_duration: Duration::MAX, stats: stats::Session::default(), @@ -3320,7 +3433,62 @@ struct Selected { path: PathOwned, via: Hop, instance: Instance, + /// The prefix of the route the instance came through. + prefix: PathOwned, + /// The selection period, closed when the peer stops selecting the instance. current: Option>, + /// The [`Release`] that ends `current`: the peer replaced the instance with a + /// route without an epoch, whose `Restart` is held until then. + retiring: Option, +} + +/// How a [`Selected`] period changes. +enum Change { + /// A new period: nothing refused in the last one revives. + Start, + /// The instance won again before its replacement was released. + Continue, + /// Retire the period once the replacement's held restart is released. + Retire(Release), + /// The peer no longer selects the instance, effective at once. + End, +} + +impl Selected { + /// How the period changes now that the peer's best route is `best`, if it does. + fn change(&self, best: Option<&RouteEntry>, hold: &Hold) -> Option { + if best.is_some_and(|entry| entry.instance() == self.instance) { + return match (&self.current, &self.retiring) { + (Some(_), None) => None, + (Some(_), Some(release)) if !release.released() => Some(Change::Continue), + _ => Some(Change::Start), + }; + } + self.current.as_ref()?; + // Announce cursors hold a restart at the same prefix onto a route without an + // epoch, so the period lasts as long. Anything else they deliver at once. + let held = best.is_some_and(|entry| entry.epoch.is_none() && entry.prefix == self.prefix); + match (held, &self.retiring) { + (true, Some(_)) => None, + (true, None) => Some(hold.release().map_or(Change::End, Change::Retire)), + (false, _) => Some(Change::End), + } + } + + fn apply(&mut self, change: Change) { + match change { + Change::Start => { + self.current = Some(kio::Producer::default()); + self.retiring = None; + } + Change::Continue => self.retiring = None, + Change::Retire(release) => self.retiring = Some(release), + Change::End => { + self.current = None; + self.retiring = None; + } + } + } } /// The lifetime of path requests while this peer still selects the held instance. @@ -3340,6 +3508,18 @@ impl Selection { .map(kio::Producer::consume) .unwrap_or_else(|| kio::Producer::new(()).consume()) } + + /// `Ready` once the current period's replacement is released, registering + /// `waiter` until then. + pub(crate) fn poll_retired(&self, waiter: &kio::Waiter) -> Poll<()> { + match self.signal.poll(waiter, |selected| match &selected.retiring { + Some(release) => release.poll(waiter), + None => Poll::Pending, + }) { + Poll::Pending => Poll::Pending, + Poll::Ready(_) => Poll::Ready(()), + } + } } impl Drop for Selection { @@ -3449,7 +3629,7 @@ impl RouteNode { } /// Wake route watchers, and revoke only leases whose selected instance changed. - fn poke(&self, routes: Option<&RouteTable>) { + fn poke(&self, routes: Option<(&RouteTable, &Hold)>) { for (_, watch) in &self.watches { match watch { WatchProducer::Changes(watch) => { @@ -3459,9 +3639,9 @@ impl RouteNode { } WatchProducer::Selection(watch) => { if let Ok(mut selected) = watch.write() { - let active = routes - .and_then(|routes| { - routes.best_route( + let change = match routes { + Some((routes, hold)) => { + let best = routes.best_route( &selected.path, Horizon { exclude: None, @@ -3470,11 +3650,14 @@ impl RouteNode { }, &HashSet::new(), |entry| entry.via == selected.via, - ) - }) - .is_some_and(|entry| entry.instance() == selected.instance); - if active != selected.current.is_some() { - selected.current = active.then(kio::Producer::default); + ); + selected.change(best, hold) + } + None => selected.current.is_some().then_some(Change::End), + }; + // Only a change wakes the requests waiting on the selection. + if let Some(change) = change { + selected.apply(change); } } } @@ -3483,13 +3666,13 @@ impl RouteNode { } /// Wake only the watched subtrees covered by this route. - fn poke_below(&self, routes: &RouteTable) { + fn poke_below(&self, routes: &RouteTable, hold: &Hold) { if self.watches_below == 0 { return; } - self.poke(Some(routes)); + self.poke(Some((routes, hold))); for child in self.children.values() { - child.poke_below(routes); + child.poke_below(routes, hold); } } @@ -3662,9 +3845,9 @@ impl RouteTable { } /// Wake the watches of every path a route at `prefix` covers. - fn poke_below(&self, prefix: &Path) { + fn poke_below(&self, prefix: &Path, hold: &Hold) { if let (_, Some(node)) = self.split(prefix) { - node.poke_below(self); + node.poke_below(self, hold); } } @@ -3699,6 +3882,8 @@ struct OriginState { // See [`Config::update_hold`]. No `Default`: every origin takes it from its // config, so no path silently disables the hold. update_hold: Duration, + // The driver's clock, which a held restart's [`Release`] counts on. + timers: Clock, // The announced routes, keyed by prefix. The table holds one entry per live // advertisement, not one per broadcast consumer. @@ -3729,9 +3914,10 @@ struct OriginState { } impl OriginState { - fn new(update_hold: Duration) -> Self { + fn new(update_hold: Duration, timers: Clock) -> Self { Self { update_hold, + timers, routes: RouteTable::default(), next_route: 0, next_watch: 0, @@ -3816,6 +4002,7 @@ impl OriginState { // Split borrows: the recompute reads `routes` while mutating a cursor. let routes = &self.routes; let now = self.next_route; + let hold = Hold::new(&self.timers, self.update_hold); let mut candidates = None; for id in routes.cursors_touching(prefix) { let Some(cursor) = self.cursors.get_mut(&id) else { @@ -3824,7 +4011,7 @@ impl OriginState { if let Some(presented) = cursor.presented(prefix, claim) { if presented.is_empty() { // A cursor root can collapse several covering prefixes into one. - Self::sync_cursor(routes, cursor, &presented, now); + Self::sync_cursor(routes, cursor, &presented, now, &hold); } else { // Every non-root presentation refers to this exact prefix. Rank // its routes once, then take each cursor's first visible entry. @@ -3840,12 +4027,12 @@ impl OriginState { .iter() .map(|(_, entry)| *entry) .find(|entry| cursor.visible(entry)); - cursor.update(&presented, best, now); + cursor.update(&presented, best, now, &hold); } } } // The fronts and requesters under the prefix re-select from the table. - routes.poke_below(prefix); + routes.poke_below(prefix, &hold); } /// A path beneath `prefix` moved to another of its routes without an epoch, while @@ -3881,7 +4068,7 @@ impl OriginState { /// Recompute the best visible route presenting at `presented` (relative) for /// one cursor and deliver the change, if any, as of `now`. - fn sync_cursor(routes: &RouteTable, cursor: &mut TableCursor, presented: &PathOwned, now: u64) { + fn sync_cursor(routes: &RouteTable, cursor: &mut TableCursor, presented: &PathOwned, now: u64, hold: &Hold) { // The entries presenting here are the ones announced at the absolute // prefix, or, for the cursor's own root, at the root and every prefix // above it (all of which present as the empty path). Among them, the @@ -3905,7 +4092,7 @@ impl OriginState { .min_by_key(|entry| route_order(&entry.prefix, entry)) }); - cursor.update(presented, best, now); + cursor.update(presented, best, now, hold); } /// Register a cursor and replay the current best route per presented prefix. @@ -3925,8 +4112,10 @@ impl OriginState { } } } + // A cursor's first sight of a route is no restart, so nothing is held. + let hold = Hold::new(&self.timers, Duration::ZERO); for p in &presented { - Self::sync_cursor(&self.routes, &mut cursor, p, self.next_route); + Self::sync_cursor(&self.routes, &mut cursor, p, self.next_route, &hold); } for head in &cursor.heads { self.routes.add_cursor(head, id); @@ -4096,6 +4285,7 @@ impl Dynamic { )?; let via = entry.via; let instance = entry.instance(); + let prefix = entry.prefix.clone(); let active = table .best_route( path, @@ -4112,7 +4302,9 @@ impl Dynamic { path: path.to_owned(), via, instance, + prefix, current: active.then(kio::Producer::default), + retiring: None, }); let signal = producer.consume(); let id = table.next_watch; diff --git a/rs/moq-net/tests/upgrade.rs b/rs/moq-net/tests/upgrade.rs new file mode 100644 index 00000000000..8530127336c --- /dev/null +++ b/rs/moq-net/tests/upgrade.rs @@ -0,0 +1,169 @@ +//! A transport upgrade, end to end over real sessions: a second session from the +//! same peer re-announces every route while the first one is still open. +//! +//! Without an epoch the re-announced routes are new instances, so announce consumers +//! get a `Restart`. A handle resolved before the upgrade keeps serving new requests +//! until that `Restart` is deliverable, so its holder can follow the `Restart` instead +//! of failing `Unroutable`. With an epoch the upgrade continues the instance: no +//! `Restart`, and the old handle keeps working. + +mod support; + +use std::time::Duration; + +use moq_net::{Epoch, Error, Hop, Timestamp, Version, announce, broadcast, origin, track}; +use support::harness::{MockConnectOptions, MockPair, connect_mock}; + +/// A millisecond track: the frames written here carry timestamps. +fn timed() -> moq_net::track::Info { + moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI) +} + +const TEST_TIMEOUT: Duration = Duration::from_secs(60); + +fn produce_origin(hop: u64) -> origin::Producer { + let (producer, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + support::harness::spawn(driver); + producer +} + +/// Have `to` pull everything `from` publishes, over a new session. +async fn link(version: Version, from: &origin::Producer, to: &origin::Producer) -> MockPair { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(from.consume()); + options.client_subscribe = Some(to.clone()); + connect_mock(options).await +} + +/// The next announce event, failing on a hang. +async fn next_event(announced: &mut announce::Consumer) -> announce::Event { + moq_net_sim::timeout(Duration::from_secs(10), announced.next()) + .await + .expect("announce hung") + .expect("announce ended") +} + +/// Write one group of one frame tagged `tag`. +fn write(track: &track::Producer, tag: &str) { + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, tag.as_bytes().to_vec()).unwrap(); + group.finish().unwrap(); +} + +/// The next frame a subscription delivers, as text. +async fn read(subscription: &mut track::Subscriber) -> String { + let mut group = moq_net_sim::timeout(Duration::from_secs(10), subscription.recv_group()) + .await + .expect("subscription hung") + .unwrap() + .expect("subscription ended"); + let frame = group.read_frame().await.unwrap().expect("a frame"); + String::from_utf8(frame.payload.to_vec()).unwrap() +} + +async fn subscribe(broadcast: &broadcast::Consumer, name: &str) -> Result { + moq_net_sim::timeout(Duration::from_secs(10), broadcast.track(name)?.subscribe(None)) + .await + .expect("subscribe hung") +} + +/// Shorter than the default update hold, so the upgrade's `Restart` is still held. +const WITHIN_HOLD: Duration = Duration::from_millis(10); + +/// Longer than the default update hold. +const PAST_HOLD: Duration = Duration::from_millis(500); + +async fn upgrade(version: &str, epoch: Option) { + let version: Version = version.parse().unwrap(); + let publisher = produce_origin(1); + let viewer = produce_origin(3); + + let mut route = origin::Route::default(); + route.epoch = epoch.clone(); + let cam = publisher.publish("cam", route).unwrap(); + let video = cam.create_track("video", timed()).unwrap(); + let audio = cam.create_track("audio", timed()).unwrap(); + let meta = cam.create_track("meta", timed()).unwrap(); + + let _websocket = link(version, &publisher, &viewer).await; + let mut announced = viewer.consume().announced(); + match next_event(&mut announced).await { + announce::Event::Start(announce) => assert_eq!(announce.prefix.as_str(), "cam"), + other => panic!("{version}: expected the broadcast, got {other:?}"), + } + + let old = viewer.consume().request_broadcast("cam", None).await.unwrap(); + let mut sticky = subscribe(&old, "video").await.unwrap(); + write(&video, "video:0"); + assert_eq!(read(&mut sticky).await, "video:0", "{version}"); + + // The upgrade: the same peer re-announces everything over a second session, + // while the first one is still open. + let _quic = link(version, &publisher, &viewer).await; + moq_net_sim::sleep(WITHIN_HOLD).await; + + // A request through the old handle right after the upgrade is still served. + let mut late = subscribe(&old, "audio") + .await + .unwrap_or_else(|err| panic!("{version}: the old handle refused a request before any restart: {err}")); + write(&audio, "audio:0"); + assert_eq!(read(&mut late).await, "audio:0", "{version}"); + + if epoch.is_some() { + // The epoch continues across the upgrade: nothing to restart, and the old + // handle keeps serving. + if let Ok(event) = moq_net_sim::timeout(PAST_HOLD, announced.next()).await { + panic!("{version}: an epoch route restarted on upgrade: {event:?}"); + } + let mut later = subscribe(&old, "meta").await.unwrap(); + write(&meta, "meta:0"); + assert_eq!(read(&mut later).await, "meta:0", "{version}"); + return; + } + + // Without an epoch the upgrade is a restart: once the hold releases it, the old + // handle refuses new requests, and the restart is already there to follow. + moq_net_sim::sleep(PAST_HOLD).await; + let Err(err) = subscribe(&old, "meta").await else { + panic!("{version}: a retired handle served a request"); + }; + assert!(matches!(err, Error::Unroutable), "{version}: {err}"); + match announced.try_next() { + Some(announce::Event::Restart(announce)) => assert_eq!(announce.prefix.as_str(), "cam"), + other => panic!("{version}: retired before the restart was deliverable, got {other:?}"), + } + + // Following the restart resolves the path afresh, and the new handle serves. + let fresh = viewer.consume().request_broadcast("cam", None).await.unwrap(); + assert!(!fresh.is_clone(&old), "{version}: joined the retired broadcast"); + let mut followed = subscribe(&fresh, "meta").await.unwrap(); + write(&meta, "meta:0"); + assert_eq!(read(&mut followed).await, "meta:0", "{version}"); +} + +/// The epoch an upgraded route carries on both sessions. +fn epoch() -> Epoch { + "01900000-0000-7000-8000-000000000001".parse().unwrap() +} + +#[moq_net_sim::test] +async fn epochless_lite06() { + moq_net_sim::timeout(TEST_TIMEOUT, upgrade("moq-lite-06", None)) + .await + .expect("timed out"); +} + +#[moq_net_sim::test] +async fn epochless_lite07() { + moq_net_sim::timeout(TEST_TIMEOUT, upgrade("moq-lite-07-wip", None)) + .await + .expect("timed out"); +} + +/// Only lite-07 carries the epoch. +#[moq_net_sim::test] +async fn epoch_lite07() { + moq_net_sim::timeout(TEST_TIMEOUT, upgrade("moq-lite-07-wip", Some(epoch()))) + .await + .expect("timed out"); +}