From 7ff3fbd4c1864820bd6c321ed4dc5637ac33f497 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 11:38:04 -0700 Subject: [PATCH 1/6] quest(m1): a route change wakes only the fronts it affects Found by the pool churn benchmark (#4607). Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/route-wakes.md | 68 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100644 quest/m1/route-wakes.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 12f1f067c0a..10f80fa48ea 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -117,6 +117,7 @@ QUIC studies there on that rule. - [Frame slot charge](/quest/m1/frame-slot-charge.md) - a group's frame slots past the first four count against the cache pool, including capacity a released group keeps - [Front deadlines](/quest/m1/front-deadline-index.md) - a front's per-event cost stops growing with its track count: an expiry index and per-track wakes, proven by a churn benchmark - [Front parking](/quest/m1/origin-front-parks.md) - an unroutable request waits on a front instead of re-asking on every route-table move +- [Route wakes](/quest/m1/route-wakes.md) - a route change wakes only the fronts it can move, so pool churn stops scaling with served paths - [Publish channel count](/quest/m1/publish-audio-channel-count.md) - forcing a channel count on an Audio.Capture stops costing the subscriber gaps of silence - [JS abandonment](/quest/m1/js-subscribe-abandonment.md) - a viewer returning during IETF subscribe setup keeps its track across microtasks - [Epoch primitive](/quest/m1/epoch.md) - one `Epoch` type in moq-net and @moq/net, carried as a trailing `@` path segment, shared by e2ee and broadcast epochs diff --git a/quest/m1/route-wakes.md b/quest/m1/route-wakes.md new file mode 100644 index 00000000000..1664cf826d2 --- /dev/null +++ b/quest/m1/route-wakes.md @@ -0,0 +1,68 @@ +# [M] A route change wakes only the fronts it affects + +## Goal + +A route change under a prefix (join, withdraw, re-price, restale) makes the +origin driver act only on the fronts that could change their selection: +fronts still waiting for a source, fronts served by the changed route, and +publisher-pinned fronts on its first hop. Today it wakes every front below +the prefix, so one equal-cost pool member joining or leaving a prefix that +serves 10k paths costs the driver 100 to 600 ms of single-threaded work +while no front switches (`origin/pool_churn`, #4607). A root claim, such as +an archive's catch-all, makes every front on the relay pay for every change +to it. Done when `origin/pool_churn` is flat in served paths. + +## Plan + +Facts from the wildcard line (`rs/moq-net/src/model/origin.rs` there): + +- Every route change calls `sync_route`, which calls + `routes.poke_below(prefix)`. That bumps a counter-only `Watch` on every path + below the prefix, so a woken front cannot tell what changed. Each re-runs + `select` under the table read lock (`retain_routes`, then `best_route`, + which hashes every pool member), then the driver polls every track and + rescans deadlines. +- `route_order` is rendezvous-style: FNV over the path and hop ids, lowest + wins. A join moves only paths the newcomer wins, a leave only paths the + leaver was winning. +- A serving front is `Pin::Stay` and ignores joins and re-prices; it moves + only when its own route is withdrawn or its source closes. So on a join + the fronts that can act are the waiting ones (no live source, including an + upstream request in flight), publisher-pinned ones on the new route's first + hop, and local-pinned ones for a local route. On a leave, the fronts served + by or requesting through that route. A deeper prefix appearing shadows + shallower routes for the same non-Stay fronts. + +Decisions: + +- Covers every route change, not only equal-cost pools. The pool is where + the benchmark found it, but the wake is per prefix regardless of cost. + (2026-09-30) +- An index, not a payload filter on the watch: each route records the + fronts it serves, and each prefix node records only its non-Stay watches + (waiting fronts, publisher- and local-pinned fronts, `routed_broadcast` + waiters). A change wakes those plus the changed route's own fronts. A + payload filter would still walk every watch under the prefix, which stays + linear in paths. (2026-09-30) +- Builds on shared-fronts' keying, so the index hangs off the final front + identity. `origin-front-parks.md` replaces the `routed_broadcast` retry + loop, one of the watch consumers here; whichever lands second adapts it. + (2026-09-30) +- The wildcard line's `quest/m1/front-upgrade.md` will make a cheaper route + matter to Stay fronts, roughly those the new route would win. Keep the + index shaped so that set can be added without walking every served path. + +Verification: `origin/pool_churn` flat in served paths, and a unit test +that counts `select` calls per route change and asserts only affected +fronts re-select. Keep `pool_resolve` unchanged. + +Public API: none. Wire: none. + +## Required + +- [Viewer sessions share a front](/quest/m0/shared-fronts.md) - re-keys fronts, which this index hangs off; it lands after the wildcard line, which is where the pool code lives + +## Related + +- [Front deadlines](/quest/m1/front-deadline-index.md) - each spurious wake also pays that per-track poll and deadline scan +- [Front parking](/quest/m1/origin-front-parks.md) - replaces the `routed_broadcast` watch loop this wakes From 26882dba9d9173bfec9f671918a1125b41c1aaa7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 11:50:19 -0700 Subject: [PATCH 2/6] quest(m1): key route-wake watches by pin Serving pool fronts are Publisher-pinned, not Stay, so a prefix bucket of all non-Stay watches would still wake every served path. Co-Authored-By: Claude Opus 5.5 --- quest/m1/route-wakes.md | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/quest/m1/route-wakes.md b/quest/m1/route-wakes.md index 1664cf826d2..9ebe57e3fb1 100644 --- a/quest/m1/route-wakes.md +++ b/quest/m1/route-wakes.md @@ -20,18 +20,22 @@ Facts from the wildcard line (`rs/moq-net/src/model/origin.rs` there): `routes.poke_below(prefix)`. That bumps a counter-only `Watch` on every path below the prefix, so a woken front cannot tell what changed. Each re-runs `select` under the table read lock (`retain_routes`, then `best_route`, - which hashes every pool member), then the driver polls every track and - rescans deadlines. + which hashes every qualifying pool member unless a Stay front's route + still serves), then the driver polls every track and rescans deadlines. - `route_order` is rendezvous-style: FNV over the path and hop ids, lowest wins. A join moves only paths the newcomer wins, a leave only paths the leaver was winning. -- A serving front is `Pin::Stay` and ignores joins and re-prices; it moves - only when its own route is withdrawn or its source closes. So on a join - the fronts that can act are the waiting ones (no live source, including an - upstream request in flight), publisher-pinned ones on the new route's first - hop, and local-pinned ones for a local route. On a leave, the fronts served - by or requesting through that route. A deeper prefix appearing shadows - shallower routes for the same non-Stay fronts. +- A front's `Pin` limits which routes it can take. `Stay` (an origin-named + front while serving) keeps its route while it serves; `Route` qualifies + only its own route; `Publisher` only routes whose first hop matches; + `Local` only local routes; `Any` (waiting, no live source or an upstream + request in flight) every route. Pool fronts in `origin/pool_churn` never + get an origin-naming reply, so they are `Publisher`, each on a different + member's hop. So on a join the fronts that can act are the `Any` ones, + `Publisher` ones on the new route's first hop, and `Local` ones for a + local route. On a leave, the fronts served by or requesting through that + route. A deeper prefix appearing shadows shallower routes for the same + fronts. Decisions: @@ -44,6 +48,12 @@ Decisions: waiters). A change wakes those plus the changed route's own fronts. A payload filter would still walk every watch under the prefix, which stays linear in paths. (2026-09-30) +- The prefix node keys its watches by what they qualify for: `Any` and + `routed_broadcast` waiters in one set, `Publisher` by hop, `Local` in its + own set. `Route` fronts hang off their route only. A join wakes the `Any` + set, its first hop's `Publisher` set, and the `Local` set if local, so an + unrelated publisher joining wakes no serving pool front. A withdrawal + takes the route's served set before dropping the record. (2026-09-30) - Builds on shared-fronts' keying, so the index hangs off the final front identity. `origin-front-parks.md` replaces the `routed_broadcast` retry loop, one of the watch consumers here; whichever lands second adapts it. @@ -52,9 +62,10 @@ Decisions: matter to Stay fronts, roughly those the new route would win. Keep the index shaped so that set can be added without walking every served path. -Verification: `origin/pool_churn` flat in served paths, and a unit test -that counts `select` calls per route change and asserts only affected -fronts re-select. Keep `pool_resolve` unchanged. +Verification: `origin/pool_churn` flat in served paths at every pool width +it already sweeps, and a unit test that counts `select` calls per route +change and asserts only affected fronts re-select, with zero for an +unrelated publisher's join and leave. Keep `pool_resolve` unchanged. Public API: none. Wire: none. From 500774558411a43f0a601177832c58a46c24f07d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 12:05:52 -0700 Subject: [PATCH 3/6] quest(m1): spell route-wake sets for every change Aligns the index decision with the pin-keyed wake rule, covers leave, re-price, and restale, says how descendant fronts are found, and names the same-publisher limit. Co-Authored-By: Claude Opus 5.5 --- quest/m1/route-wakes.md | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/quest/m1/route-wakes.md b/quest/m1/route-wakes.md index 9ebe57e3fb1..7246bb58682 100644 --- a/quest/m1/route-wakes.md +++ b/quest/m1/route-wakes.md @@ -28,8 +28,8 @@ Facts from the wildcard line (`rs/moq-net/src/model/origin.rs` there): - A front's `Pin` limits which routes it can take. `Stay` (an origin-named front while serving) keeps its route while it serves; `Route` qualifies only its own route; `Publisher` only routes whose first hop matches; - `Local` only local routes; `Any` (waiting, no live source or an upstream - request in flight) every route. Pool fronts in `origin/pool_churn` never + `Local` only local routes; `Any` (identity not yet known, or origin-named + and not serving) every route. Pool fronts in `origin/pool_churn` never get an origin-naming reply, so they are `Publisher`, each on a different member's hop. So on a join the fronts that can act are the `Any` ones, `Publisher` ones on the new route's first hop, and `Local` ones for a @@ -45,15 +45,26 @@ Decisions: - An index, not a payload filter on the watch: each route records the fronts it serves, and each prefix node records only its non-Stay watches (waiting fronts, publisher- and local-pinned fronts, `routed_broadcast` - waiters). A change wakes those plus the changed route's own fronts. A + waiters). A change wakes the ones it qualifies for (next decision) plus + the changed route's own fronts. A payload filter would still walk every watch under the prefix, which stays linear in paths. (2026-09-30) - The prefix node keys its watches by what they qualify for: `Any` and `routed_broadcast` waiters in one set, `Publisher` by hop, `Local` in its - own set. `Route` fronts hang off their route only. A join wakes the `Any` - set, its first hop's `Publisher` set, and the `Local` set if local, so an - unrelated publisher joining wakes no serving pool front. A withdrawal - takes the route's served set before dropping the record. (2026-09-30) + own set. `Route` fronts hang off their route only. A front moves its + entries when it selects, and drops them when it closes. A route gaining + standing (join, cheaper re-price, restale to live) wakes the `Any` set, + its first hop's `Publisher` set, and the `Local` set if local, so an + unrelated publisher joining wakes no serving pool front. A route losing + standing (leave, dearer re-price, going stale) wakes only the fronts it + serves or is requesting for; a withdrawal takes that set before dropping + the record. Fronts sit at paths below the changed prefix, so the lookup + walks descendants but skips subtrees whose matching set is empty, keeping + cost in woken fronts rather than served paths. (2026-09-30) +- A same-publisher join still wakes that hop's whole `Publisher` set, even + where the new route cannot win. Narrowing it to the paths the route wins + is the same set `front-upgrade.md` needs (below), so it lands with that. + (2026-09-30) - Builds on shared-fronts' keying, so the index hangs off the final front identity. `origin-front-parks.md` replaces the `routed_broadcast` retry loop, one of the watch consumers here; whichever lands second adapts it. From 5dac5b40ad617b8b355724450926c2194da143b7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 12:27:03 -0700 Subject: [PATCH 4/6] quest(m1): wake parked waiters when a claim loses standing A withdrawn deeper claim can unshadow a broader route for a waiter it never served, so every change wakes the pin sets, not only gains. Co-Authored-By: Claude Opus 5.5 --- quest/m1/route-wakes.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/quest/m1/route-wakes.md b/quest/m1/route-wakes.md index 7246bb58682..d44a3a7dfe0 100644 --- a/quest/m1/route-wakes.md +++ b/quest/m1/route-wakes.md @@ -52,13 +52,13 @@ Decisions: - The prefix node keys its watches by what they qualify for: `Any` and `routed_broadcast` waiters in one set, `Publisher` by hop, `Local` in its own set. `Route` fronts hang off their route only. A front moves its - entries when it selects, and drops them when it closes. A route gaining - standing (join, cheaper re-price, restale to live) wakes the `Any` set, - its first hop's `Publisher` set, and the `Local` set if local, so an - unrelated publisher joining wakes no serving pool front. A route losing - standing (leave, dearer re-price, going stale) wakes only the fronts it - serves or is requesting for; a withdrawal takes that set before dropping - the record. Fronts sit at paths below the changed prefix, so the lookup + entries when it selects, and drops them when it closes. Any change to a + route wakes the `Any` set, its first hop's `Publisher` set, the `Local` + set if local, and the fronts it serves or is requesting for, so an + unrelated publisher joining or leaving wakes no serving pool front. The + pin sets wake on a loss too: withdrawing a deeper claim can unshadow a + broader route for a parked waiter it never served. A withdrawal takes + the served set before dropping the record. Fronts sit at paths below the changed prefix, so the lookup walks descendants but skips subtrees whose matching set is empty, keeping cost in woken fronts rather than served paths. (2026-09-30) - A same-publisher join still wakes that hop's whole `Publisher` set, even @@ -76,7 +76,8 @@ Decisions: Verification: `origin/pool_churn` flat in served paths at every pool width it already sweeps, and a unit test that counts `select` calls per route change and asserts only affected fronts re-select, with zero for an -unrelated publisher's join and leave. Keep `pool_resolve` unchanged. +unrelated publisher's join and leave, and a parked waiter retrying when a +deeper advertise-only claim over a served root is withdrawn. Keep `pool_resolve` unchanged. Public API: none. Wire: none. From aab92313cf4cddae338e94e46e7ec96c721dcc5a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 12:48:21 -0700 Subject: [PATCH 5/6] quest(m1): align route-wake facts with the leave rule Co-Authored-By: Claude Opus 5.5 --- quest/m1/route-wakes.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/quest/m1/route-wakes.md b/quest/m1/route-wakes.md index d44a3a7dfe0..444ba1ec0bd 100644 --- a/quest/m1/route-wakes.md +++ b/quest/m1/route-wakes.md @@ -34,8 +34,9 @@ Facts from the wildcard line (`rs/moq-net/src/model/origin.rs` there): member's hop. So on a join the fronts that can act are the `Any` ones, `Publisher` ones on the new route's first hop, and `Local` ones for a local route. On a leave, the fronts served by or requesting through that - route. A deeper prefix appearing shadows shallower routes for the same - fronts. + route, plus those same pin sets: a deeper prefix appearing shadows + shallower routes for them, and one leaving can unshadow a broader route + for a waiter it never served. Decisions: From 55982f0b1ba610ff7f2cf05a23c1ba0af0ebe750 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 13:59:01 -0700 Subject: [PATCH 6/6] quest(route-wakes): drop front-upgrade, removed by the tiers decision Co-Authored-By: Claude Opus 5.5 --- quest/m1/route-wakes.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/quest/m1/route-wakes.md b/quest/m1/route-wakes.md index 444ba1ec0bd..e90a09936da 100644 --- a/quest/m1/route-wakes.md +++ b/quest/m1/route-wakes.md @@ -64,15 +64,15 @@ Decisions: cost in woken fronts rather than served paths. (2026-09-30) - A same-publisher join still wakes that hop's whole `Publisher` set, even where the new route cannot win. Narrowing it to the paths the route wins - is the same set `front-upgrade.md` needs (below), so it lands with that. - (2026-09-30) + is left out: the `Publisher` pin goes with `--hop` removal in the cluster + routing line. (2026-09-30) - Builds on shared-fronts' keying, so the index hangs off the final front identity. `origin-front-parks.md` replaces the `routed_broadcast` retry loop, one of the watch consumers here; whichever lands second adapts it. (2026-09-30) -- The wildcard line's `quest/m1/front-upgrade.md` will make a cheaper route - matter to Stay fronts, roughly those the new route would win. Keep the - index shaped so that set can be added without walking every served path. +- Stay fronts never need a join or re-price: the edge/core tiers decision + (2026-09-30) dropped route upgrades, so a serving front moves only on + failover. Verification: `origin/pool_churn` flat in served paths at every pool width it already sweeps, and a unit test that counts `select` calls per route