diff --git a/quest/m0/README.md b/quest/m0/README.md index d59646e1cfc..42fb2c0a147 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -2,12 +2,14 @@ ## Goal -Defects in what main ships today: crashes, races, protocol violations, -wrong output, security gaps, and the missing tests that let them through. +Defects in what main or dev ships today: crashes, races, protocol violations, +wrong output, security gaps, regressions an unreleased change introduced, and +the missing tests that let them through. ## Plan -Fix on main. Security and credential exposure lead; user-visible breakage +Fix where the defect is, which is usually main; a quest says so when it is dev. +Security and credential exposure lead; user-visible breakage next; hardening, tooling, and test debt close the list. Each fix lands with a regression test per Root Cause First. @@ -15,6 +17,7 @@ regression test per Root Cause First. - [#3087](/quest/m0/3087-relay-mtls-peers-bypass-auth-api-mode-so-proxy-grants.md) - relay: mTLS peers bypass --auth-api-mode, so proxy grants can't refuse or scope them - [#2405](/quest/m0/2405-js-net-connect-logs-on-every-connection-at-the-wrong.md) - js/net: connect() logs on every connection at the wrong level and prints the JWT in the URL +- [Route resume identity](/quest/m0/route-resume.md) - a failing route ends the subscription instead of resuming through the next one, since announcements became prefix routes - [Dart leaks](/quest/m0/dart-leak.md) - the generated Dart bindings leak native memory on every call - [#3207](/quest/m0/3207-send-valid-publish-done-statuses-for-every-supported-ietf.md) - Send valid PUBLISH_DONE statuses for every supported IETF draft - [#3076](/quest/m0/3076-moq-relay-publish-is-unimplemented-by-design-make-the.md) - moq-relay: PUBLISH is unimplemented by design - make the rejection fail fast for clients that wait diff --git a/quest/m0/route-resume.md b/quest/m0/route-resume.md new file mode 100644 index 00000000000..c08b5e5bf6c --- /dev/null +++ b/quest/m0/route-resume.md @@ -0,0 +1,101 @@ +# [XL] Route resume identity + +## Goal + +A subscription whose serving route goes away resumes through the next best +route without the consumer noticing, as it did before announcements became +prefix routes. A route change that preserves the origin identity re-splices at +a group boundary; one that does not ends the subscription so the consumer +starts a new one. + +This supplies the per-path identity two other questlines are built on top of. + +## Plan + +### Root cause, and the decision + +[moq#3225](https://github.com/moq-dev/moq/pull/3225) made announcements prefix +routes and moved route state out of `broadcast::Broadcast` into `origin`. A +front used to hold the routes for its path and the identity of whoever was +serving it: `FrontState.publisher` was the first hop of the best route, and +`same_identity` decided whether an arriving route joined the existing front +(splice) or started a fresh broadcast. Both are gone, and the PR specified the +loss into the draft: "A route carries no content identity", and a relay "MUST +NOT splice a live subscription across sources reached through different +routes". Failover on dev is abort and resubscribe +(`origin::Consumer::routed_broadcast` retries), and every reader sees the +subscription end. + +Decision (2026-09-01): that spec point is reversed. Routes keep carrying the +hops and cost of what they serve, and a relay resumes/stitches a broadcast +across routes when the first hop is the same. `Epoch` stays dead; it was a +per-broadcast content generation, and nothing per-broadcast survives prefix +routes. What identifies a source is the route's first hop, which is still on +the wire, still in `RouteEntry`, and still ranked on by `route_order`. What was +lost is the structure that read it, not the value. + +### What to restore + +Give the per-path serve state an identity, derived from the first hop of the +route currently serving it, and re-splice rather than abort when a route change +preserves it: + +- Equal non-zero first hops are the same origin endpoint reached another way (a + reconnect, a cheaper path, a relay restarting). Splice at a group boundary. +- Different first hops are different sources at one path. End the subscription + rather than splicing one source's subscribers onto another's frames. +- `Hop::UNKNOWN` identifies nobody and never matches itself, which is what + `same_identity` existed to encode. Two anonymous peers must not pass for one + reconnecting. + +An empty hop chain is a local publish and keeps the local-source behavior it +has now. + +This is more than restoring a field, which is why the quest is sized XL. On dev +`ServeState` is per-announcer, and `create_source` mints a broadcast that is +deliberately not inserted into the broadcast tree, so nothing per-path spans +routes. The work includes rebuilding that front: a per-path structure that +outlives any single route, holds the serving identity, and is what a +replacement route splices into. + +### Draft + +The implementing PR amends `drafts/draft-lcurley-moq-lite.md` in the same +change, per the wire-behavior rule. The "route carries no content identity" +paragraph becomes the first-hop rule: two routes covering one path with the +same non-zero first hop are the same origin reached different ways, and a +relay MAY move a live subscription between them at a group boundary; across +differing or unknown first hops a relay MUST NOT splice, and the subscription +ends with the serving session as it does today. Record the change in the +moq-lite-06 changelog. + +### Boundaries + +Identity here is a routing-layer property, not a media one. The first hop +identifies the endpoint that originated the route, which for a prefix route is +whoever serves the subtree; equal first hops mean the same endpoint, not that +two encodings are interchangeable. That media-generation question is what +`Epoch` was reaching for and what +[plan-hls-identity](/quest/m2/plan-hls-identity.md) still owns for cacheable +URLs. + +The fix lands on `dev`, where prefix routes live; `main` has no prefix routes. + +### Tests + +The pre-#3225 goaway and cluster tests are the specification: a draining +session serves through the handover window and its subscribers move to the +surviving route without the subscription ending. Add the first-hop cases the +old front covered: a reconnecting publisher splices, a different publisher at +the same path does not, two anonymous peers never pass for one, and a +metadata-only reprice changes nothing visible. + +## Related + +- [Rank](/quest/m2/pop-skipping/rank.md) - reuses this comparison rule on the + route's *last* hop, since adopting a parent is about the adjacent relay rather + than about who produced the content +- [Resolve](/quest/m2/wildcard/resolve.md) - two publishers colliding at one + literal path resolve through this identity +- [plan-hls-identity](/quest/m2/plan-hls-identity.md) - the media-generation + identity this deliberately does not supply diff --git a/quest/m1/2714-per-subscriber-path-predicate-for-originconsumer-0-1-x.md b/quest/m1/2714-per-subscriber-path-predicate-for-originconsumer-0-1-x.md index fed5c726ced..bd91dbcb6ea 100644 --- a/quest/m1/2714-per-subscriber-path-predicate-for-originconsumer-0-1-x.md +++ b/quest/m1/2714-per-subscriber-path-predicate-for-originconsumer-0-1-x.md @@ -12,6 +12,23 @@ dev's reshaped announce/origin surface only; the 0.1.x backport half is dropped unless a maintenance branch appears. See also the pattern-scoped origin work at /quest/m2/path-patterns/origin.md. +### After prefix routes + +A per-path predicate does not mean what it did when this was filed. +[moq#3225](https://github.com/moq-dev/moq/pull/3225) made `announced()` yield +covering prefixes rather than broadcast paths, so a predicate applied to the +announce stream is deciding about a *set*: a route at `room/` carries every +path under it, including the ones a deafen rule wants blocked, and blocking the +whole prefix hides paths the subscriber may use. The filter also cannot be the +enforcement point on its own, because `request_broadcast` resolves against the +route table directly. + +So this needs the enforcement boundary settled before an API: a predicate over +announcements (discovery), a predicate at resolution (access), or the +pattern-scoped consumer in +[origin](/quest/m2/path-patterns/origin.md). The reporter's requirement, which a +forked client cannot bypass, points away from a discovery-time filter. + ### Issue context Hi @kixelated 👋 diff --git a/quest/m1/2895-add-an-atomic-readiness-gate-for-origin-broadcasts.md b/quest/m1/2895-add-an-atomic-readiness-gate-for-origin-broadcasts.md index 0a508b8dd13..977f0387acd 100644 --- a/quest/m1/2895-add-an-atomic-readiness-gate-for-origin-broadcasts.md +++ b/quest/m1/2895-add-an-atomic-readiness-gate-for-origin-broadcasts.md @@ -11,6 +11,22 @@ Use the public issue's scope, implementation notes, and acceptance criteria below as the starting plan. Reconcile paths and assumptions with the current tree before implementation. +### After prefix routes + +Half of the stated problem is gone. +[moq#3225](https://github.com/moq-dev/moq/pull/3225) made +`origin::Producer::create_broadcast` stop announcing: the blessed order is +create, populate, announce, and `announce` is a separate call returning a guard. +So a broadcast is no longer announced before its tracks exist, and +`broadcast::Route::announce = false` (which the issue correctly says is not a +readiness gate) does not exist either. + +What remains is the exact-path half: `create_broadcast` still makes the source +visible to `request_broadcast` immediately, so a concurrent consumer can find a +broadcast whose tracks and `broadcast::Dynamic` handler are not installed yet. +Re-scope to that, and weigh it against the convention now being sufficient for +the announced path. + ### Issue context #### Problem diff --git a/quest/m1/2933-moq-ffi-moqroute-round-trip-erases-a-routes-cold-cost.md b/quest/m1/2933-moq-ffi-moqroute-round-trip-erases-a-routes-cold-cost.md index a90891e0f4a..71cfb941047 100644 --- a/quest/m1/2933-moq-ffi-moqroute-round-trip-erases-a-routes-cold-cost.md +++ b/quest/m1/2933-moq-ffi-moqroute-round-trip-erases-a-routes-cold-cost.md @@ -11,6 +11,28 @@ Use the public issue's scope, implementation notes, and acceptance criteria below as the starting plan. Reconcile paths and assumptions with the current tree before implementation. +### After prefix routes + +[moq#3225](https://github.com/moq-dev/moq/pull/3225) deleted both APIs the +issue names. `MoqBroadcastConsumer::route_updates` and +`MoqBroadcastProducer::set_route` are gone; a broadcast no longer carries a +route at all. + +The bug survived the move. `MoqOriginProducer::announce(prefix, MoqRoute)` is +the input, `MoqAnnouncement.route` is the output, and the two conversions are +unchanged: `From` reports `route.cost.warm` and drops `cold`, +while `TryFrom` calls `with_cost(u64)`, which sets both halves. So +observing a route through `announced()` and re-announcing it still rewrites a +truthful `{warm: 0, cold: N}` into `{warm: 0, cold: 0}`, claiming to be the +publisher. + +The severity argument in the issue no longer holds either: it rested on +`FrontState::handover_allowed` only consulting cold at two or more hops, and +that gate was deleted with the rest of the handover machinery. Cold is still +ranked on by `route_order`, so an understated cold still wins ties it should +lose. Re-read the options below against `announce`/`announced` rather than +`set_route`/`route_updates`. + ### Issue context Found by the Codex reviewer during [#2925](https://github.com/moq-dev/moq/pull/2925), which splits the route cost into `broadcast::Cost { warm, cold }`. Filing rather than fixing in that PR because it is a public-API decision spanning five language bindings. diff --git a/quest/m1/2985-js-net-path-keyed-publisher-state-goes-stale-when-a.md b/quest/m1/2985-js-net-path-keyed-publisher-state-goes-stale-when-a.md index ec38bc9241f..357aca8a0fc 100644 --- a/quest/m1/2985-js-net-path-keyed-publisher-state-goes-stale-when-a.md +++ b/quest/m1/2985-js-net-path-keyed-publisher-state-goes-stale-when-a.md @@ -11,6 +11,30 @@ Use the public issue's scope, implementation notes, and acceptance criteria below as the starting plan. Reconcile paths and assumptions with the current tree before implementation. +### After prefix routes + +The remedy this quest points at no longer exists. +[moq#3225](https://github.com/moq-dev/moq/pull/3225) removed `Epoch` from +`draft-lcurley-moq-lite` as spec-only and never implemented, and retired +`draft-lcurley-moq-broadcast`. So "#2610 specs exactly this as the broadcast +epoch" is now a dead pointer, and TRACK_INFO epoch-keying is not coming from +there. + +Two of the three sites are also already fixed on `dev`, by keying on the routing +front rather than on the path: + +- `lite/publisher.ts`'s `runAnnounce` holds `Map` and diffs on the front, so a republish emits + ended-then-active instead of nothing. +- `ietf/publisher.ts` records the front each refusal was about in `offered`, so + a republish at the same path clears the refusal. + +What remains is the third: `#resolveTrackInfo` caches on +`` `${broadcast}\0${track}` `` with no generation, so a republish serves the old +track's `TRACK_INFO`. Rescope to that one cache. It is local state with a local +fix (invalidate on the front change the other two sites already observe), and it +needs no wire-level identity, so do not block it on the Rust side. + ### Issue context #### Problem diff --git a/quest/m1/3190-align-origin-broadcast-creation-naming-across-language.md b/quest/m1/3190-align-origin-broadcast-creation-naming-across-language.md index 380b76be629..010478f968f 100644 --- a/quest/m1/3190-align-origin-broadcast-creation-naming-across-language.md +++ b/quest/m1/3190-align-origin-broadcast-creation-naming-across-language.md @@ -11,6 +11,21 @@ Use the public issue's scope, implementation notes, and acceptance criteria below as the starting plan. Reconcile paths and assumptions with the current tree before implementation. +### After prefix routes + +This is now a semantics divergence, not only a naming one. +[moq#3225](https://github.com/moq-dev/moq/pull/3225) made Rust's +`origin::Producer::create_broadcast` stop announcing, while the FFI's +`create_broadcast` still announces the exact path on the caller's behalf (it +calls `announce(path, Default::default())` and hands back the guard), so +wrappers keep announcing by convention. Two surfaces now share one name and do +different things. + +Settle that before the rename: either the FFI's auto-announce becomes explicit +in the wrappers too, or the fused operation keeps a distinct name of its own. +Renaming both to `create_broadcast` without deciding hides the difference behind +matching spelling, which is worse than the inconsistency it fixes. + ### Issue context #### Problem diff --git a/quest/m2/3193-expose-a-cancellable-route-watch-api-in-python.md b/quest/m2/3193-expose-a-cancellable-route-watch-api-in-python.md deleted file mode 100644 index bec563a3a3c..00000000000 --- a/quest/m2/3193-expose-a-cancellable-route-watch-api-in-python.md +++ /dev/null @@ -1,50 +0,0 @@ -# [M] Expose a cancellable route watch API in Python - -## Goal - -Implement and verify the behavior tracked in [#3193](https://github.com/moq-dev/moq/issues/3193) -within the issue's stated scope and boundaries. - -## Plan - -Use the public issue's scope, implementation notes, and acceptance criteria -below as the starting plan. Reconcile paths and assumptions with the current -tree before implementation. - -### Issue context - -#### Problem - -Python handles route changes differently from the other bindings. `BroadcastConsumer.route_changed()` lazily creates a route watch, caches it in a private field, and awaits its next value: - -- [Python `BroadcastConsumer` route handling](https://github.com/moq-dev/moq/blob/7494084aaf7e2fa6abe553ac83101ac4ef19f33a/py/moq-rs/moq/subscribe.py#L376-L405) - -The watch handle is not public and has no deterministic cancellation path. A caller that abandons route observation must rely on garbage collection or broadcast shutdown to release the native watch. - -Other wrappers expose explicit lifetime ownership: - -- [Go `RouteWatch.Next(ctx)` and `Cancel`](https://github.com/moq-dev/moq/blob/7494084aaf7e2fa6abe553ac83101ac4ef19f33a/go/wrapper/subscribe.go#L15-L40) -- Swift exposes a cancellable asynchronous sequence. -- Kotlin exposes a `Flow` that cancels the watch in `finally`. - -#### Proposed direction - -Expose a public Python route watch abstraction with deterministic cleanup, following the existing wrapper conventions. Suitable shapes include: - -- an async iterator returned by `routes()` -- an async context-managed `RouteWatch` -- both, with `route_changed()` retained only as a convenience over the owned watch - -The owner should be able to cancel a pending wait and release the native resource without waiting for garbage collection. - -#### Acceptance criteria - -- Python callers can explicitly own and close a route watch. -- Abandoning iteration deterministically cancels the native watch. -- A pending route wait is cancellable by normal asyncio task cancellation. -- Tests cover early iterator exit and cancellation while waiting. -- Lifecycle documentation matches the Go, Swift, and Kotlin ownership contract. - -## Closes - -- [#3193](https://github.com/moq-dev/moq/issues/3193) - close this issue when the quest finishes diff --git a/quest/m2/README.md b/quest/m2/README.md index a4e4737de42..026c9b9228e 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -48,7 +48,6 @@ dashboards, fleet rollout) stay downstream. - [#2907](/quest/m2/2907-bind-the-browser-through-moq-ffi-uniffi-instead-of-a.md) - Bind the browser through moq-ffi/UniFFI instead of a second hand-written wasm API - [#2822](/quest/m2/2822-moq-wasm-bind-the-datagram-path-append-datagram-recv.md) - moq-wasm: bind the datagram path (append_datagram / recv_datagram) - [#2835](/quest/m2/2835-moq-wasm-bind-track-dynamic-so-a-browser-publisher-can.md) - moq-wasm: bind track::Dynamic so a browser publisher can serve cache-miss fetches -- [#3193](/quest/m2/3193-expose-a-cancellable-route-watch-api-in-python.md) - Expose a cancellable route watch API in Python - [#3189](/quest/m2/3189-add-uniffi-defaults-to-caller-constructed-configuration.md) - Add UniFFI defaults to caller-constructed configuration records - [Cluster flags](/quest/m2/cluster-flags.md) - a discovery mechanism carries its own prerequisites, so an incomplete cluster config cannot be expressed - [#3058](/quest/m2/3058-moq-relay-a-revalidation-re-check-cannot-update-a.md) - moq-relay: a revalidation re-check cannot update a session's tier, and changes its alias only by closing it diff --git a/quest/m2/pop-skipping/README.md b/quest/m2/pop-skipping/README.md index d0590517cad..3aa51953ea2 100644 --- a/quest/m2/pop-skipping/README.md +++ b/quest/m2/pop-skipping/README.md @@ -26,8 +26,69 @@ For `sjc0 -> dal0 -> iad0`, with the publisher in IAD: ## Plan +### Where this stands after prefix routes + +[moq#3225](https://github.com/moq-dev/moq/pull/3225) made an announcement a +route over a path *prefix* and deleted the machinery this questline had +already landed: the warm-cost discount, `COST_LINGER`, the +`(cold, hash)` adoption gate, the handover hold, and the per-broadcast front +that hosted all of it. A relay now forwards accumulated costs only. + +What survives is the part that was expensive to get right: + +- `origin::Cost { warm, cold }` on the wire, decoded on lite-06, with + `Cost::UNKNOWN` reading an inexpressible cold as the ceiling rather than as + free. +- `route_order`, which still breaks a warm tie on the lower cold cost ahead of + hop count. +- `DRAIN_COST`, and the hop list as the loop check. +- The link-price decisions below, which were rulings about the topology rather + than about the code that read them. + +So this questline is no longer "add a rank to a working discount". It is +re-deriving warmth on a route model that prices prefixes, then putting the +adoption gate back on top. + +### The tension prefix routes introduce + +Warmth is a property of one broadcast. A route covers a prefix, which is a +claim about a set of paths. A relay carrying `pid/foo.hang` knows nothing about +the rest of `pid/`, so there is no honest way to discount the prefix route it +already advertises: doing so would attract subscribers for every cold path +underneath it. + ### Decisions +- **A carrying relay advertises the exact broadcast path as its own route, + priced warm.** Per-broadcast warmth becomes a more specific route rather than + a discount on a broader one. Nothing new goes on the wire, and the existing + selection rule already prefers it. +- **Selection keeps specificity ahead of cost.** `best_server` filters to the + longest covering prefix and only then orders by `route_order`, and the lite + draft says the same. This matches longest-prefix-match everywhere it appears + (IP forwarding, BGP, DNS closest encloser, URL routers), and for the same + reason: routes of different prefix length describe different destination + sets, so comparing their costs asks "what does this broadcast cost" against + "what would anything under here cost". Cost decides between routes that cover + the same thing, which is exactly where the warm/cold pair was designed to + work. +- **A draining carrier retracts its exact-path route rather than repricing + it.** Under specificity-first, forgoing the discount is not enough: a route + priced at the ceiling still wins on specificity and keeps attracting + subscribers a drain is trying to move. Retraction drops the claim, and the + broader route the content is still reachable through takes over. This + replaces the draft's ceiling-exemption paragraph, which was written when the + discount rode a single per-broadcast advertisement. +- **Adoption and resume need different identities.** Adoption keys on the + *last* hop of a route, the peer that advertised it and the parent a relay + would be adopting; resuming a subscription keys on the *first* hop, who + produced the content, which alternate routes to one publisher deliberately + share. The pre-#3225 front kept both (`FrontState.publisher` for the first, + `handover_allowed` and the hold for the last), with `same_identity` comparing + either and refusing `Hop::UNKNOWN` on both. [Route resume + identity](/quest/m0/route-resume.md) restores that comparison rule and the + publisher half as a failover fix; [Rank](/quest/m2/pop-skipping/rank.md) owns + the carrier half rather than reusing the wrong one. - The operator hand-authors one undirected base graph. Same-PoP connectivity is unconditional, not a self-edge operators must remember. The radius-two closure and shortest base-graph distance are derived and validated. @@ -39,48 +100,19 @@ For `sjc0 -> dal0 -> iad0`, with the publisher in IAD: is made on price, not on the hop-count tie-break below it. - No link is free, including a same-PoP one. A local transfer still costs a NIC, a hop of latency, and a copy; what is genuinely free is bytes already flowing, and - that is the warm discount's job, not the link price's. A floor of 1 also prices + that is the warm route's job, not the link price's. A floor of 1 also prices chain *length*, so a PoP converges on a flat tree around one puller instead of daisy-chaining at no cost, and it means adopting a parent strictly increases the adopter's own cold cost, which leaves the per-broadcast hash as a tie-break between equally-placed relays rather than the only thing keeping the order strict. - Warmth is broadcast-wide: demand for any track makes the broadcast warm. When - demand drains, the discount follows the existing 30-second spliced-track warm + demand drains, the exact-path route follows the existing 30-second spliced-track lifecycle (`TRACK_IDLE_LINGER`) while at least one track copy remains retained. - Remove the separate five-second announcement-only `COST_LINGER`; do not add a - second timer with a different definition. A retained track has canceled its - upstream subscription, so "warm" here intentionally means reusable route/track - state plus hysteresis, not that every future byte is already in memory. -- The hop list remains the steady-state loop detector, but it cannot make two - simultaneous handovers safe: A and B can each advertise a clean path through IAD, - then adopt each other before either new hop list arrives. Adoption therefore - descends a per-broadcast rank `(cold route cost, hash(broadcast path, - relay origin id))`. Lower cold cost wins; equal-cost peers choose the lowest hash. - The winner keeps its cold upstream rather than adopting a higher-ranked peer. - Including the broadcast path distributes ownership instead of making one hot - relay win every broadcast. Marginal cost remains the primary route metric, and - where it ties the *lower* cold cost wins, ahead of hop count. Both tie-breaks - answer the same question, how far away the content is, and cold cost answers it - in the operator's own prices while hop count only counts relays; the priced - answer goes first, and a route straight from the publisher can never lose it, - since at equal marginal cost its cold equals its marginal. -- That rank is only a shared order while the costs behind it are. A relay reports - its own cold cost, so a report still crossing the mesh can be lower than what its - sender would say now, and rings of relays can each rank a stale neighbour below - themselves and all let go at once. Rising costs are the whole hazard: if costs - only fell, a stale value would only make a peer look worse than it is. A relay - therefore holds a re-parent onto another relay for ~500ms and re-evaluates when - the hold expires, rather than acting on the decision that armed it. The hold is - not there to stagger the relays, which a uniform delay cannot do; it outlasts the - propagation of the very costs it is deciding on, so the sizing rule is just - "longer than an announcement crosses the mesh", plus a stable per-relay spread so - a PoP does not reconsider on one instant. It covers only trading a working - upstream for a better one: an idle relay is pulling nothing, a one-hop chain is - the publisher, and leaving a drained or vanished route stays immediate. -- Cluster sessions use `moq-lite-06-wip`, explicitly. Lite05 silently drops - the cost; the MoQT Cluster extension is not the chosen cluster wire for this - quest. + Do not add a second timer with a different definition; `COST_LINGER` was that + timer and is already gone. A retained track has canceled its upstream + subscription, so "warm" here intentionally means reusable route/track state plus + hysteresis, not that every future byte is already in memory. - Provider economics are directional and dominate locality in a mixed-provider deployment. Conceptually the metric is `(serving provider egress class, topology distance)`: pulling from an unmetered relay can be cheaper than the @@ -89,86 +121,40 @@ For `sjc0 -> dal0 -> iad0`, with the publisher in IAD: choose an encoding whose economic stride exceeds the maximum accumulated topology distance allowed by the bounded hop list, which at a 32-entry chain and a 5-cost worst link is 160. +- Cluster sessions use `moq-lite-06-wip`, explicitly. Lite05 silently drops + the cost; the MoQT Cluster extension is not the chosen cluster wire for this + questline. - Fleet rollout stays downstream: moq.pro owns rendering the priced radius-two topology, the two-phase Lite06 cluster cutover, and the staging and live route-verification proofs. This questline completes when the mechanism above lands here. -### Existing foundation and gaps - -moq-relay 0.14.11 / moq-net 0.2.12 (`adad52b2`) already carry cumulative route -costs, `?cost=N` cluster peer URLs, a zero-cost advertisement for a demanded -broadcast, group-boundary route migration without announcement churn, and -hop-list loop rejection. They also carry the simultaneous-warm-peer hash gate. -The missing piece is the rank above: the current gate may keep SJC on IAD based -only on the hash, so it does not guarantee that a cheaper cold parent such as -DAL becomes the aggregation point. +### Peer reconfiguration -The URL-query reconciliation slice landed in -[moq#2874](https://github.com/moq-dev/moq/pull/2874): canonical identity now stays +The URL-backed half is complete in +[moq#2874](https://github.com/moq-dev/moq/pull/2874): canonical identity stays separate from dial configuration, so changing `?cost=` or an inline credential -replaces the active session while an identical render is a no-op. It also preserves -the last-good topology on malformed input, keeps an identical fallback session -alive, redacts credentials from parse errors, and tracks overlapping gossip paths -so an old unannounce cannot stale a replacement. The remaining reconciliation +replaces the active session while an identical render is a no-op. It also +preserves the last-good topology on malformed input, keeps an identical fallback +session alive, redacts credentials from parse errors, and tracks overlapping +gossip paths so an old unannounce cannot stale a replacement. The remaining boundary is structured policy that does not live in the URL, especially selected wire version and the two directional costs of one bidirectional session. -### rank - -Extend the Lite06 WIP route advertisement with the cold rank needed beside its -discounted marginal cost. The cold cost is the relay's best path with warm discounts -removed; it remains observable while the serving path is warm. Compute the hash from -the absolute broadcast path and stable relay origin id locally, so only cold cost -needs new wire state. A carrying relay adopts another carrying relay only when the -peer's `(cold cost, hash)` is strictly smaller. Its own rank, not the rank of a parent -it already adopted, is what it advertises, making every warm edge descend and -preventing cycles. The hop list still rejects a stale or malformed route containing -the receiver. Keep marginal cost first in route selection; where it ties, the -lower cold cost wins ahead of hop count, which is the priced version of the same -"how far away is this" question hop count only estimates. - -Prove the asymmetric chain (DAL outranks SJC regardless of hash), the equal-cost -race (exactly the lower hash keeps IAD while the other adopts it), a three-node -transitive tree, simultaneous updates, and route loss/reversion. A live track must -migrate only at a group boundary and must not see announcement churn. - -### warm-lifecycle - -Expose one broadcast-level warm signal derived from all spliced tracks: true while -any track has demand or retains its source copy inside `TRACK_IDLE_LINGER`, false -only after the last copy expires. Make both Lite and IETF publishers consume that -signal even when a cluster selects Lite06, so the model has one semantic. Delete -the five-second cost-only linger and its parallel deadline machinery. Tests cover -two tracks with staggered demand, demand returning during retention, the last warm -track expiring, a route change during retention, and no repeated re-advertisement -after expiry. - -### peer-reconfigure - -The URL-backed half is complete in -[moq#2874](https://github.com/moq-dev/moq/pull/2874): cost and credential changes -replace the active peer session, identical renders and identical source fallbacks do -not churn it, malformed lists preserve the last-good topology, and overlapping -gossip advertisements cannot tear down the new configuration. Finish the structured -half so an operator's topology renderer can grow provider policy: selected wire -version and each direction's economic cost must participate in reconciliation even -though they are not URL identity. Test both directions of an asymmetric link and a -version-only change; the old session must close once and the replacement must carry -the new values. - ## Quests +- [Warm advertise](/quest/m2/pop-skipping/warm-advertise.md) - a carrying relay + advertises the exact broadcast path as a warm route, and retracts it when + draining or idle - [Rank](/quest/m2/pop-skipping/rank.md) - rank warm relay candidates by cold - cost before hop count, on the Lite06 WIP -- [Warm lifecycle](/quest/m2/pop-skipping/warm-lifecycle.md) - one - broadcast-wide warmth lifecycle for Lite and IETF publishers, replacing the - separate five-second cost linger + cost and adopt only downhill, with a hold that outlasts cost propagation - [Peer reconfigure](/quest/m2/pop-skipping/peer-reconfigure.md) - reconfigure peer policy when negotiated versions or directional costs change, proving both sides of an asymmetric link ## Related +- [Route resume identity](/quest/m0/route-resume.md) - supplies the identity comparison rule adoption reuses on a different hop - [drain](/quest/m2/drain/README.md) - a second relay per PoP makes the same-PoP link price and its connection cardinality operationally important -- [wildcard](/quest/m2/wildcard/README.md) - it reuses this quest's route cost and rank hash, and needs a cluster on Lite06 +- [wildcard](/quest/m2/wildcard/README.md) - it reuses this questline's route cost, and needs a cluster on Lite06 +- [relay-memory](/quest/m2/relay-memory/README.md) - a denser mesh multiplies whatever a non-selected route costs diff --git a/quest/m2/pop-skipping/rank.md b/quest/m2/pop-skipping/rank.md index a63c247fdb0..2da556d86dc 100644 --- a/quest/m2/pop-skipping/rank.md +++ b/quest/m2/pop-skipping/rank.md @@ -2,9 +2,93 @@ ## Goal -Rank warm relay candidates by cold cost before hop count. Carry cold cost -beside marginal cost on Lite06 WIP, prefer the lower cold cost ahead of hop -count where marginal cost ties, make adoption descend `(cold cost, -per-broadcast relay hash)`, and hold a re-parent onto another relay long -enough for the costs it rests on to land, with asymmetric, equal-rank, -transitive, simultaneous-handover, stale-ring, and group-boundary tests. +A relay adopts another relay's warm copy only when that relay is strictly +closer to the publisher, so a PoP converges on one aggregation point instead of +a coin flip, and no two relays can adopt each other. + +## Plan + +Once [Warm advertise](/quest/m2/pop-skipping/warm-advertise.md) lands, two +relays carrying one broadcast both advertise it warm and tie: warm cost cannot +separate them, and only the deterministic hash would, so the aggregation root is +picked at random. In the `sjc0 -> dal0 -> iad0` topology that lets SJC (two +links from IAD) win over DAL (one link), and the cluster carries the extra +backhaul it was supposed to remove. + +Break that tie on cold cost, which is the relay's own distance to the publisher +with warm discounts removed and is already on the wire and already ranked below +warm in `route_order`. Adoption descends `(cold cost, hash(broadcast path, relay +hop id))`: lower cold wins, equal cold takes the lower hash, and a relay keeps +its own upstream rather than adopting a peer that ranks above it. Including the +broadcast path in the hash spreads ownership instead of making one relay win +every broadcast. A relay advertises its *own* rank, not that of a parent it +adopted, so every warm edge descends and cycles cannot form. + +Adopting a parent adds that link to the adopter's cold cost, so it can only rank +above its parent afterwards. That is what makes descent automatic, and it is why +no link may be priced free. + +### The hold, and why it is not optional + +Cold is a value each relay reports about itself, so a report still crossing the +mesh can be lower than what its sender would say now. Rings of relays can each +rank a stale neighbour below themselves and all let go at once, leaving the +broadcast with no source. Rising costs are the whole hazard; if costs only fell, +a stale value would only make a peer look worse than it is. A GOAWAY prices a +route at the ceiling while neighbours still remember it cheap, so the trigger is +a rolling restart, not an exotic race. + +Hold a re-parent onto another relay long enough for the costs it rests on to +land, and re-evaluate when the hold expires rather than committing to the +decision that armed it. The sizing rule is "longer than an announcement crosses +the mesh", plus a stable per-relay spread so a PoP does not reconsider on one +instant. The hold covers only trading a working upstream for a better one: +an idle relay is pulling nothing, a one-hop chain is the publisher itself, and +leaving a drained or vanished route stays immediate. + +### Two identities, not one + +Adoption is a statement about the adjacent relay, and that is a different +identity from the one [Route resume identity](/quest/m0/route-resume.md) +restores. The pre-#3225 front kept both, because they answer different +questions: + +- The **first** hop is who produced the content. Alternate routes to one + publisher deliberately share it, which is what makes them spliceable, and it + is what route-resume keys on. +- The **last** hop is the peer that advertised the route: the parent a relay + would be adopting. `handover_allowed` and the hold both keyed on it, and the + rank hash was taken over it (`fnv_key(name, [peer])`). + +Use the last hop here. Keying adoption on the first hop would make every carrier +of one broadcast look like the same peer, so a changed parent would go +undetected and two relays could adopt each other, which is the failure the hold +exists to prevent. + +What this quest reuses from route-resume is the comparison rule rather than the +field: `Hop::UNKNOWN` identifies nobody and never matches itself, so two +anonymous relays must not pass for one relay reconnecting and skip the gate. + +Update `drafts/draft-lcurley-moq-lite.md` in the same change, restoring the +adoption-rank rule that +[moq#3278](https://github.com/moq-dev/moq/pull/3278) removed. + +### Tests + +The asymmetric chain (DAL outranks SJC regardless of hash), the equal-cost race +(exactly the lower hash keeps its upstream while the other adopts it), a +three-node transitive tree, simultaneous updates, a ring of relays each holding +a stale cheaper report (which must not leave the broadcast sourceless), route +loss and reversion, and an unknown-cold peer losing to a known cheap one in both +hash directions. A live track must migrate only at a group boundary and must not +see announcement churn. + +Write every gate test over both hash directions, or it proves nothing beyond a +lucky hash. + +## Required + +- [Warm advertise](/quest/m2/pop-skipping/warm-advertise.md) - there is nothing + to rank until two relays can both advertise one broadcast as warm +- [Route resume identity](/quest/m0/route-resume.md) - supplies the identity + comparison rule, and the publisher half of it diff --git a/quest/m2/pop-skipping/warm-advertise.md b/quest/m2/pop-skipping/warm-advertise.md new file mode 100644 index 00000000000..55cb00bb0ad --- /dev/null +++ b/quest/m2/pop-skipping/warm-advertise.md @@ -0,0 +1,56 @@ +# [L] Warm advertise + +## Goal + +A relay carrying a broadcast advertises that broadcast's exact path as its own +route, priced warm, so later subscribers reach the copy the cluster already has +instead of opening another one. It retracts that route when the broadcast goes +idle or its own path starts draining. + +## Plan + +Warmth is per-broadcast and a route covers a prefix, so the discount cannot ride +the prefix route a relay already advertises: carrying `pid/foo.hang` says +nothing about the rest of `pid/`. Advertise the exact path instead, as a second, +more specific route. `best_server` already filters to the longest covering +prefix before ordering by cost, so the warm route wins for that one broadcast +and changes nothing for its neighbors. + +Price it the way the deleted discount did: warm zero (the ingress is already +paid for), cold forwarded accumulated, since cold prices the path this relay +would have to open if it were not already carrying. Two relays both carrying +then tie on warm and are separated by cold, which is what +[Rank](/quest/m2/pop-skipping/rank.md) builds on. + +Lifecycle follows the state that already exists rather than a new timer. +The route appears when a track under the broadcast has demand and stays while +any track retains its source copy inside `TRACK_IDLE_LINGER`; it is retracted +once the last copy expires. `COST_LINGER` was the parallel five-second timer +with a different definition and is already gone with +[moq#3225](https://github.com/moq-dev/moq/pull/3225); do not reintroduce it. + +Retract rather than reprice when the serving path drains. Under +specificity-first selection a ceiling-priced exact-path route still outranks +every broader route, so a drain that only repriced would keep attracting the +subscribers it is trying to move. Dropping the claim lets the broader route the +content is still reachable through take over. Update +`drafts/draft-lcurley-moq-lite.md` in the same change: this replaces the +ceiling-exemption paragraph that +[moq#3278](https://github.com/moq-dev/moq/pull/3278) removed, and it is a behavior change the +draft has to carry. + +Make both Lite and IETF publishers advertise from one warmth signal, so a +cluster selecting Lite06 and one on the Cluster extension mean the same thing by +warm. + +Tests: two tracks with staggered demand keep one warm route alive; demand +returning during retention does not churn the advertisement; the route is +retracted after the last copy expires and is not re-advertised afterwards; a +draining path retracts rather than repricing, and a subscriber on it moves to +the broader route; a second relay carrying the same broadcast ties on warm and +is separated by cold. + +## Required + +- [Route resume identity](/quest/m0/route-resume.md) - a subscriber moving + between a warm route and a broader one must resume rather than end diff --git a/quest/m2/pop-skipping/warm-lifecycle.md b/quest/m2/pop-skipping/warm-lifecycle.md deleted file mode 100644 index a1117f5847a..00000000000 --- a/quest/m2/pop-skipping/warm-lifecycle.md +++ /dev/null @@ -1,7 +0,0 @@ -# [M] Warm lifecycle - -## Goal - -Use one broadcast-wide warmth lifecycle for both Lite and IETF publishers. -Replace the separate five-second cost linger with the shared state of any -demanded or 30-second-retained spliced track. diff --git a/quest/m2/relay-memory/README.md b/quest/m2/relay-memory/README.md index 75ccdb895e8..6bfcf914858 100644 --- a/quest/m2/relay-memory/README.md +++ b/quest/m2/relay-memory/README.md @@ -24,11 +24,16 @@ A broadcast holds four to five of those cells (the source's `Shared` and `Alive`, the front's `Shared` and `Alive`, and `FrontState`), so ~4.2 KB of the 8.8 KB is empty slots. [Waiter slots](/quest/m2/relay-memory/waiters.md) is that struct. -The per-route half is separate: every received announcement runs the whole -`create_broadcast` path, minting its own `broadcast::Producer` and lifecycle -task, even for a standby route that is never selected. In a sparse mesh a relay -of degree `d` attaches `d` sources to one front, so degree is a straight -multiplier. [Standby routes](/quest/m2/relay-memory/standby-routes.md) is that half. +The per-route half is closed. It used to be that every received announcement ran +the whole `create_broadcast` path, minting its own `broadcast::Producer` and +lifecycle task even for a standby route that was never selected, so a relay of +degree `d` attached `d` sources to one front and degree was a straight +multiplier. [moq#3225](https://github.com/moq-dev/moq/pull/3225) removed that: +`lite::subscriber` now calls `origin::Producer::announce_served`, so an +announcement costs a `RouteEntry` plus one `ServeState` whose `served` cache +materializes a broadcast only when something requests a path. That is what the +standby-routes quest asked for, delivered as a side effect, and the quest is +closed. A group holds one of the same cells, which is why [group charge](/quest/m0/group-charge.md) is an accounting bug and not only a footprint one. @@ -46,11 +51,11 @@ RAM given to the group cache and health shedding at 75%), announce state has roughly 200 MB of headroom against a warm cache, so the node sheds itself out of serving near 7,000 concurrent cluster-wide broadcasts. -`dev` (`b39fb87`) does not help: it is 10-13% worse per broadcast (9.7 KB at -one route, 4.8 KB per extra route) because the structs grew, and it still -requests announcements at prefix `""`. It does move the per-broadcast -lifecycle off two tokio tasks onto one shared `Driver`, which helps churn but -not footprint. +Every figure above predates prefix routes and is no longer trustworthy: they +were measured on `adad52b`, where an announcement built a broadcast. A +`ServeState` still carries kio state cells, so the waiter-slot lever below still +applies to it, but the per-route multiplier those numbers describe is gone. +Remeasure before treating any of them as current. ### Why now rather than later @@ -62,21 +67,18 @@ Nothing is near the ceiling today. Two committed directions move it: (~19 GB, not a tuning problem). - [Cache-aware PoP skipping](/quest/m2/pop-skipping/README.md) densifies the relay mesh, roughly tripling average degree on the topology it was designed - against. That multiplies exactly the per-route cost [standby - routes](/quest/m2/relay-memory/standby-routes.md) removes, so landing that - quest first keeps PoP skipping from tripling announce memory as a side - effect. + against, and its warm advertisement adds a second, more specific route per + carried broadcast. Both multiply whatever a route costs, which is now a table + entry rather than an object graph. ### Expected result -Per cluster-wide broadcast on a degree-5, 1 GB relay, and the point at which -it sheds: - -| | Per broadcast | Broadcasts | -|---|---|---| -| today | ~29 KB | ~7,000 | -| after [waiter slots](/quest/m2/relay-memory/waiters.md) | ~19 KB | ~11,000 | -| after [standby routes](/quest/m2/relay-memory/standby-routes.md) | ~9 KB | ~23,000 | +The published table compared today, waiter slots, and standby routes at ~29 KB, +~19 KB, and ~9 KB per cluster-wide broadcast on a degree-5, 1 GB relay. Standby +routes has since landed by another route, and the baseline it was measured +against no longer exists, so the remaining claim is only that waiter slots +removes 840 B from every kio channel. Rebuild the harness and restate the table +before quoting a shed threshold. ### Reproducing the measurements @@ -91,8 +93,8 @@ from this quest rather than expecting them in the tree. - [Waiter slots](/quest/m2/relay-memory/waiters.md) - kio channels stop paying 840 B of empty inline waker slots: the largest single lever in the questline -- [Standby routes](/quest/m2/relay-memory/standby-routes.md) - a route a relay - never selects costs a table entry instead of a full broadcast object graph +- [Routes per broadcast gauge](/quest/m2/relay-memory/route-gauge.md) - expose + how many routes a relay holds for a path ## Related diff --git a/quest/m2/relay-memory/route-gauge.md b/quest/m2/relay-memory/route-gauge.md new file mode 100644 index 00000000000..603f507ef2d --- /dev/null +++ b/quest/m2/relay-memory/route-gauge.md @@ -0,0 +1,18 @@ +# [XS] Routes per broadcast gauge + +## Goal + +An operator can see how many routes a relay holds for a path. moq.pro's +(downstream) announce gauge wants it, and nothing exposes it. + +## Plan + +This rode the standby-routes quest as a passenger, because that quest had the +route table open anyway. [moq#3225](https://github.com/moq-dev/moq/pull/3225) +completed standby routes by making a non-selected route a table entry, so the +quest was closed and the accessor needs its own change. + +Expose the count of routes covering a path from `origin`, next to whatever +`best_server` already walks, and surface it wherever `moq-relay` reports node +state. Prefer a count over exposing the entries themselves: the caller is a +gauge, and a routes iterator is a much larger surface to keep stable. diff --git a/quest/m2/relay-memory/standby-routes.md b/quest/m2/relay-memory/standby-routes.md deleted file mode 100644 index 4674f463d1e..00000000000 --- a/quest/m2/relay-memory/standby-routes.md +++ /dev/null @@ -1,60 +0,0 @@ -# [L] Standby routes - -## Goal - -A route a relay never selects costs a table entry rather than a full broadcast -object graph. Degree stops being a straight multiplier on announce memory, which -is what makes hub relays and the denser [PoP-skipping -graph](/quest/m2/pop-skipping/README.md) expensive. - -## Plan - -### What actually happens today - -A relay of degree `d` hears about the same broadcast `d` times: once over the -shortest path and once forwarded by each other neighbor, because split horizon -only suppresses a chain that already passed through us. Every one of those -arrivals runs `Subscriber::start_announce` -> `origin::Producer::create_broadcast`, -which mints a `broadcast::Producer` (two kio state cells plus an `Alive`) and a -lifecycle task, and then `attach_source` files it as one more `FrontRoute` on the -single shared front for that path. - -Only the best route by `route_order` is ever `active`. The other `d - 1` object -graphs exist to hold three facts: this peer also has it, at this cost, via this -hop chain. Measured, that is 4.3 KB each today and ~2.9 KB after [waiter -slots](/quest/m2/relay-memory/waiters.md). - -### The change - -Represent a non-selected route as a record: hop chain, cost, announce id, and a -handle to the session that would serve it. Roughly 250 B. Mint the -`broadcast::Producer` and attach it only when the route is promoted (failover, -or a cheaper route arriving) or first subscribed. Promotion stays local: the -session already exists, so no network round trip is added to failover. - -### What has to keep working - -- `route_order` ranks on announce flag, cost, hop length, the `fnv_key` hash, - and attach recency. All of those are in the record, so ranking must not need - the producer. -- Split horizon and `FrontState::excluded`: exposure is registered when a path - is advertised to a peer, not only when they subscribe, and `taints_a_reader` - decides on hop chains. Confirm a lazy standby still participates, because - getting this wrong hands a peer back its own bytes. -- Lite-06 announce ids map to paths per stream (`announced_by_id`). A record - keeps that mapping without the producer; `ANNOUNCE_RESTART` must still - re-price a standby in place. -- `AnnouncedRoute` currently holds a `broadcast::SourceGuard` from - `create_broadcast`, and dropping it is how a retraction detaches. The record - needs the equivalent teardown with nothing attached. - -### Boundaries - -Regression test that a broadcast announced by `d` peers attaches one source and -`d - 1` records, and that killing the active peer promotes a record without a -gap. The failover path is the one that gets slower if this is done wrong, so -prove it rather than asserting it. - -Add a routes-per-broadcast accessor in the same change: this code is already -open, and moq.pro's (downstream) announce gauge wants it, so it rides this PR -rather than becoming a change of its own. diff --git a/quest/m2/wildcard/README.md b/quest/m2/wildcard/README.md index b5b961196c6..9fb4fe1fb63 100644 --- a/quest/m2/wildcard/README.md +++ b/quest/m2/wildcard/README.md @@ -38,28 +38,35 @@ would have to start working (a cold transcoder)" same exact containment check. [#2925](https://github.com/moq-dev/moq/pull/2925) has since replaced `RouteCost` with `Cost { warm, cold }`. -Matching, by contrast, is prefix-only everywhere today. The pattern matcher is -not this questline's to build: path authorization adopts the same dialect first, -and its [Matcher](/quest/m2/path-patterns/matcher.md) quest delivers the shared -matching, containment, and rebasing that [Advertise](/quest/m2/wildcard/advertise.md) now requires. - -Two things look built and are not: - -1. **A routing table.** `origin::Dynamic` (`rs/moq-net/src/model/origin.rs`) - serves an unannounced path on demand and `lite::publisher::recv_subscribe` - already falls back to it. It is not usable here. Its queue is keyed by path - alone, so requests from peers with different `exclude` origins coalesce onto - one entry; a `Request` carries no requester identity, hop list, or cost; and - every live handler drains one shared FIFO, so N wildcards cannot be ranked. - `request_broadcast` says so itself: "a handler resolves paths with no route - chain to check, so falling through would let it route around the split - horizon and rebuild the loop." -2. **Announcement `Epoch`.** [#2611](https://github.com/moq-dev/moq/pull/2611) - specced it into lite-06 and [#1920](https://github.com/moq-dev/moq/pull/1920) - had earlier removed the one that used to exist in code, so it is specified and - unimplemented. It is absent from `dev` too, whose `AnnounceBroadcast::Active` - carries `{ suffix, hops, cost }` and no epoch. `broadcast.rs`'s `route_epoch` - is an unrelated route-change counter. [Epoch](/quest/m2/wildcard/epoch.md) owns closing that gap. +[moq#3225](https://github.com/moq-dev/moq/pull/3225) moved a long way toward +this. An announcement is now a route over a path *prefix*, `origin::Prefix` is +an opaque newtype built explicitly as the extension point a pattern type slots +into, and matching is segment-wise intersection in one place. So the wire, the +model, and every binding already speak in covering claims rather than in +per-broadcast announcements. + +The routing table exists too, which it did not when this questline was written. +`Consumer::request_broadcast` resolves a local broadcast first, then +`best_server`: the longest covering prefix, filtered by the requester's excluded +hop, ordered by `route_order`, served on demand by the session that announced it +and cached per prefix in `ServeState.served`. That is the split-horizon-safe +lookup the old `origin::Dynamic` could not provide, and it is what +[Resolve](/quest/m2/wildcard/resolve.md) now extends rather than replaces. + +Matching, by contrast, is prefix-only. The pattern matcher is not this +questline's to build: path authorization adopts the same dialect first, and its +[Matcher](/quest/m2/path-patterns/matcher.md) quest delivers the shared +matching, containment, and rebasing that +[Advertise](/quest/m2/wildcard/advertise.md) requires. + +What is genuinely missing, beyond patterns themselves, is content identity. +Announcement `Epoch` was specified into lite-06 by +[#2611](https://github.com/moq-dev/moq/pull/2611), never implemented, and +removed from the draft by #3225, which retired `draft-lcurley-moq-broadcast` +with it. [Route resume identity](/quest/m0/route-resume.md) restores per-path +identity from the route's first hop, reversing #3225's no-splice rule, and +this questline builds its collision handling on that rather than on a +generation field. ### Decisions @@ -69,9 +76,12 @@ Two things look built and are not: shared matcher, so nothing resembles a second grammar. Exact set-valued rebasing preserves every match inside a rooted view, including both the root and deeper residuals when `**` consumes zero or more segments. -- **Most specific pattern wins, and its refusal is final.** When several - patterns match one path, only the tier selected by the matcher's shared - structural specificity is consulted; equal-specificity patterns +- **Most specific pattern wins, and its refusal is final.** This is the rule + routing already follows: `best_server` filters to the longest covering prefix + before it compares cost, and the lite draft says the same, matching + longest-prefix-match wherever it appears. When several patterns match one + path, only the tier selected by the matcher's shared structural specificity is + consulted; equal-specificity patterns form one pool that cost and the request hash order. A terminal refusal from the winning tier IS the answer and never falls through to a less specific pattern, so a transcoder refusing a path does not leak the request to the @@ -91,16 +101,23 @@ Two things look built and are not: same way. Cost orders the pool first, which keeps work local and makes a distant advertiser the overflow rather than an equal peer. - **A wildcard is priced, not special-cased.** Within a tier, route selection - stays one comparison on one metric; there is no "concrete beats wildcard" - rule beside it. What makes a running transcode win is that its announcement - is seeded live while a standby wildcard is seeded high (`with_cost(1000)` is - the existing convention for a production-cost seed). The seed has a floor: it - MUST exceed the maximum accumulated topology cost a bounded hop list can - reach (`MAX_HOPS` is 32 and the planned link costs are 1/3/5, so the ceiling is 160), or a nearby - standby outranks a distant running transcode and the mesh starts a second - encode of a stream it is already serving. That floor replaces the ad-hoc - standby bias the moq.pro (downstream) transcode worker carries today, and it - is the same stride discipline + stays one comparison on one metric. Concrete-versus-wildcard is not decided + by price at all: a concrete claim is maximally specific, so "most specific + wins" above already shadows every pattern behind it at any cost. The + accepted consequence follows from that rule's finality: a live session's + concrete claim shadows a healthy wildcard pool even when its service is + broken, its terminal refusal does not fall through, and the shadow lasts + exactly as long as the claiming session that carries it. + The seed still has a floor, because standby and running claims of equal + specificity do meet: a standby concrete claim (`with_cost(1000)` is the + existing per-broadcast convention) shares a tier with a running publisher's + concrete announcement and with warm-advertise's exact-path warm routes. The + floor MUST exceed the maximum accumulated topology cost a bounded hop list + can reach (`MAX_HOPS` is 32 and the planned link costs are 1/3/5, so the + ceiling is 160), or a nearby standby outranks a distant running copy and the + mesh starts a second encode of a stream it is already serving. That floor + replaces the ad-hoc standby bias the moq.pro (downstream) transcode worker + carries today, and it is the same stride discipline [pop-skipping](/quest/m2/pop-skipping/README.md) states for provider economics. - **One cost varint, not the pair.** `Cost` is `{ warm, cold }` because a relay @@ -148,20 +165,18 @@ Two things look built and are not: permanent, so a new refusal mode surfaces instead of quietly joining a retry loop. No negative cache either way; rate limiting stays with the advertiser and the per-project auth gate. -- **A double claim is settled by the service's `Epoch` contract.** Two relays can - hash one path to different workers before either concrete announcement - propagates, and both concrete announcements land at the SAME literal path. - Deterministic services such as transcode, and external processors whose - configured workers are interchangeable, derive one epoch from the source - generation so a relay may splice between them. Session-local services such as - transcription allocate distinct globally ordered epochs, so the higher - generation wins and consumers reset rather than splice. Wildcard routing does - not invent a lease or choose between those media contracts. This is why - [Epoch](/quest/m2/wildcard/epoch.md) is required by any service that can - produce colliding concrete claims, including transcode, transcription, and - external processors. A service that derives output identity from the source - requires a resolved nonzero source epoch; zero is unspecified and must refuse - derived work rather than splice across an ambiguous restart. +- **A double claim is settled by route identity, not by a lease.** Two relays + can hash one path to different workers before either concrete announcement + propagates, and both land at the SAME literal path. Whichever route wins + selection serves it, and a consumer moves between them only when the winner's + identity is preserved, per [Route resume + identity](/quest/m0/route-resume.md); two distinct workers are two identities, + so the loser's subscribers end and resubscribe rather than being spliced onto + another worker's frames mid-group. Wildcard routing invents neither a lease + nor a generation. This is weaker than the retired `Epoch` design, which could + declare two workers' output interchangeable and splice between them; a service + that needs that guarantee has to carry it in its own media contract, not in + routing. - **The spec home is moq-lite core, mirrored in moq-cluster**, following how route cost landed. moq-lite-06 is still WIP, so this goes into it rather than opening an 07. @@ -191,7 +206,7 @@ overlay, because prefix-only matching needs the variable part of a path trailing. That overlay was not a view transform: `pid/foo` and `.transcode/pid/foo` are separate tree leaves with separate broadcast fronts, so it had to build a logical front across roots that re-owned route selection, -epoch identity, the split-horizon guard, and splicing. The suffix pattern +content identity, the split-horizon guard, and splicing. The suffix pattern deletes all of it while keeping what the mirror bought: - **The grant needs no transform.** The customer addresses @@ -208,8 +223,8 @@ deletes all of it while keeping what the mirror bought: removes project discovery entirely. - **Takeover is single-front.** A worker's concrete announcement lands at the literal path the wildcard served, so wildcard-versus-concrete and - worker-versus-worker collisions are ordinary route selection plus `Epoch` - splicing at one tree node, not a cross-root front. + worker-versus-worker collisions are ordinary route selection at one tree node, + not a cross-root front. What the mirror bought and this deliberately gives up: a customer holding `publish: ["pid/"]` CAN publish `foo.hang/transcode.pro` themselves, competing @@ -238,9 +253,6 @@ than announce state. - [Demand](/quest/m2/wildcard/demand.md) - the browser player subscribes to a catalog-referenced broadcast a wildcard covers, breaking the lazy-rendition deadlock -- [Epoch](/quest/m2/wildcard/epoch.md) - implement moq-lite-06's announcement - `Epoch`, so two publishers colliding on one path resolve instead of both - serving ## Related @@ -250,3 +262,5 @@ than announce state. pattern, and its catalog names the generations a wildcard cannot - [pop-skipping](/quest/m2/pop-skipping/README.md) - it owns the route cost and the rank hash this reuses +- [Route resume identity](/quest/m0/route-resume.md) - the per-path identity two + colliding advertisers are settled by diff --git a/quest/m2/wildcard/advertise.md b/quest/m2/wildcard/advertise.md index 24c001c979f..71ce5ee8e7b 100644 --- a/quest/m2/wildcard/advertise.md +++ b/quest/m2/wildcard/advertise.md @@ -7,9 +7,14 @@ resolving one into a subscription. ## Plan -Following the draft, the message goes in `rs/moq-net/src/lite/announce.rs` -alongside `AnnounceBroadcast`, gated on the lite-06 version check the route -cost already uses so older peers neither send nor receive it. +Following the draft, this goes in `rs/moq-net/src/lite/announce.rs` alongside +`AnnounceBroadcast`, gated on the lite-06 version check the route cost already +uses so older peers neither send nor receive it. + +[Draft](/quest/m2/wildcard/draft.md) settles whether a pattern rides the +existing announce messages' prefix field or arrives as a message of its own. +Either way the skew hazard below applies, and either way the decode side lands +first. That gate is NOT sufficient on its own. `moq-lite-06-wip` is one ALPN with no sub-version, and `AnnounceBroadcast::decode` rejects an unknown message type @@ -24,11 +29,15 @@ wildcards but does not emit them: that is what a rollout actually deploys first. Testing against a receiver built before the message existed only re-demonstrates that it drops the stream, which is why emission has to wait. -The state belongs in `rs/moq-net/src/model/origin.rs` beside the announce tree -rather than inside it: a wildcard is not a node, it covers paths that mostly -do not exist. Keep it in a flat structure keyed by the pattern, matched by the -shared matcher [Matcher](/quest/m2/path-patterns/matcher.md) provides, and -the set is small (one entry per advertiser per service, not per broadcast). +The state has a home already. Since +[moq#3225](https://github.com/moq-dev/moq/pull/3225) a route is a flat +`RouteEntry` keyed by an opaque `origin::Prefix`, deliberately built as the +extension point a pattern type slots into: matching is segment-wise +intersection in one place, so a pattern extends `Prefix` internally without +touching `Route` or any signature. Teach `Prefix` the dialect the shared +[Matcher](/quest/m2/path-patterns/matcher.md) provides rather than adding a +parallel table beside it. The set stays small either way (one entry per +advertiser per service, not per broadcast). A wildcard forwards like an announcement: accumulate the sending peer's declared link cost onto its single varint (`Cost::charged` adds to both halves of a @@ -45,20 +54,22 @@ clamped, so a misconfigured worker fails loudly instead of quietly advertising less than it thinks. The same exact containment handles literal-headed and leading-star patterns without a special authorization rule. -Wildcards are visible to subscribers, so `announced` must surface them as a -DISTINCT event rather than as an available broadcast: a pattern names no -content, and presenting one as a broadcast would have consumers subscribe to -the pattern itself. Scope and rebase each one per subscriber with the matcher's -exact set-valued operation. Aggregate duplicate advertisers of one pattern into a -single presented entry whose withdrawal fires only when the last advertiser -leaves. That aggregation has no other owner, so it belongs here. +Wildcards are visible to subscribers, and #3225 already made that safe: an +announcement is a covering claim, `Consumer::announced` yields +`announce::Update { prefix, route, active }`, and it never yields a +`broadcast::Consumer`. So a pattern needs no distinct event kind; it is another +covering claim in the stream consumers already read, and no consumer can +mistake one for a broadcast. Scope and rebase each per subscriber with the +matcher's exact set-valued operation. Aggregate duplicate advertisers of one +pattern into a single presented entry whose withdrawal fires only when the last +advertiser leaves. That aggregation has no other owner, so it belongs here. Tests: encode/decode round trip and the version gate, cost accumulation across two hops, reflected-wildcard drop, per-subscriber exclusion, retraction, scope -refusal, root and descendant residuals from a `**` rebase, and the distinct -wildcard event itself: never presented as an -available broadcast, duplicates aggregated into one entry, withdrawal firing -only when the last advertiser leaves. +refusal, root and descendant residuals from a `**` rebase, duplicates +aggregated into one entry, and withdrawal firing only when the last advertiser +leaves. Cover that a pattern and a literal prefix coexist in one route table +and that a literal-only deployment behaves exactly as it does today. ## Required diff --git a/quest/m2/wildcard/demand.md b/quest/m2/wildcard/demand.md index b464cc3eb0d..5b2c3ff9e0a 100644 --- a/quest/m2/wildcard/demand.md +++ b/quest/m2/wildcard/demand.md @@ -1,4 +1,4 @@ -# [M] Demand +# [S] Demand ## Goal @@ -9,24 +9,25 @@ player never demands what it hides. ## Plan -The gate is JS-only. `js/watch/src/broadcast.ts` resolves -a catalog reference in `#relativeTarget`, and its `#isPathAnnounced` check -drops any rendition whose broadcast has no EXACT-path announcement: -`#runFiltered` removes it from `out.catalog`, so rendition selection never sees -it and nothing ever subscribes. The Rust side needs nothing here: -`moq-mux::Source` resolves references through `request_broadcast`, which -[resolve](/quest/m2/wildcard/resolve.md) teaches to consult the wildcard table. +The gate is JS-only, and it is already half fixed. +[moq#3225](https://github.com/moq-dev/moq/pull/3225) made `#isPathAnnounced` in +`js/watch/src/broadcast.ts` prefix-aware: it holds the set of announced +prefixes and accepts any that covers the path, so a route at `room/` already +makes `room/alice/cam.hang` selectable without naming it. What it cannot do is +match a pattern, since it tests with `Path.hasPrefix`. + +So the remaining work is narrow: teach the JS client the wildcard +advertisement (`js/net`'s announce handling, mirroring what +[advertise](/quest/m2/wildcard/advertise.md) does in moq-net) and make the +covering test use the shared pattern matching rather than prefix containment. +Withdrawal of the last covering wildcard hides the rendition again, the same +reactive shape announcements have today. Do not simply delete the gate. It exists so the player does not subscribe to absent broadcasts and so renditions appear and disappear reactively with -announcements, and `moq-lite`'s own convention (await `announced_broadcast`, -then `request_broadcast` downstream) exists because a blind request races the -first announce handshake. Instead, teach the JS client the wildcard -advertisement (`js/net`'s announce handling, mirroring what -[advertise](/quest/m2/wildcard/advertise.md) does in moq-net) and make `#isPathAnnounced` count a -pattern covering the path as available. Withdrawal of the last covering -wildcard hides the rendition again, the same reactive shape announcements have -today. +announcements. The Rust side needs nothing here: `moq-mux::Source` resolves +references through `request_broadcast`, which +[resolve](/quest/m2/wildcard/resolve.md) teaches to consult patterns. Two existing soft spots to not reintroduce: the first evaluation runs before the announcement stream has populated, briefly hiding cross-broadcast diff --git a/quest/m2/wildcard/draft.md b/quest/m2/wildcard/draft.md index 06e3f476df1..8cac2de2d3e 100644 --- a/quest/m2/wildcard/draft.md +++ b/quest/m2/wildcard/draft.md @@ -13,6 +13,16 @@ route cost appears in both. What the text has to settle: +- **Whether a pattern is a new message or the existing prefix field.** + [moq#3225](https://github.com/moq-dev/moq/pull/3225) reinterpreted the + announce suffix as a route prefix without changing a byte, so a pattern could + ride ANNOUNCE_START's existing field rather than arriving as its own message. + That is the smaller wire change and reuses retraction and repricing for free, + but an older lite-06 build reads `**` as a literal segment and silently + mis-covers rather than failing, where an unknown message type at least fails + loudly. Settle this explicitly, and say which failure mode is preferred; the + staged decode-tolerance rollout in + [Advertise](/quest/m2/wildcard/advertise.md) is required either way. - The message itself: a path pattern in its string form, the hop list, and ONE cost varint, retracted and replaced through the same id-referencing forms ANNOUNCE_END and ANNOUNCE_UPDATE already use. The pattern is the @@ -29,16 +39,16 @@ What the text has to settle: varint rather than `Cost`'s warm/cold pair: the halves differ only when the sender is carrying the broadcast, and a wildcard carries nothing, so a second field would be provably equal to the first. -- `Epoch` has no meaning for a wildcard, because there is no generation of - content at a pattern. Say so explicitly rather than leaving a field a - receiver might splice on. - Remove the `Ended` flag from ANNOUNCE_REQUEST in the same lite-06 edit. It is unimplemented everywhere, and its one stated use, enumerating available recordings, is per-recording announce state, the fleet-wide growth wildcards exist to stop. A client that must distinguish recording generations reads them from the catalog the archive serves, not from announcements. -- Selection is two rules, in order. First, when several patterns match one +- Selection is two rules, in order, and the first is already the draft's rule + for literal prefixes: the most specific covering claim is consulted before + cost. State that a pattern participates in that same specificity ordering + rather than in a rule of its own. When several patterns match one path, only the tier selected by the matcher's shared structural specificity is consulted, with equal-specificity patterns forming one pool. A refusal from that tier never falls through to a less specific diff --git a/quest/m2/wildcard/epoch.md b/quest/m2/wildcard/epoch.md deleted file mode 100644 index 3aafc003cfc..00000000000 --- a/quest/m2/wildcard/epoch.md +++ /dev/null @@ -1,49 +0,0 @@ -# [L] Epoch - -## Goal - -Implement the announcement `Epoch` that moq-lite-06 specifies, so two publishers -colliding on one path resolve instead of both serving. - -## Plan - -The spec exists and the code does not. -[#2611](https://github.com/moq-dev/moq/pull/2611) added epochs and ended -broadcasts to the lite-06 draft, while -[#1920](https://github.com/moq-dev/moq/pull/1920) had earlier removed the -epoch that used to be in the wire. `AnnounceBroadcast::Active` carries -`{ suffix, hops, cost }` on both `main` and `dev`; `broadcast.rs`'s `route_epoch` -and `routes_epoch` are unrelated route-change counters and are not this. - -Adding a field to ANNOUNCE_START and ANNOUNCE_UPDATE has the same skew hazard -[advertise](/quest/m2/wildcard/advertise.md) states for the wildcard message: Lite06 is one ALPN -with no sub-version, and an earlier Lite06 peer reads the extra varint as the -next message and kills the stream. Stage it the same way, decode tolerance -fleet-wide before any sender emits the field, and test the sender against the -intermediate accept-only build. - -Implement what the draft already states: a generation on ANNOUNCE_START and -ANNOUNCE_UPDATE, forwarded unchanged by relays; the highest epoch wins and a -lower one never displaces it; equal non-zero epochs declare interchangeable -content that a relay MAY splice at a group boundary; anything else is two -generations colliding, which MUST NOT splice and MUST end the lower rather than -wait for it to drop. Zero falls back to the first hop entry as identity. The -draft's recommended construction (milliseconds shifted left 16 with random low -bits, clamped above the highest observable epoch) covers restarts without -persisted state. - -For transcode workers the point is the interchangeable case: two workers -that both claim one path publish concrete announcements at the SAME literal -path (the contribution is published where it is addressed, so there is no -cross-root front to reconcile) and derive the SAME epoch from the source -generation, so the loser's viewers migrate at a group boundary rather than -being torn down. -That only holds because the transcode contract requires its redundant output to -be interchangeable. Session-local services with distinct ordered epochs are -also permitted; those replace the lower generation and never splice. -Test both generic Epoch behaviors rather than assuming either at the routing -layer. - -Cover: a higher epoch displacing a lower, a lower one being ignored, equal -epochs splicing mid-stream without a visible gap, unequal epochs ending the -loser promptly, and the zero fallback. diff --git a/quest/m2/wildcard/resolve.md b/quest/m2/wildcard/resolve.md index c41fcdc75ea..e27e34b3ae4 100644 --- a/quest/m2/wildcard/resolve.md +++ b/quest/m2/wildcard/resolve.md @@ -7,26 +7,21 @@ matching wildcard. ## Plan -`lite::publisher::recv_subscribe` already falls back to -`Consumer::request_broadcast`, which drains an `origin::Dynamic` handler that -nothing installs, so the serving half looks done. Do NOT build on it as it -stands. Its request queue is keyed by absolute path alone, so requests from -consumers with different `exclude` origins coalesce onto one entry; a `Request` -carries the path and nothing else, no requester identity, hop list, or cost; and -every live handler drains one shared FIFO, so installing one handler per -received wildcard means whichever polls first wins rather than whichever the -cost ranks. `request_broadcast`'s own comment names the consequence: "a handler -resolves paths with no route chain to check, so falling through would let it -route around the split horizon and rebuild the loop." +[moq#3225](https://github.com/moq-dev/moq/pull/3225) built the table this quest +was going to need. `Consumer::request_broadcast` resolves a local broadcast +first, then `best_server`, which filters routes to those covering the path, +drops any whose hop chain contains the requester's excluded hop, keeps the +longest covering prefix, and orders the survivors by `route_order`. The winning +session serves the request on demand, and `ServeState.served` caches the result +per path so repeat requests share one upstream subscription. The old +`origin::Dynamic` objection (one shared FIFO, no requester identity, no route +chain to check) no longer applies, because that is not the path a route takes. -So wildcard lookup is a first-class origin routing table, sitting beside the -announce tree and consulted by the same resolution path that already returns -`Found`/`Excluded`/`Missing`. It holds each wildcard's pattern, hop list, and -accumulated cost, and resolution takes the requester's -excluded origin as an argument the way `resolve_broadcast` already does. -Candidates whose hop list contains the requester are filtered BEFORE selection, -so an out-of-band request can never be served back through the peer that made -it. +So this quest extends a working table rather than standing one up: teach the +route entries to hold a pattern instead of only a literal prefix, and teach +selection the tiering and pooling below. Keep the exclusion filter where it is, +applied before selection, so an out-of-band request can never be served back +through the peer that made it. Among the survivors, only the tier selected by the matcher's shared structural specificity is consulted, with equal-specificity patterns forming one pool. A @@ -54,9 +49,9 @@ arriving later lands on that same node, competes on cost at one front, and the front's ordinary route change moves consumers at a group boundary. A cache-only answer would strand every bound consumer on the wildcard subscription with nothing able to migrate or stop it. When the concrete claim -is a DIFFERENT publisher, migrating its consumers additionally needs -[Epoch](/quest/m2/wildcard/epoch.md); this quest's obligation is the route -install that makes both possible. Repeat requests for one path +is a DIFFERENT publisher, its consumers end and resubscribe rather than +splicing, per [Route resume identity](/quest/m0/route-resume.md); this quest's +obligation is the route install that makes selection between them possible. Repeat requests for one path share that one route, keyed to survive the exclusion filter rather than handing one peer's answer to another. @@ -80,8 +75,9 @@ Tests, at the process level with real sessions rather than an in-process stand-i - Two requesters with different excluded origins do not receive each other's resolved broadcast. - A concrete announcement from the SAME publisher takes over from the wildcard - route at a group boundary, without announcement churn; the cross-publisher - case needs `Epoch` and belongs with [Epoch](/quest/m2/wildcard/epoch.md). + route at a group boundary, without announcement churn; a concrete claim from a + DIFFERENT publisher ends the wildcard-served subscription instead of splicing + into it. - A FETCH for an unannounced archived path resolves through the catch-all the same way a subscribe does. - A capacity refusal re-resolves onto another advertiser exactly once, and a @@ -101,3 +97,5 @@ Tests, at the process level with real sessions rather than an in-process stand-i ## Required - [Advertise](/quest/m2/wildcard/advertise.md) +- [Route resume identity](/quest/m0/route-resume.md) - a wildcard-served + subscription must resume or end correctly when a concrete claim arrives