diff --git a/AGENTS.md b/AGENTS.md index 288ff6983f2..a37ffaabbcc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,6 +37,7 @@ The API is the most important thing to get right: a bad shape costs a breaking c PRs target `main`, where breaking API changes are allowed. `release` is what ships; touch it only to cut a release or backport a fix or benchmark (see `CONTRIBUTING.md`). Wire changes stay backwards compatible with *published* drafts/versions; an unpublished `-wip` version changes freely. +moq.pro rolls out with relays on the newest and previous release side by side, so those two versions must interoperate as relays (today moq-lite-06 and moq-lite-07). A temporary migration for that is fine when it is clearly marked and easy to delete at the next version migration. Our libraries prefer moq-lite, so moq-transport compatibility only matters against third-party implementations. Build output may live outside the checkout and gets wiped, so scratch never goes in `target/`. diff --git a/quest/m1/README.md b/quest/m1/README.md index 11e8343ac1f..9b974ad4274 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -33,6 +33,7 @@ moves what remains of them once their claims merge. - [Drop set](/quest/m1/drop-set.md) - the drop set's track and cursor runs are billed to the cache pool, so fragmenting drops costs what the pool holds; eviction folds final runs or ends the cursor with `TooFarBehind` - [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation setup +- [A withdrawn claim ends across mixed lite-06 and lite-07 relays](/quest/m1/mixed-version-ring.md) - lite-07 carries a temporary lite-06 compatibility chain, so a rolling upgrade's withdrawn broadcast stops circling - [A reader on a failed copy continues on the held re-offer](/quest/m1/copy-recover.md) - a live reader keeps a group whose copy fails by continuing on the same track's re-offer at the same frame - [Group errors are cached by kind](/quest/m1/group-error-cache.md) - permanent group errors bind every reader, retryable ones are kept only for a bounded time, and a live SUBSCRIBE_DROP or FETCH answered `Old` no longer caches it for good - [The overcommitted receive-window cells run again](/quest/m1/receive-windows-evicted.md) - `benchmark_receive_windows` measures its 256-stream and 4 MiB-frame cells again, reporting delivered frames and evictions instead of stalling diff --git a/quest/m1/mixed-version-ring.md b/quest/m1/mixed-version-ring.md new file mode 100644 index 00000000000..34aa381e1e0 --- /dev/null +++ b/quest/m1/mixed-version-ring.md @@ -0,0 +1,92 @@ +# [L] A withdrawn claim ends across mixed lite-06 and lite-07 relays + +## Goal + +After a publisher leaves, its broadcast ends at every relay within a bounded +number of route updates, for any mix of moq-lite-06 and moq-lite-07 relay +links. That includes a rolling upgrade, where one lite-06 relay links to +three or more lite-07 relays. Today such a claim circles forever after its +withdrawal, and every subscriber sees a Restart on every lap. + +Not here: relay links over moq-transport. moq-transport is supported for +clients, not relays (decided 2026-11-08), so its anonymous chains stay out of +scope. + +## Plan + +Root cause, reproduced 2026-11-08 on main at `8065e7148f`. The loop has been +on main since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023), +which added the lite-06 translation and its single-link reflection check: + +- A route learned over lite-06 becomes the translating relay's own claim + (`OwnClaims::apply` in `rs/moq-net/src/model/origin.rs`, the `learned` + flag), and its lite-06 hop chain is dropped there. +- Translating a lite-07 route back out to lite-06 sends only its origin + node's hop (`hops()` in `rs/moq-net/src/lite/subscriber/routed.rs`). +- So `Subscriber::chain` (`rs/moq-net/src/lite/subscriber.rs`) only catches a + claim that went out and came back across one lite-06 segment. Once a claim + crosses two lite-06 to lite-07 boundaries, no relay sees its own hop again. + In A-07-B-06-C-07-D-06-A, B and D keep re-originating each other's copy. +- Minting a `~` instance per lite-06 hop only turns each lap into a Restart. + It does not cause the loop, and the rising cost saturates only near + `MAX_COST`, so in practice the loop never ends. + +Announcing settles in every topology probed; only withdrawal loops. It loops +in a ring with two lite-06 links separated by lite-07 links (three behave the +same), and when one lite-06 relay links to three or more lite-07 relays, +wherever the publisher sits. A 07-06-06 triangle, a ring with adjacent lite-06 +links, and an old relay with two lite-07 neighbours all settle. + +Decided 2026-11-08 (maintainer, /quest-plan): + +- ✅ Carry the lite-06 chain through lite-07, as a crate-private chain plus a + lite-07-wip compatibility extension and field. Rejected: a cost or hop + ceiling with no wire change, which still circles for hundreds to thousands + of laps and is an arbitrary constant; never forwarding a lite-06-learned + claim to another lite-07 relay, which breaks old-new-new chains mid-upgrade; + documenting mixed rings as unsupported, which a rolling upgrade cannot + avoid. A claim learned over lite-06 keeps that chain. lite-07 + ANNOUNCE_START, ANNOUNCE_RESTART and ANNOUNCE_UPDATE carry it, empty for a + native claim. Translating back to lite-06 sends the chain plus this node's + hop. Both receive sides drop a chain that names them: lite-06 in `chain()`, + and lite-07 when it serves the route. `max_hops` then bounds the total + across islands, and the existing route-hop check becomes one case of it. +- ✅ It is temporary: the draft and code mark the extension and field as + lite-06 compatibility, and the lite-07 to lite-08 migration deletes them. + No public API carries the chain. Keep it apart from the minted flag that + `quest/m2/instance-provenance.md` plans (on + [moq-dev/moq#8266](https://github.com/moq-dev/moq/pull/8266)'s branch + until it merges), even though a non-empty chain marks exactly the + announcements that flag would: the flag also covers moq-transport routes + and outlives lite-06. +- ✅ m1 (rejected: m0, m2). lite-07 is opt-in `-wip`, so nothing published + breaks, and the field must exist before + [lite-07 is finalized](/quest/m2/lite07-finalize.md). + +A throwaway prototype, with a process-wide table standing in for the wire +field, made every looping topology settle in 4 to 13 events after withdrawal. +All integration and lib tests still passed. It needed the lite-07-side drop, +or it tripped `origin.rs`'s "announce called with a looping hop chain" +debug assertion. A chain change on an open session also needs +ANNOUNCE_UPDATE to carry the field. + +Tests, in `rs/moq-net/tests/integration/mesh_withdraw.rs` beside +`mixed_version_ring_withdraw_retracts`. Each announces at one relay, checks +that every relay sees it start, withdraws it, and checks that every relay +sees it end: + +- The ring A-07-B-06-C-07-D-06-A. +- A rolling upgrade with the publisher at the lite-06 relay, linked to each + relay of a three-relay lite-07 full mesh. This one hung 12 of 12 runs, so + it is the reliable gate. +- The same rolling upgrade with the publisher at a lite-07 relay. It hung 11 + of 12 runs, because route identities are random and tie-breaks hash them. + Make it deterministic or drop it. + +The js/net lite-07 port in [routes/js.md](/quest/m2/cluster-routing/routes/js.md) +decodes the field; JS translates nothing, so it keeps no chain. + +## Related + +- [Route session integration](/quest/m2/cluster-routing/routes/session.md) - makes `Hops` crate-private behind a chain type, which this fills from a third source, and merges the lite-06 and lite-07 identities, which folds the two receive checks together +- [Routes acceptance](/quest/m2/cluster-routing/routes/acceptance.md) - these tests are its kind of fixture, a stale neighbour unable to revive an ended broadcast diff --git a/quest/m2/cluster-routing/routes/js.md b/quest/m2/cluster-routing/routes/js.md index 2c5126b740e..4dd74db6ba1 100644 --- a/quest/m2/cluster-routing/routes/js.md +++ b/quest/m2/cluster-routing/routes/js.md @@ -18,6 +18,7 @@ Decided 2026-10-27 (maintainer), from the cheapest one; fix it with the route-selection rework here. - Feed the availability API from [Announcement availability](/quest/m2/cluster-routing/routes/availability.md). - Rust/JS interop covers the route messages, the floor marker, and the echo. +- Decode the temporary lite-06 compatibility chain that lite-07 announcements carry ([mixed-version ring](/quest/m1/mixed-version-ring.md)). JS translates nothing, so it keeps no chain. - Restore the native JS lite-07 lanes. Decided 2026-11-06 (maintainer, /quest-complete on [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)), asked how to handle the in-tree lanes running JS on lite-07-wip once the @@ -34,6 +35,7 @@ the cheapest one; fix it with the route-selection rework here. ## Required - [Announcement availability](/quest/m2/cluster-routing/routes/availability.md) - the API this feeds +- [Mixed-version ring](/quest/m1/mixed-version-ring.md) - defines the compatibility chain field this decodes ## Related diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index e24b81c2743..4d9a6f2b015 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -25,10 +25,12 @@ checks done at that call; the Required list is the work already known to land first, not a closed freeze set. The route layer of [Cluster routing](/quest/m2/cluster-routing/README.md) -(ROUTE_START/UPDATE/END, path-less ANNOUNCE, the hop list dropped) lands in +(ROUTE_START/UPDATE/END, path-less ANNOUNCE, the native hop list dropped) lands in lite-07, decided 2026-10-05. That removes lite-07's `Hop Base`/`Hop Keep` announce compression along with the hop list, and moq.pro's lite-07 rollout -has to plan for a lite-07 without hop lists. Only +has to plan for a lite-07 without native hop lists; only the temporary lite-06 +compatibility chain from [Mixed-version ring](/quest/m1/mixed-version-ring.md) +rides announcements until the lite-07 to lite-08 migration. Only [Routes and announces](/quest/m2/cluster-routing/routes/README.md) gates the cut, not the later federation/P2P integrations. Decided 2026-10-13: routes now owns the tier/partition correctness tests, so they run before the cut. Peer @@ -117,13 +119,14 @@ whose lite-07 encoding [lite-07 by start](/quest/m2/time-addressing/lite07.md) c ## Required -- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the hop list gone +- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the native hop list gone (only the temporary lite-06 compatibility chain remains) - [Lite-07 FETCH is single-group only](/quest/m2/subscribe-ranges/lite07-fetch.md) - lite-07 FETCH reads one group or a slice of one, and SUBSCRIBE ranges of whole groups - [Lite-07 by start](/quest/m2/time-addressing/lite07.md) - lite-07 carries group starts, previous-start links, GROUP_END, and time bounds instead of sequence numbers - [@moq/net by start](/quest/m2/time-addressing/js.md) - JS speaks the same lite-07 wire - [Takeover](/quest/m1/stream-credit/takeover.md) - lite-07 lets a group stream take over for another group - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path - [Delete the old start bookkeeping](/quest/m2/subscribe-ranges/start-bookkeeping.md) - may change START per update on lite-07 +- [A withdrawn claim ends across mixed lite-06 and lite-07 relays](/quest/m1/mixed-version-ring.md) - lite-07 announcements carry a temporary lite-06 compatibility chain ## Related diff --git a/quest/m5/lite07-mesh.md b/quest/m5/lite07-mesh.md index 4376c667229..2f89d919292 100644 --- a/quest/m5/lite07-mesh.md +++ b/quest/m5/lite07-mesh.md @@ -15,5 +15,6 @@ condition cannot clear before a release carries the final version. Decided 2026-10-05: [Cluster routing](/quest/m2/cluster-routing/README.md)'s route layer lands in lite-07, so the final lite-07 carries ROUTE and -path-less ANNOUNCE instead of hop lists, and loses the `Hop Base`/`Hop Keep` +path-less ANNOUNCE instead of native hop lists (a temporary lite-06 +compatibility chain rides until lite-08), and loses the `Hop Base`/`Hop Keep` announce compression. moq.pro's rollout must plan for that wire.