Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ The API is the most important thing to get right: a bad shape costs a breaking c
PRs target `main`, where breaking API changes are allowed.
`release` is what ships; touch it only to cut a release or backport a fix or benchmark (see `CONTRIBUTING.md`).
Wire changes stay backwards compatible with *published* drafts/versions; an unpublished `-wip` version changes freely.
moq.pro rolls out with relays on the newest and previous release side by side, so those two versions must interoperate as relays (today moq-lite-06 and moq-lite-07). A temporary migration for that is fine when it is clearly marked and easy to delete at the next version migration.
Our libraries prefer moq-lite, so moq-transport compatibility only matters against third-party implementations.

Build output may live outside the checkout and gets wiped, so scratch never goes in `target/`.
Expand Down
1 change: 1 addition & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ moves what remains of them once their claims merge.

- [Drop set](/quest/m1/drop-set.md) - the drop set's track and cursor runs are billed to the cache pool, so fragmenting drops costs what the pool holds; eviction folds final runs or ends the cursor with `TooFarBehind`
- [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation setup
- [A withdrawn claim ends across mixed lite-06 and lite-07 relays](/quest/m1/mixed-version-ring.md) - lite-07 carries a temporary lite-06 compatibility chain, so a rolling upgrade's withdrawn broadcast stops circling
- [A reader on a failed copy continues on the held re-offer](/quest/m1/copy-recover.md) - a live reader keeps a group whose copy fails by continuing on the same track's re-offer at the same frame
- [Group errors are cached by kind](/quest/m1/group-error-cache.md) - permanent group errors bind every reader, retryable ones are kept only for a bounded time, and a live SUBSCRIBE_DROP or FETCH answered `Old` no longer caches it for good
- [The overcommitted receive-window cells run again](/quest/m1/receive-windows-evicted.md) - `benchmark_receive_windows` measures its 256-stream and 4 MiB-frame cells again, reporting delivered frames and evictions instead of stalling
Expand Down
92 changes: 92 additions & 0 deletions quest/m1/mixed-version-ring.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# [L] A withdrawn claim ends across mixed lite-06 and lite-07 relays

## Goal

After a publisher leaves, its broadcast ends at every relay within a bounded
number of route updates, for any mix of moq-lite-06 and moq-lite-07 relay
links. That includes a rolling upgrade, where one lite-06 relay links to
three or more lite-07 relays. Today such a claim circles forever after its
withdrawal, and every subscriber sees a Restart on every lap.

Not here: relay links over moq-transport. moq-transport is supported for
clients, not relays (decided 2026-11-08), so its anonymous chains stay out of
scope.

## Plan

Root cause, reproduced 2026-11-08 on main at `8065e7148f`. The loop has been
on main since [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023),
which added the lite-06 translation and its single-link reflection check:

- A route learned over lite-06 becomes the translating relay's own claim
(`OwnClaims::apply` in `rs/moq-net/src/model/origin.rs`, the `learned`
flag), and its lite-06 hop chain is dropped there.
- Translating a lite-07 route back out to lite-06 sends only its origin
node's hop (`hops()` in `rs/moq-net/src/lite/subscriber/routed.rs`).
- So `Subscriber::chain` (`rs/moq-net/src/lite/subscriber.rs`) only catches a
claim that went out and came back across one lite-06 segment. Once a claim
crosses two lite-06 to lite-07 boundaries, no relay sees its own hop again.
In A-07-B-06-C-07-D-06-A, B and D keep re-originating each other's copy.
- Minting a `~` instance per lite-06 hop only turns each lap into a Restart.
It does not cause the loop, and the rising cost saturates only near
`MAX_COST`, so in practice the loop never ends.

Announcing settles in every topology probed; only withdrawal loops. It loops
in a ring with two lite-06 links separated by lite-07 links (three behave the
same), and when one lite-06 relay links to three or more lite-07 relays,
wherever the publisher sits. A 07-06-06 triangle, a ring with adjacent lite-06
links, and an old relay with two lite-07 neighbours all settle.

Decided 2026-11-08 (maintainer, /quest-plan):

- ✅ Carry the lite-06 chain through lite-07, as a crate-private chain plus a
lite-07-wip compatibility extension and field. Rejected: a cost or hop
ceiling with no wire change, which still circles for hundreds to thousands
of laps and is an arbitrary constant; never forwarding a lite-06-learned
claim to another lite-07 relay, which breaks old-new-new chains mid-upgrade;
documenting mixed rings as unsupported, which a rolling upgrade cannot
avoid. A claim learned over lite-06 keeps that chain. lite-07
ANNOUNCE_START, ANNOUNCE_RESTART and ANNOUNCE_UPDATE carry it, empty for a
native claim. Translating back to lite-06 sends the chain plus this node's
hop. Both receive sides drop a chain that names them: lite-06 in `chain()`,
and lite-07 when it serves the route. `max_hops` then bounds the total
across islands, and the existing route-hop check becomes one case of it.
- ✅ It is temporary: the draft and code mark the extension and field as
lite-06 compatibility, and the lite-07 to lite-08 migration deletes them.
No public API carries the chain. Keep it apart from the minted flag that
`quest/m2/instance-provenance.md` plans (on
[moq-dev/moq#8266](https://github.com/moq-dev/moq/pull/8266)'s branch
until it merges), even though a non-empty chain marks exactly the
announcements that flag would: the flag also covers moq-transport routes
and outlives lite-06.
- ✅ m1 (rejected: m0, m2). lite-07 is opt-in `-wip`, so nothing published
breaks, and the field must exist before
[lite-07 is finalized](/quest/m2/lite07-finalize.md).

A throwaway prototype, with a process-wide table standing in for the wire
field, made every looping topology settle in 4 to 13 events after withdrawal.
All integration and lib tests still passed. It needed the lite-07-side drop,
or it tripped `origin.rs`'s "announce called with a looping hop chain"
debug assertion. A chain change on an open session also needs
ANNOUNCE_UPDATE to carry the field.

Tests, in `rs/moq-net/tests/integration/mesh_withdraw.rs` beside
`mixed_version_ring_withdraw_retracts`. Each announces at one relay, checks
that every relay sees it start, withdraws it, and checks that every relay
sees it end:

- The ring A-07-B-06-C-07-D-06-A.
- A rolling upgrade with the publisher at the lite-06 relay, linked to each
relay of a three-relay lite-07 full mesh. This one hung 12 of 12 runs, so
it is the reliable gate.
- The same rolling upgrade with the publisher at a lite-07 relay. It hung 11
of 12 runs, because route identities are random and tie-breaks hash them.
Make it deterministic or drop it.

The js/net lite-07 port in [routes/js.md](/quest/m2/cluster-routing/routes/js.md)
decodes the field; JS translates nothing, so it keeps no chain.

## Related

- [Route session integration](/quest/m2/cluster-routing/routes/session.md) - makes `Hops` crate-private behind a chain type, which this fills from a third source, and merges the lite-06 and lite-07 identities, which folds the two receive checks together
- [Routes acceptance](/quest/m2/cluster-routing/routes/acceptance.md) - these tests are its kind of fixture, a stale neighbour unable to revive an ended broadcast
2 changes: 2 additions & 0 deletions quest/m2/cluster-routing/routes/js.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Decided 2026-10-27 (maintainer), from
the cheapest one; fix it with the route-selection rework here.
- Feed the availability API from [Announcement availability](/quest/m2/cluster-routing/routes/availability.md).
- Rust/JS interop covers the route messages, the floor marker, and the echo.
- Decode the temporary lite-06 compatibility chain that lite-07 announcements carry ([mixed-version ring](/quest/m1/mixed-version-ring.md)). JS translates nothing, so it keeps no chain.
Comment thread
kixelated marked this conversation as resolved.
Resolved
- Restore the native JS lite-07 lanes. Decided 2026-11-06 (maintainer,
/quest-complete on [moq-dev/moq#8023](https://github.com/moq-dev/moq/pull/8023)),
asked how to handle the in-tree lanes running JS on lite-07-wip once the
Expand All @@ -34,6 +35,7 @@ the cheapest one; fix it with the route-selection rework here.
## Required

- [Announcement availability](/quest/m2/cluster-routing/routes/availability.md) - the API this feeds
- [Mixed-version ring](/quest/m1/mixed-version-ring.md) - defines the compatibility chain field this decodes

## Related

Expand Down
9 changes: 6 additions & 3 deletions quest/m2/lite07-finalize.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@ checks done at that call; the Required list is the work already known to land
first, not a closed freeze set.

The route layer of [Cluster routing](/quest/m2/cluster-routing/README.md)
(ROUTE_START/UPDATE/END, path-less ANNOUNCE, the hop list dropped) lands in
(ROUTE_START/UPDATE/END, path-less ANNOUNCE, the native hop list dropped) lands in
lite-07, decided 2026-10-05. That removes lite-07's `Hop Base`/`Hop Keep`
announce compression along with the hop list, and moq.pro's lite-07 rollout
has to plan for a lite-07 without hop lists. Only
has to plan for a lite-07 without native hop lists; only the temporary lite-06
compatibility chain from [Mixed-version ring](/quest/m1/mixed-version-ring.md)
rides announcements until the lite-07 to lite-08 migration. Only
[Routes and announces](/quest/m2/cluster-routing/routes/README.md) gates the cut,
not the later federation/P2P integrations. Decided 2026-10-13: routes now
owns the tier/partition correctness tests, so they run before the cut. Peer
Expand Down Expand Up @@ -117,13 +119,14 @@ whose lite-07 encoding [lite-07 by start](/quest/m2/time-addressing/lite07.md) c

## Required

- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the hop list gone
- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the native hop list gone (only the temporary lite-06 compatibility chain remains)
- [Lite-07 FETCH is single-group only](/quest/m2/subscribe-ranges/lite07-fetch.md) - lite-07 FETCH reads one group or a slice of one, and SUBSCRIBE ranges of whole groups
- [Lite-07 by start](/quest/m2/time-addressing/lite07.md) - lite-07 carries group starts, previous-start links, GROUP_END, and time bounds instead of sequence numbers
- [@moq/net by start](/quest/m2/time-addressing/js.md) - JS speaks the same lite-07 wire
- [Takeover](/quest/m1/stream-credit/takeover.md) - lite-07 lets a group stream take over for another group
- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path
- [Delete the old start bookkeeping](/quest/m2/subscribe-ranges/start-bookkeeping.md) - may change START per update on lite-07
- [A withdrawn claim ends across mixed lite-06 and lite-07 relays](/quest/m1/mixed-version-ring.md) - lite-07 announcements carry a temporary lite-06 compatibility chain
Comment thread
kixelated marked this conversation as resolved.
Resolved

## Related

Expand Down
3 changes: 2 additions & 1 deletion quest/m5/lite07-mesh.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,6 @@ condition cannot clear before a release carries the final version.

Decided 2026-10-05: [Cluster routing](/quest/m2/cluster-routing/README.md)'s
route layer lands in lite-07, so the final lite-07 carries ROUTE and
path-less ANNOUNCE instead of hop lists, and loses the `Hop Base`/`Hop Keep`
path-less ANNOUNCE instead of native hop lists (a temporary lite-06
compatibility chain rides until lite-08), and loses the `Hop Base`/`Hop Keep`
announce compression. moq.pro's rollout must plan for that wire.