Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
2a81ae4
chore(quest): claim quest/m2/cluster-routes/acceptance.md
kixelated Nov 15, 2026
c48e3b4
test(net): tiered-cluster and LAN partition routes on lite-07 sessions
kixelated Nov 15, 2026
aa88b01
test(net): route layer acceptance fixtures for topology and prior art
kixelated Nov 15, 2026
28b0301
test(net): failover acceptance measured on live readers, and a narrow…
kixelated Nov 15, 2026
45e129e
test(net): a claim covering a mount's target continues beneath the mo…
kixelated Nov 15, 2026
889df72
fix(net): a scope widened to the root keeps the routes it held
kixelated Nov 15, 2026
a06deba
test(net): a client holds routes only to origins meeting its interest
kixelated Nov 15, 2026
a4dc7d0
test(route): operation counts for a route change along each fan-out axis
kixelated Nov 15, 2026
94496e3
docs(bench): name the residual the repath and relink groups pin
kixelated Nov 15, 2026
55bf324
Merge branch 'acceptance-part/failover' into quest/m2/cluster-routes/…
kixelated Nov 15, 2026
542feef
Merge branch 'acceptance-part/cluster' into quest/m2/cluster-routes/a…
kixelated Nov 15, 2026
f27c2ca
Merge branch 'acceptance-part/bench' into quest/m2/cluster-routes/acc…
kixelated Nov 15, 2026
714e333
test(route): forwarding fixtures run on the path router, and the refe…
kixelated Nov 15, 2026
eec5830
quest: acceptance keeps the mitigations' retirement, and its findings…
kixelated Nov 15, 2026
5aadacd
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 15, 2026
8284766
test(route): acceptance follows main's doubt rule, and answers its re…
kixelated Nov 15, 2026
2b61b2d
test(route): pin only the three known Unroutable readers in the mesh …
kixelated Nov 15, 2026
ce7e254
Merge origin/main: relink goes with the RTT #8838 removed
kixelated Nov 15, 2026
c7e9d8d
quest: the request loop returns to m2's cluster-routes epic
kixelated Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,8 +222,10 @@ maintainer asked that what remains be unblocked or punted, and picked these on
[moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)). m2 keeps only
what would need a breaking release had it landed right after the cut:

- ✅ Left m2: [the request loop](/quest/m3/cluster-routing/request-loop.md)
to m3's cluster-routing epic (a fixture and a fix, no wire);
- ✅ Left m2: [the request loop](/quest/m2/cluster-routes/request-loop.md)
to m3's cluster-routing epic (a fixture and a fix, no wire), until it
returned to m2's cluster-routes epic on 2026-11-15 (maintainer, on
[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832));
[media consumers](/quest/m3/time-addressing-consumers.md) to m3 (its API
breaks ride the model; dropping hang's markers is tolerated by readers);
[moq-transport time addressing](/quest/m4/ietf-time-addressing.md) to m4
Expand Down
38 changes: 18 additions & 20 deletions quest/m2/cluster-routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,10 +93,11 @@ lite-07 session in Rust and `js/net` drives the path router
deleted with its distance, seqno requests, and echo. The handoff fixtures,
both sweeps, and the route-layer fixtures of the contract run against it in
`route/tests/`, and real sessions with mocked time in
`tests/integration/path_routes.rs`; the forwarding and first-hop selection
fixtures still run only on the Babel-era reference model in
`rs/moq-net/tests/integration/routing_contracts/`, which
[acceptance](/quest/m2/cluster-routes/acceptance.md) deletes. The children below carry the rest, in order, and land in
`tests/integration/path_routes.rs`. The forwarding fixtures run on the router
in `route/tests/forward.rs`, and the topology, failover, and cluster
acceptance fixtures on real sessions in `tests/integration/route_acceptance.rs`,
`route_failover.rs`, and `route_cluster.rs`; the Babel-era reference model is
deleted. The children below carry the rest, in order, and land in
one release: lite-07 carries the route layer as a whole
([lite-07 finalize](/quest/m2/lite07-finalize.md) waits on this epic). This
README holds the routing contract every child implements.
Expand All @@ -123,10 +124,7 @@ Settled by the core, for review:

Keep the 2026-10-01 two-layer direction, reconciled with the maintainer's
2026-10-13 [decisions](/quest/m3/cluster-routing/README.md) and the routing
contract below. The routing contract's remaining model fixtures live in
`rs/moq-net/tests/integration/routing_contracts/` and run against a small reference model;
port each topology to the implementation as acceptance, then delete the model.
The handoff contract below, decided 2026-10-16, settles claim freshness:
contract below. The handoff contract below, decided 2026-10-16, settles claim freshness:

The lite-07 wire is specified in `drafts/draft-lcurley-moq-lite.md` (Announce,
Floors, and Routing), and `rs/moq-net/src/lite/announce.rs` encodes and decodes
Expand Down Expand Up @@ -196,17 +194,16 @@ the bullets below are the contract it carries.
id, when the connection's publish grant is scoped, and bare when it is
unscoped (publish `**` at the root). Its own identity is no exception
(2026-10-26): the bare exemption relied on the dropped proof, so an
unproven own identity could otherwise claim a victim's. Extend the fixture
below to a forger declaring the victim's id as its own, and drop the
reference model's `proved` exemption. The key has its own hash domain, so
unproven own identity could otherwise claim a victim's, as the fixture
below's forger does. The key has its own hash domain, so
it cannot collide with a key-derived identity, and is collision-resistant
at the identity width, since the forwarder picks `id` and could otherwise
grind another peer's key. The relay that files it translates it back when
forwarding a request into that connection. Claims under it are refused
outside the grant's scope. Routes are shared by all of an identity's
claims, so otherwise a scoped peer forging an identity, or declaring a
victim's id as its own, captures claims outside its namespace
(`forged_identity_hijacks_claims_outside_the_forgers_scope`); with it, a
(`a_forged_identity_is_contained_by_a_scoped_admission`); with it, a
forger or a misconfigured uplink only affects its own scope. The `peer`
claim no longer decides transit trust. Legitimate scoped delivery survives
losing one of two uplinks under one key
Expand All @@ -221,7 +218,7 @@ the bullets below are the contract it carries.
and a scoped connection forwarding an already qualified identity has that
route refused, with no wire change. Otherwise a key sent down through a
misconfigured uplink comes back and is filed again as a new identity on
every lap (`scoped_connections_cannot_requalify_a_qualified_identity`).
every lap (`a_qualified_identity_is_never_qualified_again`).
Nested scoped federation is therefore out: the inner level must be
unscoped. Revisit with a wire change if a use case appears. Rejected:
keying a forwarded identity by its bare id and the narrowest scope it
Expand Down Expand Up @@ -282,13 +279,14 @@ the bullets below are the contract it carries.
when its selected route turns down-only, and the first hop re-resolves; no
request bit goes on the wire. One selection hides a cheaper exportable
route whenever the preferred route is down-only
(`preferred_down_only_route_hides_the_exportable_alternative`); this is
accepted, and ranking local peers above a backup uplink avoids it. The
two-view model it replaces lost its export restriction one LAN hop after an
upstream ingress, and identity-only aggregation deadlocked across views
(`cdn_never_reaches_its_own_relay_through_the_lan`,
`upstream_ingress_ends_a_request_whose_route_turns_down_only`,
`dead_uplink_consumes_and_publishes_through_a_lan_peer`). A
(`a_preferred_down_only_route_hides_the_exportable_alternative`); this is
accepted, and ranking local peers above a backup uplink avoids it, though a
route's priority is the lowest on its path, so it must outrank the whole
path. The two-view model it replaces lost its export restriction one LAN hop
after an upstream ingress, and identity-only aggregation deadlocked across
views (`the_cdn_never_reaches_itself_through_the_lan`,
`an_upstream_request_stops_once_the_route_turns_down_only`,
`a_dead_uplink_consumes_and_publishes_through_a_lan_peer`). A
misconfigured uplink that leaks CDN routes back is contained by
qualified identities rather than a downstream mark
(`misconfigured_uplink_is_contained_by_qualified_identities`).
Expand Down
142 changes: 41 additions & 101 deletions quest/m2/cluster-routes/acceptance.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,11 @@
# [L] Route layer acceptance
# [M] Route layer acceptance

## Goal

The route layer is proven on the implementation, not the model: the
forwarding and first-hop selection fixtures run against the router and the
session, and the reference model in
`rs/moq-net/tests/integration/routing_contracts/` is deleted. The incident
fixtures, tiered-cluster and partition end-to-end checks, failover
measurements, and benchmarks below pass in CI, and the mitigations they
replace are retired.
The route layer is proven on the implementation, and the mitigations its
fixtures replace are retired: [moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375)'s
retraction quarantine, [moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s
same-clock queue dedup, and the 300 ms update hold (`origin::Config::update_hold`).

## Plan

Expand All @@ -19,100 +16,43 @@ tier and partition tests run before the cut (rejected: moving it to m3 and
fixing late routing flaws in lite-08). A flaw found here is fixed on
lite-07-wip before it freezes.

Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for
Babel): acceptance is of the path rule
([epic](/quest/m2/cluster-routes/README.md) Plan item 27), on the
sessions [the switch](https://github.com/moq-dev/moq/pull/8731) left.
The seqno request fixtures are replaced below, since the rule has no seqno
and no request.

- Port the forwarding fixtures (`Forward`, `Aggregate`, pool ties, and the
sweeps over them) to the implementation, then delete the model.
- The earlier simulator persisted seqnos across restart, and the path rule
has none. Re-run with a fresh routing identity per restart and the
stale-state rules; do not carry the old assumption into acceptance.
- Retire [moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375)'s
quarantine, [moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s
dedup, and the 300 ms hold only once the fixtures below demonstrate their
replacement (see the epic's hold-down note).
- Add a session-level fixture for routes scoped to interested origins
(Plan item 15 of the epic, implemented on the router): a client watching
one prefix holds routes only to the origins whose claims meet it, a
request for a path outside its interest still resolves through the relay
(Node ID 0), and an origin leaving the interest ends its route after the
batch withdrawing its last claim there, closed by its floor: one that
certifies the withdrawal where the relay certifies the request, and one
certifying nothing where it knows only part of it.

The counterexamples from #4349 (closed; this epic still owns the fix) are the `residual_*` tests in
`rs/moq-net/tests/integration/mesh_withdraw.rs`. They pin today's hunting on lite-06:
a stale path outlasting the 300 ms hold, a revival the quarantine misses,
and 179 hunting relays (3 reviving) over 200 seeded meshes. Turn each into a
single retraction everywhere, and keep `cross_node_close_retracts_once`
passing. On `main` an epochless switch to a stale path is a Restart, which a
lite-06 client sees as a retraction and an announcement; the release sent an
in-place update.

Publishing toggles too, decided 2026-10-13: a longer path arrives first and
the better route then replaces it. `late_better_path_updates_in_place` pins
the lite-07 mesh, where the move keeps the instance and a lite-06 or lite-07
client sees an update. `residual_late_better_path_restarts_on_lite06` pins the
lite-06 mesh, where each relay mints its own instance and the client sees
`Start, End, Start` while the broadcast stays live; turn it into an update.

Tests, time mocked: the handoff fixtures and sweeps, already on the router;
[moq-dev/moq#4644](https://github.com/moq-dev/moq/pull/4644)'s live-graph withdrawal and failover tests (the PR closed and
its branch is gone; fetch `refs/pull/4644/head`) as acceptance, run on the wip version; a
link flap with many broadcasts behind one origin costs one ROUTE change per
link; a ring of three where a stale neighbour cannot revive an ended
broadcast; a two-uplink mesh that never carries CDN routes back to the CDN;
an island reached through the CDN after a partition; an unknown Route ID
refused; a five-node ring whose preferred origin link fails moves to its
longer backup, at once where the backup's path avoids the far end of the
closed link and once the doubt lifts where it runs through it, while that
backup never loses its session; a relay with several neighbours that dies
is named in one withdrawal per session, and no relay moves onto a route
through it before the doubt lifts; two relays whose only routes run through
each other both let go, neither holding for the other; and a relay that
holds lets go once the neighbours behind it have answered, including one
whose session closes without an answer. Failover is measured
on broadcast liveness while the next hop changes, including a new next hop
whose ROUTE arrives before its ANNOUNCE set, and reports per-path
retractions and re-announcements beside the ROUTE budget; eventual route
recovery alone does not show it is seamless.

Tiered-cluster and partition/healing end-to-end checks belong here,
not after lite-07 can be finalized: two regions with redundant cores, publish,
claim withdrawal, core loss, and matching-epoch failover reaching a far edge;
a two-uplink mesh reached through CDN during partition. Include overlapping
wildcards, a hidden exportable alternative under one selection, epochless Restart, and a
grant union crossing the scoped boundary each way (a `**` token added, then expiring). Fixtures are
public and time-mocked, wired into CI alongside Rust/JS interop.

Mounts get the same continuity checks (`rs/moq-net/tests/integration/mount.rs`):
a claim whose predecessor covers a mount's target arrives beneath the mount
point exactly as it does unmounted, on lite-06 and lite-07. This replaces
the deleted mount-takeover quest, whose `Takeover` event
[moq-dev/moq#8063](https://github.com/moq-dev/moq/pull/8063) removes in
favor of the claim's instance; if #8063 closes unmerged, restore that quest.

Benchmark origins, broadcasts per origin, neighbors, active tracks/fronts,
and announce cursors independently. A route update must not expand back into
per-broadcast table rewrites or poll unrelated fronts. Measure messages, bytes,
wakes/select calls, CPU, replay work, and retained state. One broadcast per
origin is an explicit case. Coordinate with Route wakes and Announce replay;
benchmark smoke alone does not enforce complexity, so add operation-count
regressions. Bound held routes, doubts, and remembered dead hops under
same-clock churn.

Decided 2026-11-12 (maintainer, /quest-audit; epic Plan item 30): three
fixtures from routing prior art, as tests before any design change. A
relay vanishes and its neighbours notice up to the 3 s keep-alive apart,
which shows whether the doubt timer must follow the keep-alive or relay
keep-alives must shorten. Three relays whose local priorities form a dispute
wheel settle, or the test pins how they fail (RFC 4264). A primary route
fails and returns, and every relay ends back on it.
The fixtures landed in [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832),
and the reference model is deleted: forwarding on the router
(`rs/moq-net/src/route/tests/forward.rs`), and on real sessions with time mocked
the topology and prior-art fixtures (`tests/integration/route_acceptance.rs`),
failover liveness and the live-graph tests of
[moq-dev/moq#4644](https://github.com/moq-dev/moq/pull/4644) (`route_failover.rs`),
the tiered cluster, partition, and scoped-interest checks (`route_cluster.rs`),
mount continuity (`mount.rs`), and operation-count regressions
(`route/tests/cost.rs`, `route/node/tests.rs`). They found the flaws the
quests under Required fix; the tests that pin each flaw say so in their docs.

What remains:

- Retire the quarantine, the dedup, and the hold once the Required quests
turn their pinned fixtures into the contract's behaviour. The quarantine
is what turns the flaws into a 2 s gap: a relay that shows a held claim
offline announces it again only after `RETRACT_QUARANTINE`. The hold and
the quarantine sit on the announce cursor, which every version shares, so
check what a lite-06 mesh loses without them
(`mesh_withdraw::cross_node_close_without_hold_reannounces` is its
positive control) before deleting them; if a lite-06 mesh still needs
them, retire them for lite-07 routes only.
- `mesh_withdraw::residual_late_better_path_restarts_on_lite06` pins a
lite-06 mesh where each relay mints its own instance, so a client sees
`Start, End, Start` while the broadcast stays live. Turn it into an update,
or record why a lite-06 mesh keeps it.
- A request naming Node ID 0 on a lite-07 session: Rust clients always name
a node, so only publisher unit tests cover a relay answering 0. Cover it
where `js/net` sends it, in `just test interop`.

## Required

- [Held after close](/quest/m2/cluster-routes/held-after-close.md) - a relay whose own session closed shows its held claims unavailable, not offline
- [Doubt holds](/quest/m2/cluster-routes/doubt-holds.md) - a relay keeps holding while doubt alone keeps it off its routes
- [Route fan-out](/quest/m2/cluster-routes/route-fanout.md) - a route change costs only the streams and broadcasts it touches

## Related

- [Routes and announces](/quest/m2/cluster-routes/README.md) - the contract and its fixtures
- [Doubt and keep-alive](/quest/m2/cluster-routes/doubt-keepalive.md) - whether the doubt timer follows the keep-alive
40 changes: 40 additions & 0 deletions quest/m2/cluster-routes/doubt-holds.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# [M] Doubt holds

## Goal

A relay whose only routes left are kept off by doubt holds its claims,
unavailable, until the doubt lifts and it moves onto one, instead of letting
go while its neighbours answer that they have nothing.

## Plan

Kept 2026-11-15 (maintainer, /quest-complete on
[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in
m2 (rejected: drop; moving it to m3).

Found by [acceptance](/quest/m2/cluster-routes/acceptance.md):
`route::tests::flap::residual_a_relay_kept_off_by_doubt_lets_go_before_it_lifts`
pins it on the router. The publisher sits behind a transit relay that loses
most of its links, leaving only routes through it, which every relay that
heard a link die doubts for the timer. A relay with no usable route holds,
its neighbours are in the same state and answer that they have nothing, and
it lets go a hop after the cut: the claim goes Unavailable, then gone, then
back once the doubt lifts.

`route_failover::live_mesh_link_failure_keeps_presence` measured it on a
34-relay mesh before [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734):
over 5 publishers, 98 readers lost their subscription. That fixture publishes
on the relay itself, so since #8734 doubt no longer blocks its routes, and it
loses 6. A real-session fixture for this quest needs the publisher behind the
relay whose links close.

The contract already says a node holds "while a neighbour offers a route that
only the doubt keeps it off". Find why the answers end the hold first: a
neighbour that holds only because of doubt should answer with ROUTE_HOLD, not
nothing. Check the rule in `RouteHandoff.tla` with the `verify-route` skill
before changing `rs/moq-net/src/route/path.rs`.

## Related

- [Routes and announces](/quest/m2/cluster-routes/README.md) - Plan item 27 and the hold rule
- [Held after close](/quest/m2/cluster-routes/held-after-close.md) - the same symptom at the relay whose own session closed
37 changes: 37 additions & 0 deletions quest/m2/cluster-routes/doubt-keepalive.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [S] Doubt and keep-alive

## Goal

No relay moves onto a route through a relay that died silently, however far
apart its neighbours notice the death within the keep-alive.

## Plan

Kept 2026-11-15 (maintainer, /quest-complete on
[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ keep it in
m2 (rejected: drop; moving it to m3).

Found by [acceptance](/quest/m2/cluster-routes/acceptance.md) (epic Plan
item 30's first prior-art fixture):
`route_acceptance::a_vanished_relay_is_noticed_a_keep_alive_apart` kills
`X` silently, and its neighbours `R1`, `R2`, `R3` notice at 0 s, 1.5 s, and
3 s. With the 1 s doubt, `R1` moves onto a neighbour's route through the
dead `X` at 1 s, holds again at 1.5 s, and moves onto `R3`'s route through
`X` at 2.5 s; `R2` does the same at 2.5 s. All three go offline exactly once,
at 3 s. Readers see each move as available for a moment, and a request in
that window goes to a dead relay. A 1 s doubt does not cover a 3 s spread.

## Waiting

- Maintainer: how the doubt covers a silent death. Options: (a) the doubt
lasts at least the configured idle timeout, so it covers the spread (the
recommendation: no wire change, and a relay already knows its idle
timeout); (b) relay keep-alives shorten below the doubt, which costs
traffic on every relay link; (c) accept the window and keep the test
pinning it. Check with `/quest-grill` or the PR that picks it up. The
maintainer deferred this decision on 2026-11-15, not yet sure about the
doubt design.

## Related

- [Routes and announces](/quest/m2/cluster-routes/README.md) - Plan item 27's doubt and item 30
Loading