Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/verify-route/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/rout
All three pass at the sync commit. Each takes minutes on 4 workers, so run them last. First run what answers in seconds:

- `just rs test -p moq-net --lib -E 'test(/route::/)'`. `sweep_no_route_outgrows_a_loop_free_path` delivers every message and echo in a random order on random meshes with unequal costs, flaps, restarts, and collections, and checks the metric bound, loop freedom, and the quiet state. Raise its case count for a wider search: 40,000 cases run in 15 seconds in release.
- TLC's simulation mode on the config in question, which samples deep runs the exhaustive bounds do not reach: add `-simulate num=3000 -depth 80`.
- TLC's simulation mode, which samples runs the exhaustive bounds do not reach. The config's `MaxRev`, `MaxFlaps`, and `MaxCollects` still bound a simulated run, so raise them in a copy of the config first, such as three session changes, two claim changes, and a collection, then add `-simulate num=3000 -depth 150`.

Both behind-a-triangle configs fail `SimplePaths` within seconds if `Crossed` is made false, which is the count to infinity the hops fix: a relay drops its distance once its neighbours echoed, and takes back a copy of its own route that a third relay still held.

Expand Down
3 changes: 2 additions & 1 deletion drafts/draft-lcurley-moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,7 @@ A metric that overflows 64 bits makes the route unusable.

A route carries its Hops: the Hop ID of every node it crossed, the origin node first and the sender last.
A Hop ID is a non-zero 64-bit value a node names itself with for as long as its process runs, such as the Node ID it started with; a rotation does not change it.
A receiver tolerates a zero or repeated Hop ID instead of closing the session: such a route harms only itself.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
A node sending its selected route appends its own Hop ID to the Hops it received, and an origin's own route starts with its Hop ID alone.
A relay forwards the other entries unchanged and never qualifies them.
A route that arrives with 32 Hops is unusable.
Expand Down Expand Up @@ -2273,7 +2274,7 @@ The `Message Length` describes the payload size on the wire.
- Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix.
- Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup.
- ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route.
- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route.
- Added Hops to ROUTE_START and ROUTE_UPDATE: the Hop ID of every node the route crossed, at most 32. A node never selects a route whose Hops name it, so one that dropped its feasibility distance cannot take back a copy of its own route. A zero or repeated Hop ID in a route's Hops is tolerated, where a repeated non-zero Hop ID in moq-lite-06's Hop ID list is a PROTOCOL_VIOLATION.
- Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back.
- Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible.
- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows.
Expand Down
6 changes: 4 additions & 2 deletions quest/m1/route-inductive.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@ stay constants the config states. Say both bounds in the `verify-route` skill.
replay it on `route::tests::Sim` as the skill says before calling it a bug.
- Apalache needs type annotations. Add them to the spec TLC already checks,
so both tools read one file and neither drifts.
- `NoCountToInfinity` follows from `SimplePaths`, which is inductive on its
own: every route sent adds its sender to hops that did not name it
- `NoCountToInfinity` follows from `SimplePaths`, which is inductive once
the invariant also says a route is held only from a node the holder shares
a link with: every route sent adds its sender to hops that did not name it
([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Start there.
- Run it through the `tla` recipes with Apalache from the flake. If five
relays does not fit Nightly, record the largest mesh that does and its time.
Expand All @@ -47,6 +48,7 @@ Public API: none. Wire: none.
## Required

- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - the spec parametrised over mesh shapes that this invariant is stated on
- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule for whose claims a relay believes, which the claim rules rest on; an invariant found before it is found for a rule about to go

## Related

Expand Down
4 changes: 4 additions & 0 deletions quest/m1/route-trace-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,7 @@ Related: [the mesh route model](/quest/m1/route-mesh-model.md), which grows
the spec these traces are checked against.

Public API: none. Wire: none.

## Required

- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - replaces who a relay believes and adds two prefixes and a scoped relay, so traces are validated against the rule that stays
14 changes: 14 additions & 0 deletions quest/m2/cache-routing-proof.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,16 @@ it; a SETUP extension needs neither before the cut, and
from its list. Either placement is now this quest's to record, with no
second trip to the maintainer for choosing lite-07.

Reopened 2026-11-11 (maintainer, /quest-plan on the route redesign, epic
Plan item 8): a warm route propagates, and the rule that keeps it free of
loops and stale copies is
[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to pick.
The eligibility rule and the fixtures below are for the one-hop warm route
first decided, which [moq-dev/moq#8457](https://github.com/moq-dev/moq/pull/8457)
proves. Once the prototype's pick is recorded, rewrite both for it: the
fixtures keep their meshes, and gain a relay that joins a copy at a neighbour
farther from the publisher than itself.

Build on the sans-IO `Router` in `rs/moq-net/src/route/`, which each
origin's `route::Node` drives, and the harness behind its handoff sweeps,
not the reference model in
Expand Down Expand Up @@ -70,6 +80,10 @@ Route ID, and the instance, and say which stream carries it and how a peer
that never sends one behaves. Size it against lite-07 and an additive SETUP
extension and record the choice and its reason in the epic's Plan.

## Required

- [A prototype settles what keeps a route free of stale copies](/quest/m2/cluster-routing/routes/stale-copies.md) - the rule a propagating warm route rests on, which replaces the one this proved

## Related

- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the feasibility rules and handoff sweeps this extends
12 changes: 12 additions & 0 deletions quest/m2/cluster-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,18 @@ Decided with the maintainer, 2026-10-13:
- Keep Babel. A detour longer than the lost route may wait for a sequence
refresh from the origin; instant failover is not required. Measure failure
detection, additional routing recovery, and total interruption separately.
Reopened 2026-11-11 (maintainer, /quest-plan after
[moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews):
whether Babel's distance stays is
[the prototype](/quest/m2/cluster-routing/routes/stale-copies.md)'s to
measure, against paths alone with a withdrawal that names the hop that
died. The distance refuses a longer way round until the publisher advances
its seqno, and refuses a cached copy at a neighbour no closer to the
publisher, which [cache-aware routing](/quest/m3/cache-routing/README.md)
wants to join. Decided the same day: a route is shared per publisher and
per scope, and a relay believes a route by that route's own path, whichever
rule wins. Rejected: link-state among relays, and routes to relays in
place of publishers.
- An origin declares its own routing identity, fresh per process or tab
restart and shared across its live sessions. It is unverified. Decided 2026-11-02 (maintainer, /quest-unblock): no
key proof is planned; if a use case needs a provable identity, moq-lite-07's
Expand Down
4 changes: 3 additions & 1 deletion quest/m2/cluster-routing/routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,11 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout
22. ✅ **A side's interest is a set of prefixes edited in place.** The live set never nests, and one edit may swap prefixes for a wider or narrower one without re-sending what both cover ([the interest set](/quest/m2/cluster-routing/routes/interest-set.md)). A token-scope change is such an edit, with one prefix per region of the scope. This supersedes items 16 and 17, and the handoff contract's "An interest change opens a new stream" below, once the route stream lands.
23. ✅ Asked what a floor covers with several prefixes on one stream: **one prefix, named by its id.** This amends item 10: the floor is still a marker and never a region list. Rejected: a floor over the whole set.
24. ✅ Asked how a side learns a newly added prefix's initial set is complete: **SYNCED per prefix, naming its id**, plus one marker per edit where it takes effect for both sides. Rejected: one SYNCED per edit, counted in order.
25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes.
25. ✅ Asked how a relay safely forgets a publisher that is gone, after the model showed a per-neighbour echo is not enough once the origin sits behind a relay: **each route names the nodes it crossed, and a node never uses a route that names itself** ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487), which holds the counterexamples and the model results). Feasibility still selects routes; the list is one per origin route, not the per-claim Hop ID list item 6 removed, and the echo and the drain rule stay as they are. As built: Babel's feasibility condition is unchanged, and a node never selects or certifies through such a route. Where the condition refuses it, it still counts as infeasible, so a relay whose neighbours all route through it holds its claims while one of them finds another way. Under a qualified key it counts only while its hops extend those of the route the node last sent under that key: a copy that a scoped peer hands back under a key derived from the node's own route never moves, so nothing waits on it. The origin names itself too, so a rotated identity's route is refused where it began, and a node names itself by a Hop ID that a rotation leaves alone. The hop limit is the wire's 32 and no lower. A change of hops or RTT alone lets go of no region, unless it changes whether the node may use the route: by crossing it, or by reaching the hop limit. This supersedes the claim, in the handoff contract and in decision 6 below, that the echo alone makes forgetting safe. First decided as a relay echoing a ROUTE_END only once its own downstream had; the model ran that the same day and it fails when a session closes before answering. Rejected: that deferred echo; never dropping the distance; a timer; relays as the only origin nodes.


26. ✅ Decided 2026-11-11 (maintainer, /quest-plan after [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)'s reviews), asked whose copy of a publisher's claims a relay believes: **a route is shared per publisher and per scope, and a relay believes a route by that route's own path** ([the prototype](/quest/m2/cluster-routing/routes/stale-copies.md), [the model](/quest/m2/cluster-routing/routes/scope-model.md), then [the router](/quest/m2/cluster-routing/routes/scope.md)). A scope is the prefix a relay asked a neighbour for when its grant there is narrower than its interest. Today only a feasible neighbour counts, one strictly cheaper than the distance the relay remembers, so a relay's claims move whenever a cost or seqno does: it can forget a claim for good, a mesh with two grants never settles after one seqno advance, and behind a scoped neighbour it never shows what a farther one knows. Only a broadcast some relay carries gets its own route, and that route propagates ([cache-aware routing](/quest/m3/cache-routing/README.md) Plan item 8). Open, for the prototype to measure and the maintainer to pick: whether Babel's distance or the path alone, with a withdrawal that names the hop that died, keeps a route free of loops and stale copies. The contract's bullet on which neighbours certify stands until the model quest rewrites it. First decided the same day as any neighbour whose path to the publisher does not pass through the relay; tried on the router, two relays behind a scoped relay then keep a dead claim alive through each other, and the relay behind a scoped neighbour shows a claim it has no route to request. Rejected: that rule; believing a neighbour that is cheaper than the relay is now, which fixed the storm and the forgotten claim with no wire change but rests belief on distance; a path per broadcast as lite-06 has; link-state among relays; routes to relays.

The route layer's core landed in
[moq-dev/moq#7676](https://github.com/moq-dev/moq/pull/7676): a
crate-private, sans-IO `Router` in `rs/moq-net/src/route/`, generic over the
Expand Down
37 changes: 37 additions & 0 deletions quest/m2/cluster-routing/routes/handoff-revival.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [M] The handoff sweep revives no ended claim

## Goal

`sweep_handoff` in `rs/moq-net/src/route/tests/handoff.rs` reports no revival
over 20,000 scenarios: no reader sees one life of a claim start again after
it saw that life end, unless it forgot the claim in between. A publisher that
withdraws a claim and announces it again starts a new life, which is no
revival. The scenario that revives one today is a plain test.

## Plan

Found 2026-11-11 by running the sweep past its 150 scenarios: scenario 16316
revives a claim, on `main` at `688312b9eb`, and no earlier one does. It is
the same whichever neighbours the router believes. It has not been traced.
Decided the same day (maintainer, /quest-plan): ✅ plan it.

- Reproduce it: the sweep draws every scenario from one random stream, so run
the first 16,317. Record the scenario's steps as it runs and remove them
one at a time while the revival remains, as
[the false END](/quest/m2/cluster-routing/routes/widen-false-end.md) was
shrunk.
- Decide whether it is a revival the contract forbids or one it excuses: a
replay after a collection or a restart is allowed only until the
withdrawal behind it lands
([epic](/quest/m2/cluster-routing/routes/README.md), handoff contract). If
the fixture's `revivals` count misses an excuse, fix the count and say so.
- Otherwise fix the cause and keep the shrunk case as a test.
- If the cause is in what
[the route per scope](/quest/m2/cluster-routing/routes/scope.md) deletes,
pin it as a `residual_` test naming that quest instead of fixing it twice.

Public API: none. Wire: none.

## Related

- [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - found the same way, and adds the way to run more scenarios
20 changes: 16 additions & 4 deletions quest/m2/cluster-routing/routes/interest-set.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,13 @@ Work:
([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)), and move the echo from its own queue onto the neighbour's FIFO, as
that PR's first commit (`81c0698295`) did: with one stream per side nothing
overtakes anything.
- Give the model at least two prefixes and edits in flight: add, remove,
widen, narrow, and an edit that crosses a claim change, a route change, a
ROUTE_END, and its echo. The model fixes interest at the root today, so
this is new state; keep it tractable by bounding edits as it bounds flaps.
- The model has two prefixes and a scoped relay since
[it has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md).
Give it edits in flight: add, remove, widen, narrow, and an edit that
crosses a claim change, a route change, a ROUTE_END, and its echo. Keep it
tractable by bounding edits as it bounds flaps. Where Settled item 1 says
the merge rule is unchanged, read it as that quest's rule: a floor names
its route's prefix, and the newest floor wins among one publisher's routes.
- In the router, `Peer::asked` and `Peer::watch` become sets with a floor and
a synced flag per prefix, `batch` takes the prefix its floor names, and a
neighbour is one key for both directions, as `Sim` already has it. The
Expand All @@ -97,3 +100,12 @@ one key, which `Sim` and the tests use, so this changes no behaviour on its
own.

Public API: none (the router is crate-private). Wire: none.

## Required

- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the two prefixes and the rule for whose claims a relay believes, which an edit must preserve

## Related

- [The router keeps a route per scope](/quest/m2/cluster-routing/routes/scope.md) - rewrites how the router keeps a floor per prefix; whichever lands second adapts
- [Widening an interest ends no live claim](/quest/m2/cluster-routing/routes/widen-false-end.md) - a false END on an interest change today, which the edit in place may remove
Loading