Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
b459825
chore(quest): claim quest/m2/cluster-routes/flapping-publisher.md
kixelated Nov 14, 2026
b72e395
feat(route): doubt spares a route that begins at the doubted node
kixelated Nov 14, 2026
8ceade8
feat(route): a link that returns during the doubt waits it out
kixelated Nov 14, 2026
154bb65
Merge remote-tracking branch 'origin/main' into quest/m2/cluster-rout…
kixelated Nov 14, 2026
b3ee511
fix(route): a restart reconnects a publisher's second home; credit dr…
kixelated Nov 14, 2026
51cc09b
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 14, 2026
30b3e36
test(route): pin the stale-ring cache budget to the path rule's loop
kixelated Nov 14, 2026
fa431b6
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 14, 2026
29f7544
fix(route): fingerprint a doubt's session mark, after merging main
kixelated Nov 14, 2026
9992297
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 15, 2026
255c24b
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 15, 2026
a0473ae
test(route): a link opened before a doubt is taken at once
kixelated Nov 15, 2026
805064b
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 15, 2026
9be566b
docs(quest): the draft text keeps a relay's current route exempt from…
kixelated Nov 15, 2026
f747012
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 15, 2026
685b7c3
Merge origin/main into quest/m2/cluster-routes/flapping-publisher
kixelated Nov 15, 2026
599fa8d
feat(route): sessions spare a doubted origin's other links; the draft…
kixelated Nov 15, 2026
e034e5c
fix(route): a session's later stream shares its generation under doubt
kixelated Nov 15, 2026
4c74cda
Merge origin/main into quest/m2/cluster-routes/flapping-publisher
kixelated Nov 15, 2026
6898b27
test(verify): TwoHomes.cfg checks Stable again under a patient timer
kixelated Nov 15, 2026
da31c09
docs(draft): the request-loop guarantee names the three-home exception
kixelated Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 17 additions & 11 deletions .claude/skills/verify-route/SKILL.md

Large diffs are not rendered by default.

10 changes: 6 additions & 4 deletions drafts/draft-lcurley-moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -542,7 +542,7 @@ Its priority is the lowest local priority of a session on its Path: a receiver t
A node ranks its usable routes to an origin node at a prefix: one that is not draining first, then the highest priority, then the lowest metric.
Extending a route never clears its drain, never raises its priority, and always raises its metric, so no node ranks a route above how the neighbour it came from ranked it, and selection converges.
A node selects its best-ranked usable route at once.
A route is unusable while its Path names the node, it is held, a hop on its Path is dead, a node on its Path is in doubt (see [Dead Hops](#dead-hops)) unless it is the route the node already uses, it arrived with 32 hops, or its metric overflows.
A route is unusable while its Path names the node, it is held, a hop on its Path is dead, the doubt keeps the node off it (see [Dead Hops](#dead-hops)), it arrived with 32 hops, or its metric overflows.
A node MAY mark a session upstream.
A route is down-only when it arrived on an upstream session or with its down-only flag set; a node sends a down-only route with the flag set, and never on an upstream session.

Expand All @@ -553,7 +553,7 @@ An origin never advertises a Revision of 2^64-1: a change that would reach it in

A SUBSCRIBE, FETCH, TRACK, TRACKS, or DATAGRAMS request names the origin node that serves it, and the Route Depth of the route it follows: how many leading segments of its Broadcast Path make up the route's prefix.
A subscriber that names 0 lets the publisher select one (see [Resolution](#resolution)), and a relay forwarding a request names the origin node and Route Depth of the route it selected or was given.
A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin or loops.
A relay forwards a request naming an origin node only along the route it sent the requester to that node at that prefix, or a SUBSCRIBE through the neighbor [Warm Routes](#warm-routes) chooses for its broadcast, and resets it with UNROUTABLE when that route is no longer usable or its Path names the requester, so the request never reaches another origin, and loops only briefly after an origin on three or more relays leaves them all at once (see [Dead Hops](#dead-hops)).

Claims with the same Instance serve the same Broadcast, so a relay MAY move a live subscription between them, including onto a higher Epoch, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber.
A relay MUST NOT move a subscription to a claim with a different Instance: it stays on its Instance's claims and ends with the last of them.
Expand All @@ -575,7 +575,9 @@ Then every route through the hop is dead, whoever sent it: the sender's routes c
The receiver remembers the dead hop, so a route through it that arrives late stays unusable.

A closed session also casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time: a neighbour that has not heard yet may still offer a stale route through it.
A receiver doubts the far node for one second, during which it moves onto no route through that node; the route it already uses stays usable.
A receiver doubts the far node for one second, during which it moves onto no route whose Path names that node after the origin node's hop, and onto no route the doubted node originates whose Path is that node's hop alone, received on a session that opened since the doubt began; the route it already uses stays usable.
Any other route the doubted node originates stays usable, since a route cannot loop through its own origin: an origin on several relays stays reachable through the others while its session to one of them flaps, and a session that reopens waits out the doubt, which damps the flaps.
A node cannot tell an origin leaving from its session to one relay closing, so an origin on three or more relays that leaves them all at once can send a request round them, each relay on a neighbour's route that the neighbour's own close made stale, until that neighbour's DEAD arrives.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Nothing on the wire carries the doubt.
A node that sees a session close doubts its far end in the same way.

Expand Down Expand Up @@ -2552,7 +2554,7 @@ An application-specific payload.
- ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route.
- ROUTE_START and ROUTE_UPDATE carry a Route Offer that leads with the route's Path: one hop per node the route crossed, each a Hop ID and the Session Number that node learned the route on, at most 32. A node never uses a route whose Path names it. ROUTE_START also carries a Route Suffix naming the prefix the route covers, and the offer carries the route's drain (Flags 0x4) and priority, which rank before its metric.
- Added ROUTE_HOLD (0xB), which holds a route while it may recover, for at most 10 seconds; ROUTE_END lets it go. Neither withdraws the origin's claims.
- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of loops and stale copies, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there.
- Added DEAD (0xF): a node that loses a session names that hop once to each neighbour it sent a route through it, listing the routes that moved, and every node drops every route through the hop and doubts the node at the far end for a second, keeping nodes off routes through it but not off one it originates, except over a session to it opened since; a DRAIN names a node that sent a GOAWAY, whose routes rank draining. The path, the dead hop, holds, and the doubt keep routes free of stale copies, and of loops except briefly when an origin on three or more relays leaves them all at once, with no distance, sequence number, route request, or echo. On a scoped session a DEAD applies only to the routes heard there.
- A node closes a session whose peer leaves an Announce Stream undrained for 10 seconds, and the subscriber sends nothing on the stream after ANNOUNCE_REQUEST.
- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the route's prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows.
- A route presents only its most specific claim covering the requested prefix, moved to the next covering claim when it is withdrawn rather than ended.
Expand Down
4 changes: 3 additions & 1 deletion quest/m2/cluster-routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,14 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout
- A hop names a relay and the session it learned the route on. A relay uses the cheapest route whose path does not name it, at once.
- A relay that loses a session names that hop, and every relay drops every route through it, whoever offered it. One message per session covers every publisher and lists the routes that moved to another path.
- A relay with no route left holds its claims, unavailable, until each neighbour that learned the route from it has answered: with another way, or with nothing once those behind it have answered. Then it lets go.
- A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt.
- A closed session casts doubt on the node at its far end, since a dead relay is reported one neighbour at a time. No relay moves to a route through that node until a timer lifts the doubt. The route a relay already uses is exempt, and so is a route that begins at that node, unless it comes over a session to that node opened since the doubt began (below).

The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review."

Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it.

Decided 2026-11-14 (taken by the agent under `/quest-afk`, for the maintainer's review, on [the flapping publisher](https://github.com/moq-dev/moq/pull/8734)): ✅ **doubt blocks a route through a node, not one that begins at it, except the node's own route on a session to it that opened since the doubt began.** A publisher on two relays whose link to one flaps faster than the timer stays routed through the other: a relay that already hears a route through the other home that does not name it keeps one the whole time, as every relay does in the measured mesh (the two homes joined, a third relay behind both), and a relay whose neighbours all route through it is routed again once one of them re-offers, a round trip after the news, instead of after the timer. A publisher that moves to another relay is routed in 50 ms in a 24-relay clique and 325 ms in a ring, where it waited 1,100 and 1,850 ms. The returning link still waits out the timer, so no relay hears it until the relay it returned to takes it, which keeps the timer's damping of a flapping link: in 400 churn cases (120,000 steps) with the publisher on one relay, the rule costs 3% more route messages than timer-only doubt with no fewer relays routed, and with the publisher on two relays it routes relays 15% more often for 22% more route messages, 6% more claims sent, and 4% more claims taken offline. A relay cannot tell the publisher dying from one of its links closing, so the model shows the cost: a publisher on three or more relays that leaves them all at once can send a request round them, each on a neighbour's direct route that neighbour's own close made stale, until the news arrives one hop later (`Homes.cfg`, `LoopsLostThePublisher`). The two-home churn, which flaps random links and restarts relays but never takes a publisher off three relays, shows a route loop in 51 of its 120,000 steps. With two homes the model finds no loop and no claim dropped (`TwoHomes.cfg`), given a timer that outlasts every stale offer as well as the DEAD (`Patient`). This also settles the fresh-publisher quest, which proposed the opposite exemption and is deleted: its measurements showed that taking a returning link at once costs more traffic and claims offline than it saves. Rejected: keeping timer-only doubt, under which a dual-homed publisher's other link is unusable for the whole timer; sparing every route that begins at the doubted node, which routes relays 67% more often with two homes but costs 121% more route messages, 76% more claims sent, and 69% more claims offline, since each return of a flapping link reaches every relay; sparing only links heard before the doubt began, which keeps the damping but rarely helps a relay whose neighbours had not chosen the other link; exempting the publisher only at relays that heard of the close in a DEAD, which leaves the relay on the flapping link dark; a liveness message the publisher floods along its routes when it loses a session, which keeps requests loop-free but adds a wire message, still leaves relays dark for a round trip, and waited on [the switch](https://github.com/moq-dev/moq/pull/8731); holding under a next hop that has yet to answer a dead route whatever its path, which the router's sweeps showed can hold for good.

Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages").

Declined 2026-11-14 (maintainer), timer-only doubt retained over it: a fresh publisher's exemption, under which a relay moves at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route:
Expand Down
66 changes: 0 additions & 66 deletions quest/m2/cluster-routes/flapping-publisher.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m3/cluster-routing/request-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ pending request (the front in `rs/moq-net/src/model/front.rs`, and how

## Related

- [A dual-homed publisher stays reachable while one link flaps](/quest/m2/cluster-routes/flapping-publisher.md) - another failure mode from the same BGP comparison
- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives
Loading