Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion quest/m2/cluster-routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,27 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout

The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review."

Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it.
Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it.

Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages").

Declined 2026-11-14 (maintainer), timer-only doubt retained over it: a fresh publisher's exemption, under which a relay moves at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route:

| Case | Rule | Recovery | Dark | Messages | Claims |
| --- | --- | ---: | ---: | ---: | ---: |
| Clique rejoin | timer | 1,050 | 984 | 1,105 | 96 |
| Clique rejoin | exempt | 1,025 | 921 | 1,105 | 96 |
| Ring rejoin | timer | 1,875 | 1,512 | 118 | 96 |
| Ring rejoin | exempt | 1,800 | 1,380 | 120 | 88 |
| Sparse rejoin | timer | 1,372 | 1,194 | 164 | 100 |
| Sparse rejoin | exempt | 1,285 | 1,056 | 187 | 91 |
| Clique move | either | 1,100 | 1,032 | 1,612 | 2,124 |
| Ring move | either | 1,850 | 1,488 | 117 | 92 |
| Churn | timer | | 162,654 | 8,785 | 5,698 |
| Churn | exempt | | 116,404 | 13,646 | 10,524 |

Neither rule looped, and churn's stale observations stayed near 3,300 under both.

28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](https://github.com/moq-dev/moq/pull/8728) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused).

29. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), asked how the path rule keeps the `upstream` guarantee, which the prototype left out and [the switch](/quest/m2/cluster-routes/switch.md) had listed for removal with Babel's machinery: **a route keeps one sticky down-only flag beside its path.** A relay sets it on a route it learns over a link marked `upstream`, it is never cleared on a later hop, and a relay never sends a flagged route over an upstream link. The flag is export policy, not loop avoidance, so the path does not replace it: a relay that hears a CDN route from a LAN peer learned it on a link that is not upstream, and only the flag tells it the route came down from an upstream. That case is a local cluster with more than one uplink ([outbound federation](/quest/m3/cluster-routing/federation.md)), where a relay whose own uplink died reaches the CDN through a peer; an edge that dials only its cores needs just the local rule. This is the Only-To-Customer attribute of BGP roles (RFC 9234). The router (`rs/moq-net/src/route/path.rs`, built in [#8622](https://github.com/moq-dev/moq/pull/8622)) carries the flag and its export check, [the switch](/quest/m2/cluster-routes/switch.md) adds them to `RouteHandoff.tla` and keeps the flag on ROUTE_START and ROUTE_UPDATE, and [acceptance](/quest/m2/cluster-routes/acceptance.md)'s two-uplink mesh proves it. Sender side only, as today. Rejected: a mark on each hop of the path, which gives the same answer from more bytes and model state; the local rule with no flag, which either leaks through a LAN peer or gives up the dead-uplink fallback; also refusing a flagged route that arrives from a link whose grant marks the sender as a customer, which can be added later without a wire change.
Expand Down
51 changes: 0 additions & 51 deletions quest/m2/cluster-routes/fresh-publisher.md

This file was deleted.

2 changes: 2 additions & 0 deletions rs/moq-srt/src/server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -868,6 +868,8 @@ mod tests {
use moq_mux::container::{Producer, ts};
use moq_net::Timestamp;

kio::clock::host_time(); // real SRT sockets

// Short enough to keep the test quick, long enough that the release instants
// either side of the marker are separated by more than clock noise.
const LATENCY: Duration = Duration::from_millis(300);
Expand Down