From 4114171a129df4ea2b3a09592140e5ce2167767b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 03:14:13 -0800 Subject: [PATCH 1/8] chore(quest): claim quest/m2/cluster-routes/fresh-publisher.md Quest-Claim: 0c8dc5ad7da97ac5 From 4babb59da7c11157acbab50231a45b1932ba6b18 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 05:54:01 -0800 Subject: [PATCH 2/8] fix(moq-srt): the publisher-marker test runs on the host clock it uses The test drives real SRT sockets, so its jitter engine waits in real time. Under load that wait ran out, and KIO_DENY_HOST_TIMERS failed the test, as its sibling already declares with `kio::clock::host_time()`. Co-Authored-By: Claude Opus 5.5 --- rs/moq-srt/src/server.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rs/moq-srt/src/server.rs b/rs/moq-srt/src/server.rs index 1af2858c62d..4b365bb9b08 100644 --- a/rs/moq-srt/src/server.rs +++ b/rs/moq-srt/src/server.rs @@ -868,6 +868,8 @@ mod tests { use moq_mux::container::{Producer, ts}; use moq_net::Timestamp; + kio::clock::host_time(); // real SRT sockets + // Short enough to keep the test quick, long enough that the release instants // either side of the marker are separated by more than clock noise. const LATENCY: Duration = Duration::from_millis(300); From ad444d7f2a8e13c6bec42d0add9183f468ca20e6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 05:54:32 -0800 Subject: [PATCH 3/8] test(moq-net): model and measure a fresh publisher's doubt exemption The route model gains `Exempt`: "fresh" lets a relay move at once onto a doubted publisher's own route, offered on a session the publisher opened after the doubt began, and "own" and "session" each drop one condition. `Fresh.cfg` checks it beside its timer-only control `Reconnect.cfg`. The router runs it as the comparison rule `Rule::FRESH` in test builds only, and the route benchmarks compare it. It is safe in every bound checked, but frees only the publisher's own relay: the 24-relay reconnect cases end 25 to 75 ms sooner, a publisher that moves gains nothing, and churn sends 55% more route messages. The quest now waits on the maintainer's call, recommending timer-only doubt. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 9 ++- quest/m2/cluster-routes/README.md | 2 +- quest/m2/cluster-routes/fresh-publisher.md | 94 +++++++++++++--------- rs/moq-net/src/route/path.rs | 57 +++++++++++-- rs/moq-net/src/route/tests/compare.rs | 5 +- rs/moq-net/src/route/tests/mod.rs | 7 ++ rs/moq-net/verify/route/Blind.cfg | 1 + rs/moq-net/verify/route/Dead.cfg | 1 + rs/moq-net/verify/route/Drain.cfg | 1 + rs/moq-net/verify/route/DrainRing.cfg | 1 + rs/moq-net/verify/route/Early.cfg | 1 + rs/moq-net/verify/route/Fresh.cfg | 34 ++++++++ rs/moq-net/verify/route/HoldCycle.cfg | 1 + rs/moq-net/verify/route/Lag.cfg | 1 + rs/moq-net/verify/route/LagWide.cfg | 1 + rs/moq-net/verify/route/Loop.cfg | 1 + rs/moq-net/verify/route/Overlap.cfg | 1 + rs/moq-net/verify/route/Prefer.cfg | 1 + rs/moq-net/verify/route/Reconnect.cfg | 35 ++++++++ rs/moq-net/verify/route/Ring.cfg | 1 + rs/moq-net/verify/route/RouteHandoff.cfg | 1 + rs/moq-net/verify/route/RouteHandoff.tla | 47 +++++++++-- rs/moq-net/verify/route/Twice.cfg | 1 + 23 files changed, 247 insertions(+), 57 deletions(-) create mode 100644 rs/moq-net/verify/route/Fresh.cfg create mode 100644 rs/moq-net/verify/route/Reconnect.cfg diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index c26aa3e6551..4fccf3bc47b 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. `Exempt` compares exemptions from doubt for [the fresh publisher](/quest/m2/cluster-routes/fresh-publisher.md): `"none"` is the router's rule; `"fresh"` lets a relay move at once onto the publisher's own route, offered on a session the publisher opened while the relay doubted it, as `Rule::FRESH` runs it in the router's tests (`Doubt::fresh`); `"own"` drops the session's freshness, and `"session"` drops that the route is the publisher's own. ## Rules @@ -31,7 +31,7 @@ Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regres No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. -What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). With two failures a stale route can return whatever the timer (`Twice.cfg` drops `NoStaleReturn`), and `StaleIsChased` is the bound. +What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). With two failures a stale route can return whatever the timer (`Twice.cfg` drops `NoStaleReturn`), and `StaleIsChased` is the bound. Exempting the publisher's own route on a session it opened while doubted breaks no rule (`Fresh.cfg`, beside its control `Reconnect.cfg`), and neither does `"own"` or `"session"` on the same bounds, so these bounds do not show what the exemption's two conditions buy. ## Run @@ -79,6 +79,8 @@ change; faster checkers are separate work. | `Drain.cfg` | The publisher joined to two relays joined to each other; both hear its GOAWAY. Ranking only a relay's own drain exceeded `Budget`. | 1 claim change, 1 session change, GOAWAY on each session | 648,218 | two minutes | | `DrainRing.cfg` | The publisher joined to three relays in a ring that is cheap one way round (`Ring`); all hear its GOAWAY. Without the DRAIN news and its doubt, a request looped round the ring with no session closed, breaking `LoopsAreStale`. Simulated. | 1 claim change, GOAWAY on each session | 1,000 behaviours per worker, depth 150, seed 1 | about a minute | | `Prefer.cfg` | The same triangle, where both relays prefer the link between them. Ranking only a relay's own link priority exceeded `Budget`. | 1 claim change, 1 session change | 14,630 | seconds | +| `Reconnect.cfg` | The publisher on relays 1 and 2, which are joined. Its sessions close and open again, so it rejoins a relay that doubts it or moves to the other. The control for `Fresh.cfg`. | 1 claim change, 3 session changes | 1,490,147 | three minutes | +| `Fresh.cfg` | The same, with `Exempt = "fresh"`. | 1 claim change, 3 session changes | 1,822,789 | five minutes | | `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes | Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. @@ -89,13 +91,14 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::FRESH` is `Exempt = "fresh"`. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | | `rs/moq-net/src/route/path.rs` `Router::run`, `Router::select`, `Router::retain` | `Select` plans live routes and their reflected sources before `SelectKey` decides holds. | | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | | `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` | +| `rs/moq-net/src/route/path.rs` `Router::barred`, `Router::fresh`, `Router::suspect` | `Barred`, `Exempted`, `Own`, `Opened`: a doubt records how many sessions were open when it began (`since`), so a session opened later is fresh. | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | | `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` | | `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 2e7e945cab0..7bc2aa358a4 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -53,7 +53,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." - Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it. + Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it: [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) found it safe but freeing only the publisher's own relay, and the call waits on the maintainer. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/quest/m2/cluster-routes/fresh-publisher.md b/quest/m2/cluster-routes/fresh-publisher.md index 10e5a978847..aaca1dcf06d 100644 --- a/quest/m2/cluster-routes/fresh-publisher.md +++ b/quest/m2/cluster-routes/fresh-publisher.md @@ -1,51 +1,69 @@ -# [L] Model and measure a fresh publisher's doubt exemption +# [S] Adopt or retain a fresh publisher's doubt exemption ## Goal -Determine whether a fresh authenticated direct publisher session can recover -its own route before the one-second doubt expires, without admitting a stale -return or a request loop. Compare it with timer-only doubt before deciding -whether to adopt it. +Settle whether a relay moves at once onto a doubted publisher's own route, +offered on an authenticated session the publisher opened after the doubt +began, or keeps timer-only doubt, and leave the tree matching the choice. ## Plan -Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, -while the maintainer was away): -timer-only doubt is the baseline in -[moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622), and this exemption -is modeled and measured separately. Early collection is rejected because it -permits the request loop recorded by `Loop.cfg`. These choices remain open -to change in review. - -The fixed 24-relay reconnect cases take 1,050 ms in a clique and 1,875 ms in -a ring, including propagation at 25 ms per hop, versus 50 ms and 325 ms for -the distance rule. Both path policies wait because heard routes still name -the doubted publisher. The exemption could remove that wait for a publisher -returning directly, but not for a relay presenting somebody else's route. - -Model what authenticates the publisher and binds the offer to the new direct -session, including delayed offers from the old session, reconnects, and -concurrent link failures. Keep the timer-only model as the control. Reproduce -counterexamples on the router and measure the same reconnect and churn -scripts on both policies, reporting traffic, request recovery, stale routes, -and reachable but unrouted observations. Preserve the baseline's safety -checks; a faster reconnect alone does not justify the exemption. - -Recommend limiting any exemption to the new authenticated publisher's own -direct offer. A forwarded offer or a new transport session alone does not -establish that the origin is alive. Record the evidence and adoption decision -before changing the default, and mirror an adopted rule in Rust, JS, and the -draft wherever the wire switch has already landed. +[moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and +measured the exemption beside timer-only doubt, the baseline of epic Plan +item 27. The router runs it as the comparison rule `Rule::FRESH` +(`Doubt::fresh`, test builds only), and `RouteHandoff.tla` as +`Exempt = "fresh"`. A route qualifies when its path names only the session's +authenticated peer, and that session opened after the relay's doubt of the +peer began. + +- Safety: every rule holds exhaustively on `Fresh.cfg`, as on its control + `Reconnect.cfg` (the publisher on two joined relays, three session + changes), `NoStaleReturn` included. With a third relay behind both, the + control and the exemption both break `NoStaleReturn` through two closes + and no reconnect, `Twice.cfg`'s accepted case; without it, the exemption + showed no violation in 12.9M states, every ordering to depth 14, before a + 45-minute cap. The router's fixtures and sweeps pass under the rule + (2,000 scenarios a sweep), and churn shows no loop. +- Reconnect cases, 24 relays: the publisher's own relay recovers in one hop + of delay instead of after the timer, but every other relay still doubts the + publisher and refuses the forwarded route, so recovery ends 25 ms sooner in + the clique (1,025 ms) and 75 ms sooner in the ring (1,800 ms). A publisher + that moves to another relay gains nothing: that relay opened its session + before the DEAD reached it. In the ring, DEADs echoing back round it extend + each relay's doubt to as late as 1,575 ms. +- Churn (400 cases, the publisher's link flapping most): 28% fewer + reachable but unrouted observations, but 55% more route messages, 85% more + claims sent, and 80% more claims seen going offline. Timer-only doubt damps + a flapping publisher; the exemption shows each flap. +- Neither broader variant breaks a rule within these bounds: `"own"`, the + publisher's own route on a session open before the doubt, and `"session"`, + any route on a session opened while doubting. The model's honest nodes and + atomic closes cannot show why the exemption needs both conditions. + +Adopting: move `Router::fresh` and `since` out of the test build, make the +exemption the default, mirror it in `js/net` once +[the switch](/quest/m2/cluster-routes/switch.md) runs the path router there, +and update epic Plan item 27 and the `verify-route` skill. Doubt stays local, +so the wire and the draft are unchanged. + +Retaining: delete this quest. `Rule::FRESH` stays a comparison rule beside +`Rule::EARLY`, and `Exempt` in the model. + +[The flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md) +lets no doubt block a route at its own origin. That also frees the relays +this exemption cannot, so if it lands first, delete this quest. ## Waiting -- Maintainer: after the model and measurements, adopt the narrowly authenticated - direct-publisher exemption or retain timer-only doubt. Recommend adoption only - if it preserves the safety checks and improves the measured reconnect cases; - otherwise retain the baseline. An agent can gather that evidence, but adopting - an exception changes the coordinator's accepted routing policy and needs an - explicit decision. +- Maintainer: adopt the narrowly authenticated direct-publisher exemption, + or retain timer-only doubt. Recommended: retain. Safety holds, but the + measured reconnect cases improve by one to three hops of delay at most and + not at all when the publisher moves, while a flapping publisher costs much + more traffic. An agent cannot take it: adopting changes the accepted + routing policy of epic Plan item 27. Done when this quest is deleted, or + rewritten to the adopting work. ## Related - [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - supplies the authenticated session boundary when it puts the rule on the wire +- [A dual-homed publisher stays reachable while one link flaps](/quest/m2/cluster-routes/flapping-publisher.md) - a broader rule that would cover this exemption diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 0c318bfa13c..702757b97c2 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -180,6 +180,10 @@ pub(crate) struct Doubt { pub timer: Duration, #[cfg(test)] pub lift: Lift, + /// The comparison rule exempting a doubted node's own route, offered on a + /// session it opened after the doubt began. + #[cfg(test)] + pub fresh: bool, } impl Doubt { @@ -188,6 +192,8 @@ impl Doubt { timer: Duration::from_secs(1), #[cfg(test)] lift: Lift::Timer, + #[cfg(test)] + fresh: false, }; } @@ -511,6 +517,10 @@ pub(crate) struct Router { dead: BTreeMap, /// Each node's doubt and deadline. doubted: BTreeMap, + /// For each doubted node, the sessions opened when the doubt began: a later one + /// is fresh for [`Doubt::fresh`]. + #[cfg(test)] + since: BTreeMap, /// Nodes that sent GOAWAY, kept while a heard route names one or it is in doubt. draining: BTreeSet, watch: Path, @@ -584,6 +594,8 @@ impl Router { filed: BTreeMap::new(), dead: BTreeMap::new(), doubted: BTreeMap::new(), + #[cfg(test)] + since: BTreeMap::new(), draining: BTreeSet::new(), watch: Path::empty(), interest: 0, @@ -1066,6 +1078,8 @@ impl Router { let mut ids = BTreeSet::new(); for node in due { self.doubted.remove(&node); + #[cfg(test)] + self.since.remove(&node); ids.extend(self.naming(node)); } for hop in expired { @@ -1301,6 +1315,30 @@ impl Router { .any(|hop| matches!(self.doubted.get(&hop.node), Some(Suspicion::Doubted(_)))) } + /// Whether doubt keeps this node from moving onto `path`, which `peer` offered. + fn barred(&self, peer: N, path: &[Hop]) -> bool { + #[cfg(test)] + if self.fresh(peer, path) { + return false; + } + #[cfg(not(test))] + let _ = peer; + self.doubts_path(path) + } + + /// [`Doubt::fresh`]: `path` is `peer`'s own route, offered on a session opened + /// after this node began doubting it. + #[cfg(test)] + fn fresh(&self, peer: N, path: &[Hop]) -> bool { + let session = &self.sessions[&peer]; + self.doubt.fresh + && matches!(path, [hop] if hop.node == session.hop) + && self + .since + .get(&session.hop) + .is_some_and(|since| session.number > *since) + } + fn hear(&mut self, key: RouteId, from: N, heard: Heard) { self.scopes.entry(key.0).or_default().insert(key.1.clone()); for hop in heard.offer.path.iter() { @@ -1390,10 +1428,19 @@ impl Router { if !self.draining.insert(node) { return BTreeSet::new(); } + self.suspect(node, now); + self.naming(node) + } + + /// Doubt `node` until the timer fires from `now`, or later if it already runs. + fn suspect(&mut self, node: Id, now: Instant) { let at = now.saturating_add(self.doubt.timer); let at = self.doubted.get(&node).map_or(at, |state| state.deadline().max(at)); + #[cfg(test)] + if !matches!(self.doubted.get(&node), Some(Suspicion::Doubted(_))) { + self.since.insert(node, self.opened); + } self.doubted.insert(node, Suspicion::Doubted(at)); - self.naming(node) } /// The session `hop` names closed. `from` reported it, with what became of its own @@ -1461,9 +1508,7 @@ impl Router { } } if far != self.hop { - let at = now.saturating_add(self.doubt.timer); - let at = self.doubted.get(&far).map_or(at, |state| state.deadline().max(at)); - self.doubted.insert(far, Suspicion::Doubted(at)); + self.suspect(far, now); // Any route through that node may have become unusable. ids.extend(self.naming(far)); } @@ -1514,7 +1559,7 @@ impl Router { let Some(metric) = heard.offer.metric.checked_add(session.link.cost) else { continue; }; - if Some(peer) != using && self.doubts_path(&heard.offer.path) { + if Some(peer) != using && self.barred(peer, &heard.offer.path) { // It may come back once the doubt lifts. continue; } @@ -1630,7 +1675,7 @@ impl Router { heard.state == State::Live && !names(&heard.offer.path, self.hop) && heard.offer.metric.checked_add(self.sessions[peer].link.cost).is_some() - && self.doubts_path(&heard.offer.path) + && self.barred(*peer, &heard.offer.path) }) } diff --git a/rs/moq-net/src/route/tests/compare.rs b/rs/moq-net/src/route/tests/compare.rs index 400997f50a1..b0e743b458e 100644 --- a/rs/moq-net/src/route/tests/compare.rs +++ b/rs/moq-net/src/route/tests/compare.rs @@ -1,6 +1,7 @@ //! One script on both rules: [`Router`], which keeps Babel's distance, and //! [`path::Router`], which keeps its doubt until the timer fires. The ignored -//! benchmarks also compare the rejected early-collection policy. +//! benchmarks also compare the rejected early-collection policy and the +//! fresh-publisher exemption. //! //! A [`Lab`] builds the same mesh on each, applies one change at a time, and //! delivers in rounds: every message queued when a round starts crosses its link @@ -17,7 +18,7 @@ use super::*; const RULES: [Rule; 2] = [Rule::Distance, Rule::PATH]; /// The rules compared, in the report's order. -pub(super) const COMPARED: [Rule; 3] = [Rule::Distance, Rule::EARLY, Rule::PATH]; +pub(super) const COMPARED: [Rule; 4] = [Rule::Distance, Rule::EARLY, Rule::PATH, Rule::FRESH]; /// A queue of messages, or of echoed ROUTE_ENDs, from one node to another. #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 85f7f5d9667..9a573913eb9 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -51,6 +51,12 @@ impl Rule { lift: Lift::Unnamed, ..Doubt::DEFAULT }); + /// The comparison rule exempting a doubted publisher's own route on a session it + /// opened after the doubt began. + pub const FRESH: Self = Self::Path(Doubt { + fresh: true, + ..Doubt::DEFAULT + }); pub fn as_str(self) -> &'static str { match self { @@ -58,6 +64,7 @@ impl Rule { Self::Path(Doubt { lift: Lift::Unnamed, .. }) => "path+early", + Self::Path(Doubt { fresh: true, .. }) => "path+fresh", Self::Path(Doubt { lift: Lift::Timer, .. }) => "path", } } diff --git a/rs/moq-net/verify/route/Blind.cfg b/rs/moq-net/verify/route/Blind.cfg index aa709100bdb..cf9b777d93b 100644 --- a/rs/moq-net/verify/route/Blind.cfg +++ b/rs/moq-net/verify/route/Blind.cfg @@ -18,6 +18,7 @@ CONSTANTS MaxRev = 2 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Dead.cfg b/rs/moq-net/verify/route/Dead.cfg index 1e1fc4126c8..2e7e1bf350a 100644 --- a/rs/moq-net/verify/route/Dead.cfg +++ b/rs/moq-net/verify/route/Dead.cfg @@ -19,6 +19,7 @@ CONSTANTS MaxRev = 3 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Drain.cfg b/rs/moq-net/verify/route/Drain.cfg index f3da411c468..999583a56e1 100644 --- a/rs/moq-net/verify/route/Drain.cfg +++ b/rs/moq-net/verify/route/Drain.cfg @@ -18,6 +18,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/DrainRing.cfg b/rs/moq-net/verify/route/DrainRing.cfg index 2698259d236..2eff34c64e2 100644 --- a/rs/moq-net/verify/route/DrainRing.cfg +++ b/rs/moq-net/verify/route/DrainRing.cfg @@ -21,6 +21,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 0 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Early.cfg b/rs/moq-net/verify/route/Early.cfg index 96b9e2e89f4..2f2c907e438 100644 --- a/rs/moq-net/verify/route/Early.cfg +++ b/rs/moq-net/verify/route/Early.cfg @@ -15,6 +15,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 1 Patient = FALSE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Fresh.cfg b/rs/moq-net/verify/route/Fresh.cfg new file mode 100644 index 00000000000..fd10ae685a8 --- /dev/null +++ b/rs/moq-net/verify/route/Fresh.cfg @@ -0,0 +1,34 @@ +\* `Reconnect.cfg` where a relay takes at once the publisher's own route on a +\* session the publisher opened while the relay doubted it. +CONSTANTS + Nodes = {0, 1, 2} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}} + Dear = {} + Ring = {} + Preferred = {} + Leaving = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 3 + Patient = TRUE + Exempt = "fresh" + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/HoldCycle.cfg b/rs/moq-net/verify/route/HoldCycle.cfg index 6e26a5171ae..ece248e008e 100644 --- a/rs/moq-net/verify/route/HoldCycle.cfg +++ b/rs/moq-net/verify/route/HoldCycle.cfg @@ -20,6 +20,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Lag.cfg b/rs/moq-net/verify/route/Lag.cfg index 1b13967a558..e8d08537e67 100644 --- a/rs/moq-net/verify/route/Lag.cfg +++ b/rs/moq-net/verify/route/Lag.cfg @@ -16,6 +16,7 @@ CONSTANTS MaxRev = 2 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/LagWide.cfg b/rs/moq-net/verify/route/LagWide.cfg index 4ebc496667d..057c1cad474 100644 --- a/rs/moq-net/verify/route/LagWide.cfg +++ b/rs/moq-net/verify/route/LagWide.cfg @@ -14,6 +14,7 @@ CONSTANTS MaxRev = 2 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index 5dc279c44d1..49ec66b81e9 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -16,6 +16,7 @@ CONSTANTS MaxRev = 0 MaxFlaps = 3 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Overlap.cfg b/rs/moq-net/verify/route/Overlap.cfg index aeb6101abc3..536b5d0764b 100644 --- a/rs/moq-net/verify/route/Overlap.cfg +++ b/rs/moq-net/verify/route/Overlap.cfg @@ -17,6 +17,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 0 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Prefer.cfg b/rs/moq-net/verify/route/Prefer.cfg index d2047f90906..6a3daf60e06 100644 --- a/rs/moq-net/verify/route/Prefer.cfg +++ b/rs/moq-net/verify/route/Prefer.cfg @@ -17,6 +17,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/Reconnect.cfg b/rs/moq-net/verify/route/Reconnect.cfg new file mode 100644 index 00000000000..ce710a5d4b6 --- /dev/null +++ b/rs/moq-net/verify/route/Reconnect.cfg @@ -0,0 +1,35 @@ +\* The publisher (0) on relays 1 and 2, which are joined. Its sessions close +\* and open again, so it rejoins a relay that doubts it, or moves to the other, +\* and a second session may close meanwhile. The control for `Fresh.cfg`. +CONSTANTS + Nodes = {0, 1, 2} + Origin = 0 + Links = {{0, 1}, {0, 2}, {1, 2}} + Dear = {} + Ring = {} + Preferred = {} + Leaving = {} + Scoped = {} + Claims = {"x"} + Narrow = {} + MaxRev = 1 + MaxFlaps = 3 + Patient = TRUE + Exempt = "none" + +SPECIFICATION Spec + +INVARIANTS + Truthful + NoRevival + NoFalseEnd + NoRegression + Stable + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + NoStaleReturn + QuietIsSettled + Budget diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg index c8234dd7ae1..81b262093ec 100644 --- a/rs/moq-net/verify/route/Ring.cfg +++ b/rs/moq-net/verify/route/Ring.cfg @@ -16,6 +16,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 1 Patient = FALSE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index 0c5dcdd120d..96d4ab44e71 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -16,6 +16,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 1 Patient = TRUE + Exempt = "none" SPECIFICATION Spec diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index e5f2ec9ae35..a32906598b4 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -20,7 +20,8 @@ (* neighbour that learned the route from it has answered. *) (* - A closed session casts doubt on the node at its far end: no relay *) (* moves to a route through it until a timer lifts the doubt. The timer *) -(* is the step `Lift`, which may fire at any moment. *) +(* is the step `Lift`, which may fire at any moment. `Exempt` compares *) +(* letting a relay move sooner onto some routes through a doubted node. *) (* *) (* Each node's every entry point is one atomic step ending in `Update`, *) (* as in the router. Messages travel on one FIFO per direction. A *) @@ -44,7 +45,9 @@ CONSTANTS Narrow, \* the claims inside prefix "a" MaxRev, \* the publisher's claim changes MaxFlaps, \* session opens and closes - Patient \* TRUE: a doubt lifts only once no news about its node is in flight + Patient, \* TRUE: a doubt lifts only once no news about its node is in flight + Exempt \* which routes doubt does not keep a relay off (`Exempted`): + \* "none", the router's rule, or a comparison rule Relays == Nodes \ {Origin} @@ -163,6 +166,25 @@ Live(r) == r.on /\ ~r.held (* Selection on node n's state s (path.rs: `select`, `pending`, `hopes`). *) Doubted(s, path) == \E i \in DOMAIN path : path[i][1] \in s.doubt + +\* The route neighbour m offered for key k is m's own: its path names m alone. +\* Sessions authenticate their peer, so only m can offer it on m's session. +Own(s, m, k) == Len(s.rx[m][k].path) = 1 /\ s.rx[m][k].path[1][1] = m + +\* Doubt does not keep this node off the route m offered for key k. "fresh": +\* m's own route, on a session m opened while this node doubted it, so m was +\* alive after the news that cast the doubt. The comparison rules each drop one +\* of those conditions: "own" takes m's own route on any session, and "session" +\* any route on a session opened while this node doubted any node. +Exempted(s, m, k) == + CASE Exempt = "fresh" -> Own(s, m, k) /\ m \in s.fresh + [] Exempt = "own" -> Own(s, m, k) + [] Exempt = "session" -> m \in s.fresh + [] OTHER -> FALSE + +\* Only doubt keeps this node off the route m offered for key k. +Barred(s, m, k) == Doubted(s, s.rx[m][k].path) /\ ~Exempted(s, m, k) + \* The path crosses a node this node knows sent GOAWAY. Drains(s, path) == \E i \in DOMAIN path : path[i][1] \in s.draining @@ -199,7 +221,7 @@ Pending(n, s, k) == /\ s.tx[p][j].src = k \* A neighbour offers a route that only doubt keeps this node off. -Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Doubted(s, s.rx[m][k].path) +Hopes(n, s, k) == \E m \in Nodes : Usable(n, s, m, k) /\ Barred(s, m, k) \* `select` for one key: the best-ranked usable route, staying on the next hop \* among equals, then the lowest key. Doubt keeps a node from moving to a route, @@ -208,7 +230,7 @@ SelectKey(n, s, k, gens) == LET was == s.rt[k] current == IF was.on THEN was.next ELSE None using == IF Live(was) THEN was.next ELSE None - C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Doubted(s, s.rx[m][k].path))} + C == {m \in Nodes : Usable(n, s, m, k) /\ (m = using \/ ~Barred(s, m, k))} Path(m) == Append(s.rx[m][k].path, <>) Drain(m) == s.rx[m][k].drain \/ m \in s.drain \/ Drains(s, s.rx[m][k].path) R(m) == Rank(Path(m), Drain(m)) @@ -502,6 +524,7 @@ InitNode == [ expiry |-> {}, \* Pending timers belong to individual dead hops. drain |-> {}, \* the neighbours that sent GOAWAY on the session open to them draining |-> {}, \* nodes known to have sent GOAWAY, while named or doubted + fresh |-> {}, \* neighbours whose session opened while this node doubted them tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(k)]], told |-> [m \in Nodes |-> {}], toldDrain |-> [m \in Nodes |-> {}], @@ -623,7 +646,9 @@ Deliver(d) == \* the news to cross the mesh. Lift(n, x) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, - !.expiry = {h \in @ : h[2] # x}], + !.expiry = {h \in @ : h[2] # x}, + \* A later doubt of x postdates its session. + !.fresh = IF Exempt = "fresh" THEN @ \ {x} ELSE @], OV, up, gen, ended[n], NoReport, TRUE)] IN /\ x \in st[n].wait /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : @@ -665,14 +690,22 @@ Closed(s, y) == !.tx[y] = [k \in Keys |-> NoSent(k)], !.told[y] = {}, !.toldDrain[y] = {}, - !.drain = @ \ {y}] + !.drain = @ \ {y}, + !.fresh = @ \ {y}] + +\* A session to y opens, fresh as `Exempted` says. Only a comparison rule tracks +\* it, so the router's rule keeps the state space it had. +Opened(s, y) == + IF \/ Exempt = "fresh" /\ y \in s.doubt + \/ Exempt = "session" /\ s.doubt # {} + THEN [s EXCEPT !.fresh = @ \cup {y}] ELSE s Connect(l) == LET up1 == up \cup {l} gen1 == [gen EXCEPT ![l] = @ + 1] rs == [x \in l |-> LET y == CHOOSE y \in l : y # x - IN Update(x, Closed(st[x], y), OV, up1, gen1, ended[x], NoReport, FALSE)] + IN Update(x, Opened(Closed(st[x], y), y), OV, up1, gen1, ended[x], NoReport, FALSE)] IN /\ l \notin up /\ flaps < MaxFlaps /\ up' = up1 diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index 2b277a48b58..b596e3cd0a6 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -17,6 +17,7 @@ CONSTANTS MaxRev = 1 MaxFlaps = 2 Patient = TRUE + Exempt = "none" SPECIFICATION Spec From edadef35279de72e2f2cdba1688306432009e542 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 06:17:08 -0800 Subject: [PATCH 4/8] docs(verify-route): the model is in sync with the scoped path router Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 4fccf3bc47b..b152743f961 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -91,7 +91,7 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on th ## Coverage -Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::FRESH` is `Exempt = "fresh"`. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the scoped path router of [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622), which [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735)'s exemption extends. Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model; the comparison-only `Rule::FRESH` is `Exempt = "fresh"`. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | From 3c47bcff37e54205e8fbcadae6005fda5770b4e9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 09:34:59 -0800 Subject: [PATCH 5/8] test(moq-net): reject an unknown Exempt value in RouteHandoff.tla Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/verify/route/RouteHandoff.tla | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index a32906598b4..31801b14e09 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -49,6 +49,8 @@ CONSTANTS Exempt \* which routes doubt does not keep a relay off (`Exempted`): \* "none", the router's rule, or a comparison rule +ASSUME Exempt \in {"none", "fresh", "own", "session"} + Relays == Nodes \ {Origin} Cost == [l \in Links |-> IF l \in Dear THEN 2 ELSE 1] From a8625e486e063e835d61dd98811d100d97f45431 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 10:47:59 -0800 Subject: [PATCH 6/8] quest: retain timer-only doubt and delete the fresh-publisher quest The maintainer declined the exemption on 2026-11-14. Rule::FRESH and the model's Exempt stay as comparisons. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 2 +- quest/m2/cluster-routes/README.md | 2 +- quest/m2/cluster-routes/fresh-publisher.md | 69 ---------------------- 3 files changed, 2 insertions(+), 71 deletions(-) delete mode 100644 quest/m2/cluster-routes/fresh-publisher.md diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index b152743f961..c785d85f71e 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. `Exempt` compares exemptions from doubt for [the fresh publisher](/quest/m2/cluster-routes/fresh-publisher.md): `"none"` is the router's rule; `"fresh"` lets a relay move at once onto the publisher's own route, offered on a session the publisher opened while the relay doubted it, as `Rule::FRESH` runs it in the router's tests (`Doubt::fresh`); `"own"` drops the session's freshness, and `"session"` drops that the route is the publisher's own. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. `Exempt` compares a fresh publisher's exemptions from doubt, which [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) measured and the maintainer declined: `"none"` is the router's rule; `"fresh"` lets a relay move at once onto the publisher's own route, offered on a session the publisher opened while the relay doubted it, as `Rule::FRESH` runs it in the router's tests (`Doubt::fresh`); `"own"` drops the session's freshness, and `"session"` drops that the route is the publisher's own. ## Rules diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 7bc2aa358a4..743e4ef7556 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -53,7 +53,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." - Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it: [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) found it safe but freeing only the publisher's own relay, and the call waits on the maintainer. + Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was modeled and measured separately: [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) found it safe but freeing only the publisher's own relay, and the maintainer declined it on 2026-11-14. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). diff --git a/quest/m2/cluster-routes/fresh-publisher.md b/quest/m2/cluster-routes/fresh-publisher.md deleted file mode 100644 index aaca1dcf06d..00000000000 --- a/quest/m2/cluster-routes/fresh-publisher.md +++ /dev/null @@ -1,69 +0,0 @@ -# [S] Adopt or retain a fresh publisher's doubt exemption - -## Goal - -Settle whether a relay moves at once onto a doubted publisher's own route, -offered on an authenticated session the publisher opened after the doubt -began, or keeps timer-only doubt, and leave the tree matching the choice. - -## Plan - -[moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and -measured the exemption beside timer-only doubt, the baseline of epic Plan -item 27. The router runs it as the comparison rule `Rule::FRESH` -(`Doubt::fresh`, test builds only), and `RouteHandoff.tla` as -`Exempt = "fresh"`. A route qualifies when its path names only the session's -authenticated peer, and that session opened after the relay's doubt of the -peer began. - -- Safety: every rule holds exhaustively on `Fresh.cfg`, as on its control - `Reconnect.cfg` (the publisher on two joined relays, three session - changes), `NoStaleReturn` included. With a third relay behind both, the - control and the exemption both break `NoStaleReturn` through two closes - and no reconnect, `Twice.cfg`'s accepted case; without it, the exemption - showed no violation in 12.9M states, every ordering to depth 14, before a - 45-minute cap. The router's fixtures and sweeps pass under the rule - (2,000 scenarios a sweep), and churn shows no loop. -- Reconnect cases, 24 relays: the publisher's own relay recovers in one hop - of delay instead of after the timer, but every other relay still doubts the - publisher and refuses the forwarded route, so recovery ends 25 ms sooner in - the clique (1,025 ms) and 75 ms sooner in the ring (1,800 ms). A publisher - that moves to another relay gains nothing: that relay opened its session - before the DEAD reached it. In the ring, DEADs echoing back round it extend - each relay's doubt to as late as 1,575 ms. -- Churn (400 cases, the publisher's link flapping most): 28% fewer - reachable but unrouted observations, but 55% more route messages, 85% more - claims sent, and 80% more claims seen going offline. Timer-only doubt damps - a flapping publisher; the exemption shows each flap. -- Neither broader variant breaks a rule within these bounds: `"own"`, the - publisher's own route on a session open before the doubt, and `"session"`, - any route on a session opened while doubting. The model's honest nodes and - atomic closes cannot show why the exemption needs both conditions. - -Adopting: move `Router::fresh` and `since` out of the test build, make the -exemption the default, mirror it in `js/net` once -[the switch](/quest/m2/cluster-routes/switch.md) runs the path router there, -and update epic Plan item 27 and the `verify-route` skill. Doubt stays local, -so the wire and the draft are unchanged. - -Retaining: delete this quest. `Rule::FRESH` stays a comparison rule beside -`Rule::EARLY`, and `Exempt` in the model. - -[The flapping publisher](/quest/m2/cluster-routes/flapping-publisher.md) -lets no doubt block a route at its own origin. That also frees the relays -this exemption cannot, so if it lands first, delete this quest. - -## Waiting - -- Maintainer: adopt the narrowly authenticated direct-publisher exemption, - or retain timer-only doubt. Recommended: retain. Safety holds, but the - measured reconnect cases improve by one to three hops of delay at most and - not at all when the publisher moves, while a flapping publisher costs much - more traffic. An agent cannot take it: adopting changes the accepted - routing policy of epic Plan item 27. Done when this quest is deleted, or - rewritten to the adopting work. - -## Related - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - supplies the authenticated session boundary when it puts the rule on the wire -- [A dual-homed publisher stays reachable while one link flaps](/quest/m2/cluster-routes/flapping-publisher.md) - a broader rule that would cover this exemption From 9e77656f46a5c42d9faf0b1452ef44620bc4df36 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 23:48:50 -0800 Subject: [PATCH 7/8] quest: keep the exemption's results beside item 27's measurements Co-Authored-By: Claude Opus 5.5 --- quest/m2/cluster-routes/README.md | 6 +++--- quest/m2/cluster-routes/flapping-publisher.md | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 3ae96f14ead..be5b513851d 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -53,7 +53,9 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." - Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. + Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. + + Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). Timer-only doubt retained (maintainer, 2026-11-14), declining a fresh publisher's exemption: a relay would move at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route: @@ -72,8 +74,6 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Neither rule looped, and churn's stale observations stayed near 3,300 under both. - Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). - 28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](https://github.com/moq-dev/moq/pull/8728) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). 29. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), asked how the path rule keeps the `upstream` guarantee, which the prototype left out and [the switch](/quest/m2/cluster-routes/switch.md) had listed for removal with Babel's machinery: **a route keeps one sticky down-only flag beside its path.** A relay sets it on a route it learns over a link marked `upstream`, it is never cleared on a later hop, and a relay never sends a flagged route over an upstream link. The flag is export policy, not loop avoidance, so the path does not replace it: a relay that hears a CDN route from a LAN peer learned it on a link that is not upstream, and only the flag tells it the route came down from an upstream. That case is a local cluster with more than one uplink ([outbound federation](/quest/m3/cluster-routing/federation.md)), where a relay whose own uplink died reaches the CDN through a peer; an edge that dials only its cores needs just the local rule. This is the Only-To-Customer attribute of BGP roles (RFC 9234). The router (`rs/moq-net/src/route/path.rs`, built in [#8622](https://github.com/moq-dev/moq/pull/8622)) carries the flag and its export check, [the switch](/quest/m2/cluster-routes/switch.md) adds them to `RouteHandoff.tla` and keeps the flag on ROUTE_START and ROUTE_UPDATE, and [acceptance](/quest/m2/cluster-routes/acceptance.md)'s two-uplink mesh proves it. Sender side only, as today. Rejected: a mark on each hop of the path, which gives the same answer from more bytes and model state; the local rule with no flag, which either leaks through a LAN peer or gives up the dead-uplink fallback; also refusing a flagged route that arrives from a link whose grant marks the sender as a customer, which can be added later without a wire change. diff --git a/quest/m2/cluster-routes/flapping-publisher.md b/quest/m2/cluster-routes/flapping-publisher.md index 561e78d92bf..960b68df961 100644 --- a/quest/m2/cluster-routes/flapping-publisher.md +++ b/quest/m2/cluster-routes/flapping-publisher.md @@ -38,10 +38,10 @@ chased out (`StaleIsChased`) and that requests stay loop-free relay originates a route for a broadcast it carries ([cache-aware routing](/quest/m3/cache-routing/README.md)). -Exempting only a publisher's own new direct session from doubt was measured -and declined ([moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735), -epic Plan item 27): it never helps a relay further away, which still doubts -the publisher until its timer fires. This rule has to reach those relays. +`fresh-publisher.md`, added by #8622, exempts only a publisher's own new +direct session from doubt, so it never helps a relay further away. If this +rule holds, it covers that case too: fold that quest into this one or delete +it, whichever has landed first. Work: From 5722a22ef67ae1733d707bff24c636cf0f475cac Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 23:49:41 -0800 Subject: [PATCH 8/8] quest: name the declined exemption in item 27 Co-Authored-By: Claude Opus 5.5 --- quest/m2/cluster-routes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index be5b513851d..fb6cabff2c4 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -57,7 +57,7 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). - Timer-only doubt retained (maintainer, 2026-11-14), declining a fresh publisher's exemption: a relay would move at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route: + Declined 2026-11-14 (maintainer), timer-only doubt retained over it: a fresh publisher's exemption, under which a relay moves at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route: | Case | Rule | Recovery | Dark | Messages | Claims | | --- | --- | ---: | ---: | ---: | ---: |