From 1800db2c3cacb4f4f41bbf6ea3da8afe202b028e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 04:50:56 -0800 Subject: [PATCH 1/2] chore(quest): claim quest/m2/cluster-routes/holdcycle-stale-return.md Quest-Claim: 2de13792fb33a170 From 06a461a233e39b5443afb42ed3c39761542575dc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 06:59:40 -0800 Subject: [PATCH 2/2] fix(verify): a patient route timer waits for stale offers too A patient timer is one long enough for the news of a closed session to cross the mesh, so it also outlasts every offer through that session sent before the news reached its sender. The model let it fire once no DEAD was in flight, while an older offer naming the dead hop could still be queued, so a relay forgot the hop and took that offer back. HoldCycle broke NoStaleReturn that way under random seeds, and Twice and Loop within seconds. Lift and Expire now wait for every message naming the hop, so all three check NoStaleReturn and HoldCycle samples a new seed each run. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 6 ++-- .../cluster-routes/holdcycle-stale-return.md | 30 ------------------- rs/moq-net/verify/route/HoldCycle.cfg | 2 +- rs/moq-net/verify/route/Loop.cfg | 1 + rs/moq-net/verify/route/RouteHandoff.tla | 25 +++++++++++----- rs/moq-net/verify/route/Twice.cfg | 1 + 6 files changed, 24 insertions(+), 41 deletions(-) delete mode 100644 quest/m2/cluster-routes/holdcycle-stale-return.md diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 20f307cce39..0e5723ea3a2 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no message naming a closed session it covers is in flight: no DEAD, and no stale offer sent before the news. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). ## Rules @@ -33,7 +33,7 @@ Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regres No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. -What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). With two failures a stale route can return whatever the timer (`Twice.cfg` drops `NoStaleReturn`), and `StaleIsChased` is the bound. +What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). Patient means the timer outlasts every message sent before it fired, not only the DEAD: one that fires while an older offer through the closed session is still queued is early, and lets that offer return. Until `Patient` waited for those offers too, `HoldCycle.cfg` broke `NoStaleReturn` under random seeds, and `Twice.cfg` and `Loop.cfg` within seconds; all three check it now. A relay that has not yet heard the news may still use a stale route, and `StaleIsChased` is the bound. ## Run @@ -86,7 +86,7 @@ change; faster checkers are separate work. | `EditNarrow.cfg` | The publisher with sessions to relays 1 and 2, the one to relay 2 dear; relay 2 watches everything and narrows to `"a"` or drops it once. | 1 claim change, 1 session change, 1 edit | 768,483 | three minutes | | `EditWiden.cfg` | The same, where relay 2 watches `"a"` and widens to everything or drops it once. | 1 claim change, 1 session change, 1 edit | 3,655,573 | a quarter of an hour | | `Edit.cfg` | The same with two edits, such as widening then narrowing back, or dropping everything then asking again. Before the rekey kept the route the sender would reflect, a widening then a narrowing crossing the close of relay 2's own session kept a route relay 1 learned through it and took a claim offline, breaking `Stable`. Simulated; an exhaustive pass covered every behaviour to depth 18 (10.6M states) before it was stopped. | 1 claim change, 1 session change, 2 edits | 40,000 behaviours, 4.8M states | four minutes | -| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes | +| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, a new seed each run | a few minutes | Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. diff --git a/quest/m2/cluster-routes/holdcycle-stale-return.md b/quest/m2/cluster-routes/holdcycle-stale-return.md deleted file mode 100644 index d9826b823ba..00000000000 --- a/quest/m2/cluster-routes/holdcycle-stale-return.md +++ /dev/null @@ -1,30 +0,0 @@ -# [M] HoldCycle keeps a dead path from returning off its pinned seed - -## Goal - -`HoldCycle.cfg` holds `NoStaleReturn` under any simulation seed, as its -bounds promise: one session change and a patient timer, where the -[verify-route skill](/.claude/skills/verify-route/SKILL.md) says the rule holds. -Today Nightly simulates it only with `-seed 1`, which passes. Random seeds -break `NoStaleReturn` within a few hundred thousand states (a 35-step -behaviour). The base of -[the interest set](https://github.com/moq-dev/moq/pull/8728) breaks it the -same way, so no change to interest edits caused it. - -## Plan - -Decided 2026-11-14 (maintainer, /quest-complete of [the interest set](https://github.com/moq-dev/moq/pull/8728)): ✅ keep in m2 (rejected: move to m3, drop it). - -Recommended: - -1. Rerun `HoldCycle.cfg` with `-simulate num=3000 -depth 100` and no seed - until it fails, then read the behaviour as the skill's Reading a failure - says. Decide whether the model misdescribes the router or the router lets a - relay use a route through a hop it already learned died. -2. Replay it on `route::tests::Sim` under `Rule::PATH`. If the code follows - the path, fix the code and pin the replay as a fixture. If it does not, - fix the model. -3. Drop `-seed 1` from `HoldCycle.cfg`'s `Simulate:` line once random seeds - pass, so Nightly samples new orderings each night. Do not drop - `NoStaleReturn` from the config unless the behaviour shows a second session - change that makes this `Twice.cfg`'s accepted case. diff --git a/rs/moq-net/verify/route/HoldCycle.cfg b/rs/moq-net/verify/route/HoldCycle.cfg index b00b16c71fe..81f9deaeb8a 100644 --- a/rs/moq-net/verify/route/HoldCycle.cfg +++ b/rs/moq-net/verify/route/HoldCycle.cfg @@ -1,4 +1,4 @@ -\* Simulate: -simulate num=2000 -depth 200 -seed 1 +\* Simulate: -simulate num=2000 -depth 200 \* Holds that would close a cycle (`holds_never_close_a_cycle`): the publisher \* (0) on hub 1, open relays 2 and 3 on the hub, and relays 4 and 5 asking for \* "a" alone, each joined to both open relays and nearer the other's choice. diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index ed0a1ad1a93..cb660433b81 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -36,3 +36,4 @@ INVARIANTS RequestsLoopFree QuietIsSettled Budget + NoStaleReturn diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index c5f41ddebb2..443bfe54d07 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -56,7 +56,7 @@ CONSTANTS MaxRev, \* the publisher's claim changes MaxFlaps, \* session opens and closes MaxEdits, \* changes of what a relay watches - Patient \* TRUE: a doubt lifts only once no news about its node is in flight + Patient \* TRUE: a timer fires only once no news or stale offer about its node is in flight Relays == Nodes \ {Origin} @@ -752,17 +752,28 @@ Deliver(d) == /\ Internal(Sent(rs)) /\ UNCHANGED <> +\* The hops a message names: an offer's path, or a DEAD's hop and moved paths. +MsgHops(m) == + CASE m.t = "offer" -> {m.path[i] : i \in DOMAIN m.path} + [] m.t = "dead" -> {m.hop} \cup UNION {{mv[2][i] : i \in DOMAIN mv[2]} : mv \in m.moved} + [] OTHER -> {} +\* The session hop `h` names has closed. +Severed(h) == h[1] # h[2] /\ ~({h[1], h[2]} \in up /\ gen[{h[1], h[2]}] = h[3]) +\* Some message in flight names a hop `P` picks. +InFlight(P(_)) == \E d \in Dirs : \E i \in DOMAIN q[d] : \E h \in MsgHops(q[d][i]) : P(h) + \* The timer node n started on x fires: the doubt on x lifts, and the dead hops -\* whose session ended at x may be forgotten. With `Patient`, only once no DEAD -\* about such a hop, and no DRAIN of x, is in flight: a timer long enough for -\* the news to cross the mesh. +\* whose session ended at x may be forgotten. With `Patient`, only once no +\* message naming such a hop, and no DRAIN of x, is in flight: a timer long +\* enough for the news to cross the mesh outlasts the DEAD and every stale +\* offer sent before the news reached its sender. Lift(n, x) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, !.expiry = {h \in @ : h[2] # x}], OV, up, gen, ended[n], NoReport, Claims)] IN /\ x \in st[n].wait - /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : - ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) /\ ~(q[d][i].t = "drain" /\ q[d][i].node = x) + /\ Patient => /\ ~InFlight(LAMBDA h : h[2] = x /\ Severed(h)) + /\ \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "drain" /\ q[d][i].node = x) /\ Commit(rs, q) /\ Internal(Sent(rs)) /\ UNCHANGED <> @@ -773,7 +784,7 @@ Expire(n, hop) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.expiry = @ \ {hop}], OV, up, gen, ended[n], NoReport, Claims)] IN /\ hop \in st[n].expiry - /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop = hop) + /\ Patient => ~InFlight(LAMBDA h : h = hop) /\ Commit(rs, q) /\ Internal(Sent(rs)) /\ UNCHANGED <> diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index fef277fe656..6ed75be9bdd 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -37,3 +37,4 @@ INVARIANTS RequestsLoopFree QuietIsSettled Budget + NoStaleReturn