diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 20f307cce39..0e5723ea3a2 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no message naming a closed session it covers is in flight: no DEAD, and no stale offer sent before the news. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt). ## Rules @@ -33,7 +33,7 @@ Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regres No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there. -What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). With two failures a stale route can return whatever the timer (`Twice.cfg` drops `NoStaleReturn`), and `StaleIsChased` is the bound. +What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). Patient means the timer outlasts every message sent before it fired, not only the DEAD: one that fires while an older offer through the closed session is still queued is early, and lets that offer return. Until `Patient` waited for those offers too, `HoldCycle.cfg` broke `NoStaleReturn` under random seeds, and `Twice.cfg` and `Loop.cfg` within seconds; all three check it now. A relay that has not yet heard the news may still use a stale route, and `StaleIsChased` is the bound. ## Run @@ -86,7 +86,7 @@ change; faster checkers are separate work. | `EditNarrow.cfg` | The publisher with sessions to relays 1 and 2, the one to relay 2 dear; relay 2 watches everything and narrows to `"a"` or drops it once. | 1 claim change, 1 session change, 1 edit | 768,483 | three minutes | | `EditWiden.cfg` | The same, where relay 2 watches `"a"` and widens to everything or drops it once. | 1 claim change, 1 session change, 1 edit | 3,655,573 | a quarter of an hour | | `Edit.cfg` | The same with two edits, such as widening then narrowing back, or dropping everything then asking again. Before the rekey kept the route the sender would reflect, a widening then a narrowing crossing the close of relay 2's own session kept a route relay 1 learned through it and took a claim offline, breaking `Stable`. Simulated; an exhaustive pass covered every behaviour to depth 18 (10.6M states) before it was stopped. | 1 claim change, 1 session change, 2 edits | 40,000 behaviours, 4.8M states | four minutes | -| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes | +| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, a new seed each run | a few minutes | Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. diff --git a/quest/m2/cluster-routes/holdcycle-stale-return.md b/quest/m2/cluster-routes/holdcycle-stale-return.md deleted file mode 100644 index d9826b823ba..00000000000 --- a/quest/m2/cluster-routes/holdcycle-stale-return.md +++ /dev/null @@ -1,30 +0,0 @@ -# [M] HoldCycle keeps a dead path from returning off its pinned seed - -## Goal - -`HoldCycle.cfg` holds `NoStaleReturn` under any simulation seed, as its -bounds promise: one session change and a patient timer, where the -[verify-route skill](/.claude/skills/verify-route/SKILL.md) says the rule holds. -Today Nightly simulates it only with `-seed 1`, which passes. Random seeds -break `NoStaleReturn` within a few hundred thousand states (a 35-step -behaviour). The base of -[the interest set](https://github.com/moq-dev/moq/pull/8728) breaks it the -same way, so no change to interest edits caused it. - -## Plan - -Decided 2026-11-14 (maintainer, /quest-complete of [the interest set](https://github.com/moq-dev/moq/pull/8728)): ✅ keep in m2 (rejected: move to m3, drop it). - -Recommended: - -1. Rerun `HoldCycle.cfg` with `-simulate num=3000 -depth 100` and no seed - until it fails, then read the behaviour as the skill's Reading a failure - says. Decide whether the model misdescribes the router or the router lets a - relay use a route through a hop it already learned died. -2. Replay it on `route::tests::Sim` under `Rule::PATH`. If the code follows - the path, fix the code and pin the replay as a fixture. If it does not, - fix the model. -3. Drop `-seed 1` from `HoldCycle.cfg`'s `Simulate:` line once random seeds - pass, so Nightly samples new orderings each night. Do not drop - `NoStaleReturn` from the config unless the behaviour shows a second session - change that makes this `Twice.cfg`'s accepted case. diff --git a/rs/moq-net/verify/route/HoldCycle.cfg b/rs/moq-net/verify/route/HoldCycle.cfg index b00b16c71fe..81f9deaeb8a 100644 --- a/rs/moq-net/verify/route/HoldCycle.cfg +++ b/rs/moq-net/verify/route/HoldCycle.cfg @@ -1,4 +1,4 @@ -\* Simulate: -simulate num=2000 -depth 200 -seed 1 +\* Simulate: -simulate num=2000 -depth 200 \* Holds that would close a cycle (`holds_never_close_a_cycle`): the publisher \* (0) on hub 1, open relays 2 and 3 on the hub, and relays 4 and 5 asking for \* "a" alone, each joined to both open relays and nearer the other's choice. diff --git a/rs/moq-net/verify/route/Loop.cfg b/rs/moq-net/verify/route/Loop.cfg index ed0a1ad1a93..cb660433b81 100644 --- a/rs/moq-net/verify/route/Loop.cfg +++ b/rs/moq-net/verify/route/Loop.cfg @@ -36,3 +36,4 @@ INVARIANTS RequestsLoopFree QuietIsSettled Budget + NoStaleReturn diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index c5f41ddebb2..443bfe54d07 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -56,7 +56,7 @@ CONSTANTS MaxRev, \* the publisher's claim changes MaxFlaps, \* session opens and closes MaxEdits, \* changes of what a relay watches - Patient \* TRUE: a doubt lifts only once no news about its node is in flight + Patient \* TRUE: a timer fires only once no news or stale offer about its node is in flight Relays == Nodes \ {Origin} @@ -752,17 +752,28 @@ Deliver(d) == /\ Internal(Sent(rs)) /\ UNCHANGED <> +\* The hops a message names: an offer's path, or a DEAD's hop and moved paths. +MsgHops(m) == + CASE m.t = "offer" -> {m.path[i] : i \in DOMAIN m.path} + [] m.t = "dead" -> {m.hop} \cup UNION {{mv[2][i] : i \in DOMAIN mv[2]} : mv \in m.moved} + [] OTHER -> {} +\* The session hop `h` names has closed. +Severed(h) == h[1] # h[2] /\ ~({h[1], h[2]} \in up /\ gen[{h[1], h[2]}] = h[3]) +\* Some message in flight names a hop `P` picks. +InFlight(P(_)) == \E d \in Dirs : \E i \in DOMAIN q[d] : \E h \in MsgHops(q[d][i]) : P(h) + \* The timer node n started on x fires: the doubt on x lifts, and the dead hops -\* whose session ended at x may be forgotten. With `Patient`, only once no DEAD -\* about such a hop, and no DRAIN of x, is in flight: a timer long enough for -\* the news to cross the mesh. +\* whose session ended at x may be forgotten. With `Patient`, only once no +\* message naming such a hop, and no DRAIN of x, is in flight: a timer long +\* enough for the news to cross the mesh outlasts the DEAD and every stale +\* offer sent before the news reached its sender. Lift(n, x) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, !.expiry = {h \in @ : h[2] # x}], OV, up, gen, ended[n], NoReport, Claims)] IN /\ x \in st[n].wait - /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : - ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) /\ ~(q[d][i].t = "drain" /\ q[d][i].node = x) + /\ Patient => /\ ~InFlight(LAMBDA h : h[2] = x /\ Severed(h)) + /\ \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "drain" /\ q[d][i].node = x) /\ Commit(rs, q) /\ Internal(Sent(rs)) /\ UNCHANGED <> @@ -773,7 +784,7 @@ Expire(n, hop) == LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.expiry = @ \ {hop}], OV, up, gen, ended[n], NoReport, Claims)] IN /\ hop \in st[n].expiry - /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop = hop) + /\ Patient => ~InFlight(LAMBDA h : h = hop) /\ Commit(rs, q) /\ Internal(Sent(rs)) /\ UNCHANGED <> diff --git a/rs/moq-net/verify/route/Twice.cfg b/rs/moq-net/verify/route/Twice.cfg index fef277fe656..6ed75be9bdd 100644 --- a/rs/moq-net/verify/route/Twice.cfg +++ b/rs/moq-net/verify/route/Twice.cfg @@ -37,3 +37,4 @@ INVARIANTS RequestsLoopFree QuietIsSettled Budget + NoStaleReturn