Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .claude/skills/verify-route/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH

# Verify the route rule

`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt).
`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. A route learned on a link its learner marks upstream, or from a down-only route, is down-only, and no relay sends one over a link it marks upstream (`Upstream`, the links whose higher-numbered end marks them), so no relay carries traffic between two upstreams. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` start out watching alone), and keeps dead hops until their timer fires. A relay asks each neighbour for the set of prefixes it watches, none inside another; a relay in `Editors` may change that set up to `MaxEdits` times. Each change is an edit, sent as an ask on the FIFO and answered by a marker on the neighbour's FIFO, where both sides rekey the routes between them alike (`Rekey`): a key the new set covers stays, the route at a removed `"a"` moves to an added `"root"` without its floor, and the route at a removed `"root"` carries its floor, presented at `"a"`, to an added `"a"`. A heard route is believed for a prefix only while the relay asked for it at the last marker and in every edit still in flight. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no message naming a closed session it covers is in flight: no DEAD, and no stale offer sent before the news. Every lite-07 session drives the path router through `route::node::Node` since [the switch](https://github.com/moq-dev/moq/pull/8731), and `route::tests::Sim` drives it under `Rule::PATH` (timer-only doubt).

## Rules

Expand Down Expand Up @@ -33,7 +33,7 @@ Every config checks `RequestsLoopFree`; `Loop.cfg` pins the three-failure regres

No config checks a liveness property: `Spec` has no fairness, so TLC does not show that a hold ends. That one does is an argument over what TLC does check. `Budget` bounds the messages a burst sends, each delivery takes one message off a queue, and a doubt starts only at a close or a delivery, so a burst is a finite number of steps. A delivery is enabled while a queue holds a message and a timer may fire once no news about its node is in flight, so a burst left alone reaches the state with nothing in flight and no doubt. `QuietIsSettled` says no relay holds there.

What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). With two failures a stale route can return whatever the timer (`Twice.cfg` drops `NoStaleReturn`), and `StaleIsChased` is the bound.
What the timer costs, as the configs show it: fired late, it costs only the wait, and every rule holds (`RouteHandoff.cfg`). Fired early, a stale route can return, as with no doubt at all, and nothing else breaks (`Early.cfg`, which passes without `NoStaleReturn` and fails it within seconds with it). Patient means the timer outlasts every message sent before it fired, not only the DEAD: one that fires while an older offer through the closed session is still queued is early, and lets that offer return. Until `Patient` waited for those offers too, `HoldCycle.cfg` broke `NoStaleReturn` under random seeds, and `Twice.cfg` and `Loop.cfg` within seconds; all three check it now. A relay that has not yet heard the news may still use a stale route, and `StaleIsChased` is the bound.

## Run

Expand Down Expand Up @@ -86,7 +86,7 @@ change; faster checkers are separate work.
| `EditNarrow.cfg` | The publisher with sessions to relays 1 and 2, the one to relay 2 dear; relay 2 watches everything and narrows to `"a"` or drops it once. | 1 claim change, 1 session change, 1 edit | 768,483 | three minutes |
| `EditWiden.cfg` | The same, where relay 2 watches `"a"` and widens to everything or drops it once. | 1 claim change, 1 session change, 1 edit | 3,655,573 | a quarter of an hour |
| `Edit.cfg` | The same with two edits, such as widening then narrowing back, or dropping everything then asking again. Before the rekey kept the route the sender would reflect, a widening then a narrowing crossing the close of relay 2's own session kept a route relay 1 learned through it and took a claim offline, breaking `Stable`. Simulated; an exhaustive pass covered every behaviour to depth 18 (10.6M states) before it was stopped. | 1 claim change, 1 session change, 2 edits | 40,000 behaviours, 4.8M states | four minutes |
| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, seed 1 | a few minutes |
| `HoldCycle.cfg` | The publisher on a hub with two open relays, and two relays asking for `"a"` alone, each joined to both open relays and nearer the other's choice. Before a hold followed only the path it was built on, the publisher's session closing left four holds waiting on each other for good, breaking `QuietIsSettled` within seconds. Simulated. | 1 claim change, 1 session change | 2,000 behaviours per worker, depth 200, a new seed each run | a few minutes |

Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`.

Expand Down
30 changes: 0 additions & 30 deletions quest/m2/cluster-routes/holdcycle-stale-return.md

This file was deleted.

2 changes: 1 addition & 1 deletion rs/moq-net/verify/route/HoldCycle.cfg
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
\* Simulate: -simulate num=2000 -depth 200 -seed 1
\* Simulate: -simulate num=2000 -depth 200
\* Holds that would close a cycle (`holds_never_close_a_cycle`): the publisher
\* (0) on hub 1, open relays 2 and 3 on the hub, and relays 4 and 5 asking for
\* "a" alone, each joined to both open relays and nearer the other's choice.
Expand Down
1 change: 1 addition & 0 deletions rs/moq-net/verify/route/Loop.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ INVARIANTS
RequestsLoopFree
QuietIsSettled
Budget
NoStaleReturn
25 changes: 18 additions & 7 deletions rs/moq-net/verify/route/RouteHandoff.tla
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ CONSTANTS
MaxRev, \* the publisher's claim changes
MaxFlaps, \* session opens and closes
MaxEdits, \* changes of what a relay watches
Patient \* TRUE: a doubt lifts only once no news about its node is in flight
Patient \* TRUE: a timer fires only once no news or stale offer about its node is in flight

Relays == Nodes \ {Origin}

Expand Down Expand Up @@ -752,17 +752,28 @@ Deliver(d) ==
/\ Internal(Sent(rs))
/\ UNCHANGED <<up, gen, rev, own, hist, flaps, edits>>

\* The hops a message names: an offer's path, or a DEAD's hop and moved paths.
MsgHops(m) ==
CASE m.t = "offer" -> {m.path[i] : i \in DOMAIN m.path}
[] m.t = "dead" -> {m.hop} \cup UNION {{mv[2][i] : i \in DOMAIN mv[2]} : mv \in m.moved}
[] OTHER -> {}
\* The session hop `h` names has closed.
Severed(h) == h[1] # h[2] /\ ~({h[1], h[2]} \in up /\ gen[{h[1], h[2]}] = h[3])
\* Some message in flight names a hop `P` picks.
InFlight(P(_)) == \E d \in Dirs : \E i \in DOMAIN q[d] : \E h \in MsgHops(q[d][i]) : P(h)

\* The timer node n started on x fires: the doubt on x lifts, and the dead hops
\* whose session ended at x may be forgotten. With `Patient`, only once no DEAD
\* about such a hop, and no DRAIN of x, is in flight: a timer long enough for
\* the news to cross the mesh.
\* whose session ended at x may be forgotten. With `Patient`, only once no
\* message naming such a hop, and no DRAIN of x, is in flight: a timer long
\* enough for the news to cross the mesh outlasts the DEAD and every stale
\* offer sent before the news reached its sender.
Lift(n, x) ==
LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x},
!.expiry = {h \in @ : h[2] # x}],
OV, up, gen, ended[n], NoReport, Claims)]
IN /\ x \in st[n].wait
/\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] :
~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) /\ ~(q[d][i].t = "drain" /\ q[d][i].node = x)
/\ Patient => /\ ~InFlight(LAMBDA h : h[2] = x /\ Severed(h))
/\ \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "drain" /\ q[d][i].node = x)
/\ Commit(rs, q)
/\ Internal(Sent(rs))
/\ UNCHANGED <<up, gen, rev, own, hist, flaps, edits>>
Expand All @@ -773,7 +784,7 @@ Expire(n, hop) ==
LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.expiry = @ \ {hop}],
OV, up, gen, ended[n], NoReport, Claims)]
IN /\ hop \in st[n].expiry
/\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop = hop)
/\ Patient => ~InFlight(LAMBDA h : h = hop)
/\ Commit(rs, q)
/\ Internal(Sent(rs))
/\ UNCHANGED <<up, gen, rev, own, hist, flaps, edits>>
Expand Down
1 change: 1 addition & 0 deletions rs/moq-net/verify/route/Twice.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,4 @@ INVARIANTS
RequestsLoopFree
QuietIsSettled
Budget
NoStaleReturn