From 725928161ad9ca7e01243791a0e079bb85c9b219 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 16:21:29 -0800 Subject: [PATCH 1/6] chore(quest): claim quest/m1/route-mesh-model.md Quest-Claim: bb4dd9407519d139 From a034344c02a4c1f0669c95189ca6f264b0496496 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 21:22:00 -0800 Subject: [PATCH 2/6] test(net): the route model counts each burst's messages and covers lines, rings, and clients Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/verify/route/Chord.cfg | 29 +++++ rs/moq-net/verify/route/Line.cfg | 31 +++++ rs/moq-net/verify/route/Ring.cfg | 26 +++++ rs/moq-net/verify/route/RouteHandoff.cfg | 1 + rs/moq-net/verify/route/RouteHandoff.tla | 116 +++++++++++++++---- rs/moq-net/verify/route/RouteHandoff.view.ts | 23 ++-- 6 files changed, 194 insertions(+), 32 deletions(-) create mode 100644 rs/moq-net/verify/route/Chord.cfg create mode 100644 rs/moq-net/verify/route/Line.cfg create mode 100644 rs/moq-net/verify/route/Ring.cfg diff --git a/rs/moq-net/verify/route/Chord.cfg b/rs/moq-net/verify/route/Chord.cfg new file mode 100644 index 00000000000..3456fac50bb --- /dev/null +++ b/rs/moq-net/verify/route/Chord.cfg @@ -0,0 +1,29 @@ +\* Too large to check every ordering, so run it with TLC's random simulation: +\* just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Chord.cfg -simulate num= -depth 600 +\* A client (0) publishes through relay 1 of a ring of five relays (1 to 5) +\* with a chord from 2 to 4, and two clients (6 on 3, 7 on 4) watch. Enough +\* flaps cut the ring into a line. +CONSTANTS + Nodes = {0, 1, 2, 3, 4, 5, 6, 7} + Origin = 0 + Leaves = {6, 7} + Links = {{0, 1}, {1, 2}, {2, 3}, {3, 4}, {4, 5}, {5, 1}, {2, 4}, {3, 6}, {4, 7}} + Claims = {"x", "y"} + MaxRev = 4 + MaxFlaps = 4 + MaxCollects = 2 + +SPECIFICATION Spec + +INVARIANTS + Truthful + AllCertified + NoRevival + NoFalseEnd + NoRegression + LoopFree + QuietIsSettled + Budget + +CONSTRAINT + Bounded diff --git a/rs/moq-net/verify/route/Line.cfg b/rs/moq-net/verify/route/Line.cfg new file mode 100644 index 00000000000..c8d9bc55c92 --- /dev/null +++ b/rs/moq-net/verify/route/Line.cfg @@ -0,0 +1,31 @@ +\* A line: the origin (0) publishes through relays 1, 2, and 3 to a client +\* (4) that only watches. Every route has one path, so a cut leaves the far +\* side with nothing, and the client must echo every ROUTE_END it hears. +\* Converges checks under fairness what Budget implies: routing goes quiet. +CONSTANTS + Nodes = {0, 1, 2, 3, 4} + Origin = 0 + Leaves = {4} + Links = {{0, 1}, {1, 2}, {2, 3}, {3, 4}} + Claims = {"x"} + MaxRev = 2 + MaxFlaps = 1 + MaxCollects = 1 + +SPECIFICATION LiveSpec + +INVARIANTS + Truthful + AllCertified + NoRevival + NoFalseEnd + NoRegression + LoopFree + QuietIsSettled + Budget + +PROPERTIES + Converges + +CONSTRAINT + Bounded diff --git a/rs/moq-net/verify/route/Ring.cfg b/rs/moq-net/verify/route/Ring.cfg new file mode 100644 index 00000000000..ba0b1e246ad --- /dev/null +++ b/rs/moq-net/verify/route/Ring.cfg @@ -0,0 +1,26 @@ +\* A ring: the origin (0) and relays 1, 2, and 3, so relay 2 has two equal +\* paths and a cut anywhere leaves a longer way around. +CONSTANTS + Nodes = {0, 1, 2, 3} + Origin = 0 + Leaves = {} + Links = {{0, 1}, {1, 2}, {2, 3}, {3, 0}} + Claims = {"x"} + MaxRev = 2 + MaxFlaps = 1 + MaxCollects = 0 + +SPECIFICATION Spec + +INVARIANTS + Truthful + AllCertified + NoRevival + NoFalseEnd + NoRegression + LoopFree + QuietIsSettled + Budget + +CONSTRAINT + Bounded diff --git a/rs/moq-net/verify/route/RouteHandoff.cfg b/rs/moq-net/verify/route/RouteHandoff.cfg index d6020df9941..1c3043855c3 100644 --- a/rs/moq-net/verify/route/RouteHandoff.cfg +++ b/rs/moq-net/verify/route/RouteHandoff.cfg @@ -21,6 +21,7 @@ INVARIANTS NoRegression LoopFree QuietIsSettled + Budget CONSTRAINT Bounded diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index ccca07819cc..401b93f0dea 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -70,10 +70,12 @@ VARIABLES falseEnds, \* reader ENDs while the origin still has that item (`false_ends`) regressions,\* reader UPDATEs to an older item flaps, - collects + collects, + burst, \* ROUTE, batch, request, and echo messages sent since the last quiet point + churn \* external changes since the last quiet point vars == <> + regressions, flaps, collects, burst, churn>> \* The origin's view and seqno. OV == [rev |-> rev, items |-> own, seq |-> seqno] @@ -269,6 +271,30 @@ Commit(rs, q0) == /\ falseEnds' = falseEnds + Sum(LAMBDA n : rs[n].ev.falseEnds) /\ regressions' = regressions + Sum(LAMBDA n : rs[n].ev.regressions) +Quiet == \A d \in Dirs : q[d] = <<>> /\ acks[d] = 0 + +\* The messages the updates `rs` send, counted before `Heeded` drops any. +Sent(rs) == + LET ns == DOMAIN rs + F[S \in SUBSET ns] == + IF S = {} THEN 0 + ELSE LET x == CHOOSE x \in S : TRUE + G[T \in SUBSET Nodes] == + IF T = {} THEN 0 ELSE LET m == CHOOSE m \in T : TRUE IN Len(rs[x].out[m]) + G[T \ {m}] + IN G[Nodes] + F[S \ {x}] + IN F[ns] + +\* A burst runs from one quiet point to the next. An external change (a claim +\* change, a session opening or closing, a collection) made while quiet starts +\* a fresh one; every message sent in it counts against the changes made in it. +External(sent) == + /\ burst' = (IF Quiet THEN 0 ELSE burst) + sent + /\ churn' = (IF Quiet THEN 0 ELSE churn) + 1 + +Internal(sent) == + /\ burst' = burst + sent + /\ UNCHANGED churn + ----------------------------------------------------------------------------- (* Initial state: every session open (`Sim::join`), the origin declared *) (* with no claims (`originate`), and nothing sent. *) @@ -300,6 +326,8 @@ Init == /\ regressions = 0 /\ flaps = 0 /\ collects = 0 + /\ burst = 0 + /\ churn = 0 ----------------------------------------------------------------------------- (* The origin changes a claim (`announce` / `unannounce` -> `change`). *) @@ -307,10 +335,12 @@ Init == ChangeClaim(c, item) == LET own1 == [own EXCEPT ![c] = item] ov1 == [rev |-> rev + 1, items |-> own1, seq |-> seqno] + rs == [n \in {Origin} |-> Update(Origin, st[Origin], ov1, up, ended[Origin])] IN /\ rev' = rev + 1 /\ own' = own1 /\ hist' = Append(hist, own1) - /\ Commit([n \in {Origin} |-> Update(Origin, st[Origin], ov1, up, ended[Origin])], q) + /\ Commit(rs, q) + /\ External(Sent(rs)) /\ UNCHANGED <> \* A new claim, or new attributes for one, under a fresh revision. @@ -377,26 +407,32 @@ Deliver(d) == ELSE CASE msg.type = "route" -> OnRoute(st[n], m, msg.reach) [] msg.type = "batch" -> OnBatch(st[n], m, msg.view) [] OTHER -> OnRequest(st[n], m, msg.seq) + rs == [x \in {n} |-> Update(n, s1, ov1, up, ended[n])] + echoes == IF isEnd THEN 1 ELSE 0 IN /\ q[d] # <<>> \* Every ROUTE_END is echoed once applied. /\ acks' = IF isEnd THEN [acks EXCEPT ![<>] = @ + 1] ELSE acks /\ seqno' = ov1.seq /\ IF skip THEN /\ q' = q0 + /\ Internal(echoes) /\ UNCHANGED <> - ELSE Commit([x \in {n} |-> Update(n, s1, ov1, up, ended[n])], q0) + ELSE /\ Commit(rs, q0) + /\ Internal(echoes + Sent(rs)) /\ UNCHANGED <> \* `echo`: an echoed ROUTE_END comes back (`Sim::deliver_ack`). DeliverAck(d) == LET m == d[1] n == d[2] + rs == [x \in {n} |-> Update(n, [st[n] EXCEPT !.unacked[m] = @ - 1], OV, up, ended[n])] IN /\ acks[d] > 0 /\ acks' = [acks EXCEPT ![d] = @ - 1] /\ IF st[n].unacked[m] = 0 - THEN UNCHANGED <> - ELSE Commit([x \in {n} |-> - Update(n, [st[n] EXCEPT !.unacked[m] = @ - 1], OV, up, ended[n])], q) + THEN /\ Internal(0) + /\ UNCHANGED <> + ELSE /\ Commit(rs, q) + /\ Internal(Sent(rs)) /\ UNCHANGED <> ----------------------------------------------------------------------------- @@ -417,43 +453,48 @@ Closed(s, y) == Connect(l) == LET up1 == up \cup {l} + rs == [x \in l |-> + LET y == CHOOSE y \in l : y # x + IN Update(x, Opened(st[x], y), OV, up1, ended[x])] IN /\ l \notin up /\ flaps < MaxFlaps /\ up' = up1 /\ flaps' = flaps + 1 - /\ Commit([x \in l |-> - LET y == CHOOSE y \in l : y # x - IN Update(x, Opened(st[x], y), OV, up1, ended[x])], q) + /\ Commit(rs, q) + /\ External(Sent(rs)) /\ UNCHANGED <> Disconnect(l) == LET up1 == up \ {l} q0 == [d \in Dirs |-> IF {d[1], d[2]} = l THEN <<>> ELSE q[d]] + rs == [x \in l |-> + LET y == CHOOSE y \in l : y # x + IN Update(x, Closed(st[x], y), OV, up1, ended[x])] IN /\ l \in up /\ flaps < MaxFlaps /\ up' = up1 /\ flaps' = flaps + 1 /\ acks' = [d \in Dirs |-> IF {d[1], d[2]} = l THEN 0 ELSE acks[d]] - /\ Commit([x \in l |-> - LET y == CHOOSE y \in l : y # x - IN Update(x, Closed(st[x], y), OV, up1, ended[x])], q0) + /\ Commit(rs, q0) + /\ External(Sent(rs)) /\ UNCHANGED <> \* `collect`, allowed once the node holds no route (`Sim::holds`). It \* forgets the views, so the reader's earlier ENDs no longer count. Collect(n) == + LET rs == [x \in {n} |-> + Update(n, [st[n] EXCEPT !.view = NoView, + !.heard = [m \in Nodes |-> NoHeard], + !.pending = NoPending], + OV, up, [c \in Claims |-> None])] + IN /\ collects < MaxCollects /\ \A m \in Nodes : ~st[n].rx[m].on /\ collects' = collects + 1 - /\ Commit([x \in {n} |-> - Update(n, [st[n] EXCEPT !.view = NoView, - !.heard = [m \in Nodes |-> NoHeard], - !.pending = NoPending], - OV, up, [c \in Claims |-> None])], q) + /\ Commit(rs, q) + /\ External(Sent(rs)) /\ UNCHANGED <> -Quiet == \A d \in Dirs : q[d] = <<>> /\ acks[d] = 0 - \* Nothing is in flight and no more churn is allowed: the run is over. Done == /\ Quiet @@ -504,17 +545,19 @@ Hop(n, k) == IF k = 0 THEN n ELSE Hop(NextHop(n), k - 1) \* Following selected next hops from any relay never cycles. LoopFree == \A n \in Relays : Hop(n, Cardinality(Nodes)) \notin Relays -\* Every link costs 1, so a loop-free path to the origin has at most this metric. -Longest == Cardinality(Nodes) - 1 +\* Every link costs 1 and only forwarding nodes are on a path, so a loop-free +\* path from the origin to `n` has at most this metric. +Forwarders == {n \in Nodes : Serves(n)} +Longest(n) == Cardinality(Forwarders) - (IF Serves(n) THEN 1 ELSE 0) \* No relay selects a route longer than any loop-free path: a longer one is a \* stale route counting up as it bounces between relays (Babel's feasibility \* condition exists to rule this out). -NoCountToInfinity == \A n \in Relays : Selected(n, st[n], OV).met <= Longest +NoCountToInfinity == \A n \in Relays : Selected(n, st[n], OV).met <= Longest(n) \* Explore one step past NoCountToInfinity and no further, so a count to \* infinity is reported rather than explored forever. -Bounded == \A n, m \in Nodes : st[n].rx[m].met <= Longest +Bounded == \A n, m \in Nodes : st[n].rx[m].met <= Longest(m) \* Nodes the origin reaches over open sessions, through nodes that forward. RECURSIVE Grow(_, _) @@ -540,4 +583,29 @@ Settled == QuietIsSettled == Quiet => Settled +\* What one external change may cost: about one message of each kind (a ROUTE, +\* a batch, a seqno request, an echo) per direction of each configured session. +PerChange == 4 * Cardinality(Dirs) + +\* The messages sent since the last quiet point stay within the budget of the +\* external changes made since then, so a change never sets off announce spam. +\* `burst` is part of the state, so a cycle of deliveries that never goes quiet +\* would grow it past any budget: Budget also rules out routing that never +\* converges once the changes stop. +Budget == burst <= PerChange * churn + +\* While anything is in flight, something is eventually delivered. This is +\* weaker than fairness per queue (whose tableau TLC builds in time exponential +\* in the queues), so Converges fails on any cycle of deliveries, starving a +\* queue or not. +Delivery == \E d \in Dirs : Deliver(d) \/ DeliverAck(d) +\* WF_vars(Delivery), spelled out: Delivery is enabled exactly when not Quiet, +\* and TLC evaluates ENABLED of this action far too slowly. +Fairness == []<>Quiet \/ []<><>_vars + +LiveSpec == Spec /\ Fairness + +\* After the last external change, routing goes quiet and stays settled. +Converges == <>[](Quiet /\ Settled) + ============================================================================= diff --git a/rs/moq-net/verify/route/RouteHandoff.view.ts b/rs/moq-net/verify/route/RouteHandoff.view.ts index e02b18361bc..e7c444541b0 100644 --- a/rs/moq-net/verify/route/RouteHandoff.view.ts +++ b/rs/moq-net/verify/route/RouteHandoff.view.ts @@ -14,6 +14,8 @@ type Vars = { acks: Record; up: number[][]; own: Record; + burst: number; + churn: number; }; const ORIGIN = "0"; @@ -45,17 +47,22 @@ const relay = (s: Node) => { return `${how}${shows ? ", shows it" : ""}`; }; -const sessions = (v: Vars) => { - const down = ["0,1", "0,2", "1,2"].filter((l) => !v.up.some((u) => [...u].sort().join(",") === l)); - return down.length === 0 ? "all open" : down.map((l) => `${l.replace(",", "–")} closed`).join(", "); -}; +const sessions = (v: Vars) => + v.up + .map((l) => [...l].sort((a, b) => a - b).join("–")) + .sort() + .join(", ") || "none"; -export const title = "One origin identity across a triangle of relays"; +export const title = "One origin identity across a mesh of relays"; export const state = (v: Vars) => ({ - Sessions: sessions(v), - "Relay 1": relay(v.st["1"]), - "Relay 2": relay(v.st["2"]), + "Open sessions": sessions(v), + ...Object.fromEntries( + Object.keys(v.st) + .filter((n) => n !== ORIGIN) + .map((n) => [`Relay ${n}`, relay(v.st[n])]), + ), + "Messages this burst": `${v.burst} for ${v.churn} change${v.churn === 1 ? "" : "s"}`, }); const len = (s: Record | undefined) => (s ? Object.keys(s).length : 0); From 5c2831896d8551bddc4a683a4cb7c462848f3b4b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 21:38:21 -0800 Subject: [PATCH 3/6] test(net): the route model checks an announce budget and convergence on lines, rings, and a chorded ring with clients Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 29 +++++++++++++++--- quest/m1/model-checks-nightly.md | 4 ++- quest/m1/route-inductive.md | 4 --- quest/m1/route-mesh-model.md | 37 ----------------------- quest/m1/route-mesh-sweep.md | 12 +++++--- quest/m1/route-trace-validation.md | 7 ++--- quest/m2/cluster-routing/routes/README.md | 1 - rs/moq-net/verify/route/RouteHandoff.tla | 28 +++++++---------- 8 files changed, 49 insertions(+), 73 deletions(-) delete mode 100644 quest/m1/route-mesh-model.md diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index d2a2a815011..be8f50c835b 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,9 @@ description: Model-check the lite-07 route layer (`rs/moq-net/src/route/`) with # Verify the route layer -`rs/moq-net/verify/route/RouteHandoff.tla` models one origin identity in `route::Router` as `route::tests::Sim` drives it: every router entry point is one atomic step ending in `update`, ROUTE, batches, and ROUTE_REQUEST share one FIFO per direction, and echoes travel on their own queue. TLC checks every ordering of those messages in a triangle of the origin and two relays, under the bounds in the config. +`rs/moq-net/verify/route/RouteHandoff.tla` models one origin identity in `route::Router` as `route::tests::Sim` drives it: every router entry point is one atomic step ending in `update`, ROUTE, batches, and ROUTE_REQUEST share one FIFO per direction, and echoes travel on their own queue. A config picks the mesh: which nodes there are, which sessions join them, and which are clients (`Leaves`, as `Sim::leaf`) that forward no route but echo every ROUTE_END. TLC checks every ordering of those messages under each config's bounds, or samples them at random for a mesh too large to exhaust. + +One identity is enough for meshes with several origins: the router keys every piece of state by identity and recomputes one identity from that identity's state alone, so each identity's messages, read on their own, are a run of this model with that origin. Sharing one FIFO per direction only removes orderings, and a session flap is a flap for each. The configs put the origin at the end of a line, in a ring, and on a client off a ring instead. ## Rules @@ -20,18 +22,36 @@ Each rule is an INVARIANT; the harm says why it matters. | `NoRegression` | A reader never sees an UPDATE to an older item. | A stale announcement replaces a newer one. | | `LoopFree` | Selected next hops never cycle. | Requests loop between relays. | | `QuietIsSettled` | Once nothing is in flight, each relay reaching the origin shows its claims, available, and routes only while it has any; each cut-off relay shows nothing and keeps no route, distance, or request. | Routing never converges, or a dead origin stays shown. | +| `Budget` | From one quiet point to the next, the ROUTE, batch, ROUTE_REQUEST, and echo messages sent stay within 4 per direction of each configured session for each external change made (a claim change, a session opening or closing, a collection). | One change sets off announce spam. | | `NoCountToInfinity` | No relay selects a route longer than a loop-free path. | Stale routes bounce between relays with no bound. | +`Converges` is a PROPERTY: under fairness that something in flight is eventually delivered, routing goes quiet after the last external change and stays settled. `Budget` implies it, since the message count is part of the state: a cycle of deliveries that never goes quiet would grow the count past any budget. So every config checks `Budget`, and only `Line.cfg`, small enough for TLC's liveness checker, checks `Converges` directly. + +The budget is about one message of each kind per directed session. The most one change costs in the triangle is 20 of the 24 it allows: withdrawing the last claim sends a batch, a ROUTE_END, and an echo across each of the six directions, plus two seqno requests that chase routes whose ends are still in flight. `route-mesh-sweep` holds the implementation to the same budget, counted from the Sim's `route_msgs + claim_msgs + requests`. + The constraint `Bounded` stops exploring one hop past `NoCountToInfinity`, so a count to infinity shows up there instead of running TLC forever. ## Run ```sh just rs tla moq-net/verify/route/RouteHandoff.tla +just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Line.cfg +just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg +just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Chord.cfg -simulate num=1000 -depth 600 just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/CountToInfinity.cfg ``` -The first checks every rule but `NoCountToInfinity`: about 3.7M states, two minutes on 32 cores, and it passes at the sync commit. The second fails at the sync commit on the open finding below. +Each but the last checks every rule but `NoCountToInfinity` and passes at the sync commit. Times are on 24 shared cores. + +| Config | Mesh | Bounds | Run | +| --- | --- | --- | --- | +| `RouteHandoff.cfg` | The origin and two relays in a triangle. | 2 revisions, 2 flaps, 1 collection | 24M states, 22 min | +| `Line.cfg` | The origin, three relays, and a watching client in a line; also checks `Converges`. | 2 revisions, 1 flap, 1 collection | 2M states, 7 min | +| `Ring.cfg` | The origin and three relays in a ring. | 2 revisions, 1 flap | 24M states, 34 min | +| `Chord.cfg` | A publishing client on a ring of five relays with a chord, and two watching clients. Simulated: `num` counts behaviours per worker. | 2 claims, 4 revisions, 4 flaps, 2 collections | 24K behaviours (28M states) in 15 min | +| `CountToInfinity.cfg` | The triangle. | As `RouteHandoff.cfg` | Fails in 9 steps | + +Larger exhaustive meshes do not fit: `Line.cfg` at two flaps passed 12M states in 55 minutes without finishing. TLC's `SYMMETRY` does not help, because selection breaks ties by neighbour key, so no two relays are interchangeable; only watching clients on the same relay are, and one per relay covers them. `Chord.cfg` samples the larger meshes instead; its flaps can cut the ring into a line. Read a counterexample with `/statechart`, passing it the spec, a config, and `RouteHandoff.view.ts`, which labels each relay's route and each step as the message that arrives. It gives the shortest path to each broken rule in those words. @@ -45,7 +65,7 @@ A counterexample is a path through the model, not yet a bug. Replay it on the ro ## Coverage -Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun both configs, and move the sync commit forward only once the model describes the code at the new commit. +Sync commit: `529fa6ac9138c6f7b09ce8380fdb2407334bd6ca`. Run `git log 529fa6ac91.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun both configs, and move the sync commit forward only once the model describes the code at the new commit. | Code (`rs/moq-net/src/route/`) | Model | | --- | --- | @@ -57,7 +77,8 @@ Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07 | `mod.rs` `connect`, `peer_interest`, `disconnect`, `collect`, `change` | `Connect`, `Opened`, `Disconnect`, `Closed`, `Collect`, `ChangeClaim` | | `view.rs` `View::merge`, `View::restrict` | `Merge`, `Certify` | | `tests/mod.rs` `Sim` queues and `revivals`, `false_ends` | `q`, `acks`, `revivals`, `falseEnds` | +| `tests/mod.rs` `Sim::leaf`, `Stats` | `Leaves`, `burst` | ## Not covered -The model fixes every interest at the root, so it leaves out wildcards, `present`/`represent`/`retire`, interest changes, and the hidden `current` regions; prune's region bookkeeping reduces to "no certifier has a floor". It also leaves out scoped grants and qualified keys, GOAWAY, link priority, costs other than 1, upstream links, identity rotation and restart, and `node.rs`'s per-stream keys. A batch carries the sender's whole presentation, equal to applying its diff on a FIFO stream. The origin drops the ROUTEs and batches it hears for its own identity, which change nothing there. A pass covers one claim, two revisions, two session opens or closes, and one collection in that triangle. `sweep_handoff` and `sweep_scoped_handoff` still own the rest. +The model fixes every interest at the root, so it leaves out wildcards, `present`/`represent`/`retire`, interest changes, and the hidden `current` regions; prune's region bookkeeping reduces to "no certifier has a floor". It also leaves out scoped grants and qualified keys, GOAWAY, link priority, costs other than 1, upstream links, identity rotation and restart, `node.rs`'s per-stream keys, and how the origin above the router ranks two identities claiming the same broadcast. A batch carries the sender's whole presentation, equal to applying its diff on a FIFO stream. The origin drops the ROUTEs and batches it hears for its own identity, which change nothing there; `Budget` still counts them. An exhaustive pass covers one claim and only the bounds in the table above. `sweep_handoff` and `sweep_scoped_handoff` still own the rest. diff --git a/quest/m1/model-checks-nightly.md b/quest/m1/model-checks-nightly.md index b6e282bb7c2..b3937e2e1b8 100644 --- a/quest/m1/model-checks-nightly.md +++ b/quest/m1/model-checks-nightly.md @@ -16,7 +16,9 @@ added the models (`rs/moq-net/verify/*`, `js/*/verify/`) and their `verify-*` skills. - Nightly runs every config through the `tla` recipes. TLC runs take 20 s - to 2 min on 32 cores, so they stay out of the merge gate. An open-finding + to 35 min on 24 cores, so they stay out of the merge gate. A config its + skill says is simulated (the route model's `Chord.cfg`) runs with the + `-simulate` arguments the skill gives. An open-finding config is one its skill lists under "Open finding"; the job expects TLC's invariant-violation exit for it and fails on a pass. - The JS flow models (`js/*/verify/`) check every rule in one config, so diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index 91dcece786b..0ccce75cce4 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -44,10 +44,6 @@ stay constants the config states. Say both bounds in the `verify-route` skill. Public API: none. Wire: none. -## Required - -- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - the spec parametrised over mesh shapes that this invariant is stated on - ## Related - [The TLA+ models run nightly and reviews resync them](/quest/m1/model-checks-nightly.md) - the Nightly job this check joins diff --git a/quest/m1/route-mesh-model.md b/quest/m1/route-mesh-model.md deleted file mode 100644 index cc0e93fc9b5..00000000000 --- a/quest/m1/route-mesh-model.md +++ /dev/null @@ -1,37 +0,0 @@ -# [L] The route model covers a mesh with clients and an announce budget - -## Goal - -The route model checks lite-07 routing on meshes larger than a triangle, with -more than one origin and with clients, and adds the two rules production has -broken most: a change costs a bounded number of announce messages, and -routing goes quiet after the last change. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan), after the triangle model in -[moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) found count to -infinity. Routing, announce spam, and resume failover have caused many -production issues; this and [the lite-07 mesh sweep](/quest/m1/route-mesh-sweep.md) -are routing's confidence gates for finalizing lite-07. - -- Grow `RouteHandoff.tla` (or a sibling spec sharing its operators) to 4 and - 5 relays in a line, a ring, and a ring with a chord; two origin identities; - and client leaves that advertise nothing but echo every ROUTE_END. -- An invariant bounds the ROUTE, batch, and request messages sent between - external changes (an origin's claim change, a session open or close) by a - budget derived from the links, so announce spam is a counterexample. -- A liveness property, under weak fairness of delivery: after the last - external change, the queues empty and every relay shows what - `QuietIsSettled` requires. -- Keep explicit-state runs tractable with `SYMMETRY` over interchangeable - relays; run larger meshes with TLC's random simulation in Nightly - ([nightly models](/quest/m1/model-checks-nightly.md)). -- Count to infinity stays in its own open-finding config until - [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) lands, so - the other rules still run on larger meshes. - -Replay every counterexample on `route::tests::Sim` before calling it a bug, as -the `verify-route` skill says, and pin confirmed ones as `residual_*` tests. - -Public API: none. Wire: none. diff --git a/quest/m1/route-mesh-sweep.md b/quest/m1/route-mesh-sweep.md index f78238e55ad..5730deb4ede 100644 --- a/quest/m1/route-mesh-sweep.md +++ b/quest/m1/route-mesh-sweep.md @@ -9,9 +9,9 @@ budget. Nightly sweeps fresh seeds, and a failing seed replays bit-exact. ## Plan -Decided 2026-11-10 (maintainer, /quest-plan). The model -([mesh route model](/quest/m1/route-mesh-model.md)) checks every ordering on -small meshes; this checks the real stack, sessions included, on larger random +Decided 2026-11-10 (maintainer, /quest-plan). The route model (the +`verify-route` skill) checks every ordering on meshes of up to four relays and +samples a chorded ring of five; this checks the real stack, sessions included, on larger random ones. `rs/moq-net/tests/integration/mesh_withdraw.rs` already sweeps 200 seeded meshes, but on lite-06; [route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) ports specific @@ -24,7 +24,11 @@ fixtures, not a random sweep. and relay sees: truthful, no revival, no false end, no regression, loop-free, and settled. Count ROUTE and announce messages per external change against the same budget the model states, and report the counts as a benchmark swept - over mesh size and churn. + over mesh size and churn. The model's `Budget`: from one quiet point to the + next, the ROUTE, batch, ROUTE_REQUEST, and echo messages sent (the Sim's + `route_msgs + claim_msgs + requests`) stay within 4 per direction of each + configured session per external change made in that span (a claim change, a + session open or close, a collection). - Seeds follow the repo's `MOQ_*_SEED` convention; the merge gate runs a few fixed seeds only if they stay fast, and Nightly sweeps fresh ones. diff --git a/quest/m1/route-trace-validation.md b/quest/m1/route-trace-validation.md index cf16769e271..0092dd2fd71 100644 --- a/quest/m1/route-trace-validation.md +++ b/quest/m1/route-trace-validation.md @@ -14,8 +14,8 @@ the route model drifting from the code ([moq-dev/moq#8447](https://github.com/mo Decided 2026-11-10 (maintainer, a later /quest-plan the same day): -- ✅ m1 (rejected: staying in m4). The [mesh model](/quest/m1/route-mesh-model.md), - the [inductive invariant](/quest/m1/route-inductive.md), and the +- ✅ m1 (rejected: staying in m4). The mesh configs beside it, the + [inductive invariant](/quest/m1/route-inductive.md), and the [cache-aware routing proof](/quest/m2/cache-routing-proof.md) all build on `RouteHandoff.tla`, so drift there costs the most. - ✅ The other three models get trace validation too, each its own m4 quest @@ -36,7 +36,4 @@ or the model grows to cover them; pick whichever costs less and say which. Run it on the sweeps' existing seeds in CI only if it stays fast; otherwise in Nightly with the sweeps' fresh seeds. -Related: [the mesh route model](/quest/m1/route-mesh-model.md), which grows -the spec these traces are checked against. - Public API: none. Wire: none. diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 8d4fe86ef09..1fa2125df70 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -541,7 +541,6 @@ benchmarks. ## Required -- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence - [Seeded lite-07 meshes hold the route rules and an announce budget](/quest/m1/route-mesh-sweep.md) - the route rules and message budget checked on real sessions over random meshes ## Related diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index 401b93f0dea..eeda65a2867 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -256,13 +256,16 @@ Update(n, s0, ov, ups, eb) == Heeded(d, m) == d[2] # Origin \/ m.type = "request" \/ (m.type = "route" /\ m.reach.kind = "none") +\* The sum of f(x) over the finite set S. +SumOver(S, f(_)) == + LET F[T \in SUBSET S] == IF T = {} THEN 0 ELSE LET x == CHOOSE x \in T : TRUE IN f(x) + F[T \ {x}] + IN F[S] + \* Commit the updates `rs` (a function from nodes to `Update` results), appending \* each node's messages to `q0`. Nodes in one step send on disjoint queues. Commit(rs, q0) == LET ns == DOMAIN rs - Sum(f(_)) == LET F[S \in SUBSET ns] == - IF S = {} THEN 0 ELSE LET x == CHOOSE x \in S : TRUE IN f(x) + F[S \ {x}] - IN F[ns] + Sum(f(_)) == SumOver(ns, f) IN /\ st' = [n \in Nodes |-> IF n \in ns THEN rs[n].s ELSE st[n]] /\ q' = [d \in Dirs |-> IF d[1] \in ns THEN q0[d] \o SelectSeq(rs[d[1]].out[d[2]], LAMBDA m : Heeded(d, m)) ELSE q0[d]] @@ -274,15 +277,7 @@ Commit(rs, q0) == Quiet == \A d \in Dirs : q[d] = <<>> /\ acks[d] = 0 \* The messages the updates `rs` send, counted before `Heeded` drops any. -Sent(rs) == - LET ns == DOMAIN rs - F[S \in SUBSET ns] == - IF S = {} THEN 0 - ELSE LET x == CHOOSE x \in S : TRUE - G[T \in SUBSET Nodes] == - IF T = {} THEN 0 ELSE LET m == CHOOSE m \in T : TRUE IN Len(rs[x].out[m]) + G[T \ {m}] - IN G[Nodes] + F[S \ {x}] - IN F[ns] +Sent(rs) == SumOver(DOMAIN rs, LAMBDA n : SumOver(Nodes, LAMBDA m : Len(rs[n].out[m]))) \* A burst runs from one quiet point to the next. An external change (a claim \* change, a session opening or closing, a collection) made while quiet starts @@ -595,12 +590,11 @@ PerChange == 4 * Cardinality(Dirs) Budget == burst <= PerChange * churn \* While anything is in flight, something is eventually delivered. This is -\* weaker than fairness per queue (whose tableau TLC builds in time exponential -\* in the queues), so Converges fails on any cycle of deliveries, starving a -\* queue or not. +\* WF_vars(Delivery) written out, since Delivery is enabled exactly when not +\* Quiet and TLC's ENABLED on these actions does not finish. It is weaker than +\* fairness per queue, so Converges fails on any cycle of deliveries, starving +\* a queue or not. Delivery == \E d \in Dirs : Deliver(d) \/ DeliverAck(d) -\* WF_vars(Delivery), spelled out: Delivery is enabled exactly when not Quiet, -\* and TLC evaluates ENABLED of this action far too slowly. Fairness == []<>Quiet \/ []<><>_vars LiveSpec == Spec /\ Fairness From 1ea1c91daacb15a1edf92049fd553cda878ea604 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 23:23:43 -0800 Subject: [PATCH 4/6] build: TLC runs on JDK 25 with 8 workers Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 2 +- js/justfile | 2 +- rs/justfile | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index be8f50c835b..aab5196deec 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -41,7 +41,7 @@ just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/rout just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/CountToInfinity.cfg ``` -Each but the last checks every rule but `NoCountToInfinity` and passes at the sync commit. Times are on 24 shared cores. +Each but the last checks every rule but `NoCountToInfinity` and passes at the sync commit. Times were measured with every core of a loaded 24-core host, before the recipe dropped to 8 workers. | Config | Mesh | Bounds | Run | | --- | --- | --- | --- | diff --git a/js/justfile b/js/justfile index ea47805d793..7a8a3f9b27e 100644 --- a/js/justfile +++ b/js/justfile @@ -56,7 +56,7 @@ fix-changed list: # Model-check a TLA+ spec with TLC, e.g. `just js tla watch/verify/Watch.tla`. # Pass `-config js/.cfg` for a config not named after the spec. tla spec *args: - cd .. && nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} js/{{ spec }} + cd .. && nix shell --inputs-from . nixpkgs#jdk25 --command java -XX:+UseParallelGC -cp "$(nix build --inputs-from . nixpkgs#tlaplus --no-link --print-out-paths)/share/java/tla2tools.jar" tlc2.TLC -workers 8 -cleanup -metadir "$(mktemp -d)" {{ args }} js/{{ spec }} # Run all unit tests. test: diff --git a/rs/justfile b/rs/justfile index 391e57fb7cf..b30fcabdcbf 100644 --- a/rs/justfile +++ b/rs/justfile @@ -21,7 +21,7 @@ default: # Model-check a TLA+ spec with TLC, e.g. `just rs tla moq-net/verify/route/RouteHandoff.tla`. # Pass `-config rs/.cfg` for a config not named after the spec. tla spec *args: - nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }} + nix shell --inputs-from . nixpkgs#jdk25 --command java -XX:+UseParallelGC -cp "$(nix build --inputs-from . nixpkgs#tlaplus --no-link --print-out-paths)/share/java/tla2tools.jar" tlc2.TLC -workers 8 -cleanup -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }} # Default features for the selected packages; `features` covers the feature # matrix nightly and on release PRs. No `cargo check` pass: clippy is a From 04dcf47bad649f90f75246d4006eb03eca2f3f16 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 16:33:09 -0800 Subject: [PATCH 5/6] docs(verify-route): runtimes for every config on 4 workers Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 8 ++++---- quest/m1/model-checks-nightly.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index bd2ff2f4ba1..15a17746cc6 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -41,15 +41,15 @@ just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/rout just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Chord.cfg -simulate num=1000 -depth 600 ``` -Times are TLC 1.8.0 on Java 21 with `-workers 4` at `nice -n 19`, on a 24-core host. +Times are TLC 1.8.0 on Java 21 with `-workers 4` at `nice -n 19`, on a 24-core host under load from other work. | Config | Mesh | Bounds | Distinct states | Run | | --- | --- | --- | --- | --- | | `RouteHandoff.cfg` | The origin and two relays in a triangle. | 2 claim changes, 2 session changes, 1 collection | 24,192,399 | 39 min | -| `Behind.cfg` | The origin behind relay 1, which sits in a triangle with relays 2 and 3. | 1 claim change, 1 session change | BEHIND_STATES | BEHIND_TIME | -| `BehindDear.cfg` | The same, with relay 3 nearer the origin than relay 2. | 1 claim change, 1 session change | DEAR_STATES | DEAR_TIME | +| `Behind.cfg` | The origin behind relay 1, which sits in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 9,384,796 | 25 min | +| `BehindDear.cfg` | The same, with relay 3 nearer the origin than relay 2. | 1 claim change, 1 session change | 17,863,476 | 61 min | | `Line.cfg` | The origin, three relays, and a watching client in a line; also checks `Converges`. | 2 claim changes, 1 session change, 1 collection | 1,543,073 | 13 min | -| `Ring.cfg` | The origin and three relays in a ring. | 2 claim changes, 1 session change | RING_STATES | RING_TIME | +| `Ring.cfg` | The origin and three relays in a ring. | 2 claim changes, 1 session change | 18,437,833 | 61 min | | `Chord.cfg` | A publishing client on a ring of five relays with a chord, and two watching clients. Simulated: `num` counts behaviours per worker. | 2 claims, 4 claim changes, 4 session changes, 2 collections | 4,000 behaviours, 4.6M states | 9 min | All pass at the sync commit. Run them last, one at a time. First run what answers in seconds: diff --git a/quest/m1/model-checks-nightly.md b/quest/m1/model-checks-nightly.md index b3937e2e1b8..31cd0e46211 100644 --- a/quest/m1/model-checks-nightly.md +++ b/quest/m1/model-checks-nightly.md @@ -16,7 +16,7 @@ added the models (`rs/moq-net/verify/*`, `js/*/verify/`) and their `verify-*` skills. - Nightly runs every config through the `tla` recipes. TLC runs take 20 s - to 35 min on 24 cores, so they stay out of the merge gate. A config its + to an hour on 4 workers, so they stay out of the merge gate. A config its skill says is simulated (the route model's `Chord.cfg`) runs with the `-simulate` arguments the skill gives. An open-finding config is one its skill lists under "Open finding"; the job expects TLC's From 3f3a771639c7adaf3e71b1e98751b1b91789ee02 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 16:53:11 -0800 Subject: [PATCH 6/6] docs(verify-route): restore the coverage rows the merge dropped Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 3b54e331092..836272c7805 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -71,7 +71,14 @@ Sync commit: `caa3dd8926f16fbed15760ab22298085df73b655`. Run `git log caa3dd8926 | Code | Model | | --- | --- | -Sync commit: `c916855d163ff5641ad18cfb8e14b76dababd99a`, plus the change that added a route's hops ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Run `git log c916855d16.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it. +| `rs/moq-net/src/route/mod.rs` `feasible`, `Entry::crossed`, `eligible`, `metric`, `selected`, `infeasible`, `recoverable`, `live`, `certifiers` | `Feasible`, `Crossed`, `Eligible`, `Through`, `Selected`, `InfeasibleSet`, `Recoverable`, `Live`, `Certifiers` | +| `rs/moq-net/src/route/mod.rs` `certified`, `prune`, `adopted` | `Certified`, `Prune`, `Adopted` | +| `rs/moq-net/src/route/mod.rs` `update`, `recompute`, `record`, `advertise`, `requests` | `Update` and its `prune`, `Record`, `Plan`, `Requests` | +| `rs/moq-net/src/route/mod.rs` `show`, `shown_now` | `Events`, `ShownNow` | +| `rs/moq-net/src/route/mod.rs` `route`, `batch`, `request`, `echo` | `OnRoute`, `OnBatch`, `OnRequest`, `Deliver`, `DeliverAck` | +| `rs/moq-net/src/route/mod.rs` `connect`, `peer_interest`, `disconnect`, `collect`, `change` | `Connect`, `Opened`, `Disconnect`, `Closed`, `Collect`, `ChangeClaim` | +| `rs/moq-net/src/route/view.rs` `View::merge`, `View::restrict` | `Merge`, `Certify` | +| `rs/moq-net/src/route/tests/mod.rs` `Sim` queues and `revivals`, `false_ends` | `q`, `acks`, `revivals`, `falseEnds` | | `rs/moq-net/src/route/tests/mod.rs` `Sim::leaf`, `Stats` | `Leaves`, `burst` | ## Not covered