Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 24 additions & 10 deletions .claude/skills/verify-route/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ description: Model-check the lite-07 route layer (`rs/moq-net/src/route/`) with

# Verify the route layer

`rs/moq-net/verify/route/RouteHandoff.tla` models one origin identity in `route::Router` as `route::tests::Sim` drives it: every router entry point is one atomic step ending in `update`, ROUTE, batches, and ROUTE_REQUEST share one FIFO per direction, and echoes travel on their own queue, free to overtake. TLC checks every ordering of those messages on the mesh and bounds each config names.
`rs/moq-net/verify/route/RouteHandoff.tla` models one origin identity in `route::Router` as `route::tests::Sim` drives it: every router entry point is one atomic step ending in `update`, ROUTE, batches, and ROUTE_REQUEST share one FIFO per direction, and echoes travel on their own queue, free to overtake. A config picks the mesh: which nodes there are, which sessions join them, which cost 2, and which nodes are clients (`Leaves`, as `Sim::leaf`) that forward no route but echo every ROUTE_END. TLC checks every ordering of those messages under each config's bounds, or samples them at random for a mesh too large to exhaust.

One identity is enough for meshes with several origins: the router keys every piece of state by identity and recomputes one identity from that identity's state alone, so each identity's messages, read on their own, are a run of this model with that origin. Sharing one FIFO per direction only removes orderings, and a session flap is a flap for each. The configs put the origin at the end of a line, in a ring, behind a triangle, and on a client off a ring instead.

## Rules

Expand All @@ -22,39 +24,50 @@ Each rule is an INVARIANT; the harm says why it matters.
| `QuietIsSettled` | Once nothing is in flight, each relay reaching the origin shows its claims, available, and routes only while it has any; each cut-off relay shows nothing and keeps no route, distance, or request. | Routing never converges, or a dead origin stays shown. |
| `SimplePaths` | Every route held or in flight names a loop-free path from the origin to its sender, at that path's cost. | A node took back a route that crossed it, the first step of a count to infinity. |
| `NoCountToInfinity` | No relay selects a route longer than a loop-free path. | Stale routes bounce between relays with no bound. |
| `Budget` | From one quiet point to the next, the ROUTE, batch, ROUTE_REQUEST, and echo messages sent stay within 4 each way over each configured session, per route identity and per claim, for each external change made (a claim change, a session opening or closing, a collection). | One change sets off announce spam. |

`SimplePaths` needs no search to believe: a node sends a route only after adding itself to hops that did not name it. It holds whatever a node forgot, which is why `NoCountToInfinity` no longer rests on the echo.

`Budget` is the one `mesh_sweep.rs` holds the implementation to, with the model's one identity. `Converges` is a PROPERTY: under fairness that something in flight is eventually delivered, routing goes quiet after the last external change and stays settled. `Budget` implies it, since the message count is part of the state: a cycle of deliveries that never goes quiet would grow the count past any budget. So every config checks `Budget`, and only `Line.cfg`, small enough for TLC's liveness checker, checks `Converges` directly.

## Run

```sh
just rs tla moq-net/verify/route/RouteHandoff.tla
just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Behind.cfg
just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/BehindDear.cfg
just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Line.cfg
just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg
just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Chord.cfg -simulate num=1000 -depth 600
```

| Config | Mesh | Bounds | Distinct states |
| --- | --- | --- | --- |
| `RouteHandoff.cfg` | The origin and two relays in a triangle. | 2 claim changes, 2 session changes, 1 collection | 3,545,311 |
| `Behind.cfg` | The origin behind relay 1, which sits in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 3,033,111 |
| `BehindDear.cfg` | The same, with relay 3 nearer the origin than relay 2. | 1 claim change, 1 session change | 5,377,597 |
Times are TLC 1.8.0 on Java 21 with `-workers 4` at `nice -n 19`, on a 24-core host under load from other work.

| Config | Mesh | Bounds | Distinct states | Run |
| --- | --- | --- | --- | --- |
| `RouteHandoff.cfg` | The origin and two relays in a triangle. | 2 claim changes, 2 session changes, 1 collection | 24,192,399 | 39 min |
| `Behind.cfg` | The origin behind relay 1, which sits in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 9,384,796 | 25 min |
| `BehindDear.cfg` | The same, with relay 3 nearer the origin than relay 2. | 1 claim change, 1 session change | 17,863,476 | 61 min |
| `Line.cfg` | The origin, three relays, and a watching client in a line; also checks `Converges`. | 2 claim changes, 1 session change, 1 collection | 1,543,073 | 13 min |
| `Ring.cfg` | The origin and three relays in a ring. | 2 claim changes, 1 session change | 18,437,833 | 61 min |
| `Chord.cfg` | A publishing client on a ring of five relays with a chord, and two watching clients. Simulated: `num` counts behaviours per worker. | 2 claims, 4 claim changes, 4 session changes, 2 collections | 4,000 behaviours, 4.6M states | 9 min |

All three pass at the sync commit, and `just test tla rs/moq-net/verify/route` runs them all, as Nightly does. Each takes minutes on 4 workers, so run them last. First run what answers in seconds:
All pass at the sync commit, and `just test tla rs/moq-net/verify/route` runs them all, as Nightly does, simulating `Chord.cfg` as its `Simulate:` line says. Run them last, one at a time. First run what answers in seconds:

- `just rs test -p moq-net --lib -E 'test(/route::/)'`. `sweep_no_route_outgrows_a_loop_free_path` delivers every message and echo in a random order on random meshes with unequal costs, flaps, restarts, and collections, and checks the metric bound, loop freedom, and the quiet state. Raise its case count for a wider search: 40,000 cases run in 15 seconds in release.
- TLC's simulation mode, which samples runs the exhaustive bounds do not reach. The config's `MaxRev`, `MaxFlaps`, and `MaxCollects` still bound a simulated run, so raise them in a copy of the config first, such as three session changes, two claim changes, and a collection, then add `-simulate num=3000 -depth 150`.

Both behind-a-triangle configs fail `SimplePaths` within seconds if `Crossed` is made false, which is the count to infinity the hops fix: a relay drops its distance once its neighbours echoed, and takes back a copy of its own route that a third relay still held.

Read a counterexample with `/statechart`, passing it the spec, a config, and `RouteHandoff.view.ts`, which labels each relay's route and each step as the message that arrives. It gives the shortest path to each broken rule in those words. The view names the triangle's sessions and assumes equal costs.
Read a counterexample with `/statechart`, passing it the spec, a config, and `RouteHandoff.view.ts`, which labels each relay's route and each step as the message that arrives. It gives the shortest path to each broken rule in those words. The view labels any mesh and assumes equal costs.

## Reading a failure

A counterexample is a path through the model, not yet a bug. Replay it on the router with `Sim` (`hold`, `deliver`, `deliver_ack`, `disconnect`, `collect`), one call per step. If the router does not follow the path, the model is wrong: fix the model to match the code. If it does, report the path and the rule in the code's words. Do not weaken a rule, add a constraint, or shrink the bounds until the check passes.

## Coverage

Sync commit: `c916855d163ff5641ad18cfb8e14b76dababd99a`, plus the change that added a route's hops ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Run `git log c916855d16.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it.
Sync commit: `caa3dd8926f16fbed15760ab22298085df73b655`. Run `git log caa3dd8926.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it.

| Code | Model |
| --- | --- |
Expand All @@ -66,7 +79,8 @@ Sync commit: `c916855d163ff5641ad18cfb8e14b76dababd99a`, plus the change that ad
| `rs/moq-net/src/route/mod.rs` `connect`, `peer_interest`, `disconnect`, `collect`, `change` | `Connect`, `Opened`, `Disconnect`, `Closed`, `Collect`, `ChangeClaim` |
| `rs/moq-net/src/route/view.rs` `View::merge`, `View::restrict` | `Merge`, `Certify` |
| `rs/moq-net/src/route/tests/mod.rs` `Sim` queues and `revivals`, `false_ends` | `q`, `acks`, `revivals`, `falseEnds` |
| `rs/moq-net/src/route/tests/mod.rs` `Sim::leaf`, `Stats` | `Leaves`, `burst` |

## Not covered

The model fixes every interest at the root, so it leaves out wildcards, `present`/`represent`/`retire`, interest changes, and the hidden `current` regions; prune's region bookkeeping reduces to "no certifier has a floor". It also leaves out scoped grants and qualified keys (so `advertised`, which only they read), GOAWAY, link priority, costs other than 1 and 2, the hop limit, the informational RTT, upstream links, identity rotation and restart, and `node.rs`'s per-stream keys. A batch carries the sender's whole presentation, equal to applying its diff on a FIFO stream. The origin drops the ROUTEs and batches it hears for its own identity, which change nothing there. A pass covers one claim at the bounds in the table above, and no mesh of more than three relays. `sweep_no_route_outgrows_a_loop_free_path`, `sweep_handoff`, and `sweep_scoped_handoff` still own the rest: larger meshes, more churn, restarts, and scoped grants.
The model fixes every interest at the root, so it leaves out wildcards, `present`/`represent`/`retire`, interest changes, and the hidden `current` regions; prune's region bookkeeping reduces to "no certifier has a floor". It also leaves out scoped grants and qualified keys (so `advertised`, which only they read), GOAWAY, link priority, costs other than 1 and 2, the hop limit, the informational RTT, upstream links, identity rotation and restart, `node.rs`'s per-stream keys and the seqno it advances when a route link closes, and how the origin above the router ranks two identities claiming the same broadcast. A batch carries the sender's whole presentation, equal to applying its diff on a FIFO stream. The origin drops the ROUTEs and batches it hears for its own identity, which change nothing there; `Budget` still counts them. An exhaustive pass covers one claim at the bounds in the table above, and no mesh of more than four relays. `sweep_no_route_outgrows_a_loop_free_path`, `sweep_handoff`, `sweep_scoped_handoff`, and `mesh_sweep.rs` still own the rest: larger meshes, more churn, restarts, and scoped grants.
16 changes: 12 additions & 4 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -411,17 +411,25 @@ jobs:
tla:
name: TLA+ (${{ matrix.name }})
runs-on: ubuntu-24.04-arm
timeout-minutes: 120
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
# The two largest models get a runner each; the last leg takes every
# other config, so a new model is checked without editing this list.
# The largest models get a runner each, and the route model's three
# largest configs (about an hour each on 4 workers) one per config; the
# `route` and `rest` legs take every other config, so a new one is
# checked without editing this list.
include:
- name: resume
paths: rs/moq-net/verify/resume
- name: route triangle
paths: rs/moq-net/verify/route/RouteHandoff.cfg
- name: route behind
paths: rs/moq-net/verify/route/BehindDear.cfg
- name: route ring
paths: rs/moq-net/verify/route/Ring.cfg
- name: route
paths: rs/moq-net/verify/route
paths: "rs/moq-net/verify/route :!rs/moq-net/verify/route/RouteHandoff.cfg :!rs/moq-net/verify/route/BehindDear.cfg :!rs/moq-net/verify/route/Ring.cfg"
- name: rest
paths: ":!rs/moq-net/verify/resume :!rs/moq-net/verify/route"
steps:
Expand Down
1 change: 0 additions & 1 deletion quest/m1/route-inductive.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ Public API: none. Wire: none.

## Required

- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - the spec parametrised over mesh shapes that this invariant is stated on
- [The route model has a route per scope](/quest/m2/cluster-routing/routes/scope-model.md) - the rule for whose claims a relay believes, which the claim rules rest on; an invariant found before it is found for a rule about to go

## Related
Expand Down
40 changes: 0 additions & 40 deletions quest/m1/route-mesh-model.md

This file was deleted.

7 changes: 2 additions & 5 deletions quest/m1/route-trace-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ the route model drifting from the code ([moq-dev/moq#8447](https://github.com/mo

Decided 2026-11-10 (maintainer, a later /quest-plan the same day):

- ✅ m1 (rejected: staying in m4). The [mesh model](/quest/m1/route-mesh-model.md),
the [inductive invariant](/quest/m1/route-inductive.md), and the
- ✅ m1 (rejected: staying in m4). The mesh configs beside it, the
[inductive invariant](/quest/m1/route-inductive.md), and the
[cache-aware routing proof](/quest/m2/cache-routing-proof.md) all build on
`RouteHandoff.tla`, so drift there costs the most.
- ✅ The other three models get trace validation too, each its own m4 quest
Expand All @@ -36,9 +36,6 @@ or the model grows to cover them; pick whichever costs less and say which.
Run it on the sweeps' existing seeds in CI only if it stays fast; otherwise in
Nightly with the sweeps' fresh seeds.

Related: [the mesh route model](/quest/m1/route-mesh-model.md), which grows
the spec these traces are checked against.

Public API: none. Wire: none.

## Required
Expand Down
4 changes: 0 additions & 4 deletions quest/m2/cluster-routing/routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -565,10 +565,6 @@ benchmarks.
subscriptions through the radio mesh and the split refuses, and after a
partition each island reaches the other through its uplink.

## Required

- [The route model covers a mesh with clients and an announce budget](/quest/m1/route-mesh-model.md) - every ordering on 4-5 relay meshes with two origins and clients, with an announce budget and convergence

## Related

- [Route wakes](/quest/m4/route-wakes.md) - index work follows the settled source-selection model
1 change: 1 addition & 0 deletions rs/moq-net/verify/route/Behind.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,4 @@ INVARIANTS
SimplePaths
NoCountToInfinity
QuietIsSettled
Budget
1 change: 1 addition & 0 deletions rs/moq-net/verify/route/BehindDear.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,4 @@ INVARIANTS
SimplePaths
NoCountToInfinity
QuietIsSettled
Budget
29 changes: 29 additions & 0 deletions rs/moq-net/verify/route/Chord.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
\* Too large to check every ordering, so TLC samples it at random.
\* Simulate: -simulate num=1000 -depth 600
\* A client (0) publishes through relay 1 of a ring of five relays (1 to 5)
\* with a chord from 2 to 4, and two clients (6 on 3, 7 on 4) watch. Enough
\* flaps cut the ring into a line.
CONSTANTS
Nodes = {0, 1, 2, 3, 4, 5, 6, 7}
Origin = 0
Leaves = {6, 7}
Links = {{0, 1}, {1, 2}, {2, 3}, {3, 4}, {4, 5}, {5, 1}, {2, 4}, {3, 6}, {4, 7}}
Dear = {}
Claims = {"x", "y"}
MaxRev = 4
MaxFlaps = 4
MaxCollects = 2

SPECIFICATION Spec

INVARIANTS
Truthful
AllCertified
NoRevival
NoFalseEnd
NoRegression
LoopFree
SimplePaths
NoCountToInfinity
QuietIsSettled
Budget
31 changes: 31 additions & 0 deletions rs/moq-net/verify/route/Line.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
\* A line: the origin (0) publishes through relays 1, 2, and 3 to a client
\* (4) that only watches. Every route has one path, so a cut leaves the far
\* side with nothing, and the client must echo every ROUTE_END it hears.
\* Converges checks under fairness what Budget implies: routing goes quiet.
CONSTANTS
Nodes = {0, 1, 2, 3, 4}
Origin = 0
Leaves = {4}
Links = {{0, 1}, {1, 2}, {2, 3}, {3, 4}}
Dear = {}
Claims = {"x"}
MaxRev = 2
MaxFlaps = 1
MaxCollects = 1

SPECIFICATION LiveSpec

INVARIANTS
Truthful
AllCertified
NoRevival
NoFalseEnd
NoRegression
LoopFree
SimplePaths
NoCountToInfinity
QuietIsSettled
Budget

PROPERTIES
Converges
26 changes: 26 additions & 0 deletions rs/moq-net/verify/route/Ring.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
\* A ring: the origin (0) and relays 1, 2, and 3, so relay 2 has two equal
\* paths and a cut anywhere leaves a longer way around.
CONSTANTS
Nodes = {0, 1, 2, 3}
Origin = 0
Leaves = {}
Links = {{0, 1}, {1, 2}, {2, 3}, {3, 0}}
Dear = {}
Claims = {"x"}
MaxRev = 2
MaxFlaps = 1
MaxCollects = 0

SPECIFICATION Spec

INVARIANTS
Truthful
AllCertified
NoRevival
NoFalseEnd
NoRegression
LoopFree
SimplePaths
NoCountToInfinity
QuietIsSettled
Budget
1 change: 1 addition & 0 deletions rs/moq-net/verify/route/RouteHandoff.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,4 @@ INVARIANTS
SimplePaths
NoCountToInfinity
QuietIsSettled
Budget
Loading