diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md
index c26aa3e6551..d7a35204be9 100644
--- a/.claude/skills/verify-route/SKILL.md
+++ b/.claude/skills/verify-route/SKILL.md
@@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH
# Verify the route rule
-`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire.
+`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. A route goes to a stream only while a claim meets it: one that meets none withdraws the claims sent and ends. `Timeout` fires the timers due together on one node in one update, as `Router::timeout` does: doubts lift, and each dead hop also expires on its own timer, so repeated failures of one node cannot retain all its old hops. A timer may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire.
## Rules
@@ -83,13 +83,19 @@ change; faster checkers are separate work.
Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`.
+## Trace validation
+
+`quest gate just test route-trace` checks that the router's own runs are behaviours of the model, as Trace validation in `rs/moq-net/verify/README.md` describes: `route::tests::Sim` records 150 runs each of `sweep_handoff` and `sweep_scoped_handoff` under `Rule::PATH` (`rs/moq-net/src/route/tests/trace.rs`), and `trace/RouteTrace.tla` follows each with every message and every touched relay's route and shown claims compared, checking the model's rules too except `Stable` (`quest/m2/cluster-routes/stable-repath.md`). Nightly picks a fresh window of scenarios (`MOQ_ROUTE_TRACE_FROM`); the printed settings replay it. Each sweep stays inside the model: one publisher whose every claim change changes a claim, relays asking for the root, no restart, collection only as a step the model does not see. A leaf that publishes nothing, the client, offers no route, so the traces drop it, its grant, and its interest edits; `the_rules_agree_on_every_script` keeps the full script, with a second origin, restarts, and collection. Costs come from the mesh (`Cost <- TCost`), and the timer is impatient (`Patient = FALSE`), since the sweeps hold messages past it.
+
+A failure leaves its trace in the `.tmp/route-trace.*` directory the run names: a deadlock is a step the model cannot take, and `Agrees` breaks where its messages or state differ, in a state whose `step` names what the router did and whose `route`, `shown`, and `q` show what the model did. Fix the side that is wrong, as Reading a failure says; a broken rule on an accepted trace is a finding in the rule.
+
## Reading a failure
-Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Lift(n, x)` fires one timer on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until that doubt lifts, and where another doubt lifted with it, the replay took a different path, which is no defect in the model.
+Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Timeout(n, X, H)` fires timers on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until those timers fire, and where others fired with them, the replay took a different path, which is no defect in the model.
## Coverage
-Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
+Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the merge of the scoped path router ([moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)), which trace validation checked the model against ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)). Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since; `test route-trace` fails when a change there takes a step the model does not. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Timeout`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
| Code | Model |
| --- | --- |
@@ -97,9 +103,9 @@ Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped
| `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` |
| `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` |
| `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` |
-| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` |
+| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::meets`, `Router::advertise`, `Router::end_sent`, `Router::report` | `Want`, `Meets`, `Withdraw`, `Advertise`, `Report`, `Collect` (an ended stream forgets what it reflected) |
| `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` |
-| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout` | `Collect`, `Lift`, `Expire`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. |
+| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout`, `Router::due` | `Collect`, `Timeout`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. |
| `rs/moq-net/src/route/path.rs` `Router::know`, `Router::shown_now`, `Router::show` | `Know`, `ShownNow` |
| `rs/moq-net/src/route/reader.rs` `Reader::apply` | `Events` |
| `rs/moq-net/src/route/tests/mod.rs` `Sim::deliver` | `Deliver` |
@@ -111,4 +117,4 @@ Concurrent disjoint regions of one permit need the
[interest set](/quest/m2/cluster-routes/interest-set.md), including per-prefix
SYNCED; neither this model nor the current scoped sweep covers them.
-One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn.
+One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. Every claim change changes a claim: the router's revision bump with no change is outside the model. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn.
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
index e2ac69a1828..8a1fe21c5fc 100644
--- a/.github/workflows/nightly.yml
+++ b/.github/workflows/nightly.yml
@@ -359,12 +359,13 @@ jobs:
# viewer: diff-independent, since a break can arrive through the relay,
# js/net, or the wire between them.
# `rs quic-sim` sweeps moq-quic's simulated workloads from a fresh seed; PRs run a fixed 64.
+ # It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases.
# `rs rung-disable` proves FFI simulcast disable and recovery on a shaped BBR3 uplink.
# `test resume-model` sweeps resume against its model from a fresh seed; PRs run a fixed 64.
# `test route-key-model` proves the route key proof with Tamarin, and that each variant breaks what it names.
# `test route-sweep` checks random lite-07 meshes against the route rules from a fresh seed; PRs run a fixed 8.
# It also runs the router's handoff sweeps over 20,000 fixed scenarios; PRs run 150.
- # It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases.
+ # `test route-trace` checks with TLC that 150 runs of those sweeps, from a fresh window, are behaviours of the route model.
# `test drill impaired` runs the relay drills over a freshly seeded shaper; PRs run loopback.
# `rs cli-export` runs moq-cli's export process tests, which take tens of seconds of real time.
# `rs hls-conformance` probes every segment moq-hls serves with ffprobe, against its playlists and manifest.
@@ -379,7 +380,7 @@ jobs:
# `rs v4l2` streams moq-video's V4L2 capture from the kernel's vivid
# virtual camera, a module load no pull request job should make.
# `js worklet-matrix` loads `?worklet` output through the bundlers pinned in test/worklet.
- recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "test route-key-model", "test route-sweep", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
+ recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "test route-key-model", "test route-sweep", "test route-trace", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"]
include:
# Miri interprets kio's and moq-net's in-memory unsafe; too slow for the merge gate.
- recipe: rs miri
diff --git a/quest/m2/cluster-routes/stable-repath.md b/quest/m2/cluster-routes/stable-repath.md
new file mode 100644
index 00000000000..c928e968a8d
--- /dev/null
+++ b/quest/m2/cluster-routes/stable-repath.md
@@ -0,0 +1,40 @@
+# [M] A relay keeps its claims when its next hop re-paths through a dead hop
+
+## Goal
+
+A relay whose next hop sends a new path through a session the relay already
+knows has closed keeps showing its claims, unavailable, until that next hop
+answers, instead of dropping them offline on that one message. The route
+model's `Stable` holds on every run of the router's sweeps, and
+`RouteTrace.cfg` checks it again.
+
+## Plan
+
+Found by trace validation ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)):
+in `sweep_handoff` scenario 139 (`MOQ_ROUTE_TRACE=
just rs test -p moq-net --lib -E 'test(=route::tests::handoff::path::sweep_handoff)'`,
+then `python3 sh/tla-trace.py rs/moq-net/verify/route/trace/RouteTrace.tla /handoff-139.ndjson`
+with `Stable` back in `RouteTrace.cfg`), the router and the model take the same
+steps, and at step 87 relay 4 drops two available claims
+offline. Relay 4 routes through relay 0 and has already heard the DEAD for
+relay 2's closed session to the publisher. Relay 0 has not, and re-paths
+through relay 2's closed session. The new offer is dead at relay 4, so it has no
+usable route. It holds nothing, because its next hop's path is no longer the
+one its route was built on, and it has no child or doubted offer to wait for.
+The claims come back once relay 0 hears the DEAD and moves. That is the flap
+`Stable` rules out. It needs six nodes and a message order no config reaches.
+
+Recommended: a relay holds under its next hop while that hop's offer is dead,
+as it does while the hop holds the path the route was built on, since the DEAD
+the relay heard is on its way to that hop (`StaleIsChased`). Check that holds
+still never close a cycle (`HoldCycle.cfg`, `QuietIsSettled`), add a config
+that reaches the flap, and change the router and model together.
+
+Public API: none. Wire: none.
+
+## Waiting
+
+- Maintainer: hold under a next hop whose new path is dead (recommended, as
+ above), or accept the flap and narrow `Stable` to exempt a path change that
+ makes the route dead, which keeps the rule as it is but lets a stale offer
+ flap what a reader sees until the DEAD catches up. Decide in the pick round
+ for moq-dev/moq#8733.
diff --git a/quest/m3/route-trace-validation.md b/quest/m3/route-trace-validation.md
index 5f2ec74099d..68e48802906 100644
--- a/quest/m3/route-trace-validation.md
+++ b/quest/m3/route-trace-validation.md
@@ -47,4 +47,37 @@ of it, so this waits for the router too.
Run it on the sweeps' existing seeds in CI only if it stays fast; otherwise in
Nightly with the sweeps' fresh seeds.
+Paused 2026-11-14 when the quest moved to m3, with the work done on its
+branch ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)), taken
+under /quest-afk for the maintainer's review:
+
+- Done: `route::tests::Sim` records each path-rule run of both sweeps
+ (`rs/moq-net/src/route/tests/trace.rs`), `rs/moq-net/verify/route/trace/RouteTrace.tla`
+ follows it step by step comparing every message and each touched relay's
+ route and shown claims, `sh/tla-trace.py` runs TLC per trace, and
+ `test route-trace` (Nightly, a fresh window of 150 scenarios) wires it up.
+ All 300 traces of the gate's scenarios pass on the pinned TLC, and two of
+ four planted router bugs fail traces while the sweeps pass.
+- ✅ Restrict the sweeps, and grow the model only where the router had
+ drifted (rejected: growing the model to several publishers, restarts,
+ grants, and interest edits, which four open PRs on `RouteHandoff.tla`
+ would conflict with). `sweep_handoff` runs one origin whose every claim change
+ changes a claim, with no restart; `the_rules_agree_on_every_script` keeps the
+ full script. The client, a leaf that offers no route, is dropped from the
+ traces. The model grew three ways: a route goes to a stream only while a
+ claim meets it, timers due together fire in one `Timeout`, and an ended
+ stream forgets what it reflected. Every config still passes.
+- ✅ Nightly only (rejected: the merge gate): 300 traces cost about 20
+ core-minutes of TLC.
+- Found: `Stable` breaks on an accepted trace, a behaviour of the model the
+ configs miss ([stable-repath](/quest/m2/cluster-routes/stable-repath.md)),
+ so `RouteTrace.cfg` leaves it out.
+
+Left: run `just fix` and `just check`, refresh the skill's state-count table
+for the grown model (`RouteHandoff.cfg` now 29,229 distinct states; the
+configs' new counts are in each run's output), run `test route-trace` through
+`sh/tlc.py` once Nix fetches TLC again (this host's daemon hung on it, so the
+runs used the same jars by hand), review with `/quest-review`, and finish the
+PR body.
+
Public API: none. Wire: none.
diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs
index 0c318bfa13c..5aae30c42bc 100644
--- a/rs/moq-net/src/route/path.rs
+++ b/rs/moq-net/src/route/path.rs
@@ -1048,18 +1048,24 @@ impl Router {
self.update(ids, Vec::new(), None);
}
- /// Fire every doubt and dead-hop timer due by `now`.
- pub fn timeout(&mut self, now: Instant) {
- let due: Vec = self
+ /// The doubts and dead hops whose timers are due by `now`.
+ pub fn due(&self, now: Instant) -> (Vec, Vec) {
+ let due = self
.doubted
.iter()
.filter_map(|(node, state)| (state.deadline() <= now).then_some(*node))
.collect();
- let expired: Vec = self
+ let expired = self
.dead
.iter()
.filter_map(|(hop, state)| state.deadline().filter(|at| *at <= now).map(|_| *hop))
.collect();
+ (due, expired)
+ }
+
+ /// Fire every doubt and dead-hop timer due by `now`.
+ pub fn timeout(&mut self, now: Instant) {
+ let (due, expired) = self.due(now);
if due.is_empty() && expired.is_empty() {
return;
}
@@ -1099,6 +1105,12 @@ impl Router {
// --- Queries ---
+ /// How this node numbers its session to `neighbor` in the hops it names.
+ #[cfg(test)]
+ pub fn session(&self, neighbor: N) -> Option {
+ self.sessions.get(&neighbor).map(|session| session.number)
+ }
+
/// The route this node uses for `id` under `prefix`.
pub fn route_for(&self, id: Id, prefix: &Path) -> Option<&Route> {
self.routes.get(&(id, prefix.clone()))
diff --git a/rs/moq-net/src/route/tests/handoff.rs b/rs/moq-net/src/route/tests/handoff.rs
index 1d1ddcebebf..25a9ee40309 100644
--- a/rs/moq-net/src/route/tests/handoff.rs
+++ b/rs/moq-net/src/route/tests/handoff.rs
@@ -1147,7 +1147,9 @@ fn announcement_cost_follows_sessions_and_live_claims(rule: Rule) {
/// grant under one subject, so both feed one qualified view. Under churn every
/// view stays some revision of the origin's truth, the widening guard stays
/// bounded, no claim ends while the origin still has it, and once churn stops
-/// the client shows exactly the claims its interest and the scopes admit.
+/// the client shows exactly the claims its interest and the scopes admit. Each
+/// claim change changes a claim, as the route model's do, so every run under the
+/// path rule is a trace `RouteTrace.tla` checks (`MOQ_ROUTE_TRACE`).
fn sweep_scoped_handoff(rule: Rule) {
const CLAIMS: [&str; 5] = ["live/a/x", "live/a/y/", "live/b/z", "live/c", "live/"];
const WATCHES: [&str; 5] = ["", "live/", "live/a/", "live/b/z", "live/c"];
@@ -1172,13 +1174,16 @@ fn sweep_scoped_handoff(rule: Rule) {
let links = [(origin, r1), (origin, r2), (r1, r2), (client, r1), (client, r2)];
sim.originate(origin, 7, 0);
assert!(sim.run());
+ sim.trace("scoped", scenario, origin, &CLAIMS);
+ let mut seed = 0;
for _ in 0..40 {
match rng.next(8) {
0..=2 => {
let claim = CLAIMS[rng.next(CLAIMS.len() as u64) as usize];
if rng.chance(60) {
- sim.announce(origin, claim, rng.next(3), None);
- } else {
+ seed += 1;
+ sim.announce(origin, claim, seed, None);
+ } else if sim.truth(7, claim, sim.revision(7)).is_some() {
sim.unannounce(origin, claim);
}
}
@@ -1223,6 +1228,7 @@ fn sweep_scoped_handoff(rule: Rule) {
}
}
assert!(sim.run());
+ sim.finish_trace();
sim.check().unwrap_or_else(|err| panic!("scenario {scenario}: {err}"));
let watch = sim.watching(client).clone();
let rev = sim.revision(7);
@@ -1262,16 +1268,17 @@ fn sweep_scoped_handoff(rule: Rule) {
const RELAYS: usize = 5;
const CLAIMS: [&str; 4] = ["live/", "live/a/", "live/a/x", "live/b"];
-/// Random meshes with two origins and a client under churn: claims change,
-/// sessions drop and return, relays send GOAWAY, messages are held and replayed
-/// out of step, the client changes interest, nodes collect route-less
-/// identities, and an origin restarts with a fresh identity. Selected routes
-/// never loop, every view is always some revision of the origin's truth,
-/// nothing revives and nothing ends falsely, and once churn stops every node
-/// shows every live origin's current claims and holds nothing for a dead one.
+/// A random mesh under churn, as the route model has it: one origin's claims
+/// change, sessions drop and return, relays send GOAWAY, messages are held and
+/// replayed out of step, and a client changes interest. Selected routes never
+/// loop, every view is always some revision of the origin's truth, nothing
+/// revives and nothing ends falsely, and once churn stops every node shows the
+/// origin's current claims. Under the path rule every run is a trace
+/// `RouteTrace.tla` checks (`MOQ_ROUTE_TRACE`); `the_rules_agree_on_every_script`
+/// runs the full script, with a second origin, restarts, and collection.
fn sweep_handoff(rule: Rule) {
for scenario in 0..scenarios() {
- let mut sim = churn(rule, scenario);
+ let mut sim = churn(rule, scenario, Shape::Model);
let label = format!("scenario {scenario}");
assert_eq!(revivals(&sim.events), 0, "{label}");
assert_eq!(false_ends(&sim.events), 0, "{label}");
@@ -1318,7 +1325,7 @@ fn the_rules_agree_on_every_script() {
for scenario in 0..scenarios() {
let label = format!("scenario {scenario}");
let sims = [Rule::Distance, Rule::PATH].map(|rule| {
- let mut sim = churn(rule, scenario);
+ let mut sim = churn(rule, scenario, Shape::Full);
assert_eq!(revivals(&sim.events), 0, "{label}: {rule:?}");
assert_eq!(false_ends(&sim.events), 0, "{label}: {rule:?}");
settled(&mut sim, &label);
@@ -1330,10 +1337,22 @@ fn the_rules_agree_on_every_script() {
}
}
+/// What a [`churn`] script does.
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum Shape {
+ /// Two origins, either of which restarts with a fresh identity, and nodes
+ /// collect route-less identities.
+ Full,
+ /// What the route model has: one origin, whose every claim change changes a
+ /// claim, and no restart. The second origin's node is a leaf with nothing to
+ /// publish. The run is recorded as a trace when `MOQ_ROUTE_TRACE` is set.
+ Model,
+}
+
/// Origins sit at `RELAYS` and `RELAYS + 1`, the client at `RELAYS + 2`. The script,
-/// every change made from outside, depends on `scenario` alone, so both rules run
-/// the same one; which queued message arrives next is drawn apart from it.
-fn churn(rule: Rule, scenario: usize) -> Sim {
+/// every change made from outside, depends on `scenario` and `shape` alone, so both
+/// rules run the same one; which queued message arrives next is drawn apart from it.
+fn churn(rule: Rule, scenario: usize, shape: Shape) -> Sim {
let rng = &mut XorShift(0x4a4d_0ff5 ^ (scenario as u64 + 1).wrapping_mul(0x9e37_79b9_7f4a_7c15) | 1);
let mut pick = XorShift(0x5eed_0003 + scenario as u64);
let (first, client) = (RELAYS, RELAYS + 2);
@@ -1358,18 +1377,32 @@ fn churn(rule: Rule, scenario: usize) -> Sim {
}
let mut ids = [100, 200];
sim.originate(first, ids[0], 0);
- sim.originate(first + 1, ids[1], 0);
+ if shape == Shape::Full {
+ sim.originate(first + 1, ids[1], 0);
+ }
assert!(sim.run());
+ sim.trace("handoff", scenario, first, &CLAIMS);
+ let mut seed = 0;
for _ in 0..40 {
match rng.next(10) {
0..=3 => {
- let origin = rng.next(2) as usize;
+ let origin = match shape {
+ Shape::Full => rng.next(2) as usize,
+ Shape::Model => 0,
+ };
let claim = CLAIMS[rng.next(CLAIMS.len() as u64) as usize];
- if rng.chance(50) {
- sim.announce(first + origin, claim, rng.next(3), None);
- } else {
- sim.unannounce(first + origin, claim);
+ let announce = rng.chance(50);
+ let id = ids[origin];
+ match shape {
+ Shape::Full if announce => sim.announce(first + origin, claim, rng.next(3), None),
+ Shape::Full => sim.unannounce(first + origin, claim),
+ Shape::Model if announce => {
+ seed += 1;
+ sim.announce(first, claim, seed, None);
+ }
+ Shape::Model if sim.truth(id, claim, sim.revision(id)).is_some() => sim.unannounce(first, claim),
+ Shape::Model => {}
}
}
4 if rng.chance(20) => sim.goaway(rng.next(RELAYS as u64) as usize),
@@ -1393,6 +1426,7 @@ fn churn(rule: Rule, scenario: usize) -> Sim {
6 => {
sim.watch(client, ["", "live/a/", "live/b"][rng.next(3) as usize]);
}
+ 7 if shape == Shape::Model => {}
7 if rng.chance(50) => {
// Collection is allowed once a node holds no route to an identity.
let node = rng.next(RELAYS as u64 + 3) as usize;
@@ -1438,6 +1472,7 @@ fn churn(rule: Rule, scenario: usize) -> Sim {
}
}
assert!(sim.run());
+ sim.finish_trace();
sim
}
@@ -1446,10 +1481,12 @@ fn churn(rule: Rule, scenario: usize) -> Sim {
/// for an identity whose process is gone.
fn settled(sim: &mut Sim, label: &str) {
sim.check().unwrap();
- let live: Vec = (RELAYS..RELAYS + 2).map(|origin| sim.proven(origin)).collect();
+ let live: Vec<(Node, Id)> = (RELAYS..RELAYS + 2)
+ .filter_map(|origin| Some((origin, sim.nodes[origin].declared()?)))
+ .collect();
for node in 0..RELAYS + 3 {
let mut expected = BTreeMap::new();
- for (origin, &id) in (RELAYS..).zip(&live) {
+ for &(origin, id) in &live {
let presented = sim.presented(id, sim.watching(node), sim.revision(id));
// A route reaches a node only while a claim meets its interest.
assert_eq!(
@@ -1480,7 +1517,10 @@ fn settled(sim: &mut Sim, label: &str) {
"{label}: node {node} {id} {claim}"
);
}
- for id in (100..110).chain(200..210).filter(|id| !live.contains(id)) {
+ for id in (100..110)
+ .chain(200..210)
+ .filter(|id| live.iter().all(|(_, live)| live != id))
+ {
assert!(!sim.holds(node, id), "{label}: node {node} holds dead {id}");
assert!(
sim.paths() || !sim.remembers(node, id),
diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs
index 85f7f5d9667..d6430bfb29f 100644
--- a/rs/moq-net/src/route/tests/mod.rs
+++ b/rs/moq-net/src/route/tests/mod.rs
@@ -17,6 +17,7 @@ mod residual;
mod routing;
mod rtt;
mod scope;
+mod trace;
use std::{
collections::{BTreeMap, BTreeSet, VecDeque},
@@ -174,7 +175,7 @@ impl Relay {
/// A claim's attributes, plus the revision its life began at so fixtures can
/// tell a revival from a fresh announcement.
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct Item {
pub seed: Metric,
pub epoch: Option,
@@ -365,6 +366,12 @@ pub(super) struct Sim {
/// The work of routers a restart replaced.
spent: usize,
pub events: Vec,
+ /// Under the path rule, each node's session number for a hop, as the neighbour and
+ /// the link's generation: how many times it opened before.
+ numbers: BTreeMap<(Node, u64), (Node, u32)>,
+ generations: BTreeMap<(Node, Node), u32>,
+ /// The run being recorded as a trace of `RouteHandoff.tla`.
+ trace: Option,
}
impl Sim {
@@ -398,6 +405,9 @@ impl Sim {
stats: Stats::default(),
spent: 0,
events: Vec::new(),
+ numbers: BTreeMap::new(),
+ generations: BTreeMap::new(),
+ trace: None,
}
}
@@ -475,6 +485,13 @@ impl Sim {
pub fn connect(&mut self, a: Node, b: Node) {
assert!(!self.connected(a, b), "{a}-{b} already up");
self.up.insert(Self::pair(a, b));
+ let generation = match self.generations.get_mut(&Self::pair(a, b)) {
+ Some(generation) => {
+ *generation += 1;
+ *generation
+ }
+ None => *self.generations.entry(Self::pair(a, b)).or_default(),
+ };
for (x, y) in [(a, b), (b, a)] {
let link = self.config(x, y);
let named = hop(y, self.restarts[y]);
@@ -483,11 +500,16 @@ impl Sim {
Relay::Path(router) => router.connect(y, named, link),
}
.expect("a positive cost");
+ if let Relay::Path(router) = &self.nodes[x] {
+ let number = router.session(y).expect("a session");
+ self.numbers.insert((x, number), (y, generation));
+ }
}
for (x, y) in [(a, b), (b, a)] {
self.ask(y, x);
}
self.flush();
+ self.note_link("connect", a, b);
}
/// `node` asks `peer` for announcements under its interest. Under the path rule a
@@ -534,10 +556,12 @@ impl Sim {
}
}
self.flush();
+ self.note_link("disconnect", a, b);
}
/// Every session of `node` closes at once, as a node shutting down closes them.
pub fn isolate(&mut self, node: Node) {
+ self.untraced("isolate");
let peers: Vec = self.peers(node).collect();
for &peer in &peers {
self.up.remove(&Self::pair(node, peer));
@@ -560,6 +584,7 @@ impl Sim {
/// The process at `node` exits: its sessions close and its state is lost.
pub fn kill(&mut self, node: Node) {
+ self.untraced("kill");
let peers: Vec = self.peers(node).collect();
for peer in peers {
self.disconnect(node, peer);
@@ -671,9 +696,14 @@ impl Sim {
let id = router.declared().unwrap();
let rev = router.revision().unwrap();
if self.declared[node] == Some(id) {
- self.history.entry(id).or_default().push(Change { rev, claim, item });
+ self.history.entry(id).or_default().push(Change {
+ rev,
+ claim: claim.clone(),
+ item,
+ });
}
self.flush();
+ self.note_claim(node, claim, item, rev);
}
/// The prefix `node` asks its neighbors for announcements under; each
@@ -690,6 +720,8 @@ impl Sim {
self.ask(node, peer);
}
self.flush();
+ let peers: Vec = self.peers(node).collect();
+ self.note_skip(&peers);
}
pub fn watching(&self, node: Node) -> &Path {
@@ -704,8 +736,10 @@ impl Sim {
Relay::Distance(router) => router.goaway(node),
Relay::Path(router) => router.goaway(node, now),
}
+ // One step per node that hears it, as the model takes them.
+ self.flush();
+ self.note_goaway(peer, node);
}
- self.flush();
}
pub fn collect(&mut self, node: Node, id: Id) {
@@ -714,6 +748,7 @@ impl Sim {
Relay::Path(router) => router.collect(id),
}
self.flush();
+ self.note_skip(&[node]);
}
/// `node` measures its link to `peer` at `ms`, or nothing.
@@ -749,6 +784,7 @@ impl Sim {
let Some(msg) = self.queues.get_mut(&(from, to)).and_then(VecDeque::pop_front) else {
return false;
};
+ let modeled = !matches!(msg, Msg::Synced { .. });
let now = self.now;
match (&mut self.nodes[to], msg) {
(Relay::Distance(router), Msg::Route { id, reach }) => router.route(from, id, reach),
@@ -787,6 +823,7 @@ impl Sim {
}
self.stats.delivered += 1;
self.flush();
+ self.note_deliver(from, to, modeled);
true
}
@@ -889,11 +926,15 @@ impl Sim {
/// Fire every doubt timer due by now.
fn tick(&mut self) {
let now = self.now;
- for relay in &mut self.nodes {
- if let Relay::Path(router) = relay
+ for node in 0..self.nodes.len() {
+ if let Relay::Path(router) = &mut self.nodes[node]
&& router.deadline().is_some_and(|at| at <= now)
{
+ let due = router.due(now);
router.timeout(now);
+ // One step per node, as the model takes them.
+ self.flush();
+ self.note_timeout(node, due);
}
}
self.flush();
@@ -1051,6 +1092,7 @@ impl Sim {
(to, Msg::Synced { interest })
}
};
+ self.note_sent(node, to, &msg);
self.queues.entry((node, to)).or_default().push_back(msg);
}
}
diff --git a/rs/moq-net/src/route/tests/trace.rs b/rs/moq-net/src/route/tests/trace.rs
new file mode 100644
index 00000000000..db9cc968cd6
--- /dev/null
+++ b/rs/moq-net/src/route/tests/trace.rs
@@ -0,0 +1,331 @@
+//! Records a [`Sim`] run under the path rule as a trace of
+//! `rs/moq-net/verify/route/RouteHandoff.tla`, for TLC to check that the run is a
+//! behaviour of the model (`rs/moq-net/verify/route/trace/RouteTrace.tla`).
+//!
+//! A trace is newline-delimited JSON. The first line names the model's constants:
+//! the nodes it has, the publisher, the links with their costs, and the claims.
+//! Every later line is one step: the model action it is (`op`), the messages it sent
+//! between modelled nodes, and what each node it touched now routes and shows.
+//!
+//! The model has one publisher and relays that pass routes on, so a trace keeps
+//! only those nodes. A leaf that publishes nothing, like a client, offers no route,
+//! so no path crosses it and nothing it does reaches another node's routes: the
+//! trace drops it, and steps that only it takes are `skip`, a step the model does not
+//! see, which must send nothing between modelled nodes. Messages to the publisher are
+//! dropped too, as the model drops them: it ignores what it hears.
+//!
+//! [`Sim::trace`] starts recording when `MOQ_ROUTE_TRACE` names a directory, and
+//! [`Sim::finish_trace`] writes the trace there. `MOQ_ROUTE_TRACE_FROM` skips the
+//! scenarios before it, so a run can trace a window of a long sweep.
+//! `test/route-trace.sh` records the sweeps and has TLC check every trace.
+
+use std::path::PathBuf;
+
+use serde_json::{Value, json};
+
+use super::*;
+
+/// A run being recorded.
+pub(super) struct Recorder {
+ file: PathBuf,
+ /// The publisher's identity, and the node publishing it.
+ id: Id,
+ origin: Node,
+ /// The nodes the trace keeps: the publisher and every relay.
+ nodes: BTreeSet,
+ claims: BTreeSet,
+ /// The model's attributes for each item announced: the revision that set it.
+ items: BTreeMap<(Path, Item), Rev>,
+ /// The messages sent since the last step.
+ sent: Vec,
+ steps: Vec,
+}
+
+impl Sim {
+ /// Record the rest of this run as `-` if `MOQ_ROUTE_TRACE` names a
+ /// directory. `origin` publishes the one identity, and `claims` are the claims it
+ /// may make: the trace starts here, as the model's initial state does, with every
+ /// configured link open, the publisher's identity declared with no claims, and
+ /// nothing in flight.
+ pub fn trace(&mut self, sweep: &str, scenario: usize, origin: Node, claims: &[&str]) {
+ let Some(dir) = std::env::var_os("MOQ_ROUTE_TRACE") else {
+ return;
+ };
+ let from = std::env::var("MOQ_ROUTE_TRACE_FROM").map_or(0, |from| from.parse().expect("MOQ_ROUTE_TRACE_FROM"));
+ if !self.paths() || self.rule != Rule::PATH || scenario < from {
+ return;
+ }
+ let id = self.proven(origin);
+ assert!(self.busy().is_empty(), "a trace starts with nothing in flight");
+ assert_eq!(self.revision(id), 0, "a trace starts with no claims");
+ let nodes = (0..self.nodes.len())
+ .filter(|&node| node == origin || !self.leaf[node])
+ .collect();
+ self.trace = Some(Recorder {
+ file: PathBuf::from(dir).join(format!("{sweep}-{scenario}.ndjson")),
+ id,
+ origin,
+ nodes,
+ claims: claims.iter().map(|claim| p(claim)).collect(),
+ items: BTreeMap::new(),
+ sent: Vec::new(),
+ steps: Vec::new(),
+ });
+ for &(a, b) in &self.up {
+ assert!(
+ !(self.modelled(a) && self.modelled(b)) || self.generations[&(a, b)] == 0,
+ "a trace starts on sessions never reopened"
+ );
+ }
+ }
+
+ /// Write the recorded trace, if any.
+ pub fn finish_trace(&mut self) {
+ let Some(trace) = self.trace.take() else {
+ return;
+ };
+ assert!(trace.sent.is_empty(), "messages sent outside a step");
+ let links: Vec = self
+ .up
+ .iter()
+ .filter(|(a, b)| trace.nodes.contains(a) && trace.nodes.contains(b))
+ .map(|&(a, b)| {
+ let cost = self.links[a][b].expect("configured").cost;
+ assert_eq!(cost, self.links[b][a].expect("configured").cost, "a symmetric cost");
+ json!([a, b, cost])
+ })
+ .collect();
+ let header = json!({
+ "nodes": trace.nodes,
+ "origin": trace.origin,
+ "links": links,
+ "claims": trace.claims.iter().map(Path::as_str).collect::>(),
+ });
+ let mut out = String::new();
+ for line in std::iter::once(&header).chain(&trace.steps) {
+ out += &line.to_string();
+ out.push('\n');
+ }
+ std::fs::write(&trace.file, out).expect("write the trace");
+ }
+
+ /// Fail a run that does something the trace cannot express.
+ pub(super) fn untraced(&self, what: &str) {
+ assert!(self.trace.is_none(), "a traced run cannot {what}");
+ }
+
+ fn modelled(&self, node: Node) -> bool {
+ self.trace.as_ref().is_some_and(|trace| trace.nodes.contains(&node))
+ }
+
+ /// Record a message `from` queued for `to`, if the model has it.
+ pub(super) fn note_sent(&mut self, from: Node, to: Node, msg: &Msg) {
+ let Some(trace) = &self.trace else {
+ return;
+ };
+ if !self.modelled(from) || !self.modelled(to) || to == trace.origin {
+ return;
+ }
+ let key = |id: Id, prefix: &Path| {
+ assert_eq!(id, trace.id, "one publisher");
+ Self::key(prefix)
+ };
+ let msg = match msg {
+ Msg::Path { id, prefix, reach, .. } => match reach {
+ Some(path::Reach::Finite(offer)) => json!({
+ "t": "offer",
+ "k": key(*id, prefix),
+ "path": self.hops(&offer.path),
+ "drain": offer.draining,
+ }),
+ Some(path::Reach::Held) => json!({"t": "hold", "k": key(*id, prefix)}),
+ None => json!({"t": "end", "k": key(*id, prefix)}),
+ },
+ Msg::Claims { id, prefix, floor, .. } => json!({
+ "t": "claims",
+ "k": key(*id, prefix),
+ "rev": floor.map_or(-1, |floor| floor as i64),
+ }),
+ Msg::Dead(path::Dead::Closed { hop, far, moved, gone }) => {
+ let named = self.hop_of(*hop);
+ assert_eq!(
+ named[1],
+ Self::node_of(*far) as u64,
+ "a DEAD's far end is its hop's peer"
+ );
+ let moved: Vec = moved
+ .iter()
+ .map(|moved| {
+ json!([
+ key(moved.id, &moved.prefix),
+ self.hops(&moved.offer.path),
+ moved.offer.draining
+ ])
+ })
+ .collect();
+ json!({"t": "dead", "hop": named, "moved": moved, "gone": gone})
+ }
+ Msg::Dead(path::Dead::Draining { node }) => json!({"t": "drain", "node": Self::node_of(*node)}),
+ Msg::Synced { .. } => return,
+ msg => panic!("the path rule sent {msg:?}"),
+ };
+ let sent = json!([from, to, msg]);
+ self.trace.as_mut().unwrap().sent.push(sent);
+ }
+
+ /// The model's key for a route's prefix: the root, or the prefix itself.
+ fn key(prefix: &Path) -> String {
+ match prefix.as_str() {
+ "" => "root".to_string(),
+ prefix => prefix.to_string(),
+ }
+ }
+
+ /// The node a hop id names, which never restarted in a traced run.
+ fn node_of(id: Id) -> Node {
+ assert!((1_000..2_000).contains(&id), "{id} names a node that never restarted");
+ (id - 1_000) as Node
+ }
+
+ /// The model's hop: the node, the neighbour it learned the route from, and that
+ /// session's generation. A publisher names itself with generation 0.
+ fn hop_of(&self, hop: path::Hop) -> [u64; 3] {
+ let node = Self::node_of(hop.node);
+ if hop.session == 0 {
+ return [node as u64, node as u64, 0];
+ }
+ let (peer, generation) = self.numbers[&(node, hop.session)];
+ [node as u64, peer as u64, u64::from(generation)]
+ }
+
+ fn hops(&self, path: &[path::Hop]) -> Vec<[u64; 3]> {
+ path.iter().map(|hop| self.hop_of(*hop)).collect()
+ }
+
+ /// What `node` routes and shows, as the model keeps it.
+ fn node_state(&self, node: Node) -> Value {
+ let trace = self.trace.as_ref().unwrap();
+ let router = self.path(node);
+ let route = match router.route_for(trace.id, &Path::empty()) {
+ None => json!({"on": false}),
+ Some(path::Route::Live { next, offer }) => json!({
+ "on": true,
+ "held": false,
+ "next": next,
+ "path": self.hops(&offer.path),
+ "drain": offer.draining,
+ }),
+ Some(path::Route::Held { under, path }) => json!({
+ "on": true,
+ "held": true,
+ "next": under.map_or(-1, |under| under as i64),
+ "path": self.hops(path),
+ }),
+ Some(route) => panic!("node {node} routes {route:?} between updates"),
+ };
+ let others: Vec<&Path> = router
+ .routes(trace.id)
+ .map(|(prefix, _)| prefix)
+ .filter(|prefix| !prefix.is_empty())
+ .collect();
+ assert!(
+ others.is_empty(),
+ "node {node} routes {others:?}, which the model has not"
+ );
+ let shown: Vec = router
+ .shown()
+ .iter()
+ .filter(|((id, _), _)| *id == trace.id)
+ .map(|((_, claim), shown)| {
+ let item = trace.items[&(claim.clone(), shown.item)];
+ json!([claim.as_str(), item, shown.available])
+ })
+ .collect();
+ json!({"node": node, "route": route, "shown": shown})
+ }
+
+ /// Record a step: `op`, the messages sent since the last one, and what each
+ /// modelled node in `touched` now routes and shows.
+ fn note(&mut self, mut op: Value, touched: &[Node]) {
+ let Some(trace) = &self.trace else {
+ return;
+ };
+ let origin = trace.origin;
+ let state: Vec = touched
+ .iter()
+ .copied()
+ .filter(|&node| self.modelled(node) && node != origin)
+ .map(|node| self.node_state(node))
+ .collect();
+ let trace = self.trace.as_mut().unwrap();
+ op["sent"] = Value::Array(std::mem::take(&mut trace.sent));
+ op["state"] = Value::Array(state);
+ trace.steps.push(op);
+ }
+
+ /// A step the model does not see: it must send nothing between modelled nodes.
+ pub(super) fn note_skip(&mut self, touched: &[Node]) {
+ self.note(json!({"op": "skip"}), touched);
+ }
+
+ pub(super) fn note_link(&mut self, op: &str, a: Node, b: Node) {
+ match self.modelled(a) && self.modelled(b) {
+ true => self.note(json!({"op": op, "a": a, "b": b}), &[a, b]),
+ false => self.note_skip(&[a, b]),
+ }
+ }
+
+ pub(super) fn note_deliver(&mut self, from: Node, to: Node, modelled: bool) {
+ let origin = self.trace.as_ref().map(|trace| trace.origin);
+ match modelled && self.modelled(from) && self.modelled(to) && Some(to) != origin {
+ true => self.note(json!({"op": "deliver", "from": from, "to": to}), &[to]),
+ false => self.note_skip(&[to]),
+ }
+ }
+
+ pub(super) fn note_goaway(&mut self, node: Node, from: Node) {
+ let origin = self.trace.as_ref().map(|trace| trace.origin);
+ match self.modelled(node) && self.modelled(from) && Some(node) != origin {
+ true => self.note(json!({"op": "goaway", "node": node, "from": from}), &[node]),
+ false => self.note_skip(&[node]),
+ }
+ }
+
+ pub(super) fn note_timeout(&mut self, node: Node, (lifted, expired): (Vec, Vec)) {
+ if self.trace.is_none() {
+ return;
+ }
+ let lift: Vec = (lifted.into_iter().map(Self::node_of))
+ .filter(|&node| self.modelled(node))
+ .collect();
+ let expire: Vec<[u64; 3]> = (expired.into_iter().map(|hop| self.hop_of(hop)))
+ .filter(|hop| self.modelled(hop[0] as Node) && self.modelled(hop[1] as Node))
+ .collect();
+ // The model's publisher keeps no doubt: it ignores what it hears.
+ let origin = self.trace.as_ref().map(|trace| trace.origin);
+ match self.modelled(node) && Some(node) != origin && !(lift.is_empty() && expire.is_empty()) {
+ true => self.note(
+ json!({"op": "timeout", "node": node, "lift": lift, "expire": expire}),
+ &[node],
+ ),
+ false => self.note_skip(&[node]),
+ }
+ }
+
+ pub(super) fn note_claim(&mut self, node: Node, claim: Path, item: Option- , rev: Rev) {
+ let Some(trace) = &mut self.trace else {
+ return;
+ };
+ assert_eq!(node, trace.origin, "one publisher");
+ assert!(
+ trace.claims.contains(&claim),
+ "{claim} is not one of the trace's claims"
+ );
+ if let Some(item) = item {
+ let was = trace.items.insert((claim.clone(), item), rev);
+ assert!(was.is_none(), "each announcement in a trace sets a new item");
+ }
+ let op = json!({"op": "claim", "claim": claim.as_str(), "item": item.is_some(), "rev": rev});
+ self.note(op, &[]);
+ }
+}
diff --git a/rs/moq-net/verify/README.md b/rs/moq-net/verify/README.md
index 17bd335594d..eb555d7cede 100644
--- a/rs/moq-net/verify/README.md
+++ b/rs/moq-net/verify/README.md
@@ -9,3 +9,12 @@ A counterexample is a path through the model, not yet a bug. Replay it on the co
## Coverage
After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it.
+
+## Trace validation
+
+A model the code has drifted from keeps passing. Trace validation (Cirstea, Kuppe, Loillier, Merz, 2024) checks the code's own runs against it: a test records each run as the model actions it takes, and TLC checks that the run is a behaviour of the spec. The route model has it (`verify-route`); the others follow the same shape.
+
+- The test writes one trace per run, as newline-delimited JSON, into the directory its environment variable names. The first line gives the constants the run needs, such as the mesh; each later line is one step: `op`, the action and its arguments, then what the step sent and what each node it touched now holds, in the model's terms.
+- A trace spec in the model's `trace/` directory EXTENDS the model, reads the trace TLC's environment names (`ndJsonDeserialize(IOEnv.TRACE)`), and takes each step through the model's own action. Its config substitutes the trace's constants (`Nodes <- TNodes`). Each action must be deterministic given its arguments, so TLC follows one state per step. A step the model cannot take is a deadlock; a step whose messages or state differ breaks `Agrees`, in a state its `ALIAS` shows beside the step.
+- `sh/tla-trace.py ...` runs TLC on each trace, four at once on one worker each, and keeps only the traces that fail. `sh/tla.sh` skips `trace/` directories.
+- Where the code does more than the model, either restrict the recorded runs to what the model has or grow the model; say which, and drop from the trace only what cannot reach the rest of the state. Planting a bug in the code and seeing a trace fail checks that the comparison has teeth.
diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla
index e5f2ec9ae35..6f4a4d892a2 100644
--- a/rs/moq-net/verify/route/RouteHandoff.tla
+++ b/rs/moq-net/verify/route/RouteHandoff.tla
@@ -20,7 +20,7 @@
(* neighbour that learned the route from it has answered. *)
(* - A closed session casts doubt on the node at its far end: no relay *)
(* moves to a route through it until a timer lifts the doubt. The timer *)
-(* is the step `Lift`, which may fire at any moment. *)
+(* is the step `Timeout`, which may fire at any moment. *)
(* *)
(* Each node's every entry point is one atomic step ending in `Update`, *)
(* as in the router. Messages travel on one FIFO per direction. A *)
@@ -93,7 +93,9 @@ NoHeard == [on |-> FALSE, path |-> <<>>, drain |-> FALSE, state |-> "live", rev
\* route's, and `next` is the neighbour it holds under, or None on its own account.
NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>, drain |-> FALSE]
\* What a node last sent a neighbour for a key (`Sent`). `src` is the key of
-\* the node's own route the offer reflects, kept after an end.
+\* the node's own route the offer reflects, kept after an end while the
+\* neighbour's route there may be a child yet to answer; NoSrc once forgotten.
+NoSrc == "none"
NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, drain |-> FALSE, rev |-> None, items |-> NoItems, src |-> k]
NotShown == [item |-> None, avail |-> FALSE]
@@ -290,27 +292,47 @@ ShownNow(s) ==
-----------------------------------------------------------------------------
(* What a node sends (path.rs: `want`, `advertise`, `report`). *)
+\* `covering_view`: the keys of the narrowest view at or above k, at most one.
+CoverKey(s, k) == IF s.view[k].floor # Absent THEN {k}
+ ELSE IF s.view["root"].floor # Absent THEN {"root"}
+ ELSE {}
+
+\* `meets`: whether this node presents any claim at k.
+Meets(s, k) == \E ck \in CoverKey(s, k) : \E c \in Claims : Covers(k, c) /\ s.view[ck].items[c] # None
+
\* What this node wants a neighbour p to hold for key k: a path, a hold, an
\* end, or to keep what it was sent while the next hop this node holds under
-\* waits for those behind it.
+\* waits for those behind it. A route goes to a stream only while a claim meets
+\* it: a path that meets none is "empty", which withdraws the claims sent and
+\* then ends the route.
Want(n, s, k, p) ==
LET src == SrcKey(s, p, k)
r == s.rt[src]
- IN IF ~r.on THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src]
- ELSE IF ~r.held THEN [kind |-> "path", path |-> r.path, drain |-> r.drain, src |-> src]
+ End == [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src]
+ IN IF ~r.on THEN End
+ ELSE IF ~r.held THEN [kind |-> IF Meets(s, k) THEN "path" ELSE "empty",
+ path |-> r.path, drain |-> r.drain, src |-> src]
ELSE IF r.next = p
- THEN [kind |-> IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end",
- path |-> <<>>, drain |-> FALSE, src |-> src]
+ THEN [End EXCEPT !.kind = IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end"]
\* A hold names no path, so a stream holds only the path it last sent.
- ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path
- THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src]
- ELSE [kind |-> "hold", path |-> <<>>, drain |-> FALSE, src |-> src]
+ ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path THEN End
+ ELSE [End EXCEPT !.kind = IF Meets(s, k) THEN "hold" ELSE "end"]
+
+\* The view a stream for k presents from, empty while there is none.
+Presented(s, k) == IF s.view[k].floor = Absent THEN EmptyView ELSE s.view[k]
+
+\* The batch that withdraws what was sent t for k before an "empty" ends it:
+\* sent whenever there is a view, unless it would say nothing at all.
+Withdraw(s, k, t) ==
+ LET v == Presented(s, k)
+ IN IF s.view[k].floor # Absent /\ (v.floor # None \/ t.items # v.items)
+ THEN <> ELSE <<>>
\* `advertise` for one key on the stream to p.
Advertise1(n, s, p, k) ==
LET t == s.tx[p][k]
w == Want(n, s, k, p)
- v == IF s.view[k].floor = Absent THEN EmptyView ELSE s.view[k]
+ v == Presented(s, k)
IN CASE k \notin Streams(p) \/ w.kind = "keep" -> [tx |-> t, out |-> <<>>]
[] w.kind = "hold" ->
IF t.on /\ ~t.held THEN [tx |-> [t EXCEPT !.held = TRUE], out |-> <>]
@@ -318,6 +340,9 @@ Advertise1(n, s, p, k) ==
[] w.kind = "end" ->
IF t.on THEN [tx |-> NoSent(t.src), out |-> <>]
ELSE [tx |-> t, out |-> <<>>]
+ [] w.kind = "empty" ->
+ IF t.on THEN [tx |-> NoSent(t.src), out |-> Withdraw(s, k, t) \o <>]
+ ELSE [tx |-> t, out |-> <<>>]
[] OTHER ->
LET offer == ~t.on \/ t.held \/ t.path # w.path \/ t.drain # w.drain
claims == t.rev # v.floor \/ t.items # v.items
@@ -351,17 +376,20 @@ Report1(n, s, p, hop) ==
\* The neighbour this node learned the route from takes silence to mean
\* it holds under it, so it is told outright.
parents == {k \in holds : Child(t[k].path, p)}
- ends == {k \in through : w[k].kind = "end"}
- gone == \A k \in through : w[k].kind \in {"path", "end"}
+ ends == {k \in through : w[k].kind \in {"end", "empty"}}
+ gone == \A k \in through : w[k].kind \in {"path", "end", "empty"}
first == hop \notin s.told[p]
tx1 == [k \in Keys |->
IF k \notin through THEN t[k]
ELSE CASE w[k].kind = "path" -> [t[k] EXCEPT !.path = w[k].path, !.drain = w[k].drain,
!.held = FALSE, !.src = w[k].src]
[] w[k].kind = "hold" -> [t[k] EXCEPT !.held = TRUE]
- [] w[k].kind = "end" -> NoSent(t[k].src)
+ [] w[k].kind \in {"end", "empty"} -> NoSent(t[k].src)
[] OTHER -> t[k]]
- out == IF first \/ (gone /\ ends # {})
+ \* An "empty" route's withdrawal goes out at once, before the news.
+ withdrawn == Each({k \in through : w[k].kind = "empty"}, LAMBDA k : Withdraw(s, k, t[k]))
+ out == withdrawn \o
+ IF first \/ (gone /\ ends # {})
THEN <>
\o (IF gone THEN <<>>
ELSE Each(parents, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>))
@@ -388,10 +416,15 @@ ReportDrain(n, s, ups, x) ==
out |-> [p \in Nodes |-> IF p \in tell THEN <> ELSE <<>>]]
\* `collect`: forget each dead hop no route names once its timer has fired,
-\* and each told hop no sent route names. Doubt lasts until Lift fires.
-Collect(s) ==
+\* and each told hop no sent route names. Doubt lasts until its timer fires.
+\* An ended stream forgets what it reflected unless the neighbour's route there
+\* extends this node's, as `advertise` drops such an entry.
+Collect(n, s) ==
LET heard == {s.rx[x[1]][x[2]].path : x \in {y \in Nodes \X Keys : s.rx[y[1]][y[2]].on}}
+ Kept(m, k) == s.tx[m][k].on \/ (s.rx[m][k].on /\ s.rx[m][k].state # "held" /\ Child(s.rx[m][k].path, n))
IN [s EXCEPT
+ !.tx = [m \in Nodes |-> [k \in Keys |->
+ IF Kept(m, k) THEN s.tx[m][k] ELSE NoSent(NoSrc)]],
!.dead = {h \in s.dead : h \in s.expiry \/ \E p \in heard : HasHop(p, h)},
!.draining = {x \in s.draining : x \in s.doubt \/ \E p \in heard : Names(p, x)},
!.told = [m \in Nodes |-> {h \in s.told[m] :
@@ -438,7 +471,7 @@ Update(n, s0, ov, ups, gens, eb, rep, quiet) ==
ELSE IF rep.node # None THEN ReportDrain(n, s2, ups, rep.node)
ELSE [s |-> s2, out |-> [p \in Nodes |-> <<>>]]
a == Advertise(n, r.s, ups)
- s3 == Collect(a.s)
+ s3 == Collect(n, a.s)
now == IF n = Origin THEN [c \in Claims |-> NotShown] ELSE ShownNow(s3)
ev == Events(s0, s3, s0.shown, now, ov, eb, quiet)
IN [s |-> [s3 EXCEPT !.shown = now],
@@ -502,7 +535,7 @@ InitNode == [
expiry |-> {}, \* Pending timers belong to individual dead hops.
drain |-> {}, \* the neighbours that sent GOAWAY on the session open to them
draining |-> {}, \* nodes known to have sent GOAWAY, while named or doubted
- tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(k)]],
+ tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(NoSrc)]],
told |-> [m \in Nodes |-> {}],
toldDrain |-> [m \in Nodes |-> {}],
shown |-> [c \in Claims |-> NotShown]
@@ -617,28 +650,23 @@ Deliver(d) ==
/\ Internal(Sent(rs))
/\ UNCHANGED <>
-\* The timer node n started on x fires: the doubt on x lifts, and the dead hops
-\* whose session ended at x may be forgotten. With `Patient`, only once no DEAD
-\* about such a hop, and no DRAIN of x, is in flight: a timer long enough for
-\* the news to cross the mesh.
-Lift(n, x) ==
- LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x},
- !.expiry = {h \in @ : h[2] # x}],
+\* `timeout`: the timers on node n that come due together fire in one update.
+\* Each doubt in X lifts, and the dead hops whose session ended there may be
+\* forgotten; each hop in H expires on its own timer, which can fire before a
+\* later failure's timer on the same far node, so a new DEAD cannot extend every
+\* old hop's retention indefinitely. With `Patient`, a timer fires only once no
+\* DEAD about its hop, and no DRAIN of its node, is in flight: a timer long
+\* enough for the news to cross the mesh.
+Timeout(n, X, H) ==
+ LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ X, !.wait = @ \ X,
+ !.expiry = {h \in @ : h \notin H /\ h[2] \notin X}],
OV, up, gen, ended[n], NoReport, TRUE)]
- IN /\ x \in st[n].wait
+ IN /\ X \subseteq st[n].wait
+ /\ H \subseteq st[n].expiry
+ /\ X # {} \/ H # {}
/\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] :
- ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) /\ ~(q[d][i].t = "drain" /\ q[d][i].node = x)
- /\ Commit(rs, q)
- /\ Internal(Sent(rs))
- /\ UNCHANGED <>
-
-\* A hop's own timer can fire before a later failure's timer on the same far
-\* node. A new DEAD cannot extend every old hop's retention indefinitely.
-Expire(n, hop) ==
- LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.expiry = @ \ {hop}],
- OV, up, gen, ended[n], NoReport, TRUE)]
- IN /\ hop \in st[n].expiry
- /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop = hop)
+ /\ ~(q[d][i].t = "dead" /\ (q[d][i].hop[2] \in X \/ q[d][i].hop \in H))
+ /\ ~(q[d][i].t = "drain" /\ q[d][i].node \in X)
/\ Commit(rs, q)
/\ Internal(Sent(rs))
/\ UNCHANGED <>
@@ -662,7 +690,7 @@ GoAway(n, m) ==
\* A closed session: everything learned or sent on it goes.
Closed(s, y) ==
[s EXCEPT !.rx[y] = [k \in Keys |-> NoHeard],
- !.tx[y] = [k \in Keys |-> NoSent(k)],
+ !.tx[y] = [k \in Keys |-> NoSent(NoSrc)],
!.told[y] = {},
!.toldDrain[y] = {},
!.drain = @ \ {y}]
@@ -711,8 +739,7 @@ Done ==
Next ==
\/ \E c \in Claims : Announce(c) \/ Unannounce(c)
\/ \E d \in Dirs : Deliver(d)
- \/ \E n \in Relays, x \in Nodes : Lift(n, x)
- \/ \E n \in Relays : \E hop \in st[n].expiry : Expire(n, hop)
+ \/ \E n \in Relays : \E X \in SUBSET st[n].wait, H \in SUBSET st[n].expiry : Timeout(n, X, H)
\/ \E l \in Links : Connect(l) \/ Disconnect(l)
\/ \E n \in Relays, m \in Leaving : GoAway(n, m)
\/ Done
diff --git a/rs/moq-net/verify/route/trace/RouteTrace.cfg b/rs/moq-net/verify/route/trace/RouteTrace.cfg
new file mode 100644
index 00000000000..c8f20d24831
--- /dev/null
+++ b/rs/moq-net/verify/route/trace/RouteTrace.cfg
@@ -0,0 +1,37 @@
+\* Checks the trace TRACE names; `just test route-trace` records and checks
+\* every sweep run. The trace gives the mesh; the rest is what the sweeps have.
+CONSTANTS
+ Nodes <- TNodes
+ Origin <- TOrigin
+ Links <- TLinks
+ Cost <- TCost
+ Leaving <- TLeaving
+ Claims <- TClaims
+ Dear = {}
+ Ring = {}
+ Preferred = {}
+ Scoped = {}
+ Narrow = {}
+ MaxRev = 1000000
+ MaxFlaps = 1000000
+ \* The sweeps hold messages past any timer.
+ Patient = FALSE
+
+SPECIFICATION TraceSpec
+ALIAS Failure
+
+\* The model's rules, on the router's own runs. `Stable` is left out: the
+\* sweep's handoff scenario 139 breaks it, in a behaviour of the model too
+\* (quest/m2/cluster-routes/stable-repath.md).
+INVARIANTS
+ Agrees
+ Truthful
+ NoRevival
+ NoFalseEnd
+ NoRegression
+ Seen
+ SimplePaths
+ LoopsAreStale
+ StaleIsChased
+ RequestsLoopFree
+ QuietIsSettled
diff --git a/rs/moq-net/verify/route/trace/RouteTrace.tla b/rs/moq-net/verify/route/trace/RouteTrace.tla
new file mode 100644
index 00000000000..d34b85b1847
--- /dev/null
+++ b/rs/moq-net/verify/route/trace/RouteTrace.tla
@@ -0,0 +1,143 @@
+----------------------------- MODULE RouteTrace -----------------------------
+(***************************************************************************)
+(* Trace validation: whether a run of the router, as `route::tests::Sim` *)
+(* records it (rs/moq-net/src/route/tests/trace.rs), is a behaviour of *)
+(* RouteHandoff. *)
+(* *)
+(* The trace's first line gives the model's constants; each later line is *)
+(* one step of `Next` with its arguments, the messages the router sent, *)
+(* and what each node it touched routes and shows. Each step takes that *)
+(* action, and its messages and the touched nodes' routes and shown claims *)
+(* must be the router's. Every action is deterministic given its *)
+(* arguments, so TLC follows one state per step. A step the model cannot *)
+(* take leaves no successor, and TLC reports a deadlock; one whose *)
+(* messages or state differ breaks `Agrees`, in a state that shows the *)
+(* model's queues and nodes. Either way the last state's `l` is one past *)
+(* the step that failed. A run that matches every step ends in `Matched`, *)
+(* which stutters. *)
+(* *)
+(* `skip` is a step the model does not see, such as a delivery of SYNCED *)
+(* or one only a client takes: nothing may change and nothing be sent. *)
+(***************************************************************************)
+EXTENDS RouteHandoff, Json, IOUtils, TLC
+
+\* The trace named by the environment variable TRACE.
+Log == ndJsonDeserialize(IOEnv.TRACE)
+Header == Log[1]
+Steps == [i \in 1..Len(Log) - 1 |-> Log[i + 1]]
+
+ToSet(s) == {s[i] : i \in DOMAIN s}
+
+\* The constants the trace gives, which the config substitutes for the model's.
+TNodes == ToSet(Header.nodes)
+TOrigin == Header.origin
+TLinks == {{e[1], e[2]} : e \in ToSet(Header.links)}
+TClaims == ToSet(Header.claims)
+\* Every relay may send GOAWAY.
+TLeaving == TNodes \ {TOrigin}
+\* Each link's configured cost, in place of `Dear`'s 1 or 2.
+TCost == [l \in TLinks |-> (CHOOSE e \in ToSet(Header.links) : {e[1], e[2]} = l)[3]]
+
+VARIABLES
+ l, \* the step to take next
+ ok \* whether the last step's messages and state matched the router's
+traceVars == <>
+
+-----------------------------------------------------------------------------
+(* Matching what the router did. *)
+
+\* A model message is the router's. A claims message is matched by its floor:
+\* `Truthful` and the sweep's own check pin the claims at a floor.
+Match(m, j) ==
+ /\ m.t = j.t
+ /\ CASE j.t = "offer" -> m.k = j.k /\ m.path = j.path /\ m.drain = j.drain
+ [] j.t \in {"hold", "end"} -> m.k = j.k
+ [] j.t = "claims" -> m.k = j.k /\ m.rev = j.rev
+ [] j.t = "dead" -> m.hop = j.hop /\ m.gone = j.gone /\ m.moved = ToSet(j.moved)
+ [] j.t = "drain" -> m.node = j.node
+
+\* The messages step e sent on d, in order.
+SentOn(e, d) == LET mine == SelectSeq(e.sent, LAMBDA x : x[1] = d[1] /\ x[2] = d[2])
+ IN [i \in DOMAIN mine |-> mine[i][3]]
+
+\* What d held before step e appended to it: less the message it delivered, or
+\* nothing once its session closed.
+Before(e, d) == IF e.op = "deliver" /\ d = <> THEN Tail(q[d])
+ ELSE IF e.op = "disconnect" /\ {d[1], d[2]} = {e.a, e.b} THEN <<>>
+ ELSE q[d]
+
+\* Every queue gained exactly what the router sent on it. The model drops what
+\* is sent to the publisher, and so does the trace.
+SentMatches(e) ==
+ /\ \A x \in ToSet(e.sent) : <> \in Dirs
+ /\ \A d \in Dirs : d[2] # Origin =>
+ LET b == Before(e, d)
+ c == SentOn(e, d)
+ IN /\ Len(q'[d]) = Len(b) + Len(c)
+ /\ \A i \in DOMAIN c : Match(q'[d][Len(b) + i], c[i])
+
+RouteMatches(r, j) ==
+ /\ r.on = j.on
+ /\ j.on => /\ r.held = j.held
+ /\ r.next = j.next
+ /\ r.path = j.path
+ /\ ~j.held => r.drain = j.drain
+
+ShownMatches(shown, j) ==
+ \A c \in Claims :
+ shown[c] = IF \E x \in ToSet(j) : x[1] = c
+ THEN LET x == CHOOSE x \in ToSet(j) : x[1] = c
+ IN [item |-> x[2], avail |-> x[3]]
+ ELSE NotShown
+
+\* Each node the step touched routes and shows what the router's does. The
+\* router keeps routes for the root alone: a trace has no scoped relay.
+StateMatches(e) ==
+ \A x \in ToSet(e.state) :
+ /\ RouteMatches(st'[x.node].rt["root"], x.route)
+ /\ ~st'[x.node].rt["a"].on
+ /\ ShownMatches(st'[x.node].shown, x.shown)
+
+-----------------------------------------------------------------------------
+(* Following the trace. *)
+
+Skip == UNCHANGED vars
+
+\* The action step e names.
+Act(e) ==
+ CASE e.op = "claim" -> /\ IF e.item THEN Announce(e.claim) ELSE Unannounce(e.claim)
+ /\ rev' = e.rev
+ [] e.op = "deliver" -> Deliver(<>)
+ [] e.op = "connect" -> Connect({e.a, e.b})
+ [] e.op = "disconnect" -> Disconnect({e.a, e.b})
+ [] e.op = "timeout" -> Timeout(e.node, ToSet(e.lift), ToSet(e.expire))
+ \* GOAWAY again on a session already draining: `GoAway` is not enabled.
+ [] e.op = "goaway" -> IF e.from \in st[e.node].drain THEN Skip ELSE GoAway(e.node, e.from)
+ [] e.op = "skip" -> Skip
+
+TraceInit == Init /\ l = 1 /\ ok = [sent |-> TRUE, state |-> TRUE]
+
+TraceNext ==
+ /\ l <= Len(Steps)
+ /\ Act(Steps[l])
+ /\ ok' = [sent |-> SentMatches(Steps[l]), state |-> StateMatches(Steps[l])]
+ /\ l' = l + 1
+
+\* The router sent and holds what the model does.
+Agrees == ok.sent /\ ok.state
+
+\* What a failing state shows: the step it took, what matched, and the model's
+\* routes, shown claims, doubts, dead hops, and queues after it.
+Failure == [step |-> IF l > 1 THEN Steps[l - 1] ELSE "none", ok |-> ok,
+ route |-> [n \in Nodes |-> st[n].rt["root"]],
+ shown |-> [n \in Nodes |-> {<> :
+ c \in {x \in Claims : st[n].shown[x] # NotShown}}],
+ doubt |-> [n \in Nodes |-> st[n].doubt], dead |-> [n \in Nodes |-> st[n].dead],
+ q |-> [d \in {x \in Dirs : q[x] # <<>>} |-> q[d]]]
+
+\* Every step matched.
+Matched == l > Len(Steps) /\ UNCHANGED traceVars
+
+TraceSpec == TraceInit /\ [][TraceNext \/ Matched]_traceVars
+
+=============================================================================
diff --git a/sh/tla-trace.py b/sh/tla-trace.py
new file mode 100644
index 00000000000..0a7b8ced0c2
--- /dev/null
+++ b/sh/tla-trace.py
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+"""Check recorded traces against a TLA+ model: `sh/tla-trace.py ...`.
+
+Trace validation: a test records each run as newline-delimited JSON, and a trace
+spec beside the model (`verify//trace/.tla`, with `.cfg`)
+follows one trace, named by the environment variable TRACE, as a behaviour of the
+model it EXTENDS. TLC reports a deadlock where the run leaves the model, so each
+trace must pass with TLC's deadlock check on.
+
+The trace spec and the model's modules are linked into one directory under the
+checkout's `.tmp/`, since TLC finds a module only beside the spec. Each trace is
+one TLC run on one worker through sh/tlc.py; four run at once. A trace that
+passes is removed, so what is left is what failed.
+"""
+
+import os
+import subprocess
+import sys
+import tempfile
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+JOBS = 4
+
+
+def check(spec, cfg, trace):
+ env = dict(os.environ, TRACE=str(trace.resolve()))
+ command = [sys.executable, str(ROOT / "sh/tlc.py"), str(spec), "-workers", "1", "-config", str(cfg)]
+ run = subprocess.run(command, cwd=ROOT, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+ return trace, run.returncode, run.stdout
+
+
+def main(argv):
+ if len(argv) < 2:
+ sys.exit("usage: sh/tla-trace.py ...")
+ spec, traces = Path(argv[0]).resolve(), [Path(trace) for trace in argv[1:]]
+ model = spec.parent.parent
+ (ROOT / ".tmp").mkdir(exist_ok=True)
+ with tempfile.TemporaryDirectory(dir=ROOT / ".tmp", prefix="tla-trace-") as run:
+ run = Path(run)
+ for module in [*model.glob("*.tla"), spec]:
+ (run / module.name).symlink_to(module)
+ cfg = spec.with_suffix(".cfg")
+ failed = []
+ with ThreadPoolExecutor(JOBS) as pool:
+ for trace, status, out in pool.map(lambda trace: check(run / spec.name, cfg, trace), traces):
+ if status == 0:
+ trace.unlink()
+ else:
+ failed.append(trace)
+ print(f"::group::{trace}: exit {status}\n{out}::endgroup::", flush=True)
+ print(f"{len(traces) - len(failed)} of {len(traces)} traces are behaviours of the model")
+ if failed:
+ print("error: left the model:", *failed, sep="\n ", file=sys.stderr)
+ return 1
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/sh/tla.sh b/sh/tla.sh
index 742441e63e8..26a426ed3a0 100755
--- a/sh/tla.sh
+++ b/sh/tla.sh
@@ -4,13 +4,14 @@
# a rule (a deadlock, invariant, property, or assertion), so a fix that should
# promote its rule is noticed; every other config must pass. A config with a
# `\* Simulate:` line is too large to exhaust: TLC samples it at random with the
-# arguments that line gives.
+# arguments that line gives. A `trace/` directory holds a trace spec, which
+# checks recorded runs rather than exploring (`sh/tla-trace.py`).
set -euo pipefail
shopt -s nullglob
cd "$(git rev-parse --show-toplevel)"
[ $# -ne 0 ] || set -- rs js
-mapfile -t configs < <(git ls-files --cached --others --exclude-standard -- "$@" | grep -E '^(rs|js)/.*/verify/.*\.cfg$' || true)
+mapfile -t configs < <(git ls-files --cached --others --exclude-standard -- "$@" | grep -E '^(rs|js)/.*/verify/.*\.cfg$' | grep -v '/trace/' || true)
if [ ${#configs[@]} -eq 0 ]; then
echo "error: no TLA+ config under: $*" >&2
exit 1
diff --git a/test/justfile b/test/justfile
index 23bb2a28c40..1a4a83db763 100644
--- a/test/justfile
+++ b/test/justfile
@@ -310,6 +310,10 @@ tla *paths:
route-key-model:
bash route-key-model.sh
+# Check that the router's handoff sweeps, from a fresh replayable window, are behaviours of the route model.
+route-trace:
+ bash ../sh/local.sh bash route-trace.sh
+
# Check random lite-07 meshes against the route rules from a fresh replayable seed.
route-sweep:
bash ../sh/local.sh bash route-sweep.sh
diff --git a/test/route-trace.sh b/test/route-trace.sh
new file mode 100755
index 00000000000..11aae5a5ae1
--- /dev/null
+++ b/test/route-trace.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# Trace validation of the route model: record the router's handoff sweeps under the
+# path rule and check with TLC that each run is a behaviour of RouteHandoff.tla
+# (rs/moq-net/verify/route/trace/). Nightly traces a fresh window of the sweeps'
+# scenarios; the printed settings replay it.
+set -euo pipefail
+cd "$(git rev-parse --show-toplevel)"
+
+MOQ_ROUTE_TRACE_FROM=${MOQ_ROUTE_TRACE_FROM:-$(python3 -c 'import secrets; print(secrets.randbelow(20000))')}
+MOQ_ROUTE_SCENARIOS=${MOQ_ROUTE_SCENARIOS:-$((MOQ_ROUTE_TRACE_FROM + 150))}
+export MOQ_ROUTE_TRACE_FROM MOQ_ROUTE_SCENARIOS
+echo "Route traces: MOQ_ROUTE_TRACE_FROM=$MOQ_ROUTE_TRACE_FROM MOQ_ROUTE_SCENARIOS=$MOQ_ROUTE_SCENARIOS"
+
+mkdir -p .tmp
+MOQ_ROUTE_TRACE=$(mktemp -d "$PWD/.tmp/route-trace.XXXXXX")
+export MOQ_ROUTE_TRACE
+trap 'rm -rf "$MOQ_ROUTE_TRACE"' EXIT
+# Release keeps a window far into the sweeps inside nextest's hang budget.
+export CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true
+just rs test -P sweep -p moq-net --release -E 'test(/^route::tests::handoff::path::(sweep_handoff|sweep_scoped_handoff)$/)'
+python3 sh/tla-trace.py rs/moq-net/verify/route/trace/RouteTrace.tla "$MOQ_ROUTE_TRACE"/*.ndjson