From 1d1e37d5e65e3ad850c6d812daa579b571954034 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 03:13:57 -0800 Subject: [PATCH 1/4] chore(quest): claim quest/m1/route-trace-validation.md Quest-Claim: cd6ce8f06d2b131e From d9294c5054dc267fd76d7f797a5bcf78c33f919e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 04:51:13 -0800 Subject: [PATCH 2/4] test(moq-net): check route sweep runs against RouteHandoff.tla Record sweep_handoff and sweep_scoped_handoff under the path rule as traces of RouteHandoff.tla and check each with TLC (RouteTrace.tla, sh/tla-trace.py, test route-trace). The model grows where the router had drifted from it: a route goes to a stream only while a claim meets it, timers due together fire in one update, and an ended stream forgets what it reflected. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-route/SKILL.md | 18 +- .github/workflows/nightly.yml | 5 +- quest/m2/cluster-routes/stable-repath.md | 40 +++ rs/moq-net/src/route/path.rs | 20 +- rs/moq-net/src/route/tests/handoff.rs | 88 +++-- rs/moq-net/src/route/tests/mod.rs | 52 ++- rs/moq-net/src/route/tests/trace.rs | 331 +++++++++++++++++++ rs/moq-net/verify/README.md | 9 + rs/moq-net/verify/route/RouteHandoff.tla | 111 ++++--- rs/moq-net/verify/route/trace/RouteTrace.cfg | 37 +++ rs/moq-net/verify/route/trace/RouteTrace.tla | 143 ++++++++ sh/tla-trace.py | 61 ++++ sh/tla.sh | 5 +- test/justfile | 4 + test/route-trace.sh | 21 ++ 15 files changed, 860 insertions(+), 85 deletions(-) create mode 100644 quest/m2/cluster-routes/stable-repath.md create mode 100644 rs/moq-net/src/route/tests/trace.rs create mode 100644 rs/moq-net/verify/route/trace/RouteTrace.cfg create mode 100644 rs/moq-net/verify/route/trace/RouteTrace.tla create mode 100644 sh/tla-trace.py create mode 100755 test/route-trace.sh diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index c26aa3e6551..d7a35204be9 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -5,7 +5,7 @@ description: Model-check the lite-07 route rule (`rs/moq-net/verify/route/RouteH # Verify the route rule -`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. The node timer is `Lift`; each dead hop also expires independently through `Expire`, so repeated failures of one node cannot retain all its old hops. Either may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. +`rs/moq-net/verify/route/RouteHandoff.tla` models one publisher's routes across a mesh, under the rule the routes epic's Plan item 27 picked: a route names every relay it crossed with the session it learned it on, a closed session is named in one DEAD message per neighbour, a relay with no route holds until those that learned it from the relay answer, and only along the path it was built on, so holds never close a cycle, and a closed session casts doubt on its far end until a timer fires. A relay ranks routes by drain, then the lowest link priority on the path, then cost; an offer carries its drain, so no relay ranks a path above what its neighbour did. A relay that hears GOAWAY sends a DRAIN naming the leaving node, the DEAD message's other entry, to each neighbour it sent a route through that node, and each relay doubts that node, so none takes a stale offer through it. A draining path ranks worst but stays usable. It follows `route::path::Router` in `rs/moq-net/src/route/path.rs` with one publisher and a route per prefix (`"root"`, and `"a"` inside it, which relays in `Scoped` ask for alone), and keeps dead hops until their timer fires. Every entry point is one atomic step ending in `Update`; messages share one FIFO per direction. A route goes to a stream only while a claim meets it: one that meets none withdraws the claims sent and ends. `Timeout` fires the timers due together on one node in one update, as `Router::timeout` does: doubts lift, and each dead hop also expires on its own timer, so repeated failures of one node cannot retain all its old hops. A timer may fire at any moment, or with `Patient` only once no relevant DEAD is in flight. Sessions still run Babel's distance in `route::Router`: only `route::tests::Sim` drives the path router, under `Rule::PATH` (timer-only doubt), until [the switch](/quest/m2/cluster-routes/switch.md) puts it on the wire. ## Rules @@ -83,13 +83,19 @@ change; faster checkers are separate work. Run the affected config last, with at most one TLC process at a time. First run what answers in seconds: the router's tests (`quest gate just rs test -p moq-net --lib -E 'test(/route::tests::(handoff|residual|scope|compare)/)'`, with `MOQ_ROUTE_SCENARIOS` raising the sweeps' count), then TLC's simulation mode with raised bounds in a copy of a config, such as `-simulate num=3000 -depth 150`. +## Trace validation + +`quest gate just test route-trace` checks that the router's own runs are behaviours of the model, as Trace validation in `rs/moq-net/verify/README.md` describes: `route::tests::Sim` records 150 runs each of `sweep_handoff` and `sweep_scoped_handoff` under `Rule::PATH` (`rs/moq-net/src/route/tests/trace.rs`), and `trace/RouteTrace.tla` follows each with every message and every touched relay's route and shown claims compared, checking the model's rules too except `Stable` (`quest/m2/cluster-routes/stable-repath.md`). Nightly picks a fresh window of scenarios (`MOQ_ROUTE_TRACE_FROM`); the printed settings replay it. Each sweep stays inside the model: one publisher whose every claim change changes a claim, relays asking for the root, no restart, collection only as a step the model does not see. A leaf that publishes nothing, the client, offers no route, so the traces drop it, its grant, and its interest edits; `the_rules_agree_on_every_script` keeps the full script, with a second origin, restarts, and collection. Costs come from the mesh (`Cost <- TCost`), and the timer is impatient (`Patient = FALSE`), since the sweeps hold messages past it. + +A failure leaves its trace in the `.tmp/route-trace.*` directory the run names: a deadlock is a step the model cannot take, and `Agrees` breaks where its messages or state differ, in a state whose `step` names what the router did and whose `route`, `shown`, and `q` show what the model did. Fix the side that is wrong, as Reading a failure says; a broken rule on an accepted trace is a finding in the rule. + ## Reading a failure -Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Lift(n, x)` fires one timer on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until that doubt lifts, and where another doubt lifted with it, the replay took a different path, which is no defect in the model. +Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path on the router with `Sim::with(Rule::PATH, nodes)` (`deliver`, `disconnect`, `connect`, `announce`, `wait`), one call per step, except a timer. `Timeout(n, X, H)` fires timers on one relay, where `wait(HOP)` moves every relay's clock by one hop of delay: call it until those timers fire, and where others fired with them, the replay took a different path, which is no defect in the model. ## Coverage -Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped path router in [moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622). Run `git log 31b215b11.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Lift`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `12537d1c2272c39db154fbd210bf60b8f438da4d`, the merge of the scoped path router ([moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622)), which trace validation checked the model against ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)). Run `git log 12537d1c2.. -- rs/moq-net/src/route/path.rs rs/moq-net/src/route/reader.rs` to see what changed since; `test route-trace` fails when a change there takes a step the model does not. Prefix and reader indexes make the router's updates incremental; test-only full recomputation checks their result against the complete view. The model recomputes every prefix atomically and leaves that optimization out. `Rank` orders routes as `Offer::rank` does: drain, carried with each offer or known from a DRAIN (`Drains`), then the lowest link priority on the path (`Preferred`), then cost; `GoAway` marks a neighbour's session draining. `SrcKey` mirrors the router's preference for the best-ranked covering live path that does not name its receiver, with the narrower scope breaking ties; `Select` plans live routes and their reflected sources before deciding holds; a held route keeps its last live path, so `SelectKey` holds under a next hop only on the path it was built on and `Want` ends a held stream whose source now has another path; `Advertise1` forwards a newer floor even when the claims did not change. The router keeps doubt until its one-second timer fires (`Lift::Timer`, `Doubt::DEFAULT`); the model's `Collect` leaves it for `Timeout`. The comparison-only `Rule::EARLY` lifts doubt once no heard route names its node and is outside this timer-only model. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | @@ -97,9 +103,9 @@ Sync commit: `31b215b112fb649cc3f63d4de3c76b5e87769a06`, the base of the scoped | `rs/moq-net/src/route/path.rs` `Offer::rank`, `Router::goaway`, `Router::drains`, `Router::report_draining`, `Router::dead` (`Dead::Draining`) | `Rank`, `Below`, `PathPrio`, `CostAt`, `GoAway`, `MarkDrain`, `Credited`, `ReportDrain`, `Recv` | | `rs/moq-net/src/route/path.rs` `Router::pending`, `Router::hopes`, `Router::doubts_path` | `Pending`, `Hopes`, `Doubted` | | `rs/moq-net/src/route/path.rs` `Router::src` | `SrcKey` | -| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::advertise`, `Router::report` | `Want`, `Advertise`, `Report` | +| `rs/moq-net/src/route/path.rs` `Router::want`, `Router::meets`, `Router::advertise`, `Router::end_sent`, `Router::report` | `Want`, `Meets`, `Withdraw`, `Advertise`, `Report`, `Collect` (an ended stream forgets what it reflected) | | `rs/moq-net/src/route/path.rs` `Router::route`, `Router::batch`, `Router::dead`, `Router::fail`, `Router::disconnect`, `Router::connect`, `Router::change` | `Recv`, `Fail`, `Disconnect`, `Connect`, `ChangeClaim` | -| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout` | `Collect`, `Lift`, `Expire`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. | +| `rs/moq-net/src/route/path.rs` `Router::collect_doubt`, `Router::timeout`, `Router::due` | `Collect`, `Timeout`: each dead hop's own timer retires it independently of later failures of the same node; reporting an already retained hop again does not restart its expiry. | | `rs/moq-net/src/route/path.rs` `Router::know`, `Router::shown_now`, `Router::show` | `Know`, `ShownNow` | | `rs/moq-net/src/route/reader.rs` `Reader::apply` | `Events` | | `rs/moq-net/src/route/tests/mod.rs` `Sim::deliver` | `Deliver` | @@ -111,4 +117,4 @@ Concurrent disjoint regions of one permit need the [interest set](/quest/m2/cluster-routes/interest-set.md), including per-prefix SYNCED; neither this model nor the current scoped sweep covers them. -One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. A route is sent on every stream whether or not the publisher has a claim there. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. +One publisher, so routes for several publishers are left out, and so are the copies and copy rank of cache-aware routing, which `rs/moq-net/src/route/tests/paths.rs` prototypes; the DEAD message's per-publisher batching reduces to one route per prefix. Two prefixes, one nested in the other, and a scoped relay asks every neighbour for the narrow one: no wildcards, interest edits, stream generations or cancellation, grants, qualified keys, or hidden claims. Leaves out restarts, priorities other than 0 and 1, a priority that differs between a link's two ends, costs other than 1 and 2, RTT, upstream/down-only links, and the hop limit. Messages come only from honest routers: validation of an unrelated DEAD or its far node, and empty or overlong replacement paths, is outside the model's input domain. Rust fixtures cover those guards and exportable covering-source selection, including keeping a covering hold when every live source is unexportable. Interest-edit fixtures also cover a cancelled stream retaining its pending child replies; the model has no stream cancellation. A claims message carries the sender's whole view for its prefix, equal to applying its changes on a FIFO stream. Every claim change changes a claim: the router's revision bump with no change is outside the model. The publisher ignores what it hears. `Overlap.cfg` also simulates five relays with every session up. No exhaustive pass has more than four relays or two session changes; `Sim`'s sweeps on the path rule (`sweep_handoff`, `sweep_scoped_handoff`, `sweep_scoped_meshes_show_what_the_rule_says`, and `compare.rs`'s churn) cover larger meshes, grants, qualified keys, interest edits, restarts, and churn. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index e2ac69a1828..8a1fe21c5fc 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -359,12 +359,13 @@ jobs: # viewer: diff-independent, since a break can arrive through the relay, # js/net, or the wire between them. # `rs quic-sim` sweeps moq-quic's simulated workloads from a fresh seed; PRs run a fixed 64. + # It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases. # `rs rung-disable` proves FFI simulcast disable and recovery on a shaped BBR3 uplink. # `test resume-model` sweeps resume against its model from a fresh seed; PRs run a fixed 64. # `test route-key-model` proves the route key proof with Tamarin, and that each variant breaks what it names. # `test route-sweep` checks random lite-07 meshes against the route rules from a fresh seed; PRs run a fixed 8. # It also runs the router's handoff sweeps over 20,000 fixed scenarios; PRs run 150. - # It also runs the bandwidth probe's acceptance suite, of which PRs run the short cases. + # `test route-trace` checks with TLC that 150 runs of those sweeps, from a fresh window, are behaviours of the route model. # `test drill impaired` runs the relay drills over a freshly seeded shaper; PRs run loopback. # `rs cli-export` runs moq-cli's export process tests, which take tens of seconds of real time. # `rs hls-conformance` probes every segment moq-hls serves with ffprobe, against its playlists and manifest. @@ -379,7 +380,7 @@ jobs: # `rs v4l2` streams moq-video's V4L2 capture from the kernel's vivid # virtual camera, a module load no pull request job should make. # `js worklet-matrix` loads `?worklet` output through the bundlers pinned in test/worklet. - recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "test route-key-model", "test route-sweep", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"] + recipe: ["gate", "gh test-integration", "rs doctest --workspace", "rs send-depth", "test drill-sensitivity", "rs uring", "test drain", "test json-properties", "test resume-model", "test route-key-model", "test route-sweep", "test route-trace", "rs quic-sim", "rs rung-disable", "test drill impaired", "rs cli-export", "rs hls-conformance", "rs quic-stress", "rs audio-devices", "rs multicast", "rs render-sw", "test qmux", "rs v4l2", "js worklet-matrix"] include: # Miri interprets kio's and moq-net's in-memory unsafe; too slow for the merge gate. - recipe: rs miri diff --git a/quest/m2/cluster-routes/stable-repath.md b/quest/m2/cluster-routes/stable-repath.md new file mode 100644 index 00000000000..c928e968a8d --- /dev/null +++ b/quest/m2/cluster-routes/stable-repath.md @@ -0,0 +1,40 @@ +# [M] A relay keeps its claims when its next hop re-paths through a dead hop + +## Goal + +A relay whose next hop sends a new path through a session the relay already +knows has closed keeps showing its claims, unavailable, until that next hop +answers, instead of dropping them offline on that one message. The route +model's `Stable` holds on every run of the router's sweeps, and +`RouteTrace.cfg` checks it again. + +## Plan + +Found by trace validation ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)): +in `sweep_handoff` scenario 139 (`MOQ_ROUTE_TRACE= just rs test -p moq-net --lib -E 'test(=route::tests::handoff::path::sweep_handoff)'`, +then `python3 sh/tla-trace.py rs/moq-net/verify/route/trace/RouteTrace.tla /handoff-139.ndjson` +with `Stable` back in `RouteTrace.cfg`), the router and the model take the same +steps, and at step 87 relay 4 drops two available claims +offline. Relay 4 routes through relay 0 and has already heard the DEAD for +relay 2's closed session to the publisher. Relay 0 has not, and re-paths +through relay 2's closed session. The new offer is dead at relay 4, so it has no +usable route. It holds nothing, because its next hop's path is no longer the +one its route was built on, and it has no child or doubted offer to wait for. +The claims come back once relay 0 hears the DEAD and moves. That is the flap +`Stable` rules out. It needs six nodes and a message order no config reaches. + +Recommended: a relay holds under its next hop while that hop's offer is dead, +as it does while the hop holds the path the route was built on, since the DEAD +the relay heard is on its way to that hop (`StaleIsChased`). Check that holds +still never close a cycle (`HoldCycle.cfg`, `QuietIsSettled`), add a config +that reaches the flap, and change the router and model together. + +Public API: none. Wire: none. + +## Waiting + +- Maintainer: hold under a next hop whose new path is dead (recommended, as + above), or accept the flap and narrow `Stable` to exempt a path change that + makes the route dead, which keeps the rule as it is but lets a stale offer + flap what a reader sees until the DEAD catches up. Decide in the pick round + for moq-dev/moq#8733. diff --git a/rs/moq-net/src/route/path.rs b/rs/moq-net/src/route/path.rs index 0c318bfa13c..5aae30c42bc 100644 --- a/rs/moq-net/src/route/path.rs +++ b/rs/moq-net/src/route/path.rs @@ -1048,18 +1048,24 @@ impl Router { self.update(ids, Vec::new(), None); } - /// Fire every doubt and dead-hop timer due by `now`. - pub fn timeout(&mut self, now: Instant) { - let due: Vec = self + /// The doubts and dead hops whose timers are due by `now`. + pub fn due(&self, now: Instant) -> (Vec, Vec) { + let due = self .doubted .iter() .filter_map(|(node, state)| (state.deadline() <= now).then_some(*node)) .collect(); - let expired: Vec = self + let expired = self .dead .iter() .filter_map(|(hop, state)| state.deadline().filter(|at| *at <= now).map(|_| *hop)) .collect(); + (due, expired) + } + + /// Fire every doubt and dead-hop timer due by `now`. + pub fn timeout(&mut self, now: Instant) { + let (due, expired) = self.due(now); if due.is_empty() && expired.is_empty() { return; } @@ -1099,6 +1105,12 @@ impl Router { // --- Queries --- + /// How this node numbers its session to `neighbor` in the hops it names. + #[cfg(test)] + pub fn session(&self, neighbor: N) -> Option { + self.sessions.get(&neighbor).map(|session| session.number) + } + /// The route this node uses for `id` under `prefix`. pub fn route_for(&self, id: Id, prefix: &Path) -> Option<&Route> { self.routes.get(&(id, prefix.clone())) diff --git a/rs/moq-net/src/route/tests/handoff.rs b/rs/moq-net/src/route/tests/handoff.rs index 1d1ddcebebf..25a9ee40309 100644 --- a/rs/moq-net/src/route/tests/handoff.rs +++ b/rs/moq-net/src/route/tests/handoff.rs @@ -1147,7 +1147,9 @@ fn announcement_cost_follows_sessions_and_live_claims(rule: Rule) { /// grant under one subject, so both feed one qualified view. Under churn every /// view stays some revision of the origin's truth, the widening guard stays /// bounded, no claim ends while the origin still has it, and once churn stops -/// the client shows exactly the claims its interest and the scopes admit. +/// the client shows exactly the claims its interest and the scopes admit. Each +/// claim change changes a claim, as the route model's do, so every run under the +/// path rule is a trace `RouteTrace.tla` checks (`MOQ_ROUTE_TRACE`). fn sweep_scoped_handoff(rule: Rule) { const CLAIMS: [&str; 5] = ["live/a/x", "live/a/y/", "live/b/z", "live/c", "live/"]; const WATCHES: [&str; 5] = ["", "live/", "live/a/", "live/b/z", "live/c"]; @@ -1172,13 +1174,16 @@ fn sweep_scoped_handoff(rule: Rule) { let links = [(origin, r1), (origin, r2), (r1, r2), (client, r1), (client, r2)]; sim.originate(origin, 7, 0); assert!(sim.run()); + sim.trace("scoped", scenario, origin, &CLAIMS); + let mut seed = 0; for _ in 0..40 { match rng.next(8) { 0..=2 => { let claim = CLAIMS[rng.next(CLAIMS.len() as u64) as usize]; if rng.chance(60) { - sim.announce(origin, claim, rng.next(3), None); - } else { + seed += 1; + sim.announce(origin, claim, seed, None); + } else if sim.truth(7, claim, sim.revision(7)).is_some() { sim.unannounce(origin, claim); } } @@ -1223,6 +1228,7 @@ fn sweep_scoped_handoff(rule: Rule) { } } assert!(sim.run()); + sim.finish_trace(); sim.check().unwrap_or_else(|err| panic!("scenario {scenario}: {err}")); let watch = sim.watching(client).clone(); let rev = sim.revision(7); @@ -1262,16 +1268,17 @@ fn sweep_scoped_handoff(rule: Rule) { const RELAYS: usize = 5; const CLAIMS: [&str; 4] = ["live/", "live/a/", "live/a/x", "live/b"]; -/// Random meshes with two origins and a client under churn: claims change, -/// sessions drop and return, relays send GOAWAY, messages are held and replayed -/// out of step, the client changes interest, nodes collect route-less -/// identities, and an origin restarts with a fresh identity. Selected routes -/// never loop, every view is always some revision of the origin's truth, -/// nothing revives and nothing ends falsely, and once churn stops every node -/// shows every live origin's current claims and holds nothing for a dead one. +/// A random mesh under churn, as the route model has it: one origin's claims +/// change, sessions drop and return, relays send GOAWAY, messages are held and +/// replayed out of step, and a client changes interest. Selected routes never +/// loop, every view is always some revision of the origin's truth, nothing +/// revives and nothing ends falsely, and once churn stops every node shows the +/// origin's current claims. Under the path rule every run is a trace +/// `RouteTrace.tla` checks (`MOQ_ROUTE_TRACE`); `the_rules_agree_on_every_script` +/// runs the full script, with a second origin, restarts, and collection. fn sweep_handoff(rule: Rule) { for scenario in 0..scenarios() { - let mut sim = churn(rule, scenario); + let mut sim = churn(rule, scenario, Shape::Model); let label = format!("scenario {scenario}"); assert_eq!(revivals(&sim.events), 0, "{label}"); assert_eq!(false_ends(&sim.events), 0, "{label}"); @@ -1318,7 +1325,7 @@ fn the_rules_agree_on_every_script() { for scenario in 0..scenarios() { let label = format!("scenario {scenario}"); let sims = [Rule::Distance, Rule::PATH].map(|rule| { - let mut sim = churn(rule, scenario); + let mut sim = churn(rule, scenario, Shape::Full); assert_eq!(revivals(&sim.events), 0, "{label}: {rule:?}"); assert_eq!(false_ends(&sim.events), 0, "{label}: {rule:?}"); settled(&mut sim, &label); @@ -1330,10 +1337,22 @@ fn the_rules_agree_on_every_script() { } } +/// What a [`churn`] script does. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Shape { + /// Two origins, either of which restarts with a fresh identity, and nodes + /// collect route-less identities. + Full, + /// What the route model has: one origin, whose every claim change changes a + /// claim, and no restart. The second origin's node is a leaf with nothing to + /// publish. The run is recorded as a trace when `MOQ_ROUTE_TRACE` is set. + Model, +} + /// Origins sit at `RELAYS` and `RELAYS + 1`, the client at `RELAYS + 2`. The script, -/// every change made from outside, depends on `scenario` alone, so both rules run -/// the same one; which queued message arrives next is drawn apart from it. -fn churn(rule: Rule, scenario: usize) -> Sim { +/// every change made from outside, depends on `scenario` and `shape` alone, so both +/// rules run the same one; which queued message arrives next is drawn apart from it. +fn churn(rule: Rule, scenario: usize, shape: Shape) -> Sim { let rng = &mut XorShift(0x4a4d_0ff5 ^ (scenario as u64 + 1).wrapping_mul(0x9e37_79b9_7f4a_7c15) | 1); let mut pick = XorShift(0x5eed_0003 + scenario as u64); let (first, client) = (RELAYS, RELAYS + 2); @@ -1358,18 +1377,32 @@ fn churn(rule: Rule, scenario: usize) -> Sim { } let mut ids = [100, 200]; sim.originate(first, ids[0], 0); - sim.originate(first + 1, ids[1], 0); + if shape == Shape::Full { + sim.originate(first + 1, ids[1], 0); + } assert!(sim.run()); + sim.trace("handoff", scenario, first, &CLAIMS); + let mut seed = 0; for _ in 0..40 { match rng.next(10) { 0..=3 => { - let origin = rng.next(2) as usize; + let origin = match shape { + Shape::Full => rng.next(2) as usize, + Shape::Model => 0, + }; let claim = CLAIMS[rng.next(CLAIMS.len() as u64) as usize]; - if rng.chance(50) { - sim.announce(first + origin, claim, rng.next(3), None); - } else { - sim.unannounce(first + origin, claim); + let announce = rng.chance(50); + let id = ids[origin]; + match shape { + Shape::Full if announce => sim.announce(first + origin, claim, rng.next(3), None), + Shape::Full => sim.unannounce(first + origin, claim), + Shape::Model if announce => { + seed += 1; + sim.announce(first, claim, seed, None); + } + Shape::Model if sim.truth(id, claim, sim.revision(id)).is_some() => sim.unannounce(first, claim), + Shape::Model => {} } } 4 if rng.chance(20) => sim.goaway(rng.next(RELAYS as u64) as usize), @@ -1393,6 +1426,7 @@ fn churn(rule: Rule, scenario: usize) -> Sim { 6 => { sim.watch(client, ["", "live/a/", "live/b"][rng.next(3) as usize]); } + 7 if shape == Shape::Model => {} 7 if rng.chance(50) => { // Collection is allowed once a node holds no route to an identity. let node = rng.next(RELAYS as u64 + 3) as usize; @@ -1438,6 +1472,7 @@ fn churn(rule: Rule, scenario: usize) -> Sim { } } assert!(sim.run()); + sim.finish_trace(); sim } @@ -1446,10 +1481,12 @@ fn churn(rule: Rule, scenario: usize) -> Sim { /// for an identity whose process is gone. fn settled(sim: &mut Sim, label: &str) { sim.check().unwrap(); - let live: Vec = (RELAYS..RELAYS + 2).map(|origin| sim.proven(origin)).collect(); + let live: Vec<(Node, Id)> = (RELAYS..RELAYS + 2) + .filter_map(|origin| Some((origin, sim.nodes[origin].declared()?))) + .collect(); for node in 0..RELAYS + 3 { let mut expected = BTreeMap::new(); - for (origin, &id) in (RELAYS..).zip(&live) { + for &(origin, id) in &live { let presented = sim.presented(id, sim.watching(node), sim.revision(id)); // A route reaches a node only while a claim meets its interest. assert_eq!( @@ -1480,7 +1517,10 @@ fn settled(sim: &mut Sim, label: &str) { "{label}: node {node} {id} {claim}" ); } - for id in (100..110).chain(200..210).filter(|id| !live.contains(id)) { + for id in (100..110) + .chain(200..210) + .filter(|id| live.iter().all(|(_, live)| live != id)) + { assert!(!sim.holds(node, id), "{label}: node {node} holds dead {id}"); assert!( sim.paths() || !sim.remembers(node, id), diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 85f7f5d9667..d6430bfb29f 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -17,6 +17,7 @@ mod residual; mod routing; mod rtt; mod scope; +mod trace; use std::{ collections::{BTreeMap, BTreeSet, VecDeque}, @@ -174,7 +175,7 @@ impl Relay { /// A claim's attributes, plus the revision its life began at so fixtures can /// tell a revival from a fresh announcement. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] pub(super) struct Item { pub seed: Metric, pub epoch: Option, @@ -365,6 +366,12 @@ pub(super) struct Sim { /// The work of routers a restart replaced. spent: usize, pub events: Vec, + /// Under the path rule, each node's session number for a hop, as the neighbour and + /// the link's generation: how many times it opened before. + numbers: BTreeMap<(Node, u64), (Node, u32)>, + generations: BTreeMap<(Node, Node), u32>, + /// The run being recorded as a trace of `RouteHandoff.tla`. + trace: Option, } impl Sim { @@ -398,6 +405,9 @@ impl Sim { stats: Stats::default(), spent: 0, events: Vec::new(), + numbers: BTreeMap::new(), + generations: BTreeMap::new(), + trace: None, } } @@ -475,6 +485,13 @@ impl Sim { pub fn connect(&mut self, a: Node, b: Node) { assert!(!self.connected(a, b), "{a}-{b} already up"); self.up.insert(Self::pair(a, b)); + let generation = match self.generations.get_mut(&Self::pair(a, b)) { + Some(generation) => { + *generation += 1; + *generation + } + None => *self.generations.entry(Self::pair(a, b)).or_default(), + }; for (x, y) in [(a, b), (b, a)] { let link = self.config(x, y); let named = hop(y, self.restarts[y]); @@ -483,11 +500,16 @@ impl Sim { Relay::Path(router) => router.connect(y, named, link), } .expect("a positive cost"); + if let Relay::Path(router) = &self.nodes[x] { + let number = router.session(y).expect("a session"); + self.numbers.insert((x, number), (y, generation)); + } } for (x, y) in [(a, b), (b, a)] { self.ask(y, x); } self.flush(); + self.note_link("connect", a, b); } /// `node` asks `peer` for announcements under its interest. Under the path rule a @@ -534,10 +556,12 @@ impl Sim { } } self.flush(); + self.note_link("disconnect", a, b); } /// Every session of `node` closes at once, as a node shutting down closes them. pub fn isolate(&mut self, node: Node) { + self.untraced("isolate"); let peers: Vec = self.peers(node).collect(); for &peer in &peers { self.up.remove(&Self::pair(node, peer)); @@ -560,6 +584,7 @@ impl Sim { /// The process at `node` exits: its sessions close and its state is lost. pub fn kill(&mut self, node: Node) { + self.untraced("kill"); let peers: Vec = self.peers(node).collect(); for peer in peers { self.disconnect(node, peer); @@ -671,9 +696,14 @@ impl Sim { let id = router.declared().unwrap(); let rev = router.revision().unwrap(); if self.declared[node] == Some(id) { - self.history.entry(id).or_default().push(Change { rev, claim, item }); + self.history.entry(id).or_default().push(Change { + rev, + claim: claim.clone(), + item, + }); } self.flush(); + self.note_claim(node, claim, item, rev); } /// The prefix `node` asks its neighbors for announcements under; each @@ -690,6 +720,8 @@ impl Sim { self.ask(node, peer); } self.flush(); + let peers: Vec = self.peers(node).collect(); + self.note_skip(&peers); } pub fn watching(&self, node: Node) -> &Path { @@ -704,8 +736,10 @@ impl Sim { Relay::Distance(router) => router.goaway(node), Relay::Path(router) => router.goaway(node, now), } + // One step per node that hears it, as the model takes them. + self.flush(); + self.note_goaway(peer, node); } - self.flush(); } pub fn collect(&mut self, node: Node, id: Id) { @@ -714,6 +748,7 @@ impl Sim { Relay::Path(router) => router.collect(id), } self.flush(); + self.note_skip(&[node]); } /// `node` measures its link to `peer` at `ms`, or nothing. @@ -749,6 +784,7 @@ impl Sim { let Some(msg) = self.queues.get_mut(&(from, to)).and_then(VecDeque::pop_front) else { return false; }; + let modeled = !matches!(msg, Msg::Synced { .. }); let now = self.now; match (&mut self.nodes[to], msg) { (Relay::Distance(router), Msg::Route { id, reach }) => router.route(from, id, reach), @@ -787,6 +823,7 @@ impl Sim { } self.stats.delivered += 1; self.flush(); + self.note_deliver(from, to, modeled); true } @@ -889,11 +926,15 @@ impl Sim { /// Fire every doubt timer due by now. fn tick(&mut self) { let now = self.now; - for relay in &mut self.nodes { - if let Relay::Path(router) = relay + for node in 0..self.nodes.len() { + if let Relay::Path(router) = &mut self.nodes[node] && router.deadline().is_some_and(|at| at <= now) { + let due = router.due(now); router.timeout(now); + // One step per node, as the model takes them. + self.flush(); + self.note_timeout(node, due); } } self.flush(); @@ -1051,6 +1092,7 @@ impl Sim { (to, Msg::Synced { interest }) } }; + self.note_sent(node, to, &msg); self.queues.entry((node, to)).or_default().push_back(msg); } } diff --git a/rs/moq-net/src/route/tests/trace.rs b/rs/moq-net/src/route/tests/trace.rs new file mode 100644 index 00000000000..db9cc968cd6 --- /dev/null +++ b/rs/moq-net/src/route/tests/trace.rs @@ -0,0 +1,331 @@ +//! Records a [`Sim`] run under the path rule as a trace of +//! `rs/moq-net/verify/route/RouteHandoff.tla`, for TLC to check that the run is a +//! behaviour of the model (`rs/moq-net/verify/route/trace/RouteTrace.tla`). +//! +//! A trace is newline-delimited JSON. The first line names the model's constants: +//! the nodes it has, the publisher, the links with their costs, and the claims. +//! Every later line is one step: the model action it is (`op`), the messages it sent +//! between modelled nodes, and what each node it touched now routes and shows. +//! +//! The model has one publisher and relays that pass routes on, so a trace keeps +//! only those nodes. A leaf that publishes nothing, like a client, offers no route, +//! so no path crosses it and nothing it does reaches another node's routes: the +//! trace drops it, and steps that only it takes are `skip`, a step the model does not +//! see, which must send nothing between modelled nodes. Messages to the publisher are +//! dropped too, as the model drops them: it ignores what it hears. +//! +//! [`Sim::trace`] starts recording when `MOQ_ROUTE_TRACE` names a directory, and +//! [`Sim::finish_trace`] writes the trace there. `MOQ_ROUTE_TRACE_FROM` skips the +//! scenarios before it, so a run can trace a window of a long sweep. +//! `test/route-trace.sh` records the sweeps and has TLC check every trace. + +use std::path::PathBuf; + +use serde_json::{Value, json}; + +use super::*; + +/// A run being recorded. +pub(super) struct Recorder { + file: PathBuf, + /// The publisher's identity, and the node publishing it. + id: Id, + origin: Node, + /// The nodes the trace keeps: the publisher and every relay. + nodes: BTreeSet, + claims: BTreeSet, + /// The model's attributes for each item announced: the revision that set it. + items: BTreeMap<(Path, Item), Rev>, + /// The messages sent since the last step. + sent: Vec, + steps: Vec, +} + +impl Sim { + /// Record the rest of this run as `-` if `MOQ_ROUTE_TRACE` names a + /// directory. `origin` publishes the one identity, and `claims` are the claims it + /// may make: the trace starts here, as the model's initial state does, with every + /// configured link open, the publisher's identity declared with no claims, and + /// nothing in flight. + pub fn trace(&mut self, sweep: &str, scenario: usize, origin: Node, claims: &[&str]) { + let Some(dir) = std::env::var_os("MOQ_ROUTE_TRACE") else { + return; + }; + let from = std::env::var("MOQ_ROUTE_TRACE_FROM").map_or(0, |from| from.parse().expect("MOQ_ROUTE_TRACE_FROM")); + if !self.paths() || self.rule != Rule::PATH || scenario < from { + return; + } + let id = self.proven(origin); + assert!(self.busy().is_empty(), "a trace starts with nothing in flight"); + assert_eq!(self.revision(id), 0, "a trace starts with no claims"); + let nodes = (0..self.nodes.len()) + .filter(|&node| node == origin || !self.leaf[node]) + .collect(); + self.trace = Some(Recorder { + file: PathBuf::from(dir).join(format!("{sweep}-{scenario}.ndjson")), + id, + origin, + nodes, + claims: claims.iter().map(|claim| p(claim)).collect(), + items: BTreeMap::new(), + sent: Vec::new(), + steps: Vec::new(), + }); + for &(a, b) in &self.up { + assert!( + !(self.modelled(a) && self.modelled(b)) || self.generations[&(a, b)] == 0, + "a trace starts on sessions never reopened" + ); + } + } + + /// Write the recorded trace, if any. + pub fn finish_trace(&mut self) { + let Some(trace) = self.trace.take() else { + return; + }; + assert!(trace.sent.is_empty(), "messages sent outside a step"); + let links: Vec = self + .up + .iter() + .filter(|(a, b)| trace.nodes.contains(a) && trace.nodes.contains(b)) + .map(|&(a, b)| { + let cost = self.links[a][b].expect("configured").cost; + assert_eq!(cost, self.links[b][a].expect("configured").cost, "a symmetric cost"); + json!([a, b, cost]) + }) + .collect(); + let header = json!({ + "nodes": trace.nodes, + "origin": trace.origin, + "links": links, + "claims": trace.claims.iter().map(Path::as_str).collect::>(), + }); + let mut out = String::new(); + for line in std::iter::once(&header).chain(&trace.steps) { + out += &line.to_string(); + out.push('\n'); + } + std::fs::write(&trace.file, out).expect("write the trace"); + } + + /// Fail a run that does something the trace cannot express. + pub(super) fn untraced(&self, what: &str) { + assert!(self.trace.is_none(), "a traced run cannot {what}"); + } + + fn modelled(&self, node: Node) -> bool { + self.trace.as_ref().is_some_and(|trace| trace.nodes.contains(&node)) + } + + /// Record a message `from` queued for `to`, if the model has it. + pub(super) fn note_sent(&mut self, from: Node, to: Node, msg: &Msg) { + let Some(trace) = &self.trace else { + return; + }; + if !self.modelled(from) || !self.modelled(to) || to == trace.origin { + return; + } + let key = |id: Id, prefix: &Path| { + assert_eq!(id, trace.id, "one publisher"); + Self::key(prefix) + }; + let msg = match msg { + Msg::Path { id, prefix, reach, .. } => match reach { + Some(path::Reach::Finite(offer)) => json!({ + "t": "offer", + "k": key(*id, prefix), + "path": self.hops(&offer.path), + "drain": offer.draining, + }), + Some(path::Reach::Held) => json!({"t": "hold", "k": key(*id, prefix)}), + None => json!({"t": "end", "k": key(*id, prefix)}), + }, + Msg::Claims { id, prefix, floor, .. } => json!({ + "t": "claims", + "k": key(*id, prefix), + "rev": floor.map_or(-1, |floor| floor as i64), + }), + Msg::Dead(path::Dead::Closed { hop, far, moved, gone }) => { + let named = self.hop_of(*hop); + assert_eq!( + named[1], + Self::node_of(*far) as u64, + "a DEAD's far end is its hop's peer" + ); + let moved: Vec = moved + .iter() + .map(|moved| { + json!([ + key(moved.id, &moved.prefix), + self.hops(&moved.offer.path), + moved.offer.draining + ]) + }) + .collect(); + json!({"t": "dead", "hop": named, "moved": moved, "gone": gone}) + } + Msg::Dead(path::Dead::Draining { node }) => json!({"t": "drain", "node": Self::node_of(*node)}), + Msg::Synced { .. } => return, + msg => panic!("the path rule sent {msg:?}"), + }; + let sent = json!([from, to, msg]); + self.trace.as_mut().unwrap().sent.push(sent); + } + + /// The model's key for a route's prefix: the root, or the prefix itself. + fn key(prefix: &Path) -> String { + match prefix.as_str() { + "" => "root".to_string(), + prefix => prefix.to_string(), + } + } + + /// The node a hop id names, which never restarted in a traced run. + fn node_of(id: Id) -> Node { + assert!((1_000..2_000).contains(&id), "{id} names a node that never restarted"); + (id - 1_000) as Node + } + + /// The model's hop: the node, the neighbour it learned the route from, and that + /// session's generation. A publisher names itself with generation 0. + fn hop_of(&self, hop: path::Hop) -> [u64; 3] { + let node = Self::node_of(hop.node); + if hop.session == 0 { + return [node as u64, node as u64, 0]; + } + let (peer, generation) = self.numbers[&(node, hop.session)]; + [node as u64, peer as u64, u64::from(generation)] + } + + fn hops(&self, path: &[path::Hop]) -> Vec<[u64; 3]> { + path.iter().map(|hop| self.hop_of(*hop)).collect() + } + + /// What `node` routes and shows, as the model keeps it. + fn node_state(&self, node: Node) -> Value { + let trace = self.trace.as_ref().unwrap(); + let router = self.path(node); + let route = match router.route_for(trace.id, &Path::empty()) { + None => json!({"on": false}), + Some(path::Route::Live { next, offer }) => json!({ + "on": true, + "held": false, + "next": next, + "path": self.hops(&offer.path), + "drain": offer.draining, + }), + Some(path::Route::Held { under, path }) => json!({ + "on": true, + "held": true, + "next": under.map_or(-1, |under| under as i64), + "path": self.hops(path), + }), + Some(route) => panic!("node {node} routes {route:?} between updates"), + }; + let others: Vec<&Path> = router + .routes(trace.id) + .map(|(prefix, _)| prefix) + .filter(|prefix| !prefix.is_empty()) + .collect(); + assert!( + others.is_empty(), + "node {node} routes {others:?}, which the model has not" + ); + let shown: Vec = router + .shown() + .iter() + .filter(|((id, _), _)| *id == trace.id) + .map(|((_, claim), shown)| { + let item = trace.items[&(claim.clone(), shown.item)]; + json!([claim.as_str(), item, shown.available]) + }) + .collect(); + json!({"node": node, "route": route, "shown": shown}) + } + + /// Record a step: `op`, the messages sent since the last one, and what each + /// modelled node in `touched` now routes and shows. + fn note(&mut self, mut op: Value, touched: &[Node]) { + let Some(trace) = &self.trace else { + return; + }; + let origin = trace.origin; + let state: Vec = touched + .iter() + .copied() + .filter(|&node| self.modelled(node) && node != origin) + .map(|node| self.node_state(node)) + .collect(); + let trace = self.trace.as_mut().unwrap(); + op["sent"] = Value::Array(std::mem::take(&mut trace.sent)); + op["state"] = Value::Array(state); + trace.steps.push(op); + } + + /// A step the model does not see: it must send nothing between modelled nodes. + pub(super) fn note_skip(&mut self, touched: &[Node]) { + self.note(json!({"op": "skip"}), touched); + } + + pub(super) fn note_link(&mut self, op: &str, a: Node, b: Node) { + match self.modelled(a) && self.modelled(b) { + true => self.note(json!({"op": op, "a": a, "b": b}), &[a, b]), + false => self.note_skip(&[a, b]), + } + } + + pub(super) fn note_deliver(&mut self, from: Node, to: Node, modelled: bool) { + let origin = self.trace.as_ref().map(|trace| trace.origin); + match modelled && self.modelled(from) && self.modelled(to) && Some(to) != origin { + true => self.note(json!({"op": "deliver", "from": from, "to": to}), &[to]), + false => self.note_skip(&[to]), + } + } + + pub(super) fn note_goaway(&mut self, node: Node, from: Node) { + let origin = self.trace.as_ref().map(|trace| trace.origin); + match self.modelled(node) && self.modelled(from) && Some(node) != origin { + true => self.note(json!({"op": "goaway", "node": node, "from": from}), &[node]), + false => self.note_skip(&[node]), + } + } + + pub(super) fn note_timeout(&mut self, node: Node, (lifted, expired): (Vec, Vec)) { + if self.trace.is_none() { + return; + } + let lift: Vec = (lifted.into_iter().map(Self::node_of)) + .filter(|&node| self.modelled(node)) + .collect(); + let expire: Vec<[u64; 3]> = (expired.into_iter().map(|hop| self.hop_of(hop))) + .filter(|hop| self.modelled(hop[0] as Node) && self.modelled(hop[1] as Node)) + .collect(); + // The model's publisher keeps no doubt: it ignores what it hears. + let origin = self.trace.as_ref().map(|trace| trace.origin); + match self.modelled(node) && Some(node) != origin && !(lift.is_empty() && expire.is_empty()) { + true => self.note( + json!({"op": "timeout", "node": node, "lift": lift, "expire": expire}), + &[node], + ), + false => self.note_skip(&[node]), + } + } + + pub(super) fn note_claim(&mut self, node: Node, claim: Path, item: Option, rev: Rev) { + let Some(trace) = &mut self.trace else { + return; + }; + assert_eq!(node, trace.origin, "one publisher"); + assert!( + trace.claims.contains(&claim), + "{claim} is not one of the trace's claims" + ); + if let Some(item) = item { + let was = trace.items.insert((claim.clone(), item), rev); + assert!(was.is_none(), "each announcement in a trace sets a new item"); + } + let op = json!({"op": "claim", "claim": claim.as_str(), "item": item.is_some(), "rev": rev}); + self.note(op, &[]); + } +} diff --git a/rs/moq-net/verify/README.md b/rs/moq-net/verify/README.md index 17bd335594d..eb555d7cede 100644 --- a/rs/moq-net/verify/README.md +++ b/rs/moq-net/verify/README.md @@ -9,3 +9,12 @@ A counterexample is a path through the model, not yet a bug. Replay it on the co ## Coverage After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it. + +## Trace validation + +A model the code has drifted from keeps passing. Trace validation (Cirstea, Kuppe, Loillier, Merz, 2024) checks the code's own runs against it: a test records each run as the model actions it takes, and TLC checks that the run is a behaviour of the spec. The route model has it (`verify-route`); the others follow the same shape. + +- The test writes one trace per run, as newline-delimited JSON, into the directory its environment variable names. The first line gives the constants the run needs, such as the mesh; each later line is one step: `op`, the action and its arguments, then what the step sent and what each node it touched now holds, in the model's terms. +- A trace spec in the model's `trace/` directory EXTENDS the model, reads the trace TLC's environment names (`ndJsonDeserialize(IOEnv.TRACE)`), and takes each step through the model's own action. Its config substitutes the trace's constants (`Nodes <- TNodes`). Each action must be deterministic given its arguments, so TLC follows one state per step. A step the model cannot take is a deadlock; a step whose messages or state differ breaks `Agrees`, in a state its `ALIAS` shows beside the step. +- `sh/tla-trace.py ...` runs TLC on each trace, four at once on one worker each, and keeps only the traces that fail. `sh/tla.sh` skips `trace/` directories. +- Where the code does more than the model, either restrict the recorded runs to what the model has or grow the model; say which, and drop from the trace only what cannot reach the rest of the state. Planting a bug in the code and seeing a trace fail checks that the comparison has teeth. diff --git a/rs/moq-net/verify/route/RouteHandoff.tla b/rs/moq-net/verify/route/RouteHandoff.tla index e5f2ec9ae35..6f4a4d892a2 100644 --- a/rs/moq-net/verify/route/RouteHandoff.tla +++ b/rs/moq-net/verify/route/RouteHandoff.tla @@ -20,7 +20,7 @@ (* neighbour that learned the route from it has answered. *) (* - A closed session casts doubt on the node at its far end: no relay *) (* moves to a route through it until a timer lifts the doubt. The timer *) -(* is the step `Lift`, which may fire at any moment. *) +(* is the step `Timeout`, which may fire at any moment. *) (* *) (* Each node's every entry point is one atomic step ending in `Update`, *) (* as in the router. Messages travel on one FIFO per direction. A *) @@ -93,7 +93,9 @@ NoHeard == [on |-> FALSE, path |-> <<>>, drain |-> FALSE, state |-> "live", rev \* route's, and `next` is the neighbour it holds under, or None on its own account. NoRoute == [on |-> FALSE, held |-> FALSE, next |-> None, path |-> <<>>, drain |-> FALSE] \* What a node last sent a neighbour for a key (`Sent`). `src` is the key of -\* the node's own route the offer reflects, kept after an end. +\* the node's own route the offer reflects, kept after an end while the +\* neighbour's route there may be a child yet to answer; NoSrc once forgotten. +NoSrc == "none" NoSent(k) == [on |-> FALSE, held |-> FALSE, path |-> <<>>, drain |-> FALSE, rev |-> None, items |-> NoItems, src |-> k] NotShown == [item |-> None, avail |-> FALSE] @@ -290,27 +292,47 @@ ShownNow(s) == ----------------------------------------------------------------------------- (* What a node sends (path.rs: `want`, `advertise`, `report`). *) +\* `covering_view`: the keys of the narrowest view at or above k, at most one. +CoverKey(s, k) == IF s.view[k].floor # Absent THEN {k} + ELSE IF s.view["root"].floor # Absent THEN {"root"} + ELSE {} + +\* `meets`: whether this node presents any claim at k. +Meets(s, k) == \E ck \in CoverKey(s, k) : \E c \in Claims : Covers(k, c) /\ s.view[ck].items[c] # None + \* What this node wants a neighbour p to hold for key k: a path, a hold, an \* end, or to keep what it was sent while the next hop this node holds under -\* waits for those behind it. +\* waits for those behind it. A route goes to a stream only while a claim meets +\* it: a path that meets none is "empty", which withdraws the claims sent and +\* then ends the route. Want(n, s, k, p) == LET src == SrcKey(s, p, k) r == s.rt[src] - IN IF ~r.on THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] - ELSE IF ~r.held THEN [kind |-> "path", path |-> r.path, drain |-> r.drain, src |-> src] + End == [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] + IN IF ~r.on THEN End + ELSE IF ~r.held THEN [kind |-> IF Meets(s, k) THEN "path" ELSE "empty", + path |-> r.path, drain |-> r.drain, src |-> src] ELSE IF r.next = p - THEN [kind |-> IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end", - path |-> <<>>, drain |-> FALSE, src |-> src] + THEN [End EXCEPT !.kind = IF Pending(n, s, src) \/ Hopes(n, s, src) THEN "keep" ELSE "end"] \* A hold names no path, so a stream holds only the path it last sent. - ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path - THEN [kind |-> "end", path |-> <<>>, drain |-> FALSE, src |-> src] - ELSE [kind |-> "hold", path |-> <<>>, drain |-> FALSE, src |-> src] + ELSE IF s.tx[p][k].on /\ s.tx[p][k].path # r.path THEN End + ELSE [End EXCEPT !.kind = IF Meets(s, k) THEN "hold" ELSE "end"] + +\* The view a stream for k presents from, empty while there is none. +Presented(s, k) == IF s.view[k].floor = Absent THEN EmptyView ELSE s.view[k] + +\* The batch that withdraws what was sent t for k before an "empty" ends it: +\* sent whenever there is a view, unless it would say nothing at all. +Withdraw(s, k, t) == + LET v == Presented(s, k) + IN IF s.view[k].floor # Absent /\ (v.floor # None \/ t.items # v.items) + THEN <> ELSE <<>> \* `advertise` for one key on the stream to p. Advertise1(n, s, p, k) == LET t == s.tx[p][k] w == Want(n, s, k, p) - v == IF s.view[k].floor = Absent THEN EmptyView ELSE s.view[k] + v == Presented(s, k) IN CASE k \notin Streams(p) \/ w.kind = "keep" -> [tx |-> t, out |-> <<>>] [] w.kind = "hold" -> IF t.on /\ ~t.held THEN [tx |-> [t EXCEPT !.held = TRUE], out |-> <>] @@ -318,6 +340,9 @@ Advertise1(n, s, p, k) == [] w.kind = "end" -> IF t.on THEN [tx |-> NoSent(t.src), out |-> <>] ELSE [tx |-> t, out |-> <<>>] + [] w.kind = "empty" -> + IF t.on THEN [tx |-> NoSent(t.src), out |-> Withdraw(s, k, t) \o <>] + ELSE [tx |-> t, out |-> <<>>] [] OTHER -> LET offer == ~t.on \/ t.held \/ t.path # w.path \/ t.drain # w.drain claims == t.rev # v.floor \/ t.items # v.items @@ -351,17 +376,20 @@ Report1(n, s, p, hop) == \* The neighbour this node learned the route from takes silence to mean \* it holds under it, so it is told outright. parents == {k \in holds : Child(t[k].path, p)} - ends == {k \in through : w[k].kind = "end"} - gone == \A k \in through : w[k].kind \in {"path", "end"} + ends == {k \in through : w[k].kind \in {"end", "empty"}} + gone == \A k \in through : w[k].kind \in {"path", "end", "empty"} first == hop \notin s.told[p] tx1 == [k \in Keys |-> IF k \notin through THEN t[k] ELSE CASE w[k].kind = "path" -> [t[k] EXCEPT !.path = w[k].path, !.drain = w[k].drain, !.held = FALSE, !.src = w[k].src] [] w[k].kind = "hold" -> [t[k] EXCEPT !.held = TRUE] - [] w[k].kind = "end" -> NoSent(t[k].src) + [] w[k].kind \in {"end", "empty"} -> NoSent(t[k].src) [] OTHER -> t[k]] - out == IF first \/ (gone /\ ends # {}) + \* An "empty" route's withdrawal goes out at once, before the news. + withdrawn == Each({k \in through : w[k].kind = "empty"}, LAMBDA k : Withdraw(s, k, t[k])) + out == withdrawn \o + IF first \/ (gone /\ ends # {}) THEN <> \o (IF gone THEN <<>> ELSE Each(parents, LAMBDA k : <>) \o Each(ends, LAMBDA k : <>)) @@ -388,10 +416,15 @@ ReportDrain(n, s, ups, x) == out |-> [p \in Nodes |-> IF p \in tell THEN <> ELSE <<>>]] \* `collect`: forget each dead hop no route names once its timer has fired, -\* and each told hop no sent route names. Doubt lasts until Lift fires. -Collect(s) == +\* and each told hop no sent route names. Doubt lasts until its timer fires. +\* An ended stream forgets what it reflected unless the neighbour's route there +\* extends this node's, as `advertise` drops such an entry. +Collect(n, s) == LET heard == {s.rx[x[1]][x[2]].path : x \in {y \in Nodes \X Keys : s.rx[y[1]][y[2]].on}} + Kept(m, k) == s.tx[m][k].on \/ (s.rx[m][k].on /\ s.rx[m][k].state # "held" /\ Child(s.rx[m][k].path, n)) IN [s EXCEPT + !.tx = [m \in Nodes |-> [k \in Keys |-> + IF Kept(m, k) THEN s.tx[m][k] ELSE NoSent(NoSrc)]], !.dead = {h \in s.dead : h \in s.expiry \/ \E p \in heard : HasHop(p, h)}, !.draining = {x \in s.draining : x \in s.doubt \/ \E p \in heard : Names(p, x)}, !.told = [m \in Nodes |-> {h \in s.told[m] : @@ -438,7 +471,7 @@ Update(n, s0, ov, ups, gens, eb, rep, quiet) == ELSE IF rep.node # None THEN ReportDrain(n, s2, ups, rep.node) ELSE [s |-> s2, out |-> [p \in Nodes |-> <<>>]] a == Advertise(n, r.s, ups) - s3 == Collect(a.s) + s3 == Collect(n, a.s) now == IF n = Origin THEN [c \in Claims |-> NotShown] ELSE ShownNow(s3) ev == Events(s0, s3, s0.shown, now, ov, eb, quiet) IN [s |-> [s3 EXCEPT !.shown = now], @@ -502,7 +535,7 @@ InitNode == [ expiry |-> {}, \* Pending timers belong to individual dead hops. drain |-> {}, \* the neighbours that sent GOAWAY on the session open to them draining |-> {}, \* nodes known to have sent GOAWAY, while named or doubted - tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(k)]], + tx |-> [m \in Nodes |-> [k \in Keys |-> NoSent(NoSrc)]], told |-> [m \in Nodes |-> {}], toldDrain |-> [m \in Nodes |-> {}], shown |-> [c \in Claims |-> NotShown] @@ -617,28 +650,23 @@ Deliver(d) == /\ Internal(Sent(rs)) /\ UNCHANGED <> -\* The timer node n started on x fires: the doubt on x lifts, and the dead hops -\* whose session ended at x may be forgotten. With `Patient`, only once no DEAD -\* about such a hop, and no DRAIN of x, is in flight: a timer long enough for -\* the news to cross the mesh. -Lift(n, x) == - LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ {x}, !.wait = @ \ {x}, - !.expiry = {h \in @ : h[2] # x}], +\* `timeout`: the timers on node n that come due together fire in one update. +\* Each doubt in X lifts, and the dead hops whose session ended there may be +\* forgotten; each hop in H expires on its own timer, which can fire before a +\* later failure's timer on the same far node, so a new DEAD cannot extend every +\* old hop's retention indefinitely. With `Patient`, a timer fires only once no +\* DEAD about its hop, and no DRAIN of its node, is in flight: a timer long +\* enough for the news to cross the mesh. +Timeout(n, X, H) == + LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.doubt = @ \ X, !.wait = @ \ X, + !.expiry = {h \in @ : h \notin H /\ h[2] \notin X}], OV, up, gen, ended[n], NoReport, TRUE)] - IN /\ x \in st[n].wait + IN /\ X \subseteq st[n].wait + /\ H \subseteq st[n].expiry + /\ X # {} \/ H # {} /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : - ~(q[d][i].t = "dead" /\ q[d][i].hop[2] = x) /\ ~(q[d][i].t = "drain" /\ q[d][i].node = x) - /\ Commit(rs, q) - /\ Internal(Sent(rs)) - /\ UNCHANGED <> - -\* A hop's own timer can fire before a later failure's timer on the same far -\* node. A new DEAD cannot extend every old hop's retention indefinitely. -Expire(n, hop) == - LET rs == [y \in {n} |-> Update(n, [st[n] EXCEPT !.expiry = @ \ {hop}], - OV, up, gen, ended[n], NoReport, TRUE)] - IN /\ hop \in st[n].expiry - /\ Patient => \A d \in Dirs : \A i \in DOMAIN q[d] : ~(q[d][i].t = "dead" /\ q[d][i].hop = hop) + /\ ~(q[d][i].t = "dead" /\ (q[d][i].hop[2] \in X \/ q[d][i].hop \in H)) + /\ ~(q[d][i].t = "drain" /\ q[d][i].node \in X) /\ Commit(rs, q) /\ Internal(Sent(rs)) /\ UNCHANGED <> @@ -662,7 +690,7 @@ GoAway(n, m) == \* A closed session: everything learned or sent on it goes. Closed(s, y) == [s EXCEPT !.rx[y] = [k \in Keys |-> NoHeard], - !.tx[y] = [k \in Keys |-> NoSent(k)], + !.tx[y] = [k \in Keys |-> NoSent(NoSrc)], !.told[y] = {}, !.toldDrain[y] = {}, !.drain = @ \ {y}] @@ -711,8 +739,7 @@ Done == Next == \/ \E c \in Claims : Announce(c) \/ Unannounce(c) \/ \E d \in Dirs : Deliver(d) - \/ \E n \in Relays, x \in Nodes : Lift(n, x) - \/ \E n \in Relays : \E hop \in st[n].expiry : Expire(n, hop) + \/ \E n \in Relays : \E X \in SUBSET st[n].wait, H \in SUBSET st[n].expiry : Timeout(n, X, H) \/ \E l \in Links : Connect(l) \/ Disconnect(l) \/ \E n \in Relays, m \in Leaving : GoAway(n, m) \/ Done diff --git a/rs/moq-net/verify/route/trace/RouteTrace.cfg b/rs/moq-net/verify/route/trace/RouteTrace.cfg new file mode 100644 index 00000000000..0701875283a --- /dev/null +++ b/rs/moq-net/verify/route/trace/RouteTrace.cfg @@ -0,0 +1,37 @@ +\* Checks the trace TRACE names; `just rs route-trace` records and checks +\* every sweep run. The trace gives the mesh; the rest is what the sweeps have. +CONSTANTS + Nodes <- TNodes + Origin <- TOrigin + Links <- TLinks + Cost <- TCost + Leaving <- TLeaving + Claims <- TClaims + Dear = {} + Ring = {} + Preferred = {} + Scoped = {} + Narrow = {} + MaxRev = 1000000 + MaxFlaps = 1000000 + \* The sweeps hold messages past any timer. + Patient = FALSE + +SPECIFICATION TraceSpec +ALIAS Failure + +\* The model's rules, on the router's own runs. `Stable` is left out: the +\* sweep's handoff scenario 139 breaks it, in a behaviour of the model too +\* (quest/m2/cluster-routes/stable-repath.md). +INVARIANTS + Agrees + Truthful + NoRevival + NoFalseEnd + NoRegression + Seen + SimplePaths + LoopsAreStale + StaleIsChased + RequestsLoopFree + QuietIsSettled diff --git a/rs/moq-net/verify/route/trace/RouteTrace.tla b/rs/moq-net/verify/route/trace/RouteTrace.tla new file mode 100644 index 00000000000..d34b85b1847 --- /dev/null +++ b/rs/moq-net/verify/route/trace/RouteTrace.tla @@ -0,0 +1,143 @@ +----------------------------- MODULE RouteTrace ----------------------------- +(***************************************************************************) +(* Trace validation: whether a run of the router, as `route::tests::Sim` *) +(* records it (rs/moq-net/src/route/tests/trace.rs), is a behaviour of *) +(* RouteHandoff. *) +(* *) +(* The trace's first line gives the model's constants; each later line is *) +(* one step of `Next` with its arguments, the messages the router sent, *) +(* and what each node it touched routes and shows. Each step takes that *) +(* action, and its messages and the touched nodes' routes and shown claims *) +(* must be the router's. Every action is deterministic given its *) +(* arguments, so TLC follows one state per step. A step the model cannot *) +(* take leaves no successor, and TLC reports a deadlock; one whose *) +(* messages or state differ breaks `Agrees`, in a state that shows the *) +(* model's queues and nodes. Either way the last state's `l` is one past *) +(* the step that failed. A run that matches every step ends in `Matched`, *) +(* which stutters. *) +(* *) +(* `skip` is a step the model does not see, such as a delivery of SYNCED *) +(* or one only a client takes: nothing may change and nothing be sent. *) +(***************************************************************************) +EXTENDS RouteHandoff, Json, IOUtils, TLC + +\* The trace named by the environment variable TRACE. +Log == ndJsonDeserialize(IOEnv.TRACE) +Header == Log[1] +Steps == [i \in 1..Len(Log) - 1 |-> Log[i + 1]] + +ToSet(s) == {s[i] : i \in DOMAIN s} + +\* The constants the trace gives, which the config substitutes for the model's. +TNodes == ToSet(Header.nodes) +TOrigin == Header.origin +TLinks == {{e[1], e[2]} : e \in ToSet(Header.links)} +TClaims == ToSet(Header.claims) +\* Every relay may send GOAWAY. +TLeaving == TNodes \ {TOrigin} +\* Each link's configured cost, in place of `Dear`'s 1 or 2. +TCost == [l \in TLinks |-> (CHOOSE e \in ToSet(Header.links) : {e[1], e[2]} = l)[3]] + +VARIABLES + l, \* the step to take next + ok \* whether the last step's messages and state matched the router's +traceVars == <> + +----------------------------------------------------------------------------- +(* Matching what the router did. *) + +\* A model message is the router's. A claims message is matched by its floor: +\* `Truthful` and the sweep's own check pin the claims at a floor. +Match(m, j) == + /\ m.t = j.t + /\ CASE j.t = "offer" -> m.k = j.k /\ m.path = j.path /\ m.drain = j.drain + [] j.t \in {"hold", "end"} -> m.k = j.k + [] j.t = "claims" -> m.k = j.k /\ m.rev = j.rev + [] j.t = "dead" -> m.hop = j.hop /\ m.gone = j.gone /\ m.moved = ToSet(j.moved) + [] j.t = "drain" -> m.node = j.node + +\* The messages step e sent on d, in order. +SentOn(e, d) == LET mine == SelectSeq(e.sent, LAMBDA x : x[1] = d[1] /\ x[2] = d[2]) + IN [i \in DOMAIN mine |-> mine[i][3]] + +\* What d held before step e appended to it: less the message it delivered, or +\* nothing once its session closed. +Before(e, d) == IF e.op = "deliver" /\ d = <> THEN Tail(q[d]) + ELSE IF e.op = "disconnect" /\ {d[1], d[2]} = {e.a, e.b} THEN <<>> + ELSE q[d] + +\* Every queue gained exactly what the router sent on it. The model drops what +\* is sent to the publisher, and so does the trace. +SentMatches(e) == + /\ \A x \in ToSet(e.sent) : <> \in Dirs + /\ \A d \in Dirs : d[2] # Origin => + LET b == Before(e, d) + c == SentOn(e, d) + IN /\ Len(q'[d]) = Len(b) + Len(c) + /\ \A i \in DOMAIN c : Match(q'[d][Len(b) + i], c[i]) + +RouteMatches(r, j) == + /\ r.on = j.on + /\ j.on => /\ r.held = j.held + /\ r.next = j.next + /\ r.path = j.path + /\ ~j.held => r.drain = j.drain + +ShownMatches(shown, j) == + \A c \in Claims : + shown[c] = IF \E x \in ToSet(j) : x[1] = c + THEN LET x == CHOOSE x \in ToSet(j) : x[1] = c + IN [item |-> x[2], avail |-> x[3]] + ELSE NotShown + +\* Each node the step touched routes and shows what the router's does. The +\* router keeps routes for the root alone: a trace has no scoped relay. +StateMatches(e) == + \A x \in ToSet(e.state) : + /\ RouteMatches(st'[x.node].rt["root"], x.route) + /\ ~st'[x.node].rt["a"].on + /\ ShownMatches(st'[x.node].shown, x.shown) + +----------------------------------------------------------------------------- +(* Following the trace. *) + +Skip == UNCHANGED vars + +\* The action step e names. +Act(e) == + CASE e.op = "claim" -> /\ IF e.item THEN Announce(e.claim) ELSE Unannounce(e.claim) + /\ rev' = e.rev + [] e.op = "deliver" -> Deliver(<>) + [] e.op = "connect" -> Connect({e.a, e.b}) + [] e.op = "disconnect" -> Disconnect({e.a, e.b}) + [] e.op = "timeout" -> Timeout(e.node, ToSet(e.lift), ToSet(e.expire)) + \* GOAWAY again on a session already draining: `GoAway` is not enabled. + [] e.op = "goaway" -> IF e.from \in st[e.node].drain THEN Skip ELSE GoAway(e.node, e.from) + [] e.op = "skip" -> Skip + +TraceInit == Init /\ l = 1 /\ ok = [sent |-> TRUE, state |-> TRUE] + +TraceNext == + /\ l <= Len(Steps) + /\ Act(Steps[l]) + /\ ok' = [sent |-> SentMatches(Steps[l]), state |-> StateMatches(Steps[l])] + /\ l' = l + 1 + +\* The router sent and holds what the model does. +Agrees == ok.sent /\ ok.state + +\* What a failing state shows: the step it took, what matched, and the model's +\* routes, shown claims, doubts, dead hops, and queues after it. +Failure == [step |-> IF l > 1 THEN Steps[l - 1] ELSE "none", ok |-> ok, + route |-> [n \in Nodes |-> st[n].rt["root"]], + shown |-> [n \in Nodes |-> {<> : + c \in {x \in Claims : st[n].shown[x] # NotShown}}], + doubt |-> [n \in Nodes |-> st[n].doubt], dead |-> [n \in Nodes |-> st[n].dead], + q |-> [d \in {x \in Dirs : q[x] # <<>>} |-> q[d]]] + +\* Every step matched. +Matched == l > Len(Steps) /\ UNCHANGED traceVars + +TraceSpec == TraceInit /\ [][TraceNext \/ Matched]_traceVars + +============================================================================= diff --git a/sh/tla-trace.py b/sh/tla-trace.py new file mode 100644 index 00000000000..0a7b8ced0c2 --- /dev/null +++ b/sh/tla-trace.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Check recorded traces against a TLA+ model: `sh/tla-trace.py ...`. + +Trace validation: a test records each run as newline-delimited JSON, and a trace +spec beside the model (`verify//trace/.tla`, with `.cfg`) +follows one trace, named by the environment variable TRACE, as a behaviour of the +model it EXTENDS. TLC reports a deadlock where the run leaves the model, so each +trace must pass with TLC's deadlock check on. + +The trace spec and the model's modules are linked into one directory under the +checkout's `.tmp/`, since TLC finds a module only beside the spec. Each trace is +one TLC run on one worker through sh/tlc.py; four run at once. A trace that +passes is removed, so what is left is what failed. +""" + +import os +import subprocess +import sys +import tempfile +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +JOBS = 4 + + +def check(spec, cfg, trace): + env = dict(os.environ, TRACE=str(trace.resolve())) + command = [sys.executable, str(ROOT / "sh/tlc.py"), str(spec), "-workers", "1", "-config", str(cfg)] + run = subprocess.run(command, cwd=ROOT, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + return trace, run.returncode, run.stdout + + +def main(argv): + if len(argv) < 2: + sys.exit("usage: sh/tla-trace.py ...") + spec, traces = Path(argv[0]).resolve(), [Path(trace) for trace in argv[1:]] + model = spec.parent.parent + (ROOT / ".tmp").mkdir(exist_ok=True) + with tempfile.TemporaryDirectory(dir=ROOT / ".tmp", prefix="tla-trace-") as run: + run = Path(run) + for module in [*model.glob("*.tla"), spec]: + (run / module.name).symlink_to(module) + cfg = spec.with_suffix(".cfg") + failed = [] + with ThreadPoolExecutor(JOBS) as pool: + for trace, status, out in pool.map(lambda trace: check(run / spec.name, cfg, trace), traces): + if status == 0: + trace.unlink() + else: + failed.append(trace) + print(f"::group::{trace}: exit {status}\n{out}::endgroup::", flush=True) + print(f"{len(traces) - len(failed)} of {len(traces)} traces are behaviours of the model") + if failed: + print("error: left the model:", *failed, sep="\n ", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/sh/tla.sh b/sh/tla.sh index 742441e63e8..26a426ed3a0 100755 --- a/sh/tla.sh +++ b/sh/tla.sh @@ -4,13 +4,14 @@ # a rule (a deadlock, invariant, property, or assertion), so a fix that should # promote its rule is noticed; every other config must pass. A config with a # `\* Simulate:` line is too large to exhaust: TLC samples it at random with the -# arguments that line gives. +# arguments that line gives. A `trace/` directory holds a trace spec, which +# checks recorded runs rather than exploring (`sh/tla-trace.py`). set -euo pipefail shopt -s nullglob cd "$(git rev-parse --show-toplevel)" [ $# -ne 0 ] || set -- rs js -mapfile -t configs < <(git ls-files --cached --others --exclude-standard -- "$@" | grep -E '^(rs|js)/.*/verify/.*\.cfg$' || true) +mapfile -t configs < <(git ls-files --cached --others --exclude-standard -- "$@" | grep -E '^(rs|js)/.*/verify/.*\.cfg$' | grep -v '/trace/' || true) if [ ${#configs[@]} -eq 0 ]; then echo "error: no TLA+ config under: $*" >&2 exit 1 diff --git a/test/justfile b/test/justfile index 23bb2a28c40..1a4a83db763 100644 --- a/test/justfile +++ b/test/justfile @@ -310,6 +310,10 @@ tla *paths: route-key-model: bash route-key-model.sh +# Check that the router's handoff sweeps, from a fresh replayable window, are behaviours of the route model. +route-trace: + bash ../sh/local.sh bash route-trace.sh + # Check random lite-07 meshes against the route rules from a fresh replayable seed. route-sweep: bash ../sh/local.sh bash route-sweep.sh diff --git a/test/route-trace.sh b/test/route-trace.sh new file mode 100755 index 00000000000..11aae5a5ae1 --- /dev/null +++ b/test/route-trace.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Trace validation of the route model: record the router's handoff sweeps under the +# path rule and check with TLC that each run is a behaviour of RouteHandoff.tla +# (rs/moq-net/verify/route/trace/). Nightly traces a fresh window of the sweeps' +# scenarios; the printed settings replay it. +set -euo pipefail +cd "$(git rev-parse --show-toplevel)" + +MOQ_ROUTE_TRACE_FROM=${MOQ_ROUTE_TRACE_FROM:-$(python3 -c 'import secrets; print(secrets.randbelow(20000))')} +MOQ_ROUTE_SCENARIOS=${MOQ_ROUTE_SCENARIOS:-$((MOQ_ROUTE_TRACE_FROM + 150))} +export MOQ_ROUTE_TRACE_FROM MOQ_ROUTE_SCENARIOS +echo "Route traces: MOQ_ROUTE_TRACE_FROM=$MOQ_ROUTE_TRACE_FROM MOQ_ROUTE_SCENARIOS=$MOQ_ROUTE_SCENARIOS" + +mkdir -p .tmp +MOQ_ROUTE_TRACE=$(mktemp -d "$PWD/.tmp/route-trace.XXXXXX") +export MOQ_ROUTE_TRACE +trap 'rm -rf "$MOQ_ROUTE_TRACE"' EXIT +# Release keeps a window far into the sweeps inside nextest's hang budget. +export CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true +just rs test -P sweep -p moq-net --release -E 'test(/^route::tests::handoff::path::(sweep_handoff|sweep_scoped_handoff)$/)' +python3 sh/tla-trace.py rs/moq-net/verify/route/trace/RouteTrace.tla "$MOQ_ROUTE_TRACE"/*.ndjson From fca59e7034548e140f24e250063a53cefcbeefb7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 04:53:16 -0800 Subject: [PATCH 3/4] quest: record route trace validation's progress, paused at the move to m3 Co-Authored-By: Claude Opus 5.5 --- quest/m3/route-trace-validation.md | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/quest/m3/route-trace-validation.md b/quest/m3/route-trace-validation.md index 5f2ec74099d..68e48802906 100644 --- a/quest/m3/route-trace-validation.md +++ b/quest/m3/route-trace-validation.md @@ -47,4 +47,37 @@ of it, so this waits for the router too. Run it on the sweeps' existing seeds in CI only if it stays fast; otherwise in Nightly with the sweeps' fresh seeds. +Paused 2026-11-14 when the quest moved to m3, with the work done on its +branch ([moq-dev/moq#8733](https://github.com/moq-dev/moq/pull/8733)), taken +under /quest-afk for the maintainer's review: + +- Done: `route::tests::Sim` records each path-rule run of both sweeps + (`rs/moq-net/src/route/tests/trace.rs`), `rs/moq-net/verify/route/trace/RouteTrace.tla` + follows it step by step comparing every message and each touched relay's + route and shown claims, `sh/tla-trace.py` runs TLC per trace, and + `test route-trace` (Nightly, a fresh window of 150 scenarios) wires it up. + All 300 traces of the gate's scenarios pass on the pinned TLC, and two of + four planted router bugs fail traces while the sweeps pass. +- ✅ Restrict the sweeps, and grow the model only where the router had + drifted (rejected: growing the model to several publishers, restarts, + grants, and interest edits, which four open PRs on `RouteHandoff.tla` + would conflict with). `sweep_handoff` runs one origin whose every claim change + changes a claim, with no restart; `the_rules_agree_on_every_script` keeps the + full script. The client, a leaf that offers no route, is dropped from the + traces. The model grew three ways: a route goes to a stream only while a + claim meets it, timers due together fire in one `Timeout`, and an ended + stream forgets what it reflected. Every config still passes. +- ✅ Nightly only (rejected: the merge gate): 300 traces cost about 20 + core-minutes of TLC. +- Found: `Stable` breaks on an accepted trace, a behaviour of the model the + configs miss ([stable-repath](/quest/m2/cluster-routes/stable-repath.md)), + so `RouteTrace.cfg` leaves it out. + +Left: run `just fix` and `just check`, refresh the skill's state-count table +for the grown model (`RouteHandoff.cfg` now 29,229 distinct states; the +configs' new counts are in each run's output), run `test route-trace` through +`sh/tlc.py` once Nix fetches TLC again (this host's daemon hung on it, so the +runs used the same jars by hand), review with `/quest-review`, and finish the +PR body. + Public API: none. Wire: none. From 3ff726125b08d1170f20bafdc420f53b147ee72b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 04:54:50 -0800 Subject: [PATCH 4/4] test(moq-net): name the route-trace recipe right in RouteTrace.cfg Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/verify/route/trace/RouteTrace.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-net/verify/route/trace/RouteTrace.cfg b/rs/moq-net/verify/route/trace/RouteTrace.cfg index 0701875283a..c8f20d24831 100644 --- a/rs/moq-net/verify/route/trace/RouteTrace.cfg +++ b/rs/moq-net/verify/route/trace/RouteTrace.cfg @@ -1,4 +1,4 @@ -\* Checks the trace TRACE names; `just rs route-trace` records and checks +\* Checks the trace TRACE names; `just test route-trace` records and checks \* every sweep run. The trace gives the mesh; the rest is what the sweeps have. CONSTANTS Nodes <- TNodes