From eb65d86a376a27aaa7882244126ce6214a93d5b3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 00:43:35 -0800 Subject: [PATCH 1/3] chore: TLC runs on Java 21 at low priority, and plan two faster-checking spikes The tla recipes move from TLC 1.7.4 on Java 8 to TLC 1.8.0 on Java 21 (nixpkgs#tlaplus18), run niced with a 4 GB heap, and write no trace spec beside the model. The verify skills run TLC through quest gate. Plans a spike of Rust TLA+ checkers and a spike of exhaustive search over route::tests::Sim. Co-Authored-By: Claude --- .claude/skills/verify-reconnect/SKILL.md | 8 +-- .claude/skills/verify-resume/SKILL.md | 4 +- .claude/skills/verify-route/SKILL.md | 4 +- .claude/skills/verify-subscribe/SKILL.md | 4 +- js/justfile | 3 +- quest/m1/route-exhaustive-sim.md | 55 +++++++++++++++++++++ quest/m1/tla-native-checker.md | 62 ++++++++++++++++++++++++ rs/justfile | 7 ++- 8 files changed, 134 insertions(+), 13 deletions(-) create mode 100644 quest/m1/route-exhaustive-sim.md create mode 100644 quest/m1/tla-native-checker.md diff --git a/.claude/skills/verify-reconnect/SKILL.md b/.claude/skills/verify-reconnect/SKILL.md index d6bb2cfc4c2..af3bcbc0d29 100644 --- a/.claude/skills/verify-reconnect/SKILL.md +++ b/.claude/skills/verify-reconnect/SKILL.md @@ -23,10 +23,10 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan` and ## Run ```sh -just rs tla moq-net/verify/reconnect/Handover.tla -just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg -just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg -just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg ``` The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute on 32 cores. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. diff --git a/.claude/skills/verify-resume/SKILL.md b/.claude/skills/verify-resume/SKILL.md index a0ba5e2f688..bd46d3b233b 100644 --- a/.claude/skills/verify-resume/SKILL.md +++ b/.claude/skills/verify-resume/SKILL.md @@ -24,8 +24,8 @@ Each rule is an INVARIANT; the harm says why it matters. ## Run ```sh -just rs tla moq-net/verify/resume/ResumeReader.tla -just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg +quest gate just rs tla moq-net/verify/resume/ResumeReader.tla +quest gate just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg ``` The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, three minutes on 32 cores. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, under two minutes. Both pass at the sync commit. diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index d2a2a815011..d95f0c03059 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -27,8 +27,8 @@ The constraint `Bounded` stops exploring one hop past `NoCountToInfinity`, so a ## Run ```sh -just rs tla moq-net/verify/route/RouteHandoff.tla -just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/CountToInfinity.cfg +quest gate just rs tla moq-net/verify/route/RouteHandoff.tla +quest gate just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/CountToInfinity.cfg ``` The first checks every rule but `NoCountToInfinity`: about 3.7M states, two minutes on 32 cores, and it passes at the sync commit. The second fails at the sync commit on the open finding below. diff --git a/.claude/skills/verify-subscribe/SKILL.md b/.claude/skills/verify-subscribe/SKILL.md index b706ad43c76..f6509441418 100644 --- a/.claude/skills/verify-subscribe/SKILL.md +++ b/.claude/skills/verify-subscribe/SKILL.md @@ -21,8 +21,8 @@ Each rule is an INVARIANT; the harm says why it matters. ## Run ```sh -just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg +quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla +quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg ``` The first checks every rule but `StreamWins`: about 3.0M states, 20 to 30 seconds on 32 cores, and it passes at the sync commit. The second fails at the sync commit on the open finding below. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race. diff --git a/js/justfile b/js/justfile index ea47805d793..df716f0d55c 100644 --- a/js/justfile +++ b/js/justfile @@ -55,8 +55,9 @@ fix-changed list: # Model-check a TLA+ spec with TLC, e.g. `just js tla watch/verify/Watch.tla`. # Pass `-config js/.cfg` for a config not named after the spec. +# TLC runs at the lowest priority with a 4 GB heap, as `just rs tla` explains. tla spec *args: - cd .. && nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} js/{{ spec }} + cd .. && JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} js/{{ spec }} # Run all unit tests. test: diff --git a/quest/m1/route-exhaustive-sim.md b/quest/m1/route-exhaustive-sim.md new file mode 100644 index 00000000000..ac38079e526 --- /dev/null +++ b/quest/m1/route-exhaustive-sim.md @@ -0,0 +1,55 @@ +# [M] Every delivery order of a small mesh runs on the real route layer + +## Goal + +A test explores every order in which `route::tests::Sim` can deliver, hold, +disconnect, collect, and change claims on a triangle of the origin and two +relays, on the real `route::Router`, and checks the route rules in every state +it reaches. A broken rule prints the path to it as `Sim` calls. The PR reports +its states and time beside TLC's for the same bounds. If the real state space +does not fit, the PR records the numbers and lands no test. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan), as a spike: `RouteHandoff.tla` +is a hand-kept mirror of `route/mod.rs` that TLC interprets, while `Sim` +already drives the router one atomic step at a time. Searching `Sim` itself +runs compiled code, has no model to drift, and reaches what the model leaves +out. + +- ✅ A hand-written breadth-first search beside `Sim` in `route/tests/`: a + fingerprint set, a queue, and a parent map for the path (rejected: the + `stateright` crate, whose 12 direct dependencies include dashmap, rand, + serde_json, and tiny_http; building both and keeping the better). Add + threads or symmetry only if the measurements call for them. +- Bound the run as the model does: one claim, two revisions, two session opens + or closes, one collection (`MaxRev`, `MaxFlaps`, `MaxCollects` in + `RouteHandoff.cfg`), so the two counts compare. +- The search converges only if equal situations hash equal. `Sim` and `Router` + need a clone and a fingerprint, and the fingerprint must leave out what only + records the past, such as `events`, `stats`, and `requested`. Check each + rule at the step that would break it rather than from that log. +- While [the echo fix](/quest/m2/cluster-routing/routes/echo-order.md) is + open, the search must find a count to infinity, as + `residual_an_overtaking_echo_counts_to_infinity` replays; once it lands, the + search must pass. Either way that is the check that the search works. +- Then say which of the model's "Not covered" list (`verify-route` skill) the + search can afford: wildcards, scoped grants, costs other than 1, a fourth + node. +- A search that takes seconds joins the merge gate; a larger one runs in + Nightly. + +This sits beside [trace validation](/quest/m1/route-trace-validation.md), which +keeps the TLA+ model honest for [the inductive invariant](/quest/m1/route-inductive.md) +and the design work done on the spec before code exists. It replaces neither +that nor the random sweeps on larger meshes +([the mesh sweep](/quest/m1/route-mesh-sweep.md)). If the spike shows the +search covers what a model check does, recommend in the PR what to do with the +route model's TLC runs. + +Public API: none. Wire: none. + +## Related + +- [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself +- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - the other spike, a faster engine for the same specs diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md new file mode 100644 index 00000000000..33ac992091b --- /dev/null +++ b/quest/m1/tla-native-checker.md @@ -0,0 +1,62 @@ +# [M] A Rust TLA+ checker runs the models locally if it agrees with TLC + +## Goal + +The `tla` recipes run a native TLA+ checker locally when one gives TLC's +verdict and distinct-state count on every config in the tree and costs fewer +core-seconds on the route and resume models. Nightly then runs both engines on +every config and fails when they disagree. If no checker agrees, the PR +records what each one failed on and adopts nothing. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan), as a spike: TLC interprets a +spec on the JVM, and the route model's 4-node runs take tens of minutes of +many cores. + +- ✅ A checker that agrees runs locally, and Nightly cross-checks it against + TLC (rejected: local runs with a TLC-only Nightly, where a checker bump or + spec change could make it wrong unnoticed; replacing TLC everywhere, where + nothing checks the checker again; reporting the numbers and adopting + nothing). + +Candidates found that day, each young, at least one AI-written, and benchmarked +only by its authors: + +- [tla-mc](https://github.com/ddalton/tla-mc) compiles a spec to a native + parallel checker and claims about 4x TLC at 8 workers. +- [tlaplusplus](https://github.com/zoratu/tlaplusplus) claims about 10x on + many cores and passes 174 of 182 of tlaplus/Examples. +- [tla-rs](https://github.com/bbcvc/tla-rs) claims state-for-state agreement + with TLC on 823 recorded runs and 2x to 3x on large models with one worker. +- [tlx](https://github.com/Medfin-AI/tlx) reports some models slower than TLC. + +A checker bug shows as a rule passing that should break, so agreement is the +bar, not speed: + +- Every config: the same verdict as TLC, the same broken rule for an + open-finding config, and the same distinct-state count. The specs use + `RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`, + `CONSTRAINT`, and `VIEW`, unmodified. +- Measure both engines at equal worker counts through `quest gate` on a quiet + host, against TLC as the `tla` recipes run it. That baseline, measured + 2026-11-10 on a 24-core desktop: `RouteHandoff.cfg` (17.7M states generated, + 3.67M distinct) in about 50 s and 930 CPU-seconds on TLC 1.8.0 with Java 21, + down from about 118 s and 1,730 on TLC 1.7.4 with Java 8. CPU-seconds stayed + near flat from 2 to 24 workers. Report the table in the PR. +- A config the checker cannot run, such as a liveness property or random + simulation, which [the mesh model](/quest/m1/route-mesh-model.md) plans, + stays on TLC; the recipe picks the engine per config and errors rather than + falling back at runtime. +- A broken rule must still give `/statechart` a trace it can read with the + spec's `*.view.ts`. Say how. +- Build each candidate from a flake input pinned to a revision, inside the Nix + sandbox. Run none of their build scripts or binaries outside it. +- Report upstream bugs to the maintainer rather than filing them. + +Public API: none. Wire: none. + +## Related + +- [The TLA+ models run nightly and reviews resync them](/quest/m1/model-checks-nightly.md) - the Nightly job the cross-check joins +- [Every delivery order of a small mesh runs on the real route layer](/quest/m1/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model diff --git a/rs/justfile b/rs/justfile index 391e57fb7cf..96a07727bb3 100644 --- a/rs/justfile +++ b/rs/justfile @@ -19,9 +19,12 @@ default: just check # Model-check a TLA+ spec with TLC, e.g. `just rs tla moq-net/verify/route/RouteHandoff.tla`. -# Pass `-config rs/.cfg` for a config not named after the spec. +# Pass `-config rs/.cfg` for a config not named after the spec, and +# `../.tla` for a spec outside `rs/`, such as a generated one in `.tmp/`. +# TLC runs at the lowest priority with a 4 GB heap, which no model here outgrows: +# the JVM otherwise takes a quarter of RAM per run. tla spec *args: - nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }} + JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }} # Default features for the selected packages; `features` covers the feature # matrix nightly and on release PRs. No `cargo check` pass: clippy is a From 3d302aa54cda89830fb2372b4bb6b9aad0576149 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 14:20:22 -0800 Subject: [PATCH 2/3] quest: record the pick round's answers for the TLC spikes Co-Authored-By: Claude Opus 5.5 --- quest/m1/tla-native-checker.md | 2 +- .../routes}/route-exhaustive-sim.md | 21 ++++++++----------- 2 files changed, 10 insertions(+), 13 deletions(-) rename quest/{m1 => m2/cluster-routing/routes}/route-exhaustive-sim.md (79%) diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md index 41ce0fd0960..9a4e033fb21 100644 --- a/quest/m1/tla-native-checker.md +++ b/quest/m1/tla-native-checker.md @@ -59,4 +59,4 @@ Public API: none. Wire: none. ## Related -- [Every delivery order of a small mesh runs on the real route layer](/quest/m1/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model +- [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routing/routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model diff --git a/quest/m1/route-exhaustive-sim.md b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md similarity index 79% rename from quest/m1/route-exhaustive-sim.md rename to quest/m2/cluster-routing/routes/route-exhaustive-sim.md index 3f915918ce9..ad4023478e6 100644 --- a/quest/m1/route-exhaustive-sim.md +++ b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md @@ -46,21 +46,18 @@ that nor the seeded sweeps on larger meshes search covers what a model check does, recommend in the PR what to do with the route model's TLC runs. +Decided 2026-11-12 (maintainer, pick round): the search runs on the `Router` +after [the router quest](/quest/m2/cluster-routing/routes/scope.md)'s rewrite to +the path rule, so it checks the code that ships and nothing is built twice +(rejected: today's Babel `Router`, which the rewrite throws away; the test-only +`paths.rs` prototype). The storm self-check above then runs against a router +from before the rewrite. + Public API: none. Wire: none. -## Waiting +## Required -- Maintainer: which router the search runs on. Since this quest was planned, - the model moved to the path rule ([#8487](https://github.com/moq-dev/moq/pull/8487)), - which `route/tests/paths.rs` prototypes, and - [the router quest](/quest/m2/cluster-routing/routes/scope.md) rewrites - `route::Router` to it. Options: wait for that rewrite and search the new - `Router` (recommended: the search then checks the code that ships, and - nothing is built twice), which moves this quest beside it in - `quest/m2/cluster-routing/routes/` with the router quest under `Required`; - search today's Babel `Router` now, which the rewrite discards; or search the - `paths.rs` prototype, which only tests exercise. An agent cannot pick: it - sets whether this spike stays in m1. +- [The router quest](/quest/m2/cluster-routing/routes/scope.md) - rewrites `route::Router` to the path rule, the router this searches ## Related From 28028e23c22b6d1fb3ca59fde526894e6cdf725a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 14:21:42 -0800 Subject: [PATCH 3/3] quest: settle coderabbit findings on the two spikes' bounds and agreement bar Co-Authored-By: Claude Opus 5.5 --- quest/m1/tla-native-checker.md | 10 +++++++--- .../m2/cluster-routing/routes/route-exhaustive-sim.md | 9 +++++++-- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md index 9a4e033fb21..0af2acd2669 100644 --- a/quest/m1/tla-native-checker.md +++ b/quest/m1/tla-native-checker.md @@ -3,7 +3,8 @@ ## Goal The `tla` recipes run a native TLA+ checker locally when one gives TLC's -verdict and distinct-state count on every config in the tree and costs fewer +verdict and distinct-state count on every config in the tree it can run +(the rest stay on TLC, as the Plan says) and costs fewer core-seconds on the route and resume models. Nightly (`just test tla`, which `sh/tla.sh` drives) then runs both engines on every config and fails when they disagree. If no checker agrees, the PR @@ -35,8 +36,11 @@ only by its authors: A checker bug shows as a rule passing that should break, so agreement is the bar, not speed: -- Every config: the same verdict as TLC, the same broken rule for an - open-finding config, and the same distinct-state count. The specs use +- Every config the checker runs: the same verdict as TLC, and the same broken + rule for an open-finding config. A passing config also matches TLC's + distinct-state count; a violating one does not, since each engine stops at + its first violation after exploring a different, order-dependent part of + the space. The specs use `RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`, `CONSTRAINT`, and `VIEW`, unmodified. - Measure both engines at equal worker counts through `quest gate` on a quiet diff --git a/quest/m2/cluster-routing/routes/route-exhaustive-sim.md b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md index ad4023478e6..0a11a1c296a 100644 --- a/quest/m2/cluster-routing/routes/route-exhaustive-sim.md +++ b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md @@ -23,7 +23,9 @@ out. serde_json, and tiny_http; building both and keeping the better). Add threads or symmetry only if the measurements call for them. - Bound the run as `RouteHandoff.cfg` does (`MaxRev`, `MaxFlaps`, one - claim), so the two counts compare. + claim), but on the triangle: `RouteHandoff.cfg` has four nodes, so the PR + runs TLC on a copy of that config cut to the same three nodes, and the two + counts compare. - The search converges only if equal situations hash equal. `Sim` and `Router` need a clone and a fingerprint, and the fingerprint must leave out what only records the past, such as `events`, `stats`, and `requested`. Check each @@ -31,7 +33,10 @@ out. - The check that the search works: run on today's router before the switch to the path rule, it must find the storm `residual_a_seqno_advance_storms_between_two_grants` replays, or another - `residual_` test in `route/tests/residual.rs` that fits the bounds. + `residual_` test in `route/tests/residual.rs`. Those fixtures use 6 to 10 + nodes, beyond the triangle, so the self-check searches the fixture's own + mesh, bounded by depth and by the fixture's scripted claims, rather than + the triangle's bounds; the search code takes the mesh as a parameter. - Then say which of the model's "Not covered" list (`verify-route` skill) the search can afford: wildcards, scoped grants, costs other than 1, a fourth node.