diff --git a/.claude/skills/verify-failover/SKILL.md b/.claude/skills/verify-failover/SKILL.md index ebab99ef0a9..a38fc98b46d 100644 --- a/.claude/skills/verify-failover/SKILL.md +++ b/.claude/skills/verify-failover/SKILL.md @@ -22,9 +22,9 @@ Each rule is an INVARIANT, except `Settles`, which is TLC's deadlock check: ever ## Run ```sh -just rs tla moq-net/verify/failover/FailoverChain.tla -just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Leave.cfg -just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/EndedRelay.cfg +quest gate just rs tla moq-net/verify/failover/FailoverChain.tla +quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Leave.cfg +quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/EndedRelay.cfg ``` The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.2M states. Each takes seconds and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop: it fails at the sync commit on the open finding below at depth 31, after about 50M distinct states and minutes. diff --git a/.claude/skills/verify-reconnect/SKILL.md b/.claude/skills/verify-reconnect/SKILL.md index d47042a487c..4637c7c5772 100644 --- a/.claude/skills/verify-reconnect/SKILL.md +++ b/.claude/skills/verify-reconnect/SKILL.md @@ -24,10 +24,10 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re ## Run ```sh -just rs tla moq-net/verify/reconnect/Handover.tla -just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg -just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg -just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg +quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg ``` The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. `just test tla rs/moq-net/verify/reconnect` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. diff --git a/.claude/skills/verify-resume/SKILL.md b/.claude/skills/verify-resume/SKILL.md index 6d9eeaa9d2e..91b1c1611a0 100644 --- a/.claude/skills/verify-resume/SKILL.md +++ b/.claude/skills/verify-resume/SKILL.md @@ -24,8 +24,8 @@ Each rule is an INVARIANT; the harm says why it matters. ## Run ```sh -just rs tla moq-net/verify/resume/ResumeReader.tla -just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg +quest gate just rs tla moq-net/verify/resume/ResumeReader.tla +quest gate just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg ``` The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, in minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, in minutes. Both pass at the sync commit. `just test tla rs/moq-net/verify/resume` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. diff --git a/.claude/skills/verify-route-key/SKILL.md b/.claude/skills/verify-route-key/SKILL.md index e7929f50402..714fc7dfe5c 100644 --- a/.claude/skills/verify-route-key/SKILL.md +++ b/.claude/skills/verify-route-key/SKILL.md @@ -32,8 +32,8 @@ Each variant drops one assumption the draft states and must break exactly the le ## Run ```sh -just test route-key-model -just rs tamarin moq-net/verify/route-key/RouteKey.spthy -D=NO_SIDE +quest gate just test route-key-model +quest gate just rs tamarin moq-net/verify/route-key/RouteKey.spthy -D=NO_SIDE ``` The first proves the model and checks every variant's `Breaks:` line, in seconds; Nightly runs it. The second proves one variant. To read a counterexample, run `nix shell --inputs-from . nixpkgs#tamarin-prover --command tamarin-prover interactive rs/moq-net/verify/route-key/` and open the falsified lemma. diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 2f772191f50..f8446466d75 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -34,9 +34,9 @@ What the timer costs, as the configs show it: fired late, it costs only the wait ## Run ```sh -just rs tla moq-net/verify/route/RouteHandoff.tla -just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg -just test tla rs/moq-net/verify/route +quest gate just rs tla moq-net/verify/route/RouteHandoff.tla +quest gate just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg +quest gate just test tla rs/moq-net/verify/route ``` `just test tla rs/moq-net/verify/route` runs every config, as Nightly does, simulating each one with a `Simulate:` line and expecting `Loop.cfg` to break. Times are with `-workers 4`. `Patient` is TRUE unless the table says otherwise. diff --git a/.claude/skills/verify-subscribe/SKILL.md b/.claude/skills/verify-subscribe/SKILL.md index 2a5042886ec..4fc848d4fcc 100644 --- a/.claude/skills/verify-subscribe/SKILL.md +++ b/.claude/skills/verify-subscribe/SKILL.md @@ -21,8 +21,8 @@ Each rule is an INVARIANT; the harm says why it matters. ## Run ```sh -just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg +quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla +quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg ``` The first checks every rule but `StreamWins` under `Pressure`: about 13.1M states, a minute on 4 workers, and it passes at the sync commit. The second checks `StreamWins` without `Pressure` and fails on the open finding below. `just test tla rs/moq-net/verify/subscribe` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race. diff --git a/js/justfile b/js/justfile index 31ea383e3d6..fa3ab58dd38 100644 --- a/js/justfile +++ b/js/justfile @@ -55,8 +55,9 @@ fix-changed list: # Model-check a TLA+ spec with TLC, e.g. `just js tla watch/verify/Watch.tla`. # Pass `-config js/.cfg` for a config not named after the spec. +# TLC runs at the lowest priority with a 4 GB heap, as `just rs tla` explains. tla spec *args: - cd .. && nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} js/{{ spec }} + cd .. && JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} js/{{ spec }} # Run all unit tests. test: diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md new file mode 100644 index 00000000000..0af2acd2669 --- /dev/null +++ b/quest/m1/tla-native-checker.md @@ -0,0 +1,66 @@ +# [M] A Rust TLA+ checker runs the models locally if it agrees with TLC + +## Goal + +The `tla` recipes run a native TLA+ checker locally when one gives TLC's +verdict and distinct-state count on every config in the tree it can run +(the rest stay on TLC, as the Plan says) and costs fewer +core-seconds on the route and resume models. Nightly (`just test tla`, which +`sh/tla.sh` drives) then runs both engines on every config and fails when they +disagree. If no checker agrees, the PR +records what each one failed on and adopts nothing. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan), as a spike: TLC interprets a +spec on the JVM, and the route model's 4-node runs take tens of minutes of +many cores. + +- ✅ A checker that agrees runs locally, and Nightly cross-checks it against + TLC (rejected: local runs with a TLC-only Nightly, where a checker bump or + spec change could make it wrong unnoticed; replacing TLC everywhere, where + nothing checks the checker again; reporting the numbers and adopting + nothing). + +Candidates found that day, each young, at least one AI-written, and benchmarked +only by its authors: + +- [tla-mc](https://github.com/ddalton/tla-mc) compiles a spec to a native + parallel checker and claims about 4x TLC at 8 workers. +- [tlaplusplus](https://github.com/zoratu/tlaplusplus) claims about 10x on + many cores and passes 174 of 182 of tlaplus/Examples. +- [tla-rs](https://github.com/bbcvc/tla-rs) claims state-for-state agreement + with TLC on 823 recorded runs and 2x to 3x on large models with one worker. +- [tlx](https://github.com/Medfin-AI/tlx) reports some models slower than TLC. + +A checker bug shows as a rule passing that should break, so agreement is the +bar, not speed: + +- Every config the checker runs: the same verdict as TLC, and the same broken + rule for an open-finding config. A passing config also matches TLC's + distinct-state count; a violating one does not, since each engine stops at + its first violation after exploring a different, order-dependent part of + the space. The specs use + `RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`, + `CONSTRAINT`, and `VIEW`, unmodified. +- Measure both engines at equal worker counts through `quest gate` on a quiet + host, against TLC as the `tla` recipes run it. The 2026-11-10 baseline, on a + 24-core desktop and the route model as it stood then (3.67M distinct + states): about 50 s and 930 CPU-seconds on TLC 1.8.0 with Java 21, down from + about 118 s and 1,730 on TLC 1.7.4 with Java 8, with CPU-seconds near flat + from 2 to 24 workers. The model has since changed, so remeasure TLC and + report the table in the PR. +- A config the checker cannot run, such as one with a `\* Simulate:` line + that `sh/tla.sh` samples at random, stays on TLC; the recipe picks the engine per config and errors rather than + falling back at runtime. +- A broken rule must still print a trace as readable as TLC's, which the + Reading a failure in `rs/moq-net/verify/README.md` relies on. Say how. +- Build each candidate from a flake input pinned to a revision, inside the Nix + sandbox. Run none of their build scripts or binaries outside it. +- Report upstream bugs to the maintainer rather than filing them. + +Public API: none. Wire: none. + +## Related + +- [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routing/routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model diff --git a/quest/m2/cluster-routing/routes/route-exhaustive-sim.md b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md new file mode 100644 index 00000000000..0a11a1c296a --- /dev/null +++ b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md @@ -0,0 +1,70 @@ +# [M] Every delivery order of a small mesh runs on the real route layer + +## Goal + +A test explores every order in which `route::tests::Sim` can deliver, hold, +disconnect, collect, and change claims on a triangle of the origin and two +relays, on the real `route::Router`, and checks the route rules in every state +it reaches. A broken rule prints the path to it as `Sim` calls. The PR reports +its states and time beside TLC's for the same bounds. If the real state space +does not fit, the PR records the numbers and lands no test. + +## Plan + +Decided 2026-11-10 (maintainer, /quest-plan), as a spike: `RouteHandoff.tla` +is a hand-kept mirror of the rule `route/tests/paths.rs` prototypes, which TLC interprets, while `Sim` +already drives the router one atomic step at a time. Searching `Sim` itself +runs compiled code, has no model to drift, and reaches what the model leaves +out. + +- ✅ A hand-written breadth-first search beside `Sim` in `route/tests/`: a + fingerprint set, a queue, and a parent map for the path (rejected: the + `stateright` crate, whose 12 direct dependencies include dashmap, rand, + serde_json, and tiny_http; building both and keeping the better). Add + threads or symmetry only if the measurements call for them. +- Bound the run as `RouteHandoff.cfg` does (`MaxRev`, `MaxFlaps`, one + claim), but on the triangle: `RouteHandoff.cfg` has four nodes, so the PR + runs TLC on a copy of that config cut to the same three nodes, and the two + counts compare. +- The search converges only if equal situations hash equal. `Sim` and `Router` + need a clone and a fingerprint, and the fingerprint must leave out what only + records the past, such as `events`, `stats`, and `requested`. Check each + rule at the step that would break it rather than from that log. +- The check that the search works: run on today's router before the switch to + the path rule, it must find the storm + `residual_a_seqno_advance_storms_between_two_grants` replays, or another + `residual_` test in `route/tests/residual.rs`. Those fixtures use 6 to 10 + nodes, beyond the triangle, so the self-check searches the fixture's own + mesh, bounded by depth and by the fixture's scripted claims, rather than + the triangle's bounds; the search code takes the mesh as a parameter. +- Then say which of the model's "Not covered" list (`verify-route` skill) the + search can afford: wildcards, scoped grants, costs other than 1, a fourth + node. +- A search that takes seconds joins the merge gate; a larger one runs in + Nightly. + +This sits beside [trace validation](/quest/m1/route-trace-validation.md), which +keeps the TLA+ model honest for [the inductive invariant](/quest/m1/route-inductive.md) +and the design work done on the spec before code exists. It replaces neither +that nor the seeded sweeps on larger meshes +(`tests/integration/mesh_sweep.rs`). If the spike shows the +search covers what a model check does, recommend in the PR what to do with the +route model's TLC runs. + +Decided 2026-11-12 (maintainer, pick round): the search runs on the `Router` +after [the router quest](/quest/m2/cluster-routing/routes/scope.md)'s rewrite to +the path rule, so it checks the code that ships and nothing is built twice +(rejected: today's Babel `Router`, which the rewrite throws away; the test-only +`paths.rs` prototype). The storm self-check above then runs against a router +from before the rewrite. + +Public API: none. Wire: none. + +## Required + +- [The router quest](/quest/m2/cluster-routing/routes/scope.md) - rewrites `route::Router` to the path rule, the router this searches + +## Related + +- [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself +- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - the other spike, a faster engine for the same specs diff --git a/rs/justfile b/rs/justfile index a211ef3fdc3..b57a995a81a 100644 --- a/rs/justfile +++ b/rs/justfile @@ -19,9 +19,12 @@ default: just check # Model-check a TLA+ spec with TLC, e.g. `just rs tla moq-net/verify/route/RouteHandoff.tla`. -# Pass `-config rs/.cfg` for a config not named after the spec. +# Pass `-config rs/.cfg` for a config not named after the spec, and +# `../.tla` for a spec outside `rs/`, such as a generated one in `.tmp/`. +# TLC runs at the lowest priority with a 4 GB heap, which no model here outgrows: +# the JVM otherwise takes a quarter of RAM per run. tla spec *args: - nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }} + JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }} # Prove a Tamarin model's lemmas, e.g. `just rs tamarin moq-net/verify/route-key/RouteKey.spthy`. # Pass `-D=` for a variant the model defines.