diff --git a/quest/m0/ci/tlc-cost.md b/quest/m0/ci/tlc-cost.md index 0728361d3b7..349a5c1b6c0 100644 --- a/quest/m0/ci/tlc-cost.md +++ b/quest/m0/ci/tlc-cost.md @@ -28,8 +28,8 @@ Decided 2026-11-12 (maintainer, /quest-plan): - ✅ Cap TLC's workers in `rs/justfile` and `js/justfile`, which hard-code `-workers auto`. Recommended: 4, which the `verify-*` skills' times - already assume and Nightly's runners have; the - [native checker quest](/quest/m1/tla-native-checker.md) measured + already assume and Nightly's runners have; the native checker spike + ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600)) measured core-seconds near flat from 2 to 24 workers, so the cap costs wall time, not CPU. - ✅ TLC 1.8 on a modern JDK. The recipes moved to `nixpkgs#tlaplus18` in @@ -69,8 +69,8 @@ Decided 2026-11-12 (maintainer, /quest-plan): `just test tla` over a directory. - Rejected: Apalache, which is no cheaper here and ran out of memory on the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)); - waiting for the Rust checker, which the m1 quest - spikes and this complements. + waiting for a Rust checker, none of which ran the route model in + [moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600). Not decided: the route-inductive work in progress adds a refinement check, `RefinesClaims`, to the simulated `Blind.cfg`, where every sampled trace @@ -89,5 +89,5 @@ Public API: none. Wire: none. ## Related -- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - a faster engine for the same specs; its TLC baseline moves with this +- [A Rust TLA+ checker is rechecked once the route model drops SumOver](/quest/m2/tla-native-recheck.md) - a faster engine for the same specs, waiting on this quest's `SumOver` rewrite - [The route rules hold from any state, not only the runs TLC reaches](/quest/m1/route-inductive.md) - rewrites `SumOver` and adds the refinement check named above diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md deleted file mode 100644 index 1b105446786..00000000000 --- a/quest/m1/tla-native-checker.md +++ /dev/null @@ -1,67 +0,0 @@ -# [M] A Rust TLA+ checker runs the models locally if it agrees with TLC - -## Goal - -The `tla` recipes run a native TLA+ checker locally when one gives TLC's -verdict and distinct-state count on every config in the tree it can run -(the rest stay on TLC, as the Plan says) and costs fewer -core-seconds on the route and resume models. Nightly (`just test tla`, which -`sh/tla.sh` drives) then runs both engines on every config and fails when they -disagree. If no checker agrees, the PR -records what each one failed on and adopts nothing. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan), as a spike: TLC interprets a -spec on the JVM, and the route model's 4-node runs take tens of minutes of -many cores. - -- ✅ A checker that agrees runs locally, and Nightly cross-checks it against - TLC (rejected: local runs with a TLC-only Nightly, where a checker bump or - spec change could make it wrong unnoticed; replacing TLC everywhere, where - nothing checks the checker again; reporting the numbers and adopting - nothing). - -Candidates found that day, each young, at least one AI-written, and benchmarked -only by its authors: - -- [tla-mc](https://github.com/ddalton/tla-mc) compiles a spec to a native - parallel checker and claims about 4x TLC at 8 workers. -- [tlaplusplus](https://github.com/zoratu/tlaplusplus) claims about 10x on - many cores and passes 174 of 182 of tlaplus/Examples. -- [tla-rs](https://github.com/bbcvc/tla-rs) claims state-for-state agreement - with TLC on 823 recorded runs and 2x to 3x on large models with one worker. -- [tlx](https://github.com/Medfin-AI/tlx) reports some models slower than TLC. - -A checker bug shows as a rule passing that should break, so agreement is the -bar, not speed: - -- Every config the checker runs: the same verdict as TLC, and the same broken - rule for an open-finding config. A passing config also matches TLC's - distinct-state count; a violating one does not, since each engine stops at - its first violation after exploring a different, order-dependent part of - the space. The specs use - `RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`, - `CONSTRAINT`, and `VIEW`, unmodified. -- Measure both engines at equal worker counts through `quest gate` on a quiet - host, against TLC as the `tla` recipes run it. The 2026-11-10 baseline, on a - 24-core desktop and the route model as it stood then (3.67M distinct - states): about 50 s and 930 CPU-seconds on TLC 1.8.0 with Java 21, down from - about 118 s and 1,730 on TLC 1.7.4 with Java 8, with CPU-seconds near flat - from 2 to 24 workers. The model has since changed, so remeasure TLC and - report the table in the PR. -- A config the checker cannot run, such as one with a `\* Simulate:` line - that `sh/tla.sh` samples at random, stays on TLC; the recipe picks the engine per config and errors rather than - falling back at runtime. -- A broken rule must still print a trace as readable as TLC's, which the - Reading a failure in `rs/moq-net/verify/README.md` relies on. Say how. -- Build each candidate from a flake input pinned to a revision, inside the Nix - sandbox. Run none of their build scripts or binaries outside it. -- Report upstream bugs to the maintainer rather than filing them. - -Public API: none. Wire: none. - -## Related - -- [TLC runs cost fewer core-seconds and share the host](/quest/m0/ci/tlc-cost.md) - makes TLC itself cheaper, which moves this quest's baseline -- [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model diff --git a/quest/m2/cluster-routes/route-exhaustive-sim.md b/quest/m2/cluster-routes/route-exhaustive-sim.md index c57e9700a79..db0a00f04d1 100644 --- a/quest/m2/cluster-routes/route-exhaustive-sim.md +++ b/quest/m2/cluster-routes/route-exhaustive-sim.md @@ -67,4 +67,3 @@ Public API: none. Wire: none. ## Related - [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself -- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - the other spike, a faster engine for the same specs diff --git a/quest/m2/tla-native-recheck.md b/quest/m2/tla-native-recheck.md new file mode 100644 index 00000000000..31e4e9b2ee7 --- /dev/null +++ b/quest/m2/tla-native-recheck.md @@ -0,0 +1,44 @@ +# [M] A Rust TLA+ checker is rechecked once the route model drops SumOver + +## Goal + +The `tla` recipes run a native TLA+ checker locally when one gives TLC's +verdict and distinct-state count on every exhaustive config and costs fewer +core-seconds than TLC on the route and resume models; Nightly then runs both +engines on every config and fails when they disagree. If none does, the PR +records what each failed on and adopts nothing. + +## Plan + +[moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600) compared tlx, +tla-mc, tla-rs, and tlaplusplus with TLC 1.8.0 on every exhaustive config and +adopted none. Each agreed with TLC wherever it finished, and tlx and tla-mc +cost a fifth to a third of TLC's CPU on the resume model, but on the route +model: + +- tla-mc rejects `RouteHandoff.tla` (`unknown operator f`) where `SumOver`'s + operator argument is passed on (`Sum(f(_)) == SumOver(ns, f)`). +- tlx and tla-rs exceed a 16 GB cap where TLC fits in 4 GB, tlx after + finding 5% to 50% of TLC's states. + +`SumOver` is a recursive function over `SUBSET S`, the likely cost for both, +and [TLC's cost quest](/quest/m0/ci/tlc-cost.md) replaces it with +`FoldSet`. Once that lands, rerun #8600's comparison at each checker's newest +revision: route and resume first, since they decide cost, then every +other exhaustive config, since adoption needs agreement on all of them: + +- Build each from a pinned flake input inside the Nix sandbox, and run each + check there as a derivation under `prlimit --as=16G`, one at a time through + `quest gate`; an uncapped tla-rs run took the host's memory in #8600. +- Every config a checker runs gives TLC's verdict and broken rule, and a + passing one TLC's distinct-state count. `Simulate:` configs stay on TLC, + picked per config by the recipe. +- A broken rule prints TLC's trace format, with each step's action, which the + verify skills' replay steps read; tla-mc prints states alone. +- Report upstream bugs to the maintainer rather than filing them. + +Public API: none. Wire: none. + +## Required + +- [TLC runs cost fewer core-seconds and share the host](/quest/m0/ci/tlc-cost.md) - replaces `SumOver` in the route model, the construct every candidate failed on, and moves TLC's baseline