From 53239029dc84774a08f54c515e9186b5fb2c9014 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 12 Nov 2026 16:18:16 -0800 Subject: [PATCH 1/4] chore(quest): claim quest/m1/tla-native-checker.md Quest-Claim: 9ab31f91a93eb531 From 0b8c93a88cee93532856b6ecb1189157a545b161 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 12 Nov 2026 22:50:48 -0800 Subject: [PATCH 2/4] chore(quest): no Rust TLA+ checker agrees with TLC on the route model tla-rs, tla-mc, tlx, and tlaplusplus were each run against TLC on every exhaustive config (moq-dev/moq#8600). None checks the route model within TLC's cost and memory, so nothing is adopted. A follow-up rechecks tla-mc, the closest, once it parses the route spec and names actions in traces. Co-Authored-By: Claude Opus 5.5 --- quest/m1/tla-native-checker.md | 66 ------------------- .../routes/route-exhaustive-sim.md | 1 - quest/m2/tla-native-recheck.md | 41 ++++++++++++ 3 files changed, 41 insertions(+), 67 deletions(-) delete mode 100644 quest/m1/tla-native-checker.md create mode 100644 quest/m2/tla-native-recheck.md diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md deleted file mode 100644 index 0af2acd2669..00000000000 --- a/quest/m1/tla-native-checker.md +++ /dev/null @@ -1,66 +0,0 @@ -# [M] A Rust TLA+ checker runs the models locally if it agrees with TLC - -## Goal - -The `tla` recipes run a native TLA+ checker locally when one gives TLC's -verdict and distinct-state count on every config in the tree it can run -(the rest stay on TLC, as the Plan says) and costs fewer -core-seconds on the route and resume models. Nightly (`just test tla`, which -`sh/tla.sh` drives) then runs both engines on every config and fails when they -disagree. If no checker agrees, the PR -records what each one failed on and adopts nothing. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan), as a spike: TLC interprets a -spec on the JVM, and the route model's 4-node runs take tens of minutes of -many cores. - -- ✅ A checker that agrees runs locally, and Nightly cross-checks it against - TLC (rejected: local runs with a TLC-only Nightly, where a checker bump or - spec change could make it wrong unnoticed; replacing TLC everywhere, where - nothing checks the checker again; reporting the numbers and adopting - nothing). - -Candidates found that day, each young, at least one AI-written, and benchmarked -only by its authors: - -- [tla-mc](https://github.com/ddalton/tla-mc) compiles a spec to a native - parallel checker and claims about 4x TLC at 8 workers. -- [tlaplusplus](https://github.com/zoratu/tlaplusplus) claims about 10x on - many cores and passes 174 of 182 of tlaplus/Examples. -- [tla-rs](https://github.com/bbcvc/tla-rs) claims state-for-state agreement - with TLC on 823 recorded runs and 2x to 3x on large models with one worker. -- [tlx](https://github.com/Medfin-AI/tlx) reports some models slower than TLC. - -A checker bug shows as a rule passing that should break, so agreement is the -bar, not speed: - -- Every config the checker runs: the same verdict as TLC, and the same broken - rule for an open-finding config. A passing config also matches TLC's - distinct-state count; a violating one does not, since each engine stops at - its first violation after exploring a different, order-dependent part of - the space. The specs use - `RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`, - `CONSTRAINT`, and `VIEW`, unmodified. -- Measure both engines at equal worker counts through `quest gate` on a quiet - host, against TLC as the `tla` recipes run it. The 2026-11-10 baseline, on a - 24-core desktop and the route model as it stood then (3.67M distinct - states): about 50 s and 930 CPU-seconds on TLC 1.8.0 with Java 21, down from - about 118 s and 1,730 on TLC 1.7.4 with Java 8, with CPU-seconds near flat - from 2 to 24 workers. The model has since changed, so remeasure TLC and - report the table in the PR. -- A config the checker cannot run, such as one with a `\* Simulate:` line - that `sh/tla.sh` samples at random, stays on TLC; the recipe picks the engine per config and errors rather than - falling back at runtime. -- A broken rule must still print a trace as readable as TLC's, which the - Reading a failure in `rs/moq-net/verify/README.md` relies on. Say how. -- Build each candidate from a flake input pinned to a revision, inside the Nix - sandbox. Run none of their build scripts or binaries outside it. -- Report upstream bugs to the maintainer rather than filing them. - -Public API: none. Wire: none. - -## Related - -- [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routing/routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model diff --git a/quest/m2/cluster-routing/routes/route-exhaustive-sim.md b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md index 0a11a1c296a..3e09f97dadb 100644 --- a/quest/m2/cluster-routing/routes/route-exhaustive-sim.md +++ b/quest/m2/cluster-routing/routes/route-exhaustive-sim.md @@ -67,4 +67,3 @@ Public API: none. Wire: none. ## Related - [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself -- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - the other spike, a faster engine for the same specs diff --git a/quest/m2/tla-native-recheck.md b/quest/m2/tla-native-recheck.md new file mode 100644 index 00000000000..31fa33b490d --- /dev/null +++ b/quest/m2/tla-native-recheck.md @@ -0,0 +1,41 @@ +# [M] tla-mc runs the TLA+ models locally once it checks the route model + +## Goal + +The `tla` recipes run [tla-mc](https://github.com/ddalton/tla-mc) on every +exhaustive config once a pinned revision checks the route model, prints a +trace that names each step's action, and keeps TLC's verdict and +distinct-state count on every config. Nightly then runs both engines on every +config and fails when they disagree, as the +[first comparison](https://github.com/moq-dev/moq/pull/8600) planned. + +## Plan + +[moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600) compared four +Rust checkers against TLC 1.8.0 on every exhaustive config and adopted none. +tla-mc agreed with TLC everywhere it ran and cost a third of TLC's CPU on the +resume model (672 against 2,057 CPU-s on `ResumeReader.cfg`), but: + +- It rejects `RouteHandoff.tla` with `unknown operator f`, where an operator + argument is passed on to another operator + (`Sum(f(_)) == SumOver(ns, f)`), so it runs no route config. +- Its counterexample lists states without the action that led to each, which + the verify skills' replay steps read (`` in TLC). + +Recheck both at tla-mc's newest revision, built from a pinned flake input in +the Nix sandbox and run there under a memory cap, as #8600 did. If both are +fixed, rerun #8600's sweep on the route and resume configs and adopt it when +it agrees and costs fewer CPU-seconds than TLC on both. Try `-compile` too: +it builds the generated checker with `cargo` at run time, so it needs the +generated crate built as a Nix derivation instead (`-codegen DIR`). + +Public API: none. Wire: none. + +## Waiting + +- Maintainer: report the two tla-mc gaps above upstream (the quest's rule is + to report upstream bugs to the maintainer rather than file them); done when + issues exist on ddalton/tla-mc +- ddalton/tla-mc: support passing an operator argument to another operator, + and name each step's action in a trace; check by running a newer revision on + `rs/moq-net/verify/route/RouteHandoff.cfg` and `EndedRelay.cfg` From e6a1356c3c06e5aff1af3616d6de6a258f8352b3 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 12 Nov 2026 22:56:53 -0800 Subject: [PATCH 3/4] chore(quest): recheck the Rust checkers once the route model drops SumOver TLC's cost quest, new on main, replaces SumOver, the construct tla-mc rejects and the likely cost behind tlx's and tla-rs's route blowups, so the follow-up waits on it rather than on tla-mc alone. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/tlc-cost.md | 2 +- quest/m2/tla-native-recheck.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/quest/m0/ci/tlc-cost.md b/quest/m0/ci/tlc-cost.md index 4133cfd8934..349a5c1b6c0 100644 --- a/quest/m0/ci/tlc-cost.md +++ b/quest/m0/ci/tlc-cost.md @@ -89,5 +89,5 @@ Public API: none. Wire: none. ## Related -- [A Rust TLA+ checker is rechecked once the route model drops `SumOver`](/quest/m2/tla-native-recheck.md) - a faster engine for the same specs, waiting on this quest's `SumOver` rewrite +- [A Rust TLA+ checker is rechecked once the route model drops SumOver](/quest/m2/tla-native-recheck.md) - a faster engine for the same specs, waiting on this quest's `SumOver` rewrite - [The route rules hold from any state, not only the runs TLC reaches](/quest/m1/route-inductive.md) - rewrites `SumOver` and adds the refinement check named above diff --git a/quest/m2/tla-native-recheck.md b/quest/m2/tla-native-recheck.md index 9491e1828f3..8cb4fbd7c44 100644 --- a/quest/m2/tla-native-recheck.md +++ b/quest/m2/tla-native-recheck.md @@ -1,4 +1,4 @@ -# [M] A Rust TLA+ checker is rechecked once the route model drops `SumOver` +# [M] A Rust TLA+ checker is rechecked once the route model drops SumOver ## Goal From 2122acd974e3e72ea0da3df60258a60b7feb1d08 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 23:06:22 -0800 Subject: [PATCH 4/4] chore(quest): the recheck covers every exhaustive config, route and resume first Co-Authored-By: Claude Opus 5.5 --- quest/m2/tla-native-recheck.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/quest/m2/tla-native-recheck.md b/quest/m2/tla-native-recheck.md index 8cb4fbd7c44..31e4e9b2ee7 100644 --- a/quest/m2/tla-native-recheck.md +++ b/quest/m2/tla-native-recheck.md @@ -24,7 +24,8 @@ model: `SumOver` is a recursive function over `SUBSET S`, the likely cost for both, and [TLC's cost quest](/quest/m0/ci/tlc-cost.md) replaces it with `FoldSet`. Once that lands, rerun #8600's comparison at each checker's newest -revision, on route and resume first: +revision: route and resume first, since they decide cost, then every +other exhaustive config, since adoption needs agreement on all of them: - Build each from a pinned flake input inside the Nix sandbox, and run each check there as a derivation under `prlimit --as=16G`, one at a time through