Repository navigation
chore: TLC runs on Java 21 at low priority, and plan two faster-checking spikes #8499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
eb65d86
chore: TLC runs on Java 21 at low priority, and plan two faster-check…
kixelated 05558f1
Merge main: gate the new verify skills' runs, and refresh the two spi…
kixelated 3d302aa
quest: record the pick round's answers for the TLC spikes
kixelated 28028e2
quest: settle coderabbit findings on the two spikes' bounds and agree…
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # [M] A Rust TLA+ checker runs the models locally if it agrees with TLC | ||
|
|
||
| ## Goal | ||
|
|
||
| The `tla` recipes run a native TLA+ checker locally when one gives TLC's | ||
| verdict and distinct-state count on every config in the tree it can run | ||
| (the rest stay on TLC, as the Plan says) and costs fewer | ||
| core-seconds on the route and resume models. Nightly (`just test tla`, which | ||
| `sh/tla.sh` drives) then runs both engines on every config and fails when they | ||
| disagree. If no checker agrees, the PR | ||
| records what each one failed on and adopts nothing. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-10 (maintainer, /quest-plan), as a spike: TLC interprets a | ||
| spec on the JVM, and the route model's 4-node runs take tens of minutes of | ||
| many cores. | ||
|
|
||
| - ✅ A checker that agrees runs locally, and Nightly cross-checks it against | ||
| TLC (rejected: local runs with a TLC-only Nightly, where a checker bump or | ||
| spec change could make it wrong unnoticed; replacing TLC everywhere, where | ||
| nothing checks the checker again; reporting the numbers and adopting | ||
| nothing). | ||
|
|
||
| Candidates found that day, each young, at least one AI-written, and benchmarked | ||
| only by its authors: | ||
|
|
||
| - [tla-mc](https://github.com/ddalton/tla-mc) compiles a spec to a native | ||
| parallel checker and claims about 4x TLC at 8 workers. | ||
| - [tlaplusplus](https://github.com/zoratu/tlaplusplus) claims about 10x on | ||
| many cores and passes 174 of 182 of tlaplus/Examples. | ||
| - [tla-rs](https://github.com/bbcvc/tla-rs) claims state-for-state agreement | ||
| with TLC on 823 recorded runs and 2x to 3x on large models with one worker. | ||
| - [tlx](https://github.com/Medfin-AI/tlx) reports some models slower than TLC. | ||
|
|
||
| A checker bug shows as a rule passing that should break, so agreement is the | ||
| bar, not speed: | ||
|
|
||
| - Every config the checker runs: the same verdict as TLC, and the same broken | ||
| rule for an open-finding config. A passing config also matches TLC's | ||
| distinct-state count; a violating one does not, since each engine stops at | ||
| its first violation after exploring a different, order-dependent part of | ||
| the space. The specs use | ||
| `RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`, | ||
| `CONSTRAINT`, and `VIEW`, unmodified. | ||
| - Measure both engines at equal worker counts through `quest gate` on a quiet | ||
| host, against TLC as the `tla` recipes run it. The 2026-11-10 baseline, on a | ||
| 24-core desktop and the route model as it stood then (3.67M distinct | ||
| states): about 50 s and 930 CPU-seconds on TLC 1.8.0 with Java 21, down from | ||
| about 118 s and 1,730 on TLC 1.7.4 with Java 8, with CPU-seconds near flat | ||
| from 2 to 24 workers. The model has since changed, so remeasure TLC and | ||
| report the table in the PR. | ||
| - A config the checker cannot run, such as one with a `\* Simulate:` line | ||
| that `sh/tla.sh` samples at random, stays on TLC; the recipe picks the engine per config and errors rather than | ||
| falling back at runtime. | ||
| - A broken rule must still print a trace as readable as TLC's, which the | ||
| Reading a failure in `rs/moq-net/verify/README.md` relies on. Say how. | ||
| - Build each candidate from a flake input pinned to a revision, inside the Nix | ||
| sandbox. Run none of their build scripts or binaries outside it. | ||
| - Report upstream bugs to the maintainer rather than filing them. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routing/routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # [M] Every delivery order of a small mesh runs on the real route layer | ||
|
|
||
| ## Goal | ||
|
|
||
| A test explores every order in which `route::tests::Sim` can deliver, hold, | ||
| disconnect, collect, and change claims on a triangle of the origin and two | ||
| relays, on the real `route::Router`, and checks the route rules in every state | ||
| it reaches. A broken rule prints the path to it as `Sim` calls. The PR reports | ||
| its states and time beside TLC's for the same bounds. If the real state space | ||
| does not fit, the PR records the numbers and lands no test. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-10 (maintainer, /quest-plan), as a spike: `RouteHandoff.tla` | ||
| is a hand-kept mirror of the rule `route/tests/paths.rs` prototypes, which TLC interprets, while `Sim` | ||
| already drives the router one atomic step at a time. Searching `Sim` itself | ||
| runs compiled code, has no model to drift, and reaches what the model leaves | ||
| out. | ||
|
|
||
| - ✅ A hand-written breadth-first search beside `Sim` in `route/tests/`: a | ||
| fingerprint set, a queue, and a parent map for the path (rejected: the | ||
| `stateright` crate, whose 12 direct dependencies include dashmap, rand, | ||
| serde_json, and tiny_http; building both and keeping the better). Add | ||
| threads or symmetry only if the measurements call for them. | ||
| - Bound the run as `RouteHandoff.cfg` does (`MaxRev`, `MaxFlaps`, one | ||
| claim), but on the triangle: `RouteHandoff.cfg` has four nodes, so the PR | ||
| runs TLC on a copy of that config cut to the same three nodes, and the two | ||
| counts compare. | ||
| - The search converges only if equal situations hash equal. `Sim` and `Router` | ||
| need a clone and a fingerprint, and the fingerprint must leave out what only | ||
| records the past, such as `events`, `stats`, and `requested`. Check each | ||
| rule at the step that would break it rather than from that log. | ||
| - The check that the search works: run on today's router before the switch to | ||
| the path rule, it must find the storm | ||
| `residual_a_seqno_advance_storms_between_two_grants` replays, or another | ||
| `residual_` test in `route/tests/residual.rs`. Those fixtures use 6 to 10 | ||
| nodes, beyond the triangle, so the self-check searches the fixture's own | ||
| mesh, bounded by depth and by the fixture's scripted claims, rather than | ||
| the triangle's bounds; the search code takes the mesh as a parameter. | ||
| - Then say which of the model's "Not covered" list (`verify-route` skill) the | ||
| search can afford: wildcards, scoped grants, costs other than 1, a fourth | ||
| node. | ||
| - A search that takes seconds joins the merge gate; a larger one runs in | ||
| Nightly. | ||
|
|
||
| This sits beside [trace validation](/quest/m1/route-trace-validation.md), which | ||
| keeps the TLA+ model honest for [the inductive invariant](/quest/m1/route-inductive.md) | ||
| and the design work done on the spec before code exists. It replaces neither | ||
| that nor the seeded sweeps on larger meshes | ||
| (`tests/integration/mesh_sweep.rs`). If the spike shows the | ||
| search covers what a model check does, recommend in the PR what to do with the | ||
| route model's TLC runs. | ||
|
|
||
| Decided 2026-11-12 (maintainer, pick round): the search runs on the `Router` | ||
| after [the router quest](/quest/m2/cluster-routing/routes/scope.md)'s rewrite to | ||
| the path rule, so it checks the code that ships and nothing is built twice | ||
| (rejected: today's Babel `Router`, which the rewrite throws away; the test-only | ||
| `paths.rs` prototype). The storm self-check above then runs against a router | ||
| from before the rewrite. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Required | ||
|
|
||
| - [The router quest](/quest/m2/cluster-routing/routes/scope.md) - rewrites `route::Router` to the path rule, the router this searches | ||
|
|
||
| ## Related | ||
|
|
||
| - [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself | ||
| - [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - the other spike, a faster engine for the same specs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.