Repository navigation
quest: plan cheaper TLC runs and a cached JS API reference #8620
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| # [M] TLC runs cost fewer core-seconds and share the host | ||
|
|
||
| ## Goal | ||
|
|
||
| The `tla` recipes and their callers check every TLA+ config with the same | ||
| verdict as before for measurably fewer core-seconds, a TLC run leaves room on | ||
| the host for other gated commands, and no run, killed ones included, leaves | ||
| its working directory behind. Agents check the one config a change touches; | ||
| whole directories are Nightly's. | ||
|
|
||
| ## Plan | ||
|
|
||
| Measured 2026-11-12 on the maintainer's agent hosts: | ||
|
|
||
| - An ungated `just test tla rs/moq-net` matched one agent session's whole | ||
| usage: 21.4 CPU-hours over 1.8 hours. | ||
| - A gated `just test tla rs/moq-net/verify/route` took about 24 CPU-hours in | ||
| 53 minutes at about 31 cores, while 35 other gated commands queued behind | ||
| it, the oldest for 53 minutes. `quest gate` admits by load average, so one | ||
| run that takes every core stalls everything else. | ||
| - Another session ran TLC by hand, outside the recipes, for 2.5 hours: | ||
| 15.6 CPU-hours and 17.8 GB, on nixpkgs `tlaplus` 1.7.4 and a Java 8 JRE. | ||
| - Nightly splits these configs across five runners at about an hour each on | ||
| 4 workers. | ||
| - A killed run's `-metadir` under `/tmp` still held 26 GB. | ||
|
|
||
| Decided 2026-11-12 (maintainer, /quest-plan): | ||
|
|
||
| - ✅ Cap TLC's workers in `rs/justfile` and `js/justfile`, which hard-code | ||
| `-workers auto`. Recommended: 4, which the `verify-*` skills' times | ||
| already assume and Nightly's runners have; the | ||
| [native checker quest](/quest/m1/tla-native-checker.md) measured | ||
| core-seconds near flat from 2 to 24 workers, so the cap costs wall time, | ||
| not CPU. | ||
| - ✅ TLC 1.8 on a modern JDK. The recipes moved to `nixpkgs#tlaplus18` in | ||
| [moq-dev/moq#8499](https://github.com/moq-dev/moq/pull/8499); the 1.7.4 run | ||
| above bypassed them. Leave no doc, skill, or script pointing at the 1.7.4 | ||
| package, `nixpkgs#tlaplus`. | ||
| - ✅ Replace `SumOver` in `rs/moq-net/verify/route/RouteHandoff.tla`, a | ||
| recursive function over `SUBSET S` that TLC re-evaluates on every use, with | ||
| the Community Modules' `FoldSet` (`FiniteSetsExt`), folding `f(x)` over | ||
| `S`. Not `SumSet({f(x) : x \in S})`, which sums each value once and so drops | ||
| two nodes' equal counts. TLC evaluates `FoldSet` natively only through the | ||
| Java override in `CommunityModules-deps.jar`, which nixpkgs does not | ||
| package: take it from a pinned release, put it on TLC's classpath in both | ||
| recipes, and confirm TLC's startup log reports the override loaded. | ||
| [moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564) rewrites | ||
| `SumOver` as an Apalache fold; whichever lands second keeps one fold for the | ||
| model. | ||
| - ✅ Add `SYMMETRY` where sound. No config checks a `PROPERTY`, so every one | ||
| checks safety only, which symmetry preserves. A permutation is sound only if | ||
| it fixes every constant the config gives, such as `Links`, `Origin`, or | ||
| `Start`, and the spec never orders those values or does arithmetic on them; | ||
| so turn | ||
| a fully interchangeable set, such as `Relays` or `Endpoints`, into model | ||
| values and permute it, or the members no constant singles out. That can cut | ||
| a state space by up to n!, though `Endpoints = {"a", "b"}` with | ||
| `Start = "a"` gains nothing. Say per config whether it applies and why. | ||
| - ✅ Pass `-checkpoint 0`, and make sure no metadir outlives its run. TLC never | ||
| removes one it was killed in, and a signal handler cannot run on SIGKILL. | ||
| Recommended: put the metadir under the checkout's `.tmp/`, remove it when | ||
| the recipe exits, and at start remove the metadirs earlier runs left there | ||
| that no live run owns, so a checkout that outlives killed runs does not | ||
| collect them. Mark ownership so a concurrent run's metadir survives, such | ||
| as a lock each run holds on a file in its metadir, which the kernel drops | ||
| when the run dies. | ||
| - ✅ Agents check one config locally: the `verify-*` skills, `REVIEW.md`, and | ||
| the `js/*/verify/` READMEs name single configs, and only Nightly runs | ||
| `just test tla` over a directory. | ||
| - Rejected: Apalache, which is no cheaper here and ran out of memory on the | ||
| route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)); | ||
| waiting for the Rust checker, which the m1 quest | ||
| spikes and this complements. | ||
|
|
||
| Not decided: the route-inductive work in progress adds a refinement check, | ||
| `RefinesClaims`, to the simulated `Blind.cfg`, where every sampled trace | ||
| re-checks it. Recommended: check a refinement on the smallest exhaustive config | ||
| that exercises it, where each distinct state is checked once, unless measuring | ||
| shows the simulated config is cheaper. | ||
|
|
||
| Verification: every config keeps its verdict, and every exhaustive config | ||
| without `SYMMETRY` keeps its distinct-state count (the `verify-*` skills list them). | ||
| Measure each config's core-seconds through `quest gate` at equal worker counts | ||
| before and after, report the table in the PR, and update the times the skills | ||
| and Nightly's `tla` matrix comment quote; rebalance that matrix if its legs | ||
| change a lot. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - a faster engine for the same specs; its TLC baseline moves with this | ||
| - [The route rules hold from any state, not only the runs TLC reaches](/quest/m1/route-inductive.md) - rewrites `SumOver` and adds the refinement check named above |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # [XS] The JS API reference regenerates only when its inputs change | ||
|
|
||
| ## Goal | ||
|
|
||
| `doc/.vitepress/api/index.ts` skips TypeDoc when a hash of everything the | ||
| reference is built from matches the last run's, in the docs build and the dev | ||
| server alike, so a JS-module check that changes no input spends no TypeDoc | ||
| time. | ||
|
|
||
| ## Plan | ||
|
|
||
| Measured 2026-11-12 on one of the maintainer's agent hosts: the reference | ||
| regenerated on every JS-module check (`doc`'s `check` script ends in | ||
| `vitepress build`, which never reuses it), 50 runs in 4.8 hours at a median | ||
| of 74 CPU-seconds, about 6 CPU-hours a day. | ||
|
|
||
| Decided 2026-11-12 (maintainer, /quest-plan): ✅ skip when a hash of the | ||
| inputs is unchanged. | ||
|
|
||
| - The hash replaces the `--cached` modification-time stamp, which every fresh | ||
| checkout defeats; one rule serves build and dev server. Keep | ||
| `MOQ_API_REFERENCE=regenerate` only if something still needs it. | ||
| - Inputs are what can change the output: the published packages' sources and | ||
| `package.json`s, their tsconfigs and every config those reach through | ||
| `extends`, such as `js/tsconfig.json`, the generator under | ||
| `doc/.vitepress/api/`, and the TypeDoc versions and patches in `bun.lock` | ||
| and `patches/`. A missed input serves a stale reference, so derive the | ||
| list from what `generate` reads rather than by hand where possible. | ||
| - Recommended: keep the output keyed by its hash in a cache outside the | ||
| checkout, written atomically, so each new agent worktree reuses it rather | ||
| than regenerating once per worktree. Measure the hit rate across worktrees. | ||
|
|
||
| Public API: none. Wire: none. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.