From e0e706c985e66c7969baf5dc3c0712a1e1332dc0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 19:45:09 -0800 Subject: [PATCH 1/2] quest: plan cheaper TLC runs and a cached JS API reference Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/tlc-cost.md | 89 +++++++++++++++++++++++++++++++++ quest/m1/api-reference-cache.md | 32 ++++++++++++ quest/m1/tla-native-checker.md | 1 + 3 files changed, 122 insertions(+) create mode 100644 quest/m0/ci/tlc-cost.md create mode 100644 quest/m1/api-reference-cache.md diff --git a/quest/m0/ci/tlc-cost.md b/quest/m0/ci/tlc-cost.md new file mode 100644 index 00000000000..690b1a4e88f --- /dev/null +++ b/quest/m0/ci/tlc-cost.md @@ -0,0 +1,89 @@ +# [M] TLC runs cost fewer core-seconds and share the host + +## Goal + +The `tla` recipes and their callers check every TLA+ config with the same +verdict as before for measurably fewer core-seconds, a TLC run leaves room on +the host for other gated commands, and no run, killed ones included, leaves +its working directory behind. Agents check the one config a change touches; +whole directories are Nightly's. + +## Plan + +Measured 2026-11-12 on the maintainer's agent hosts: + +- An ungated `just test tla rs/moq-net` matched one agent session's whole + usage: 21.4 CPU-hours over 1.8 hours. +- A gated `just test tla rs/moq-net/verify/route` took about 24 CPU-hours in + 53 minutes at about 31 cores, while 35 other gated commands queued behind + it, the oldest for 53 minutes. `quest gate` admits by load average, so one + run that takes every core stalls everything else. +- Another session ran TLC by hand, outside the recipes, for 2.5 hours: + 15.6 CPU-hours and 17.8 GB, on nixpkgs `tlaplus` 1.7.4 and a Java 8 JRE. +- Nightly splits these configs across five runners at about an hour each on + 4 workers. +- A killed run's `-metadir` under `/tmp` still held 26 GB. + +Decided 2026-11-12 (maintainer, /quest-plan): + +- ✅ Cap TLC's workers in `rs/justfile` and `js/justfile`, which hard-code + `-workers auto`. Recommended: 4, which the `verify-*` skills' times + already assume and Nightly's runners have; the + [native checker quest](/quest/m1/tla-native-checker.md) measured + core-seconds near flat from 2 to 24 workers, so the cap costs wall time, + not CPU. +- ✅ TLC 1.8 on a modern JDK. The recipes moved to `nixpkgs#tlaplus18` in + [moq-dev/moq#8499](https://github.com/moq-dev/moq/pull/8499); the 1.7.4 run + above bypassed them. Leave no doc, skill, or script pointing at the 1.7.4 + package, `nixpkgs#tlaplus`. +- ✅ Replace `SumOver` in `rs/moq-net/verify/route/RouteHandoff.tla`, a + recursive function over `SUBSET S` that TLC re-evaluates on every use, with + the Community Modules' `FoldSet` (`FiniteSetsExt`), folding `f(x)` over + `S`. Not `SumSet({f(x) : x \in S})`, which sums each value once and so drops + two nodes' equal counts. TLC evaluates `FoldSet` natively only through the + Java override in `CommunityModules-deps.jar`, which nixpkgs does not + package: take it from a pinned release, put it on TLC's classpath in both + recipes, and confirm TLC's startup log reports the override loaded. + [moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564) rewrites + `SumOver` as an Apalache fold; whichever lands second keeps one fold for the + model. +- ✅ Add `SYMMETRY` where sound. No config checks a `PROPERTY`, so every one + checks safety only, which symmetry preserves. A permutation is sound only if + it fixes every constant the config gives, such as `Links`, `Origin`, or + `Start`, and the spec never orders those values or does arithmetic on them; + so turn + a fully interchangeable set, such as `Relays` or `Endpoints`, into model + values and permute it, or the members no constant singles out. That can cut + a state space by up to n!, though `Endpoints = {"a", "b"}` with + `Start = "a"` gains nothing. Say per config whether it applies and why. +- ✅ Pass `-checkpoint 0`, and make sure no metadir outlives its run. TLC never + removes one it was killed in, and a signal handler cannot run on SIGKILL. + Recommended: put the metadir under the checkout's `.tmp/`, which goes with + the worktree, and remove it when the recipe exits. +- ✅ Agents check one config locally: the `verify-*` skills, `REVIEW.md`, and + the `js/*/verify/` READMEs name single configs, and only Nightly runs + `just test tla` over a directory. +- Rejected: Apalache, which is no cheaper here and ran out of memory on the + route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)); + waiting for the Rust checker, which the m1 quest + spikes and this complements. + +Not decided: the route-inductive work in progress adds a refinement check, +`RefinesClaims`, to the simulated `Blind.cfg`, where every sampled trace +re-checks it. Recommended: check a refinement on the smallest exhaustive config +that exercises it, where each distinct state is checked once, unless measuring +shows the simulated config is cheaper. + +Verification: every config keeps its verdict, and every exhaustive config +without `SYMMETRY` keeps its distinct-state count (the `verify-*` skills list them). +Measure each config's core-seconds through `quest gate` at equal worker counts +before and after, report the table in the PR, and update the times the skills +and Nightly's `tla` matrix comment quote; rebalance that matrix if its legs +change a lot. + +Public API: none. Wire: none. + +## Related + +- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - a faster engine for the same specs; its TLC baseline moves with this +- [The route rules hold from any state, not only the runs TLC reaches](/quest/m1/route-inductive.md) - rewrites `SumOver` and adds the refinement check named above diff --git a/quest/m1/api-reference-cache.md b/quest/m1/api-reference-cache.md new file mode 100644 index 00000000000..3c156d01c0e --- /dev/null +++ b/quest/m1/api-reference-cache.md @@ -0,0 +1,32 @@ +# [XS] The JS API reference regenerates only when its inputs change + +## Goal + +`doc/.vitepress/api/index.ts` skips TypeDoc when a hash of everything the +reference is built from matches the last run's, in the docs build and the dev +server alike, so a JS-module check that changes no input spends no TypeDoc +time. + +## Plan + +Measured 2026-11-12 on one of the maintainer's agent hosts: the reference +regenerated on every JS-module check (`doc`'s `check` script ends in +`vitepress build`, which never reuses it), 50 runs in 4.8 hours at a median +of 74 CPU-seconds, about 6 CPU-hours a day. + +Decided 2026-11-12 (maintainer, /quest-plan): ✅ skip when a hash of the +inputs is unchanged. + +- The hash replaces the `--cached` modification-time stamp, which every fresh + checkout defeats; one rule serves build and dev server. Keep + `MOQ_API_REFERENCE=regenerate` only if something still needs it. +- Inputs are what can change the output: the published packages' sources and + `package.json`s, their tsconfigs, the generator under + `doc/.vitepress/api/`, and the TypeDoc versions and patches in `bun.lock` + and `patches/`. A missed input serves a stale reference, so derive the + list from what `generate` reads rather than by hand where possible. +- Recommended: keep the output keyed by its hash in a cache outside the + checkout, written atomically, so each new agent worktree reuses it rather + than regenerating once per worktree. Measure the hit rate across worktrees. + +Public API: none. Wire: none. diff --git a/quest/m1/tla-native-checker.md b/quest/m1/tla-native-checker.md index 0af2acd2669..6c0b053165f 100644 --- a/quest/m1/tla-native-checker.md +++ b/quest/m1/tla-native-checker.md @@ -64,3 +64,4 @@ Public API: none. Wire: none. ## Related - [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routing/routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model +- [TLC runs cost fewer core-seconds and share the host](/quest/m0/ci/tlc-cost.md) - makes TLC itself cheaper, which moves this quest's baseline From 36eff2b7db0d7ad1a11865a56305763849e5e13b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 20:42:02 -0800 Subject: [PATCH 2/2] quest: clean stale TLC metadirs and hash inherited tsconfigs Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/tlc-cost.md | 8 ++++++-- quest/m1/api-reference-cache.md | 3 ++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/quest/m0/ci/tlc-cost.md b/quest/m0/ci/tlc-cost.md index 690b1a4e88f..0728361d3b7 100644 --- a/quest/m0/ci/tlc-cost.md +++ b/quest/m0/ci/tlc-cost.md @@ -58,8 +58,12 @@ Decided 2026-11-12 (maintainer, /quest-plan): `Start = "a"` gains nothing. Say per config whether it applies and why. - ✅ Pass `-checkpoint 0`, and make sure no metadir outlives its run. TLC never removes one it was killed in, and a signal handler cannot run on SIGKILL. - Recommended: put the metadir under the checkout's `.tmp/`, which goes with - the worktree, and remove it when the recipe exits. + Recommended: put the metadir under the checkout's `.tmp/`, remove it when + the recipe exits, and at start remove the metadirs earlier runs left there + that no live run owns, so a checkout that outlives killed runs does not + collect them. Mark ownership so a concurrent run's metadir survives, such + as a lock each run holds on a file in its metadir, which the kernel drops + when the run dies. - ✅ Agents check one config locally: the `verify-*` skills, `REVIEW.md`, and the `js/*/verify/` READMEs name single configs, and only Nightly runs `just test tla` over a directory. diff --git a/quest/m1/api-reference-cache.md b/quest/m1/api-reference-cache.md index 3c156d01c0e..c50aa47656e 100644 --- a/quest/m1/api-reference-cache.md +++ b/quest/m1/api-reference-cache.md @@ -21,7 +21,8 @@ inputs is unchanged. checkout defeats; one rule serves build and dev server. Keep `MOQ_API_REFERENCE=regenerate` only if something still needs it. - Inputs are what can change the output: the published packages' sources and - `package.json`s, their tsconfigs, the generator under + `package.json`s, their tsconfigs and every config those reach through + `extends`, such as `js/tsconfig.json`, the generator under `doc/.vitepress/api/`, and the TypeDoc versions and patches in `bun.lock` and `patches/`. A missed input serves a stale reference, so derive the list from what `generate` reads rather than by hand where possible.