Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .claude/skills/verify-failover/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ Each rule is an INVARIANT, except `Settles`, which is TLC's deadlock check: ever
## Run

```sh
just rs tla moq-net/verify/failover/FailoverChain.tla
just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Leave.cfg
just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/EndedRelay.cfg
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/Leave.cfg
quest gate just rs tla moq-net/verify/failover/FailoverChain.tla -config rs/moq-net/verify/failover/EndedRelay.cfg
```

The first checks every rule with `E` failing over between `A` and `B`, one death, and one switch per relay, with a client that stays, so a deadlock is a playback hang: about 2.0M states. The second lets the client leave at any point, so a deadlock is a relay still holding a copy after it left; leaving is always possible there, which is why it cannot stand in for the first: about 2.2M states. Each takes seconds and passes at the sync commit. The third adds `B`'s route through `A`, so the route also moves at the middle hop: it fails at the sync commit on the open finding below at depth 31, after about 50M distinct states and minutes.
Expand Down
8 changes: 4 additions & 4 deletions .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ Each rule is an INVARIANT; the harm says why it matters. All but `NoOrphan`, `Re
## Run

```sh
just rs tla moq-net/verify/reconnect/Handover.tla
just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg
just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg
just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RestartWait.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/RedirectLost.cfg
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg
```

The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. `just test tla rs/moq-net/verify/reconnect` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/verify-resume/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ Each rule is an INVARIANT; the harm says why it matters.
## Run

```sh
just rs tla moq-net/verify/resume/ResumeReader.tla
just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg
quest gate just rs tla moq-net/verify/resume/ResumeReader.tla
quest gate just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg
```

The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, in minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, in minutes. Both pass at the sync commit. `just test tla rs/moq-net/verify/resume` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes.
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/verify-route-key/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ Each variant drops one assumption the draft states and must break exactly the le
## Run

```sh
just test route-key-model
just rs tamarin moq-net/verify/route-key/RouteKey.spthy -D=NO_SIDE
quest gate just test route-key-model
quest gate just rs tamarin moq-net/verify/route-key/RouteKey.spthy -D=NO_SIDE
```

The first proves the model and checks every variant's `Breaks:` line, in seconds; Nightly runs it. The second proves one variant. To read a counterexample, run `nix shell --inputs-from . nixpkgs#tamarin-prover --command tamarin-prover interactive rs/moq-net/verify/route-key/` and open the falsified lemma.
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/verify-route/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ What the timer costs, as the configs show it: fired late, it costs only the wait
## Run

```sh
just rs tla moq-net/verify/route/RouteHandoff.tla
just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg
just test tla rs/moq-net/verify/route
quest gate just rs tla moq-net/verify/route/RouteHandoff.tla
quest gate just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg
quest gate just test tla rs/moq-net/verify/route
```

`just test tla rs/moq-net/verify/route` runs every config, as Nightly does, simulating each one with a `Simulate:` line and expecting `Loop.cfg` to break. Times are with `-workers 4`. `Patient` is TRUE unless the table says otherwise.
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/verify-subscribe/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ Each rule is an INVARIANT; the harm says why it matters.
## Run

```sh
just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla
just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg
quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla
quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg
```

The first checks every rule but `StreamWins` under `Pressure`: about 13.1M states, a minute on 4 workers, and it passes at the sync commit. The second checks `StreamWins` without `Pressure` and fails on the open finding below. `just test tla rs/moq-net/verify/subscribe` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race.
Expand Down
3 changes: 2 additions & 1 deletion js/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,9 @@ fix-changed list:

# Model-check a TLA+ spec with TLC, e.g. `just js tla watch/verify/Watch.tla`.
# Pass `-config js/<path>.cfg` for a config not named after the spec.
# TLC runs at the lowest priority with a 4 GB heap, as `just rs tla` explains.
tla spec *args:
cd .. && nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} js/{{ spec }}
cd .. && JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} js/{{ spec }}

# Run all unit tests.
test:
Expand Down
66 changes: 66 additions & 0 deletions quest/m1/tla-native-checker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# [M] A Rust TLA+ checker runs the models locally if it agrees with TLC

## Goal

The `tla` recipes run a native TLA+ checker locally when one gives TLC's
verdict and distinct-state count on every config in the tree it can run
(the rest stay on TLC, as the Plan says) and costs fewer
core-seconds on the route and resume models. Nightly (`just test tla`, which
`sh/tla.sh` drives) then runs both engines on every config and fails when they
disagree. If no checker agrees, the PR
Comment thread
kixelated marked this conversation as resolved.
Resolved
records what each one failed on and adopts nothing.

## Plan

Decided 2026-11-10 (maintainer, /quest-plan), as a spike: TLC interprets a
spec on the JVM, and the route model's 4-node runs take tens of minutes of
many cores.

- ✅ A checker that agrees runs locally, and Nightly cross-checks it against
TLC (rejected: local runs with a TLC-only Nightly, where a checker bump or
spec change could make it wrong unnoticed; replacing TLC everywhere, where
nothing checks the checker again; reporting the numbers and adopting
nothing).

Candidates found that day, each young, at least one AI-written, and benchmarked
only by its authors:

- [tla-mc](https://github.com/ddalton/tla-mc) compiles a spec to a native
parallel checker and claims about 4x TLC at 8 workers.
- [tlaplusplus](https://github.com/zoratu/tlaplusplus) claims about 10x on
many cores and passes 174 of 182 of tlaplus/Examples.
- [tla-rs](https://github.com/bbcvc/tla-rs) claims state-for-state agreement
with TLC on 823 recorded runs and 2x to 3x on large models with one worker.
- [tlx](https://github.com/Medfin-AI/tlx) reports some models slower than TLC.

A checker bug shows as a rule passing that should break, so agreement is the
bar, not speed:

- Every config the checker runs: the same verdict as TLC, and the same broken
rule for an open-finding config. A passing config also matches TLC's
distinct-state count; a violating one does not, since each engine stops at
its first violation after exploring a different, order-dependent part of
the space. The specs use
`RECURSIVE`, `CHOOSE`, `CASE`, `LAMBDA`, `Sequences`, `FiniteSets`,
`CONSTRAINT`, and `VIEW`, unmodified.
- Measure both engines at equal worker counts through `quest gate` on a quiet
host, against TLC as the `tla` recipes run it. The 2026-11-10 baseline, on a
24-core desktop and the route model as it stood then (3.67M distinct
states): about 50 s and 930 CPU-seconds on TLC 1.8.0 with Java 21, down from
about 118 s and 1,730 on TLC 1.7.4 with Java 8, with CPU-seconds near flat
from 2 to 24 workers. The model has since changed, so remeasure TLC and
report the table in the PR.
- A config the checker cannot run, such as one with a `\* Simulate:` line
that `sh/tla.sh` samples at random, stays on TLC; the recipe picks the engine per config and errors rather than
falling back at runtime.
- A broken rule must still print a trace as readable as TLC's, which the
Reading a failure in `rs/moq-net/verify/README.md` relies on. Say how.
- Build each candidate from a flake input pinned to a revision, inside the Nix
sandbox. Run none of their build scripts or binaries outside it.
- Report upstream bugs to the maintainer rather than filing them.

Public API: none. Wire: none.

## Related

- [Every delivery order of a small mesh runs on the real route layer](/quest/m2/cluster-routing/routes/route-exhaustive-sim.md) - the other spike, which checks the Rust instead of a faster model
70 changes: 70 additions & 0 deletions quest/m2/cluster-routing/routes/route-exhaustive-sim.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# [M] Every delivery order of a small mesh runs on the real route layer

## Goal

A test explores every order in which `route::tests::Sim` can deliver, hold,
disconnect, collect, and change claims on a triangle of the origin and two
relays, on the real `route::Router`, and checks the route rules in every state
it reaches. A broken rule prints the path to it as `Sim` calls. The PR reports
its states and time beside TLC's for the same bounds. If the real state space
does not fit, the PR records the numbers and lands no test.

## Plan

Decided 2026-11-10 (maintainer, /quest-plan), as a spike: `RouteHandoff.tla`
is a hand-kept mirror of the rule `route/tests/paths.rs` prototypes, which TLC interprets, while `Sim`
already drives the router one atomic step at a time. Searching `Sim` itself
runs compiled code, has no model to drift, and reaches what the model leaves
out.

- ✅ A hand-written breadth-first search beside `Sim` in `route/tests/`: a
fingerprint set, a queue, and a parent map for the path (rejected: the
`stateright` crate, whose 12 direct dependencies include dashmap, rand,
serde_json, and tiny_http; building both and keeping the better). Add
threads or symmetry only if the measurements call for them.
- Bound the run as `RouteHandoff.cfg` does (`MaxRev`, `MaxFlaps`, one
claim), but on the triangle: `RouteHandoff.cfg` has four nodes, so the PR
runs TLC on a copy of that config cut to the same three nodes, and the two
counts compare.
- The search converges only if equal situations hash equal. `Sim` and `Router`
need a clone and a fingerprint, and the fingerprint must leave out what only
records the past, such as `events`, `stats`, and `requested`. Check each
rule at the step that would break it rather than from that log.
- The check that the search works: run on today's router before the switch to
the path rule, it must find the storm
`residual_a_seqno_advance_storms_between_two_grants` replays, or another
`residual_` test in `route/tests/residual.rs`. Those fixtures use 6 to 10
nodes, beyond the triangle, so the self-check searches the fixture's own
mesh, bounded by depth and by the fixture's scripted claims, rather than
the triangle's bounds; the search code takes the mesh as a parameter.
- Then say which of the model's "Not covered" list (`verify-route` skill) the
search can afford: wildcards, scoped grants, costs other than 1, a fourth
node.
- A search that takes seconds joins the merge gate; a larger one runs in
Nightly.

This sits beside [trace validation](/quest/m1/route-trace-validation.md), which
keeps the TLA+ model honest for [the inductive invariant](/quest/m1/route-inductive.md)
and the design work done on the spec before code exists. It replaces neither
that nor the seeded sweeps on larger meshes
(`tests/integration/mesh_sweep.rs`). If the spike shows the
search covers what a model check does, recommend in the PR what to do with the
route model's TLC runs.

Decided 2026-11-12 (maintainer, pick round): the search runs on the `Router`
after [the router quest](/quest/m2/cluster-routing/routes/scope.md)'s rewrite to
the path rule, so it checks the code that ships and nothing is built twice
(rejected: today's Babel `Router`, which the rewrite throws away; the test-only
`paths.rs` prototype). The storm self-check above then runs against a router
from before the rewrite.

Public API: none. Wire: none.

## Required

- [The router quest](/quest/m2/cluster-routing/routes/scope.md) - rewrites `route::Router` to the path rule, the router this searches

## Related

- [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - guards the model against the code; this checks the code itself
- [A Rust TLA+ checker runs the models locally if it agrees with TLC](/quest/m1/tla-native-checker.md) - the other spike, a faster engine for the same specs
7 changes: 5 additions & 2 deletions rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@ default:
just check

# Model-check a TLA+ spec with TLC, e.g. `just rs tla moq-net/verify/route/RouteHandoff.tla`.
# Pass `-config rs/<path>.cfg` for a config not named after the spec.
# Pass `-config rs/<path>.cfg` for a config not named after the spec, and
# `../<path>.tla` for a spec outside `rs/`, such as a generated one in `.tmp/`.
# TLC runs at the lowest priority with a 4 GB heap, which no model here outgrows:
# the JVM otherwise takes a quarter of RAM per run.
tla spec *args:
nix shell --inputs-from . nixpkgs#tlaplus --command tlc -workers auto -cleanup -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }}
JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }}

# Prove a Tamarin model's lemmas, e.g. `just rs tamarin moq-net/verify/route-key/RouteKey.spthy`.
# Pass `-D=<VARIANT>` for a variant the model defines.
Expand Down