Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/verify-reconnect/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/
quest gate just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg
```

The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. `just test tla rs/moq-net/verify/reconnect` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.
The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed.

## Open findings

Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/verify-resume/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ quest gate just rs tla moq-net/verify/resume/ResumeReader.tla
quest gate just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg
```

The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, in minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, in minutes. Both pass at the sync commit. `just test tla rs/moq-net/verify/resume` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes.
The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, in minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, in minutes. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes.

## Open findings

Expand Down
5 changes: 2 additions & 3 deletions .claude/skills/verify-route/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,9 @@ What the timer costs, as the configs show it: fired late, it costs only the wait
```sh
quest gate just rs tla moq-net/verify/route/RouteHandoff.tla
quest gate just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/route/Ring.cfg
quest gate just test tla rs/moq-net/verify/route
```

`just test tla rs/moq-net/verify/route` runs every config, as Nightly does, simulating each one with a `Simulate:` line and expecting `Loop.cfg` to break. Times are with `-workers 4`. `Patient` is TRUE unless the table says otherwise.
Run the configs the change touches, one at a time. Nightly runs every config (`just test tla rs/moq-net/verify/route`), simulating each one with a `Simulate:` line and expecting `Loop.cfg` to break. Times are on the recipe's 4 workers. `Patient` is TRUE unless the table says otherwise.

| Config | Mesh | Bounds | Distinct states | Run |
| --- | --- | --- | --- | --- |
Expand All @@ -48,7 +47,7 @@ quest gate just test tla rs/moq-net/verify/route
| `Twice.cfg` | The same, where two sessions close, as when relay 1 dies. Simulated: `num` counts behaviours per worker. | 1 claim change, 2 session changes | 40,000 behaviours, 4.0M states | a few minutes |
| `Ring.cfg` | The publisher and three relays in a ring, with a timer that fires at any moment. | 1 claim change, 1 session change | 426,409 | a minute or two |
| `Lag.cfg` | Relay 3 hears the publisher through relays 1 and 2: the lagging neighbour. | 2 claim changes, 1 session change | 1,299,405 | a few minutes |
| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 4,194,747 | twenty minutes |
| `LagWide.cfg` | The same, with relay 1 asking for `"a"` alone. | 2 claim changes, 1 session change | 4,194,747 | ten minutes |
| `Dead.cfg` | Two relays behind a relay whose only uplink asks for `"a"` alone, one with a dear session of its own to the publisher. Simulated. | 3 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes |
| `Blind.cfg` | A relay whose nearest neighbour asks for `"a"` alone, and a farther one knows everything. Simulated. | 2 claim changes, 1 session change | 40,000 behaviours, 4.8M states | ten minutes |
| `Loop.cfg` | The publisher with a session to each of three relays in a triangle, all closing. Open finding: `RequestsLoopFree` breaks. | 3 session changes | 979,474 when it breaks | a few minutes |
Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/verify-subscribe/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config
quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg
```

The first checks every rule but `StreamWins` and `ContinuesWithinHeld` under `Pressure`, with one YIELDED reset and frames left out: about 76.5M distinct states, five minutes on 24 workers. The second gives the group one frame and two YIELDED resets, without `Pressure` and over two updates, so it also checks where each continuation starts: about 29.4M distinct states, two to three minutes. Both pass. The third checks `StreamWins` without `Pressure` or YIELDED and fails on the open finding below. `just test tla rs/moq-net/verify/subscribe` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race.
The first checks every rule but `StreamWins` and `ContinuesWithinHeld` under `Pressure`, with one YIELDED reset and frames left out: about 77.7M distinct states, fifteen minutes. The second gives the group one frame and two YIELDED resets, without `Pressure` and over two updates, so it also checks where each continuation starts: about 29.6M distinct states, five minutes. Both pass. The third checks `StreamWins` without `Pressure` or YIELDED and fails on the open finding below. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race.

## Open finding: a group read whole is named in a stale drop

Expand Down
1 change: 1 addition & 0 deletions .github/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ test:
python3 sh/migrate.test.py
python3 sh/text.test.py
python3 sh/coverage-tables.test.py
python3 sh/tlc.test.py
python3 sh/dispatch-query.test.py
python3 sh/gh/test-inventory.test.py
bun test ./sh/gh/setup.test.ts
Expand Down
14 changes: 6 additions & 8 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -435,17 +435,15 @@ jobs:
fail-fast: false
matrix:
# The largest models get a runner each, and the route model's two
# largest configs (about 45 and 70 minutes on 4 workers) one per
# config; the `route` and `rest` legs take every other config, so a
# new one is checked without editing this list. sh/gh/nightly.test.py
# fails on a path that no longer exists.
# largest configs (about 30 and 15 minutes) share one; the `route`
# and `rest` legs take every other config, so a new one is checked
# without editing this list. sh/gh/nightly.test.py fails on a path
# that no longer exists.
include:
- name: resume
paths: rs/moq-net/verify/resume
- name: route early
paths: rs/moq-net/verify/route/Early.cfg
- name: route wide
paths: rs/moq-net/verify/route/LagWide.cfg
- name: route large
paths: rs/moq-net/verify/route/Early.cfg rs/moq-net/verify/route/LagWide.cfg
- name: route
paths: "rs/moq-net/verify/route :!rs/moq-net/verify/route/Early.cfg :!rs/moq-net/verify/route/LagWide.cfg"
- name: rest
Expand Down
2 changes: 1 addition & 1 deletion REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ The runtime guards fail Rust and bun tests only when a timer fires on the host c
A TLA+ model that no longer describes its code keeps passing while proving nothing, and Nightly cannot tell.
When a PR changes a file in a `verify-*` skill's Coverage table, run that skill on the head.
Flag a covered change the model does not mirror, a config whose result changed without its `\* Open finding:` line following, and a sync commit not moved to the PR's base once the model matches (PRs squash, so the base is the latest commit it can name).
The flow models under `js/*/verify/` have no skill: give a PR that changes the code their README names the same check with `just test tla`.
The flow models under `js/*/verify/` have no skill: give a PR that changes the code their README names the same check, running the configs it touches as the README shows.

# Not findings

Expand Down
4 changes: 2 additions & 2 deletions js/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ fix-changed list:

# Model-check a TLA+ spec with TLC, e.g. `just js tla watch/verify/Watch.tla`.
# Pass `-config js/<path>.cfg` for a config not named after the spec.
# TLC runs at the lowest priority with a 4 GB heap, as `just rs tla` explains.
# sh/tlc.py says how a run shares the host.
tla spec *args:
cd .. && JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} js/{{ spec }}
python3 ../sh/tlc.py js/{{ spec }} {{ args }}

# Run all unit tests.
test:
Expand Down
2 changes: 1 addition & 1 deletion js/publish/verify/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ FAILED once the reopen budget runs out.
## Run

```sh
just test tla js/publish
quest gate just js tla publish/verify/Publish.tla
```

This must pass, as Nightly checks.
Expand Down
2 changes: 1 addition & 1 deletion js/room/verify/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ comes back once it reconnects.
## Run

```sh
just test tla js/room
quest gate just js tla room/verify/Room.tla
```

`Room.cfg` plays only Bob's camera; add `"screen"` to
Expand Down
7 changes: 5 additions & 2 deletions js/watch/verify/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,13 @@ through `a redirected connection falls back to the configured URL once its targe
## Run

```sh
just test tla js/watch
quest gate just js tla watch/verify/Watch.tla
quest gate just js tla watch/verify/Failover.tla
quest gate just js tla watch/verify/Failover.tla -config js/watch/verify/FailoverRedirect.cfg
quest gate just js tla watch/verify/Failover.tla -config js/watch/verify/FailoverRedirectRefused.cfg
```

This runs every config, as Nightly does. `Watch.cfg` and `Failover.cfg` check the rules that hold
Run the configs the change touches; Nightly runs every one (`just test tla js/watch`). `Watch.cfg` and `Failover.cfg` check the rules that hold
and must pass; each rule that breaks has its own config, marked `\* Open finding:`, which must
fail until the fix moves its rule into the passing one. A config checks the spec its name starts
with.
Expand Down
93 changes: 0 additions & 93 deletions quest/m0/ci/tlc-cost.md

This file was deleted.

6 changes: 6 additions & 0 deletions quest/m1/route-inductive.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ stay constants the config states. Say both bounds in the `verify-route` skill.
`Budget` and `QuietIsSettled`. Leave it there.
- Run it through the `tla` recipes with Apalache from the flake. If five
relays does not fit Nightly, record the largest mesh that does and its time.
- A refinement check such as `RefinesClaims` runs on the smallest exhaustive
config that exercises it, where each distinct state is checked once, not on
a simulated config such as `Blind.cfg`, where every sampled trace re-checks
it, unless measuring shows the simulated config is cheaper. `SumOver` folds
with `FoldSet` through the Community Modules' Java override the `tla`
recipes load; keep one fold for both checkers.

Public API: none. Wire: none.

Expand Down
17 changes: 8 additions & 9 deletions quest/m2/tla-native-recheck.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,14 @@ model:
- tlx and tla-rs exceed a 16 GB cap where TLC fits in 4 GB, tlx after
finding 5% to 50% of TLC's states.

`SumOver` is a recursive function over `SUBSET S`, the likely cost for both,
and [TLC's cost quest](/quest/m0/ci/tlc-cost.md) replaces it with
`FoldSet`. Once that lands, rerun #8600's comparison at each checker's newest
revision: route and resume first, since they decide cost, then every
other exhaustive config, since adoption needs agreement on all of them:
`SumOver` was a recursive function over `SUBSET S`, the likely cost for both.
[moq-dev/moq#8628](https://github.com/moq-dev/moq/pull/8628) replaced it with
the Community Modules' `FoldSet` (`FiniteSetsExt`, through `INSTANCE`), which
`sh/tlc.py` puts on TLC's classpath, so a checker must also resolve that
module. Rerun #8600's comparison at each checker's newest revision, against
TLC remeasured after #8628: route and resume first, since they decide cost,
then every other exhaustive config, since adoption needs agreement on all of
them:

- Build each from a pinned flake input inside the Nix sandbox, and run each
check there as a derivation under `prlimit --as=16G`, one at a time through
Expand All @@ -38,7 +41,3 @@ other exhaustive config, since adoption needs agreement on all of them:
- Report upstream bugs to the maintainer rather than filing them.

Public API: none. Wire: none.

## Required

- [TLC runs cost fewer core-seconds and share the host](/quest/m0/ci/tlc-cost.md) - replaces `SumOver` in the route model, the construct every candidate failed on, and moves TLC's baseline
5 changes: 2 additions & 3 deletions rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,9 @@ default:
# Model-check a TLA+ spec with TLC, e.g. `just rs tla moq-net/verify/route/RouteHandoff.tla`.
# Pass `-config rs/<path>.cfg` for a config not named after the spec, and
# `../<path>.tla` for a spec outside `rs/`, such as a generated one in `.tmp/`.
# TLC runs at the lowest priority with a 4 GB heap, which no model here outgrows:
# the JVM otherwise takes a quarter of RAM per run.
# sh/tlc.py says how a run shares the host.
tla spec *args:
JAVA_TOOL_OPTIONS=-Xmx4g nice -n 19 nix shell --inputs-from . nixpkgs#tlaplus18 --command tlc -workers auto -cleanup -noGenerateSpecTE -metadir "$(mktemp -d)" {{ args }} rs/{{ spec }}
python3 sh/tlc.py rs/{{ spec }} {{ args }}

# Prove a Tamarin model's lemmas, e.g. `just rs tamarin moq-net/verify/route-key/RouteKey.spthy`.
# Pass `-D=<VARIANT>` for a variant the model defines.
Expand Down
Loading