diff --git a/.claude/skills/verify-failover/SKILL.md b/.claude/skills/verify-failover/SKILL.md index f83159c7cb5..907d88e9bf7 100644 --- a/.claude/skills/verify-failover/SKILL.md +++ b/.claude/skills/verify-failover/SKILL.md @@ -46,17 +46,17 @@ A counterexample is a path through the model, not yet a bug. Replay it on the re ## Coverage -Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. +Sync commit: `88572c583e2f64c7f5fc9bfa59df90706308497e`. Run `git log 88572c583e.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/front.rs rs/moq-net/src/model/origin.rs rs/moq-net/src/model/track.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. -| Code (`rs/moq-net/src/model/`) | Model | +| Code | Model | | --- | --- | -| `resume.rs` `Reader::sync`, `backfill_gaps`, `Producer::subscription` | `ResumeAsk`, `Demand`, `Switch`, `Subscribe` | -| `resume.rs` `Reader::take`, `poll_end` | `HandOut`, `Drained`, `EndOk` | -| `resume.rs` `Reader::let_go`, `Copy::retains`, `Producer::park` | `LetGo`, `Park` | -| `resume.rs` `Recover::wants`, `poll`, `held`, `poll_serving` | `Recover`, `Holders`, `Exhausted`, `Abort` | -| `resume.rs` `Recover::poll_fetch`, `Fetching`; `track.rs` `fetch_group` | `Ask` | -| `front.rs` `track_ended`; `origin.rs` `Action::Finish`, `TrackIo::end` | `FrontFinish`, `Whole` | -| `front.rs` failover and its unroutable end | `Switch`, `FrontFail` | +| `rs/moq-net/src/model/resume.rs` `Reader::sync`, `backfill_gaps`, `Producer::subscription` | `ResumeAsk`, `Demand`, `Switch`, `Subscribe` | +| `rs/moq-net/src/model/resume.rs` `Reader::take`, `poll_end` | `HandOut`, `Drained`, `EndOk` | +| `rs/moq-net/src/model/resume.rs` `Reader::let_go`, `Copy::retains`, `Producer::park` | `LetGo`, `Park` | +| `rs/moq-net/src/model/resume.rs` `Recover::wants`, `poll`, `held`, `poll_serving` | `Recover`, `Holders`, `Exhausted`, `Abort` | +| `rs/moq-net/src/model/resume.rs` `Recover::poll_fetch`, `Fetching`; `rs/moq-net/src/model/track.rs` `fetch_group` | `Ask` | +| `rs/moq-net/src/model/front.rs` `track_ended`, `Action::Finish`; `rs/moq-net/src/model/origin.rs` `TrackIo::end` | `FrontFinish`, `Whole` | +| `rs/moq-net/src/model/front.rs` failover and its unroutable end | `Switch`, `FrontFail` | ## Not covered diff --git a/.claude/skills/verify-reconnect/SKILL.md b/.claude/skills/verify-reconnect/SKILL.md index 513aad6eec2..883a61457d2 100644 --- a/.claude/skills/verify-reconnect/SKILL.md +++ b/.claude/skills/verify-reconnect/SKILL.md @@ -30,7 +30,7 @@ just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reco just rs tla moq-net/verify/reconnect/Handover.tla -config rs/moq-net/verify/reconnect/OfflineWhileServing.cfg ``` -The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute on 32 cores. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. +The first checks every rule but the two with open findings, over four sessions: about 1.2M distinct states, under a minute on 32 cores. The second checks the same rules under `Restart::Wait` over three sessions: about 240k states. Both pass at the sync commit. The last two fail at the sync commit, each on its finding below, within seconds. `just test tla rs/moq-net/verify/reconnect` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. The `VIEW` counts states that differ only in what nothing reads again as one: a closed session's details unless it is still the live one, a deadline of a phase the policy left, and where no dial is headed. Read a counterexample with `/statechart`, passing it the spec, a config, and `Handover.view.ts`, which labels what the handle reports, what the policy holds, and each step as the event that happens. @@ -50,17 +50,17 @@ A counterexample is a path through the model, not yet a bug. Replay it on the po ## Coverage -Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. +Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/reconnect.rs rs/moq-net/src/reconnect/ rs/moq-tokio/src/connection.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it. | Code | Model | | --- | --- | -| `reconnect.rs` `Reconnect::poll` (queue, predecessor, phases) | `Pop`, `Retire`, `IdleClose`, `Dial`, `Abandon`, `BackoffEnds`, `LiveEnds`, `ProbeAsk`, `ProbeAbandon` | -| `reconnect.rs` `established`, and `walk_failed` ending the first walk | `Established`, `Failed`, `Abandon` | -| `reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` | -| `reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` | -| `reconnect.rs` `Restart::Wait` | `LiveEnds` with `Restart = "wait"` (`RestartWait.cfg`) | -| `reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` | -| `moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `predecessor`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` | +| `rs/moq-net/src/reconnect.rs` `Reconnect::poll` (queue, predecessor, phases) | `Pop`, `Retire`, `IdleClose`, `Dial`, `Abandon`, `BackoffEnds`, `LiveEnds`, `ProbeAsk`, `ProbeAbandon` | +| `rs/moq-net/src/reconnect.rs` `established`, and `walk_failed` ending the first walk | `Established`, `Failed`, `Abandon` | +| `rs/moq-net/src/reconnect.rs` `connected`, `fallback`, `failed`, `probed` | `Connected`, `Failed`, `ProbeFails`, `ProbeGetsThrough` | +| `rs/moq-net/src/reconnect.rs` `end`, `closed`, `done`, `Predecessor` | `EndGoaway`, `EndClosed`, `Done`, `Retire`, `IdleClose` | +| `rs/moq-net/src/reconnect.rs` `Restart::Wait` | `LiveEnds` with `Restart = "wait"` (`RestartWait.cfg`) | +| `rs/moq-net/src/reconnect/redirect.rs` `Redirect::target` | `Refused` and `Endpoints` | +| `rs/moq-tokio/src/connection.rs` `run`, `Shared::connected`, `migrating`, `predecessor`, `offline`, `end` | `Connected`, `Apply`, `OnMigrating`, `OnRetired`, `OnOffline`, `OnDone` | ## Not covered diff --git a/.claude/skills/verify-resume/SKILL.md b/.claude/skills/verify-resume/SKILL.md index a0ba5e2f688..0276e5b97c6 100644 --- a/.claude/skills/verify-resume/SKILL.md +++ b/.claude/skills/verify-resume/SKILL.md @@ -28,7 +28,7 @@ just rs tla moq-net/verify/resume/ResumeReader.tla just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg ``` -The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, three minutes on 32 cores. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, under two minutes. Both pass at the sync commit. +The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, three minutes on 32 cores. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, under two minutes. Both pass at the sync commit. `just test tla rs/moq-net/verify/resume` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. `Chart.cfg` is for `/statechart` only: a story (`Story`) small enough to draw whole, where the reader switches to a route that holds an older group and asks it for that group. Pass it the spec, `Chart.cfg`, and `ResumeReader.view.ts`, which labels which route serves, what the reader has, and each step as what happens to a route or the reader. For a counterexample, pass the config that broke instead. @@ -38,21 +38,21 @@ None at the sync commit. ## Reading a failure -A counterexample is a path through the model, not yet a bug. Replay it as a test in `model/resume.rs`'s `mod test`, one call per step: `received_copy` for a route, `routes.serve(copy.consume())` for a switch, `write` for a landing group, `finish()` for a declared end, `abort(..)` for a route failing, and `drain` or `poll_recv_group` for a poll. If the code does not follow the path, the model is wrong: fix the model to match the code. If it does, report the path and the rule in the code's words, and pin it as a `residual_*` test with its own config so the others still pass. Do not weaken a rule, add a constraint, or shrink the bounds until the check passes. +A counterexample is a path through the model, not yet a bug. Replay it as a test in `model/resume.rs`'s `mod test`, one call per step: `received_copy` for a route, `routes.serve(copy.consume())` for a switch, `write` for a landing group, `finish()` for a declared end, `abort(..)` for a route failing, and `drain` or `poll_recv_group` for a poll. If the code does not follow the path, the model is wrong: fix the model to match the code. If it does, report the path and the rule in the code's words, and pin it as a `residual_*` test with its own config, marked with an `\* Open finding:` line, so the others still pass. Do not weaken a rule, add a constraint, or shrink the bounds until the check passes. ## Coverage -Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. +Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it. -| Code (`rs/moq-net/src/model/`) | Model | +| Code | Model | | --- | --- | -| `resume.rs` `Reader::sync` (reuse of a route's copy, new copy at `newest`) | `Sync`, `Serve` | -| `resume.rs` `Reader::backfill`, `backfill_gaps` | `Backfill`, `BackfillGaps` | -| `resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe` | `Poll`, `Scan`, `Peek`, `Deliverable`, `Take` | -| `resume.rs` `Reader::bound_at_end`, `let_go`, `Copy::retains` | `BoundAtEnd`, `LetGo`, `Retains` | -| `resume.rs` `Reader::poll_end` | `PollEnd` | -| `resume.rs` `Producer::splice`, `serve_from`, the front's end | `Serve`, `FrontFinish`, `FrontFail` | -| `track.rs` `TrackState::is_complete`, `may_land`, `latest_sequence`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` | +| `rs/moq-net/src/model/resume.rs` `Reader::sync` (reuse of a route's copy, new copy at `newest`) | `Sync`, `Serve` | +| `rs/moq-net/src/model/resume.rs` `Reader::backfill`, `backfill_gaps` | `Backfill`, `BackfillGaps` | +| `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe` | `Poll`, `Scan`, `Peek`, `Deliverable`, `Take` | +| `rs/moq-net/src/model/resume.rs` `Reader::bound_at_end`, `let_go`, `Copy::retains` | `BoundAtEnd`, `LetGo`, `Retains` | +| `rs/moq-net/src/model/resume.rs` `Reader::poll_end` | `PollEnd` | +| `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, the front's end | `Serve`, `FrontFinish`, `FrontFail` | +| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `latest_sequence`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` | ## Not covered diff --git a/.claude/skills/verify-route/SKILL.md b/.claude/skills/verify-route/SKILL.md index 5771104341a..7b2795b1b5b 100644 --- a/.claude/skills/verify-route/SKILL.md +++ b/.claude/skills/verify-route/SKILL.md @@ -39,7 +39,7 @@ just rs tla moq-net/verify/route/RouteHandoff.tla -config rs/moq-net/verify/rout | `Behind.cfg` | The origin behind relay 1, which sits in a triangle with relays 2 and 3. | 1 claim change, 1 session change | 3,033,111 | | `BehindDear.cfg` | The same, with relay 3 nearer the origin than relay 2. | 1 claim change, 1 session change | 5,377,597 | -All three pass at the sync commit. Each takes minutes on 4 workers, so run them last. First run what answers in seconds: +All three pass at the sync commit, and `just test tla rs/moq-net/verify/route` runs them all, as Nightly does. Each takes minutes on 4 workers, so run them last. First run what answers in seconds: - `just rs test -p moq-net --lib -E 'test(/route::/)'`. `sweep_no_route_outgrows_a_loop_free_path` delivers every message and echo in a random order on random meshes with unequal costs, flaps, restarts, and collections, and checks the metric bound, loop freedom, and the quiet state. Raise its case count for a wider search: 40,000 cases run in 15 seconds in release. - TLC's simulation mode on the config in question, which samples deep runs the exhaustive bounds do not reach: add `-simulate num=3000 -depth 80`. @@ -54,18 +54,18 @@ A counterexample is a path through the model, not yet a bug. Replay it on the ro ## Coverage -Sync commit: `c916855d163ff5641ad18cfb8e14b76dababd99a`, plus the change that added a route's hops ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Run `git log c916855d16.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. +Sync commit: `c916855d163ff5641ad18cfb8e14b76dababd99a`, plus the change that added a route's hops ([moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487)). Run `git log c916855d16.. -- rs/moq-net/src/route/` to see what changed since. After a change to a covered function, mirror it in the model, rerun every config, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it. -| Code (`rs/moq-net/src/route/`) | Model | +| Code | Model | | --- | --- | -| `mod.rs` `feasible`, `Entry::crossed`, `eligible`, `metric`, `selected`, `infeasible`, `recoverable`, `live`, `certifiers` | `Feasible`, `Crossed`, `Eligible`, `Through`, `Selected`, `InfeasibleSet`, `Recoverable`, `Live`, `Certifiers` | -| `mod.rs` `certified`, `prune`, `adopted` | `Certified`, `Prune`, `Adopted` | -| `mod.rs` `update`, `recompute`, `record`, `advertise`, `requests` | `Update` and its `prune`, `Record`, `Plan`, `Requests` | -| `mod.rs` `show`, `shown_now` | `Events`, `ShownNow` | -| `mod.rs` `route`, `batch`, `request`, `echo` | `OnRoute`, `OnBatch`, `OnRequest`, `Deliver`, `DeliverAck` | -| `mod.rs` `connect`, `peer_interest`, `disconnect`, `collect`, `change` | `Connect`, `Opened`, `Disconnect`, `Closed`, `Collect`, `ChangeClaim` | -| `view.rs` `View::merge`, `View::restrict` | `Merge`, `Certify` | -| `tests/mod.rs` `Sim` queues and `revivals`, `false_ends` | `q`, `acks`, `revivals`, `falseEnds` | +| `rs/moq-net/src/route/mod.rs` `feasible`, `Entry::crossed`, `eligible`, `metric`, `selected`, `infeasible`, `recoverable`, `live`, `certifiers` | `Feasible`, `Crossed`, `Eligible`, `Through`, `Selected`, `InfeasibleSet`, `Recoverable`, `Live`, `Certifiers` | +| `rs/moq-net/src/route/mod.rs` `certified`, `prune`, `adopted` | `Certified`, `Prune`, `Adopted` | +| `rs/moq-net/src/route/mod.rs` `update`, `recompute`, `record`, `advertise`, `requests` | `Update` and its `prune`, `Record`, `Plan`, `Requests` | +| `rs/moq-net/src/route/mod.rs` `show`, `shown_now` | `Events`, `ShownNow` | +| `rs/moq-net/src/route/mod.rs` `route`, `batch`, `request`, `echo` | `OnRoute`, `OnBatch`, `OnRequest`, `Deliver`, `DeliverAck` | +| `rs/moq-net/src/route/mod.rs` `connect`, `peer_interest`, `disconnect`, `collect`, `change` | `Connect`, `Opened`, `Disconnect`, `Closed`, `Collect`, `ChangeClaim` | +| `rs/moq-net/src/route/view.rs` `View::merge`, `View::restrict` | `Merge`, `Certify` | +| `rs/moq-net/src/route/tests/mod.rs` `Sim` queues and `revivals`, `false_ends` | `q`, `acks`, `revivals`, `falseEnds` | ## Not covered diff --git a/.claude/skills/verify-subscribe/SKILL.md b/.claude/skills/verify-subscribe/SKILL.md index b706ad43c76..95f60bd7e1c 100644 --- a/.claude/skills/verify-subscribe/SKILL.md +++ b/.claude/skills/verify-subscribe/SKILL.md @@ -25,7 +25,7 @@ just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla -config rs/moq-net/verify/subscribe/StreamWins.cfg ``` -The first checks every rule but `StreamWins`: about 3.0M states, 20 to 30 seconds on 32 cores, and it passes at the sync commit. The second fails at the sync commit on the open finding below. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race. +The first checks every rule but `StreamWins`: about 3.0M states, 20 to 30 seconds on 32 cores, and it passes at the sync commit. The second fails at the sync commit on the open finding below. `just test tla rs/moq-net/verify/subscribe` runs every config here, as Nightly does, and fails if a config with an `\* Open finding:` line passes. One group is enough: groups share only the order of the Subscribe Stream, and a second group multiplies the states (60M and more) without adding a race. Read a counterexample with `/statechart`, passing it the spec, a config, and `SubscribeAccounting.view.ts`, which labels what each side holds for the group, which update each side is on, and each step as the event that happens. It gives the shortest path to each broken rule in those words. @@ -39,20 +39,20 @@ A counterexample is a path through the model, not yet a bug. Replay it on the co ## Coverage -Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/lite/publisher.rs rs/moq-net/src/lite/subscriber.rs rs/moq-net/src/tail.rs rs/moq-net/src/model/track.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun both configs, and move the sync commit forward only once the model describes the code at the new commit. +Sync commit: `103052ed074a87f00269ecb0d639905482ecfb6f`. Run `git log 103052ed07.. -- rs/moq-net/src/lite/publisher.rs rs/moq-net/src/lite/subscriber.rs rs/moq-net/src/tail.rs rs/moq-net/src/model/track.rs` to see what changed since. After a change to a covered function, mirror it in the model, rerun both configs, and move the sync commit forward only once the model describes the code at the new commit. `just check` fails when a function or type the table names no longer exists in its file (`sh/coverage-tables.py`), so each Code cell starts with a repo-relative path, then the symbols in it. -| Code (`rs/moq-net/src/`) | Model | +| Code | Model | | --- | --- | -| `lite/publisher.rs` `TrackRun::poll_track` (`Recv::Group`, the re-offer pop), `serve`, `mark_served` | `CursorHand`, `PubReoffer`, `Serve` | -| `lite/publisher.rs` `poll_forward_drop`, `forward_range`, `drop_range`, `replace_stale`, `sent_stale`, `hole` | `CursorGiveUp`, `ReplaceStale` | -| `lite/publisher.rs` `TrackRun::poll_step` (outcomes, `redrop`) | `PubOutcomes`, `PubRedrop` | -| `lite/publisher.rs` `TrackRun::update`, `owe`; `model/track.rs` `reoffer_stale`, `reoffer_groups` | `PubUpdate` | -| `lite/publisher.rs` `GroupServe::poll_serve` (`Serve`, `Closed`), `settle_failed` | `StreamFin`, `StreamFail`, `StreamAck` | -| `lite/subscriber.rs` `TrackEntry::open`, `reopen`, `place`; `GroupRecv::poll_serve` (the duplicate wait) | `SubHeader`, `SubFin`, `SubReset`, `SubResetHeld` | -| `lite/subscriber.rs` `ServeLoop::apply_response` (`Drop`, `Skip`) | `OnDrop`, `OnSkip`, `SubRecv` | -| `lite/subscriber.rs` `buffer_update`; `model/track.rs` `supersede_stale` | `SubUpdate` | -| `tail.rs` `Reading::open`, `Reading::finish`, `Tail::finished`, `updated`, `account`, `unaccount` | `acct`, `finished` | -| `model/track.rs` `Producer::give_up`, `skip_received` | `rec`, `cached` | +| `rs/moq-net/src/lite/publisher.rs` `TrackRun::poll_track` (`Recv::Group`, the re-offer pop), `serve`, `mark_served` | `CursorHand`, `PubReoffer`, `Serve` | +| `rs/moq-net/src/lite/publisher.rs` `poll_forward_drop`, `forward_range`, `drop_range`, `replace_stale`, `sent_stale`, `hole` | `CursorGiveUp`, `ReplaceStale` | +| `rs/moq-net/src/lite/publisher.rs` `TrackRun::poll_step` (outcomes, `redrop`) | `PubOutcomes`, `PubRedrop` | +| `rs/moq-net/src/lite/publisher.rs` `TrackRun::update`, `owe`; `rs/moq-net/src/model/track.rs` `reoffer_stale`, `reoffer_groups` | `PubUpdate` | +| `rs/moq-net/src/lite/publisher.rs` `GroupServe::poll_serve` (`Serve`, `Closed`), `settle_failed` | `StreamFin`, `StreamFail`, `StreamAck` | +| `rs/moq-net/src/lite/subscriber.rs` `TrackEntry::open`, `reopen`, `place`; `GroupRecv::poll_serve` (the duplicate wait) | `SubHeader`, `SubFin`, `SubReset`, `SubResetHeld` | +| `rs/moq-net/src/lite/subscriber.rs` `ServeLoop::apply_response` (`Drop`, `Skip`) | `OnDrop`, `OnSkip`, `SubRecv` | +| `rs/moq-net/src/lite/subscriber.rs` `buffer_update`; `rs/moq-net/src/model/track.rs` `supersede_stale` | `SubUpdate` | +| `rs/moq-net/src/tail.rs` `Reading::open`, `Reading::finish`, `Tail::finished`, `updated`, `account`, `unaccount` | `acct`, `finished` | +| `rs/moq-net/src/model/track.rs` `Producer::give_up`, `skip_received` | `rec`, `cached` | ## Not covered diff --git a/.github/justfile b/.github/justfile index 09edd9e6b3d..13dadeeeb60 100644 --- a/.github/justfile +++ b/.github/justfile @@ -79,6 +79,7 @@ test: python3 sh/large-files.test.py python3 sh/migrate.test.py python3 sh/text.test.py + python3 sh/coverage-tables.test.py python3 sh/dispatch-query.test.py python3 sh/gh/test-inventory.test.py bun test ./sh/gh/setup.test.ts diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 43faaa7675a..ac5fc8d813a 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -27,6 +27,11 @@ name: Nightly # exception: deterministic and seconds long, so `just ci` gates every diff that # can move them, and audio-quality.yml reruns them here. # +# `tla` model-checks every TLA+ config: TLC explores every ordering a model +# allows, which takes minutes per config, and a model only changes when the code +# it covers does, which review checks (see REVIEW.md). It also fails when an +# open finding's config starts passing, so the fix promotes its rule. +# # Main PRs run only the landing gate's selection as a signal, so every # path-filtered gate (Android, C, C++, X11, the audio replays, the OSS # Scanner image, platform.yml and the release dry-runs) runs here on main too, @@ -403,6 +408,32 @@ jobs: NEXTEST_PROFILE: ci CI_STRICT: 1 + tla: + name: TLA+ (${{ matrix.name }}) + runs-on: ubuntu-24.04-arm + timeout-minutes: 120 + strategy: + fail-fast: false + matrix: + # The two largest models get a runner each; the last leg takes every + # other config, so a new model is checked without editing this list. + include: + - name: resume + paths: rs/moq-net/verify/resume + - name: route + paths: rs/moq-net/verify/route + - name: rest + paths: ":!rs/moq-net/verify/resume :!rs/moq-net/verify/route" + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: ./.github/actions/nix + # No dev shell: the recipes fetch TLC themselves. + - name: Model-check + run: nix shell --inputs-from . nixpkgs#just --command just test tla ${{ matrix.paths }} + # Media QA with a Firefox player against a Chromium fixture. The Chromium # player already runs in interop.yml, nightly and on release PRs. WebKit has # engine defects this cannot pass; see test/interop/README.md. diff --git a/REVIEW.md b/REVIEW.md index 4a74fc09d19..9801589395a 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -47,6 +47,13 @@ The fix is a mocked clock or an awaited event, never a shorter or longer real wa The exception is a test that drives real sockets, devices, or processes, which may wait on the real clock and opts out with `host_time()` or `hostTime()`; that opt-out belongs nowhere else. The runtime guards fail Rust and bun tests only when a timer fires on the host clock, so they miss Go and Python tests, elapsed-time assertions, and any wait that bypasses those clocks; and Rust's `host_time()` lasts for the whole process, so it hides other tests' waits outside nextest's one-test-per-process runs. +# Models + +A TLA+ model that no longer describes its code keeps passing while proving nothing, and Nightly cannot tell. +When a PR changes a file in a `verify-*` skill's Coverage table, run that skill on the head. +Flag a covered change the model does not mirror, a config whose result changed without its `\* Open finding:` line following, and a sync commit not moved to the PR's base once the model matches (PRs squash, so the base is the latest commit it can name). +The flow models under `js/*/verify/` have no skill: give a PR that changes the code their README names the same check with `just test tla`. + # Not findings Each of these has been raised and refuted repeatedly here; the rules in `AGENTS.md` already decide them: diff --git a/js/net/src/connection/migrate.test.ts b/js/net/src/connection/migrate.test.ts index fa4432e9be7..b7f7df0a599 100644 --- a/js/net/src/connection/migrate.test.ts +++ b/js/net/src/connection/migrate.test.ts @@ -245,7 +245,7 @@ test("a redirect moves the pool key while the handle keeps its origin", async () } }); -// Replays RedialsALiveRelay in js/watch/verify/Failover.tla under Redirect.cfg: a redirect is +// Replays RedialsALiveRelay in js/watch/verify/Failover.tla under FailoverRedirect.cfg: a redirect is // sticky, so once the relay it named dies, every redial goes back to that dead relay and never // to the configured URL, which still answers. test("residual: a redirected connection redials only its dead target", async () => { diff --git a/js/publish/verify/Publish.cfg b/js/publish/verify/Publish.cfg index e189698cd40..d2b70dc29b4 100644 --- a/js/publish/verify/Publish.cfg +++ b/js/publish/verify/Publish.cfg @@ -1,4 +1,5 @@ +\* LiveMeansWatchable is left out: its break, a badge that leads the announce by +\* milliseconds, is accepted (2026-11-10). SPECIFICATION Spec -INVARIANT LiveMeansWatchable INVARIANT StopStopsPublishing INVARIANT DenialIsShown diff --git a/js/publish/verify/README.md b/js/publish/verify/README.md index 30c982c5882..5e9b0b3539c 100644 --- a/js/publish/verify/README.md +++ b/js/publish/verify/README.md @@ -9,7 +9,7 @@ broadcast. | Rule | Holds when | Today | | --- | --- | --- | -| `LiveMeansWatchable` | The badge says LIVE only while viewers can find the broadcast. | Breaks: the badge reads the connection and the captured sources, not the announce gate, so it says LIVE while the gate waits for the encoders to settle. Accepted 2026-11-10 (maintainer): the window is usually milliseconds. | +| `LiveMeansWatchable` | The badge says LIVE only while viewers can find the broadcast. | Breaks: the badge reads the connection and the captured sources, not the announce gate, so it says LIVE while the gate waits for the encoders to settle. Accepted 2026-11-10 (maintainer): the window is usually milliseconds, so `Publish.cfg` leaves it out. | | `StopStopsPublishing` | A publisher who stopped the camera publishes nothing. | Holds. | | `DenialIsShown` | A publisher who refused the camera is told so. | Holds. | @@ -21,10 +21,10 @@ state has. ## Run ```sh -just js tla publish/verify/Publish.tla +just test tla js/publish ``` -TLC exits non-zero while a rule above breaks. To draw the chart, run `/statechart` with +This must pass, as Nightly checks. To draw the chart, run `/statechart` with `Publish.tla`, `Publish.cfg`, and `Publish.view.ts`. ## Left out diff --git a/js/room/verify/README.md b/js/room/verify/README.md index bb29efa4751..f443d9138dd 100644 --- a/js/room/verify/README.md +++ b/js/room/verify/README.md @@ -16,7 +16,7 @@ coming back once it reconnects. ## Run ```sh -just js tla room/verify/Room.tla +just test tla js/room ``` To draw the chart, run `/statechart` with diff --git a/js/watch/verify/Redirect.cfg b/js/watch/verify/FailoverRedirect.cfg similarity index 53% rename from js/watch/verify/Redirect.cfg rename to js/watch/verify/FailoverRedirect.cfg index b102a390dd3..107141cf692 100644 --- a/js/watch/verify/Redirect.cfg +++ b/js/watch/verify/FailoverRedirect.cfg @@ -1,7 +1,8 @@ \* Relays that name a URI in their GOAWAY, such as a third-party relay. moq-relay never does. +\* Open finding: a followed redirect is sticky, so RedialsALiveRelay fails. Once +\* fixed, delete this file; FailoverRedirectRefused.cfg keeps checking these relays. SPECIFICATION Spec CONSTANTS Relays = {1, 2} Redirects = TRUE INVARIANT RedialsALiveRelay -INVARIANT KeepsReconnecting diff --git a/js/watch/verify/FailoverRedirectRefused.cfg b/js/watch/verify/FailoverRedirectRefused.cfg new file mode 100644 index 00000000000..5ea3d7ac2c6 --- /dev/null +++ b/js/watch/verify/FailoverRedirectRefused.cfg @@ -0,0 +1,9 @@ +\* Relays that name a URI in their GOAWAY, such as a third-party relay. moq-relay never does. +\* Open finding: the default `same-host` policy refuses a redirect to another host +\* and stops the loop, so KeepsReconnecting fails. Once fixed, move the rule into +\* FailoverRedirect.cfg, or rename this file to it if that is gone. +SPECIFICATION Spec +CONSTANTS + Relays = {1, 2} + Redirects = TRUE +INVARIANT KeepsReconnecting diff --git a/js/watch/verify/README.md b/js/watch/verify/README.md index 692485bc8ee..91114251ff9 100644 --- a/js/watch/verify/README.md +++ b/js/watch/verify/README.md @@ -27,12 +27,12 @@ come back, drain with a GOAWAY, and lose or move their path to the origin. | Rule | Holds when | Today | | --- | --- | --- | | `OfflineOnlyWhenUnreachable` | The viewer sees OFFLINE only when no relay can reach the broadcast. | Holds: the viewer's relay crashing reads CONNECTING until the redial lands and its announcements arrive. | -| `KeepsReconnecting` | A viewer whose relay left keeps reconnecting while some relay is up. | Holds with moq-relay (`Failover.cfg`). Breaks with a relay whose GOAWAY names another host (`Redirect.cfg`): the default `same-host` policy refuses it and stops the loop. | -| `RedialsALiveRelay` | A reconnecting viewer can land on a relay that is up. | Holds with moq-relay. Breaks under `Redirect.cfg`: a followed redirect is sticky, so once that relay dies every redial goes back to it, never to the configured URL. | +| `KeepsReconnecting` | A viewer whose relay left keeps reconnecting while some relay is up. | Holds with moq-relay (`Failover.cfg`). Breaks with a relay whose GOAWAY names another host (`FailoverRedirectRefused.cfg`): the default `same-host` policy refuses it and stops the loop. | +| `RedialsALiveRelay` | A reconnecting viewer can land on a relay that is up. | Holds with moq-relay. Breaks under `FailoverRedirect.cfg`: a followed redirect is sticky, so once that relay dies every redial goes back to it, never to the configured URL. | A GOAWAY whose replacement connects within the handover keeps playing, and a move to another path to the origin only stalls. `Failover.cfg` plays relays as moq-relay runs them, draining with -an empty-URI GOAWAY; `Redirect.cfg` lets relays name a URI. The crash replays through +an empty-URI GOAWAY; `FailoverRedirect.cfg` and `FailoverRedirectRefused.cfg` let relays name a URI. The crash replays through `src/offline.test.ts` (a dropped session reads connecting), the refused redirect through `a refused redirect ends the connection instead of reconnecting` in `@moq/net`, and the sticky redirect through `residual: a redirected connection redials only its dead target` there. @@ -40,13 +40,14 @@ through `residual: a redirected connection redials only its dead target` there. ## Run ```sh -just js tla watch/verify/Watch.tla -just js tla watch/verify/Failover.tla -just js tla watch/verify/Failover.tla -config js/watch/verify/Redirect.cfg +just test tla js/watch ``` -TLC exits non-zero while a rule above breaks. To draw a chart, run `/statechart` with the spec, -its config, and its `.view.ts`. +This runs every config, as Nightly does. `Watch.cfg` and `Failover.cfg` check the rules that hold +and must pass; each rule that breaks has its own config, marked `\* Open finding:`, which must +fail until the fix moves its rule into the passing one. A config checks the spec its name starts +with. To draw a chart, run `/statechart` with the spec, a config, and its `.view.ts`; pass a +finding's config to chart its break. ## Left out diff --git a/quest/m1/model-checks-nightly.md b/quest/m1/model-checks-nightly.md deleted file mode 100644 index b6e282bb7c2..00000000000 --- a/quest/m1/model-checks-nightly.md +++ /dev/null @@ -1,42 +0,0 @@ -# [M] The TLA+ models run nightly and reviews resync them - -## Goal - -Every TLA+ model in the tree runs in Nightly: each passing config must pass, -and each open-finding config must still fail, so a fix that should promote -its rule is noticed. A PR that changes code a model covers gets that model -rerun and resynced in review, and renaming or deleting a function a model -covers fails `just check` until its skill names the new one, so no model -keeps passing on code that no longer exists. - -## Plan - -Decided 2026-11-10 (maintainer, /quest-plan), after [moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447) -added the models (`rs/moq-net/verify/*`, `js/*/verify/`) and their `verify-*` -skills. - -- Nightly runs every config through the `tla` recipes. TLC runs take 20 s - to 2 min on 32 cores, so they stay out of the merge gate. An open-finding - config is one its skill lists under "Open finding"; the job expects TLC's - invariant-violation exit for it and fails on a pass. -- The JS flow models (`js/*/verify/`) check every rule in one config, so - each breaks on its open findings. Split them as the Rust models are: a - passing config plus one per open finding. Publish's `LiveMeansWatchable` - break is accepted (2026-11-10, a badge that leads the announce by - milliseconds), so it leaves the passing config. -- ✅ Drift is flagged by a review rule (rejected: a CI check that fails a PR - touching covered files without touching the model, which taxes every - refactor and comment edit; and both). `REVIEW.md` tells the reviewer to run - the matching `verify-*` skill when a PR changes a file in that skill's - Coverage table, and the PR updates the model and its sync commit. -- The periodic resync for changes that skipped review is a check-first item - in the [test coverage audit](/quest/m3/audit-test-coverage.md). -- ✅ `just check` also fails when a function a `verify-*` skill's Coverage - table names no longer exists in the file the row names (decided 2026-11-10, - maintainer, a later /quest-plan the same day; rejected: the review rule - alone). Unlike the touched-file check rejected above, it fires only when - modelled code is renamed or deleted, which is when the table's mapping is - already wrong. Give the tables one parseable form (a file, then its - symbols) and test the check on a renamed symbol. - -Public API: none. Wire: none. diff --git a/quest/m1/route-inductive.md b/quest/m1/route-inductive.md index ac1efca1f11..f732536f30d 100644 --- a/quest/m1/route-inductive.md +++ b/quest/m1/route-inductive.md @@ -50,5 +50,4 @@ Public API: none. Wire: none. ## Related -- [The TLA+ models run nightly and reviews resync them](/quest/m1/model-checks-nightly.md) - the Nightly job this check joins - [Route sweeps are checked against the TLA+ model](/quest/m1/route-trace-validation.md) - ties the same spec to the code diff --git a/quest/m1/route-mesh-model.md b/quest/m1/route-mesh-model.md index 6fb85c07bba..a58ea119887 100644 --- a/quest/m1/route-mesh-model.md +++ b/quest/m1/route-mesh-model.md @@ -28,8 +28,9 @@ for finalizing lite-07. external change, the queues empty and every relay shows what `QuietIsSettled` requires. - Keep explicit-state runs tractable with `SYMMETRY` over interchangeable - relays; run larger meshes with TLC's random simulation in Nightly - ([nightly models](/quest/m1/model-checks-nightly.md)). + relays; run larger meshes with TLC's random simulation in Nightly, whose + `just test tla` (`sh/tla.sh`) runs each config exhaustively today and needs + a way to mark a config for simulation. - Check `SimplePaths` and `NoCountToInfinity` on every mesh: a route names the nodes it crossed since [moq-dev/moq#8487](https://github.com/moq-dev/moq/pull/8487). diff --git a/quest/m3/audit-test-coverage.md b/quest/m3/audit-test-coverage.md index 92c91915ada..7eca1714560 100644 --- a/quest/m3/audit-test-coverage.md +++ b/quest/m3/audit-test-coverage.md @@ -42,7 +42,7 @@ Previous run, with its per-crate and per-package baselines: - Each TLA+ model's `verify-*` skill against the code it covers: run `/maintain-formal-verification-skill` on every one whose Coverage files changed since its sync commit, so a change that skipped review is resynced - (decided 2026-11-10, [nightly models](/quest/m1/model-checks-nightly.md)). + (decided 2026-11-10). ## Required diff --git a/quest/m3/net-redirect-fallback.md b/quest/m3/net-redirect-fallback.md index 84f7a726339..78cee04847e 100644 --- a/quest/m3/net-redirect-fallback.md +++ b/quest/m3/net-redirect-fallback.md @@ -9,7 +9,7 @@ still answers. ## Plan Found 2026-11-10 by the failover chart ([moq-dev/moq#8447](https://github.com/moq-dev/moq/pull/8447)): -`js/watch/verify/Redirect.cfg` breaks `RedialsALiveRelay` in 4 steps. In +`js/watch/verify/FailoverRedirect.cfg` breaks `RedialsALiveRelay` in 4 steps. In `js/net/src/connection/reload.ts` every redial after a redirect targets only the relay it was sent to, with no retry timeout by default, so the viewer redials a dead relay indefinitely. `residual: a redirected connection redials diff --git a/rs/moq-net/verify/reconnect/OfflineWhileServing.cfg b/rs/moq-net/verify/reconnect/OfflineWhileServing.cfg index f599a536433..a36b9efa1df 100644 --- a/rs/moq-net/verify/reconnect/OfflineWhileServing.cfg +++ b/rs/moq-net/verify/reconnect/OfflineWhileServing.cfg @@ -1,4 +1,4 @@ -\* The open finding a replacement dropping mid-handover: +\* Open finding: a replacement dropping mid-handover reports Disconnected, so \* OfflineMeansNothingServes fails. Once fixed, move the rule into \* Handover.cfg and delete this file. CONSTANTS diff --git a/rs/moq-net/verify/reconnect/RedirectLost.cfg b/rs/moq-net/verify/reconnect/RedirectLost.cfg index fb22ee51b6d..6ce1c02840c 100644 --- a/rs/moq-net/verify/reconnect/RedirectLost.cfg +++ b/rs/moq-net/verify/reconnect/RedirectLost.cfg @@ -1,5 +1,6 @@ -\* The open finding a GOAWAY after a probe got through: RedirectFollowed fails. -\* Once fixed, move the rule into Handover.cfg and delete this file. +\* Open finding: a GOAWAY after a probe got through is lost, so +\* RedirectFollowed fails. Once fixed, move the rule into Handover.cfg and +\* delete this file. CONSTANTS Endpoints = {"a", "b"} Start = "a" diff --git a/rs/moq-net/verify/subscribe/StreamWins.cfg b/rs/moq-net/verify/subscribe/StreamWins.cfg index c9a8214310a..bd829855150 100644 --- a/rs/moq-net/verify/subscribe/StreamWins.cfg +++ b/rs/moq-net/verify/subscribe/StreamWins.cfg @@ -1,4 +1,4 @@ -\* The open finding: a group read whole is recorded stale. Once it is fixed, move +\* Open finding: a group read whole is recorded stale. Once it is fixed, move \* StreamWins into SubscribeAccounting.cfg and delete this file. CONSTANTS Groups = {0} diff --git a/sh/coverage-tables.py b/sh/coverage-tables.py new file mode 100644 index 00000000000..db1090fcf73 --- /dev/null +++ b/sh/coverage-tables.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Fail when a `verify-*` skill's Coverage table names code that no longer exists. + +Each row's Code cell is a repo-relative Rust path, then the symbols it covers, +each in backticks; another path starts that file's symbols. Parenthesized notes +are prose and go unchecked. A symbol `A::b` must define `A` and `b` in its file: +an item (`fn`, `struct`, `enum`, ...), a field, or a variant, and `b` must sit +inside `A`'s own definition or `impl` blocks, so a same-named member of another +type does not count. Lowercase leading segments name modules and are skipped. +A rename or deletion of modelled code fails here until the table, and so the +model's mapping, follows it. + +`--root DIR` checks another tree, for the tests. +""" + +import re +import sys +from pathlib import Path + +TICKED = re.compile(r"`([^`]+)`") +PAREN = re.compile(r"\([^()]*\)") +ITEM = r"\b(?:fn|struct|enum|trait|type|mod|const|static|union)\s+{}\b" +# Where a type's members live: its definition, and each `impl` whose header `implements` names it. +DEFINITION = r"\b(?:struct|enum|union|trait)\s+{}\b[^;{{]*\{{" +IMPL = re.compile(r"\bimpl\b([^;{]*)\{") +# A field `name: T`, or a variant `Name,`, `Name(..)`, `Name {{`, or `Name` alone; a call such as `name(..)` is neither. +FIELD = r"^\s*(?:pub(?:\([^)]*\))?\s+)?{}\s*:(?!:)" +VARIANT = r"^\s*{}\s*(?:,|\(|\{{|$)" + + +def rows(skill: Path): + """Yield (line number, Code cell) for each body row of the Coverage table.""" + inside = False + for number, line in enumerate(skill.read_text().splitlines(), 1): + if line.startswith("## "): + inside = line.strip() == "## Coverage" + continue + if not inside or not line.startswith("|"): + continue + cell = line.split("|")[1].strip() + if cell == "Code" or set(cell) <= set("-: "): + continue + yield number, cell + + +def defines(source: str, name: str) -> bool: + escaped = re.escape(name) + member = VARIANT if name[:1].isupper() else FIELD + return bool(re.search(ITEM.format(escaped), source) or re.search(member.format(escaped), source, re.MULTILINE)) + + +def implements(header: str) -> str: + """The last segment of the type an `impl` header implements, ignoring generics, bounds, and `where`.""" + flat, depth, prev = "", 0, "" + for char in header: + if not (char == ">" and prev == "-"): + depth += {"<": 1, ">": -1}.get(char, 0) + prev = char + if depth == 0 and char != ">": + flat += char + flat = re.split(r"\bwhere\b", flat)[0] + flat = re.split(r"\bfor\b", flat)[-1] + path = re.findall(r"[A-Za-z_][\w:]*", flat) + return path[0].split("::")[-1] if path else "" + + +def bodies(source: str, owner: str) -> str: + """Join the brace-matched bodies of `owner`'s definition and `impl` blocks.""" + spans = [] + impls = (m for m in IMPL.finditer(source) if implements(m.group(1)) == owner) + for match in sorted([*re.finditer(DEFINITION.format(re.escape(owner)), source), *impls], key=lambda m: m.start()): + depth, start = 0, match.end() - 1 + for end in range(start, len(source)): + depth += {"{": 1, "}": -1}.get(source[end], 0) + if depth == 0: + spans.append(source[start + 1 : end]) + break + return "\n".join(spans) + + +def check(root: Path) -> list[str]: + errors = [] + skills = sorted(root.glob(".claude/skills/verify-*/SKILL.md")) + for skill in skills: + where = skill.relative_to(root) + found = False + for number, cell in rows(skill): + found = True + tokens = TICKED.findall(PAREN.sub("", cell)) + source = None + for token in tokens: + if "/" in token: + path = root / token + if not token.endswith(".rs") or not path.is_file(): + errors.append(f"{where}:{number}: `{token}` is not a Rust file in the tree") + source = False + else: + source = path.read_text() + continue + if source is None: + errors.append(f"{where}:{number}: the Code cell must start with a repo-relative path") + break + if source is False: + continue + segments = token.split("::") + for index, segment in enumerate(segments): + if index < len(segments) - 1 and not segment[:1].isupper(): + continue + owner = segments[index - 1] if index and segments[index - 1][:1].isupper() else None + if not defines(bodies(source, owner) if owner else source, segment): + place = f"`{owner}`" if owner else "file" + errors.append(f"{where}:{number}: `{token}`: no `{segment}` defined in its {place}") + if not tokens: + errors.append(f"{where}:{number}: the Code cell names no code") + if not found: + errors.append(f"{where}: no Coverage table") + return errors + + +def main() -> int: + root = Path(sys.argv[2]) if sys.argv[1:2] == ["--root"] else Path(__file__).resolve().parent.parent + errors = check(root) + for error in errors: + print(f"error: {error}", file=sys.stderr) + return 1 if errors else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/sh/coverage-tables.test.py b/sh/coverage-tables.test.py new file mode 100644 index 00000000000..353fb8b4098 --- /dev/null +++ b/sh/coverage-tables.test.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +"""Check sh/coverage-tables.py against a scratch tree.""" + +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parent / "coverage-tables.py" +SOURCE = """\ +pub enum Restart { + Redial, + Wait, +} +pub struct Reader { + pub(crate) bound: u64, +} +impl u64> Reader { + pub fn sync(&mut self) {} +} +pub struct Mirror(T); +impl> Mirror where T: Into { + fn sync(&self) {} +} +impl std::fmt::Debug for Mirror> {} +""" +SKILL = """\ +# Verify + +## Coverage + +| Code | Model | +| --- | --- | +| `src/lib.rs` `Reader::sync` (`Unmodelled::thing`), `bound` | `Sync` | +| `src/lib.rs` `Restart::Wait`; `src/other.rs` `run` | `Wait` | + +## Left out +""" + + +class CoverageTablesTests(unittest.TestCase): + def setUp(self): + temp = tempfile.TemporaryDirectory() + self.addCleanup(temp.cleanup) + self.root = Path(temp.name) + self.write("src/lib.rs", SOURCE) + self.write("src/other.rs", "async fn run() {}\n") + self.write(".claude/skills/verify-x/SKILL.md", SKILL) + + def write(self, path, text): + (self.root / path).parent.mkdir(parents=True, exist_ok=True) + (self.root / path).write_text(text) + + def check(self): + return subprocess.run([sys.executable, SCRIPT, "--root", self.root], capture_output=True, text=True) + + def test_items_fields_and_variants_pass(self): + result = self.check() + self.assertEqual(result.returncode, 0, result.stderr) + + def test_a_renamed_function_fails(self): + self.write("src/lib.rs", SOURCE.replace("fn sync", "fn resync")) + result = self.check() + self.assertEqual(result.returncode, 1) + self.assertIn("SKILL.md:7: `Reader::sync`: no `sync` defined", result.stderr) + + def test_a_rename_is_not_hidden_by_another_types_member(self): + self.write("src/lib.rs", SOURCE.replace("pub fn sync", "pub fn resync")) + self.assertIn("`Reader::sync`: no `sync` defined in its `Reader`", self.check().stderr) + + def test_a_renamed_type_fails(self): + self.write("src/lib.rs", SOURCE.replace("enum Restart", "enum Policy")) + self.assertIn("no `Restart` defined", self.check().stderr) + + def test_a_moved_file_fails(self): + (self.root / "src/other.rs").unlink() + self.assertIn("`src/other.rs` is not a Rust file", self.check().stderr) + + def test_a_cell_without_a_path_fails(self): + self.write(".claude/skills/verify-x/SKILL.md", SKILL.replace("`src/lib.rs` `Reader", "`Reader")) + self.assertIn("must start with a repo-relative path", self.check().stderr) + + def test_a_call_is_not_a_definition(self): + caller = "pub fn other(&mut self) {\n sync();\n }" + self.write("src/lib.rs", SOURCE.replace("pub fn sync(&mut self) {}", caller)) + self.assertIn("`Reader::sync`: no `sync` defined", self.check().stderr) + + def test_an_empty_file_fails(self): + self.write("src/other.rs", "") + self.assertIn("`run`: no `run` defined", self.check().stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/sh/dispatch.sh b/sh/dispatch.sh index 6853e11ac2a..32f7664eef9 100755 --- a/sh/dispatch.sh +++ b/sh/dispatch.sh @@ -231,6 +231,10 @@ module large_files '' # check; only an explicit --all has none. module migrate '(^|/)MIGRATE\.md$' module text '' +# A verify-* skill's Coverage table against the Rust it names, so a rename +# or deletion of modelled code fails until the table follows it. +module coverage_tables '^(rs/|\.claude/skills/verify-|sh/coverage-tables\.py$)' \ + tools='python3' module toml '' \ tools='taplo' fix module nix '' \ @@ -453,6 +457,7 @@ step() { check:large_files) run sh/large-files.sh "$large_base" ;; check:migrate) [[ "$base" == --all ]] || run sh/migrate.sh "$base" ;; check:text) run sh/text.sh ;; + check:coverage_tables) run python3 sh/coverage-tables.py ;; check:python) files '*.py' '*.pyi' run xargs -0 ruff check --force-exclude <"$listed" diff --git a/sh/tla.sh b/sh/tla.sh new file mode 100755 index 00000000000..bb817b9c3b9 --- /dev/null +++ b/sh/tla.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Model-check every TLA+ config under the given paths (default: all of them) +# through the `tla` recipes. A config with a `\* Open finding:` line must break +# a rule (a deadlock, invariant, property, or assertion), so a fix that should +# promote its rule is noticed; every other config must pass. +set -euo pipefail +shopt -s nullglob +cd "$(git rev-parse --show-toplevel)" + +[ $# -ne 0 ] || set -- rs js +mapfile -t configs < <(git ls-files --cached --others --exclude-standard -- "$@" | grep -E '^(rs|js)/.*/verify/.*\.cfg$' || true) +if [ ${#configs[@]} -eq 0 ]; then + echo "error: no TLA+ config under: $*" >&2 + exit 1 +fi + +failed=() +for cfg in "${configs[@]}"; do + # The index still lists a config deleted but not yet staged. + [ -f "$cfg" ] || continue + # The directory's spec, or where it holds several, the one the config's name + # starts with. TLC's trace-exploration output (`*_TTrace_*.tla`) is no spec. + dir=$(dirname "$cfg") + name=$(basename "$cfg") + specs=() + for spec in "$dir"/*.tla; do + [[ "$spec" == *_TTrace_* ]] || specs+=("$spec") + done + if [ ${#specs[@]} -gt 1 ]; then + all=("${specs[@]}") + specs=() + for spec in "${all[@]}"; do + [[ "$name" != "$(basename "$spec" .tla)"* ]] || specs+=("$spec") + done + fi + if [ ${#specs[@]} -ne 1 ]; then + failed+=("$cfg: no single spec in $dir") + continue + fi + lang=${cfg%%/*} + finding=false + ! grep -q '^\\\* Open finding:' "$cfg" || finding=true + + echo "::group::$cfg" + start=$SECONDS + status=0 + just "$lang" tla "${specs[0]#"$lang"/}" -config "$cfg" || status=$? + echo "::endgroup::" + echo "$cfg: exit $status in $((SECONDS - start))s" + + # TLC exits 11 to 14 for a deadlock, a violated invariant, a violated + # property, and a failed assertion; anything else nonzero is an error. + if $finding; then + case $status in + 11 | 12 | 13 | 14) ;; + 0) failed+=("$cfg: passes, so its finding is fixed: promote its rule as the config says") ;; + *) failed+=("$cfg: exit $status, not a broken rule") ;; + esac + elif [ "$status" -ne 0 ]; then + failed+=("$cfg: exit $status") + fi +done + +if [ ${#failed[@]} -ne 0 ]; then + printf 'error: %s\n' "${failed[@]}" >&2 + exit 1 +fi diff --git a/test/justfile b/test/justfile index e247f2a116e..1c89797c302 100644 --- a/test/justfile +++ b/test/justfile @@ -294,6 +294,10 @@ json-properties: resume-model: bash ../sh/local.sh bash resume-model.sh +# Model-check every TLA+ config under the given paths (default: all): open findings must still break. +tla *paths: + bash ../sh/tla.sh {{ paths }} + # Prove the route key model, and that each variant breaks the lemmas it names. route-key-model: bash route-key-model.sh