Repository navigation
No mutation of the route and resume code survives its tests #8712
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
40 commits
Select commit
Hold shift + click to select a range
a596c16
chore(quest): claim quest/m1/mutants.md
kixelated e7cc192
ci(mutants): run cargo-mutants over the route and resume code
kixelated 4b30ee3
ci(mutants): shard Nightly into changed code and a rotation
kixelated bf3e928
ci(mutants): TEMPORARY full sweep on the PR
kixelated 8ecf117
test(route): rotated identities are fresh and in range for any seed
kixelated f858e2c
test(route): retire keeps an outside floor only while it is newer tha…
kixelated f747bc7
test(route): a batch replacing a stream's only claim keeps its route
kixelated 2fd555b
ci(mutants): the integration binary joins the suite
kixelated 1915925
test(route): pin a churn whose batch moves the merged floors
kixelated 66261de
test(route): pin which interests a route meets and whose uncertified …
kixelated 6e8ef6f
test(route): a forwarded request goes with either session it rides on
kixelated ff0521c
test(route): a relay keeps nothing once its only route closes
kixelated 46eb7ff
test(route): an uncertified claim dropped goes offline while a certif…
kixelated 6aef15c
refactor(route): move test-only checks and the stale merge shortcut i…
kixelated 266ee5f
test(route): pin the node driver, hop folding, and scoped-key holds
kixelated af05e3f
chore(mutants): exclude the awaits match a test cannot tell apart, an…
kixelated db3c650
test(resume): pin the reader, recovery, and fetch paths past line 1300
kixelated dac1602
chore(mutants): exclude the resume mutants a test cannot tell apart
kixelated cd2d247
test(resume): pin the front, copy, and reader paths below the read loop
kixelated a0fed04
chore(mutants): exclude the remaining resume mutants a test cannot te…
kixelated d3b845a
quest: finish mutants, split the other modelled code into its own quest
kixelated 74d2427
test(route): pin held-certifier availability, retiring outside the in…
kixelated 2c93875
chore(mutants): exclude the batch mark a new stream's interest makes …
kixelated bfa724a
style(mutants): format the exclusions
kixelated 1eb0fbb
test(mutants): kill the last survivors and drop the temporary sweep
kixelated 4992ff7
Merge origin/main: route by path (#8622)
kixelated ad4851e
test(route): the path router's rtt sum and changed claims
kixelated 0745079
test(route): the path router's guards, doubt, drain, and rotation
kixelated 61ab393
chore(mutants): exclude the path router's work counter and empty collect
kixelated a2d7f9e
test(route): the path router's holds, reflected sources, newest floor…
kixelated e3d6c6d
test(route): share the path offer helper; exclude know's tie and incr…
kixelated 1ad74dc
test(route): a narrow route at a newer floor shows over a wider one
kixelated e37b884
refactor(route): drop Reader::touch, which apply already does
kixelated 1034bf3
test(route): the path router's reports, ended streams, and work per c…
kixelated 495d0f6
chore(mutants): exclude the path router's equivalent survivors; file …
kixelated 59e9c47
fix(mutants): mutant builds read the shared cache and publish nothing
kixelated 12b404e
Merge remote-tracking branch 'origin/main' into quest/m1/mutants
kixelated 7783554
quest: resume-redesign's wait no longer asks about the finished mutat…
kixelated 594ae01
Merge remote-tracking branch 'origin/main' into quest/m1/mutants
kixelated 57f8da6
Merge remote-tracking branch 'origin/main' into quest/m1/mutants
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| # cargo-mutants: Nightly fails on any mutant of these files the tests miss. | ||
| examine_globs = ["rs/moq-net/src/route/**/*.rs", "rs/moq-net/src/model/resume.rs"] | ||
| exclude_globs = ["rs/moq-net/src/route/tests/**"] | ||
| test_tool = "nextest" | ||
| all_features = true | ||
| additional_cargo_args = ["--locked"] | ||
| # The unit tests, and the integration binary, which drives the route driver | ||
| # (`route/node.rs`) through real sessions. | ||
| additional_cargo_test_args = ["--lib", "--test", "integration"] | ||
|
|
||
| # Mutants no test can catch, because the result is the same either way. Each | ||
| # pattern names its function and replacement, not its line. A column pins it | ||
| # only where the same function has another such mutant that tests do catch. | ||
| exclude_re = [ | ||
| # Debug output only feeds logs. | ||
| '<impl (std::)?fmt::Debug for (Scope|Node|Stream|RouteKey|Proof)>::fmt', | ||
| # The rotation's xorshift step only picks which fresh identity comes next, | ||
| # and the guard after it enforces freshness and range. | ||
| 'replace (\^= with \|=|<< with >>|>> with <<) in Router<N, A>::rotate', | ||
| # An index of a neighbor's interest: `carries()` filters each entry again. | ||
| 'replace Router<N, A>::asked -> Path with Default::default\(\)', | ||
| # `full(old) == full(new)` is the same with both sides negated. | ||
| 'replace >= with < in Router<N, A>::route$', | ||
| # A larger `differs` set only costs work: it is compared again before use. | ||
| 'replace && with \|\| in Heard<A>::compare$', | ||
| # The test-only `Proof::LEN`. | ||
| 'proof\.rs:\d+:\d+: replace \+ with \*$', | ||
| # The stale-view shortcut only saves work: without it, or at equal revisions, | ||
| # where both views hold the same claims, the full merge changes nothing. | ||
| 'replace (> with (==|>=) in View<A>::outdates|View<A>::outdates -> bool with false)$', | ||
| # Dropping the cached context more often makes the next pass full, which | ||
| # reaches the same result. | ||
| ':\d+:(25|38): replace && with \|\| in Router<N, A>::recompute$', | ||
| # A claim an answering certifier certifies is always in the merged view. | ||
| ':\d+:35: replace && with \|\| in Router<N, A>::meets$', | ||
| # A new stream's first batch either flips its certifier's answer, which marks | ||
| # every claim, or reaches the reader only through the merge, which marks its own | ||
| # changes; marking all only reorders independent changes in a forwarded batch. | ||
| ':\d+:32: replace != with == in Router<N, A>::batch$', | ||
| # At an equal metric both arms assign the same distance. | ||
| 'replace < with <= in Router<N, A>::record$', | ||
| # The full presentation of a stream the interest misses is empty, and a full | ||
| # presentation equals the incremental one. | ||
| 'replace match guard (!self\.meets\(id, ctx, watch\)|!dirty\.all) with false in Router<N, A>::advertise$', | ||
| # `advertised` is read only for qualified identities. | ||
| 'replace & with (\^|\|) in Router<N, A>::advertise$', | ||
| # For its own identity, the node's own view answers every claim, `current` | ||
| # never holds it, and nothing serves it. | ||
| ':\d+:31: replace && with \|\| in Router<N, A>::shown_now$', | ||
| # The nearest home with a slot at or above a presented claim always serves | ||
| # that claim's instance and epoch, so a looser match picks the same home. | ||
| 'node\.rs:\d+:57: replace && with \|\| in awaits$', | ||
| # resume.rs: the sweep of handed-out groups only bounds memory, since | ||
| # `let_go` drops dead entries itself before anything reads `out`. | ||
| 'replace (== with !=|> with >=) in Reader::hand_out$', | ||
| # Polling a covered or finished copy hands out nothing, so polling it anyway | ||
| # only costs work. | ||
| ':\d+:36: replace \|\| with && in Reader::poll_lowest$', | ||
| # A run starting at `until` clips to an empty range, which has no gaps. | ||
| ':\d+:36: replace < with <= in Reader::exhausted$', | ||
| # `learn_end` just polled the same copy and returned on an `Ok` end. | ||
| 'delete match arm Poll::Ready\(Ok\(fin\)\) in Reader::poll_finished$', | ||
| # The one extra copy searched is the route's own, which never has an end. | ||
| ':\d+:32: replace - with / in Reader::backfill$', | ||
| # At the same generation the serving copy is the one that just failed the | ||
| # group, so it never answers at that index. | ||
| 'replace match guard !same with true in Recover::poll_serving$', | ||
| # A replacement starting at frame 0 already holds the head. | ||
| 'replace > with >= in lend_head$', | ||
| # Race-only, not equivalent: each call follows a read of the route in the same | ||
| # poll, so these differ only when another thread writes the route between that | ||
| # read and this check, a window no deterministic test can stage. There the | ||
| # original retries on the new copy and the arm prevents a lost wake. | ||
| 'replace \|\| with && in Fetching::poll_moved$', | ||
| ':\d+:6: delete match arm Some\(Some\(Err\(err\)\)\) in <impl kio::Task for Fetching>::poll$', | ||
| # The reader list's sweep only bounds memory: a dead entry fails every | ||
| # `Weak::upgrade`, and `subscribe` sweeps the list again. | ||
| 'replace > with >= in Producer::serve_from$', | ||
| 'replace == with != in Consumer::subscribe$', | ||
| # Dead: every write of `until` also writes `since`, so the superseded check | ||
| # never runs. `ResumeReader.tla` models it, so it goes with the model. | ||
| 'replace >= with < in Copy::retains$', | ||
| # A run starting at the end clips to an empty range, on which nothing lands | ||
| # and nothing is missing. | ||
| 'replace < with <= in Copy::retains$', | ||
| 'replace < with <= in Retain<.*>::holds_window$', | ||
| # Walking again before anything the read parked on fires returns the same | ||
| # `Pending`, so the idle shortcut only saves work. | ||
| 'replace Subscriber::parked -> bool with false$', | ||
| # path.rs: `step` only counts visits for the tests' work budgets, which | ||
| # read the count as an upper bound. | ||
| 'path\.rs:\d+:\d+: replace Router<N, A>::step with \(\)$', | ||
| # A path-rule identity nothing holds has no route left, and the update that | ||
| # removed its last one already dropped its views, so `collect` changes nothing. | ||
| 'path\.rs:\d+:\d+: replace Router<N, A>::collect with \(\)$', | ||
| # Tied candidates share a floor, so by the identity rule they present the | ||
| # same content at the prefix. | ||
| 'path\.rs:\d+:\d+: replace > with >= in Router<N, A>::know$', | ||
| # A route's change updates every view it covers, so a batch's only newer | ||
| # candidate is its own route and peer: these only turn off the incremental | ||
| # presentation, which the test build asserts equals the full one. | ||
| 'path\.rs:\d+:(51|74): replace == with != in Router<N, A>::know$', | ||
| # An ended stream's final batch always has a floor and withdraws claims: an | ||
| # offer is sent only while a claim meets it. | ||
| 'path\.rs:\d+:\d+: (replace \|\| with &&|delete !) in Router<N, A>::end_sent$', | ||
| # The full diff it skips holds the same changes in another order, and a source | ||
| # switch has already marked every claim that differs. | ||
| 'path\.rs:\d+:80: replace != with == in Router<N, A>::present$', | ||
| # NOT equivalent: Rust lets an ended stream's child wait go where | ||
| # RouteHandoff.tla keeps it. quest/m2/cluster-routes/ended-child-wait.md | ||
| # aligns the two and replaces this with a test. | ||
| 'path\.rs:\d+:55: replace && with \|\| in Router<N, A>::advertise$', | ||
| ] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| name: Mutants | ||
|
|
||
| # cargo-mutants edits the route and resume code one small change at a time | ||
| # (`.cargo/mutants.toml` names the files and the tests), and a job fails on any | ||
| # edit its tests miss. Each mutant costs a rebuild of moq-net's tests, too many | ||
| # for a merge gate, so Nightly runs it; sh/rs/mutants.sh says what each shard takes. | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| workflow_call: | ||
|
|
||
| jobs: | ||
| mutants: | ||
| name: Mutants (${{ matrix.shard }}/6) | ||
| runs-on: ubuntu-24.04-arm | ||
| # A day of heavy route churn adds about an hour of changed mutants to the rotation's half hour. | ||
| timeout-minutes: 150 | ||
| permissions: | ||
| contents: read | ||
| # The changed mutants are those since the previous scheduled Nightly's commit. | ||
| actions: read | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| shard: [0, 1, 2, 3, 4, 5] | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: ./.github/actions/nix | ||
| # Every mutant's build is a one-off, so the job reads the shared cache | ||
| # with the repository's key and publishes nothing. | ||
| - uses: ./.github/actions/rust-cache | ||
| with: | ||
| access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }} | ||
| secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }} | ||
| write: false | ||
| # The checkout is thrown away, so mutating it in place keeps the build | ||
| # incremental and cached instead of copying the tree per mutant. | ||
| - name: Mutate | ||
| run: nix develop --command just rs mutants nightly ${{ matrix.shard }} 6 | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| - name: Keep the outcomes | ||
| if: ${{ !cancelled() }} | ||
| uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 | ||
| with: | ||
| name: mutants-${{ matrix.shard }} | ||
| path: | | ||
| target/mutants/*/mutants.out | ||
| target/mutants/missed.txt | ||
| retention-days: 14 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # [S] An ended stream still waits on the child it taught | ||
|
|
||
| ## Goal | ||
|
|
||
| `path::Router` holds a lost route until every neighbour that learned it from | ||
| this node has answered, as `RouteHandoff.tla`'s `Pending` does, including a | ||
| neighbour whose offer back arrives after this node ended the route on its | ||
| stream. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-14 (maintainer, /quest-complete): ✅ keep in m2 (rejected: drop it, move to m3). | ||
|
|
||
| Found 2026-11-14 by the mutation run ([moq-dev/moq#8712](https://github.com/moq-dev/moq/pull/8712)), | ||
| which excludes the one mutant it exposes (`&&` to `||` in | ||
| `path::Router::advertise`'s cleanup of ended streams) until this lands. | ||
|
|
||
| When a stream ends, `advertise` removes its sent entry unless the peer's route | ||
| is a pending child right then. The model keeps every ended entry's source | ||
| (`NoSent(t.src)`), so its `Pending` also counts a child offer that the | ||
| neighbour sent before it heard the END. With these steps, Rust sends peer 1 | ||
| `Route { reach: None }` at the last step, while the model keeps waiting on | ||
| peer 2: | ||
|
|
||
| ``` | ||
| router 99; peers 1, 2, 3 on Link::default | ||
| route(1, 7, "", through(1)); batch(1, 7, "", [a/x = 1, b/x = 1], Some(1)) | ||
| peer_interest 1: "", 2: "b", 3: "" | ||
| route(3, 7, "", along[(7,0), (6,1), (4,1), (1,1), (99,3), (3,1)]) // a child of the "" stream | ||
| route(2, 7, "b", along[(7,0), (99,9), (5,1), (2,1)]) // not a child | ||
| batch(1, 7, "", [b/x = None], Some(2)) // the "b" stream to 2 ends | ||
| dead(1, closed([], false)) // "" holds under 1 | ||
| route(2, 7, "b", along[(7,0), (6,1), (4,1), (1,1), (99,2), (2,1)]) // 2's offer back, sent before the END reached it | ||
| route(3, 7, "", None) // peer 3 answers | ||
| ``` | ||
|
|
||
| (`through`, `along`, `closed` as in `route/tests/scope.rs`.) No loop or hold | ||
| cycle follows, since peer 2's stale path names this node, but the code departs | ||
| from the module doc's rule and from the checked model. | ||
|
|
||
| Recommendation, for the maintainer's review (taken by the agent under | ||
| /quest-afk): **align Rust with the model.** Keep an ended entry's source until | ||
| its stream is replaced or its session closes, make the steps above the test | ||
| that the route still holds, and drop the exclusion from `.cargo/mutants.toml`. | ||
| The model needs no change, so no new TLC run is required beyond the usual | ||
| `/verify-route`. Rejected: narrowing the model's `Pending` to the children | ||
| current when the stream ended, which weakens the checked rule to fit the code | ||
| and needs a new TLC run. | ||
|
|
||
| Public API: none. Wire: none. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # [L] No mutation of the other modelled code survives its tests | ||
|
|
||
| ## Goal | ||
|
|
||
| The Nightly mutation run that guards the route and resume code (#8712) also | ||
| covers the other code a TLA+ model checks: `rs/moq-net/src/reconnect.rs`, | ||
| `lite/publisher.rs`, `lite/subscriber.rs`, and `tail.rs`. The tests fail on | ||
| every edit to them that changes behaviour. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-14 (maintainer, /quest-complete): ✅ keep in m4 (rejected: drop it). | ||
|
|
||
| Split from the route and resume run (#8712, taken by the agent under | ||
| /quest-afk, for the maintainer's review), which measured what these files | ||
| would add. Its quest said they join once its cost was known, and they do not | ||
| fit as they are: | ||
|
|
||
| - They hold 1681 mutants against the route and resume code's 1300, so the | ||
| rotation (`sh/rs/mutants.sh`) would more than double each night's runner | ||
| time, or need twice the rotation's nights to cover everything once. | ||
| - Their survivors from a first full run have to be killed or excluded before | ||
| Nightly can fail on them, as #8712 did for route and resume. | ||
| - Their tests are the lite session and reconnect suites, not the route ones, | ||
| so the config needs a second test selection or a second `.cargo/mutants.toml` | ||
| run. | ||
|
|
||
| Decide which nights the run gets (more shards, a longer rotation, or a | ||
| weekly job) once a first full sweep says how long their tests take per | ||
| mutant; then sweep, kill or exclude every survivor, and add them to | ||
| `examine_globs`. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Test coverage audit](/quest/m3/audit-test-coverage.md) - the line-coverage reports this complements |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.