Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
a596c16
chore(quest): claim quest/m1/mutants.md
kixelated Nov 14, 2026
e7cc192
ci(mutants): run cargo-mutants over the route and resume code
kixelated Nov 14, 2026
4b30ee3
ci(mutants): shard Nightly into changed code and a rotation
kixelated Nov 14, 2026
bf3e928
ci(mutants): TEMPORARY full sweep on the PR
kixelated Nov 14, 2026
8ecf117
test(route): rotated identities are fresh and in range for any seed
kixelated Nov 14, 2026
f858e2c
test(route): retire keeps an outside floor only while it is newer tha…
kixelated Nov 14, 2026
f747bc7
test(route): a batch replacing a stream's only claim keeps its route
kixelated Nov 14, 2026
2fd555b
ci(mutants): the integration binary joins the suite
kixelated Nov 14, 2026
1915925
test(route): pin a churn whose batch moves the merged floors
kixelated Nov 14, 2026
66261de
test(route): pin which interests a route meets and whose uncertified …
kixelated Nov 14, 2026
6e8ef6f
test(route): a forwarded request goes with either session it rides on
kixelated Nov 14, 2026
ff0521c
test(route): a relay keeps nothing once its only route closes
kixelated Nov 14, 2026
46eb7ff
test(route): an uncertified claim dropped goes offline while a certif…
kixelated Nov 14, 2026
6aef15c
refactor(route): move test-only checks and the stale merge shortcut i…
kixelated Nov 14, 2026
266ee5f
test(route): pin the node driver, hop folding, and scoped-key holds
kixelated Nov 14, 2026
af05e3f
chore(mutants): exclude the awaits match a test cannot tell apart, an…
kixelated Nov 14, 2026
db3c650
test(resume): pin the reader, recovery, and fetch paths past line 1300
kixelated Nov 14, 2026
dac1602
chore(mutants): exclude the resume mutants a test cannot tell apart
kixelated Nov 14, 2026
cd2d247
test(resume): pin the front, copy, and reader paths below the read loop
kixelated Nov 14, 2026
a0fed04
chore(mutants): exclude the remaining resume mutants a test cannot te…
kixelated Nov 14, 2026
d3b845a
quest: finish mutants, split the other modelled code into its own quest
kixelated Nov 14, 2026
74d2427
test(route): pin held-certifier availability, retiring outside the in…
kixelated Nov 14, 2026
2c93875
chore(mutants): exclude the batch mark a new stream's interest makes …
kixelated Nov 14, 2026
bfa724a
style(mutants): format the exclusions
kixelated Nov 14, 2026
1eb0fbb
test(mutants): kill the last survivors and drop the temporary sweep
kixelated Nov 14, 2026
4992ff7
Merge origin/main: route by path (#8622)
kixelated Nov 14, 2026
ad4851e
test(route): the path router's rtt sum and changed claims
kixelated Nov 14, 2026
0745079
test(route): the path router's guards, doubt, drain, and rotation
kixelated Nov 14, 2026
61ab393
chore(mutants): exclude the path router's work counter and empty collect
kixelated Nov 14, 2026
a2d7f9e
test(route): the path router's holds, reflected sources, newest floor…
kixelated Nov 14, 2026
e3d6c6d
test(route): share the path offer helper; exclude know's tie and incr…
kixelated Nov 14, 2026
1ad74dc
test(route): a narrow route at a newer floor shows over a wider one
kixelated Nov 14, 2026
e37b884
refactor(route): drop Reader::touch, which apply already does
kixelated Nov 14, 2026
1034bf3
test(route): the path router's reports, ended streams, and work per c…
kixelated Nov 14, 2026
495d0f6
chore(mutants): exclude the path router's equivalent survivors; file …
kixelated Nov 14, 2026
59e9c47
fix(mutants): mutant builds read the shared cache and publish nothing
kixelated Nov 14, 2026
12b404e
Merge remote-tracking branch 'origin/main' into quest/m1/mutants
kixelated Nov 14, 2026
7783554
quest: resume-redesign's wait no longer asks about the finished mutat…
kixelated Nov 14, 2026
594ae01
Merge remote-tracking branch 'origin/main' into quest/m1/mutants
kixelated Nov 14, 2026
57f8da6
Merge remote-tracking branch 'origin/main' into quest/m1/mutants
kixelated Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions .cargo/mutants.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# cargo-mutants: Nightly fails on any mutant of these files the tests miss.
examine_globs = ["rs/moq-net/src/route/**/*.rs", "rs/moq-net/src/model/resume.rs"]
exclude_globs = ["rs/moq-net/src/route/tests/**"]
test_tool = "nextest"
all_features = true
additional_cargo_args = ["--locked"]
# The unit tests, and the integration binary, which drives the route driver
# (`route/node.rs`) through real sessions.
additional_cargo_test_args = ["--lib", "--test", "integration"]

# Mutants no test can catch, because the result is the same either way. Each
# pattern names its function and replacement, not its line. A column pins it
# only where the same function has another such mutant that tests do catch.
exclude_re = [
# Debug output only feeds logs.
'<impl (std::)?fmt::Debug for (Scope|Node|Stream|RouteKey|Proof)>::fmt',
# The rotation's xorshift step only picks which fresh identity comes next,
# and the guard after it enforces freshness and range.
'replace (\^= with \|=|<< with >>|>> with <<) in Router<N, A>::rotate',
# An index of a neighbor's interest: `carries()` filters each entry again.
'replace Router<N, A>::asked -> Path with Default::default\(\)',
# `full(old) == full(new)` is the same with both sides negated.
'replace >= with < in Router<N, A>::route$',
# A larger `differs` set only costs work: it is compared again before use.
'replace && with \|\| in Heard<A>::compare$',
# The test-only `Proof::LEN`.
'proof\.rs:\d+:\d+: replace \+ with \*$',
# The stale-view shortcut only saves work: without it, or at equal revisions,
# where both views hold the same claims, the full merge changes nothing.
'replace (> with (==|>=) in View<A>::outdates|View<A>::outdates -> bool with false)$',
# Dropping the cached context more often makes the next pass full, which
# reaches the same result.
':\d+:(25|38): replace && with \|\| in Router<N, A>::recompute$',
# A claim an answering certifier certifies is always in the merged view.
':\d+:35: replace && with \|\| in Router<N, A>::meets$',
# A new stream's first batch either flips its certifier's answer, which marks
# every claim, or reaches the reader only through the merge, which marks its own
# changes; marking all only reorders independent changes in a forwarded batch.
':\d+:32: replace != with == in Router<N, A>::batch$',
# At an equal metric both arms assign the same distance.
'replace < with <= in Router<N, A>::record$',
# The full presentation of a stream the interest misses is empty, and a full
# presentation equals the incremental one.
'replace match guard (!self\.meets\(id, ctx, watch\)|!dirty\.all) with false in Router<N, A>::advertise$',
# `advertised` is read only for qualified identities.
'replace & with (\^|\|) in Router<N, A>::advertise$',
# For its own identity, the node's own view answers every claim, `current`
# never holds it, and nothing serves it.
':\d+:31: replace && with \|\| in Router<N, A>::shown_now$',
# The nearest home with a slot at or above a presented claim always serves
# that claim's instance and epoch, so a looser match picks the same home.
'node\.rs:\d+:57: replace && with \|\| in awaits$',
# resume.rs: the sweep of handed-out groups only bounds memory, since
# `let_go` drops dead entries itself before anything reads `out`.
'replace (== with !=|> with >=) in Reader::hand_out$',
# Polling a covered or finished copy hands out nothing, so polling it anyway
# only costs work.
':\d+:36: replace \|\| with && in Reader::poll_lowest$',
# A run starting at `until` clips to an empty range, which has no gaps.
':\d+:36: replace < with <= in Reader::exhausted$',
# `learn_end` just polled the same copy and returned on an `Ok` end.
'delete match arm Poll::Ready\(Ok\(fin\)\) in Reader::poll_finished$',
# The one extra copy searched is the route's own, which never has an end.
':\d+:32: replace - with / in Reader::backfill$',
# At the same generation the serving copy is the one that just failed the
# group, so it never answers at that index.
'replace match guard !same with true in Recover::poll_serving$',
# A replacement starting at frame 0 already holds the head.
'replace > with >= in lend_head$',
# Race-only, not equivalent: each call follows a read of the route in the same
# poll, so these differ only when another thread writes the route between that
# read and this check, a window no deterministic test can stage. There the
# original retries on the new copy and the arm prevents a lost wake.
'replace \|\| with && in Fetching::poll_moved$',
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
':\d+:6: delete match arm Some\(Some\(Err\(err\)\)\) in <impl kio::Task for Fetching>::poll$',
# The reader list's sweep only bounds memory: a dead entry fails every
# `Weak::upgrade`, and `subscribe` sweeps the list again.
'replace > with >= in Producer::serve_from$',
'replace == with != in Consumer::subscribe$',
# Dead: every write of `until` also writes `since`, so the superseded check
# never runs. `ResumeReader.tla` models it, so it goes with the model.
'replace >= with < in Copy::retains$',
# A run starting at the end clips to an empty range, on which nothing lands
# and nothing is missing.
'replace < with <= in Copy::retains$',
'replace < with <= in Retain<.*>::holds_window$',
# Walking again before anything the read parked on fires returns the same
# `Pending`, so the idle shortcut only saves work.
'replace Subscriber::parked -> bool with false$',
# path.rs: `step` only counts visits for the tests' work budgets, which
# read the count as an upper bound.
'path\.rs:\d+:\d+: replace Router<N, A>::step with \(\)$',
# A path-rule identity nothing holds has no route left, and the update that
# removed its last one already dropped its views, so `collect` changes nothing.
'path\.rs:\d+:\d+: replace Router<N, A>::collect with \(\)$',
# Tied candidates share a floor, so by the identity rule they present the
# same content at the prefix.
'path\.rs:\d+:\d+: replace > with >= in Router<N, A>::know$',
# A route's change updates every view it covers, so a batch's only newer
# candidate is its own route and peer: these only turn off the incremental
# presentation, which the test build asserts equals the full one.
'path\.rs:\d+:(51|74): replace == with != in Router<N, A>::know$',
# An ended stream's final batch always has a floor and withdraws claims: an
# offer is sent only while a claim meets it.
'path\.rs:\d+:\d+: (replace \|\| with &&|delete !) in Router<N, A>::end_sent$',
# The full diff it skips holds the same changes in another order, and a source
# switch has already marked every claim that differs.
'path\.rs:\d+:80: replace != with == in Router<N, A>::present$',
# NOT equivalent: Rust lets an ended stream's child wait go where
# RouteHandoff.tla keeps it. quest/m2/cluster-routes/ended-child-wait.md
# aligns the two and replaces this with a test.
'path\.rs:\d+:55: replace && with \|\| in Router<N, A>::advertise$',
]
8 changes: 6 additions & 2 deletions .github/actions/rust-cache/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ inputs:
secret-access-key:
description: R2 secret access key matching `access-key-id`.
default: ""
write:
description: Whether a run on `main` publishes its compilations. A job whose builds nothing reuses, such as one per mutant, passes false and reads with the repository's key.
default: "true"

runs:
using: composite
Expand All @@ -21,7 +24,8 @@ runs:
echo "::error::Persistent self-hosted caches require Queue on main's merge queue." >&2
exit 1

# Hosted runs on `main` write; everything else reads. The merge queue
# Hosted runs on `main` write, unless the job passes `write: false`;
# everything else reads. The merge queue
# uses only its local store and receives no remote credentials.
#
# The write key is the boundary. It lives in the `cache-write`
Expand All @@ -40,7 +44,7 @@ runs:
env:
KEY_ID: ${{ inputs.access-key-id }}
SECRET: ${{ inputs.secret-access-key }}
WRITE: ${{ github.ref == 'refs/heads/main' }}
WRITE: ${{ github.ref == 'refs/heads/main' && inputs.write == 'true' }}
run: |
if [ -z "$KEY_ID" ] && [ -z "$SECRET" ]; then
if [ "$WRITE" = "true" ]; then
Expand Down
54 changes: 54 additions & 0 deletions .github/workflows/mutants.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: Mutants

# cargo-mutants edits the route and resume code one small change at a time
# (`.cargo/mutants.toml` names the files and the tests), and a job fails on any
# edit its tests miss. Each mutant costs a rebuild of moq-net's tests, too many
# for a merge gate, so Nightly runs it; sh/rs/mutants.sh says what each shard takes.

permissions:
contents: read

on:
workflow_call:

jobs:
mutants:
name: Mutants (${{ matrix.shard }}/6)
runs-on: ubuntu-24.04-arm
# A day of heavy route churn adds about an hour of changed mutants to the rotation's half hour.
timeout-minutes: 150
permissions:
contents: read
# The changed mutants are those since the previous scheduled Nightly's commit.
actions: read
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3, 4, 5]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/nix
# Every mutant's build is a one-off, so the job reads the shared cache
# with the repository's key and publishes nothing.
- uses: ./.github/actions/rust-cache
with:
access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }}
secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }}
write: false
# The checkout is thrown away, so mutating it in place keeps the build
# incremental and cached instead of copying the tree per mutant.
- name: Mutate
run: nix develop --command just rs mutants nightly ${{ matrix.shard }} 6
env:
GH_TOKEN: ${{ github.token }}
- name: Keep the outcomes
if: ${{ !cancelled() }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: mutants-${{ matrix.shard }}
path: |
target/mutants/*/mutants.out
target/mutants/missed.txt
retention-days: 14
11 changes: 11 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ name: Nightly
# exception: deterministic and seconds long, so `just ci` gates every diff that
# can move them, and audio-quality.yml reruns them here.
#
# `mutants` edits the route and resume code one change at a time and fails on an
# edit no test notices; each edit rebuilds moq-net's tests, about 27 runner-hours
# for the full set, so it rotates (see sh/rs/mutants.sh).
#
# `tla` model-checks every TLA+ config: TLC explores every ordering a model
# allows, which takes minutes per config, and a model only changes when the code
# it covers does, which review checks (see REVIEW.md). It also fails when an
Expand Down Expand Up @@ -69,6 +73,13 @@ jobs:
runner-vm:
uses: ./.github/workflows/runner-vm.yml

mutants:
uses: ./.github/workflows/mutants.yml
secrets: inherit
permissions:
contents: read
actions: read

# The path-filtered gates main PRs skip; release PRs still run them.
x11:
uses: ./.github/workflows/x11.yml
Expand Down
2 changes: 2 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,8 @@
# The workspace-hack that gives every build one feature set per dependency.
cargo-hakari
cargo-nextest
# Mutation testing of the route and resume code (`just rs mutants`).
cargo-mutants
cargo-codspeed
# Sets versions for a release cut; `sh/version.py` needs 0.4.1 or newer.
release-plz
Expand Down
2 changes: 1 addition & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ m4 default, so changes are proven before they ship: the
[QUIC Interop Runner](/quest/m1/quic-interop.md), sim rule checks,
[relay parser fuzz targets](/quest/m1/fuzz-relay-parsers.md), and from m4
the relay on the sim (landed),
[mutation testing](/quest/m1/mutants.md), and peer-reachable unsafe. OSS-Fuzz stays in m4 (maintainer, same round): it mostly waits on
mutation testing (#8712), and peer-reachable unsafe. OSS-Fuzz stays in m4 (maintainer, same round): it mostly waits on
google/oss-fuzz.
Left in m4 without a pick: Kani, Miri seeds and TSan, the JS corpus, and
bindings interop.
Expand Down
43 changes: 0 additions & 43 deletions quest/m1/mutants.md

This file was deleted.

50 changes: 50 additions & 0 deletions quest/m2/cluster-routes/ended-child-wait.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# [S] An ended stream still waits on the child it taught

## Goal

`path::Router` holds a lost route until every neighbour that learned it from
this node has answered, as `RouteHandoff.tla`'s `Pending` does, including a
neighbour whose offer back arrives after this node ended the route on its
stream.

## Plan

Decided 2026-11-14 (maintainer, /quest-complete): ✅ keep in m2 (rejected: drop it, move to m3).

Found 2026-11-14 by the mutation run ([moq-dev/moq#8712](https://github.com/moq-dev/moq/pull/8712)),
which excludes the one mutant it exposes (`&&` to `||` in
`path::Router::advertise`'s cleanup of ended streams) until this lands.

When a stream ends, `advertise` removes its sent entry unless the peer's route
is a pending child right then. The model keeps every ended entry's source
(`NoSent(t.src)`), so its `Pending` also counts a child offer that the
neighbour sent before it heard the END. With these steps, Rust sends peer 1
`Route { reach: None }` at the last step, while the model keeps waiting on
peer 2:

```
router 99; peers 1, 2, 3 on Link::default
route(1, 7, "", through(1)); batch(1, 7, "", [a/x = 1, b/x = 1], Some(1))
peer_interest 1: "", 2: "b", 3: ""
route(3, 7, "", along[(7,0), (6,1), (4,1), (1,1), (99,3), (3,1)]) // a child of the "" stream
route(2, 7, "b", along[(7,0), (99,9), (5,1), (2,1)]) // not a child
batch(1, 7, "", [b/x = None], Some(2)) // the "b" stream to 2 ends
dead(1, closed([], false)) // "" holds under 1
route(2, 7, "b", along[(7,0), (6,1), (4,1), (1,1), (99,2), (2,1)]) // 2's offer back, sent before the END reached it
route(3, 7, "", None) // peer 3 answers
```

(`through`, `along`, `closed` as in `route/tests/scope.rs`.) No loop or hold
cycle follows, since peer 2's stale path names this node, but the code departs
from the module doc's rule and from the checked model.

Recommendation, for the maintainer's review (taken by the agent under
/quest-afk): **align Rust with the model.** Keep an ended entry's source until
its stream is replaced or its session closes, make the steps above the test
that the route still holds, and drop the exclusion from `.cargo/mutants.toml`.
The model needs no change, so no new TLC run is required beyond the usual
`/verify-route`. Rejected: narrowing the model's `Pending` to the children
current when the stream ended, which weakens the checked rule to fit the code
and needs a new TLC run.

Public API: none. Wire: none.
37 changes: 37 additions & 0 deletions quest/m4/mutants-modelled.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [L] No mutation of the other modelled code survives its tests

## Goal

The Nightly mutation run that guards the route and resume code (#8712) also
covers the other code a TLA+ model checks: `rs/moq-net/src/reconnect.rs`,
`lite/publisher.rs`, `lite/subscriber.rs`, and `tail.rs`. The tests fail on
every edit to them that changes behaviour.

## Plan

Decided 2026-11-14 (maintainer, /quest-complete): ✅ keep in m4 (rejected: drop it).

Split from the route and resume run (#8712, taken by the agent under
/quest-afk, for the maintainer's review), which measured what these files
would add. Its quest said they join once its cost was known, and they do not
fit as they are:

- They hold 1681 mutants against the route and resume code's 1300, so the
rotation (`sh/rs/mutants.sh`) would more than double each night's runner
time, or need twice the rotation's nights to cover everything once.
- Their survivors from a first full run have to be killed or excluded before
Nightly can fail on them, as #8712 did for route and resume.
- Their tests are the lite session and reconnect suites, not the route ones,
so the config needs a second test selection or a second `.cargo/mutants.toml`
run.

Decide which nights the run gets (more shards, a longer rotation, or a
weekly job) once a first full sweep says how long their tests take per
mutant; then sweep, kill or exclude every survivor, and add them to
`examine_globs`.

Public API: none. Wire: none.

## Related

- [Test coverage audit](/quest/m3/audit-test-coverage.md) - the line-coverage reports this complements
Loading