Repository navigation
tri mutate spec: an opt-in non-zero exit when mutants survive #7303
Description
Problem
tri mutate spec (cli/tri/src/mutate.rs, mutate_spec) exits 0 when mutants SURVIVE. It fails only when the unmutated spec is red or when some mutants could not be run. A script or a CI step cannot gate on survivors without grepping the text SURVIVED.
Measured 2026-10-07 on the Railway lab, file specs/tri/mutate/lab.t27 (#7050 slice 1): the tool printed 48 of 52 killed ... 4 survived and exited 0.
Expected
An opt-in exit code for survivors, so the default behaviour of existing callers does not change. For example, --fail-on-survived makes the tool exit non-zero when at least one mutant survived.
The tool cannot tell an equivalent survivor from a test gap. A caller that opts in needs a way to list the survivors it has accepted, as file:line kind keys in a file it passes in.
Acceptance
tri mutate spec --file specs/tri/mutate/lab.t27 --fail-on-survivedexits non-zero and names the 4 survivors.- The same command, given the 4 survivors as accepted, exits 0.
- Without the flag, the exit code is unchanged on the same file (0).
Notes
- This is a Rust change in cli/tri, so it needs the
owner-approved-foreignlabel and an entry intools/policy/foreign-exceptions.txt. Under the only-t27 rule, the decision ("exit non-zero when survived > accepted") belongs in a t27 spec first, in the shape ofspecs/tri/mutate/lab.t27'slab_exit. - Related: tri mutate spec --lab: run the mutation pass on the Railway lab from a Mac worktree #7050 (the
--labflag; its exit codes are inspecs/tri/mutate/lab.t27).
Activity
gHashTag commentedon Oct 7, 2026
Decision on the exit code, made in the spec before any Rust (commit 39e711d4d on branch
claude/mutate-lab-7050-land, not yet a PR):tri mutate speckeeps 1 for "no verdict" (red unmutated spec, mutants not run) and that still wins first.not_killed = survived + hung; a hung mutant fails the gate unless the accepted file names itsfile:line [kind]. Unviable mutants are counted nowhere, as in cargo-mutants. (cargo-mutants gives timeouts their own code 3; one code for "not killed" keeps the accepted file the single place a caller explains any of them.)specs/tri/mutate/lab.t27follows. Its "no result" was 2 and would have collided. The tool's 0..2 now pass through--labunchanged; the lab's own codes are NO_RESULT 3, ORPHANS 4, STILL_RUNNING 5. Any tool code it does not define is a failed tool, never a verdict.Measured this tick:
survivors.t2710 tests pass, 0 vacuous; labtri mutate spec20 of 20 killed; 40 of 40 hand mutants killed by assertions.lab.t2719 tests, lab run 50 of 54 killed (the same 4 equivalent survivors), 22 of 22 hand mutants on the changed code killed.Slice 2 (the Rust:
--fail-on-survived,--accepted FILE, and a test asserting each scenario row by row) is unchanged in scope; the acceptance's first row now expects exit 2.gHashTag commentedon Oct 7, 2026
Slice 2 is done and committed locally as
0d915d2e4onclaude/mutate-gate-7303-s2. It is stacked on #7050 slice 1 and is not pushed yet: it lands as one PR after #7322 merges, so only one PR is open at a time while the Actions queue is long.The rule came first, in t27, as
specs/tri/mutate/survivors.t27. The Rust copies its five functions. The testthe_gate_agrees_with_every_assert_row_of_its_specreads the spec when it runs and evaluates all 38 of its assert rows against that copy.Measured on the Railway lab, file
specs/tri/mutate/lab.t27(now 50 of 54 killed, 4 survived, 0 hung, after its revision):--fail-on-survived(row 1)--acceptednaming the 4 (row 2)Gate passed.--acceptednaming 2 of them--fail-on-survivedonsurvivors.t27itselfcargo test --release -p tri mutate::: 38 passed.accepted_killedfails the agreement test, and so does changing one spec row.fpga::tests::test_smoke_gate_json_*failures in the lab's fullcargo test -p trialso happen on the base commit without this change. The lab has noyosys, andcli-tri.ymlinstalls it for that reason.The exit codes follow cargo-mutants rather than Stryker: 2 means the gate failed, and 1 stays the tool's own error exit.
tri mutate spec: --lab, the survivor gate, and zig's -j per mutant (C…