Repository navigation
tri mutate spec --lab: run the mutation pass on the Railway lab from a Mac worktree #7050
Description
Problem
Every tick of the actors loop (epic #6971) runs tri mutate spec on the Railway lab by hand, because t27c and tri builds run there and never on the Mac. Each tick repeats the same five steps:
- base64-encode the spec and a shell script;
- send them over
railway ssh, with 3 retries and a grep that drops the CLI's banner lines; - start the run with
nohupso a dropped ssh does not kill it; - read the output on a later ssh call;
- remove the files and check that no
zig testprocess is left (trap T9 in the queen-actors-loop skill: a timed-out run once left orphans spinning at 100% CPU).
Tick 7 spent 6 of its ssh calls on this plumbing. One remote pkill with an unbracketed pattern also once killed its own shell (trap T10).
Evidence
scripts/tri_loop/t27b.pyis the only tool that already speaksrailway ssh, and only for t27b.tri --helpon the Mac build has no lab command (tri lab-execis unrecognized there).
Proposal
Add tri mutate spec --lab [--fn NAME]. It ships the local spec to a fresh temp dir on the lab, runs the lab's own tri mutate spec with --jobs 8, waits, prints the report, removes the temp dir and reports any zig test process left behind as an error.
- The lab ids come from the environment, never from the source.
- No secret is printed.
- Under
specs/policy/own_language.t27, the command is specified first as a.t27contract: steps, exit codes, and the cleanup rule. Any Rust carries theowner-approved-foreignlabel.
Acceptance
- From a Mac worktree,
tri mutate spec --lab --file specs/queen/actors.t27 --fn start_answerprints4 of 4 killed. The lab command it runs by hand prints the same. - After the command,
ls /data | grep -c tri-mutate-on the lab is 0, andps aux | grep -c '[z]ig test'is 0. - A run cut by a dropped ssh is still read to the end by a second call, because the run sits under
nohupand its output file has a known name.
Part of epic #6971. Depends on #7043 (tri mutate spec) merging.
Boundary
specs/policy/own_language.t27specs/queen/actors.t27scripts/tri_loop/t27b.py
User Scenarios
-
Given a Mac worktree with a spec file at
specs/queen/actors.t27and a Railway lab configured via environment, When the user runstri mutate spec --lab --file specs/queen/actors.t27 --fn start_answer, Then the command ships the spec to a fresh temp dir on the lab, runs the lab'stri mutate spec --jobs 8, prints the mutation report4 of 4 killed, removes the temp dir, and reports any leftoverzig testprocesses as an error. -
Given an SSH connection to the lab drops during a mutation run, When the user re-runs the command or reconnects, Then the nohup'd run continues on the lab and its output is read to completion on the next SSH call because the output file has a known name.
Requirements
- FR-001: The command
tri mutate spec --labMUST accept--fileand optional--fnarguments and MUST ship the local spec to a fresh temp directory on the Railway lab. - FR-002: The lab command MUST be invoked as
tri mutate spec --jobs 8with the shipped spec and function name. - FR-003: Lab identifiers (host, user, paths) MUST come from environment variables only, never from source code.
- FR-004: No secret material MUST be printed to stdout or stderr at any point.
- FR-005: After the remote run completes, the temp directory on the lab MUST be removed.
- FR-006: Any
zig testprocess left running on the lab after the mutation run MUST be reported as an error. - FR-007: The remote mutation run MUST be started under
nohupso that a dropped SSH connection does not kill it, and its output MUST be written to a known file for later retrieval. - FR-008: The behavior of
tri mutate spec --labMUST be specified first inspecs/policy/own_language.t27as a.t27contract covering steps, exit codes, and the cleanup rule; any Rust implementation carries theowner-approved-foreignlabel.
Success Criteria
tri mutate spec --lab --file specs/queen/actors.t27 --fn start_answerprints4 of 4 killedand exits withexit 0.- After the command completes,
ssh lab "ls /data | grep -c tri-mutate-"outputs0and exits withexit 0. - After the command completes,
ssh lab "ps aux | grep -c '[z]ig test'"outputs0and exits withexit 0. - Simulating a dropped SSH during the run, a second SSH call retrieves the full mutation report from the known output file and the command exits with
exit 0. - The spec
specs/policy/own_language.t27contains the contract for the--labmutation pass with steps, exit codes, and cleanup rule.
Activity
gHashTag commentedon Oct 7, 2026
Decomposition, and the first slice.
Slice 1: the contract in t27. Committed locally as
e8702ee0don branchclaude/mutate-lab-7050. The PR opens after the PRs queued ahead of it merge, one at a time. The new specspecs/tri/mutate/lab.t27(moduleTriMutateLab) states what--labdecides:run_state(dir_exists, exit_written, runner_alive)returns NONE, RUNNING, DONE or LOST. The run is a nohup'd job in its own directory, so a dropped ssh is read back from the pid file, the exit file and the output file. This is the issue's third acceptance line.launch_allowed(state)holds only for NONE. A launch over a running run, an unread result or an uncleaned lost run is refused.may_remove(state, orphans)holds only for DONE or LOST with no orphan left. Orphans are killed and recounted first (trap T9).lab_exit(state, tool_rc, orphans): 0 OK, 1 tool failed, 2 no result, 3 orphans, 4 still running.ssh_should_retry(attempts, transient, is_write): reads retry, 3 tries in all; writes never retry.poll_wait_seconds: 5, 10, 20, 40, then 60.zig_j(nproc, jobs) = max(1, nproc / jobs).lab_jobs(requested, pids_max, pids_used, per_job)cuts the request to the pids left after a reserve of 104. 0 means do not start.The spec has 18 tests and 4 invariants.
zig testpasses 18/18 and test-report shows 0 vacuous. On the lab,tri mutate speckilled 48 of 52 mutants; the 4 survivors are equivalent, each reason written in the commit body. All 38 hand mutants were killed.Measurement behind
zig_jand the reserve. On the lab, 2026-10-07: 48 cores, pids.max 1000, pids.current 713 before the probe. 8 parallelzig testcompiles ofspecs/queen/actors.t27, two runs at each setting, 8/8 passing every time:-j6-j1-j6(48 / 8) keeps the default's wall time with about a fifth of the pids. With around 700 of the lab's 1000 pids held by zombies (#7090), that is the difference between a run that starts and one that hits a spawn failure (trap T11).Slice 2: the Rust. The
--labflag incli/tri/src/mutate.rs, with lab ids taken from the environment, upload, launch, poll and cleanup. It includes a test that asserts each scenario oflab.t27row by row; that test is where the two meet. It needs theowner-approved-foreignlabel.Slice 3: pass
-j zig_j(nproc, jobs)to each mutant'szig test. This applies to local runs too, not only--lab.Separate:
tri mutate specexits 0 with survivors, filed as #7303.gHashTag commentedon Oct 7, 2026
Slice 3 is up as #7366: the
--labflag, itsrailway sshplumbing, and zig's-jfor every mutant. It is stacked with slices 1 and 2 and with #7303's two slices, and has auto-merge on.Measured on the lab (2026-10-07, after its tini redeploy):
tri mutate spec --file specs/queen/actors.t27 --fn start_answer --labprints4 of 4 killedand exits 0. That was with--lab-wait 0(exit 5, run left going) followed by a read-back.T27C_LAB_LOCAL=1on the lab itself, not from a Mac. tri is not built on the Mac (builds run on the lab), so the Mac-side half is unproven until a Mac has a tri built from master.railway sshpath was covered by the scripted tests. It was not: the only non-local test pointed at a missingrailwayand checked the error.a_lab_run_goes_through_railway_ssh(test(tri): a lab run through a stand-in railway ssh (Closes #7390) #7392, closes test: tri mutate spec --lab's railway ssh call has no test #7390; tri mutate spec: --lab, the survivor gate, and zig's -j per mutant (Closes #7303) #7366 merged before it reached that branch) runs a whole lab run through a stand-inrailwayand checks every call readsssh -p p1 -e production -s t27c-lab; droppingsshor-pfrom the call fails it. No run has gone through the realrailwayCLI from a Mac.zig testprocesses and 0 zombies./tmp, not/data:/datahad 1.4G free (97% used), so the check isls -d /tmp/tri-mutate-*.--lab-waitkeeps going, and the same command reads it to the end. The directory name is a hash of the request.Weak point found and fixed in this slice: the pid model was wrong.
tri mutate specnow passeszig_jas-j(or--zig-threads N), and the spec gainsjob_pids(-j) = -j + 3.PID_RESERVEgoes from 104 to 408, which is one unbounded batch:8 * (48 + 3).--labrun of lab.t27 peaked at 60 added pids.This issue stays open for the Mac-side run.
tri mutate spec: --lab, the survivor gate, and zig's -j per mutant (C…
test(tri): a lab run through a stand-in railway ssh (Refs #7050)
test(tri): a lab run through a stand-in railway ssh (Closes #7390, Refs
gHashTag commentedon Oct 7, 2026
Reopened 2026-10-07: the bulk close of issues without a
## Boundarysection was reverted by the owner. Nothing to do here; this comment replaces the earlier close notice.