Repository navigation
t27c-lab: PID 1 never reaps orphans -- 417 zombies of pids.max 1000 #7090
Description
Problem
The t27c lab container on Railway (infra/t27c-lab, PID 1 = python3 /app/lab.py) never reaps orphaned processes. Measured over railway ssh at 2026-10-06T20:35Z:
zombies=417 # ps -eo stat= | grep -c '^Z'
procs=424
pids.max 1000 # /sys/fs/cgroup/pids.max
pids.current 424
417 1 # every zombie's PPID is 1
115 .. 191075 # zombie ages in seconds: the oldest is ~53 h old
A zombie holds a PID until its parent waits on it, and PID 1 here never waits on a process it did not start. The cgroup allows 1000 PIDs, so the lab stops spawning when zombies plus live processes reach 1000. That would fail every gate and every hand run with Resource temporarily unavailable. On 2026-10-06 a 16-job mutation run already hit the PID limit once (one spawn failed); zombies make that limit lower for everyone.
Cause
Two things combine:
lab.pysh()kills only its direct child on a timeout (p.kill()). The grandchildren (cargo -> rustc, t27c -> zig, a test binary) are reparented to PID 1.- The Dockerfile runs
CMD ["python3", "/app/lab.py"]with no init process. When a reparented process exits, nothing reaps it. Processes started overrailway sshthat outlive their shell take the same path; for example,tri mutate speckilling a hungzig testleft 2 zombies at 20:34Z.
Fix
- Run an init as PID 1:
apt-get install -y tiniandENTRYPOINT ["/usr/bin/tini", "--"]ininfra/t27c-lab/Dockerfile(Docker's--initis not available on Railway). tini reaps every orphan. - In
sh(), start the child in its own session (start_new_session=True) and kill the whole group on a timeout (os.killpg(p.pid, SIGKILL)), so a timed-out gate leaves no running grandchild behind. - Do not reap with a
SIGCHLDhandler that callswaitpid(-1)inside lab.py: that would also take the exit status thatsubprocess.Popen.wait()needs.
Acceptance criteria
ps -eo stat= | grep -c '^Z'on the lab is 0, or stays below 5 one hour after a redeploy and a mutation run with hangs.ps -o args= -p 1on the lab printstini.- A gate killed at its timeout leaves no process whose
/proc/PID/cwdis under its run directory.
Note
A redeploy clears today's zombies, but it also stops every lane's running work on the lab. It belongs in the deploy of this fix, not done on its own.
Found by the queen-actors loop (#6971) while checking for orphans after a lab run (trap T9). Related: lab disk at 99% (#7062).
Activity
gHashTag commentedon Oct 7, 2026
Measured again over
railway sshat 2026-10-07T02:16Z: 543 zombies of 551 processes (pids.max 1000), up from 417 of 424 at 2026-10-06T20:35Z; 43 of them are[spec_tests] <defunct>with PPID 1.One source of those is filed as #7255:
t27c test-reportwaited on each test binary with no limit, so a looping test (specs/fpga/testbench/clock_domain_tb.t27,gf16_accel_tb.t27on master) hung it until an outer timeout killedt27cand left the test binary to PID 1. The fix there stops and reaps the test itself; this issue's init is still what reaps anything reparented to PID 1.Lab runs from this lane now go through a small subreaper wrapper (
prctl(PR_SET_CHILD_SUBREAPER), kill the session on the limit, thenwaitpid(-1)until none is left), so they add no zombies while this stays open.t27c seal: the test record names the spec, not zig's temp dir or a mi…
gHashTag commentedon Oct 7, 2026
Measured again over
railway sshat 2026-10-07T05:52:31Z:That is 171 more than at 02:16Z, about 47 an hour. At that rate the remaining 273 PIDs run out near 11:40Z today, and after that nothing on the lab can spawn.
It already costs runs. In a full
cargo test --release -p t27c --no-fail-faston master e1ede4a,zig_primitive_bindingsnow fails 1 or 2 of its 6 tests every time withthe following command failed with SystemResources(zig spawning its test binary). The same binary passed at 04:58Z, and a master-based tree from before fails the same way now, so this is the PID budget, not code.A restart of the lab service clears all of them at once. The lasting fix is still the init process in the Dockerfile above. #7311 (this issue's #7255 half) stops
test-reportfrom leaving looping test binaries behind.t27b: tuple locals, @sqrt, out-of-range compares, unreached &repeat a…
self-host: t27core accepts compound assignment, byte-identical to gen…
t27core: numeric literal forms and pub var, byte-identical to gen-c (C…
t27c lab: reap idle /data/<name>-target dirs when the disk runs low (#…
t27c lab: reap idle /data/<name>-target dirs when the disk runs low (#…