Repository navigation
t27c test-report waits on each test with no time limit: a looping test hangs it and seal --save forever #7255
Description
Problem
t27c test-report runs each test of a spec as its own process (spec_tests <index>) and waits for it with no time limit. A test that never returns hangs test-report, and with it seal --save, which runs the same report for every spec it seals (#5577), forever.
Measured on 2026-10-07:
- On the owner's Mac, two
spec_tests 1processes had been running for 1 day 11 h 29 min at about 89% CPU each, fromt27c test-report specs/fpga/testbench/clock_domain_tb.t27and.../gf16_accel_tb.t27started by a t27b reference run (t27b-reference-87617). That run's parent was gone (botht27chad PPID 1). Nothing would ever have stopped them; they were killed by hand at 02:15Z. - With master's
t27c(Mac release build), both specs still hang:test-reporton each was alive at 60 s with onespec_testschild, and had to be killed together with that child. - A probe spec whose test loops (
while (i < n) { i = i * 1; }) shows the rest of the failure: when an outer runner kills the hungt27c, itsspec_testschild keeps running with PPID 1, and the work dirt27c-test-report-<stem>-<pid>is left behind. - On the Railway lab at 02:16Z: 543 zombies of 551 processes (pids.max 1000), 43 of them
[spec_tests] <defunct>reparented to PID 1, and 114 leaked/tmp/t27c-test-report-*dirs. The zombies are t27c-lab: PID 1 never reaps orphans -- 417 zombies of pids.max 1000 #7090; the orphaned test binaries are what reaches it from this side.
Why it matters
Every bulk caller (lab-exec, the t27b reference, seal --save over many specs, the Queen's criteria runs) needs its own outer timeout, and an outer kill leaves a spinning orphan behind. A seal of either testbench above cannot be refreshed on master at all: the command never ends.
What others do
Bazel stops a test at its size's timeout (60 s for small) and reports TIMEOUT as a failure; cargo-nextest has slow-timeout with terminate-after and counts TIMEOUT as failed; go test stops the binary at -timeout (10 min by default); pytest-timeout fails the test. Zig's own runner has no limit, which is the gap here.
Fix
test_reportruns eachspec_testscall with a limit (60 s per test by default,T27C_TEST_TIMEOUTseconds to change it). A test that outruns it is killed, reaped, and counted as failed; the per-test lines saytimed out after N s.- The
FAIL <name>line and the totals stay as they are, so the readers of the report (cli/t27b/src/blockers.rs,contrib/railway/t27b-lab/lab.py) read the same text. - Out of scope: a
t27ckilled from outside still cannot stop its child (that needs a process-group or parent-death signal); with the limit in place an outer kill should no longer be needed for a looping test.
Criteria
T27C_TEST_TIMEOUT=2 t27c test-report <looping probe>exits 0 within 30 s, printsFAIL spins_foreverandtimed out after 2 s, and leaves nospec_testsprocess and no work dir.t27c test-report specs/fpga/testbench/clock_domain_tb.t27finishes on the lab with the default limit.- Every other spec's
test-reportoutput is byte-identical to master's (lab A/B over the tracked.t27files).
Activity
gHashTag commentedon Oct 7, 2026
Local commit 47257d418 on
claude/test-timeout-7255, stacked on the #7243 commit (2271df9). It opens as a PR after #7242 and #7243 land (one PR in the queue at a time).Change. Each test, and the listing that names the tests, runs under a limit: 60 s, which is Bazel's limit for a
smalltest, orT27C_TEST_TIMEOUTseconds. A test that outruns the limit is killed and reaped, then counted as failed, and its line says(timed out after N s, not counted). The spec's other tests still run.Measured on the Railway lab (403b27f tree,
test-report --verboseon each of the 1484 tracked.t27files, 4 at a time; master's binary ran under an outer limit of 150 s, this change's under 900 s):specs/fpga/testbench/{apb_bridge,axi4,clock_domain,gf16_accel}_tbspecs/ml/layers/residual_connection(failed, not counted)(timed out after 60 s, not counted)/tmp/t27c-test-report-<stem>-<pid>/prefix (#7243) is cutA fifth hang, and why the lab hid it.
residual_connectiondoes not end because itsforward_basic_casewalks a slice built fromundefined:default_input()is declared nowhere (default_input has no author: the converter generated 169 specs' worth of hollow tests #5166)./data/ref/reaper.shkills anyt27c-test-report-*/spec_testsolder than 60 s, every 20 s. On any host without that reaper, master hangs on this spec for good.Tests.
test_report:the_timeout_is_a_whole_number_of_seconds_above_zero,a_timed_out_test_says_so_in_its_lineanda_looping_test_is_stopped_and_counted_as_failed.bootstrap/tests/test_report_timeout.rsruns the CLI withT27C_TEST_TIMEOUT=2.test_report::unit tests and 1/1 integration test pass (2.3 s).Negative controls.
Trap found on the way. procps-ng 4.0.2 reads
kill -9 -<pgid>as no target: it exits 0 and kills nothing. The test useskill -9 -- -<pgid>.Still out of scope:
t27ckilled from outside still leaves its running test behind (a process group or parent-death signal would fix that);gHashTag commentedon Oct 7, 2026
Lab evidence from coverage lane 1 (#6063), 2026-10-07: on the Railway t27b lab, the lab's reference timeout kills
t27c test-reportbut not thespec_testschild it started, so each hung testbench is reparented to PID 1 and keeps running. I found 99 orphanedspec_testsin state R, up to 11 h old (apb_bridge_tb,axi4_tb,clock_domain_tb,gf16_accel_tb,residual_connection). With the zombies they leave, the lab sat at 962 of its 1000-pid cap, and corpus runs failed withfork: retry: Resource temporarily unavailableand spuriouslab_errorresults. Killing the orphans older than 20 min brought it down to 578.A time limit inside
test-reportthat kills its own child (this issue) fixes the cause. Until then, the lab could start the reference in its own process group and kill the group on timeout.gHashTag commentedon Oct 7, 2026
Root cause of the lab's pid exhaustion, measured 2026-10-07 04:55Z on t27b-lab:
ps -p 1reportspython3 -u /opt/t27b-lab/lab.py. Inside the container, lab.py is PID 1.spec_tests. The orphan is reparented to PID 1. lab.py never callswaiton children it did not spawn, so the orphan stays a zombie.[spec_tests] <defunct>, ppid 1), some up to 13 hours old.pids.currentwas 856 againstpids.max1000.kill -9has no effect on a zombie. Only reaping by PID 1, or restarting the container, frees those pids.Each concurrent corpus or
cargo testthen works with the roughly 140 pids left. That is what produces the intermittentzig not on PATH,test binary would not list its tests,can't start new thread, andResource temporarily unavailablefailures seen in lane runs.Either fix closes the leak:
tini, ordocker run --init). This needs no code change.SIGCHLDhandler that loopsos.waitpid(-1, os.WNOHANG).A third fix also stops the orphans from being created: kill the whole process group on timeout (
start_new_session=Trueplusos.killpg), not only t27c.Lane 1 did not restart the container, because other lanes' runs are live on it.
t27c test-report: a test that outruns its limit is stopped and counte…
self-host: t27core accepts compound assignment, byte-identical to gen…
t27core: numeric literal forms and pub var, byte-identical to gen-c (C…