Repository navigation
test: fourth side-effect audit: orphan reaps, worktree scans, budget - #11010
Conversation
|
Intent: Close what a fourth five-run side-effect audit found: processes, Not a goal: changing any product behaviour, or re-fixing the three findings |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the Windows CI job actually exercises the new [DESIGN-REVIEWED] 9392c92 |
|
Why One added line trips the Cross-Platform Portability scan: os.killpg(process.pid, signal.SIGKILL)in It is the POSIX arm of a two-arm split — the Windows arm right below it shells to Every other caller in So the label, which is the escape hatch the gate's own message names, rather than |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've verified the sole candidate against the actual code. The normal-completion path ( Falsifying candidate 1: For the No grounded Step-2 finding survives on the production hunks (the No findings. [OPUS-REVIEWED] 9392c92 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One rider to eye: the real-kiro-cli precedence guard went opt-in ( Not justified as shipped
What this change shipsInventory (10 items, capped — the change has more) — 9 justifiedIntent: FIX — stop the test suite from leaving orphaned processes, leaked temp dirs, leaked permits, host-only failures, and wasted time/memory on developer machines.
The rewritten [FIRST-PRINCIPLES-REVIEWED] 9392c92 |
764d4cb to
34e4137
Compare
34e4137 to
5212685
Compare
|
F1 — PATH-resolved The adjudicator is right that the stated vector does not reach: The underlying point stands on its own though, so it is fixed rather than
The Windows arm is |
5212685 to
1bd8511
Compare
1bd8511 to
43d7176
Compare
|
F1 — the new test could orphan its own sleeper — fixed. Correct, and it is this PR's own subject turning up in the test written for it: The poll now sits in a Verified by re-running the mutation that makes the test fail (production reap |
|
|
f462d8f to
179d1c7
Compare
179d1c7 to
ef21677
Compare
ef21677 to
fe59d00
Compare
fe59d00 to
3ff1370
Compare
3ff1370 to
316483b
Compare
316483b to
3671d95
Compare
|
|
|
Five instrumented full backend runs plus five frontend runs on Linux, with an in-process audit hook attributing every write, spawn, connect and kill to a test, and a per-test census of duration, RSS, threads and descriptors. Zero flaky tests across 5 x 106,199, so what this closes is latent: tests that fail only on a developer's host, side effects that outlive the run, and cost nobody was charged for. The worst finding predated the runs. Nine python3 processes were spinning at 1464% CPU between them -- 14.6 cores -- reparented for six days, left by four earlier runs of the security-conductor tests; killing them moved the host's load average from 21 to 9.6. The cause is two individually-correct decisions meeting: child_env inherits PATH (a proof needs an interpreter) while repointing HOME at a throwaway worktree (the blast-radius bound). Where PATH leads with a version-manager shim, the bare name python3 IS the manager, and a manager that cannot find its tool state under the substituted HOME never execs anything -- it spins. verify_finding now reaps the process GROUP, the shape run_bounded already used one layer up, and the tests pin the real interpreter's directory first on PATH. The same class was then found independently in the symbolizer tests, where it blocks an xdist worker outright rather than merely leaking, which is a lost run. Also closed: repo-root walkers descending .worktrees/ and auditing other branches' checkouts (3 deterministic failures CI cannot see); a node gate that checked presence and not the version floor the scripts need (2 more); a hook permit and in-flight claim leaked forever because the only code that released them was the stub, plus the production leak window the same audit exposed; eight dev-fleet tests leaking staged sync temp dirs; and eleven slack-gateway tests each paying a real 30s drain, ~330s of sleeping a run, none of it asserted on. The memory model had gone stale under the suite and the per-worker reservation with it: the collection floor is 1,499 MiB for 106,491 items, not ~747 for ~57,000, and sampled worker peaks at -n 12 read 1,879 / 2,042 / 2,771 MiB against a 2 GiB reservation. _GIB_PER_WORKER is 3, and the tests that assert on it now derive from the constant instead of restating it. The remaining 43 fixes are the verified findings of a parallel static sweep -- bounded waits for parked events, patch targets that missed, `assert x == y or z` reading as always-true, ratchets retaining parsed source, and process-globals restored to what a test inherited. One finding is left open on purpose and documented as such: the interrupted-rebootstrap installer test is vacuous rather than merely expensive, and a rewrite that named its window -- proven non-vacuous by mutation -- passed at -n0 and failed under the full suite, so it is reverted rather than shipped red. Testing-conventions carries the round-4 section, the corrected memory table and five new rules; the writing-tests skill and the AGENTS.md index row point at them.
Problem / Motivation
Nine
python3processes were burning 14.6 cores for six days on a developerbox, left behind by earlier runs of this repo's own test suite. Killing them took
the host's load average from 21 to 9.6.
They were not the only thing the suite left on the machine. Five instrumented
full backend runs plus five frontend runs turned up host writes, leaked temp
directories, a permit leaked for the life of an xdist worker, six tests that fail
on a developer host and cannot fail on CI, and ~330s a run of pure sleeping.
Why it matters
Each of these costs somebody something and none of them show up as a red test.
An orphaned shim eats cores until a human notices by hand. A gate that scans
another branch's checkout reports offenders nobody can fix, and CI stays green
while the developer is blocked. A leaked permit makes a later capacity test pass
for the wrong reason, or hang until pytest-timeout kills the worker — which on
the Windows job, at
--max-worker-restart=0, ends the whole run and deletesevery result it had not reached.
The per-worker memory reservation matters for the same reason: it is what stops
-n autoswapping a laptop, and the suite had roughly doubled underneath thenumbers that justified it.
What changed (motivation → approach → change)
A proof of concept, or any test-spawned tool, is often reached by its bare name.
On a developer host
PATHleads with a version-manager shim directory (mise,asdf, pyenv, volta), and a shim reads its tool set from
HOME.child_envrepoints
HOMEat a throwaway worktree on purpose — that is the blast-radiusbound — so the shim finds no tool state and never execs an interpreter. It spins
instead, forever.
So
HOMEandPATHare treated as a pair.verify_findingreaps the processgroup rather than the one pid it spawned, the shape
installer_test_helpers.run_boundedalready used; the spawning tests pin thereal interpreter's directory first on
PATH; and the symbolizer helpers, wherethe same collision blocks a worker outright, resolve node before building the
child env and gate on the version floor the scripts need rather than on
which("node").The rest follows the same rule — make the failure name itself. Awaits a test must
itself unblock carry a bounded ceiling, including when the OUTER pytest-timeout is
what fires: that arrives as
BaseException, so a reap hung only offsubprocess.TimeoutExpiredis skipped andPopen.__exit__blocks unbounded on alive child.
dev_fleet's staged sync temp dirs get an owner.hooks.pymoves its contextload inside the
trythat releases the permit, andtest_webhooks_event_loopgives back what it stubs.
slack/gatewaynames its 30s pre-restart drain so thetests about the fetch/reset sequence can shorten it while the test about the
value still pins it.
xdist_budgetreserves 3 GiB, and the tests that assert onthat reservation derive from the constant instead of restating it.
Ledger — every area this touches
src/kiro_crew/builtin_skills/security-conductor/scripts/verify_finding.pysrc/kiro_crew/dashboard/handlers/hooks.pytrythat releases the permitsrc/kiro_crew/slack/gateway.pyxdist_budget.pyconftest.py_test_owned_rootsis memoisedtest/(46 files)BaseExceptionpath), restored globals, corpus caches routed through the shared helper, pid-reuse guards on every confirmed-dead reapdocs/system-specs/common/testing-conventions.mdsrc/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.mdHOME/PATHpair, group reaps, the remeasured worker cost, six checklist itemsAGENTS.md.github/black-baseline.txttest/windows-expected-failures.txtTests
Every fix was verified by running its file, and the whole suite was run after
them: 106,269 passed, 0 failed.
New or rewritten coverage locks in the parts that were unpinned:
test_a_wedged_grandchild_does_not_outlive_the_deadline— a proof that forks achild which outlives its parent must be gone after the deadline. Proven by
mutation: swapping the
killpgback forPopen.kill()fails it by name._sweep_staged_sync_tempdirs— an autouse owner for the sync snapshots, so thenext test that stubs
_start_runinline cannot leak them either.store_dirrestores the hook permit and in-flight claim to what the testinherited, checked by probing both globals after the file runs.
their own line instead of hanging.
Manual verification
The audit itself: five backend runs at
-n 12and five frontend runs(
vitest --coverage+ the Electronnode:testsuite on Node 24.21.0), with anin-process audit hook attributing every write, spawn, connect and kill to a test,
and a per-test census of duration, RSS, threads and descriptors. Zero flaky tests
across 5 × 106,199. The collection floor and worker peaks were measured directly
(
--collect-only -n0: 1,499 MiB for 106,491 items; sampled peaks at-n 12:1,879 / 2,042 / 2,771 MiB) and the reservation follows those numbers.
Overlap with open PRs — please sequence
Three open PRs independently fix findings this audit also reached. Flagging
rather than guessing which lands first:
python3PoC token inpoc_argv. Complementary: thatremoves the shim from the argv, the group reap here removes the orphan a
forking wrapper leaves whatever the argv says.
.worktreescorpus reds with abetter mechanism — the corpus asks git, so an ignored tree is out of scope by
construction. Those two files are reverted here to main's version; the finding is
closed, just not by this diff.
fixes
resource_status.py, which this does not touch. Its approach supersedesthe constant bump here; the measured floor and worker peaks recorded in
testing-conventionsstand either way.Issue link
N/A — no tracking issue; the findings are recorded in
docs/system-specs/common/testing-conventions.mdunder the fourth-pass section.Screenshots / video
N/A — no user-visible UI change.