Skip to content

test: fourth side-effect audit: orphan reaps, worktree scans, budget - #11010

Merged
bolichen97 merged 1 commit into
mainfrom
test/fourth-side-effect-audit
Sep 16, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/fourth-side-effect-audit

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Nine python3 processes were burning 14.6 cores for six days on a developer
box, left behind by earlier runs of this repo's own test suite. Killing them took
the host's load average from 21 to 9.6.

They were not the only thing the suite left on the machine. Five instrumented
full backend runs plus five frontend runs turned up host writes, leaked temp
directories, a permit leaked for the life of an xdist worker, six tests that fail
on a developer host and cannot fail on CI, and ~330s a run of pure sleeping.

Why it matters

Each of these costs somebody something and none of them show up as a red test.
An orphaned shim eats cores until a human notices by hand. A gate that scans
another branch's checkout reports offenders nobody can fix, and CI stays green
while the developer is blocked. A leaked permit makes a later capacity test pass
for the wrong reason, or hang until pytest-timeout kills the worker — which on
the Windows job, at --max-worker-restart=0, ends the whole run and deletes
every result it had not reached.

The per-worker memory reservation matters for the same reason: it is what stops
-n auto swapping a laptop, and the suite had roughly doubled underneath the
numbers that justified it.

What changed (motivation → approach → change)

A proof of concept, or any test-spawned tool, is often reached by its bare name.
On a developer host PATH leads with a version-manager shim directory (mise,
asdf, pyenv, volta), and a shim reads its tool set from HOME. child_env
repoints HOME at a throwaway worktree on purpose — that is the blast-radius
bound — so the shim finds no tool state and never execs an interpreter. It spins
instead, forever.

So HOME and PATH are treated as a pair. verify_finding reaps the process
group rather than the one pid it spawned, the shape
installer_test_helpers.run_bounded already used; the spawning tests pin the
real interpreter's directory first on PATH; and the symbolizer helpers, where
the same collision blocks a worker outright, resolve node before building the
child env and gate on the version floor the scripts need rather than on
which("node").

The rest follows the same rule — make the failure name itself. Awaits a test must
itself unblock carry a bounded ceiling, including when the OUTER pytest-timeout is
what fires: that arrives as BaseException, so a reap hung only off
subprocess.TimeoutExpired is skipped and Popen.__exit__ blocks unbounded on a
live child.
dev_fleet's staged sync temp dirs get an owner. hooks.py moves its context
load inside the try that releases the permit, and test_webhooks_event_loop
gives back what it stubs. slack/gateway names its 30s pre-restart drain so the
tests about the fetch/reset sequence can shorten it while the test about the
value still pins it. xdist_budget reserves 3 GiB, and the tests that assert on
that reservation derive from the constant instead of restating it.

Ledger — every area this touches

Area What changed there
src/kiro_crew/builtin_skills/security-conductor/scripts/verify_finding.py reaps the proof's process group; spawns it in its own session
src/kiro_crew/dashboard/handlers/hooks.py the saved-context load moves inside the try that releases the permit
src/kiro_crew/slack/gateway.py the 30s pre-restart drain becomes a named constant the tests can shorten
xdist_budget.py per-worker reservation 2 → 3 GiB, with the remeasured floor and peaks
conftest.py the import-time home floor is named after the run; the per-test temp fixture is registered only in that mode; _test_owned_roots is memoised
test/ (46 files) bounded waits, group reaps (including on the outer-timeout BaseException path), restored globals, corpus caches routed through the shared helper, pid-reuse guards on every confirmed-dead reap
docs/system-specs/common/testing-conventions.md the fourth-pass section, the corrected memory table, five new rules, and the one finding left open on purpose
src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md the HOME/PATH pair, group reaps, the remeasured worker cost, six checklist items
AGENTS.md the testing index row names the new topics
.github/black-baseline.txt one file graduated to black-clean, pruned as the gate asks
test/windows-expected-failures.txt one entry burned down: the ReDoS test now passes on Windows, and a strict XPASS is how this list asks to be pruned

Tests

Every fix was verified by running its file, and the whole suite was run after
them: 106,269 passed, 0 failed.

New or rewritten coverage locks in the parts that were unpinned:

  • test_a_wedged_grandchild_does_not_outlive_the_deadline — a proof that forks a
    child which outlives its parent must be gone after the deadline. Proven by
    mutation: swapping the killpg back for Popen.kill() fails it by name.
  • _sweep_staged_sync_tempdirs — an autouse owner for the sync snapshots, so the
    next test that stubs _start_run inline cannot leak them either.
  • store_dir restores the hook permit and in-flight claim to what the test
    inherited, checked by probing both globals after the file runs.
  • 32 parked-event awaits across the slot-race and guardrail suites now fail at
    their own line instead of hanging.

Manual verification

The audit itself: five backend runs at -n 12 and five frontend runs
(vitest --coverage + the Electron node:test suite on Node 24.21.0), with an
in-process audit hook attributing every write, spawn, connect and kill to a test,
and a per-test census of duration, RSS, threads and descriptors. Zero flaky tests
across 5 × 106,199. The collection floor and worker peaks were measured directly
(--collect-only -n0: 1,499 MiB for 106,491 items; sampled peaks at -n 12:
1,879 / 2,042 / 2,771 MiB) and the reservation follows those numbers.

Overlap with open PRs — please sequence

Three open PRs independently fix findings this audit also reached. Flagging
rather than guessing which lands first:

Issue link

N/A — no tracking issue; the findings are recorded in
docs/system-specs/common/testing-conventions.md under the fourth-pass section.

Screenshots / video

N/A — no user-visible UI change.

@bolichen97
bolichen97 requested review from a team and smeyffret September 15, 2026 08:52
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Close what a fourth five-run side-effect audit found: processes,
files and permits the suite leaves on the machine, tests that can only fail on a
developer's host, and the per-worker memory model the suite outgrew — and write
each class into testing-conventions so the next round starts from the rules
rather than from the measurements.

Not a goal: changing any product behaviour, or re-fixing the three findings
that open PRs #10825, #10978 and #10840 already address — those are flagged in
the description for sequencing, not duplicated on purpose.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 15, 2026
@bolichen97 bolichen97 added the posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only label Sep 15, 2026
@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 9392c920e75428f683c560d02687d2823cb5240d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the Windows CI job actually exercises the new run_poc group-reap path — all ten manual audit runs were Linux; the taskkill branch is no cover.

[DESIGN-REVIEWED] 9392c92

@bolichen97

Copy link
Copy Markdown
Contributor Author

Why posix-only-approved is on this PR.

One added line trips the Cross-Platform Portability scan:

os.killpg(process.pid, signal.SIGKILL)

in src/kiro_crew/builtin_skills/security-conductor/scripts/verify_finding.py.

It is the POSIX arm of a two-arm split — the Windows arm right below it shells to
taskkill /T /F /PID — so the code path is not POSIX-only, but the flagged line
is, and the scan is a line regex with no view of the branch around it.

Every other caller in src/kiro_crew/ routes this through
platform_compat.kill_process_tree, and that is the right answer for all of them.
It is not available here: this script is executed as a bare
python3 verify_finding.py against an untrusted scratch checkout, with no
guarantee that kiro_crew is importable, and its whole point is that what it
does cannot change with the revision merged under it. Importing the package to
reach one helper would trade that property away.

So the label, which is the escape hatch the gate's own message names, rather than
getattr(os, "killpg") to slip past the regex.

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 9392c920e75428f683c560d02687d2823cb5240d — this comment is updated in place on each push.

Review details

I've verified the sole candidate against the actual code. The normal-completion path (verify_finding.py:756) reaps the leader via process.wait(), then reap() at line 775 calls os.killpg(process.pid, SIGKILL) at line 641.

Falsifying candidate 1: For the killpg to land on an unrelated process, all of the following must coincide inside the sub-microsecond window between wait() returning and killpg(): (1) the just-freed leader pid must be reassigned to a brand-new process — on Linux pid allocation is roughly monotonic, so this needs a full pid_max wraparound to hit that exact number; and (2) that new process must itself become a process-group leader (setsid/setpgid(0,0)) so that a group with pgid == process.pid even exists — a freshly-forked process inherits its parent's group, so killpg on its pid otherwise returns ESRCH and falls through to the guarded process.kill() no-op. The observable wrong outcome (c) is thus purely conditional ("if an unrelated process becomes a group leader on the freed pid in the window"), not something that occurs in practice; the candidate itself scores it low. The harm is also bounded to a SIGKILL on the operator's own machine inside the operator's own verifier — the RFC trust boundary the code repeatedly names. This lands well below the 80 bar and is dropped.

No grounded Step-2 finding survives on the production hunks (the hooks.py load-inside-try move and the slack/gateway.py named constant are fixes/benign; the verify_finding.py change is a net improvement whose only regression is the non-reachable race above).

No findings.

[OPUS-REVIEWED] 9392c92

Verdict parsed from the review's SHA-scoped output markers for commit 9392c920e75428f683c560d02687d2823cb5240d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9392c920e75428f683c560d02687d2823cb5240d: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 9392c920e75428f683c560d02687d2823cb5240d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9392c92

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9392c920e75428f683c560d02687d2823cb5240d: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 9392c920e75428f683c560d02687d2823cb5240d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One rider to eye: the real-kiro-cli precedence guard went opt-in (KIROCREW_E2E_REAL_KIRO_CLI=1) — coverage that ran on any PATH-equipped host now runs only on request, and the description never says so.

Not justified as shipped

  • Item 7 — undeclared: the description's ledger ("bounded waits, group reaps, restored globals…") never mentions the new KIROCREW_E2E_REAL_KIRO_CLI env var or that the anti-drift precedence test no longer runs automatically. The gate itself is derived — testing-conventions.md:51/:351 ("Never spawn real kiro-cli in tests") forbade the old PATH-presence trigger, and the credentialed-spawn/browser-login harm is named — so this is an inventory tag, not a defect; it mirrors the existing KIROCREW_E2E_SCENARIOS_REAL_AGENT lever (5 consuming files) without duplicating it (different suite, different consent scope).

What this change ships

Inventory (10 items, capped — the change has more) — 9 justified

Intent: FIX — stop the test suite from leaving orphaned processes, leaked temp dirs, leaked permits, host-only failures, and wasted time/memory on developer machines.

  1. The security-conductor verifier kills a proof's whole process group on every exit path, including clean exit — justified
  2. Hook-agent runs release their permit and in-flight claim even when the saved-context load raises or is cancelled — justified
  3. The Slack pre-restart drain is a named constant (value unchanged, pinned by its own test); ~330s of sleeping removed per run — justified
  4. Local runs reserve 3 GiB per xdist worker instead of 2, so small-RAM hosts get fewer workers — justified
  5. Leaked test-floor directories now carry user+pid so a stray one is attributable — justified
  6. The per-test temp fixture exists only when KIROCREW_TMP_PER_TEST is set, dropping its per-test cost — justified
  7. The real-kiro-cli precedence test runs only under KIROCREW_E2E_REAL_KIRO_CLI=1, not on PATH presence — undeclared (new env var and changed trigger the description never mentions; harm derived from the documented no-real-kiro-cli invariant)
  8. ~46 test files gain bounded waits, group reaps with pid-reuse identity guards, restored globals, and shared-corpus reads — justified
  9. The ReDoS gate test defers to pytest-timeout and its Windows expected-failure entry is burned down — justified
  10. Testing conventions, the writing-tests skill, and the AGENTS index record the new rules and remeasured memory model in the same commit — justified

The rewritten reap docstring reverses a recorded direct-child-only decision, with the required evidence in-diff: the measured six-day orphan incident plus test_a_wedged_grandchild_does_not_outlive_the_deadline, proven by mutation. The 2→3 GiB bump is remeasured (1,499 MiB collection floor, 2,771 MiB worst peak) and the author flags #10840 as superseding it — sequencing, not duplication.

[FIRST-PRINCIPLES-REVIEWED] 9392c92

@bolichen97 bolichen97 added posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only and removed posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only labels Sep 15, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch 2 times, most recently from 764d4cb to 34e4137 Compare September 15, 2026 09:50
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 15, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from 34e4137 to 5212685 Compare September 15, 2026 10:04
@bolichen97

Copy link
Copy Markdown
Contributor Author

F1 — PATH-resolved taskkill — fixed, not argued.

The adjudicator is right that the stated vector does not reach: reap's
subprocess.run passes no cwd, so it resolves in the verifier's directory,
and the untrusted worktree is only ever the PoC child's cwd. Any write that
could land a taskkill.exe where reap looks already implies the operator-level
RCE the finding says it grants.

The underlying point stands on its own though, so it is fixed rather than
overridden: a bare argv[0] on Windows is resolved starting from the calling
process's own directory, and this line fires while attacker-controlled code is
running. The rest of the tree already refuses to reach git or gh through
PATH for exactly that reason.

reap now resolves %SystemRoot%\System32\taskkill.exe and fails closed to
process.kill() when that binary is not there, rather than letting the search
order pick a substitute. SystemRoot is read with a C:\Windows fallback and not
trusted for anything else — the verifier holds the operator's own environment, so
a poisoned value is not this script's threat model, but an absent one is ordinary.

The Windows arm is # pragma: no cover on the POSIX matrix, so
test_the_windows_reap_names_taskkill_by_absolute_path pins it on the source: a
bare "taskkill" may not reappear, the System32 binary must be named, and the
isfile fail-closed check must stay. No override requested.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from 5212685 to 1bd8511 Compare September 15, 2026 10:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from 1bd8511 to 43d7176 Compare September 15, 2026 10:22
@bolichen97

Copy link
Copy Markdown
Contributor Author

F1 — the new test could orphan its own sleeper — fixed.

Correct, and it is this PR's own subject turning up in the test written for it:
the assertion only fails while the grandchild is still alive, so a bare assert
left a 300-second sleeper behind on exactly the runs that report the bug. I hit it
by hand during the mutation check and should have closed it then.

The poll now sits in a try whose finally SIGKILLs that pid on every exit path,
including the assertion's, with contextlib.suppress(OSError) because
ProcessLookupError is the success path arriving there and a pid this test never
owned is not ours to signal.

Verified by re-running the mutation that makes the test fail (production reap
swapped back to Popen.kill()): the test still fails by name — so the cleanup did
not weaken it — and pgrep -cf "time.sleep(300)" reads the same before and after,
where previously the failing run left one behind.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 15, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • hooks.py:1304 — the comment's failure examples were wrong — fixed in f462d8feb3972acf0ffee3d269acb7afb093607b

Both halves of the finding hold, verified in the code.
_read_json_file catches FileNotFoundError/ValueError/OSError and answers
None, so "a corrupt or unreadable hooks.json" cannot raise out of
_load_hook_context and was never an escape. And CancelledError derives from
BaseException, so a cancellation does not reach the except Exception leg the
comment pointed at.
The code change itself stands on a different, real pair: resolve_context parses
what the store holds, so malformed stored data DOES raise into except Exception;
and cancellation is covered by the finally, which only runs if the await is
inside the try. The comment now says exactly that and names which door each escape
leaves by.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • verify_finding.py — normal completion bypassed the group sweep — fixed in 179d1c7a061933b498d813e12cc769b30f1fa87b

Real, and it is the shape the measured incident actually took rather than a
variant of it: a version-manager shim FORKS, the fork is what spins, and the
wrapper can exit with a perfectly good status while its child keeps a core busy.
Hanging the sweep off the deadline path alone meant the one run where there is no
deadline -- verdict decided, status in hand -- abandoned the survivor.
run_poc now reaps the group on every exit path: the timeout path as before, a
new except BaseException so a signal reaching the conductor cannot turn the
proof loose, and normal completion. Safe after wait(): POSIX will not reassign
a pgid while the group is non-empty, so it reaches survivors or fails ESRCH -- it
cannot land on an unrelated process that inherited the number.
Pinned by test_a_fork_outliving_a_SUCCEEDING_proof_is_still_reaped, and proven
non-vacuous: removing only the normal-path sweep fails it by name.

@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from f462d8f to 179d1c7 Compare September 15, 2026 21:20
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 15, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from 179d1c7 to ef21677 Compare September 15, 2026 23:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from ef21677 to fe59d00 Compare September 15, 2026 23:59
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 16, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from fe59d00 to 3ff1370 Compare September 16, 2026 05:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 16, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from 3ff1370 to 316483b Compare September 16, 2026 05:29
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 16, 2026
@bolichen97
bolichen97 force-pushed the test/fourth-side-effect-audit branch from 316483b to 3671d95 Compare September 16, 2026 05:44
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • stale descendant pids could still be killed on the failing path — fixed in 3671d952d6f45cd723dd58167d10e124d8d94f4a

Right, and this is the residual my previous round left: skipping the kill when the
body had PROVEN the death removed the systematic case (every passing run), but on
the failing path the pid can still be reassigned between the last poll and the
SIGKILL, and that lands on an unrelated host process.
All four sites now capture platform_compat.get_process_start_id(pid) WITH the pid
and revalidate it immediately before the pid-scoped signal --
test_cloud_connect.py, test_run_tests_reap.py, and both sites in
test_security_conductor_scripts.py. None means identity unknown and therefore no
signal, which is the rule that helper's own docstring states. It is in-process and
non-blocking on Linux/macOS/Windows, which is the "pinned helper on Windows" the
finding asks for.
Verified rather than argued: after a captured process exits the identity reads
None (so the guard refuses), and a different live process never matches the stale
identity. The group kill above is unchanged -- a pgid cannot be reassigned while
its group is non-empty.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 16, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Windows reap: nonzero taskkill read as success, single hardcoded root — fixed in 42b749138d92eee2528f75e2fd83dd4f095c9901

Both halves hold. check=False with the status discarded meant a taskkill that
answered 128 -- which it does for an access denial, not only for "no such process"
-- returned as though the tree had been killed, and the descendants kept running.
And one hardcoded root abandoned the tree entirely on any Windows not installed at
C:\Windows.
The status is now READ: only returncode == 0 counts as reaped, and a missing
binary, a raise or a nonzero exit all fall through to the direct kill, which at
least reaps the wrapper. The binary is resolved from a tuple of trusted absolute
literals -- System32, Sysnative (what a 32-bit interpreter needs to see the real
System32), and the legacy WINNT root -- each probed with isfile, so it still
fails closed and no environment value reaches argv[0].
The fallback stays best-effort rather than becoming a verdict error, deliberately:
once the OS tool is unavailable there is no portable tree kill left, and the RFC's
trust boundary already places "the operator's checkout left a process on the
operator's machine" outside this script. Pinned on the source by
test_the_windows_reap_names_taskkill_by_absolute_path, extended to cover the
status read and the no-environment rule.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Windows cannot reap descendants after the parent exits — rebutted as not a defect of this change

The mechanism is real: on Windows taskkill /T /PID walks a parent link that is
gone once the direct child has exited, so descendants can survive. I am not
disputing that. What I dispute is that it is a defect HERE.
It is the standard every production caller in this repo already meets and none
exceeds. platform_compat.kill_process_tree IS "killpg on POSIX, taskkill /T /F
on Windows", and it is what power.py, cron.py, task_executor.py,
hooks.py and kiro_prerequisite.py all use to reap a tree. A Job Object appears
exactly once in the tree, in test/installer_test_helpers.py::_KillOnCloseJob, and
even there it is a best-effort upgrade that FALLS BACK to taskkill /T /F and then
to the direct kill. So this asks a dev-skill script to exceed the bar its own
production runtime sets.
This change also strictly improved the Windows path rather than regressing it:
before it there was no tree kill on the normal-completion path at all, and now
there is one, resolved from trusted absolute literals with its status read.
verify_finding.py cannot reach kill_process_tree_pinned -- it runs as a bare
python3 verify_finding.py against an untrusted checkout with no guarantee that
kiro_crew is importable, which is the property that keeps it independent of the
revision merged under it.
I am declining to hand-port a Job Object here for a reason I would rather state
than hide: it is ~80 lines of Windows ctypes (job creation, assignment, and a
Thread32First/ResumeThread resume for the suspended-spawn ordering the finding
asks for) that I have no Windows host to execute, in a branch marked
# pragma: no cover. Untestable security code is a worse outcome than the
documented residual.
Raising the Windows tree-kill floor is a repo-wide decision about
platform_compat, not this PR's to take. If a maintainer agrees, the reason to
paste is: /ai-review override gpt 42b749138d92eee2528f75e2fd83dd4f095c9901: the Windows residual is platform_compat's documented taskkill /T standard, which every production caller shares; this change improves that path and adding a Job Object belongs in platform_compat.
I am not self-authorizing that override.

Five instrumented full backend runs plus five frontend runs on Linux,
with an in-process audit hook attributing every write, spawn, connect
and kill to a test, and a per-test census of duration, RSS, threads and
descriptors. Zero flaky tests across 5 x 106,199, so what this closes is
latent: tests that fail only on a developer's host, side effects that
outlive the run, and cost nobody was charged for.

The worst finding predated the runs. Nine python3 processes were
spinning at 1464% CPU between them -- 14.6 cores -- reparented for six
days, left by four earlier runs of the security-conductor tests; killing
them moved the host's load average from 21 to 9.6. The cause is two
individually-correct decisions meeting: child_env inherits PATH (a proof
needs an interpreter) while repointing HOME at a throwaway worktree (the
blast-radius bound). Where PATH leads with a version-manager shim, the
bare name python3 IS the manager, and a manager that cannot find its
tool state under the substituted HOME never execs anything -- it spins.
verify_finding now reaps the process GROUP, the shape run_bounded
already used one layer up, and the tests pin the real interpreter's
directory first on PATH. The same class was then found independently in
the symbolizer tests, where it blocks an xdist worker outright rather
than merely leaking, which is a lost run.

Also closed: repo-root walkers descending .worktrees/ and auditing other
branches' checkouts (3 deterministic failures CI cannot see); a node
gate that checked presence and not the version floor the scripts need (2
more); a hook permit and in-flight claim leaked forever because the only
code that released them was the stub, plus the production leak window
the same audit exposed; eight dev-fleet tests leaking staged sync temp
dirs; and eleven slack-gateway tests each paying a real 30s drain, ~330s
of sleeping a run, none of it asserted on.

The memory model had gone stale under the suite and the per-worker
reservation with it: the collection floor is 1,499 MiB for 106,491
items, not ~747 for ~57,000, and sampled worker peaks at -n 12 read
1,879 / 2,042 / 2,771 MiB against a 2 GiB reservation. _GIB_PER_WORKER
is 3, and the tests that assert on it now derive from the constant
instead of restating it.

The remaining 43 fixes are the verified findings of a parallel static
sweep -- bounded waits for parked events, patch targets that missed,
`assert x == y or z` reading as always-true, ratchets retaining parsed
source, and process-globals restored to what a test inherited.

One finding is left open on purpose and documented as such: the
interrupted-rebootstrap installer test is vacuous rather than merely
expensive, and a rewrite that named its window -- proven non-vacuous by
mutation -- passed at -n0 and failed under the full suite, so it is
reverted rather than shipped red.

Testing-conventions carries the round-4 section, the corrected memory
table and five new rules; the writing-tests skill and the AGENTS.md
index row point at them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants