Repository navigation
test: Close what the eleventh hygiene sweep found - #14388
Conversation
|
Intent: Make the backend suite green and honest on a shared host by closing what a five-run hygiene sweep measured: the process-global dispatcher leak behind the moving 409 red set, four host-dependent or racy tests, and five unit tests that reached real host programs or a fixed host path -- each fixed at the seam that produced it, and recorded in the testing spec. |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The macOS-26 What this change shipsInventory (10 items, capped — 13 raw differences) — 10 justifiedIntent: FIX — make five full-suite macOS runs green at the seam of each red and stop unit tests reaching real host programs.
[FIRST-PRINCIPLES-REVIEWED] f5caa04 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. The one candidate concern — that the new autouse pins ( [DESIGN-REVIEWED] f5caa04 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
1b5684a to
b0db423
Compare
b0db423 to
5c25235
Compare
|
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
5c25235 to
30f46c4
Compare
30f46c4 to
ba24d9e
Compare
|
…s ceiling; thirteenth test-hygiene pass PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
…s ceiling; thirteenth test-hygiene pass PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
…s ceiling; thirteenth test-hygiene pass PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
…s ceiling; thirteenth test-hygiene pass PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
…s ceiling; thirteenth test-hygiene pass PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
Five full rounds on macOS 26 (138,593 backend / 39,693 vitest / 2,199 electron tests per round, eight workers, a live gateway on the host). The backend was red every round -- 12, 12, 11, 13, 13 -- with a moving red set; vitest and electron were green five times. Every red is either fixed at the seam that produced it or recorded in the spec as the probe's. The leak behind the largest red set. hooks_integration._lifecycle_dispatcher is a process-wide slot init_hooks_system assigns and production never clears, so it carried the last app-building test's LifecycleDispatcher -- MagicMock cron service included -- into test_trusted_apps_api's revoke teardown, which awaited the mock and answered 409 teardown_incomplete (7-9 tests every round, all green alone). A rootdir conftest floor snapshots and restores both hooks-integration globals around every test, in the shape of the existing log-record-factory and queue-listener floors, with its own floor tests. The other reds. test_darwin_spawn_marker spawns the test interpreter, not `sleep`: macOS 26 answers KERN_PROCARGS2 argv-only for Apple platform binaries, so the marker was unreadable (5/5) and the refusal test vacuous. test_crew_log_session_tree_projection drops the emitter's kept handles after each test (they held write leases process-wide) and test_eventlog_hooks asserts the lease DELTA, not an empty global table. test_ci_ipv6_routing runs its five nested collect-only pytests concurrently under the outer timeout budget instead of 5 x 60 s in series inside 120 s, each with --basetemp under tmp_path. test_cron_manual_run_claim_window takes the first wrapper task from attach_run_task instead of racing _run_claimed_manual for claim.task. Real host programs reached from unit tests, each pinned at the seam the product reads and probe-verified to zero spawns: gh (a live GitHub API call from test_review_driver's post read-back), kiro-cli --version (32 per round from test_acp_client), ssh -V (three sandbox launcher suites), launchctl and ps (test_pod_api). The DeepSeek launch golden's fixed /opt/scratch window is no longer rmtree'd on the host. Spec: testing-conventions gains the eleventh-pass section, including the two probe defects that manufactured the largest classes (dir_fd-relative operands read as checkout writes; the probe's own os.path calls observed by a spied test) and what was reviewed and deliberately left. writing-tests gains five checklist items.
ba24d9e to
f5caa04
Compare
…s ceiling; thirteenth test-hygiene pass (#14424) PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
Problem / Motivation
Goal: A full backend run on a shared macOS host comes back green for the right reasons -- every red of a five-run sweep is fixed at the seam that caused it, and no unit test reaches a real host program.
Five full rounds of the suite on a macOS 26 host (138,593 backend tests per round, eight workers) were red in every round -- 12, 12, 11, 13, 13 failures -- with a red set that changed from round to round. Seven to nine of those were
test_trusted_apps_api.pyrevoke tests answering409 teardown_incomplete, every one of which passes alone. Four unit tests also reached real host programs (gh,kiro-cli,ssh,launchctl) every round, and one spawned ashutil.rmtreeat a fixed absolute host path.Why it matters
A test that is red only on some worker orderings ships as green from a single run and then fails someone else's PR. The
ghreach is a live GitHub API call with the developer's token from a unit test; the/opt/scratchremoval is onemkdiron some host away from deleting real files.Not a goal
platform_compat.py.--collect-onlyexceeded 60 s on a loaded Mac; this PR removes the timeout stack that turned it into a red, not the slowness.git cwd=NoneCLI-script contract the tenth-pass section already ruled on.What changed (motivation → approach → change)
The moving red set had one cause.
hooks_integration._lifecycle_dispatcheris a process-wide slot thatinit_hooks_systemassigns and production never clears, so the last dashboard-app-building test on a worker (first seen:test_dashboard_route_table.py) left itsLifecycleDispatcher--MagicMockcron service included -- for the revoke teardown to await. A rootdirconftest.pyfloor now snapshots and restores both hooks-integration globals around every test, in the shape of the existing log-record-factory and queue-listener floors, with its own tests intest_host_isolation_floor.py.Each remaining red is fixed at the seam that produced it:
test_darwin_spawn_marker.pyspawns the test interpreter (macOS 26 answersKERN_PROCARGS2argv-only for Apple platform binaries such assleep, so the marker was unreadable and the refusal test vacuous;platform_compat.darwin_process_environdocuments the case);test_crew_log_session_tree_projection.pyandtest_crew_log_session_tree_adopt.pydrop the emitter's kept handles after each test, so the leases they held stop leaking intotest_eventlog_hooks.py, whose strict empty-table assertion is kept as the ratchet it is;test_ci_ipv6_routing.pyruns its five nested collect-only pytests concurrently, each still under_run's 60 s cap, instead of in series inside a 120 s test;test_cron_manual_run_claim_window.pytakes the first wrapper task fromattach_run_taskinstead of racing the run forclaim.task.The host reaches are pinned where the product reads them and probe-verified to zero spawns:
discovery._run_ghintest_review_driver.py,installed_kiro_cli_versionintest_acp_client.py,_ssh_supports_accept_newin three sandbox launcher suites,main_pid/process_start_timeintest_pod_api.py. The DeepSeek golden's fixed/opt/scratchwindow was closed on main by #14421 (a real directory undertmp_path); this branch keeps only the spec entry for it.testing-conventions.mdgains the eleventh-pass section, including the two probe defects that manufactured the sweep's largest classes and what was reviewed and left; thewriting-testsskill (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md) gains five checklist items.Backwards compatibility
Compatible: test-only changes plus one docstring in
platform_compat.py; no production behaviour changes.Tests
test/test_host_isolation_floor.py::TestTheHooksDispatcherIsRestored-- three tests driving the new floor's own setup/teardown cycle through the realinit_hooks_system; the restore assertion fails when the restore line is removed (mutation-checked).-n 4(1,272 tests), and under the sweep probe with zerogh/kiro-cli/ssh/launchctlspawns and zero host writes.local-gate.py: 52,129 backend + 185 + 9,014 frontend related tests green; black baseline, isort, flake8,mypy --platform linux, docs lint green.Manual verification
The 409 cause was reproduced by replaying one worker's 1,874 files in order at
-n0with a temporary debug hook on the revoke handler, which named the stale dispatcher and itsMagicMockcron service; a sentinel plugin then stopped at the first test that left the slot set. Both reproductions and the probe re-measurements are described in the spec section.Related Issues
Follow-up to the tenth-pass sweep section in
testing-conventions.md.Pattern harvest
Rule candidate: review-prompt
Pattern: a module-global assigned by a production boot path (
init_hooks_system, a resolver'slru_cache) and read by a later unit test needs a rootdir floor, not a per-file pin -- the leaker is any of the ~170 files that reach the boot path.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)