Skip to content

test: Close what the eleventh hygiene sweep found - #14388

Merged
bolichen97 merged 1 commit into
mainfrom
test/hygiene-sweep-20260926
Sep 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/hygiene-sweep-20260926

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A full backend run on a shared macOS host comes back green for the right reasons -- every red of a five-run sweep is fixed at the seam that caused it, and no unit test reaches a real host program.

Five full rounds of the suite on a macOS 26 host (138,593 backend tests per round, eight workers) were red in every round -- 12, 12, 11, 13, 13 failures -- with a red set that changed from round to round. Seven to nine of those were test_trusted_apps_api.py revoke tests answering 409 teardown_incomplete, every one of which passes alone. Four unit tests also reached real host programs (gh, kiro-cli, ssh, launchctl) every round, and one spawned a shutil.rmtree at a fixed absolute host path.

Why it matters

A test that is red only on some worker orderings ships as green from a single run and then fails someone else's PR. The gh reach is a live GitHub API call with the developer's token from a unit test; the /opt/scratch removal is one mkdir on some host away from deleting real files.

Not a goal

  • No production behaviour change: the only non-test edit is a docstring in platform_compat.py.
  • No probe or sweep-skill code in this repository; the two probe defects the pass found are fixed in the skill and only described in the spec here.
  • No explanation for why a single nested --collect-only exceeded 60 s on a loaded Mac; this PR removes the timeout stack that turned it into a red, not the slowness.
  • No change to the git cwd=None CLI-script contract the tenth-pass section already ruled on.

What changed (motivation → approach → change)

The moving red set had one cause. hooks_integration._lifecycle_dispatcher is a process-wide slot that init_hooks_system assigns and production never clears, so the last dashboard-app-building test on a worker (first seen: test_dashboard_route_table.py) left its LifecycleDispatcher -- MagicMock cron service included -- for the revoke teardown to await. A rootdir conftest.py floor now snapshots and restores both hooks-integration globals around every test, in the shape of the existing log-record-factory and queue-listener floors, with its own tests in test_host_isolation_floor.py.

Each remaining red is fixed at the seam that produced it: test_darwin_spawn_marker.py spawns the test interpreter (macOS 26 answers KERN_PROCARGS2 argv-only for Apple platform binaries such as sleep, so the marker was unreadable and the refusal test vacuous; platform_compat.darwin_process_environ documents the case); test_crew_log_session_tree_projection.py and test_crew_log_session_tree_adopt.py drop the emitter's kept handles after each test, so the leases they held stop leaking into test_eventlog_hooks.py, whose strict empty-table assertion is kept as the ratchet it is; test_ci_ipv6_routing.py runs its five nested collect-only pytests concurrently, each still under _run's 60 s cap, instead of in series inside a 120 s test; test_cron_manual_run_claim_window.py takes the first wrapper task from attach_run_task instead of racing the run for claim.task.

The host reaches are pinned where the product reads them and probe-verified to zero spawns: discovery._run_gh in test_review_driver.py, installed_kiro_cli_version in test_acp_client.py, _ssh_supports_accept_new in three sandbox launcher suites, main_pid / process_start_time in test_pod_api.py. The DeepSeek golden's fixed /opt/scratch window was closed on main by #14421 (a real directory under tmp_path); this branch keeps only the spec entry for it.

testing-conventions.md gains the eleventh-pass section, including the two probe defects that manufactured the sweep's largest classes and what was reviewed and left; the writing-tests skill (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md) gains five checklist items.

Backwards compatibility

Compatible: test-only changes plus one docstring in platform_compat.py; no production behaviour changes.

Tests

  • test/test_host_isolation_floor.py::TestTheHooksDispatcherIsRestored -- three tests driving the new floor's own setup/teardown cycle through the real init_hooks_system; the restore assertion fails when the restore line is removed (mutation-checked).
  • Every changed test file runs green alone, together at -n 4 (1,272 tests), and under the sweep probe with zero gh / kiro-cli / ssh / launchctl spawns and zero host writes.
  • local-gate.py: 52,129 backend + 185 + 9,014 frontend related tests green; black baseline, isort, flake8, mypy --platform linux, docs lint green.

Manual verification

The 409 cause was reproduced by replaying one worker's 1,874 files in order at -n0 with a temporary debug hook on the revoke handler, which named the stale dispatcher and its MagicMock cron service; a sentinel plugin then stopped at the first test that left the slot set. Both reproductions and the probe re-measurements are described in the spec section.

Related Issues

Follow-up to the tenth-pass sweep section in testing-conventions.md.

Pattern harvest

Rule candidate: review-prompt
Pattern: a module-global assigned by a production boot path (init_hooks_system, a resolver's lru_cache) and read by a later unit test needs a rootdir floor, not a per-file pin -- the leaker is any of the ~170 files that reach the boot path.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Make the backend suite green and honest on a shared host by closing what a five-run hygiene sweep measured: the process-global dispatcher leak behind the moving 409 red set, four host-dependent or racy tests, and five unit tests that reached real host programs or a fixed host path -- each fixed at the seam that produced it, and recorded in the testing spec.
Not a goal: No production behaviour change (one docstring only), no probe or sweep-skill code in this repo, no fix for the unexplained >60 s nested collect on a loaded Mac beyond removing the timeout stack, and no change to the git cwd=None CLI-script contract the tenth pass already ruled on.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of f5caa0485e2d29730a22d59860f6ccd289261e9c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The macOS-26 KERN_PROCARGS2 platform-binary claim, now written into platform_compat's docstring, rests on one host's measurement — confirm it is OS behaviour, not that host.

What this change ships

Inventory (10 items, capped — 13 raw differences) — 10 justified

Intent: FIX — make five full-suite macOS runs green at the seam of each red and stop unit tests reaching real host programs.

  1. Every test now starts and ends with the hooks-integration dispatcher/registry restored (rootdir conftest floor) — justified
  2. Three new tests drive that floor's own snapshot/leak/restore cycle — justified
  3. The darwin spawn-marker tests spawn the test interpreter instead of sleep — justified
  4. darwin_process_environ's docstring names the platform-binary None case — justified
  5. Both session-tree suites drop emitter handles in teardown, ending cross-test lease leaks — justified
  6. The ipv6 routing test runs its five nested collects concurrently, each with its own --basetemp — justified
  7. The cron claim-window test reads the wrapper task from the attach_run_task seam, not shared state — justified
  8. Unit tests no longer spawn the host's real gh or kiro-cli (pins in two files) — justified
  9. Unit tests no longer spawn the host's real ssh or launchctl (pins in four files, per the existing checklist convention) — justified
  10. testing-conventions.md gains the eleventh-pass section and the writing-tests skill five checklist items, same-commit per AGENTS.md — justified

[FIRST-PRINCIPLES-REVIEWED] f5caa04

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of f5caa0485e2d29730a22d59860f6ccd289261e9c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

The one candidate concern — that the new autouse pins (_ssh_supports_accept_new, installed_kiro_cli_version) could make the probes themselves untested — doesn't hold: the real probe is still exercised unpinned in test_sandbox_argv.py::TestSshSupportsAcceptNew, and the pins are monkeypatch-overridable per test. Each fix lands at the seam that caused the red rather than at the victim, the conftest floor follows the repo's established restore-floor shape with mutation-checked self-tests, and the only non-test edit is a docstring. Description and diff match bidirectionally, including the explicitly deferred items (/opt/scratch via #14421, probe fixes in the skill).

[DESIGN-REVIEWED] f5caa04

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of f5caa0485e2d29730a22d59860f6ccd289261e9c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f5caa04

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f5caa0485e2d29730a22d59860f6ccd289261e9c: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the test/hygiene-sweep-20260926 branch from 1b5684a to b0db423 Compare September 27, 2026 06:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@bolichen97
bolichen97 requested review from a team and cruisercohen and removed request for a team September 27, 2026 06:17
@bolichen97
bolichen97 force-pushed the test/hygiene-sweep-20260926 branch from b0db423 to 5c25235 Compare September 27, 2026 06:43
@bolichen97

bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author
  • lease ratchet is weakened to accept inherited leaks (test/test_eventlog_hooks.py:385) span=cf1146645185

fixed in 5c25235: test/test_eventlog_hooks.py is reverted to main, strict assert not lease._held kept on every site.
Only the creator half stays: test_crew_log_session_tree_projection.py drops the emitter's kept handles in its autouse fixture, which releases the leaked leases.
Mutation-verified: with that fixture removed the strict eventlog test fails on a two-file run (2 failed); with it, 94 passed x3.
Spec paragraph in docs/system-specs/common/testing-conventions.md now says the same and names a-ratchet-may-only-tighten.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed f5caa0485e2d29730a22d59860f6ccd289261e9c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f5caa04

Verdict parsed from the review's SHA-scoped output markers for commit f5caa0485e2d29730a22d59860f6ccd289261e9c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f5caa0485e2d29730a22d59860f6ccd289261e9c: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the test/hygiene-sweep-20260926 branch from 5c25235 to 30f46c4 Compare September 27, 2026 07:26
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main's shard-3 red (test_review_repair_routing_skill.py skill byte cap, fixed by #14398) was the only failure on the previous head; no code change in this PR (94bcda2 -> f2c732e); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97
bolichen97 force-pushed the test/hygiene-sweep-20260926 branch from 30f46c4 to ba24d9e Compare September 27, 2026 08:06
@bolichen97
bolichen97 requested a review from a team as a code owner September 27, 2026 08:06
@bolichen97

bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 8 -- the creator class has a counted sibling (test_crew_log_session_tree_adopt.py) span=155421392810

fixed in ba24d9e: test_crew_log_session_tree_adopt.py::_isolated_home now calls emit.reset_caches() in teardown, same as the projection suite.
Confirmed the sibling leaked: adopt + eventlog_hooks as a two-file run was 2 failed before, 98 passed after.
The remaining on_session_opened caller without reset_caches, test_chat_send_agent_model_default.py, mocks the emitter (MagicMock) and holds no handle; the two-file run with eventlog_hooks is 67 passed.

bolichen97 added a commit that referenced this pull request Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@bolichen97
bolichen97 enabled auto-merge (rebase) September 27, 2026 09:48
bolichen97 added a commit that referenced this pull request Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
chenmingwei23
chenmingwei23 previously approved these changes Sep 27, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 27, 2026
bolichen97 added a commit that referenced this pull request Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
bolichen97 added a commit that referenced this pull request Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
bolichen97 added a commit that referenced this pull request Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
Five full rounds on macOS 26 (138,593 backend / 39,693 vitest / 2,199
electron tests per round, eight workers, a live gateway on the host). The
backend was red every round -- 12, 12, 11, 13, 13 -- with a moving red set;
vitest and electron were green five times. Every red is either fixed at the
seam that produced it or recorded in the spec as the probe's.

The leak behind the largest red set. hooks_integration._lifecycle_dispatcher
is a process-wide slot init_hooks_system assigns and production never clears,
so it carried the last app-building test's LifecycleDispatcher -- MagicMock
cron service included -- into test_trusted_apps_api's revoke teardown, which
awaited the mock and answered 409 teardown_incomplete (7-9 tests every round,
all green alone). A rootdir conftest floor snapshots and restores both
hooks-integration globals around every test, in the shape of the existing
log-record-factory and queue-listener floors, with its own floor tests.

The other reds. test_darwin_spawn_marker spawns the test interpreter, not
`sleep`: macOS 26 answers KERN_PROCARGS2 argv-only for Apple platform
binaries, so the marker was unreadable (5/5) and the refusal test vacuous.
test_crew_log_session_tree_projection drops the emitter's kept handles after
each test (they held write leases process-wide) and test_eventlog_hooks
asserts the lease DELTA, not an empty global table. test_ci_ipv6_routing runs
its five nested collect-only pytests concurrently under the outer timeout
budget instead of 5 x 60 s in series inside 120 s, each with --basetemp under
tmp_path. test_cron_manual_run_claim_window takes the first wrapper task from
attach_run_task instead of racing _run_claimed_manual for claim.task.

Real host programs reached from unit tests, each pinned at the seam the
product reads and probe-verified to zero spawns: gh (a live GitHub API call
from test_review_driver's post read-back), kiro-cli --version (32 per round
from test_acp_client), ssh -V (three sandbox launcher suites), launchctl and
ps (test_pod_api). The DeepSeek launch golden's fixed /opt/scratch window is
no longer rmtree'd on the host.

Spec: testing-conventions gains the eleventh-pass section, including the two
probe defects that manufactured the largest classes (dir_fd-relative operands
read as checkout writes; the probe's own os.path calls observed by a spied
test) and what was reviewed and deliberately left. writing-tests gains five
checklist items.
@bolichen97
bolichen97 force-pushed the test/hygiene-sweep-20260926 branch from ba24d9e to f5caa04 Compare September 27, 2026 15:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 27, 2026
chenmingwei23 pushed a commit that referenced this pull request Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass (#14424)

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97
bolichen97 merged commit c6290ba into main Sep 27, 2026
138 of 141 checks passed
@bolichen97
bolichen97 deleted the test/hygiene-sweep-20260926 branch September 27, 2026 17:31
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants