Repository navigation
fix(windows): read an exiting process's terminate refusal as its exit - #17027
Conversation
A Windows teardown asked "is this process gone?" with a zero-time look at a signal the kernel raises last. An exit publishes the exit code, then runs the process down (from which point TerminateProcess is refused with ERROR_ACCESS_DENIED), and only then signals the object. A drain member that starts exiting just before the terminate -- a console host leaving with its last client -- was therefore read as a refusal, and the drain raised with the tree's reservation charged, which red the Windows backend shard on unrelated PRs. terminate_process_handle now reads that refusal as an exit when GetExitCodeProcess no longer answers STILL_ACTIVE, or when the object signals within a 250 ms bounded wait (a zero-time look on the event loop). A genuine refusal still raises with the terminate's own error. retire_identity judges the same refusal by the retirement wait that already follows instead of a zero-time look. Two teardown helpers had the same "success reported as failure" shape: windows-process-tree.ps1 left a holder taskkill's exit code 128 in $LASTEXITCODE, which a GitHub pwsh step turns into its verdict, and a cancelled pod start deleted its Task Scheduler wrapper with a bare unlink that a transient sharing violation fails; it now uses the same _unlink_waiting_out_sharing helper as pod down. The on-loop Windows lock test pins its single-shot acquire by attempt count instead of an elapsed-time bound, which a starved runner could fail.
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the Windows backend shard and installer smoke steps pass on this head. That CI run is the only proof that the kernel exit ordering this fix assumes is real. [DESIGN-REVIEWED] 7fd8b57 Description read[DESCRIPTION-READ] 5d2e06c06f53758e638953c73aa1af2de73510eb917c97a509465b317460fcaf That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The new 250 ms refusal wait handles an exit-code-259 case that nobody has reported. The exit-code re-read alone fixes the CI failure. Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: stop Windows teardown (CI drains, installer smoke steps, pod down/up rollback) from reporting failure when a process was already exiting. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 7fd8b57 Description read[DESCRIPTION-READ] c708d70f1329455915bd01ba27ad7bdadf6dc3de8783df03e54e483a1a9c73ae That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
self-added: no
|
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Exact-SHA CI evidence for
The Windows-only tests this PR adds or changes run on the Windows shards: the real-process drain test and the on-loop lock spy are
|
Problem / Motivation
Goal: Windows teardown stops reporting failure for outcomes that are actually success, so Windows CI shards and installer steps no longer go red on a process that was already leaving.
A Windows drain asks the kernel to end each member of a process tree. A member that starts exiting on its own just before that call (a
conhost.exeleaving once its client is killed) getsERROR_ACCESS_DENIEDfromTerminateProcess, the same code a genuine refusal carries. The drain read that as a refusal and raised, which red the Windows backend shard on unrelated PRs:test/test_runtime_cleanup_windows.py::test_native_admission_and_owner_shutdown_refund_after_verified_drain[False]failed withOSError: [Errno 5] TerminateProcess failedin CI run 37251313247 (job 111579613008). The same misreading sits in three more teardown paths: the pod publisher retirement, the installer smoke scripts' process-tree helper (a taskkill exit code 128 becomes the step's verdict), and a cancelledpod uprollback that deletes its Task Scheduler wrapper with a bare unlink.Why it matters
Every Windows CI shard that drains a process tree can go red on a PR that touched nothing related, and the only remedy so far has been a rerun. The installer smoke steps can fail with every assertion passed and no message. A cancelled
pod upcan leave its rollback incomplete over a transient file hold thatpod downalready waits out. Each of these hides real failures behind noise and costs maintainers reruns and triage.Not a goal
TerminateProcessor to the ps1 callers.What changed (motivation → approach → change)
A teardown asked "is this process gone?" with a zero-time look at a signal the kernel raises last. An exit publishes the exit code, then runs the process down (from which point a terminate is refused), and only then signals the object. So a refusal can arrive while the object still looks alive.
platform_compat.pyterminate_process_handleOSErrorGetExitCodeProcessno longer answersSTILL_ACTIVE, or the object signals within_WINDOWS_TERMINATE_REFUSAL_WAIT_MS(250 ms off the event loop; no wait at all on it). Otherwise it still raises with the terminate's own errorpod/_windows_job.pyretire_identityERROR_ACCESS_DENIEDrefusal is judged by the retirement wait that already follows; a publisher still alive after it raises the captured refusal. Any other refusal stands unless the object has already signalledscripts/windows-process-tree.ps1taskkillleaves 128 in$LASTEXITCODE$global:LASTEXITCODEafter each holder kill, as the root kill already doespod/windows.py_rollback_start.cmdwrapper_unlink_waiting_out_sharing, the helperstopalready uses for[WinError 32]The terminate paragraph of
docs/system-specs/common/platform-compat.mdstates this rule.Test hygiene (no product change)
test_platform_compat.py::TestFileLockContention::test_windows_contended_lock_on_event_loop_fails_fastdrops itselapsed < 1.0wall bound for an attempt-count spy. Exactly one non-blocking attempt and no sleep is a stronger pin: it fails on a second attempt however fast the host is, where the time bound passed a short spin and could fail a starved runner that made only the one attempt.Backwards compatibility
Compatible: every path that succeeded before still succeeds, and every genuine refusal still raises the same exception type and message. The outcomes that change are a refusal of a process whose exit has begun (now success) and a held rollback wrapper (now waited out, then the same
PermissionError).terminate_process_handlekeeps its one-argument signature.Two timing deltas, both bounded and fail-closed:
retire_identity: a genuineERROR_ACCESS_DENIEDrefusal of a live publisher waits the caller's full retirementtimeout(stoppassesSTOP_TIMEOUT_SECS, 15 s) before it raises the same refusal; it used to raise at once. That wait is what lets a publisher already leaving retire instead of failingpod down. The only caller,stop, is a CLI path that already blocks in that same wait after a successful terminate._rollback_start: the wrapper delete runs insideruns.cancel_reserved, so a wrapper held by[WinError 32]keeps that run'sclaim_lockfor up to_SCRIPT_UNLINK_TIMEOUT_SECS(10 s). It cannot move after the lock release: the cancellation receipt is consumed under the lock only once every cleanup step has succeeded, and the claim lock covers cancellation through receipt deletion, so deleting the wrapper after the release would drop the receipt first and a failed delete would leave the wrapper with nothing to retry from.claim_lockis non-blocking, so a contender is refused at once rather than stalled, and no supervisor exists yet for a start that never reached/Run.Removes nothing: no key, record, path, function or validator is removed or narrowed.
Tests
Production fixes, each red on the parent and green on the fix:
test/test_platform_compat.py::TestTerminateRefusedOnAnExitingMember(all OSes, a fake kernel32 on a virtual clock): a published exit code settles the refusal with no wait (the shard's interleaving); an exit code of 259 is settled by the signal inside the 250 ms wait; a genuine refusal still raises errno 5 after exactly one bounded wait; on the event loop the look is zero-time, and an object already signalled there still reads as exited; any other error raises at once; an unreadable exit code falls back to the signal; an unwaitable handle still raises. The fake overwrites the last error on every later call, so the raise must carry the terminate's own error.test/test_runtime_cleanup_windows.py::test_a_member_refused_before_its_object_signals_reads_as_exited(Windows): a real sleeper is ended, every wait inside the terminate answers WAIT_TIMEOUT, and the drain still finishes from the real exit code without waiting; its waits on the real exit carry one named lost-run ceiling. The existing live-refusal test keeps proving a genuine refusal raises.test/test_pod_windows_job_retire.py: a publisher refused while exiting retires once its object signals inside the retirement wait; a live refusal still raises the captured refusal after that wait; any other refusal raises without it.test/test_windows_process_tree_script.py(needs pwsh; Windows CI always has PowerShell): runs the real script dot-sourced inside GitHub's ownpwshstep wrapper, with function shims fortaskkill.exe(a real native child exiting 128) andGet-CimInstance, and a holder pid no process can have. A holder that exits on its own and one that outlives the wait both leave the step at exit 0; the gateway path stays at 0 as before.test/test_pod_windows_startup_rollback.py: a wrapper held twice by[WinError 32]is still deleted and the cancellation receipt consumed; a hold past the ceiling and a plain access denial still fail the rollback closed with the receipt kept.Test hygiene, no product change:
test/test_platform_compat.py::TestFileLockContention::test_windows_contended_lock_on_event_loop_fails_fast(Windows): the elapsed-time bound becomes an attempt-count spy: exactly oneLK_NBLCKattempt on the contender and no sleep on the loop thread (a sleep raises at once, so a regression fails by name instead of spinning to the 300 s lock ceiling).Manual verification
Run on Linux (Python 3.12.13, pwsh 7.4.6 on PATH), host load 25-40 on 32 CPUs, through the campaign's test semaphore. Windows-only tests are proven on this PR's exact-SHA Windows lanes.
af1eb612d)-n0test_pod_windows_stop_race.py,test_pod_windows_job.pyand the coverageTestProcessHandlesco-tenants, 10x at-n 4 --dist loadgroupkc_force_plugin):coarse_clock+shuffleandslow_thread+shuffle, seeds 1-3 each;gil_hog+shuffleat a 5 ms switch intervalcoarse_clock+slow_thread+shuffle, seed 7OSError: [Errno 5] TerminateProcess failed(the shard's own error); both ps1 steps exit non-zero; the held wrapper raises[WinError 32]; the retiring publisher raisesmsvcrtthe on-loop acquire slept 0.01s between attemptsThe card's forced condition for each fix is the stub it names: a refusal on an object that is not yet signalled (virtual clock), and a
taskkillthat answers 128. Both are deterministic, so the parent reds every run rather than probably.Also green: 1307 adjacency and guard tests (
test_platform_compat.py,test_platform_compat_coverage.py,test_platform_compat_refactor_facade.py,test_sandbox_refactor_facade.py,test_mirrored_owner_storage.py,test_windows_tree_reap.py,test_windows_cleanup_capacity.py,test_runtime_cleanup_windows.py, thetest_pod_windows*files,test_windows_smoke_evidence.py,test_installer_test_helpers.py),mypy src/kiro_crew, black, isort, flake8, docs-lint, and the comment-history, brand, harness-parity and subprocess-encoding gates.scripts/local-gate.py --dry-runselects 654 backend and 358 frontend related files, over the shared host's cap, so CI runs the broad set; the diff touches nothing underwebsite/.Related Issues
Part of #16536
Fixes #17009
Pattern harvest
Rule candidate: review-prompt
Pattern: a Windows teardown that treats
TerminateProcessERROR_ACCESS_DENIEDas a refusal after only a zero-timeWaitForSingleObject, or that leaves a native exit code in$LASTEXITCODEinside a helper apwshstep dot-sources.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)