Skip to content

fix(windows): read an exiting process's terminate refusal as its exit - #17027

Merged
chenmingwei23 merged 1 commit into
mainfrom
refactor/df-p1-windows-teardown-races
Oct 5, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
refactor/df-p1-windows-teardown-races

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Windows teardown stops reporting failure for outcomes that are actually success, so Windows CI shards and installer steps no longer go red on a process that was already leaving.

A Windows drain asks the kernel to end each member of a process tree. A member that starts exiting on its own just before that call (a conhost.exe leaving once its client is killed) gets ERROR_ACCESS_DENIED from TerminateProcess, the same code a genuine refusal carries. The drain read that as a refusal and raised, which red the Windows backend shard on unrelated PRs: test/test_runtime_cleanup_windows.py::test_native_admission_and_owner_shutdown_refund_after_verified_drain[False] failed with OSError: [Errno 5] TerminateProcess failed in CI run 37251313247 (job 111579613008). The same misreading sits in three more teardown paths: the pod publisher retirement, the installer smoke scripts' process-tree helper (a taskkill exit code 128 becomes the step's verdict), and a cancelled pod up rollback that deletes its Task Scheduler wrapper with a bare unlink.

Why it matters

Every Windows CI shard that drains a process tree can go red on a PR that touched nothing related, and the only remedy so far has been a rerun. The installer smoke steps can fail with every assertion passed and no message. A cancelled pod up can leave its rollback incomplete over a transient file hold that pod down already waits out. Each of these hides real failures behind noise and costs maintainers reruns and triage.

Not a goal

  • No change to how a drain, retirement or rollback treats a genuine refusal: it still fails closed.
  • No change to the drain's pass budget, the sweep, or the Job-based containment.
  • No new retry, rerun marker, skip or longer timeout in any test.
  • No change to any other caller of TerminateProcess or to the ps1 callers.

What changed (motivation → approach → change)

A teardown asked "is this process gone?" with a zero-time look at a signal the kernel raises last. An exit publishes the exit code, then runs the process down (from which point a terminate is refused), and only then signals the object. So a refusal can arrive while the object still looks alive.

Where Before After
platform_compat.py terminate_process_handle refusal + unsignalled at that instant → OSError refusal reads as an exit when GetExitCodeProcess no longer answers STILL_ACTIVE, or the object signals within _WINDOWS_TERMINATE_REFUSAL_WAIT_MS (250 ms off the event loop; no wait at all on it). Otherwise it still raises with the terminate's own error
pod/_windows_job.py retire_identity refusal + unsignalled at that instant → raise when ending a pod's publisher (the process that owns the running pod), an ERROR_ACCESS_DENIED refusal is judged by the retirement wait that already follows; a publisher still alive after it raises the captured refusal. Any other refusal stands unless the object has already signalled
scripts/windows-process-tree.ps1 holder-loop taskkill leaves 128 in $LASTEXITCODE resets $global:LASTEXITCODE after each holder kill, as the root kill already does
pod/windows.py _rollback_start bare unlink of the .cmd wrapper _unlink_waiting_out_sharing, the helper stop already uses for [WinError 32]

The terminate paragraph of docs/system-specs/common/platform-compat.md states this rule.

Test hygiene (no product change)

test_platform_compat.py::TestFileLockContention::test_windows_contended_lock_on_event_loop_fails_fast drops its elapsed < 1.0 wall bound for an attempt-count spy. Exactly one non-blocking attempt and no sleep is a stronger pin: it fails on a second attempt however fast the host is, where the time bound passed a short spin and could fail a starved runner that made only the one attempt.

Backwards compatibility

Compatible: every path that succeeded before still succeeds, and every genuine refusal still raises the same exception type and message. The outcomes that change are a refusal of a process whose exit has begun (now success) and a held rollback wrapper (now waited out, then the same PermissionError). terminate_process_handle keeps its one-argument signature.

Two timing deltas, both bounded and fail-closed:

  • retire_identity: a genuine ERROR_ACCESS_DENIED refusal of a live publisher waits the caller's full retirement timeout (stop passes STOP_TIMEOUT_SECS, 15 s) before it raises the same refusal; it used to raise at once. That wait is what lets a publisher already leaving retire instead of failing pod down. The only caller, stop, is a CLI path that already blocks in that same wait after a successful terminate.
  • _rollback_start: the wrapper delete runs inside runs.cancel_reserved, so a wrapper held by [WinError 32] keeps that run's claim_lock for up to _SCRIPT_UNLINK_TIMEOUT_SECS (10 s). It cannot move after the lock release: the cancellation receipt is consumed under the lock only once every cleanup step has succeeded, and the claim lock covers cancellation through receipt deletion, so deleting the wrapper after the release would drop the receipt first and a failed delete would leave the wrapper with nothing to retry from. claim_lock is non-blocking, so a contender is refused at once rather than stalled, and no supervisor exists yet for a start that never reached /Run.

Removes nothing: no key, record, path, function or validator is removed or narrowed.

Tests

Production fixes, each red on the parent and green on the fix:

  • test/test_platform_compat.py::TestTerminateRefusedOnAnExitingMember (all OSes, a fake kernel32 on a virtual clock): a published exit code settles the refusal with no wait (the shard's interleaving); an exit code of 259 is settled by the signal inside the 250 ms wait; a genuine refusal still raises errno 5 after exactly one bounded wait; on the event loop the look is zero-time, and an object already signalled there still reads as exited; any other error raises at once; an unreadable exit code falls back to the signal; an unwaitable handle still raises. The fake overwrites the last error on every later call, so the raise must carry the terminate's own error.
  • test/test_runtime_cleanup_windows.py::test_a_member_refused_before_its_object_signals_reads_as_exited (Windows): a real sleeper is ended, every wait inside the terminate answers WAIT_TIMEOUT, and the drain still finishes from the real exit code without waiting; its waits on the real exit carry one named lost-run ceiling. The existing live-refusal test keeps proving a genuine refusal raises.
  • test/test_pod_windows_job_retire.py: a publisher refused while exiting retires once its object signals inside the retirement wait; a live refusal still raises the captured refusal after that wait; any other refusal raises without it.
  • test/test_windows_process_tree_script.py (needs pwsh; Windows CI always has PowerShell): runs the real script dot-sourced inside GitHub's own pwsh step wrapper, with function shims for taskkill.exe (a real native child exiting 128) and Get-CimInstance, and a holder pid no process can have. A holder that exits on its own and one that outlives the wait both leave the step at exit 0; the gateway path stays at 0 as before.
  • test/test_pod_windows_startup_rollback.py: a wrapper held twice by [WinError 32] is still deleted and the cancellation receipt consumed; a hold past the ceiling and a plain access denial still fail the rollback closed with the receipt kept.

Test hygiene, no product change:

  • test/test_platform_compat.py::TestFileLockContention::test_windows_contended_lock_on_event_loop_fails_fast (Windows): the elapsed-time bound becomes an attempt-count spy: exactly one LK_NBLCK attempt on the contender and no sleep on the loop thread (a sleep raises at once, so a regression fails by name instead of spinning to the 300 s lock ceiling).

Manual verification

Run on Linux (Python 3.12.13, pwsh 7.4.6 on PATH), host load 25-40 on 32 CPUs, through the campaign's test semaphore. Windows-only tests are proven on this PR's exact-SHA Windows lanes.

Check Fix Parent (af1eb612d)
The new tests, 20x at -n0 17 passed, 2 skipped (Windows-only) every run; 179 s n/a
The same plus test_pod_windows_stop_race.py, test_pod_windows_job.py and the coverage TestProcessHandles co-tenants, 10x at -n 4 --dist loadgroup 119 passed, 6 skipped every run; 130 s n/a
Forced conditions (kc_force_plugin): coarse_clock+shuffle and slow_thread+shuffle, seeds 1-3 each; gil_hog+shuffle at a 5 ms switch interval all green n/a
coarse_clock+slow_thread+shuffle, seed 7 17 passed 10 failed: both terminate interleavings OSError: [Errno 5] TerminateProcess failed (the shard's own error); both ps1 steps exit non-zero; the held wrapper raises [WinError 32]; the retiring publisher raises
Per-branch mutants of the fix (no exit-code re-read; zero-time wait off the loop; bounded wait on the loop; an on-loop signal ignored; error read after the re-read; retire raising on every refusal; retire falling through on every refusal; retire reporting a timeout instead of the refusal) each one reds at least one test by name n/a
The Windows lock test, emulated on Linux with a fake msvcrt passes; a mutant that polls on the loop reds at once with the on-loop acquire slept 0.01s between attempts n/a

The card's forced condition for each fix is the stub it names: a refusal on an object that is not yet signalled (virtual clock), and a taskkill that answers 128. Both are deterministic, so the parent reds every run rather than probably.

Also green: 1307 adjacency and guard tests (test_platform_compat.py, test_platform_compat_coverage.py, test_platform_compat_refactor_facade.py, test_sandbox_refactor_facade.py, test_mirrored_owner_storage.py, test_windows_tree_reap.py, test_windows_cleanup_capacity.py, test_runtime_cleanup_windows.py, the test_pod_windows* files, test_windows_smoke_evidence.py, test_installer_test_helpers.py), mypy src/kiro_crew, black, isort, flake8, docs-lint, and the comment-history, brand, harness-parity and subprocess-encoding gates. scripts/local-gate.py --dry-run selects 654 backend and 358 frontend related files, over the shared host's cap, so CI runs the broad set; the diff touches nothing under website/.

Related Issues

Part of #16536
Fixes #17009

Pattern harvest

Rule candidate: review-prompt
Pattern: a Windows teardown that treats TerminateProcess ERROR_ACCESS_DENIED as a refusal after only a zero-time WaitForSingleObject, or that leaves a native exit code in $LASTEXITCODE inside a helper a pwsh step dot-sources.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

A Windows teardown asked "is this process gone?" with a zero-time look at
a signal the kernel raises last. An exit publishes the exit code, then
runs the process down (from which point TerminateProcess is refused with
ERROR_ACCESS_DENIED), and only then signals the object. A drain member
that starts exiting just before the terminate -- a console host leaving
with its last client -- was therefore read as a refusal, and the drain
raised with the tree's reservation charged, which red the Windows backend
shard on unrelated PRs.

terminate_process_handle now reads that refusal as an exit when
GetExitCodeProcess no longer answers STILL_ACTIVE, or when the object
signals within a 250 ms bounded wait (a zero-time look on the event
loop). A genuine refusal still raises with the terminate's own error.
retire_identity judges the same refusal by the retirement wait that
already follows instead of a zero-time look.

Two teardown helpers had the same "success reported as failure" shape:
windows-process-tree.ps1 left a holder taskkill's exit code 128 in
$LASTEXITCODE, which a GitHub pwsh step turns into its verdict, and a
cancelled pod start deleted its Task Scheduler wrapper with a bare unlink
that a transient sharing violation fails; it now uses the same
_unlink_waiting_out_sharing helper as pod down.

The on-loop Windows lock test pins its single-shot acquire by attempt
count instead of an elapsed-time bound, which a starved runner could
fail.
@bolichen97
bolichen97 requested a review from a team as a code owner October 5, 2026 13:55
@bolichen97
bolichen97 requested a review from iamwhatever October 5, 2026 13:55
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7fd8b574c5aa14ed342d3fba881b29f7471c6b33 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the Windows backend shard and installer smoke steps pass on this head. That CI run is the only proof that the kernel exit ordering this fix assumes is real.

[DESIGN-REVIEWED] 7fd8b57

Description read

[DESCRIPTION-READ] 5d2e06c06f53758e638953c73aa1af2de73510eb917c97a509465b317460fcaf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 7fd8b574c5aa14ed342d3fba881b29f7471c6b33 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new 250 ms refusal wait handles an exit-code-259 case that nobody has reported. The exit-code re-read alone fixes the CI failure.

Not justified as shipped

  • Item 2 — oversized: the description says the wait "covers a process whose exit code is 259". No report or CI run shows that case. The shard's own regression test asserts waits == [], so the GetExitCodeProcess re-read is what removes the reported defect. Every genuine refusal now also costs 250 ms per member before it raises.
    Subtraction: delete _WINDOWS_TERMINATE_REFUSAL_WAIT_MS and the _on_event_loop() branch, and keep the prior WaitForSingleObject(process_handle, 0) look after the exit-code check.
    Clears when: the bounded wait is reduced to the zero-time look, or a reported refusal of a 259-exiting member is linked.

What this change ships

Inventory (7 items) — 6 justified

Intent: stop Windows teardown (CI drains, installer smoke steps, pod down/up rollback) from reporting failure when a process was already exiting. This is a FIX.

  1. A drain member refused while its exit code is already published now counts as exited — justified
  2. A refusal with exit code still reading 259 waits up to 250 ms off-loop for the signal — oversized
  3. A pod publisher refused with access-denied is judged by the retirement wait that follows — justified
  4. The installer smoke step no longer fails on a holder taskkill's exit code 128 — justified
  5. A cancelled pod up rollback waits out a sharing-violation hold on its wrapper — justified
  6. The lock test swaps its wall-clock bound for an attempt-count spy — rides along
  7. The platform-compat spec paragraph is updated — justified

[FIRST-PRINCIPLES-REVIEWED] 7fd8b57

Description read

[DESCRIPTION-READ] c708d70f1329455915bd01ba27ad7bdadf6dc3de8783df03e54e483a1a9c73ae

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7fd8b574c5aa14ed342d3fba881b29f7471c6b33 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7fd8b57

Verdict parsed from the review's SHA-scoped output markers for commit 7fd8b574c5aa14ed342d3fba881b29f7471c6b33.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7fd8b574c5aa14ed342d3fba881b29f7471c6b33: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no
mechanism: bounded wait for the signal after an ERROR_ACCESS_DENIED refusal whose exit code still reads STILL_ACTIVE

  • span=eee4115ded66 bounded wait judged oversized — rebutted

rebutted: the wait is not speculative scope. It completes the fix for a member shape this drain already supports and tests.
A drain member may exit with code 259, which reads exactly like STILL_ACTIVE: test/test_runtime_cleanup_windows.py::test_a_child_exiting_259_reads_as_exited_and_its_drain_finishes (added by #12005) pins that members can exit with 259 and that the drain still finishes them.
For such a member, the exit-code re-read can never settle a refusal, so only the object's signal can. With only a zero-time look, a 259-exiting member whose exit begins just before the terminate gets the same refusal-before-signal window and raises, which is the reported defect.
Cost is bounded: 250 ms, paid only on a refusal the exit code cannot settle, never on the event loop (a zero-time look there), and well inside the drain's 5 s pass. A genuine refusal still raises with the terminate's own error.
Pinned by TestTerminateRefusedOnAnExitingMember::test_a_member_whose_exit_code_is_259_reads_as_exited_once_it_signals, which is red on the parent.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7fd8b574c5aa14ed342d3fba881b29f7471c6b33 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7fd8b57

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7fd8b574c5aa14ed342d3fba881b29f7471c6b33: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Exact-SHA CI evidence for 7fd8b574c5aa14ed342d3fba881b29f7471c6b33:

Lane Result Run
Backend Tests (3.12, 1-8) and Backend Tests (Windows) (1-8) all 16 shards passed, 0 failures CI
Build Windows Installer (x64) passed Build
macOS suite (platform-tests.yml, dispatched on this branch) passed Platform Tests
E2E Gateway Boot (ubuntu-latest, windows-latest) passed CI
SAST (Semgrep), CodeQL, Internal Content Scan passed Code Review, Internal Content Scan
GPT 6.1, Opus 5.5, Design, First Principles, UX, Security Scope passed; the one First Principles item is answered above —
PR Readiness readiness: passed PR Readiness

The Windows-only tests this PR adds or changes run on the Windows shards: the real-process drain test and the on-loop lock spy are win32-gated, and the PowerShell script tests always find an interpreter there (pwsh, or powershell.exe as the fallback).

green_age.py reports one later main commit (10f96074d, #16978), which touches two apps/builtins files that import code this PR changes. Neither references terminate_process_handle, retire_identity, _rollback_start or the PowerShell helper, and git merge-tree against it is clean.

@chenmingwei23
chenmingwei23 merged commit fc0c70d into main Oct 5, 2026
101 checks passed
@chenmingwei23
chenmingwei23 deleted the refactor/df-p1-windows-teardown-races branch October 5, 2026 14:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pod: Windows pod down raises 'terminate publisher' for a publisher that is already exiting (retire_identity)

2 participants