Repository navigation
fix(acp): report the exit status of a Windows kill, not <not reaped> - #12271
chenmingwei23 wants to merge 1 commit into
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the required [DESIGN-REVIEWED] 26d0820 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Confirm the Windows shard itself goes green on this head — every new test forces the branch via monkeypatched What this change shipsInventory (5 items) — 5 justifiedIntent: FIX — make a killed Windows runtime report its exit code instead of
Both handle-drop sites now amend (grepped [FIRST-PRINCIPLES-REVIEWED] 26d0820 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
The Windows teardown branch of _kill_inner drains the tree from handles pinned at spawn and returns from it, so it never reached the amendment the POSIX ladder performs after its reap. The drain returns only once the root's exit is confirmed and reaped, so the status was knowable there -- yet every Windows kill's retained summary kept the pre-signal placeholder returncode=<not reaped> for good. That summary outlives the log: it rides AcpProcessDied into a turn's error and a cron's last_error, so the exit code was lost exactly where an operator reads it. Amend from the drained handle before it is dropped, mirroring the POSIX site. The unconfirmed-drain path is left alone: it keeps the process for maintenance to retry and the status is normally unread there anyway. The shared kill-test helper stubbed the drain with a mock that did not reap, unlike the real function, so tests reaching this branch measured a narrower contract than the code they stand for. The stub now reaps. Two parametrised tests force the branch on every platform, so the Windows path is guarded by every shard rather than only by the Windows one. Closes #12247
00b76cb to
26d0820
Compare
Round 2 dispositions, head
|
| Mutation | Result |
|---|---|
drop the IS_WINDOWS = False pin, keep the bound |
KILLED -- 1 failed with TimeoutError in 7.00s. The bound turns a lost worker into a clean red. |
| drop the pin AND unbound the stub | HUNG, killed at the 100s cap. Reproduces this shard crash exactly. |
Added to round 1's five, that is seven mutations, six killed. The one survivor is
still the widening of the amendment onto the unconfirmed-drain path, disclosed in
the description for the same reason.
No override was sought or used.
Design Review (Fable 5) -- PASS, advisory note ADDRESSED
"Confirm the required Backend Tests (Windows) (1) lane goes green on this PR's
own run." Tracked to completion rather than asserted: the previous head's run is
quoted above (4 -> 1, and that 1 is this round's fix). This head's run is the
confirmation and is reported when it lands.
First Principles Review (Fable 5) -- PASS, advisory note ADDRESSED
"Verify the four reported tests on the real Windows CI shard: local proof forced
the branch via monkeypatch, never real IS_WINDOWS." The concern was exactly
right, and the shard is what caught the residual. All four reported tests passed
on the real shard at the previous head; the only failure was the wedge this round
removes. The forced-branch runs remain useful because they guard the Windows path
on every shard, but the shard stays the arbiter.
Opus 4.8 -- no blocking findings
Nothing to disposition.
|
Closing as superseded by #12269, which merged as That change reached the same root cause independently and is a strict superset of this one. Checked element by element against
Verified rather than read off the diff: main's kill-path selection is 10 passed on POSIX, and 10 passed in 1.13s with Two things here have no counterpart on main, and neither is a fix worth a PR: the Windows amendment test was parametrised over both Nothing is left over, so there is nothing to carry forward. The branch is left on the remote for the record; no local state remains. |
Problem / Motivation
Backend Tests (Windows) (1)fails deterministically onmainwith fourfailures in
test/test_acp_runtime.py, every one of them asserting that a kill'sretained summary carries the child's exit status:
The Linux shards pass. The underlying product defect is that on Windows the exit
status of a killed runtime is never reported at all: the summary keeps the
placeholder
[returncode=<not reaped>]for good, even after a teardown thatconfirmed and reaped the exit.
Why it matters
Two costs, one shared and one per-operator.
Backend Tests (Windows) (1)is a required lane, so this redsPR Readinessonevery open pull request whose base contains the change, whatever that pull
request touches. It is a shared blocker rather than any one author's.
The retained summary also OUTLIVES the log: it rides
AcpProcessDiedinto aturn's error and a cron's
last_error. So a Windows operator asking why aruntime died was told the status could not be read, on every kill, while the same
kill on POSIX reported the code. That is the field report
#12188set out toanswer, still unanswered on one platform.
What changed (motivation -> approach -> change)
The issue reads the failure as a POSIX detail leaking into a cross-platform
assertion --
-15is-SIGTERM, which Windows cannot produce -- and suggestseither relaxing the assertions or deciding what a Windows kill should report.
Neither is the cause, and both would have hidden the defect.
_note_reaped_after_killdoes not wait for-15or for any particular value: ittakes whatever the reap read and stringifies it. The
-15in these tests is avalue the tests' own fake process sets, which is platform-neutral. The real
reason it never appears on Windows is that the amendment is never called
there.
_kill_innerhas two teardown paths and only one of them amends:_note_reaped_after_killafter itsreap, just before dropping the handle.
terminate_windows_asyncio_treeinstead -- the tree is pinned byhandle at spawn and drained from those handles -- and then
returns from thatbranch, above the amendment entirely.
The status IS knowable at that point. The drain returns only once every member's
exit is confirmed, and its success path awaits
process.wait()(
platform_compat.py), so the handle it hands back carries the root's exit code.So: call
_note_reaped_after_kill(process.returncode)in the Windows branch too,before the handle is dropped, mirroring the POSIX site. One line, plus the
comment explaining why the status is readable there.
Scope of the fix, path by path:
_kill_innerpath<no process>)SIGTERMwait returnedSIGKILL, reap returnedExactly one cell moves. The unconfirmed-drain path is deliberately left alone: it
did not confirm retirement and keeps the process for maintenance to retry, and
the status is normally unread there anyway (the executor raises before the wait,
or the wait is what timed out), which
_note_reaped_after_killalready treats asnothing owed.
The second half of the change is on the test side.
_neuter_kill_side_effects,the shared helper that keeps
kill()away from the host, stubbed the drain withAsyncMock(return_value=True)-- a stand-in that does not reap, unlike thefunction it stands for. Every test reaching the Windows branch therefore measured
a narrower contract than the real code, and the returncode the branch reads was
unknowable for a reason the product does not have. The stub now reaps, which is
what makes the four reported tests platform-neutral: their fake sets the status
inside
wait(), and both branches now read it.Backwards compatibility
Compatible. Nothing is rejected, renamed or removed. The summary's format is
unchanged; on Windows a field that always read
<not reaped>now carries theexit status when one was read, which is the same substitution POSIX has always
made. A Windows kill whose status is genuinely unread still reads
<not reaped>.Tests
Two parametrised tests added to
test/test_acp_runtime.py, both forcing theWindows branch on every platform so this path is guarded by every shard rather
than only by the Windows one:
test_the_windows_drain_amends_the_summary_like_the_posix_ladder[False-WARNING]and
[True-INFO]-- a confirmed drain puts the status in the retained summary,removes
<not reaped>, keeps the reason, and logs onereaped after killlineat the death's own severity. Uses
returncode = 1, whatTerminateProcessreports, never a signal.
test_an_unconfirmed_windows_drain_does_not_claim_a_reap-- a drain thatcannot confirm retirement re-raises, retains the process handle, leaves
<not reaped>standing and logs no reap line.The four reported tests are unchanged; they go green through the helper's stub
now reaping as the real drain does.
The load-bearing assumption -- that the drain reaps the root before returning --
is already pinned by
test/test_windows_tree_reap.py, which asserts the wait isawaited on the success path and not awaited when the handle is missing. So if
that reap is ever removed, CI reds there rather than this summary silently going
back to
<not reaped>.Mutation-verified, five mutations:
The survivor is reported rather than papered over. On the failure path
process.returncodeisNonein the scenario the test builds, so the widenedcall is silent and unobservable there; the test cannot distinguish it. It is
recorded here because a reviewer should know the "only this path" choice rests on
the argument above, not on a test.
Targeted run,
-n0, on00b76cbb12b4:Wider run of every file this change can affect
(
test_acp_runtime.py,test_acp_runtime_kill.py,test_runtime_cleanup_windows.py):406 passed, 17 skipped, 52 failed. All 52 are
kas_readiness/managed_readiness/derived_worker_identityTimeoutErrors, and the sameselection on a pristine worktree at base
4dfbfb04a8befails with abyte-identical set of 52 test ids, so they are inherited from the base on this
host and none is in the kill path.
Manual verification
N/A -- unit coverage sufficient. The branch under change is Windows-only and its
two ends are both asserted directly: this repository has no Windows host to drive
by hand, which is why the new tests force the branch rather than relying on the
Windows shard.
Related Issues
Closes #12247
Pattern harvest
Rule candidate: review-prompt
Pattern: a platform-specific branch that
returns early skips the sharedbookkeeping performed at the end of its sibling branch, and the miss is visible
only on that platform. The same shape produced this defect twice over -- once in
the product, where the Windows branch returned above the amendment, and once in
the test helper, where a stub omitted a side effect (the reap) that the real
function performs, making the branch's input unknowable for a reason the product
does not have. Worth asking of any early-returning platform branch: which of the
common tail's steps does it owe, and does its test double perform the side
effects the real callee does?
Checklist