Skip to content

fix(acp): report the exit status of a Windows kill, not <not reaped> - #12271

Closed
chenmingwei23 wants to merge 1 commit into
mainfrom
fix/windows-kill-summary-12247
Closed

chenmingwei23 wants to merge 1 commit into
mainfrom
fix/windows-kill-summary-12247

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backend Tests (Windows) (1) fails deterministically on main with four
failures in test/test_acp_runtime.py, every one of them asserting that a kill's
retained summary carries the child's exit status:

AssertionError: assert '[returncode=-15]' in
  'killed (warm mint teardown) [returncode=<not reaped>] stderr_tail: <none>'

The Linux shards pass. The underlying product defect is that on Windows the exit
status of a killed runtime is never reported at all: the summary keeps the
placeholder [returncode=<not reaped>] for good, even after a teardown that
confirmed and reaped the exit.

Why it matters

Two costs, one shared and one per-operator.

Backend Tests (Windows) (1) is a required lane, so this reds PR Readiness on
every open pull request whose base contains the change, whatever that pull
request touches. It is a shared blocker rather than any one author's.

The retained summary also OUTLIVES the log: it rides AcpProcessDied into a
turn's error and a cron's last_error. So a Windows operator asking why a
runtime died was told the status could not be read, on every kill, while the same
kill on POSIX reported the code. That is the field report #12188 set out to
answer, still unanswered on one platform.

What changed (motivation -> approach -> change)

The issue reads the failure as a POSIX detail leaking into a cross-platform
assertion -- -15 is -SIGTERM, which Windows cannot produce -- and suggests
either relaxing the assertions or deciding what a Windows kill should report.
Neither is the cause, and both would have hidden the defect.

_note_reaped_after_kill does not wait for -15 or for any particular value: it
takes whatever the reap read and stringifies it. The -15 in these tests is a
value the tests' own fake process sets, which is platform-neutral. The real
reason it never appears on Windows is that the amendment is never called
there
. _kill_inner has two teardown paths and only one of them amends:

  • POSIX runs the signal ladder and calls _note_reaped_after_kill after its
    reap, just before dropping the handle.
  • Windows runs terminate_windows_asyncio_tree instead -- the tree is pinned by
    handle at spawn and drained from those handles -- and then returns from that
    branch, above the amendment entirely.

The status IS knowable at that point. The drain returns only once every member's
exit is confirmed, and its success path awaits process.wait()
(platform_compat.py), so the handle it hands back carries the root's exit code.

So: call _note_reaped_after_kill(process.returncode) in the Windows branch too,
before the handle is dropped, mirroring the POSIX site. One line, plus the
comment explaining why the status is readable there.

Scope of the fix, path by path:

_kill_inner path Before After
no process nothing owed (label is <no process>) unchanged
Windows, drain confirmed never amended amends from the drained handle
Windows, drain raised not amended; process retained for retry unchanged
POSIX, SIGTERM wait returned amends unchanged
POSIX, escalated to SIGKILL, reap returned amends unchanged
POSIX, both waits timed out silent; status genuinely unread unchanged
child had already exited on its own nothing owed; status read at mark time unchanged

Exactly one cell moves. The unconfirmed-drain path is deliberately left alone: it
did not confirm retirement and keeps the process for maintenance to retry, and
the status is normally unread there anyway (the executor raises before the wait,
or the wait is what timed out), which _note_reaped_after_kill already treats as
nothing owed.

The second half of the change is on the test side. _neuter_kill_side_effects,
the shared helper that keeps kill() away from the host, stubbed the drain with
AsyncMock(return_value=True) -- a stand-in that does not reap, unlike the
function it stands for. Every test reaching the Windows branch therefore measured
a narrower contract than the real code, and the returncode the branch reads was
unknowable for a reason the product does not have. The stub now reaps, which is
what makes the four reported tests platform-neutral: their fake sets the status
inside wait(), and both branches now read it.

Backwards compatibility

Compatible. Nothing is rejected, renamed or removed. The summary's format is
unchanged; on Windows a field that always read <not reaped> now carries the
exit status when one was read, which is the same substitution POSIX has always
made. A Windows kill whose status is genuinely unread still reads <not reaped>.

Tests

Two parametrised tests added to test/test_acp_runtime.py, both forcing the
Windows branch on every platform so this path is guarded by every shard rather
than only by the Windows one:

  • test_the_windows_drain_amends_the_summary_like_the_posix_ladder[False-WARNING]
    and [True-INFO] -- a confirmed drain puts the status in the retained summary,
    removes <not reaped>, keeps the reason, and logs one reaped after kill line
    at the death's own severity. Uses returncode = 1, what TerminateProcess
    reports, never a signal.
  • test_an_unconfirmed_windows_drain_does_not_claim_a_reap -- a drain that
    cannot confirm retirement re-raises, retains the process handle, leaves
    <not reaped> standing and logs no reap line.

The four reported tests are unchanged; they go green through the helper's stub
now reaping as the real drain does.

The load-bearing assumption -- that the drain reaps the root before returning --
is already pinned by test/test_windows_tree_reap.py, which asserts the wait is
awaited on the success path and not awaited when the handle is missing. So if
that reap is ever removed, CI reds there rather than this summary silently going
back to <not reaped>.

Mutation-verified, five mutations:

Mutation Result
remove the Windows amendment call KILLED -- 2 failed, and the failure string reproduces the reported CI assertion verbatim
revert the drain stub to the non-reaping mock KILLED -- 2 failed
also amend on the unconfirmed-drain path SURVIVED -- see below
drop the handle before re-raising on the failure path KILLED -- 1 failed
amend BEFORE the drain instead of after KILLED -- 2 failed, so the placement is pinned, not just the presence

The survivor is reported rather than papered over. On the failure path
process.returncode is None in the scenario the test builds, so the widened
call is silent and unobservable there; the test cannot distinguish it. It is
recorded here because a reviewer should know the "only this path" choice rests on
the argument above, not on a test.

Targeted run, -n0, on 00b76cbb12b4:

pytest -n0 test/test_acp_runtime.py -k "windows_drain or amends_the_summary or
  reap_amendment or no_amendment_when or keeps_the_label_when or
  never_spawned_runtime or labels_the_unreaped"
-> 11 passed

Wider run of every file this change can affect
(test_acp_runtime.py, test_acp_runtime_kill.py, test_runtime_cleanup_windows.py):
406 passed, 17 skipped, 52 failed. All 52 are kas_readiness /
managed_readiness / derived_worker_identity TimeoutErrors, and the same
selection on a pristine worktree at base 4dfbfb04a8be fails with a
byte-identical set of 52 test ids, so they are inherited from the base on this
host and none is in the kill path.

Manual verification

N/A -- unit coverage sufficient. The branch under change is Windows-only and its
two ends are both asserted directly: this repository has no Windows host to drive
by hand, which is why the new tests force the branch rather than relying on the
Windows shard.

Related Issues

Closes #12247

Pattern harvest

Rule candidate: review-prompt

Pattern: a platform-specific branch that returns early skips the shared
bookkeeping performed at the end of its sibling branch, and the miss is visible
only on that platform. The same shape produced this defect twice over -- once in
the product, where the Windows branch returned above the amendment, and once in
the test helper, where a stub omitted a side effect (the reap) that the real
function performs, making the branch's input unknowable for a reason the product
does not have. Worth asking of any early-returning platform branch: which of the
common tail's steps does it owe, and does its test double perform the side
effects the real callee does?

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested review from a team and iamwhatever September 20, 2026 10:16
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the required Backend Tests (Windows) (1) lane is green on this head before merge — it is the failure this fix exists to clear.

[DESIGN-REVIEWED] 26d0820

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 26d0820

Verdict parsed from the review's SHA-scoped output markers for commit 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm the Windows shard itself goes green on this head — every new test forces the branch via monkeypatched IS_WINDOWS, never the real platform.

What this change ships

Inventory (5 items) — 5 justified

Intent: FIX — make a killed Windows runtime report its exit code instead of <not reaped>, un-redding the required Windows lane (base tests at test/test_acp_runtime.py:2293 assert the code and fail there; #12247).

  1. A Windows kill's retained death summary now carries the exit code — justified
  2. Shared test stub for the Windows drain now reaps, as the real drain does (platform_compat.py:4336) — justified
  3. Two new tests force the Windows teardown branch on every shard — justified
  4. test_kill_keeps_the_label_when_no_status_ever_arrives now pins the POSIX branch — undeclared; required by item 2, whose reaping stub made the branch host-dependent
  5. New meta-test pinning the stub's bounded wait — undeclared; guards the documented lost-run (class 6) invariant

Both handle-drop sites now amend (grepped self._process = None: 2, both followed by _note_reaped_after_kill) — no unfixed siblings.

[FIRST-PRINCIPLES-REVIEWED] 26d0820

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 26d0820

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
The Windows teardown branch of _kill_inner drains the tree from handles
pinned at spawn and returns from it, so it never reached the amendment the
POSIX ladder performs after its reap. The drain returns only once the
root's exit is confirmed and reaped, so the status was knowable there --
yet every Windows kill's retained summary kept the pre-signal placeholder
returncode=<not reaped> for good. That summary outlives the log: it rides
AcpProcessDied into a turn's error and a cron's last_error, so the exit
code was lost exactly where an operator reads it.

Amend from the drained handle before it is dropped, mirroring the POSIX
site. The unconfirmed-drain path is left alone: it keeps the process for
maintenance to retry and the status is normally unread there anyway.

The shared kill-test helper stubbed the drain with a mock that did not
reap, unlike the real function, so tests reaching this branch measured a
narrower contract than the code they stand for. The stub now reaps.

Two parametrised tests force the branch on every platform, so the Windows
path is guarded by every shard rather than only by the Windows one.

Closes #12247
@chenmingwei23
chenmingwei23 force-pushed the fix/windows-kill-summary-12247 branch from 00b76cb to 26d0820 Compare September 20, 2026 11:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round 2 dispositions, head 26d08208cd28de12cdd913bbbdd8aa5cdb503ebd

GPT 5.6 BLOCKING, test/test_acp_runtime.py:2049 (security-class, fenced) -- ACCEPTED AND FIXED

Real, reachable, and correctly diagnosed. It is also the single failure the
previous head's own Backend Tests (Windows) (1) run produced, so the finding
and the shard agree. That run went from the reported 4 failures to 1, and the 1
was not an assertion:

worker 'gw3' crashed while running
  'test/test_acp_runtime.py::test_kill_keeps_the_label_when_no_status_ever_arrives'
1 failed, 10249 passed, 500 skipped, 12 xfailed

Mechanism, confirmed against the code: the previous head's
_fake_windows_tree_drain awaited process.wait() UNBOUNDED, while the real
terminate_windows_asyncio_tree reaps under
asyncio.wait_for(..., _WINDOWS_TREE_REAP_TIMEOUT_SECS). So the stub was
faithful about reaping and unfaithful about bounding it. That test replaces
wait() with asyncio.sleep(3600), and the Windows branch has no escape through
_KILL_TERM_TIMEOUT / _KILL_REAP_TIMEOUT -- only the POSIX ladder reads those.
On a Windows shard it therefore wedged the worker deterministically. The previous
head traded four red assertions for one lost run, which is worse.

Fixed in two parts, the prescribed one plus its generalisation:

  1. test_kill_keeps_the_label_when_no_status_ever_arrives now pins
    IS_WINDOWS = False. It is a POSIX-ladder test -- it sets the two ladder
    timeouts, which the Windows branch never reads -- so which branch runs should
    not be left to the host. On Windows a wedged child is the unconfirmed-drain
    case instead, already covered by
    test_an_unconfirmed_windows_drain_does_not_claim_a_reap. This is exactly
    the fix the finding asked for.
  2. _fake_windows_tree_drain now bounds its reap at the real function's own
    _WINDOWS_TREE_REAP_TIMEOUT_SECS and lets the TimeoutError out, which
    _kill_inner catches as a drain that could not confirm. Part 1 alone leaves
    the same trap for the next test that stubs a slow wait(); this makes the
    hang structurally impossible rather than avoided by one test's discipline.
    Pinned by test_the_fake_windows_drain_bounds_its_reap_like_the_real_one,
    which drives the stub with a wait that never returns and requires a
    TimeoutError.

Verification. With the Windows branch forced on for the whole run (a session
plugin setting platform_compat.IS_WINDOWS = True, so the POSIX host takes the
Windows path), the kill-path selection is 12 passed in 1.02s and does not hang.
Two further mutations, both killed:

Mutation Result
drop the IS_WINDOWS = False pin, keep the bound KILLED -- 1 failed with TimeoutError in 7.00s. The bound turns a lost worker into a clean red.
drop the pin AND unbound the stub HUNG, killed at the 100s cap. Reproduces this shard crash exactly.

Added to round 1's five, that is seven mutations, six killed. The one survivor is
still the widening of the amendment onto the unconfirmed-drain path, disclosed in
the description for the same reason.

No override was sought or used.

Design Review (Fable 5) -- PASS, advisory note ADDRESSED

"Confirm the required Backend Tests (Windows) (1) lane goes green on this PR's
own run." Tracked to completion rather than asserted: the previous head's run is
quoted above (4 -> 1, and that 1 is this round's fix). This head's run is the
confirmation and is reported when it lands.

First Principles Review (Fable 5) -- PASS, advisory note ADDRESSED

"Verify the four reported tests on the real Windows CI shard: local proof forced
the branch via monkeypatch, never real IS_WINDOWS." The concern was exactly
right, and the shard is what caught the residual. All four reported tests passed
on the real shard at the previous head; the only failure was the wedge this round
removes. The forced-branch runs remain useful because they guard the Windows path
on every shard, but the shard stays the arbiter.

Opus 4.8 -- no blocking findings

Nothing to disposition.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #12269, which merged as 954fb6918594729637d17aaba08d302cb5c05a8c (now main's tip) and closed #12247 as completed.

That change reached the same root cause independently and is a strict superset of this one. Checked element by element against origin/main:

This PR On main
_note_reaped_after_kill(process.returncode) in _kill_inner's Windows branch, success path only present, same placement and same reasoning
the shared kill double's drain reaps present
that reap bounded at _WINDOWS_TREE_REAP_TIMEOUT_SECS present, off the same constant
test_kill_keeps_the_label_when_no_status_ever_arrives pinned to the POSIX ladder present
a test that the Windows branch amends, forced on every platform present as test_the_windows_branch_amends_the_summary_too
a test that an unconfirmed drain refuses to claim a reap present as test_an_unconfirmed_windows_drain_leaves_the_placeholder
the owning spec documents the amendment present, and this PR did not do it

Verified rather than read off the diff: main's kill-path selection is 10 passed on POSIX, and 10 passed in 1.13s with platform_compat.IS_WINDOWS forced true for the whole run, so the worker hang this PR spent its second round on does not exist there either. Backend Tests (Windows) shards 1 through 8 are all success on #12269's head f4784137c131.

Two things here have no counterpart on main, and neither is a fix worth a PR: the Windows amendment test was parametrised over both expected values to re-assert the reap line's severity, which _note_reaped_after_kill already decides platform-independently and the POSIX tests already cover; and a test pinning that the double's drain bounds its own wait, which guards a test helper whose bound main already has and documents.

Nothing is left over, so there is nothing to carry forward. The branch is left on the remote for the record; no local state remains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backend Tests (Windows) (1): four test_acp_runtime kill-summary tests assert a POSIX signal returncode Windows cannot produce

1 participant