Skip to content

fix(cron): retry ACP process death on the type, not the message wording - #7396

Closed
SebastianYuSun wants to merge 1 commit into
kirodotdev:mainfrom
SebastianYuSun:fix/cron-acp-death-typed-signal
Closed

SebastianYuSun wants to merge 1 commit into
kirodotdev:mainfrom
SebastianYuSun:fix/cron-acp-death-typed-signal

Conversation

@SebastianYuSun

@SebastianYuSun SebastianYuSun commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A cron job whose ACP child dies is recorded as a hard failure, even though the
failure handler already holds a retry branch built for exactly that case.

The branch looks for the words "not running" or "process exited" in the error
message. The most common death is found as a broken pipe on the next write, and it
is worded "ACP process pipe broken: <cause>". Neither word appears in it, so the
retry is skipped for the signature that produces it most often.

Observed on a real job: last_status: "error", last_error: "ACP process pipe broken: Connection lost", on a job whose previous run finished normally and whose
host was not persistently broken.

Why it matters

A paused cron job never fires again. consecutive_failures auto-pauses a job at
_AUTO_PAUSE_THRESHOLD, so a handful of one-off child deaths durably disables a
healthy job instead of self-healing.

The code reads as if this is handled, which is the worst part. The retry is right
there in the failure handler, so the behaviour looks covered.

The generic transient ladder does not catch it either. AcpError.transient is
None at that raise site and the message carries no throttle or 5xx marker, so
acp_error_is_transient falls back to marker-matching and returns False. Both
ladders miss, and control reaches record_failure().

What changed (motivation → approach → change)

The retry in slack/gateway.py now asks the exception's type, not its wording.

Symptom: a healthy cron job collects hard failures for a death it could recover
from. Root cause: the predicate reads str(exc). AcpProcessDied is raised at
every site that discovers a dead child, is an AcpError subclass, and is already
imported at the decision site. It was simply not consulted.

Change: AcpProcessDied carries a resubmit_safe verdict that the raising layer
sets, and the cron branch in slack/gateway.py reads that one attribute. This
mirrors AcpError.transient: the layer that raises knows the classification, so it
states it instead of exporting a class hierarchy for callers to re-derive. Reading
an attribute also adds no ACP import, which
scripts/check_agent_sdk_boundary.py refuses.

The verdict is about when the death was discovered, not which subsystem
noticed. With a turn in flight a tool may already have completed its side effects,
so resubmitting the prompt can repeat a mutation.

The verdict is three-valued, and the third value is the point.

value meaning who sets it
True discovered with no turn in flight; resubmitting cannot repeat a tool 2 sites, listed below
False a turn may have been in flight stated explicitly, never inferred
None (default) not classified — the site has expressed no opinion every other site

None and False must not be the same value. A two-valued flag makes silence
indistinguishable from a judgement, so an unclassified death reads as "unsafe" and a
consumer that refuses on False drops behaviour it would otherwise keep. That
population is not hypothetical: session_provider._translate_dead rebuilds every
shared-runtime death as AcpProcessDied(str(exc)), so the five runtime.py
"process not running" raises arrive unclassified. Under a refusing default they
read as unsafe and lose the retry their wording earns at the fallback — a fix for one
death signature that deletes the retry for another.

So None routes to the question the handler asked before this attribute existed,
rather than answering it. This is why AcpError.transient is also None-by-default:
an unclassified error falls back to the marker ladder instead of being declared
non-transient by omission. The precedent is in this repo, and matching it is what
makes the fallback arm's continued existence coherent rather than vestigial.

The two sites that claim safety, each one where the turn provably had not started:

site turn in flight resubmit_safe
client.py _send_prompt no, this write starts the turn True
session_provider.py runtime not alive no, no conversation yet True

One site must state False explicitly rather than lean on the default.
"Process exited during prompt" is a death that happened with a turn in flight,
and its wording contains "process exited", so leaving it unclassified would route
it to the wording fallback and retry it. That is the one regression a three-valued
flag can introduce, so it is enforced mechanically rather than by review: a test
walks the AST of acp/client.py and fails if any AcpProcessDied construction whose
message literals match the fallback's substrings omits the keyword. It is an AST walk
and not a regex because a regex also matches those phrases where they appear in
prose — including in the docstring that explains the rule.

The claim is about delivery only, so a re-prompt withdraws it. A write can
establish that its own request never reached the child. It cannot establish that
nothing ran before it, and two callers re-prompt a session that has already done
work. Each withdraws the claim from what only it can see:

caller what it sees withdraws when
src/kiro_crew/llm_helpers.py stream_and_collect tool activity in this call any tool has fired
slack/gateway.py cron post-token resume this write continues a turn the continuation prompt

stream_and_collect reuses the sticky flag its own model-fallback chain already
refuses to replay past, for the same reason: a tool can complete an external
mutation before any text streams. Without either withdrawal, a resubmit-safe death on
a re-prompt sends the cron retry back through the original message on a fresh
session and the completed tool runs twice.

Which deaths the cron retry accepts, before and after:

death the handler sees Before After
pipe broken on the prompt write 🟨 hard failure 🟩 retried
Runtime is not alive, no conversation yet 🟨 hard failure 🟩 retried
typed Process exited during prompt 🟨 retried 🟩 hard failure
pipe broken answering a mid-turn child request 🟦 hard failure 🟦 hard failure
plain AcpError("ACP process exited (code=…)") 🟦 retried 🟦 retried
plain AcpError("ACP process not running") 🟦 retried 🟦 retried
any other AcpError 🟦 hard failure 🟦 hard failure

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

The two deaths that could not be retried now can be, and the one death that could
repeat a finished tool call no longer is. Everything else answers the same as it
did.

Why a failure at _send_prompt implies nothing ran. The guarded region is
write() and drain() together, and the claim holds at either. write() failing
means no byte was queued for the child. drain() failing means bytes are still
queued, and the wire is newline-delimited JSON-RPC, so the child never got a whole
session/prompt line and cannot run a tool for a request it has not finished
reading. A child that read the whole payload drains clean even if it dies right
after. A child that read a prefix does raise here, and a prefix is not a request.

The claim sits at _send_prompt, not at the transport write, because a transport
write cannot know turn state. _send_request also carries session/steer and
commands/execute, and _send_response / _send_error exist only to answer
requests the child raises mid-turn. Tagging the shared write safe would mislabel
most of its callers: a cron turn finishes tool 1, the agent asks permission for tool
2, the child dies, and the resubmitted prompt runs tool 1 again.

The wording arm stays, scoped to plain AcpError. Five raise sites spell process
death as a plain AcpError, so dropping the arm would silently stop retrying all
five. Scoping matters the other way too: AcpProcessDied("Process exited during prompt ...") contains process exited, so an unscoped arm out-votes that death's
own refusing verdict. Adding "pipe broken" to the substring list instead would fix
this one case and leave the decision coupled to wording, which is the defect class.

Which backend the two directions reach, measured. The default
agent.acp_backend is served by the shared AcpRuntime through
AcpSessionProvider, which translates every runtime death to
AcpProcessDied(str(exc)) carrying no verdict. AcpSessionHandle does not route
through AcpClient._send_prompt, so the safe claim is not reachable there. Driving
both predicates over the shapes that path raises:

death the handler sees Before After
runtime process not running (5 raise sites) 🟨 retried 🟨 retried
runtime pipe broken: … 🟦 hard failure 🟦 hard failure
runtime runtime is dead 🟦 hard failure 🟦 hard failure

So on that backend this change costs nothing and adds nothing. All three shapes
arrive unclassified, None routes them to the wording fallback, and the fallback
answers exactly as it did before. That is the whole reason the default is None: an
earlier revision of this PR defaulted to False, which turned row one into a hard
failure and silently deleted a retry on the default backend while fixing a different
one. Measured both ways on the same death — AcpProcessDied("process not running")
retries under None and does not under False.

What this backend does not gain is a retry for its pipe broken deaths, because
the fallback it lands on does not match that wording. Giving it one means classifying
the runtime's own prompt-initiating write the way AcpClient._send_prompt is
classified here. That is a behaviour addition on a path this PR does not otherwise
touch, and it needs the delivery argument re-verified against the runtime's transport
rather than AcpClient's, so it is left out deliberately rather than overlooked.

Known residuals, deliberately out of scope. Three resubmit decisions still run
without the verdict, and the comment at the slack/gateway.py branch says so:

  1. The five plain-AcpError sites carry no verdict, so wording decides them, and
    four are reachable with a turn in flight: the EOF read raises "ACP process exited (code=...)" from the prompt loop, and the "not running" checks guard the writes
    answering a mid-turn child request. A death there is still retried — exactly as
    before this change. Typing those sites also changes the answer for the pre-turn
    callers sharing the same message, so it is a behaviour decision rather than a
    relabelling.
  2. dashboard/chat_runner.py resubmits on AcpProcessDied without reading it.
  3. task_executor.py does the same.

All three are pre-existing, in files this change does not touch, and the right
non-resubmit behaviour differs for a foreground turn versus a cron wake.

Residual 1 is also why the substring arm stays rather than being retired in this
diff. It now serves two populations: those five plain-AcpError sites, and every
shared-runtime death arriving unclassified through _translate_dead. It cannot
out-vote an explicit verdict, because the verdict is consulted first and only a None
falls through to it. The comment at the branch names both populations so its remaining
reach is not read as enforced. Retiring it means classifying both populations at their
raise sites, which is a larger behaviour change than this fix.

Tests

test/test_cron_acp_retry.py. Reverting src/ and re-running the file fails,
naming the site, so the tests carry the fix rather than describe it.

Load-bearing:

  • test_acp_pipe_broken_triggers_retry — the real pipe broken wording resets the
    session and retries once. Returning rather than raising is also what keeps the run
    off the auto-pause ladder: CronService._execute calls record_failure() only in
    its except arm, and record_success() zeroes consecutive_failures when the
    callback returns.
  • test_mid_prompt_death_is_not_retried_despite_legacy_wording — the typed
    mid-prompt death is refused even though its wording matches the legacy substring.
    Asserts the stream is entered exactly once.
  • test_only_the_audited_sites_claim_resubmit_safety — exactly 1 + 1 + 0 sites
    claim resubmit-safety across client.py, session_provider.py,
    session_handle.py. Pins the opt-in direction, because the in-flight set is not
    reliably enumerable and a new opt-in is what must justify itself.
  • test_resubmit_safe_states_the_verdict_without_exporting_the_hierarchy — the
    default is None, so a death nobody classified carries no verdict rather than a
    refusal.
  • test_an_unclassified_death_keeps_the_retry_it_had_before_the_verdict_existed —
    raises AcpProcessDied("process not running") with no keyword, byte-for-byte what
    _translate_dead builds, and asserts both that .resubmit_safe is None and that the
    session is still reset and the turn retried. This is the test that fails under a
    two-valued flag, which is how the earlier revision's regression was caught.
  • test_every_wording_matching_death_states_its_verdict_explicitly — walks the AST of
    acp/client.py and fails if any AcpProcessDied construction whose string literals
    contain "not running" or "process exited" omits the resubmit_safe keyword.
    Deleting the keyword at the mid-prompt site fails this test and nothing else, so the
    one way a tri-state can silently grant a retry is mechanically closed.
  • test_resubmit_safe_death_retries_regardless_of_wording[something nobody predicted] — a wording sharing no substring with the legacy guard still retries,
    pinning the decision to the classification rather than to today's vocabulary.
  • test_pipe_broken_retries_only_once — the _acp_retried marker bounds the typed
    arm exactly as it bounds the substring arm.
  • test_a_fired_tool_withdraws_the_claim_before_it_reaches_the_caller — a death
    raised after a tool fired in the same stream_and_collect call reaches the caller
    with the claim withdrawn.
  • test_a_death_on_the_continuation_prompt_is_not_retried — the cron resume's
    continuation prompt dies claiming safety, and the callback is not replayed. Asserts
    the stream is entered exactly twice.

Deleting either withdrawal fails exactly its own test, with the replay named in the
assertion message.

Controls. These pass on fixed and unfixed source alike, and each exists because
without it the fix could silently become a no-op or over-reach:

  • test_resubmit_safe_death_retries_regardless_of_wording[a wording that happens to say process exited] — a safe death retries because it is classified safe, not
    because its wording matched.
  • test_plain_acp_error_still_matches_the_legacy_wording_arm — the five plain
    AcpError death sites are not AcpProcessDied, so scoping the arm to plain errors
    leaves them matched. Without this the scoping could narrow the guard past the
    defect it repairs. Its docstring records that this pins the arm's reach and does
    not claim that reach is correct.
  • test_non_process_death_acp_error_still_not_retried — widening to the type did not
    widen to every AcpError.
  • TestAcpProcessDiedRaiseSites (2 tests) — anchor the wording at the real
    _send_request raise site and the transient=None classification, so a rewording
    cannot quietly invalidate the reason this fix exists.

The pre-existing tests in the file are unchanged and still pass.

Manual verification

N/A — unit coverage sufficient. The tests drive the real
GatewayOrchestrator._cron_callback and the real AcpClient._send_request (with a
real BrokenPipeError) rather than a stand-in, so the wording and the retry
decision are both measured rather than assumed.

The predicate is additionally measured directly against every death shape the code
raises, which is what the Before/After table above reports: of seven shapes, three
change and four answer as they did.

The per-site classification is pinned at source level because reaching those sites
needs a child that dies mid-turn. Two assertions run in opposite directions, and both
are needed:

  • which sites pass resubmit_safe=True — the opt-ins are few and checkable, and a new
    opt-in is what must justify itself.
  • which sites pass any verdict, among those whose wording the fallback matches —
    because with a None default, silence at such a site is not a refusal but a retry.

Neither assertion tries to enumerate the in-flight set. That set is not locally
checkable, and asserting it complete would restate the assumption the design exists to
avoid relying on.

Screenshots / video

N/A — no user-visible UI change. The diff touches the cron failure handler, the ACP
exception classification, and tests; no panel, component, layout or theme surface is
involved.

Related Issues

Fixes #7395

Pattern harvest

Rule candidate: review-prompt
Pattern: a retry or classification decision string-matches an exception message when
a typed exception carrying the same meaning is already in scope at the decision
site. The message is presentation; the type is the contract. This generalizes — the
same shape appears wherever recovery logic greps str(exc) instead of testing
isinstance, and each instance fails silently the first time somebody rewords a
message.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@SebastianYuSun
SebastianYuSun requested review from a team and hoang-phan98 August 31, 2026 23:59
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed d616eafee7182c7a9302a85111451302ae40d66b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d616eaf

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of d616eafee7182c7a9302a85111451302ae40d66b via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify no third call site re-prompts an already-worked session — an inherited resubmit_safe=True there replays a completed tool; only two withdrawals exist.

[DESIGN-REVIEWED] d616eaf

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed d616eafee7182c7a9302a85111451302ae40d66b via the fork AI-review pipeline; updated in place on each push.

Review details

I've verified the load-bearing claims against the actual source:

  • AcpError.__init__ (client.py:2306) accepts *args, transient, code, so the new AcpProcessDied.__init__'s **kw passthrough is safe for existing positional-message call sites.
  • The only AcpProcessDied raised inside _send_request are the pipe-broken ones (client.py:8159/8173/8192, default None), so _send_prompt's unconditional exc.resubmit_safe = True retag never clobbers an explicit False.
  • The "Process exited during prompt" (resubmit_safe=False) raise lives in the prompt loop, not _send_request, so it is not reachable by _send_prompt's except.
  • The gateway ternary resolves all three verdict states correctly (True→retry, False→no-retry, None→legacy wording arm still scoped to isinstance(exc, AcpError)).

The candidate list contained no candidates, and my independent falsification found no grounded defect in the changed lines.

No findings.

[OPUS-REVIEWED] d616eaf

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of d616eafee7182c7a9302a85111451302ae40d66b via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the framing claim on the real transport: a child that read the full prompt line never fails _send_prompt's drain — the resubmit-safe verdict rests on it.

Not justified as shipped

  • Item 3 — rides along: declared ("the one death that could repeat a finished tool call no longer is"); a deliberate retry removal beyond the reported defect, with the replay harm named and the raise site (client.py:8773, inside the prompt read loop) confirming it is mid-turn.

What this change ships

Inventory (7 items) — 6 justified

Intent: FIX — a cron job whose ACP child dies on the prompt write is hard-failed toward auto-pause because the retry predicate matches message wording that death doesn't carry.

  1. Cron jobs killed by a "pipe broken" child death on the prompt write now retry once instead of hard-failing — justified
  2. Cron jobs hitting a dead warm worker before any conversation starts now retry — justified
  3. A mid-prompt "Process exited during prompt" death is no longer retried — rides along (declared; prevents replaying a completed tool's side effects)
  4. Process-death exceptions carry a tri-state resubmit_safe verdict set at the raise site — justified
  5. A call that already fired a tool withdraws the safety claim before it reaches the cron retry — justified
  6. The cron continuation prompt withdraws an inherited safety claim — justified
  7. AST-walk tests pin which sites may claim safety and force wording-matching raises to state a verdict — justified

Verified against base: resubmit_safe is new (0 prior hits); the substring arm at gateway.py:5601-5607 matches the before-state; _translate_dead rebuilds deaths as AcpProcessDied(str(exc)) (session_provider.py:401); the five untyped AcpError death sites (client.py:8150-8250) and five runtime.py "process not running" raises both count as claimed; AcpError.transient is None-default (client.py:2307-2310); gateway.py:54 already imports AcpProcessDied, so no new boundary import; residual mid-turn wording retries (session_handle._died, three call sites) are declared in-code as deferred.

[FIRST-PRINCIPLES-REVIEWED] d616eaf

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 1, 2026
@SebastianYuSun
SebastianYuSun force-pushed the fix/cron-acp-death-typed-signal branch from 92c3b5f to 2621b28 Compare September 2, 2026 20:55
@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

Rebased onto a030091b4; head is now 2621b28a. This head carries the fix for the blocking finding.

The finding was correct. isinstance(exc, AcpProcessDied) alone did admit AcpToolStalled, which client.py raises after a tool has already been dispatched -- so a mutating tool that completed and then stalled would have been retried and could have run twice.

The fix is not the prescribed revert. Reverting the type arm would also drop the three pipe-broken sites this PR exists to cover, and it would put the retry decision back on exception wording, which is the defect class itself. Instead the guard now excludes the one subclass that is unsafe to retry:

(isinstance(exc, AcpProcessDied) and not isinstance(exc, AcpToolStalled))

AcpToolStalled was made a subclass of AcpProcessDied for this, so the exclusion is a type test rather than a message test, and any future stall-shaped death inherits the exclusion automatically instead of needing a new string added to a list.

The legacy substring arm is still there for the five plain-AcpError death sites (client.py:3822, :3837, :3856, :3876, :3922), so the two pre-existing retry cases are unchanged.

Tests: test/test_cron_acp_retry.py is +299, covering the pipe-broken retry, an arbitrary-wording AcpProcessDied (no legacy substring present), the two legacy cases still retrying, no auto-pause accounting on a successful retry, and a stalled-tool case asserting stream_and_collect is called exactly once so the double-mutation path stays closed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 2, 2026
@SebastianYuSun
SebastianYuSun force-pushed the fix/cron-acp-death-typed-signal branch from 2621b28 to 8125735 Compare September 2, 2026 22:22
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 3, 2026
@SebastianYuSun
SebastianYuSun force-pushed the fix/cron-acp-death-typed-signal branch from 8125735 to 20c259a Compare September 3, 2026 01:51
@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

Round 4 on 20c259a2e. All three lanes addressed; the blocking finding was correct on substance and is now closed at cause rather than by revert.

GPT 5.6 (blocking) — closed

The finding is right: resubmit_safe defaulted True on process deaths raised with a turn in flight, so a completed tool mutation could be replayed. Opus 4.8 independently argued the hazard is pre-existing (the pre-PR arm already retried "process exited (code=...)"), and that is true — but it does not excuse it, because my own docstring asserted those deaths "never reached a running child", which is false.

Enumerating every raise site rather than reasoning from the taxonomy:

raise site turn in flight? resubmit_safe
client.py transport writes, pipe broken (×3) no — write never reached a running child True
client.py exited during prompt yes False
client.py tool stalled yes — a tool was dispatched False
session_handle.py died awaiting response yes False
session_handle.py died during prompt yes False

The last two are sites First-Principles named that I had missed, and died awaiting response neither review named — I found it by enumerating grep -rn "raise AcpProcessDied" src/ instead of trusting the docstring's claim.

The invariant is now about when the death was discovered, not which subsystem noticed. Only transport-write sites are retryable, which is exactly the class this PR exists to cover.

Not the prescribed revert, for the third round: reverting the type arm drops the three pipe-broken sites this PR is for, puts the decision back on message wording (the defect class), and leaves the mid-turn hazard untouched at all four sites.

First Principles (advisory) — subtraction taken

AcpToolStalled is deleted. It had one construction site and one distinguishing consumer, so resubmit_safe=False on the raised AcpProcessDied instance does the same work with strictly less surface. grep -rn AcpToolStalled src/ is now empty and every except AcpProcessDied handler is unchanged.

The undeclared-surface point is fixed: the description now lists all four items (predicate, attribute + kwarg, two client.py raise sites, two session_handle.py raise sites) and carries the table above. It previously said "one predicate in the cron failure handler", which was wrong.

The false docstring premise is rewritten to state the real invariant, and records that the earlier wording was false so it is not reintroduced.

Deferred, with reasons

resubmit_safe is read at 1 of 3 resubmit sites — dashboard/chat_runner.py and task_executor.py also resubmit on AcpProcessDied without consulting it (grep -rn "except AcpProcessDied" src/ → 4 handlers, 2 resubmit). Left out on purpose: both are pre-existing, in files this PR does not touch, and the correct non-resubmit branch differs for a foreground turn with a user waiting versus a cron wake. That is a design decision, not a mechanical port, and bundling it would put an unmeasured behaviour change for two subsystems into a PR whose evidence covers the cron path. The classification they would need is now correct and available.

Tests

test_every_in_flight_raise_site_declares_itself_resubmit_unsafe pins all four in-flight sites. Load-bearing: removing the kwarg at the session_handle.py mid-prompt site fails it with the site named. A companion control pins that the pipe-broken sites stay True — without it the fix could silently become a no-op.

903 passed across the cron, ACP-client, ACP-error-surface and slack-gateway suites. All local gates green, including the repo's own 14 bare-invocation script gates (check_agent_sdk_boundary.py among them — an earlier round of this PR was red on exactly that, for adding an ACP import; the attribute approach adds zero boundary edges).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 3, 2026
@SebastianYuSun
SebastianYuSun force-pushed the fix/cron-acp-death-typed-signal branch from 20c259a to cb85ad9 Compare September 3, 2026 03:20
@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

Round 5 — all three lanes named the same defect, and it is real

Opus 4.8 (blocking), Design and First Principles independently converged on one
finding, so it got checked first rather than triaged by verdict severity. It
reproduces, and it is my own previous fix being a no-op at one site.

Measured before changing anything. The shipped predicate, driven with every
death shape the code actually raises:

BUG  retried=True  want=False  client.py exited during prompt
OK   retried=False want=False  client.py tool stalled
OK   retried=False want=False  session_handle awaiting response
OK   retried=False want=False  session_handle during prompt
OK   retried=True  want=True   client.py pipe broken
OK   retried=True  want=True   legacy plain AcpError "not running"
OK   retried=True  want=True   legacy plain AcpError "process exited"

1 site(s) contradict the declared contract

"Process exited during prompt (exit code N)".lower() contains
"process exited". So the retained substring arm matched and out-voted that
death's own resubmit_safe=False. The flag was set, declared in the description,
and covered by a test — and unreachable at that site. Exactly as described.

Fix, at cause. The substring arm is now scoped so a typed death is decided
solely by its flag; wording can no longer resurrect a refusal. The five plain
AcpError death sites still match it, which a new control pins.

The default is inverted, per @Design's argument

Design and First Principles both showed the audit was incomplete —
session_provider.py:262/:314 and session_handle.py:1494, none of which I had
classified, all reachable with a turn in flight. Round 4 had already missed
session_handle.py:1911.

That is two consecutive rounds of an incomplete enumeration, and it is the
argument for inverting rather than extending: the kwarg cost is identical (four
sites either way), but a fail-closed default does not depend on my list being
complete. resubmit_safe now defaults to False; only four sites claim True,
each one where the turn provably never started.

claims True why no tool can have run
client.py transport writes ×3 the write never reached the child
session_provider.py runtime not alive no conversation started yet
(7 other construction sites) default — refuses

The three sites you named are now correct without being listed, which is the point.

So the test changed direction too. Asserting "every in-flight site opts out"
restates the assumption that failed twice. It now pins the enumerable converse —
exactly 3 + 1 + 0 sites claim True per module — so a new opt-in is what has to
justify itself.

Subtractions taken

  • resubmit_safe=False kwargs deleted at all four sites; redundant against a
    refusing default. session_handle.py leaves the diff entirely.
  • Review-round narration removed from the docstrings per AGENTS.md's comments
    rule — including the test-class docstring's used to, which I had missed when
    fixing the same pattern in gateway.py earlier.

On the one point where the lanes disagreed

An earlier round had Opus calling the hazard pre-existing while another lane called
it introduced. Both were tested rather than choosing the convenient one, and both
hold: the pre-PR substring arm already retried "process exited (code=...)", and
the description asserted those deaths never reached a running child, which is false.
Pre-existing does not excuse shipping a live hazard at sites the description
declared safe.

Verification

test_mid_prompt_death_is_not_retried_despite_legacy_wording fails on the previous
published head, naming the site. Two controls guard against the fix becoming a
no-op or over-reaching: a safe death must retry because it is classified safe, not
because its wording matched; and plain AcpError deaths must still match the legacy
arm, or the scoping would have narrowed the guard past the defect it repairs.

Local gates on this head: black (repo wrapper), isort, flake8, mypy over
src/kiro_crew/ all clean; scrub-lint stage 1 clean; all 14 repo script gates
derived from ci.yml pass.

One diff-scope note: formatting the touched test file graduated it out of
.github/black-baseline.txt, and the repo's own gate asks for the entry to be
pruned, so that one-line deletion is in the diff.

The chat_runner.py / task_executor.py residual stays out of scope and is
declared in the description — though with the default now refusing, those handlers
inherit a safe verdict on any unclassified death instead of a permissive one.

@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #8215 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #8215: KEEP. Complementary verdicts for different faults; neither covers the other. The shared insertion window in client.py plus the shared session_provider.py touch means the later of the two may need a trivial textual rebase, not a design change. Files: src/kiro_crew/acp/client.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@bolichen97
bolichen97 force-pushed the fix/cron-acp-death-typed-signal branch from cb85ad9 to 79ef804 Compare September 8, 2026 13:15
@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

First Principles — confirming the trade you asked about

Scoping the substring arm removes base's retry of mid-prompt "Process exited during prompt" deaths — cron jobs that self-healed those now take a hard failure; confirm that trade.

Confirmed, and taken deliberately. Here is what the trade actually costs and buys, measured against the seven death shapes the code raises.

What is given up. One shape: AcpProcessDied("Process exited during prompt ...").
Base retried it because its wording contains process exited. That retry resets the
session and resubmits the prompt, and the death is discovered with a turn in flight, so
a tool the turn already completed can run a second time. The job did self-heal — by
possibly repeating a mutation, silently, with nothing in the record saying it happened.

What is bought. Two shapes gain a retry, and one of them is the common case:

shape base head
pipe broken on the prompt write hard failure retried
Runtime is not alive, no conversation yet hard failure retried
typed Process exited during prompt retried hard failure

The first row is the signature that produces most ACP deaths, which is the whole reason
this change exists. So the practical direction is fewer hard failures, not more, even
though one shape moves the other way.

Why the asymmetry is the right way round. A repeated mutation is silent and
unbounded: nothing reports it and the blast radius is whatever the tool did. A hard
failure is visible and bounded: it increments consecutive_failures, and it takes
_AUTO_PAUSE_THRESHOLD of them in a row to pause the job, with the error recorded each
time. Given a choice between an unobserved wrong result and an observed stop, the stop
is the one an operator can act on.

And it is not a silent removal. The refusal is the classification's own answer
rather than a side effect of scoping: that death carries resubmit_safe=False, which is
the refusing default, and test_mid_prompt_death_is_not_retried_despite_legacy_wording
asserts the stream is entered exactly once. A future decision to retry it has to opt in
at the raise site and say why the turn had not started.


A second trade you did not ask about, because I had not measured it

Checking the first one properly surfaced a larger one, on the default backend, and
it needs your ruling rather than my judgement.

agent.acp_backend defaults to the shared-runtime path: AcpSessionProvider over
AcpRuntime, not AcpClient. Two things follow, both measured by driving the base and
head predicates over the shapes that path actually raises:

death the cron handler sees base head
AcpProcessDied("process not running") — AcpRuntimeDead translated, 5 raise sites retried hard failure
AcpProcessDied("pipe broken: …") — same translation hard failure hard failure
AcpProcessDied("runtime is dead") hard failure hard failure
AcpProcessDied("ACP process pipe broken: …") from _send_prompt hard failure retried

Only the last row is served by AcpClient. AcpSessionHandle never routes through
AcpClient._send_prompt, so the safe claim is unreachable on the shared runtime, and
AcpSessionProvider._translate_dead re-raises every runtime death as
AcpProcessDied(str(exc)) with no verdict. So on the default backend this change
costs one retry and adds none.
That is a lost recovery rather than a repeated
mutation, so it fails in the safe direction — but it is the opposite of what the change
is for, and the description now says so rather than leaving it to be discovered.

The root cause is a decision two earlier rounds took deliberately. After two
consecutive incomplete enumerations of the in-flight raise sites, resubmit_safe was
made a two-state field whose default refuses. That collapses "nobody classified this
death"
and "this death is classified unsafe" into the same value, so any wrapper that
re-raises without a verdict silently drops the retry the wording used to give it.
_translate_dead is exactly such a wrapper.

Your call, not mine. Three shapes, and I do not think a contributor should pick:

  1. Make the field three-state — None means unclassified and falls through to the
    wording arm, which restores base behaviour for translated deaths. This needs the
    explicit resubmit_safe=False kwargs back at the genuinely in-flight sites, which is
    the enumeration the fail-closed default was chosen to stop depending on.
  2. Classify at the runtime raise sites, so the shared-runtime path answers with a
    verdict instead of inheriting the default. That is acp/runtime.py, which this change
    does not touch, and it has the same shared-write problem _send_request has.
  3. Ship as is and open the shared-runtime path as its own issue, accepting one lost
    retry shape on the default backend in exchange for closing the mid-turn replay.

I have not implemented any of them. Say which you want and I will do it here, or split it
out if you would rather this change stayed at the size you have already reviewed.

@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

The comment-history red is gone, and not because of anything in this diff

@bolichen97 — closing the loop on the Backend Lint & Type Check (3.12) failure I raised
twice, so nobody spends the one-liner on it.

It is fixed on main, at the mechanism. The gate no longer keeps a baseline at all.
scripts/check_comment_history.py on current main says so in its own docstring:

There is deliberately NO baseline file. A per-file count ratchet needs one shared JSON
that every cleanup PR must lower in the same change, which makes that file a
merge-conflict hotspot for as long as any cleanup is in flight anywhere in the tree.
Judging added lines needs no shared state at all.

It now judges only the lines a change adds, and it runs from fast-gate.yml rather than
from the Backend Lint & Type Check job whose step name still said (gate, baselined).
So the slack/gateway.py count of 69 against a recorded 67 cannot redden a pull request
any more, and neither of the two moves I asked you to choose between is needed: no
rewording of gateway.py:1825 / :11856, and no baseline bump.

Measured on this branch rebased onto main at 96ad35562:

comment-history gate scope: 96ad3556289fe5d5d3c44fbbe9f8b1c4d1b793ed..working tree (4 changed file(s))
comment-history gate passed: no history narration on the lines this change adds.

That was the only genuinely failing check on the previous head. The rest of the red was
code-review-7396 concurrency cancellations, and PR Readiness reporting the review
concerns as action required.

On your rebase. It was clean and content-identical, as you said. It landed on the head
whose defect three lanes had blocked, because my fix for that defect was local and
unpushed at the time — my mistake, not the rebase's. This head carries that fix, rebased
onto 96ad35562, so the base is current rather than 624 commits back.

The .github/black-baseline.txt interaction is still live and is a separate matter:
fork-workflow-guard.yml:132 greps ^\.github/ for any path, so a fork contributor who
follows the black gate's own --update-baseline instruction gets blocked and needs a
maintainer label. This diff avoids it by leaving the touched test file baselined, but the
next fork contributor who obeys that hint will hit the same wall.

@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

What unblocks this head

New head 1929838bf, rebased onto current main. Two things are outside a fork
author's reach, so flagging them rather than waiting on them.

1. The workflow runs need approval. Sixteen runs on this head sit at
action_required — Fast Gate, CI, Code Review, and all five review lanes among
them:

completed/action_required   Fast Gate, CI, Build, Code Review, PR Scope,
                            GPT 5.6 Review, Opus 4.8 Review, Design Review,
                            First Principles Review, UX Review,
                            Security Scope Review, Internal Content Scan Gate,
                            Cross-Platform Portability, Dependency License Gate,
                            Screenshot Evidence, macOS Tests (on demand)

PR Readiness itself ran and succeeded as a workflow; the check reads red because the
lanes it aggregates have not run. Nothing else on this head is red.

The previous head's only genuinely failing gate — Backend Lint & Type Check (3.12),
the baselined comment-history ratchet — is gone at the mechanism on current main, so
approving this head should not reproduce it. Detail in the comment above.

2. A fork author's dispositions do not count toward the gate. The four
ai-review-disposition comments on this head's predecessor are well formed, but
pr_status.py reports:

disposition-marked comment(s): 4  from verified writers: 0
NOTICE: none verified as a repository writer's - the disposition check is inert

The rule reads a repository writer's comment, and my access here is pull. So the
concerns are answered in substance — one comment per finding, each naming its span —
but pr-readiness.yml cannot credit them, and readiness: passed is not reachable
from the author's side on a fork PR however completely the findings are answered.
Worth knowing independently of this change: it means the disposition half of the
readiness contract is author-unsatisfiable for every fork contribution.

Local verification on this head, for what it is worth from outside CI: the repo's own
38 runnable script and lint gates pass (the 11 npm/npx gates cannot run on this
host — its glibc is 2.26 and the declared Node floor needs 2.28), test_cron_acp_retry.py
is 22/22, and prove.py reports PROVEN. The backend related-test set has 38 failures
that reproduce identically on plain origin/main here, none in a file this change
touches.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
The cron failure handler has a retry branch built for ACP process death, but
it identified that death by string-matching the exception message for "not
running" or "process exited". The most common death is discovered as a broken
pipe on the next write and is worded "ACP process pipe broken: <cause>",
which matches neither substring, so the branch built for process death is
skipped for the signature that produces it most often. A recoverable one-off
child death is recorded as a hard failure instead, marching
consecutive_failures toward auto-pause.

The generic transient ladder does not cover the gap either:
AcpError.transient defaults to None at that raise site and the message
carries no throttle or 5xx marker, so acp_error_is_transient falls back to
marker-matching and returns False.

The branch now reads the type. AcpProcessDied gains a resubmit_safe
attribute the raising layer sets, mirroring AcpError.transient: the site that
discovers a death knows whether a turn was in flight, so the cron handler
reads one attribute instead of re-deriving it from a class hierarchy.

The attribute is THREE-valued, and the third value is the one that matters.
True means the death was discovered with no turn in flight; False means a
turn may have been in flight; None, the default, means the raise site has
expressed no opinion. A two-valued flag would make silence indistinguishable
from a judgement, and silence is the common case: session_provider's
translation layer rebuilds every shared-runtime death as
AcpProcessDied(str(exc)), so the runtime's five "process not running" raises
arrive carrying no verdict. Under a refusing default they would read as
"unsafe" and lose the retry the wording arm had been giving them all along --
a fix for one death signature that silently deletes the retry for another.
None therefore means "ask the question the handler asked before this
attribute existed", not "no". This is why AcpError.transient is also
None-by-default.

Two sites claim safety. The prompt write marks its own death safe: a failure
there means the child never received a complete session/prompt line, so it
cannot have dispatched a tool. The pre-conversation liveness check marks its
death safe because no conversation exists yet.

That claim is about delivery only, and a caller may re-prompt a session that
has already done work, so the two callers that do withdraw it.
stream_and_collect withdraws it once a tool has fired in the call, reusing
the sticky flag its fallback chain already refuses to replay past. The cron
post-token resume withdraws it on the continuation prompt, which follows text
the interrupted turn already streamed. Without both, a resubmit-safe death on
a re-prompt sends the cron retry back through the ORIGINAL message on a fresh
session and repeats a completed tool call.

One site must state False explicitly rather than lean on the default:
"Process exited during prompt" is a death that happened WITH a turn in
flight, and its wording contains "process exited", so leaving it unclassified
would route it to the wording fallback and retry it. A test walks the AST of
all three modules that construct the exception and fails if any
AcpProcessDied construction whose message literals match the fallback's
substrings omits the keyword, so the one regression a three-valued flag can
introduce cannot be introduced silently. It is an AST walk and not a regex
because a regex matches the same phrases where they appear in prose.

That check sees source literals only, so a message composed at runtime is
invisible to it. session_handle._died builds its message from the runtime's
death summary, which embeds "process exited (rc=N)", and all three of its call
sites are mid-turn, so those deaths are retried by the fallback exactly as
they were before this change. The test docstring and the comment at the
consumer both name that population as a known residual rather than implying
the check covers it. Closing it means _died stating a verdict from
is_turn_active, which it can already see; that is a behaviour change on the
default backend and is left out deliberately.

The substring arm therefore stays, and its role is now the fallback for an
unclassified death. It serves two populations: the five sites that spell
process death as a plain AcpError, and every shared-runtime death that
reaches the handler through the translation layer. It cannot out-vote an
explicit verdict, because the verdict is consulted first and only a None
falls through to it. The comment at the branch records both populations, so
the arm's remaining reach is not read as vestigial.

test/test_cron_acp_retry.py covers the pipe-broken retry, an unclassified
death keeping the retry it had before the verdict existed, both sites that
claim safety, the substring arm's two populations, that a mid-prompt death is
not retried despite its wording, both withdrawals, and the AST invariant.
Removing either withdrawal fails its own test with the replay named; dropping
the explicit keyword at the mid-prompt site fails the invariant.
@SebastianYuSun
SebastianYuSun force-pushed the fix/cron-acp-death-typed-signal branch from 1929838 to d616eaf Compare September 15, 2026 12:14
@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

Round 6 — the deferred design question is now decided, and it was a regression

Three lanes on 1929838bf converged on one defect from three angles, and all three were
right. I had shipped a two-valued resubmit_safe with a refusing default and written
in the body that whether it should be three-valued was "a design decision I have put to
the maintainer rather than taken." That framing was wrong: it was not an open preference,
it was a bug in my own change, and deferring it left the bug shipped.

Measured both ways on the same death. AcpProcessDied("process not running") —
byte-for-byte what session_provider._translate_dead builds:

default consumer decision
pre-PR (wording arm only) retried
shipped two-valued, refusing default not retried
three-valued, None default retried

_translate_dead rebuilds every shared-runtime death as AcpProcessDied(str(exc)), so
the five runtime.py "process not running" raises arrive unclassified. A refusing
default read that silence as "unsafe" and deleted the retry those deaths already had. So
a fix aimed at one death signature silently removed the retry for another — on the
default backend.

The fix

resubmit_safe is now three-valued: True / False / None, where None is the
default and means not classified. None routes to the question the handler asked
before the attribute existed rather than answering it. That is also what makes the claimed
AcpError.transient parallel real instead of decorative — transient is None-by-default
for exactly this reason, and I had cited the precedent while contradicting it.

Net effect on the shared runtime, which is the population that regressed:

death the handler sees pre-PR two-valued three-valued
runtime process not running (5 sites) retried hard failure retried
runtime pipe broken: … hard failure hard failure hard failure
runtime runtime is dead hard failure hard failure hard failure

That backend now loses nothing and gains nothing. The direct-AcpClient path still gains
the fix this PR is for.

One site must now state False explicitly rather than lean on the default.
"Process exited during prompt" is a mid-turn death whose wording contains
"process exited", so leaving it unclassified would route it to the fallback and retry it.
That is the one regression a tri-state can introduce, so it is enforced mechanically: a
test walks the AST of the three modules that construct the exception and fails if any
construction whose message literals match the fallback's substrings omits the keyword.
Fault-injected — deleting that keyword fails the invariant and nothing else.

It is an AST walk rather than a regex because my first attempt was a regex, and it matched
its own explanatory prose: the docstring quotes AcpProcessDied(str(exc)) and
"process not running" while explaining the design, and the scan reported that sentence as
an unclassified raise site.

Dispositions

lane finding status
GPT BLOCKING: refusing default drops the wording-arm retry for unclassified typed deaths fixed — tri-state, above
Design CONCERNS: two-valued flag conflates silence with judgement; default backend loses 5 retries fixed — same change; "Clears when" satisfied, no retry is lost
First Principles CONCERNS 4: symptom-level, deferred fixed at the root rather than deferred

I also ran both reviewer lanes locally against the new head before pushing, and fixed
what they found. Neither returned a BLOCKING finding; these were all CONCERNS/NIT, and all
of them were stale prose from the two-valued revision I abandoned — i.e. defects I
introduced in this round, not pre-existing ones:

  • session_provider.py — comment said "Every other death here takes the fail-closed
    default
    ", which contradicts the tri-state it now sits next to. It also claimed "no
    conversation has been started yet", which is wrong: new_conversation is the warm-pool
    reset, so a prior conversation may well have run tools. The defensible claim is that no
    turn is in flight at reset time, and that is what it says now.
  • slack/gateway.py — the "WHAT THE INVARIANT DOES NOT REACH" block bounded the residual
    at one population and there are two. It omitted every shared-runtime mid-turn death
    from session_handle._died, whose message embeds the runtime's "process exited (rc=N)"
    and which is therefore retried by wording. Verified: runtime.py:2766 composes the
    reason, death_summary() embeds it, session_handle.py:836 raises it unclassified, and
    all three call sites are mid-turn. That population is larger than the five and sits on
    the default backend. Now named.
  • Test docstrings asserting is None while explaining a "fail closed" default; and
    test_only_the_audited_sites_claim_resubmit_safety argued that "opting in is the only
    route to the resubmit path", which the tri-state makes false — the wording fallback is a
    second route. Corrected, and the test now says what it does and does not bound.
  • test_plain_acp_error_still_matches_the_legacy_wording_arm reduced to
    assert not isinstance(AcpError(...), AcpProcessDied) — true of the language, and it
    would pass against a gateway that had dropped the arm entirely. It now drives the real
    cron callback.
  • Two counting tests used raw src.count(...) over module text, the technique their own
    sibling rejects for matching prose. Both are AST now.
  • The delivery argument at the prompt write claimed drain() "fails only while unflushed
    bytes remain". That is false for asyncio — drain raises once the connection is recorded
    lost, buffered or not. The conclusion holds, but on framing, not on drain: messages
    are newline-delimited JSON-RPC and the terminator is appended last, so any prefix the
    child consumed is missing its newline and cannot be parsed as a request. Comment now
    rests on the argument that actually carries it.

Deliberately not in this diff

  • session_handle._died should state a verdict from is_turn_active, which it can
    already see. Both lanes reached this independently and it closes a larger mid-turn replay
    population than the five plain-AcpError sites. It is pre-existing — the substring
    arm matched that same message before this PR for the same reason — and it is a behaviour
    change on the default backend, so it is named in the code comment as a residual rather
    than folded in here. Happy to do it as a follow-up.
  • A shared resubmit_is_safe(exc) helper beside acp_error_is_transient. One lane
    noted the tri-state copies transient's semantics but not the part that prevents drift:
    transient has exactly one owner for its ladder, whereas this inlines the equivalent at
    the consumer. The two other resubmit sites named in the comment would each have to
    re-derive it.
  • I should also correct something I implied earlier about retiring the substring arm.
    Retirement is harder after this change, not easier: the arm previously served one
    bounded population (five sites) and now additionally serves every unclassified
    AcpProcessDied, which is open-ended. The AST invariant cannot tell you when that set is
    empty, because it sees source literals only and not runtime-composed messages. A shared
    classifier is the precondition for retirement, since it gives one place to measure how
    often the fallback is reached and from where.

Verification on this head

  • test/test_cron_acp_retry.py — 24 passed.
  • Upstream prove.py --base origin/main — PROVEN: reverting the production hunks and
    re-running the changed tests fails on an assertion.
  • Repo static gates run from the resolved profile. Two failed and are fixed: black
    (my docstring edit) and check_comment_history (the ratchet caught genuine history
    narration in my new docstring — "used to get", "had been getting" — which I rewrote to
    state the mechanism in the present tense).
  • Three website/ gates fail locally for want of an installed frontend toolchain
    (node_modules absent); this diff touches no website/ file.
  • The repo's backend scoped selector picks 1444 test files here (the diff touches
    slack/gateway.py). Full run: 23 failed, 33010 passed, 119 skipped. I checked
    those 23 rather than assuming they were unrelated: all 23 reproduce in isolation, and
    running the same nine files on a clean worktree at the base commit gives a
    byte-identical failing set (diff of the sorted FAILED lines is empty). They
    are pre-existing and touch none of the files in this diff — the failing areas are
    memory-store binding, apps registry, agent sync, chat runner and spec-builder routes.
  • Rebased onto the current origin/main (534b003ee) before pushing, since the base had
    moved; the guard requires the single commit to sit directly on the freshly fetched
    base. Tests, black and the comment-history ratchet were all re-run green after the
    rebase rather than assumed to carry over.

Head is now d616eafee.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

Both PASS asks verified on d616eafee — measured, no code change needed

Both lanes flipped to PASS with one verification ask each. Neither is a separate
improvement, so neither goes to a follow-up list — they are asks about premises this
diff rests on, and both needed re-checking on the current head specifically, because
the head was rebased onto 534b003ee after the reviews of the prior round.

Design: "verify no third call site re-prompts an already-worked session"

Confirmed: there is no third call site. Enumerated structurally on the current tree
rather than by reading, because the rebase could have introduced one.

Producers and withdrawals (AST over src/kiro_crew, counting keyword args and
attribute assignments, not text matches):

site
claims True acp/client.py:9892 (_send_prompt retag), acp/session_provider.py:123
withdraws llm_helpers.py:2225, slack/gateway.py:1331

Exactly one site reads the verdict to decide a resubmit: slack/gateway.py:5674. The
other two readers are the withdrawals themselves.

The reachable graph. gateway.py:5674 sits in a handler spanning 5602-5993, whose
try protects only 5247-5601. Within that protected body there is exactly one
prompt-issuing entry, _cron_stream_with_posttoken_resume at 5284. That function
(1237-1351) is where the continuation withdrawal at 1331 lives, and beneath it
stream_and_collect's internal retry chain is covered by the withdrawal at
llm_helpers.py:2225. Two re-prompt opportunities, both covered.

The obvious candidate for a third is structurally excluded. The multi-agent
agent_sequence path also calls _cron_stream_with_posttoken_resume, at 5137 — but
that call sits outside the protected body, inside its own try at 5100-5226,
whose handlers do not read resubmit_safe. So a death on the sequential path is
handled locally and never reaches the ACP retry, which is what would be needed for
agent #1's completed tool to be replayed by a death on agent #2's prompt write.

Also checked: all three _send_prompt call sites in client.py (send_message 9062,
send_message_stream 9079, stream_events 9179) issue exactly one prompt per
invocation with no internal loop, so none is itself a re-prompt.

First Principles: "verify a child that read the full prompt line never fails _send_prompt's drain"

Confirmed by measurement on a real subprocess pipe, since this is the premise the
resubmit_safe=True claim rests on. Payload built the way _send_request builds it,
with the newline appended last:

child behaviour payload drain() result
reads the full line (terminator included), then _exit 4 KiB, unpaused returns cleanly, 3/3
reads the full line, then _exit 200 KB, past the 64 KiB high-water mark returns cleanly, 3/3
reads a prefix only (no terminator), then _exit 200 KB raises BrokenPipeError, 3/3

So the premise holds, and it holds in the paused case that the analytical argument was
least certain about. The consequence is stronger than "the claim is safe": a child that
consumed the whole line does not make _send_prompt raise at all, so no safety
claim is produced on that path. Such a death is discovered later, at a mid-turn raise
site, where it is unclassified or explicitly False.

One correction this measurement forced, already in the pushed head. The comment
justifying the claim used to assert that drain() "fails only while unflushed bytes
remain". That is false for asyncio — drain raises once the connection is recorded lost,
buffered or not. The conclusion survives, but on framing, not on drain semantics:
the terminator is written last, so any prefix the child consumed is missing its
newline. The comment now rests on the argument that actually carries it, and the table
above is what confirms the separation is clean rather than incidental.

No code change in either case. Head remains d616eafee.

Remaining state is not agent-actionable: 19 check suites are action_required with
runs=0, i.e. fork workflows awaiting "Approve and run workflows", and PR Readiness
is red as a consequence of checks never allowed to report rather than of a failure.

@SebastianYuSun

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (4) — root cause found, not this diff, and reproducible on demand

Thanks for approving the workflows. One shard is red and I traced it rather than asking
for a blind re-run. A re-run will very probably clear it, and the reason it will is
also the reason it is worth fixing properly.

What failed

Real log, not the annotation: 1 failed, 24611 passed, 1151 skipped, 38 xfailed, on
[gw1] win32 -- Python 3.12.10. Single clean assertion failure, no node-down, no
INTERNALERROR:

test/test_session_pid_reaper_shim.py::TestSweepDirectKillRoutesThroughShim::test_shared_pgid_orphan_killed_via_kill_pid
    killed = sp.kill_orphan_mcps([600])
>   assert killed == 1
E   assert 0 == 1

Root cause: the test's hard-coded pid can collide with the xdist worker's own pid

kill_orphan_mcps reads my_pid = os.getpid() and its loop begins:

if pid == my_pid:
    continue

The test sweeps [600] and patches sp.os.getpgrp and sp.os.getpgid (with
create=True, for Windows' frozen os) — but not os.getpid. So whenever the
pytest-xdist worker process happens to own pid 600, the sweep skips it, killed stays 0,
and the assertion fails with exactly assert 0 == 1.

Reproduced on Linux by injecting only that one value, which produces the Windows
signature on demand:

injected os.getpid() kill_orphan_mcps([600]) test expects
12345 1 — passes 1
600 0 — assert 0 == 1 1

Windows allocates pids in small multiples of 4, so 600 is comfortably reachable for a
process on a fresh runner; a Linux container's worker rarely lands exactly there, which
is why the test is green on Linux and green on most Windows runs.

Suggested fix (yours to take or leave): either patch os.getpid in that test to a
value that cannot collide, or sweep a pid that is not a plausible live pid. The sibling
tests in the same file are unaffected because they exercise
_kill_orphan_gatewayd directly and never reach the my_pid comparison.

Why this is not this diff

So this is an environment-dependent latent defect in an unrelated test, not a flake in
the "unexplained" sense — the mechanism is identified and reproducible. I have not
touched it, since it is outside this diff and fixing it here would both widen the change
and reset approval. Happy to send it as a standalone one-line PR if useful.

Head remains d616eafee. Everything else on this head is green: 79 checks, the only
other red is PR Readiness, which is downstream of this one shard.

@dwu96 dwu96 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 22, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 22, 2026
@NicholasRBowers NicholasRBowers added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 22, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: All four review lanes are clean on d616eafe (GPT 5.6 ✅, Opus 4.8 ✅, Design PASS, First Principles PASS). Two blockers remain and both are mechanical: (1) the branch is CONFLICTING against main and needs a rebase; (2) the only red test is test/test_session_pid_reaper_shim.py::TestSweepDirectKillRoutesThroughShim::test_shared_pgid_orphan_killed_via_kill_pid on the Windows shard, which is unrelated to this PR's ACP process-death retry typing and dates from a 2026-09-15 run. Plan: rebase onto current origin/main, resolve conflicts keeping this PR's resubmit_safe logic on main's current AcpError shape, re-run the scoped tests, and drive the fresh CI round to green. No design or scope changes.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

Copy link
Copy Markdown
Contributor

Open-PR triage (2026-10-05): PARTIALLY_COVERED by merged #15778, needs a rebase and a design decision

I compared the full merge-base diff (534b003e..d616eafe, 5 files, +995/-15) against origin/main af1eb612. The branch is 2707 commits behind. git merge-tree reports content conflicts in src/kiro_crew/acp/client.py and src/kiro_crew/slack/gateway.py.

Merged PR #15778 covers part of this PR, and only part. Its title is "retry a pre-dispatch ACP process death by type", and it was merged 2026-10-01. On main, slack/gateway.py:5568 now reads (isinstance(exc, AcpProcessDied) and not _prompt_dispatched) or (isinstance(exc, AcpError) and ("not running" in exc_msg or "process exited" in exc_msg)). _prompt_dispatched = True is set right before _cron_stream_with_posttoken_resume (gateway.py:4883, :5162).

death main (after #15778) this PR
pipe broken / Runtime is not alive before the prompt is dispatched retried ✅ retried
pipe broken on the session/prompt write (_send_prompt) not retried (post-dispatch; #15778 lists this under "Not a goal") retried (resubmit_safe=True)
typed AcpProcessDied("Process exited during prompt …") still retried through the substring arm refused (resubmit_safe=False plus the scoped wording arm)
continuation prompt / a tool already fired not retried (post-dispatch) not retried (claim withdrawn)

Issue #7395 is still open. Main still does not deliver two pieces of this PR: refusing the mid-turn Process exited during prompt replay, and the delivery-safety argument for the prompt write. It isn't complete, so I'm not closing it.

Points to settle before or during the rebase:

  1. The predicates conflict. fix(cron): retry a pre-dispatch ACP process death by type #15778 deliberately refuses any post-dispatch death, including a broken pipe on the prompt write. This PR argues framing makes that write resubmit-safe. A maintainer has to choose one rule: the _prompt_dispatched gate, or resubmit_safe. Layering both would hide the disagreement.
  2. AcpProcessDied moved and gained a delivery flag. It now lives in acp/transport_errors.py (refactor: move shared ACP transport code into focused owner modules #15153). PR #15905 also added ambiguous_delivery. On main, _send_request goes through write_request_frame_bounded and raises AcpProcessDied(..., ambiguous_delivery=True) on a DRAIN_STALL (client.py:10511-10515), meaning the frame was already buffered and may still be consumed. If _send_prompt's unconditional exc.resubmit_safe = True retag were carried over as written, it would mark that ambiguous death resubmit-safe. That is exactly the replay ambiguous_delivery exists to prevent. The retag would need to skip ambiguous_delivery deaths. The drain()/framing argument in the comment and the body would also need re-checking against the bounded writer.

Classification: PARTIALLY_COVERED (#15778), OVERLAPPING (#15905's delivery classification). No breaking change. No P0 on this head: _send_request on this PR's base only writes, so the retag cannot reach a mid-turn death there.
Recommendation: REBASE onto current main, settle point 1, then CONTINUE-DEVELOPMENT.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @SebastianYuSun for this fix and for the careful write-up of the resubmit_safe invariant.

Part of it is now on main via #15778 (merged 2026-10-01, commit 4d24fde). The cron ACP-death retry now matches AcpProcessDied by type before the prompt is dispatched, so the "ACP process pipe broken: ..." death is retried: src/kiro_crew/slack/gateway_runtime/cron_turn.py:743 (isinstance(exc, AcpProcessDied) and not _prompt_dispatched), checked on main c90dee8cce1a.

What main does not carry yet from this PR:

  • The mid-turn death refusal. src/kiro_crew/acp/client.py:3129 still raises AcpProcessDied("Process exited during prompt ...") with no verdict, and the legacy "process exited" wording arm at cron_turn.py:745 still retries it. This PR states resubmit_safe=False there so a tool that already ran is not replayed.
  • Retrying a pipe-broken death on the prompt write itself. Main sets _prompt_dispatched = True before the prompt call (cron_turn.py:610), so that death is not covered by the typed arm.
  • Withdrawing the claim after tool activity (llm_helpers.py) or on the continuation prompt (_resume_used).

So this PR stays open for a maintainer to decide on the remaining part. Nothing else was changed on it.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks a lot for this fix, @SebastianYuSun. Closing because the linked issue #7395 was closed as completed on 2026-10-12, and latest main already retries cron ACP deaths on the exception type: src/kiro_crew/slack/gateway_runtime/cron_turn.py checks isinstance(exc, AcpProcessDied) before falling back to the transient ladder (see also #9530, which classified the handshake timeout by type). If we got this wrong and a case is still not covered, please reopen or open a fresh PR.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 21, 2026
@mbriones98 mbriones98 removed the drive-to-green PR claimed by drive-to-green pipeline label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(cron): ACP-death retry never fires for the common signature -- guard string-matches the message instead of testing AcpProcessDied

6 participants