Repository navigation
fix(llm): treat a turn the backend never finished as a failure - #19034
iamwhatever merged 1 commit into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the callers the description never mentions ( The description traces what happens next for cron, the workflow pool and the heartbeat only. [DESIGN-REVIEWED] 356259b Description read[DESCRIPTION-READ] ddee3950949d5fa1601069f15c46823a5fa77803db258e31fe789bf5ee1eb226 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that raising non-transient on What this change shipsInventory (5 items) — 5 justifiedIntent: when the backend never finishes a turn, cron, heartbeat and workflow callers see a failure instead of an empty or cut-off answer. This is a FIX.
Siblings: about 11 other collectors still break on [FIRST-PRINCIPLES-REVIEWED] 356259b Description read[DESCRIPTION-READ] d6da905743792fb5ed9e31dc7058499ac2d1af88cf067465be2a5ffb663c2dd5 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
stream_and_collect returned a turn its backend never finished as a completed turn. When a backend goes silent, the shared-runtime session handle ends the turn at its ceiling with EVENT_COMPLETE and stop_reason "timeout", and an unrecovered stale turn with "stale_recover". The collector broke on any EVENT_COMPLETE without reading stop_reason and returned the text collected so far, often empty. Cron runs, heartbeat work and workflow steps recorded that as a result, and the workflow pool kept the worker as healthy. The EVENT_COMPLETE arm now raises AcpTimeoutError for those two stop reasons, after on_complete has seen the event. It is the error the per-process client raises when a prompt times out, it carries the partial text as partial_output, and it is marked not transient, so the retry ladder does not re-send the prompt to a backend that stopped answering. Every other stop reason, including "cancelled", returns the collected text as before. acp/types.py names the "timeout" stop reason as STOP_REASON_TIMEOUT. It, STOP_REASON_STALE_RECOVER and AcpTimeoutError reach llm_helpers through agent_sdk.drivers.acp_vocab, which re-exports them, so the module adds no edge to the ACP layer (scripts/check_agent_sdk_boundary.py).
599a154 to
356259b
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This PR makes stream_and_collect raise AcpTimeoutError (non-transient, carrying the partial text) when the shared-runtime handle ends a turn with stop_reason "timeout" or "stale_recover". Before, those turns came back as a successful empty or partial answer. This matches what the per-process AcpClient already does for a prompt timeout, and it is the root fix for cron, heartbeat and workflow callers recording a dead turn as a success. I checked that the raise inside the try lands in the existing except AcpError arm and is not retried. acp_error_is_transient honours the explicit transient=False, it is not prompt-busy, and the fallback chain is gated on transient. So a silent backend is not re-prompted, and on_complete still runs first. I walked the callers the description does not list (side.py, chat_threads, history_consolidation, decisions/impl_llm). Each already has to handle an AcpError from this function for any failed prompt. history_consolidation fails open, and the others take their existing failure path. So no caller sees an error type it was not already prepared for. end_turn, cancelled and every other stop reason behave as before, the tests pin both unfinished reasons and the cancelled control, all checks and AI lanes are green, and there is no open change request. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A turn its backend never finished reaches
stream_and_collect's callers as a failure, not as a completed turn with partial or empty text.When a backend goes silent, the shared-runtime session handle ends the turn at its ceiling with
EVENT_COMPLETEandstop_reason="timeout", and an unrecovered stale turn withSTOP_REASON_STALE_RECOVER(acp/session_handle.py, "a distinguishing stop_reason so callers that break on EVENT_COMPLETE can tell this apart from a normal turn end").llm_helpers.stream_and_collectbreaks on anyEVENT_COMPLETEwithout readingstop_reasonand returns the text collected so far, often''. The per-processAcpClientpath raisesAcpTimeoutErrorfor the same situation, so the two transports disagree.Why it matters
Cron runs, heartbeat work and workflow steps record a turn the backend never answered as a successful result with empty or partial output. A workflow's next steps consume it. The workflow pool releases the worker as healthy, because the runtime process is still alive, and the next task first pays a full session-start budget on the silent runtime before the pool's hard-reset fallback replaces it.
How soon a silent backend produces these terminals, by code read of
acp/session_handle.py(not exercised end to end). Thetimeoutterminal comes at the turn ceiling (agent.chat_turn_timeout_secs, 2 h by default). Thestale_recoverterminal can come much sooner. A turn that streamed text and then went silent with no tool running is checked onceWatchdogSettings.check_after_secs(60 s) has passed. If the liveness check reads the backend as dead, the turn is probed at once and ends after_CANCEL_GRACE_SECS(10 s): about 70 s. Otherwise the probe waits forstale_window_secs(600 s), ormodel_silent_probe_secs(1800 s) when the evidence is a flat wait, plus the same grace.Not a goal
EVENT_COMPLETEwithout readingstop_reason. Agent Channels (channel._stream_task, the multi-agent channels feature) andkirocrew chat(cli_chat._send_and_print) are fixed in a follow-up PR stacked on this one. Left as they are, each with its own callers and its own reading of an empty answer:task_planner.decompose,messaging/auto_title._stream_title,eval/judge,eval/runner,dashboard/handlers/{hooks,optimizer,taskrunner}and the auto_improvement app's runner.llm_helpers.run_bg_onelineris fixed in the open fix(llm): bound background one-liner calls #18908, which raises the same error for the same two stop reasons.task_executor.execute_taskand the code_review_sage app's review pool already classifystop_reasonand raise.stream_and_collect_jsoncallsstream_and_collect, so it is covered.end_turnstop reasons (max_tokens,refusal,error: tool stall,error: compaction failed). What a cron run reports for those is a separate decision.asyncio.TimeoutError; this error takes the heartbeat's generic failure arm instead (below).What changed (motivation → approach → change)
stop_reason; the collector discarded it.llm_helpers.stream_and_collectbroke on anyEVENT_COMPLETEwithout reading itsstop_reason, so the synthetic terminalsSTOP_REASON_TIMEOUT("timeout") andSTOP_REASON_STALE_RECOVERreturned the partial text as a finished turn. It now raisesAcpTimeoutErrorfor those two.src/kiro_crew/llm_helpers.py: the new_unfinished_turn_error(event, partial_output)returns anAcpTimeoutErrorfor the stop reasons in the new_UNFINISHED_TURN_STOP_REASONS(STOP_REASON_TIMEOUT,STOP_REASON_STALE_RECOVER) andNoneotherwise. The error carries the collected text aspartial_output, says which stop reason ended the turn, and hastransient = False.stream_and_collect'sEVENT_COMPLETEarm callson_completeas before, then raises that error when there is one.src/kiro_crew/acp/types.pynames the"timeout"stop reason the session handle already emits asSTOP_REASON_TIMEOUT, andsrc/kiro_crew/agent_sdk/drivers/acp_vocab.pyre-exports it withSTOP_REASON_STALE_RECOVERandAcpTimeoutError, sollm_helpersreads all three without importing the ACP layer. These lines are the same as fix(llm): bound background one-liner calls #18908's, so the two changes merge cleanly in either order.AcpTimeoutError: it is the error the per-process client already raises when a prompt times out, it subclassesAcpError, the typestream_and_collectalready raises for a failed prompt, and it keeps the partial text for a caller that wants it.stream_and_collect's transient arm would re-send the prompt to a backend that stopped answering and wait out the ceiling again, and cron dispatch's continue re-prompt does the same.slack/gateway_runtime/cron_dispatch.py): itsexcept Exceptionarm re-raises a non-transient error without the continue re-prompt, so the run takes its existing failure path.workflows/agent_pool._run_step→acp/worker_pool.WorkerPool.send): the raise setshealthy = False, soreleaseretires the worker.slack/gateway.py): the genericexcept Exceptionarm logs "Heartbeat task failed" and re-raises;_turn_landedstays False, so its re-injection bookkeeping treats the turn as not landed, asstop_reason_landedalready did for these reasons.end_turn, an empty stop reason,cancelled(a user-requested stop keeps its partial text) and every other stop reason return the collected text as before.end_turnstop reason (it changes cron reporting formax_tokensandrefusal); a new exception class (callers already handleAcpTimeoutErroras anAcpError); retrying the turn.Backwards compatibility
Compatible: a turn that ended normally or was cancelled returns its text exactly as before; only a turn whose terminal says the backend never finished it now raises instead of returning.
The error-handling spec's rule still holds:
docs/system-specs/common/error-handling.md:9says "partial output returned on timeout", and the raisedAcpTimeoutErrorcarries that partial output aspartial_output, the same field the per-process client's prompt timeout fills. The text reaches the caller with the failure instead of as the turn's result.Security: no redaction, auth, sandbox, permission or deny code changes (none in this branch's diff); the raised error carries the same partial text the per-process client's prompt timeout already carries.
Removes nothing: no config key, API, frame shape or public function changes.
Tests
In
test/test_llm_helpers.py, new classTestStreamAndCollectUnfinishedTurn(a fake provider that yields a text chunk and a terminal event; no process, no sleep):test_turn_the_backend_never_finished_raises_prompt_timeout(timeout,stale_recover): raisesAcpTimeoutErrorwithpartial_output == "partial", the stop reason in the message,transient is False, no retry delay awaited, andon_completecalled with the terminal first.test_an_empty_timed_out_turn_raises_rather_than_returning_empty: an empty turn raises withpartial_output == "".test_a_finished_or_cancelled_turn_still_returns_its_text(end_turn, empty,cancelled): returns"partial".Red on main 7bb9d57 (this branch's
test/test_llm_helpers.pycopied into a clean checkout,pytest -n 0, 3 of 3 serial runs): 3 failed, 150 passed each; the 3 control cases pass.Green on the branch (on origin/main 7bb9d57):
test/test_llm_helpers.py153 passed in 3 of 3 serial runs. The files undertest/that namellm_helpersorstream_and_collect, the changed file first, run together: 114 files, RC=0, 7075 passed (and 16 subtests). Of the 123 that match,conftest.py(which pytest loads anyway) and 8 files that callPath.home(),expanduserorpwd.getpw*were left out, because they can reach this machine's home directory.AcpTimeoutError,STOP_REASON_TIMEOUTandSTOP_REASON_STALE_RECOVERreachllm_helpersthroughagent_sdk.drivers.acp_vocab, which re-exports them, so the module adds no import edge to the ACP layer (scripts/check_agent_sdk_boundary.pypasses).Manual verification
A real-process reproducer, kept out of this PR: a real child ACP backend that answers the first turn and then goes silent, attached to a real
AcpRuntime, driven through a realAcpSessionProviderand the realstream_and_collect, with the turn ceiling pinned to 4 s. On main, three serial runs each failed withafter 4.0s the silent backend's turn returned '' as a completed turn; provider.is_process_alive()=True.Related Issues
#18908 (open) makes
llm_helpers.run_bg_onelinerraise the sameAcpTimeoutErrorfor the same two stop reasons, through its own private helper_raise_if_unfinished. Itsacp/types.pyandacp_vocab.pylines are identical to this change's, andgit merge-treeagainst its current head is clean, so the two merge in either order. Once both land, the two private helpers can become one.No issue found. Searched PRs and issues for "stream_and_collect stop_reason", "stop_reason timeout", "turn timeout reported as success", "silent backend", "timed out turn", "wedged runtime reuse" and "chat_turn_timeout_secs".
Pattern harvest
Rule candidate: review-prompt: a consumer that stops on a turn's completion event reads its
stop_reasonbefore treating the collected text as a result; a synthetic terminal (timeout,stale_recover) is not an answer.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)