Skip to content

fix(llm): treat a turn the backend never finished as a failure - #19034

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/collector-timed-out-turn-is-failure
Oct 11, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/collector-timed-out-turn-is-failure

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A turn its backend never finished reaches stream_and_collect's callers as a failure, not as a completed turn with partial or empty text.

When a backend goes silent, the shared-runtime session handle ends the turn at its ceiling with EVENT_COMPLETE and stop_reason="timeout", and an unrecovered stale turn with STOP_REASON_STALE_RECOVER (acp/session_handle.py, "a distinguishing stop_reason so callers that break on EVENT_COMPLETE can tell this apart from a normal turn end"). llm_helpers.stream_and_collect breaks on any EVENT_COMPLETE without reading stop_reason and returns the text collected so far, often ''. The per-process AcpClient path raises AcpTimeoutError for the same situation, so the two transports disagree.

Why it matters

Cron runs, heartbeat work and workflow steps record a turn the backend never answered as a successful result with empty or partial output. A workflow's next steps consume it. The workflow pool releases the worker as healthy, because the runtime process is still alive, and the next task first pays a full session-start budget on the silent runtime before the pool's hard-reset fallback replaces it.

How soon a silent backend produces these terminals, by code read of acp/session_handle.py (not exercised end to end). The timeout terminal comes at the turn ceiling (agent.chat_turn_timeout_secs, 2 h by default). The stale_recover terminal can come much sooner. A turn that streamed text and then went silent with no tool running is checked once WatchdogSettings.check_after_secs (60 s) has passed. If the liveness check reads the backend as dead, the turn is probed at once and ends after _CANCEL_GRACE_SECS (10 s): about 70 s. Otherwise the probe waits for stale_window_secs (600 s), or model_silent_probe_secs (1800 s) when the evidence is a flat wait, plus the same grace.

Not a goal

  • Other collectors that break on EVENT_COMPLETE without reading stop_reason. Agent Channels (channel._stream_task, the multi-agent channels feature) and kirocrew chat (cli_chat._send_and_print) are fixed in a follow-up PR stacked on this one. Left as they are, each with its own callers and its own reading of an empty answer: task_planner.decompose, messaging/auto_title._stream_title, eval/judge, eval/runner, dashboard/handlers/{hooks,optimizer,taskrunner} and the auto_improvement app's runner. llm_helpers.run_bg_oneliner is fixed in the open fix(llm): bound background one-liner calls #18908, which raises the same error for the same two stop reasons. task_executor.execute_task and the code_review_sage app's review pool already classify stop_reason and raise. stream_and_collect_json calls stream_and_collect, so it is covered.
  • Other non-end_turn stop reasons (max_tokens, refusal, error: tool stall, error: compaction failed). What a cron run reports for those is a separate decision.
  • The heartbeat's own timeout arm. It resets the heartbeat session only on its asyncio.TimeoutError; this error takes the heartbeat's generic failure arm instead (below).
  • Marking the runtime dead when a turn times out. The handle cannot tell a slow backend from a silent one.

What changed (motivation → approach → change)

  • Motivation: the session handle already reports an unfinished turn in stop_reason; the collector discarded it.
  • Root cause: llm_helpers.stream_and_collect broke on any EVENT_COMPLETE without reading its stop_reason, so the synthetic terminals STOP_REASON_TIMEOUT ("timeout") and STOP_REASON_STALE_RECOVER returned the partial text as a finished turn. It now raises AcpTimeoutError for those two.
  • src/kiro_crew/llm_helpers.py: the new _unfinished_turn_error(event, partial_output) returns an AcpTimeoutError for the stop reasons in the new _UNFINISHED_TURN_STOP_REASONS (STOP_REASON_TIMEOUT, STOP_REASON_STALE_RECOVER) and None otherwise. The error carries the collected text as partial_output, says which stop reason ended the turn, and has transient = False. stream_and_collect's EVENT_COMPLETE arm calls on_complete as before, then raises that error when there is one.
  • src/kiro_crew/acp/types.py names the "timeout" stop reason the session handle already emits as STOP_REASON_TIMEOUT, and src/kiro_crew/agent_sdk/drivers/acp_vocab.py re-exports it with STOP_REASON_STALE_RECOVER and AcpTimeoutError, so llm_helpers reads all three without importing the ACP layer. These lines are the same as fix(llm): bound background one-liner calls #18908's, so the two changes merge cleanly in either order.
  • Why AcpTimeoutError: it is the error the per-process client already raises when a prompt times out, it subclasses AcpError, the type stream_and_collect already raises for a failed prompt, and it keeps the partial text for a caller that wants it.
  • Why not transient: stream_and_collect's transient arm would re-send the prompt to a backend that stopped answering and wait out the ceiling again, and cron dispatch's continue re-prompt does the same.
  • What each caller does with it:
    • Cron (slack/gateway_runtime/cron_dispatch.py): its except Exception arm re-raises a non-transient error without the continue re-prompt, so the run takes its existing failure path.
    • Workflow pool (workflows/agent_pool._run_step → acp/worker_pool.WorkerPool.send): the raise sets healthy = False, so release retires the worker.
    • Heartbeat (slack/gateway.py): the generic except Exception arm logs "Heartbeat task failed" and re-raises; _turn_landed stays False, so its re-injection bookkeeping treats the turn as not landed, as stop_reason_landed already did for these reasons.
  • Unchanged: end_turn, an empty stop reason, cancelled (a user-requested stop keeps its partial text) and every other stop reason return the collected text as before.
  • Rejected: raising for every non-end_turn stop reason (it changes cron reporting for max_tokens and refusal); a new exception class (callers already handle AcpTimeoutError as an AcpError); retrying the turn.

Backwards compatibility

Compatible: a turn that ended normally or was cancelled returns its text exactly as before; only a turn whose terminal says the backend never finished it now raises instead of returning.
The error-handling spec's rule still holds: docs/system-specs/common/error-handling.md:9 says "partial output returned on timeout", and the raised AcpTimeoutError carries that partial output as partial_output, the same field the per-process client's prompt timeout fills. The text reaches the caller with the failure instead of as the turn's result.
Security: no redaction, auth, sandbox, permission or deny code changes (none in this branch's diff); the raised error carries the same partial text the per-process client's prompt timeout already carries.
Removes nothing: no config key, API, frame shape or public function changes.

Tests

In test/test_llm_helpers.py, new class TestStreamAndCollectUnfinishedTurn (a fake provider that yields a text chunk and a terminal event; no process, no sleep):

  • test_turn_the_backend_never_finished_raises_prompt_timeout (timeout, stale_recover): raises AcpTimeoutError with partial_output == "partial", the stop reason in the message, transient is False, no retry delay awaited, and on_complete called with the terminal first.
  • test_an_empty_timed_out_turn_raises_rather_than_returning_empty: an empty turn raises with partial_output == "".
  • test_a_finished_or_cancelled_turn_still_returns_its_text (end_turn, empty, cancelled): returns "partial".

Red on main 7bb9d57 (this branch's test/test_llm_helpers.py copied into a clean checkout, pytest -n 0, 3 of 3 serial runs): 3 failed, 150 passed each; the 3 control cases pass.

E           Failed: DID NOT RAISE <class 'kiro_crew.acp.client.AcpTimeoutError'>
FAILED test/test_llm_helpers.py::TestStreamAndCollectUnfinishedTurn::test_turn_the_backend_never_finished_raises_prompt_timeout[timeout]
FAILED test/test_llm_helpers.py::TestStreamAndCollectUnfinishedTurn::test_turn_the_backend_never_finished_raises_prompt_timeout[stale_recover]
FAILED test/test_llm_helpers.py::TestStreamAndCollectUnfinishedTurn::test_an_empty_timed_out_turn_raises_rather_than_returning_empty

Green on the branch (on origin/main 7bb9d57): test/test_llm_helpers.py 153 passed in 3 of 3 serial runs. The files under test/ that name llm_helpers or stream_and_collect, the changed file first, run together: 114 files, RC=0, 7075 passed (and 16 subtests). Of the 123 that match, conftest.py (which pytest loads anyway) and 8 files that call Path.home(), expanduser or pwd.getpw* were left out, because they can reach this machine's home directory. AcpTimeoutError, STOP_REASON_TIMEOUT and STOP_REASON_STALE_RECOVER reach llm_helpers through agent_sdk.drivers.acp_vocab, which re-exports them, so the module adds no import edge to the ACP layer (scripts/check_agent_sdk_boundary.py passes).

Manual verification

A real-process reproducer, kept out of this PR: a real child ACP backend that answers the first turn and then goes silent, attached to a real AcpRuntime, driven through a real AcpSessionProvider and the real stream_and_collect, with the turn ceiling pinned to 4 s. On main, three serial runs each failed with after 4.0s the silent backend's turn returned '' as a completed turn; provider.is_process_alive()=True.

Related Issues

#18908 (open) makes llm_helpers.run_bg_oneliner raise the same AcpTimeoutError for the same two stop reasons, through its own private helper _raise_if_unfinished. Its acp/types.py and acp_vocab.py lines are identical to this change's, and git merge-tree against its current head is clean, so the two merge in either order. Once both land, the two private helpers can become one.

No issue found. Searched PRs and issues for "stream_and_collect stop_reason", "stop_reason timeout", "turn timeout reported as success", "silent backend", "timed out turn", "wedged runtime reuse" and "chat_turn_timeout_secs".

Pattern harvest

Rule candidate: review-prompt: a consumer that stops on a turn's completion event reads its stop_reason before treating the collected text as a result; a synthetic terminal (timeout, stale_recover) is not an answer.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 04:33
@javenciu
javenciu requested a review from cruisercohen October 11, 2026 04:33
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 356259b402bb23aec21f7a70d095976549e46939 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 356259b402bb23aec21f7a70d095976549e46939 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the callers the description never mentions (dashboard/chat_threads.py, handlers/side.py, history_consolidation.py, decisions/impl_llm.py, meetings, issue_radar) handle the new raise.

The description traces what happens next for cron, the workflow pool and the heartbeat only. stream_and_collect has about a dozen callers. chat_threads._run_thread_turn is one example: its outer except Exception now posts _FAILED_FALLBACK instead of saving the partial reply. That looks like the intended outcome, but no one has checked it for the remaining callers.

[DESIGN-REVIEWED] 356259b

Description read

[DESCRIPTION-READ] ddee3950949d5fa1601069f15c46823a5fa77803db258e31fe789bf5ee1eb226

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 356259b402bb23aec21f7a70d095976549e46939 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that raising non-transient on stale_recover is intended: classify_stop_reason (acp/types.py:557), which every completion consumer shares, marks it recoverable.

What this change ships

Inventory (5 items) — 5 justified

Intent: when the backend never finishes a turn, cron, heartbeat and workflow callers see a failure instead of an empty or cut-off answer. This is a FIX.

  1. A turn that hits the turn ceiling (timeout) now raises AcpTimeoutError (with the text so far) instead of returning that text as the answer — justified
  2. An unrecovered stale turn (stale_recover) now raises the same error instead of returning its text — justified
  3. The error is non-transient, so neither stream_and_collect nor cron's continue re-prompt retries a silent backend — justified
  4. A named STOP_REASON_TIMEOUT constant for the "timeout" string the session handle already emits — justified
  5. acp_vocab now re-exports AcpTimeoutError and both stop reasons, so llm_helpers reaches them through the agent-SDK boundary — justified

Siblings: about 11 other collectors still break on EVENT_COMPLETE without reading stop_reason. The PR's ## Not a goal names them and puts them in a stacked follow-up and #18908, so this is accepted and deferred. Cause level: the session handle sending a completion event for a turn that never finished is out of scope, because the dashboard's stale-turn recovery depends on that event.

[FIRST-PRINCIPLES-REVIEWED] 356259b

Description read

[DESCRIPTION-READ] d6da905743792fb5ed9e31dc7058499ac2d1af88cf067465be2a5ffb663c2dd5

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 356259b402bb23aec21f7a70d095976549e46939 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 356259b

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 356259b402bb23aec21f7a70d095976549e46939 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 356259b

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
stream_and_collect returned a turn its backend never finished as a
completed turn. When a backend goes silent, the shared-runtime session
handle ends the turn at its ceiling with EVENT_COMPLETE and
stop_reason "timeout", and an unrecovered stale turn with
"stale_recover". The collector broke on any EVENT_COMPLETE without
reading stop_reason and returned the text collected so far, often
empty. Cron runs, heartbeat work and workflow steps recorded that as a
result, and the workflow pool kept the worker as healthy.

The EVENT_COMPLETE arm now raises AcpTimeoutError for those two stop
reasons, after on_complete has seen the event. It is the error the
per-process client raises when a prompt times out, it carries the
partial text as partial_output, and it is marked not transient, so
the retry ladder does not re-send the prompt to a backend that stopped
answering. Every other stop reason, including "cancelled", returns the
collected text as before.

acp/types.py names the "timeout" stop reason as STOP_REASON_TIMEOUT.
It, STOP_REASON_STALE_RECOVER and AcpTimeoutError reach llm_helpers
through agent_sdk.drivers.acp_vocab, which re-exports them, so the
module adds no edge to the ACP layer (scripts/check_agent_sdk_boundary.py).
@javenciu
javenciu force-pushed the fix/collector-timed-out-turn-is-failure branch from 599a154 to 356259b Compare October 11, 2026 14:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes stream_and_collect raise AcpTimeoutError (non-transient, carrying the partial text) when the shared-runtime handle ends a turn with stop_reason "timeout" or "stale_recover". Before, those turns came back as a successful empty or partial answer. This matches what the per-process AcpClient already does for a prompt timeout, and it is the root fix for cron, heartbeat and workflow callers recording a dead turn as a success. I checked that the raise inside the try lands in the existing except AcpError arm and is not retried. acp_error_is_transient honours the explicit transient=False, it is not prompt-busy, and the fallback chain is gated on transient. So a silent backend is not re-prompted, and on_complete still runs first. I walked the callers the description does not list (side.py, chat_threads, history_consolidation, decisions/impl_llm). Each already has to handle an AcpError from this function for any failed prompt. history_consolidation fails open, and the others take their existing failure path. So no caller sees an error type it was not already prepared for. end_turn, cancelled and every other stop reason behave as before, the tests pin both unfinished reasons and the cancelled control, all checks and AI lanes are green, and there is no open change request. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 17:35
@iamwhatever
iamwhatever merged commit f0f21bc into kirodotdev:main Oct 11, 2026
107 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants