Skip to content

fix(slack): fail and reset a turn the backend never finished - #19008

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/channel-transports-timed-out-turn
Oct 11, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/channel-transports-timed-out-turn

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Slack turn whose backend never finished is shown as timed out after whatever text it got out, and starts the next message on a fresh agent process, instead of being posted as a finished reply on the same silent process.

A silent turn ends one of two ways in slack/handler.handle_message, and neither was handled correctly:

  • The shared-runtime session handle ends the turn with one of its two stop reasons for an unfinished turn: "timeout" at its own turn ceiling, or STOP_REASON_STALE_RECOVER for a stale turn its probe confirmed wedged. The handler logged "Unexpected stop_reason ... treating as normal completion", completed the task and booked a success. The thread kept the partial text with no notice, so half an answer read as the reply; a turn with no text showed _No response._.
  • The per-process client raises AcpTimeoutError at its prompt deadline. The except AcpTimeoutError arm failed the task, but it set the reply to e.partial_output or the notice: when the stream path raised (no partial output on the error), the text that had already streamed was replaced by the notice; when the error carried partial output, that text was shown with no notice.

In both cases the runtime keeps running: the handle's terminal does not stop it, and AcpClient._dispatch_events raises without stopping its process. The session was kept (only STOP_REASON_COMPACTION_FAILED resets it), the silent runtime still reads alive, and the next message in the thread was sent to it and hung again.

Why it matters

A Slack user whose backend hangs gets a truncated answer that looks complete, or loses the part of the answer that had streamed, and every later message in the thread waits out the deadline on the same silent process until the session is reset by hand or the gateway restarts. On the stop-reason ending the turn is also counted as a success, so the session's failure breaker never sees it.

Not a goal

  • The notice text stays Slack's established "⏱️ Request timed out. Please try again.", the copy its raised-timeout arm already used. The dashboard and the messaging transports use different wording ("did not finish its answer ..."), because they had no timeout notice before; aligning Slack's copy with theirs is out of scope here.
  • The messaging transports (Telegram, Discord, Teams and the others) are a separate change in messaging/driver.py, messaging/dispatch.py and Telegram's own dispatcher, telegram/transport_dispatch.py. The two touch no common file and merge in either order.
  • The webhook run path (dashboard/handlers/hooks._run_hook_inner) is fixed in the collector change that adds unfinished_turn_error, not here.
  • The dashboard chat path is a separate change (chat_runner._run_chat).
  • The check uses the stop-reason constants on main. Once the collector change that adds unfinished_turn_error lands, the check can call it instead, so the two unfinished reasons are named in one place.

What changed (motivation → approach → change)

  • Symptom: on either ending, the thread does not read as cut off, and the next message reuses the silent runtime; the stop-reason ending is also booked a success.
  • Root cause: the post-stream decision in handle_message had three outcomes (cancelled, compaction failed, everything else completed), and the two unfinished terminals fell into "everything else". The raised-timeout arm composed its reply as e.partial_output or the notice, and neither ending reset the session.
  • Change, all in src/kiro_crew/slack/handler.py:
    • _UNFINISHED_STOP_REASONS (new module constant): frozenset({"timeout", STOP_REASON_STALE_RECOVER}), with STOP_REASON_STALE_RECOVER now imported from kiro_crew.acp.types.
    • _TIMEOUT_REPLY (new module constant): the existing "⏱️ Request timed out. Please try again." text.
    • A new branch between the cancelled and completed branches, for an unfinished terminal: sets _had_error, keeps the partial text and appends _TIMEOUT_REPLY after a blank line (or shows the notice alone), fails the task with "timeout", calls sessions.record_failure, bumps the timeout and failed-message stats, and resets the session, keeping the conversation, as the COMPACTION_FAILED branch does. The "Unexpected stop_reason" warning no longer fires for the two reasons, and the context-usage probe is skipped after the reset, as it is after a compaction-failure reset.
    • The except AcpTimeoutError arm keeps the text that streamed (or e.partial_output when the error carries it), appends the same notice, and resets the session when this turn acquired it. A timeout raised before the session was acquired (a start that never answered) resets nothing, since no session of this turn holds a runtime.
    • Both endings compose the reply and call sessions.reset in place, with no shared helper: test/test_slack_handler_composition_contract.py::test_every_definition_is_in_exactly_one_place pins the module's top-level function and class definitions, so a new module-level function would change that contract.
  • So both endings now book the same failure, show the same text and reset the session. A normal end_turn and a user's cancel take their existing branches unchanged.
  • Second commit, at the circuit breaker's threshold: there record_failure resets the session itself and answers True (session_allocation.py:1494-1507), and while that reset is awaited a newer message for this key can claim a successor, the window the compaction-failure branch's comment names (slack/handler.py:2222). A second reset would end that successor mid-turn. So both endings keep record_failure's answer and reset only when it is not True: the unfinished-turn branch (slack/handler.py:2183) and the except AcpTimeoutError arm, which still resets only a session this turn acquired (slack/handler.py:2312). Below the threshold nothing changes: the failure is booked and the session is reset once.

Rejected alternatives:

  • Re-sending the message on a fresh runtime: nothing was received, and a silent re-send would hide a backend that is down; the user decides what to send next.

Backwards compatibility

Compatible: only a turn ending with "timeout", STOP_REASON_STALE_RECOVER or a raised AcpTimeoutError behaves differently. A finished turn and a user's cancel are unchanged.

Removes nothing: no field, key, function or path is removed; the existing timeout text moves into a constant with the same value.

Tests

Class TestStopReasonUnfinishedTurn in test/test_slack_handler.py, next to TestStopReasonCompactionFailed, on the file's MockSlackClient, FakeProvider and FakeSessionManager harness with the real handle_message. A small _BookingSessions subclass records success and failure bookings, and _RaisingProvider streams its events and then raises instead of yielding a terminal.

  • test_an_unfinished_turn_is_failed_told_and_reset[timeout] and [stale_recover]: partial text then the terminal. The thread's final text holds the partial text and "Request timed out", the session is reset, a failure is booked and no success.
  • test_an_unfinished_turn_with_no_text_says_it_timed_out: the terminal alone. The thread says it timed out and the session is reset.
  • test_a_raised_timeout_keeps_the_streamed_text_and_adds_the_notice: a text chunk, then AcpTimeoutError() (the stream path's error, with no partial output). The final text is the streamed text followed by the notice.
  • test_a_raised_timeout_with_partial_output_adds_the_notice: AcpTimeoutError(partial_output=...). The final text is that output followed by the notice.
  • test_a_raised_timeout_resets_the_session_it_held: the session is reset, a failure is booked and no success.
  • test_a_finished_turn_posts_its_reply_and_keeps_the_session (control): end_turn. The reply is posted with no notice, no reset, no failure.
  • test_a_users_stop_keeps_the_session (control): STOP_REASON_CANCELLED. No reset, no timeout notice.
  • test_at_the_breaker_threshold_an_unfinished_turn_resets_once[timeout] and [stale_recover], and test_at_the_breaker_threshold_a_raised_timeout_resets_once (second commit): _BreakerAtThreshold answers record_failure as the breaker does at its threshold. It books the failure, resets the session itself and returns True, and each reset ends the live runtime while a same-key message claims a successor. The thread still says it timed out, one failure is booked, and the only runtime ended is the first one.

Red on this branch's base, 843abe3, with only the branch's test file copied in, pytest test/test_slack_handler.py -k "TestStopReasonUnfinishedTurn and not breaker_threshold" -p no:cacheprovider -q -n 0 (the first commit's eight tests): 6 failed, 2 passed (the two controls), in each of 3 serial runs. The two stop-reason tests get the partial text with no notice, the no-text test gets _No response._, the two raised-timeout tests get the notice alone or the partial output alone, and the reset test finds no sessions.reset call:

E       AssertionError: Here is the first half of the pl
E       AssertionError: assert 'Request timed out' in '_No response._'
E       assert 'Here is the first half of the pl' in '⏱️ Request timed out. Please try again.'
E       assert ('half an answer' in 'half an answer' and 'Request timed out' in 'half an answer')
E       AssertionError: []
============ 6 failed, 2 passed, 219 deselected, 1 warning in 3.82s ============

The second commit's three tests, on the first commit 8520f45 with this branch's test file copied in, -k breaker_threshold, 3 of 3 serial runs, each:

E       AssertionError: the handler reset again after the breaker did: ['first', 'successor 1']
================= 3 failed, 224 deselected, 1 warning in 3.72s =================

On the first commit each ending resets the session again after the breaker's own reset, so the successor a same-key message claimed is ended too. On main the two stop-reason tests fail as well, on the missing notice (main posts an unfinished turn as a finished reply), and the raised-timeout one passes, since main never resets after a raised timeout (2 failed, 1 passed, 3 of 3 runs).

test/test_slack_handler_composition_contract.py: the error_timeout golden digest is updated, and no other digest changes. Against main, that scenario's transcript gains only a sessions.reset call, and its posted text "half an answer" gains the notice after it. With this branch's digest, test_golden_transcript[error_timeout] fails on 843abe3 (RC=1 in 3 of 3 runs, AssertionError: GOLDEN-DIGEST error_timeout) and passes here.

Green on this branch, on 843abe3, pytest test/test_slack_handler.py -k TestStopReasonUnfinishedTurn -p no:cacheprovider -q -n 0 (all eleven tests of the class), RC=0 in 3 of 3 runs; the whole file gives 227 passed in each of 3 runs:

================ 11 passed, 216 deselected, 1 warning in 2.48s =================

The 93 test files that import kiro_crew.slack.handler, run serially: 7184 passed, exit 0. mypy src/kiro_crew/slack/handler.py: no issues. flake8, isort, black, scripts/check_black_formatting.py, scripts/check_comment_history.py (with COMMENT_HISTORY_BASE_REF=origin/main), scripts/check_harness_parity.py and scripts/docs-lint.sh pass.

Manual verification

N/A — unit coverage sufficient: the tests drive the real handle_message with only the Slack client, provider and session manager faked, and assert on the posted text and the session and booking calls.

Related Issues

Open PRs that change src/kiro_crew/slack/handler.py or test/test_slack_handler.py (#18426, #18192, #18087, #15273, #14574, #14089, #11398) were listed with gh pr list --repo kirodotdev/KiroCrew --state open --limit 500 --json number,files; each one's patch from gh api repos/kirodotdev/KiroCrew/pulls/<n>/files was dry-run applied to main and to this branch. None fails a hunk on this branch that applies on main (#14574 fails the same hunks on main already).
GitHub search (gh search prs and gh search issues --repo kirodotdev/KiroCrew, open and merged) for "Unexpected stop_reason treating as normal completion", "slack timeout partial reply", "handle_message stop_reason" and "stop_reason timeout" found nothing on this defect.

Pattern harvest

Rule candidate: review-prompt
Pattern: two endings of the same failure (a synthesized terminal and a raised exception) handled by separate code drift apart in what the user sees and what is retired; route both through the same helpers and pin both endings in tests.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 01:58
@javenciu
javenciu requested a review from CrysisDeu October 11, 2026 01:58
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 8890edc86a5c5ad70347717bd0df6b62f4cdb5f5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 8890edc86a5c5ad70347717bd0df6b62f4cdb5f5 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The local _UNFINISHED_STOP_REASONS set fixes two of the reasons Slack books as success. error: tool stall, refusal and other error:* terminals are still booked as success.

Not justified as shipped

  • Item 1 — symptom-level: the base guard reads "Unexpected stop_reason %r … treating as normal completion". handle_message never calls acp.types.classify_stop_reason, whose docstring calls it "the single mapping every completion consumer uses". This PR adds frozenset({"timeout", STOP_REASON_STALE_RECOVER}) beside that guard. I grepped _stop_reason (!=|==|not in|in ) in slack/handler.py: 3 terminal kinds that classify as failed still fall through to task.complete() and a success booking: STOP_REASON_TOOL_STALL, STOP_REASON_REFUSAL and generic error:*.
    Clears when: the description says this fix covers only the two runtime-alive reasons, and names the 3 failed-class siblings left on the success path.

What this change ships

Inventory (5 items) — 4 justified

Intent: a Slack user whose backend hung sees the partial answer marked as timed out, and their next message reaches a fresh runtime. FIX.

  1. A timeout/stale_recover terminal now fails the turn, adds the timeout notice and resets the session — symptom-level
  2. A raised timeout keeps the streamed text and adds the notice after it, instead of replacing the text — justified
  3. A raised timeout resets the session this turn held — justified
  4. When the breaker has already reset the session, the turn does not reset it again — justified
  5. The "Unexpected stop_reason" warning no longer fires for these two reasons — justified

[FIRST-PRINCIPLES-REVIEWED] 8890edc

Description read

[DESCRIPTION-READ] be8671e7238a49b2af2b69997de845d5446e86609404581e882e104b72d410e0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 8890edc86a5c5ad70347717bd0df6b62f4cdb5f5 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that sessions.reset really replaces a wedged shared runtime, so the next thread message doesn't land on the same silent process after the reset.

Suggestions

  • The unfinished-reason set is now defined once per surface (Slack here, with sibling PRs for messaging and dashboard). Have the unfinished_turn_error collector follow-up replace _UNFINISHED_STOP_REASONS here, so the surfaces can't drift apart.

[DESIGN-REVIEWED] 8890edc

Description read

[DESCRIPTION-READ] ccabaa4903683b67c933ccdf095fbd93376f9275315470198b4632900e6b0556

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 8890edc86a5c5ad70347717bd0df6b62f4cdb5f5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8890edc

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 8890edc86a5c5ad70347717bd0df6b62f4cdb5f5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8890edc

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
A Slack turn whose backend went silent was posted as a finished reply
and kept its runtime. A silent turn can end two ways in handle_message,
and neither told the user correctly or retired the runtime:

- The shared-runtime session handle ends the turn with a stop reason
  for an unfinished turn ("timeout" at its turn ceiling, "stale_recover"
  for a stale turn its probe confirmed wedged). handle_message logged
  "Unexpected stop_reason ... treating as normal completion", completed
  the task and booked a success; the thread kept the partial text with
  no notice.
- The per-process client raises AcpTimeoutError at its prompt deadline.
  The except arm failed the task, but replaced the text that had
  streamed with the timeout notice, or showed the error's partial
  output with no notice.

In both cases the runtime is still running: the handle's terminal does
not stop it, and AcpClient raises without stopping its process. It
still read alive, so the next message was sent to the same silent
runtime.

Both endings now book a failure (failed task, a failure toward the
breaker, timeout counters), show the partial text followed by the
existing "Request timed out. Please try again." notice, or the notice
alone, and reset the session, keeping the conversation. The raised arm
resets only a session this turn acquired. The facade's composition
contract pins its top-level definitions, so both endings compose the
reply and reset in place; the error_timeout golden transcript is
updated for the added reset and the notice after the partial output.
A normal end_turn and a user's cancel are unchanged.
The unfinished-turn branch and the AcpTimeoutError arm of handle_message
booked the failure and then reset the session. At the circuit-breaker
threshold record_failure already resets the session itself and answers
True, and a same-key message that arrives while that reset is awaited
can claim a successor runtime. The second reset then ended the successor
mid-turn.

Both endings now reset only when record_failure reports that the breaker
did not. Below the threshold nothing changes: the failure is booked and
the session is reset once.

Tests: test_at_the_breaker_threshold_an_unfinished_turn_resets_once
(both stop reasons) and
test_at_the_breaker_threshold_a_raised_timeout_resets_once in
test/test_slack_handler.py, with record_failure answering as the breaker
does at its threshold. Each fails on the previous commit (two resets,
the successor ended) and passes here.
@javenciu
javenciu force-pushed the fix/channel-transports-timed-out-turn branch from 1884084 to 8890edc Compare October 11, 2026 09:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes Slack treat a turn the backend never finished as a failure. That covers the session handle's timeout and STOP_REASON_STALE_RECOVER terminals, and a raised AcpTimeoutError. Before, such a turn was booked as success, half an answer read as the reply, and the next message went to the same silent runtime. Now the streamed text is kept with the timeout notice after it, the task fails, the breaker gets a failure, and the session is reset (conversation kept). The root cause is that the post-stream decision had no unfinished branch and the timeout arm overwrote the streamed text. The PR fixes both in slack/handler.py. I checked the reset against the existing COMPACTION_FAILED branch, which resets in the same way from the same place. The reset skips when record_failure has already reset at the breaker threshold, so a successor session is never ended mid-turn, and tests cover that. The raised-timeout arm resets only when _acquired is set, so a start that never answered resets nothing. end_turn and the user's cancel are unchanged, and control tests confirm it. The golden digest change for error_timeout is the intended new composed text. The First Principles CONCERNS is right that tool-stall, refusal and other error:* terminals still take the success path. That is a pre-existing gap outside this PR's stated goal, not a regression, and is worth a follow-up through classify_stop_reason. Checks are green and there are no change requests. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 18:07
@iamwhatever
iamwhatever merged commit e33c636 into kirodotdev:main Oct 11, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants