Repository navigation
fix(slack): fail and reset a turn the backend never finished - #19008
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The local Not justified as shipped
What this change shipsInventory (5 items) — 4 justifiedIntent: a Slack user whose backend hung sees the partial answer marked as timed out, and their next message reaches a fresh runtime. FIX.
[FIRST-PRINCIPLES-REVIEWED] 8890edc Description read[DESCRIPTION-READ] be8671e7238a49b2af2b69997de845d5446e86609404581e882e104b72d410e0 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that Suggestions
[DESIGN-REVIEWED] 8890edc Description read[DESCRIPTION-READ] ccabaa4903683b67c933ccdf095fbd93376f9275315470198b4632900e6b0556 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
A Slack turn whose backend went silent was posted as a finished reply
and kept its runtime. A silent turn can end two ways in handle_message,
and neither told the user correctly or retired the runtime:
- The shared-runtime session handle ends the turn with a stop reason
for an unfinished turn ("timeout" at its turn ceiling, "stale_recover"
for a stale turn its probe confirmed wedged). handle_message logged
"Unexpected stop_reason ... treating as normal completion", completed
the task and booked a success; the thread kept the partial text with
no notice.
- The per-process client raises AcpTimeoutError at its prompt deadline.
The except arm failed the task, but replaced the text that had
streamed with the timeout notice, or showed the error's partial
output with no notice.
In both cases the runtime is still running: the handle's terminal does
not stop it, and AcpClient raises without stopping its process. It
still read alive, so the next message was sent to the same silent
runtime.
Both endings now book a failure (failed task, a failure toward the
breaker, timeout counters), show the partial text followed by the
existing "Request timed out. Please try again." notice, or the notice
alone, and reset the session, keeping the conversation. The raised arm
resets only a session this turn acquired. The facade's composition
contract pins its top-level definitions, so both endings compose the
reply and reset in place; the error_timeout golden transcript is
updated for the added reset and the notice after the partial output.
A normal end_turn and a user's cancel are unchanged.
The unfinished-turn branch and the AcpTimeoutError arm of handle_message booked the failure and then reset the session. At the circuit-breaker threshold record_failure already resets the session itself and answers True, and a same-key message that arrives while that reset is awaited can claim a successor runtime. The second reset then ended the successor mid-turn. Both endings now reset only when record_failure reports that the breaker did not. Below the threshold nothing changes: the failure is booked and the session is reset once. Tests: test_at_the_breaker_threshold_an_unfinished_turn_resets_once (both stop reasons) and test_at_the_breaker_threshold_a_raised_timeout_resets_once in test/test_slack_handler.py, with record_failure answering as the breaker does at its threshold. Each fails on the previous commit (two resets, the successor ended) and passes here.
1884084 to
8890edc
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This PR makes Slack treat a turn the backend never finished as a failure. That covers the session handle's timeout and STOP_REASON_STALE_RECOVER terminals, and a raised AcpTimeoutError. Before, such a turn was booked as success, half an answer read as the reply, and the next message went to the same silent runtime. Now the streamed text is kept with the timeout notice after it, the task fails, the breaker gets a failure, and the session is reset (conversation kept). The root cause is that the post-stream decision had no unfinished branch and the timeout arm overwrote the streamed text. The PR fixes both in slack/handler.py. I checked the reset against the existing COMPACTION_FAILED branch, which resets in the same way from the same place. The reset skips when record_failure has already reset at the breaker threshold, so a successor session is never ended mid-turn, and tests cover that. The raised-timeout arm resets only when _acquired is set, so a start that never answered resets nothing. end_turn and the user's cancel are unchanged, and control tests confirm it. The golden digest change for error_timeout is the intended new composed text. The First Principles CONCERNS is right that tool-stall, refusal and other error:* terminals still take the success path. That is a pre-existing gap outside this PR's stated goal, not a regression, and is worth a follow-up through classify_stop_reason. Checks are green and there are no change requests. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A Slack turn whose backend never finished is shown as timed out after whatever text it got out, and starts the next message on a fresh agent process, instead of being posted as a finished reply on the same silent process.
A silent turn ends one of two ways in
slack/handler.handle_message, and neither was handled correctly:"timeout"at its own turn ceiling, orSTOP_REASON_STALE_RECOVERfor a stale turn its probe confirmed wedged. The handler logged "Unexpected stop_reason ... treating as normal completion", completed the task and booked a success. The thread kept the partial text with no notice, so half an answer read as the reply; a turn with no text showed_No response._.AcpTimeoutErrorat its prompt deadline. Theexcept AcpTimeoutErrorarm failed the task, but it set the reply toe.partial_output orthe notice: when the stream path raised (no partial output on the error), the text that had already streamed was replaced by the notice; when the error carried partial output, that text was shown with no notice.In both cases the runtime keeps running: the handle's terminal does not stop it, and
AcpClient._dispatch_eventsraises without stopping its process. The session was kept (onlySTOP_REASON_COMPACTION_FAILEDresets it), the silent runtime still reads alive, and the next message in the thread was sent to it and hung again.Why it matters
A Slack user whose backend hangs gets a truncated answer that looks complete, or loses the part of the answer that had streamed, and every later message in the thread waits out the deadline on the same silent process until the session is reset by hand or the gateway restarts. On the stop-reason ending the turn is also counted as a success, so the session's failure breaker never sees it.
Not a goal
messaging/driver.py,messaging/dispatch.pyand Telegram's own dispatcher,telegram/transport_dispatch.py. The two touch no common file and merge in either order.dashboard/handlers/hooks._run_hook_inner) is fixed in the collector change that addsunfinished_turn_error, not here.chat_runner._run_chat).unfinished_turn_errorlands, the check can call it instead, so the two unfinished reasons are named in one place.What changed (motivation → approach → change)
handle_messagehad three outcomes (cancelled, compaction failed, everything else completed), and the two unfinished terminals fell into "everything else". The raised-timeout arm composed its reply ase.partial_output orthe notice, and neither ending reset the session.src/kiro_crew/slack/handler.py:_UNFINISHED_STOP_REASONS(new module constant):frozenset({"timeout", STOP_REASON_STALE_RECOVER}), withSTOP_REASON_STALE_RECOVERnow imported fromkiro_crew.acp.types._TIMEOUT_REPLY(new module constant): the existing "⏱️ Request timed out. Please try again." text._had_error, keeps the partial text and appends_TIMEOUT_REPLYafter a blank line (or shows the notice alone), fails the task with"timeout", callssessions.record_failure, bumps the timeout and failed-message stats, and resets the session, keeping the conversation, as theCOMPACTION_FAILEDbranch does. The "Unexpected stop_reason" warning no longer fires for the two reasons, and the context-usage probe is skipped after the reset, as it is after a compaction-failure reset.except AcpTimeoutErrorarm keeps the text that streamed (ore.partial_outputwhen the error carries it), appends the same notice, and resets the session when this turn acquired it. A timeout raised before the session was acquired (a start that never answered) resets nothing, since no session of this turn holds a runtime.sessions.resetin place, with no shared helper:test/test_slack_handler_composition_contract.py::test_every_definition_is_in_exactly_one_placepins the module's top-level function and class definitions, so a new module-level function would change that contract.end_turnand a user's cancel take their existing branches unchanged.record_failureresets the session itself and answers True (session_allocation.py:1494-1507), and while that reset is awaited a newer message for this key can claim a successor, the window the compaction-failure branch's comment names (slack/handler.py:2222). A second reset would end that successor mid-turn. So both endings keeprecord_failure's answer and reset only when it is not True: the unfinished-turn branch (slack/handler.py:2183) and theexcept AcpTimeoutErrorarm, which still resets only a session this turn acquired (slack/handler.py:2312). Below the threshold nothing changes: the failure is booked and the session is reset once.Rejected alternatives:
Backwards compatibility
Compatible: only a turn ending with
"timeout",STOP_REASON_STALE_RECOVERor a raisedAcpTimeoutErrorbehaves differently. A finished turn and a user's cancel are unchanged.Removes nothing: no field, key, function or path is removed; the existing timeout text moves into a constant with the same value.
Tests
Class
TestStopReasonUnfinishedTurnintest/test_slack_handler.py, next toTestStopReasonCompactionFailed, on the file'sMockSlackClient,FakeProviderandFakeSessionManagerharness with the realhandle_message. A small_BookingSessionssubclass records success and failure bookings, and_RaisingProviderstreams its events and then raises instead of yielding a terminal.test_an_unfinished_turn_is_failed_told_and_reset[timeout]and[stale_recover]: partial text then the terminal. The thread's final text holds the partial text and "Request timed out", the session is reset, a failure is booked and no success.test_an_unfinished_turn_with_no_text_says_it_timed_out: the terminal alone. The thread says it timed out and the session is reset.test_a_raised_timeout_keeps_the_streamed_text_and_adds_the_notice: a text chunk, thenAcpTimeoutError()(the stream path's error, with no partial output). The final text is the streamed text followed by the notice.test_a_raised_timeout_with_partial_output_adds_the_notice:AcpTimeoutError(partial_output=...). The final text is that output followed by the notice.test_a_raised_timeout_resets_the_session_it_held: the session is reset, a failure is booked and no success.test_a_finished_turn_posts_its_reply_and_keeps_the_session(control):end_turn. The reply is posted with no notice, no reset, no failure.test_a_users_stop_keeps_the_session(control):STOP_REASON_CANCELLED. No reset, no timeout notice.test_at_the_breaker_threshold_an_unfinished_turn_resets_once[timeout]and[stale_recover], andtest_at_the_breaker_threshold_a_raised_timeout_resets_once(second commit):_BreakerAtThresholdanswersrecord_failureas the breaker does at its threshold. It books the failure, resets the session itself and returns True, and each reset ends the live runtime while a same-key message claims a successor. The thread still says it timed out, one failure is booked, and the only runtime ended is the first one.Red on this branch's base, 843abe3, with only the branch's test file copied in,
pytest test/test_slack_handler.py -k "TestStopReasonUnfinishedTurn and not breaker_threshold" -p no:cacheprovider -q -n 0(the first commit's eight tests): 6 failed, 2 passed (the two controls), in each of 3 serial runs. The two stop-reason tests get the partial text with no notice, the no-text test gets_No response._, the two raised-timeout tests get the notice alone or the partial output alone, and the reset test finds nosessions.resetcall:The second commit's three tests, on the first commit 8520f45 with this branch's test file copied in,
-k breaker_threshold, 3 of 3 serial runs, each:On the first commit each ending resets the session again after the breaker's own reset, so the successor a same-key message claimed is ended too. On main the two stop-reason tests fail as well, on the missing notice (main posts an unfinished turn as a finished reply), and the raised-timeout one passes, since main never resets after a raised timeout (2 failed, 1 passed, 3 of 3 runs).
test/test_slack_handler_composition_contract.py: theerror_timeoutgolden digest is updated, and no other digest changes. Against main, that scenario's transcript gains only asessions.resetcall, and its posted text "half an answer" gains the notice after it. With this branch's digest,test_golden_transcript[error_timeout]fails on 843abe3 (RC=1 in 3 of 3 runs,AssertionError: GOLDEN-DIGEST error_timeout) and passes here.Green on this branch, on 843abe3,
pytest test/test_slack_handler.py -k TestStopReasonUnfinishedTurn -p no:cacheprovider -q -n 0(all eleven tests of the class), RC=0 in 3 of 3 runs; the whole file gives 227 passed in each of 3 runs:The 93 test files that import
kiro_crew.slack.handler, run serially: 7184 passed, exit 0.mypy src/kiro_crew/slack/handler.py: no issues. flake8, isort, black,scripts/check_black_formatting.py,scripts/check_comment_history.py(withCOMMENT_HISTORY_BASE_REF=origin/main),scripts/check_harness_parity.pyandscripts/docs-lint.shpass.Manual verification
N/A — unit coverage sufficient: the tests drive the real
handle_messagewith only the Slack client, provider and session manager faked, and assert on the posted text and the session and booking calls.Related Issues
Open PRs that change
src/kiro_crew/slack/handler.pyortest/test_slack_handler.py(#18426, #18192, #18087, #15273, #14574, #14089, #11398) were listed withgh pr list --repo kirodotdev/KiroCrew --state open --limit 500 --json number,files; each one's patch fromgh api repos/kirodotdev/KiroCrew/pulls/<n>/fileswas dry-run applied to main and to this branch. None fails a hunk on this branch that applies on main (#14574 fails the same hunks on main already).GitHub search (
gh search prsandgh search issues --repo kirodotdev/KiroCrew, open and merged) for "Unexpected stop_reason treating as normal completion", "slack timeout partial reply", "handle_message stop_reason" and "stop_reason timeout" found nothing on this defect.Pattern harvest
Rule candidate: review-prompt
Pattern: two endings of the same failure (a synthesized terminal and a raised exception) handled by separate code drift apart in what the user sees and what is retired; route both through the same helpers and pin both endings in tests.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)