Repository navigation
fix(messaging): fail and reset a turn the backend never finished - #19026
Conversation
8d8ef97 to
a670ccd
Compare
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check one thing: on Slack's transport path these turns now show "The agent did not finish…", but What this change shipsInventory (8 items) — 8 justifiedIntent: when the backend on a messaging channel never finishes a turn, the user should see that the reply was cut off, and their next message should go to a fresh agent instead of the same hung process. FIX.
[FIRST-PRINCIPLES-REVIEWED] f3002c2 Description read[DESCRIPTION-READ] 26533d4185bc6629b8c68c99c6c9ba131a287fa3f72d95a0cfffbea38e61648c That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS
Watch
[DESIGN-REVIEWED] f3002c2 Description read[DESCRIPTION-READ] 99af2f63a3b39269ec7067d31ecffac5d31e85961fda0021ce0ec7fb2a4387a9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
Fixed in the second commit, d2d0f9e. After an unfinished turn, each dispatcher now resets the session only when |
|
Slack's transport path now keeps the notice's promise. |
On the messaging transports (Telegram, Discord, Teams and the others), a turn whose backend went silent was delivered as a finished reply. When the session handle ended the turn with "timeout" (its turn ceiling) or STOP_REASON_STALE_RECOVER (a stale turn its probe confirmed wedged), TurnDriver flushed the partial text and sent DONE with no notice: empty_turn_notice speaks only for a textless turn. The turn was then booked a success and the session kept: messaging.dispatch booked _record_success and Verdict.ANSWERED, and Telegram's own dispatcher, telegram/transport_dispatch.py, booked record_success. The runtime still read alive, so the next message was sent to the same silent runtime. TurnDriver now sends a timeout notice as the turn's last text after the partial text (or alone), so every renderer shows it through its normal text path, owes no textless notice on top, and marks the turn unfinished. Both dispatchers book such a turn as a failure (record_failure; Verdict.FAILED in messaging.dispatch, the failed- message counter in Telegram's) and reset the session, keeping the conversation, as messaging.dispatch's compaction-failure retry does. The notice lives in empty_turn_copy beside the other turn notices. The two stop reasons live in messaging.driver, its only reader, beside its other ACP stop reasons: empty_turn_copy is loaded while kiro_crew.acp.client initialises, so it imports no ACP name. A finished turn and a user's cancel are unchanged.
…ath too The unfinished-turn branch in the shared dispatcher and in Telegram's booked the failure and then reset the session unconditionally. At the circuit-breaker threshold record_failure already resets the session itself and answers True, and a same-key message that arrives while that reset is awaited can claim a successor runtime. The second reset then ended the successor mid-turn. Both now reset only when record_failure reports that the breaker did not. Slack's transport path (slack/transport_dispatch.handle_message_transport) shows the same notice, because TurnDriver.run posts it, but it booked every completed turn as a success and never reset. It now books an unfinished turn as a failure and resets once, with the same breaker check, while the turn still holds its permit, and it skips the context check on the runtime it just reset. Tests: test_at_the_breaker_threshold_an_unfinished_turn_resets_once in test/test_channel_turns.py, test/test_telegram.py and test/test_slack_transport_dispatch.py, with record_failure behaving as the breaker does at its threshold; on the Slack path also test_an_unfinished_turn_is_a_failure_and_resets_the_session and a finished-turn control. Each fails on the previous commit and passes here.
d2d0f9e to
f3002c2
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Before this PR, a turn that ended "timeout" or stale_recover on the shared, Telegram or Slack transport dispatchers looked like a finished reply. It was booked as a success, and the next message went to the same silent runtime, which hung again. This PR fixes that at the root. TurnDriver.run adds UNFINISHED_TURN_NOTICE as the turn's last text chunk and sets unfinished_turn. Each dispatcher then books record_failure and the failed counter, and resets the session (keeping the conversation) only when record_failure did not already trip the breaker. I checked record_failure (session_allocation.py:1494). Its False branch has no await, so no successor can be claimed between it and the explicit reset. Its True branch has already reset, so skipping the second reset protects a successor. I tried a user cancel and a normal end_turn against the new branch: both are untouched, and the controls pin them. The 17 new tests fail on main and pass here, and they drive the real driver, dispatchers and Discord renderer. I weighed both Design watch items. The breaker charge is harmless, because the breaker's only action is the same reset this path does anyway. The "time limit" wording for stale_recover and the Slack copy differing from #19008 are copy choices, not defects. Checks are green and there is no change request. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: On Telegram, Discord, Teams and the other messaging transports, a turn whose backend never finished is shown as cut off at its time limit and starts the next message on a fresh agent process, instead of being delivered as a finished reply on the same silent process.
When the backend goes silent, the session handle ends the turn with one of its two stop reasons for an unfinished turn:
"timeout"at its own turn ceiling, orSTOP_REASON_STALE_RECOVERfor a stale turn its probe confirmed wedged.messaging.driver.TurnDriver.runflushed whatever text had streamed and sentDONEwith no notice:empty_turn_noticespeaks only for a turn that produced no text. The partial text read as the reply.messaging.dispatchthen booked the turn with_record_successandVerdict.ANSWEREDand kept the session. Telegram runs its own dispatcher,telegram/transport_dispatch.py, on the same driver, and bookedrecord_successthe same way. The silent runtime still reads alive, so the next message in the conversation was sent to it and hung again.Why it matters
A user on a messaging channel whose backend hangs gets half an answer that looks finished, and every later message waits out the turn ceiling on the same silent process until the session is reset by hand or the gateway restarts. The turn also counts as a success, so the session's failure breaker never sees it.
Not a goal
slack/handler.handle_message, fix(slack): fail and reset a turn the backend never finished #19008). The two touch no common file and merge in either order. Slack keeps its established "⏱️ Request timed out. Please try again." copy; these transports had no timeout notice before, so they use the dashboard chat's wording ("The agent did not finish its answer ...").dashboard/handlers/hooks._run_hook_inner) is fixed in the collector change that addsunfinished_turn_error, not here.end_turnare unchanged; control tests pin both.UNFINISHED_STOP_REASONSinmessaging.driver,"timeout"andSTOP_REASON_STALE_RECOVER. One shared stop-reason set replaces these copies once the change that adds it lands.What changed (motivation → approach → change)
"timeout"orSTOP_REASON_STALE_RECOVERis delivered as a finished reply, booked as a success, and the next message reuses the silent runtime.EVENT_COMPLETEbranch decides only whether a textless turn needs a notice, and the three dispatchers that drive a visible renderer (the shared one, Telegram's own and Slack's transport path) book every completed turn that reached the channel as a success.src/kiro_crew/messaging/empty_turn_copy.py:UNFINISHED_TURN_NOTICE(new): "⏱️ The agent did not finish its answer before this turn's time limit, so the turn was stopped. Your next message starts the agent again." The module still imports no ACP name. It is loaded whilekiro_crew.acp.clientinitialises (acp/client.pyimportsacp/prompt_blocks.py, which importsmessaging.raster, andmessaging/__init__.pyloadsmessaging.driver), so the stop reasons live in the driver instead.src/kiro_crew/messaging/driver.py:UNFINISHED_STOP_REASONS(new module constant):frozenset({"timeout", STOP_REASON_STALE_RECOVER}), withSTOP_REASON_STALE_RECOVERadded to the module's existingkiro_crew.acp.typesimport.src/kiro_crew/messaging/driver.py,TurnDriver.run: for a terminal inUNFINISHED_STOP_REASONS, after the final flushes and beforeDONE, it dispatches the notice as the turn's lastTEXT_CHUNK, after a blank line when text streamed, so every renderer shows it after the partial text or alone. It sets the newunfinished_turnflag (reset at the start of each run), and setsempty_turn_noticeto""so the textless verdict is not owed on top. The returned text stays the model's partial text.src/kiro_crew/messaging/dispatch.py: after the delivery check, a turn withdriver.unfinished_turnis booked withsessions.record_failureandVerdict.FAILEDinstead of_record_successandVerdict.ANSWERED, and the session is reset, keeping the conversation, as the compaction-failure retry in the same function does. A failed reset is logged; the verdict stays FAILED. The exchange is still recorded with the partial text.src/kiro_crew/telegram/transport_dispatch.py, the post-turn bookkeeping inhandle_message: a turn withdriver.unfinished_turnis booked withsessions.record_failureand the failed-message counter instead ofrecord_success, the shared-death clear and the success counter, and the session is reset, keeping the conversation. A failed reset is logged.src/kiro_crew/slack/transport_dispatch.py, the post-turn bookkeeping inhandle_message_transport:TurnDriver.runshows the notice on this path too, and the path booked every completed turn as a success. A turn withdriver.unfinished_turnis booked withsessions.record_failureand the failed-message counter instead ofrecord_success, the shared-death clear and the success counter, and the session is reset once, keeping the conversation. A failed reset is logged. The reset runs while the turn holds its permit, like the breaker reset the same function'sexceptbranch can make, and the release in its finally block sees, throughabsorb_orphaned_release(session_allocation.py:1634), that the permit died with the reset session and leaves a successor's alone. The context check is skipped for such a turn (slack/transport_dispatch.py:1015): the runtime it would measure was just reset, and theexceptbranch skips that check too.record_failureanswers False (messaging/dispatch.py:2437-2439,telegram/transport_dispatch.py:1249-1250,slack/transport_dispatch.py:942-943). On the failure that trips the circuit breaker (the fifth in a row,session.py:758),record_failurehas already reset the session and answered True (session_allocation.py:1500-1507). A same-key message can claim a successor runtime while that reset is awaited, and a second reset would end the successor mid-turn.record_failure, thenresetunless the breaker already reset). The shared dispatcher already makes both in the same function, for an undelivered turn and for the compaction retry.Rejected alternatives:
on_done: the renderers seal their bodies differently (Telegram seals as ok whenstop_reason != "error", Discord uses the textless notice as its placeholder), so it would take one branch per transport, and the text path already renders the same way everywhere.Backwards compatibility
Compatible: only a turn ending with
"timeout"orSTOP_REASON_STALE_RECOVERbehaves differently. A finished turn, a textless turn, a refused orerror:turn and a user's cancel are unchanged.Removes nothing: the change adds two constants, one driver flag and one dispatcher branch, and removes no field, path or function.
Tests
test/test_messaging_driver.py, new classTestUnfinishedTurn, on the file's_ScriptedProviderand_RecordingRendererwith the realTurnDriver:test_the_notice_follows_the_partial_text[timeout]and[stale_recover]: the rendered text is the partial text, a blank line and the notice,DONEis last, the returned text is the partial text,empty_turn_noticeis""and the turn is marked unfinished.test_a_textless_unfinished_turn_shows_the_notice_alone.test_a_finished_or_cancelled_turn_gets_no_timeout_notice[end_turn]and[cancelled](controls).test/test_channel_turns.py, classTestDeliveryAwareAccounting, on the file'sFakeSessionsandScriptedProviderwith the real dispatcher:test_an_unfinished_turn_is_a_failure_and_resets_the_session[timeout]and[stale_recover]:Verdict.FAILED, one failure booked, no success, one reset.test_a_finished_or_cancelled_turn_keeps_the_session[end-turn]and[cancelled](controls).test_the_real_discord_renderer_shows_the_timeout_after_the_partial_text[timeout]and[stale_recover]: the realDiscordRendereron the file'sFakeDiscordClientshows the partial text followed by the notice; the turn fails and the session is reset.test_at_the_breaker_threshold_an_unfinished_turn_resets_once[timeout]and[stale_recover](second commit):record_failureacts as the breaker does at its threshold. It resets the session itself and answers True, and a same-key successor is claimed during that reset. The turn fails, the session is reset once, and the successor is not reset.test/test_telegram.py, classTestTurnLifecycleCharacterization, on the file's_dispatcher,FakeProviderand_instrumentwith the real Telegram dispatcher andTurnDriver:test_an_unfinished_turn_is_a_failure_and_resets_the_session[timeout]and[stale_recover]: no success booked, one failure, one reset.test_at_the_breaker_threshold_an_unfinished_turn_resets_once[timeout]and[stale_recover](second commit): the same breaker shape on the Telegram dispatcher; one failure, one reset, and the successor is not reset.test_a_finished_turn_is_a_success_and_keeps_the_session(control): one success, no failure, no reset.test/test_slack_transport_dispatch.py, new classTestTransportUnfinishedTurn(second commit), on the file'sFakeSessions,ScriptedProviderandRecordingSlackClientwith the realhandle_message_transportandTurnDriver:test_an_unfinished_turn_is_a_failure_and_resets_the_session[timeout]and[stale_recover]: the notice is shown in the thread; one failure is booked, no success, one reset, and no context check.test_at_the_breaker_threshold_an_unfinished_turn_resets_once[timeout]and[stale_recover]: the same breaker shape on the Slack transport path; one failure, one reset, and the successor is not reset.test_a_finished_turn_is_a_success_and_keeps_the_session(control): one success, one context check, no reset.Red on main (7bb9d57), the 23 new tests,
pytest test/test_messaging_driver.py test/test_channel_turns.py test/test_telegram.py test/test_slack_transport_dispatch.py -k "TestUnfinishedTurn or unfinished_turn_is_a_failure or finished_or_cancelled_turn_keeps_the_session or real_discord_renderer_shows_the_timeout or finished_turn_is_a_success_and_keeps or at_the_breaker_threshold_an_unfinished_turn_resets_once or TestTransportUnfinishedTurn" -p no:cacheprovider -n 0, RC=1 in 3 of 3 runs, the same failures each time, 17 failed, 6 passed (the six controls):The last
Eline is the Slack transport path booking the turn a success. On main its first test fails one step earlier, because main shows no notice in the thread.The four channel and Telegram breaker cases the second commit adds fail on main before their reset check, because main books the unfinished turn as answered. Their own red is on the first commit, 6799eed (these two test files copied into a detached checkout of it,
-k at_the_breaker_threshold_an_unfinished_turn_resets_once), 3 of 3 runs, the same failures each time:The full message is
the dispatcher reset again after the breaker did: ['first', 'successor 1']: the explicit reset ended the successor.The Slack transport tests are red on that first commit too, because the first commit leaves that path unchanged: it shows the notice and books the turn a success. The test file was copied into a detached checkout of 6799eed and run with
-k TestTransportUnfinishedTurn, 3 of 3 runs, the same failures each time:The two Slack breaker cases therefore never reach their reset check on 6799eed. They were also run on this branch with the Slack branch's breaker check removed (the reset made unconditional, in a detached checkout). They failed 3 of 3 with
the dispatcher reset again after the breaker did: ['first',..., and the other three Slack cases passed.The two Telegram failures are the fourth
Eline: the turn was booked a success.Green on this branch, on 7bb9d57: the four test files whole, RC=0 in 3 of 3 runs, the 23 new tests among them:
test/test_channel_turns.py,test/test_messaging_driver.py, the Telegram test files, every test file that namestransport_dispatchorempty_turn_copy, and the two Slack transport test files that list lacked (test/test_channel_activation.pyandtest/test_slack_events_coverage.py), 83 files run serially: 5896 passed, 2 skipped, 5 xfailed, 1 failed. The failure istest_channel_activation.py::TestChannelActivationRouting::test_group_channel_mention_mode_ignores_thread_without_session, and it does not come from this change. It fails the same way on unchangedorigin/mainat 7bb9d57 with the same 83-file list (5873 passed, the same 1 failed), and the file passes alone on both (17 passed). The test leavesget_session_for_threadunconfigured on a bareMagicMock, so whether the thread reply is ignored depends on state an earlier file in the run leaves behind.mypyon the five changed source files: no issues. flake8, isort, black,scripts/check_black_formatting.py,scripts/check_comment_history.py(withCOMMENT_HISTORY_BASE_REF=origin/main),scripts/check_agent_sdk_boundary.py,scripts/check_harness_parity.pyandscripts/docs-lint.shpass.Manual verification
N/A — unit coverage sufficient: the tests drive the real
TurnDriver, the real dispatcher and the realDiscordRendererwith only the provider, session manager and Discord client faked.Related Issues
Open PRs that change any of the seven files this branch changes were listed with
gh pr list --repo kirodotdev/KiroCrew --state open --limit 1000 --json number,files(657 open PRs; for the 13 whose file list stops at 100, the rest of their files were read page by page fromgh api repos/kirodotdev/KiroCrew/pulls/<n>/files): #6595, #6600, #7626, #10098, #10192, #10965, #11722, #11831, #14089, #15111, #15273, #17582, #18192, #18215, #18268. Each one's patch for those files was checked withgit apply --check(exact context) against main (33b67ac) and against this branch, and gives the same result on both except #15273'smessaging/driver.py: itsTurnDriver.__init__hunk sits next to the newunfinished_turnfield, so the exact-context check fails on this branch, while a three-way merge of #15273's head with this branch (git merge-tree --write-tree) is clean. #15273 also changes thedriver.runcall three lines above the changed Telegram block; that hunk applies to both. #6595, #6600, #7626, #10098, #10192 and #14089 already fail hunks on main.GitHub search (
gh search prsandgh search issues --repo kirodotdev/KiroCrew, open and merged) for "telegram timeout partial reply" and "empty_turn_notice timeout" returned nothing; "stop_reason timeout" returned only merged #5504 (native slash commands on the shared runtime, another topic). Sanity check: "empty_turn_notice" finds #12344, #13416 and #16429, none about a cut-off turn with partial text.Pattern harvest
Rule candidate: review-prompt
Pattern: a turn-end notice that only speaks for a textless turn lets a cut-off turn with partial text read as finished; every surface that books a turn must name the unfinished terminals, not only the empty one.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)