Repository navigation
fix(channels): replay a turn abandoned after a transient compaction failure - #11243
Conversation
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Discord and Telegram run their own turn loops and never read the compaction verdict — the same transient failure still eats messages there, unaddressed. Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: stop silently dropping a user's message when a transient backend throttle fails compaction, on the two chat surfaces PR #8215's dashboard fix left out — a FIX.
[FIRST-PRINCIPLES-REVIEWED] 151ab03 |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Default Slack route ( Watch
Suggestions
[DESIGN-REVIEWED] 151ab03 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
e17cb4c to
543210d
Compare
543210d to
8b4cede
Compare
8b4cede to
6a7e1f8
Compare
6a7e1f8 to
7788ed3
Compare
3b8e483 to
784f90e
Compare
784f90e to
24db81e
Compare
24db81e to
d5aa48c
Compare
d5aa48c to
8702248
Compare
…ailure Extend the transient-compaction retry to the two remaining surfaces. When the ACP layer records `last_compaction_transient` as True, nothing was emitted this turn and the per-surface budget is unspent, the abandoned message runs again instead of being dropped; a permanent verdict keeps the give-up behaviour. Slack native handler: reset, respawn the session, and put the message back on the thread's own queue (`sessions.enqueue`), which the gateway's `_on_done` drain replays through `_dispatch_queued`. Budget per thread. Shared messaging pipeline (`drive_turn`): the driver renders through a guard that holds back the abandoned attempt's DONE, the session is reset, and the message runs again into the same still-open renderer. Budget per turn. Closes #8256
… nested call A Stop landing between the abandoned attempt's reset and the successor's acquire finds no live session, so nothing recorded it and the replay would run the prompt the user just stopped. The session manager now records Stops for a key inside a "replay gap" the pipeline opens around that window (`open_replay_gap` / `close_replay_gap`, `note_stop`), the direct-cancel channel stop paths (Teams via `stop_running_turn`, Webex, WeCom, Weixin) record before their busy check, and both replay sites re-read the count right before running the message again. The Slack replay is a nested `handle_message` call with every argument unchanged instead of a queue entry: the interaction dispatch paths have no queue drain, a queue entry loses the attachment temp files to the outer task's cleanup, and the abandoned attempt must persist nothing so the conversation log carries the message once. The retry notice is posted directly and never recorded; the budget travels with the message.
8702248 to
151ab03
Compare
What
A turn abandoned after a transiently failed compaction is no longer dropped on the two surfaces PR #8215 left out. When the ACP layer reports
last_compaction_transientasTrue, nothing has been emitted this turn, and the surface's retry budget is unspent, the abandoned message runs again. A permanent verdict keeps the current give-up behaviour on both surfaces.src/kiro_crew/slack/handler.py— the native Slackhandle_messageSTOP_REASON_COMPACTION_FAILEDbranch.src/kiro_crew/messaging/dispatch.py—drive_turn'sSTOP_REASON_COMPACTION_FAILEDbranch (the shared pipeline for Teams, Webex, WhatsApp, Feishu, iMessage, WeCom, Weixin).Why
KAS runs summarization as a separate billed model call on the session's own model, so a
MODEL_TEMPORARILY_UNAVAILABLEthrottle fails the summarization, the backend abandons the turn, and the surface seesSTOP_REASON_COMPACTION_FAILED. #8215 taught the dashboard to tell a throttled compaction from an overflowed one (compaction_failure_is_transient,last_compaction_transientonLLMProvider) and to re-queue the message for the former. Both of these sites still returned unconditionally after the reset, so on Slack and every channel on the shared pipeline a throttle the next attempt would clear silently ate the user's message.Live evidence on the base (
39e732b8a):slack/handler.py:4168andmessaging/dispatch.py:818both reset and return with a "No re-queue" comment; neither reads the verdict.How
Each surface replays through its own mechanics — neither is a port of the dashboard's
_queue_recovery.Slack (native handler). The replay is a nested
handle_messagecall with every argument unchanged, made from theSTOP_REASON_COMPACTION_FAILEDbranch after the reset. That is what makes it Slack's own replay rather than a re-dispatch: it resolves the same session (linked, pinned or self-owned), keeps the same activation (review mode stays private), runs while the outer task still owns the attachment temp files the text refers to (their cleanup is the dispatcher's task-done callback, which fires after the nested call returns), and needs no queue drain from whoever dispatched the original — the interaction paths dispatchhandle_messagewith only a discard callback. The abandoned attempt sets_acquired = False(its permit died with the session the reset popped; the successor's belongs to the replay, whose ownfinallyreleases it), finalizes its own reaction ladder and stall watchdog, deletes its "Working…" block and any 💭 reasoning placeholder a thinking-only attempt left above where the answer would have gone, posts⟳ Compaction failed — retrying…directly (visible, never persisted, never mirrored), and returns as soon as the nested call does — so the conversation log records the user's message exactly once, with the reply the replay produced. The budget travels with the message as_CompactionReplay.attempt(_COMPACTION_FAILED_RETRIES = 2, same count and reasoning as the dashboard); no per-thread state. The nested call receives the user's text as it arrived (_user_text, captured before the cancelled-turn preamble is folded intotextfor the model after a soft-cancel): the replay derives its own preamble from its own one-shot gate, and the conversation log records what the user typed, never a preamble a previous attempt prepended.An earlier head of this PR did this through
sessions.enqueue+ the gateway's_on_donedrain; the GPT lane's three fenced findings on that head — interaction-path dispatches have no drain, the requeue lost attachment ownership, and the notice was persisted as a completed turn — are all properties of the queue route, and the nested call removes the route rather than patching each.Shared pipeline (
drive_turn). The pipeline has no queue of its own — five of the seven channels riding it keep none at all — and a channel's renderer is one message's output surface that finalizes on its firstDONE(_finalizedlatches in the WhatsApp/Teams renderers). So the replay happens inside the samedrive_turncall, into the same renderer: the driver renders through_TransientCompactionRetryGuard, aRendererthat forwards every event to the channel's renderer via its owndispatch(socurrent_tool_namebookkeeping lands where the handlers read it) and holds back exactly oneDONE— the abandoned attempt's, and only when the completion isSTOP_REASON_COMPACTION_FAILED, nothing was emitted through it (text, tool call or permission prompt), the provider's verdictis True, and a retry is still available. The acquire→build→drive block becomes a loop bounded by that budget: on a held completion the session is reset and the message re-runs on a freshly acquired session (the finally's singlereleaselands on the successor, as before). If the reset itself fails the heldDONEis released so the channel finalizes the reply rather than hanging. Budget: per turn,_COMPACTION_FAILED_RETRIES = 2.Everything else in the branch is unchanged: the reset still happens on every compaction-failed completion, and the post-turn bookkeeping (
record_success,persist, notice, audit) runs once for the turn that landed — with the first acquire'sis_new: a replay reacquires after a reset and may readis_new=Truefor a conversation that has existed for hours; that attempt-local value is right for building the replay's context (the fresh runtime needs the session-start injection again) and wrong for the new-conversation bookkeeping (title, dashboard surfacing), which would otherwise re-run over an existing conversation. "Emitted" also covers a consumed mid-turn steer: the backend folded a correction the replayeduser_textdoes not carry, so a steered turn keeps the give-up behaviour rather than silently dropping the accepted correction.A Stop during the reset gap keeps the message dropped (second commit). Between the abandoned attempt's
reset()and the successor'sget_or_createthe key has no live session. A channel/stop(or Slack!stop) landing there found nothing to cancel,stop_turnrecorded nothing for a session-less key, and the replay would have run the very prompt the user stopped — the GPT lane's fenced finding on the first head. The user's intent has to win over the recovery, the same way the dashboard's_stop_generationchecks make it win there, so:SessionManagergainsnote_stop(key),open_replay_gap(key),close_replay_gap(key)andawait_replay_gap(key)(session_lifecycle.py). The gap does two jobs across the reset->reacquire window: Stops stay recordable, and every other task'sget_or_createfor the key waits until the gap closes (get_or_createawaits it before taking its reservation; the owning task passes through). Without the second job a newer message for the same key arriving in that window would claim the successor first and run -- and persist -- ahead of the older message's replay. The owner closes the gap only when its whole turn has settled and the permit is released -- not at the successor claim: a waiter admitted then would park on the successor's semaphore, and a further retry's reset would pop that session from under it, stranding the message until a gateway restart. So one gap spans the whole retry sequence, and a message that waited behind it finds an idle successor.close_allreleases any waiter (their claim then meets the closing refusal), and a gap is also dropped on the per-key teardown paths. While a key sits in a replay gap, a Stop is recorded instop_requestseven though no session exists; outside one, a Stop on a session-less key still records nothing (the existing contract, kept so the dict grows only with conversations that exist, and its test still passes).stop_turnrecords throughnote_stopbefore its idle return. Gap membership and the session-less bucket are keyed canonically, so a Slack Stop issued under the bare thread ts and a turn reading underslack:<ts>meet in one bucket.stop_turnnow record first:stop_running_turn(Teams) and the Webex, WeCom and Weixin_handle_stopcallnote_user_stop(new,messaging/commands.py, probed withgetattrlike the rest of that seam) before their busy check. WhatsApp goes throughstop_turn; Feishu and iMessage offer no Stop. Slack's two!stophandlers (events.py_route_messageand thehandler.pyfallback) gatestop_turnonhas_session/ a registered task -- both false for a turn inside the gap that started from an interaction -- so they too callnote_user_stopbefore those checks -- against the thread's owning session (get_session_for_thread(thread) or key, the way the OPTIONS expiry resolves it), because a linked thread's turns and their replay run under the dashboard session that owns it and that is the key the replay reads. A discovery tripwire intest_messaging_dispatch.pyscans every*/transport_dispatch.pythat ridesdrive_turnand fails if one cancels directly without recording.sessions.stop_generation(key)when they acquire their session, open the gap before the reset, close it once the successor exists (and defensively in thefinally), and re-read the count as late as possible:drive_turnright before the replay'sdriver.run(the heldDONEis released instead, so the channel finalizes the abandoned reply), the Slack handler inside the replay, right beside the existing early-cancellation check that precedes the prompt (the replay carries the first attempt's snapshot in_CompactionReplay.stop_gen_at_entry, closes the gap once it has acquired, and ends without a turn if the count moved). The residual window is the same one every fresh turn already has between its last check and the stream opening.The same pre-replay look also covers supersession by
/new: every channel on the pipeline persists the new conversation generation before acknowledging (reserve_new_generation->SessionManager.reserve_generation), sodrive_turnsnapshotssessions.max_generation(bucket)for the key's bucket at its first acquire (session_conversation_generation, generation grammar fromsplit_dm_session_key) and re-reads it before the replay; a change means the conversation the message belonged to is over, and the heldDONEis released instead of running the retired prompt after the fresh-conversation ack. Keys without a generation grammar (Slack threads) and doubles without the reader read as 0 on both sides, so they can never manufacture a supersession. Slack has no/new; its session-removing commands (!agent,!ta,!project) change how the same thread continues rather than retiring it, so the nested replay simply runs under the new binding.Tests
Both directions per surface, plus the budget and the failure edges. Red on the unmodified base for the positive direction (verified with a
git archive maincopy: 7test_messaging_dispatch.pytests and 2test_slack_handler.pytests fail there; the negative-direction tests pass on base, as they pin existing behaviour).test/test_slack_handler.py::TestTransientCompactionRetry!stopfallback records the Stop on the manager before answering "Nothing running.", against the thread's owning session when linked (andtest_slack_events_coverage.pypins both for the primary!stophandler)text)attempt=1test_a_hook_reply_persists_only_after_the_replay_gap_resolves-- a hook auto-reply posts its canned answer at once, but its conversation-log row is written only after the key's open replay gap resolves (mutation-checked: red when the gap wait before the persist is removed)test/test_cross_surface_stop_gate.py— a Stop inside a replay gap is recorded without a session (bare and canonical Slack keys share the bucket); a closed gap records nothing again;note_stoprecords against a live session; the existing “no session records nothing” contract still holds; a second task'sget_or_createwaits behind an open gap and lands after the owner's claim and release (orderreplay, thennewer);close_allreleases a waiter. A claimant parked on a busy permit completes after that session is reset (no restart-only hang), and lands behind the successor when the resetting turn reacquires first. The torn-down turn's late key-only release cannot unlock the successor a woken waiter holds. A stranger task'sclose_replay_gapleaves the owner's gap open; the owner's close releases the waiter.test/test_messaging_commands.py—stop_running_turnrecords the Stop before its busy check.test/test_messaging_dispatch.pyDONEis delivered; the gap brackets exactly the reset→reacquire windowDONEthe channel sees is the replay's; post-turn bookkeeping onceDONEgoes throughis_new=True;persiststill receives the first acquire'sFalseandafter_persistdoes not runopen,release,close), including across the full budget/newduring the reset gap -> the message does not run again, the heldDONEis delivered, and nothing is recorded or persisted (successes == 0,persistnever called); the generation reader is a no-op for Slack-shaped keys and reader-less doublesDONEgoes throughDONEinstead of replayingdispatchand marks emitted for exactly text / tool call / prompt choicedispatchandon_turn_startguard.innertest_a_hook_reply_persists_only_after_the_replay_gap_resolves-- the hook auto-reply path rendersTEXT_CHUNK,DONEimmediately, then parks on the key's replay gap and callspersistonly after it resolves; no session is acquired or released (mutation-checked as above)Run: the four changed test files plus every channel dispatch/stop suite that rides
drive_turnorstop_turn(teams/webex/whatsapp/feishu/wecom/weixin/imessage/spool/identity/memory/session_control/stop_addresses_linked_session/discord/telegram/slack_transport_dispatch): 2698 passed, 5 xfailed; plus everytest/test_session*.py,test_subagent.pyand the Slack/discord/telegram suites: 4959 passed. Lint: black, isort, flake8, mypy,check_black_formatting.py,check_subprocess_encoding.py,check_comment_history.pyall clean.Other
no-screenshots: backend-only; there is no UI surface in this change.slack/transport_dispatch.py, the default whenmessaging.use_transportis on) does not handleSTOP_REASON_COMPACTION_FAILEDat all — no reset, no retry. Also observed while tracing: asessions.reset()discards any user messages already queued on that session (_unlink_session_queue), so on Slack a compaction-failed turn already strands queued follow-ups today; this PR does not change that.reset()now wakes tasks parked on the popped session's turn permit (_wake_turn_waiters,session_lifecycle.py). Before, a message that found the key busy waited on that session's semaphore; popping the session left it there for good, because the permit stayed held by the turn being torn down and that turn's own release lands on the successor. That was baseline behaviour on the queue-less channels whenever a turn ended in a compaction failure, and the retry loop could reach it again through waiters admitted after a gap closed. The popped permit is released once in the same tick as the pop; the woken claimant's re-validation sees the session gone and -- the second half, insession_allocation.py-- re-entersget_or_create(bounded like the won-race retry it mirrors) instead of cold-starting in place, so it meets an open replay gap or the successor at the front door. Waking waiters promptly widens a documented hazard, though:releaseis key-only, so once a woken waiter has put a successor under the key, the torn-down turn's own latereleasewould land on that successor and unlock a turn still in flight. So every acquire now records the holding task (_Session.turn_owner),resetremembers the holder of the permit it popped (orphaned_holders, keyed canonically), andreleaseabsorbs exactly that task's one owed release (absorb_orphaned_release); a task that resets and then reacquires the same key -- a replay -- has its record cleared at the successful claim (adopt_turn), so the release it makes for the successor is genuine. ThreeSessionManagertests pin it: a parked claimant completes after a reset; it lands behind the successor when the resetting turn reacquires first; and the torn-down turn's late release leaves the successor busy until its own holder releases (absorbed exactly once).finally(after the release), so a cancellation landing in the reset await (!stopcancels the handler task) cannot leave the key waiting forever. Thatfinallyruns on every turn, so the call is probed withgetattrlike the rest of the seam (the focused session-manager doubles across the suite predate the method).await_replay_gaphelper inmessaging/dispatch.py, used by both the pipeline hook path and the Slack handler's). A hook reply never callsget_or_create, so the allocation-side wait did not fence it: arriving while an older message on the key sat between its reset and its replay, its record would have filed ahead of the replayed turn the reader saw first. The reply itself still goes out immediately; only the record waits. Probed withgetattrlike the rest of the seam.mainfolded in the post-compaction re-injection bookkeeping (consume_reinjection/rearm_reinjection) and Slack's deferred permit release (success booked after delivery). Two consequences handled here: the Slackfinallydefers the gap close together with the permit (_release_permitcloses both), and a raise that leaves the turn body after a clean model completion -- a cancellation landing in the post-compaction reset -- releases the permit in thefinally(_body_completed) instead of deferring it to a_release_permitthat never runs. An attempt handed to a replay does not re-arm the one-shot re-injection flag: the replay cold-starts a new session whose first prompt carries the full session-start context.close_replay_gapcloses only when the calling task owns the gap. Both pipelines close from afinallythat runs on every turn for the key, so a concurrent same-key turn (a hook auto-reply that never acquires a session, an interaction-originated Slack turn) would otherwise end the owner's gap early and let a newer message claim the successor ahead of the replay. Teardown paths andclose_allkeep the forced discard./new) before it opens its prompt returns out of the turn after releasing the heldDONE: nothing ran, so norecord_successand nopersist-- a cancelled message is not filed as a completed turn with an empty reply. Thefinallystill closes the renderer, releases the permit and closes the gap.chat_runner._session_stop_generation_foris the dashboard's copy of the defensivestop_generationreader; the shared pipeline'ssession_stop_generation(messaging/dispatch.py) is the same read for the channel surfaces. Folding the dashboard's onto it means touchingchat_runner.py, which this PR does not; future cleanup.LLMProvider.last_compaction_transientthat fix(acp): retry a turn abandoned after a throttled compaction #8215 already centralised — so there is no new list to fold; the existing three (compaction_failure_is_transient,llm_helpers.acp_error_is_transient, the chat-runner 5xx path) remain future work.record_success/inc_message_successonce and the replay counts again, so a retried message shows as two successful turns inStats— the same over-count the dashboard's recovery has; not changed here.Pattern harvest
Rule candidate: grep
Pattern: a per-key state you open in one turn and close in a
finallyneeds an owner check on the close -- the samefinallyruns for every concurrent turn on that key, and any of them can end your window early. Also: before replaying a message through a surface's queue, grep every dispatcher of that surface for the drain callback — a queue entry is only a replay where something drains it; when dispatch paths differ, a nested call of the handler with the original arguments is the replay that works on all of them.Closes #8256