Skip to content

fix(slack): clear working indicators on CancelledError mid-stream - #20353

Merged
iamwhatever merged 2 commits into
mainfrom
fix/11448-cancelled-working-indicator-leak
Oct 29, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/11448-cancelled-working-indicator-leak

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Clear the native "…is working" Slack thread-status label and the inline Stop block on every turn exit, including when the turn coroutine is cancelled mid-stream.

A Slack turn can be cancelled at any await: a !stop, a warm-pool reclaim, or a gateway shutdown. asyncio.CancelledError is a BaseException, so it passes every except Exception arm. The clears for both indicators sit at hand-listed exits in the turn and delivery phases, so a cancel skips them. The "…is working" label and the Stop button then stay in the thread for good.

Why it matters

Users see a stuck "…is working" label and a dead Stop button after a stop, a gateway restart or a session reclaim.

Not a goal

What changed (motivation → approach → change)

The clear now lives at the one exit every phase shares. handle_message wraps its three phases in except asyncio.CancelledError:. That arm starts turn.clear_working_indicators() as a tracked task and re-raises at once. The turn records the Stop block ts as soon as it posts it, so the arm can find it from any phase. Delivery marks the indicators cleared once it takes them down, so a later cancel does not remove a review-mode status delivery chose to keep.

This replaces the first version of this PR, which patched handle_message before it was split into handler_runtime/ phases (#21040). It also answers both review findings: a cancel in the delivery region is now covered, and the shutdown case is best-effort.

Backwards compatibility

Compatible: only a cancelled turn behaves differently, and it now clears what it posted.

Removes nothing: no API, function, key or data path is removed or renamed.

Tests

TestWorkingIndicatorCleanup in test/test_slack_handler_working_indicators.py:

  • A cancel mid-stream clears the thread status and deletes the Stop block.
  • A cancel in the delivery phase, before the delivery clear, clears both.
  • A cancel after delivery cleared them adds no second clear.
  • In review mode, a cancel before the draft lands still clears the status.
  • The cancel still propagates, also when the Slack clear call raises.
  • A normal turn still clears the status.
  • A cancel that unwinds the parent after a compaction replay leaves the replay's status alone.
  • A compaction replay clears the abandoned attempt's indicators before the nested turn starts, so a cancel that lands before the nested turn sets its own status leaves no stale "…is working" status (test_a_replay_cancelled_before_its_status_clears_the_first_attempts).

test/test_handler_turn.py lists the new turn coroutine for the sync-I/O plant guard.

Proof: PROVEN test_cancelled_turn_clears_thread_status test_cancelled_turn_deletes_working_block test_cancel_in_delivery_clears_both_indicators (head e5e69e0)

Manual verification

N/A — unit coverage sufficient: the tests drive handle_message with a cancel at each phase against the mock Slack client.

Related Issues

Fixes #11448

Sibling sweep

Swept: git grep -nE "set_thread_status(.*(_STATUS_WORKING|is_typing)|build_working_blocks(" -- src/
Sibling: src/kiro_crew/slack/handler_runtime/turn.py -- changed here
Sibling: src/kiro_crew/slack/handler_runtime/stream.py -- not affected -- refreshes the status inside the same turn, cleared by the new arm
Sibling: src/kiro_crew/slack/renderer.py -- tracked in #23619

Pattern harvest

Rule candidate: review-prompt
Pattern: a working indicator cleared only at listed exits, with no clear on the shared CancelledError exit

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 14, 2026 15:26
@CrysisDeu
CrysisDeu requested a review from mbriones98 October 14, 2026 15:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f2df761c9ab83fb29aa899779724e57b25d031fa — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f2df761

Verdict parsed from the review's SHA-scoped output markers for commit f2df761c9ab83fb29aa899779724e57b25d031fa.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f2df761c9ab83fb29aa899779724e57b25d031fa false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f2df761c9ab83fb29aa899779724e57b25d031fa — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: one shared cancel exit with an idempotent, ownership-aware clear fixes the root cause, not each exit one by one.

Notes

  • The cancel arm's clear is a detached task, so it can land after the same thread's next turn has set its own "…is working" status (for example, !stop followed by a quick new message), and it would hide that label. It is worth a test that orders those two calls.

[DESIGN-REVIEWED] f2df761

Description read

[DESCRIPTION-READ] ac170d5db6e20f6a419a718485bf3202ecab8688085f9d363c78bff3538a2b51

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f2df761c9ab83fb29aa899779724e57b25d031fa touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of f2df761c9ab83fb29aa899779724e57b25d031fa — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix sits on the one exit all three phases share, and the leftover early-exit clears (turn.py:807–871) and the renderer gap are declared and deferred to #24947 and #23619.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop a cancelled Slack turn (!stop, warm-pool reclaim, shutdown) from leaving a stuck "…is working" label and a dead Stop button. FIX (#11448).

  1. A turn cancelled in any phase now clears the thread status and deletes the Stop block — justified
  2. The Stop block's ts is recorded on the turn as soon as it is posted, so a cancel in any phase can find it — justified
  3. Delivery's normal and cooperative-cancel exits clear through one helper; a failed status clear is now logged instead of raised — justified
  4. In review mode, a cancel before the draft lands now clears the status; once the draft is posted, the draft owns "Awaiting review…" — justified
  5. A compaction replay clears the abandoned attempt's status before the nested turn starts (two golden digests move) — justified

[FIRST-PRINCIPLES-REVIEWED] f2df761

Description read

[DESCRIPTION-READ] 8d8982876ee9d2d9e52a408a0f3d4cd9d54e437f1ff60790a97fc7e3cb1a5392

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@CrysisDeu
CrysisDeu force-pushed the fix/11448-cancelled-working-indicator-leak branch from a19a409 to f6bf3d9 Compare October 14, 2026 16:05
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on f2df761c9ab83fb29aa899779724e57b25d031fa; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- src/kiro_crew/slack/handler.py:1032 -- Detached cleanup clears a successor turn’s status

track_background_task(asyncio.create_task(turn.clear_working_indicators()))
Trigger: On the native Slack path, !stop cancels a turn before its indicators are cleared while another member’s reply is queued. The queue drain dispatches that reply through _dispatch_queued → handle_message → _run_turn while the abandoned cleanup awaits a Slack connection retry.
Outcome: The successor sets its working status, then the abandoned turn’s delayed set_thread_status(..., "") erases it while the successor is still running. Background-task tracking neither holds queue admission nor orders these writes.
Anchor: defect
Fix: Order each thread’s pending cleanup before the successor’s status write, while preserving cancellation propagation.
[BLOCK-MERGE-DOWNGRADED] f2df761
[GPT-REVIEWED] f2df761

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 is LOW harm. The failure needs four things at once: a !stop cancel on the native Slack path, a reply from another member already queued in the same thread, that reply's turn setting its status (turn.py:399), and the detached clear at handler.py:1032 → turn.py:287 (a single setStatus call, client.py:877) being held up by a Slack retry long enough to land after the successor's write. What breaks is only the "working" status, which is decoration: the successor turn still runs and delivers its reply, and the status is written again on its next tool call (stream.py:808) or status refresh (stream.py:736).
The real fix needs a per-thread ordering contract between the detached cleanup and the queue drain (_dispatch_queued), plus new per-thread state that has to keep cancellation propagating. That mechanism would exist only to stop a brief, self-healing status flicker, so its cost is higher than the harm it prevents.

[ADJUDICATION] f2df761c9ab83fb29aa899779724e57b25d031fa total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/slack/handler.py:1032 reason=disproportionate-remedy
[GPT-ADJUDICATED] f2df761c9ab83fb29aa899779724e57b25d031fa

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f2df761c9ab83fb29aa899779724e57b25d031fa false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 14, 2026
mbriones98
mbriones98 previously approved these changes Oct 14, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at f6bf3d9; no behavior change.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at f6bf3d9.

The new except asyncio.CancelledError arm correctly clears both working indicators for a cancel inside the main try, re-raises cleanly, and CI is green. Two current-head AI CONCERNS verdicts have no author reply yet:

  1. Design Review and First Principles both point out that a cancellation landing in the delivery region (the first yield at handler.py#L2658, or any await before the clear at L2688-L2697) exits through the structural finally at L3353, which only releases the permit. I confirmed that path in the code. The Goal says "every turn exit", so please either move the clear into a structural exit path (plus a test that cancels during delivery), or narrow the Goal and say that window is left unfixed.
  2. Design Review asks for evidence that the background cleanup task survives gateway shutdown (cancel_background_tasks plus loop teardown), or for the shutdown claim to be dropped from the description.

What clears this: one disposition per item on the PR. Minor: the Tests section names test/test_slack_handler.py; the tests are in test/test_slack_handler_turn_outcomes.py.

— crew-pr-reviewer (r101)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/11448-cancelled-working-indicator-leak branch from f6bf3d9 to dfadbdd Compare October 24, 2026 01:33
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebuilt on latest main at dfadbdd. The first version patched handle_message before #21040 split it into handler_runtime/ phases, so it no longer applied.

Dispositions for the HOLD at f6bf3d9:

  1. Delivery-region cancel — fixed. The except asyncio.CancelledError: arm now wraps all three phases in handle_message, so a cancel at the first yield in _deliver_answer or at any await before its clear is covered. The turn records the Stop block ts as soon as it posts it. Delivery marks the indicators cleared, so a later cancel does not remove a review-mode status. New test: test_cancel_in_delivery_clears_both_indicators (red on main), plus test_cancel_after_delivery_cleared_adds_no_second_clear.
  2. Shutdown claim — narrowed. The cleanup task is registered with track_background_task, so cancel_background_tasks at shutdown may stop it. The description and code comment now call the shutdown case best-effort.

Minor: the Tests section now names test/test_slack_handler_turn_outcomes.py.

Sibling found in the sweep: the transport SlackRenderer.close() has the same gap, tracked in #23619.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 24, 2026
@iamwhatever
iamwhatever force-pushed the fix/11448-cancelled-working-indicator-leak branch from dfadbdd to e5e69e0 Compare October 24, 2026 01:45
@iamwhatever

Copy link
Copy Markdown
Contributor

GPT 6.1 findings at dfadbdd, addressed at e5e69e0:

  • F1 (upheld) — fixed. A review turn now sets _indicators_cleared only after _post_review_draft succeeds. A cancel before the draft lands still clears the status. New test: test_review_cancel_before_draft_lands_clears_status (red without the delivery change).
  • F2 (downgraded to LOW by adjudication) — not-a-defect for this PR. The race needs a cancel, a queued follow-up and a late set_thread_status(""). The next turn still delivers its answer and clears its own status; only its working hint can be lost. A per-thread handoff would be a new cross-module ordering rule for that rare cosmetic case.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 24, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at e5e69e0.

The fix is in the right place now. handle_message wraps all three phases in one except asyncio.CancelledError: arm that starts a tracked clear and re-raises (handler.py#L1032-L1043), so the r101 delivery-region gap is closed and has a test. The shutdown claim is narrowed to best-effort, and the renderer sibling (#23619) is already fixed by #23662. CI is green.

These current-head review items have no reply yet:

  1. First Principles Item 5 and the Design Review note: clear_working_indicators (turn.py#L278) repeats the inline clears still in _deliver_answer, and _indicators_cleared is set by hand at three places (delivery.py#L178, L195, L342). Route those clears through turn.clear_working_indicators(), or say why the two spellings should stay.
  2. GPT 6.1 F1 on this head (downgraded to LOW): your earlier reply covers the successor-status race, but not its second case. A cancel during transcript persistence after a compaction replay (turn.py#L521) unwinds the parent turn, whose flag is still false, so it clears the replay's "Awaiting review…" status. A one-line rebuttal or a fix is enough.

To clear: one reply per item. I read the code and tests but did not run them.

— crew-pr-reviewer (r124)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the fix/11448-cancelled-working-indicator-leak branch from 74c30b7 to 415ac04 Compare October 26, 2026 14:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES at 415ac04, for follow-up tracking only.

The fix is sound. One except asyncio.CancelledError arm in handle_message covers every phase, the turn owns the clear and the flag, and the replay hand-off at turn.py#L536 answers both items from my r124 HOLD. The new tests cover each cancel phase, and CI is green.

Two items in your latest dispositions are deferred with no linked issue:

  1. The four inline status/Stop-block clears in _run_turn's early exits (turn.py#L798-L866), which First Principles flags as a second spelling of clear_working_indicators.
  2. A review turn whose draft post fails leaves the thread status set, because that exit returns before release_working_indicators without clearing.

Please either fix them here or open one issue per item and link both in the PR description. The GPT successor-status race reply is acceptable as a cosmetic trade-off.

— crew-pr-reviewer (r127)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 26, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

@iamwhatever re r127 at 415ac04: both deferred items now have their own issue, and the description links them under "Not a goal":

  1. Four hand-written status/Stop-block clears in _run_turn's early exits, a second spelling of clear_working_indicators: Slack _run_turn early exits clear the working indicators by hand instead of clear_working_indicators #24947
  2. A review turn whose draft post fails leaves "Awaiting review…" set, because that exit returns before release_working_indicators without clearing: Slack review turn leaves "Awaiting review…" status when the draft post fails #24948

The GPT successor-status race stays accepted as a cosmetic trade-off, as you noted. The code is unchanged and the head is still 415ac04.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 28, 2026
A cancel (!stop, warm-pool reclaim, shutdown) passes every
`except Exception` arm, so the hand-listed clears in the turn and
delivery phases never run and the "...is working" status and inline Stop
block stay in the thread. handle_message now catches CancelledError
around all three phases, schedules turn.clear_working_indicators() as a
tracked task and re-raises. The turn publishes _working_ts as soon as it
posts the Stop block, and delivery marks the indicators cleared so a
later cancel does not remove a review-mode status it kept.

Fixes #11448

Co-authored-by: Zezhen Xu <zezhexu@amazon.com>
@CrysisDeu
CrysisDeu force-pushed the fix/11448-cancelled-working-indicator-leak branch from 415ac04 to b2b65d7 Compare October 29, 2026 02:11
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased on current main (resolved a conflict with the new status_ctrl field) and pushed b2b65d7a0b.

GPT F1 (upheld): fixed. The compaction replay now awaits turn.clear_working_indicators() before the nested handle_message, instead of only releasing ownership. A cancel that lands before the nested turn sets its own status no longer leaves the first attempt's status behind. New test: test_a_replay_cancelled_before_its_status_clears_the_first_attempts.

GPT F2 (downgraded by adjudication): not changed. It is a cosmetic race on a successor's status, and the fix would need a new per-thread ordering contract.

The pre-stream early exits in _run_turn (#24947) are fixed in a separate PR stacked on this one.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 29, 2026
…he turn helper

_deliver_answer cleared the thread status and deleted the Stop block inline
and set _indicators_cleared by hand, a second spelling of
_SlackTurn.clear_working_indicators. Delivery now calls the helper
(keep_status for a review turn) and release_working_indicators() once a
review draft lands, so the turn owns both the clear and the flag.

A compaction replay now clears the abandoned attempt's indicators through
the helper before the nested handle_message runs. A cancel that lands
before the nested turn owns a status leaves no stale status, and a cancel
that unwinds the parent after the replay does not clear the replay's.

Refs #11448

The working-indicator tests move to test/test_slack_handler_working_indicators.py
so test/test_slack_handler_turn_outcomes.py stays under the 2,000-line cap.
@CrysisDeu
CrysisDeu force-pushed the fix/11448-cancelled-working-indicator-leak branch from b2b65d7 to f2df761 Compare October 29, 2026 02:57
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 29, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at f2df761.

A cancel in any phase now clears the "…is working" status and the Stop block. One except asyncio.CancelledError arm wraps all three phases (handler.py#L1022-L1033). Delivery and the compaction replay clear through the same idempotent helper (turn.py#L282-L302). The tests cover a cancel in each phase, review mode and both replay orders. The two deferrals from our r127 review are tracked in #24947 and #24948, and the renderer sibling is #23619. All checks are green.

AI findings judged non-blocking:

  • GPT 6.1 (downgraded) and the Design note: the detached clear can erase a queued successor's status (handler.py#L1032). The flicker is cosmetic: the successor still delivers and sets its status again on its next tool call. Fixing it would need a per-thread ordering contract.

I read the diff and CI. I did not run tests.

— crew-pr-reviewer (r134)

@iamwhatever
iamwhatever merged commit 3297d4e into main Oct 29, 2026
155 of 167 checks passed
@iamwhatever
iamwhatever deleted the fix/11448-cancelled-working-indicator-leak branch October 29, 2026 07:09
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slack "…is working" status and inline Stop button leak on turn cancellation

3 participants