Skip to content

fix(dashboard): cancel native turns before timeout unwind - #16301

Closed
junjiequ wants to merge 1 commit into
kirodotdev:mainfrom
junjiequ:fix/native-turn-timeout-cancel
Closed

junjiequ wants to merge 1 commit into
kirodotdev:mainfrom
junjiequ:fix/native-turn-timeout-cancel

Conversation

@junjiequ

@junjiequ junjiequ commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Ensure a timed-out chat turn stops its native model work before the dashboard reports the timeout.

When a guarded chat turn reaches its wall-clock ceiling, Kiro Crew cancels the local Python stream task. That local unwind can mark the ACP turn inactive without sending session/cancel, so the native turn and its built-in subagents can continue editing files or requesting tools after the dashboard has stopped listening.

Why it matters

A timeout must stop the work it reports as stopped. Otherwise invisible native work can continue for minutes, mutate the workspace, strand queued results, and leave later Stop requests with no active turn to cancel.

Not a goal

What changed (motivation → approach → change)

Observed symptom: a timed-out dashboard stream stopped locally while its native ACP turn continued unattended.

Root cause: the timeout callback cancelled _run_chat first. Stream teardown then cleared active-turn state before any native cancellation was sent, so later cancellation resolved no_turn even though the underlying prompt was still running.

Approach: use the session lease's existing owner task as the generation fence. At the deadline, request native cancellation only while that exact task still owns the session, wait for the provider's bounded acknowledgement, and only then unwind the local stream. This prevents a stale timeout from cancelling a successor turn.

Change: guarded and off-loop timeout dispatches now share the owner-fenced cancellation sequence and one timeout-reporting path. A provider-confirmed completed turn skips redundant cancellation and replay. Unfinished no_turn, timeout, and error outcomes preserve partial output and show that resuming is unsafe because native cancellation was not acknowledged.

Implementation map:

  • src/kiro_crew/dashboard/turn_dispatch.py: deadline ordering, acknowledgement handling, and shared timeout reporting.
  • src/kiro_crew/session_lifecycle.py and src/kiro_crew/session.py: exact-owner cancellation and provider-confirmed completion proof.
  • src/kiro_crew/slack/gateway.py and src/kiro_crew/apps/builtins/spec_builder/backend/runtime.py: pass session state and slot identity to the shared timeout wrapper and consume task exceptions.
  • docs/system-specs/modules/learn-cron-dashboard.md: document the owner-fenced native cancellation contract.
  • .github/black-baseline.txt: remove the cancellation consistency test after this change makes it Black-clean.

Backwards compatibility

Compatible: existing callers keep their timeout behavior and signatures remain compatible through optional cancellation context; production call sites now supply session state and slot identity for native cancellation.

Removes nothing: the change adds owner-fenced cancellation and more precise timeout reporting without deleting or rejecting an existing input, API, or recovery path.

Tests

  • test/test_turn_dispatch.py: deadline ordering, bounded acknowledgement, repeated cancellation, provider-confirmed completion, replay, and fail-closed timeout cards.
  • test/test_session.py: exact-owner cancellation, wrong-owner refusal, completion proof, and pre-await replay marking.
  • test/test_chat_turn_timeout_consistency.py: every timed dispatch uses a valid guarded form, supplies state and slot identity, and consumes task exceptions.
  • test/test_spec_builder_routes_coverage.py: Spec Builder passes the exact state and slot into the shared timeout wrapper.
  • Final affected suite: 926 passed, including the overlapping cron-routing tests.
  • Black, isort, flake8, mypy (2,063 source files), documentation lint, and git diff --check passed on the rebased head.

Manual verification

N/A — the change is an async lifecycle and ownership repair with deterministic race coverage; it does not change a rendered UI surface.

Screenshots / video

N/A — no user-visible frontend files or layout behavior changed.

Related Issues

no linked issue: #14058 also tracks stage-boundary, queued-message, and Stop recovery outside this focused native-cancellation repair.

Related but non-overlapping: #15918 reports an honest response when Stop finds an already-idle wedged slot; this PR prevents the native turn from being abandoned at the timeout boundary.

Pattern harvest

Rule candidate: review-prompt
Pattern: a timeout around a remote stream must stop and acknowledge the remote operation before cancelling the local consumer, fenced by the exact operation owner so a stale timeout cannot cancel successor work.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@junjiequ
junjiequ requested a review from a team as a code owner October 2, 2026 22:56
@junjiequ
junjiequ requested a review from bolichen97 October 2, 2026 22:56
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@junjiequ
junjiequ force-pushed the fix/native-turn-timeout-cancel branch from d74f3bd to b23ffee Compare October 2, 2026 23:39
@junjiequ

junjiequ commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in the docs system-spec module since this head's CI base (f0be829 -> 9eb318e). The same revision also replaces the Comment History Gate's flagged historical wording with a present-tense mechanism statement. The affected 899-test suite, comment-history gate, formatting, lint, mypy, docs, and whitespace checks pass locally; CI is re-running on b23ffee.

@junjiequ
junjiequ force-pushed the fix/native-turn-timeout-cancel branch from b23ffee to 4f3f57a Compare October 3, 2026 00:28
@junjiequ

junjiequ commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/slack/gateway.py and related cron-routing files since this head's CI base (9eb318e -> 8ef5ea4). The branch remains one commit and now removes test/test_chat_turn_timeout_consistency.py from the Black debt baseline because this change makes it Black-clean. The affected cancellation and cron-routing suite passes 926 tests; Black baseline, comment-history, isort, flake8, mypy, docs, and whitespace checks pass locally. CI is re-running on 4f3f57a. The prior readiness/desktop-matrix failures were GitHub installation rate limits, and the Windows installer exceeded its runner time ceiling.

@junjiequ

junjiequ commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 The current head is locally clean, but two status-publishing jobs failed only because the repository's GitHub App installation hit its API rate limit. The description check itself concluded success. I cannot rerun them because GitHub requires repository admin rights. Could you please rerun the failed jobs for Fork PR Description (https://github.com/kirodotdev/KiroCrew/actions/runs/37082233809) and PR Readiness (https://github.com/kirodotdev/KiroCrew/actions/runs/37082233908)?

Send an owner-fenced native cancellation before local timeout teardown so timed-out chat work cannot continue invisibly. Preserve partial output and fail closed when cancellation cannot be acknowledged.

Refs kirodotdev#14058
@bolichen97
bolichen97 force-pushed the fix/native-turn-timeout-cancel branch from 4f3f57a to 8c789d0 Compare October 6, 2026 09:45

Copy link
Copy Markdown
Contributor

Rebased onto latest main (cc7e79894e), which was 492 commits ahead.

  • Head: 4f3f57a113 → 8c789d0425
  • Conflicts: none (clean rebase)
  • Range-diff: 1:1; the only changes are in surrounding context in session_lifecycle.py, and the patch content is unchanged
  • Verification: py_compile passed on the changed .py files; test_chat_turn_timeout_consistency, test_session, test_spec_builder_routes_coverage, test_turn_dispatch: 899 passed

Pushed with --force-with-lease against the previous head.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 8c789d0425979e7fb7100f6f1aadfe7d667a4b3a via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that timed-out subagent and Spec Builder turns showing a new timeout card was intended. The description never says these surfaces gain a card.

Not justified as shipped

  • Item 5 — undeclared: bounded_chat_turn now calls _report_turn_timeout, so timed-out subagent-recovery and Spec Builder turns put a card in the chat. Before, the gateway _done callback only logged them. The description only says "one timeout-reporting path".
  • Item 7 — rides along: the Spec Builder _consume_turn_result logger only exists because the PR widened _is_create_task_wrapped_run_chat_dispatch to cover bounded_chat_turn.
  • Item 8 — rides along: the rewritten "Per-turn timeout" paragraph drops the list of dispatch sites and the _STALE_TURN_TIMEOUT (90s) explanation. Neither is part of documenting the cancellation contract.

What this change ships

Inventory (8 items) — 5 justified

Intent: a chat turn that times out must actually stop its native model work before the dashboard says it stopped — FIX.

  1. At the deadline, a native session/cancel goes to the exact lease owner, and the local stream unwinds only after the bounded acknowledgement — justified
  2. A timeout from a stale turn can no longer cancel the turn that follows it (turn_owner is not owner_task) — justified
  3. The timeout card now warns not to resume when the cancel was not acknowledged (timeout, no_turn or error) — justified
  4. An acknowledged timeout cancel marks the turn for replay on the next prompt, the same way soft Stop does — justified
  5. Subagent and Spec Builder timeouts now show a timeout card — undeclared
  6. Stop or shutdown of the wrapper now waits for the inner task to finish (shield) — justified
  7. Spec Builder now logs turn exceptions that would otherwise go unretrieved — rides along
  8. The spec's per-turn-timeout paragraph is rewritten and parts are dropped — rides along

[FIRST-PRINCIPLES-REVIEWED] 8c789d0

Description read

[DESCRIPTION-READ] 86f26e810d144267fbfe8e5c58deb2209f4f0dcd58ea0a0071cca17853dcd1a1

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 8c789d0425979e7fb7100f6f1aadfe7d667a4b3a via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The unacknowledged-cancel card tells users "do not resume until it becomes idle", but nothing shows idleness or stops the next prompt.

Watch

  • Fail-closed in name only. On timeout/no_turn/error, format_turn_timeout_card says "Do not resume this session until it becomes idle". Meanwhile owner_task.cancel() unwinds the slot, so the dashboard shows it idle right away. cancel_owned_turn has also already set prev_turn_cancelled = True. The user's next message therefore replays into a native turn that may still be running. The only guard is a warning the user has no signal to act on, which leaves the "invisible native work" harm from ## Why it matters in place on exactly this path.
    Clears when: the unacknowledged outcome leaves a state the user can see (for example, the slot stays busy or flagged until has_unfinished_turn reports False), or the description says the warning-only behaviour is deliberate and the card stops promising an "idle" signal the dashboard never shows.
  • Changed task identity is not listed. Gateway and Spec Builder turns now run _run_chat in an inner task created by _bounded_turn, where the wait_for they used before ran it inline in the same task. So turn_owner and orphaned_holders now point at a task other than slot.task. Removes nothing does not cover this change. decision_outbox.py:321 already expects it; no other slot.task-vs-owner comparison turned up.
    Clears when: a Compatible: line names the turn_owner ≠ slot.task change and points to a test showing that orphan-release and outbox settlement still work on the bounded_chat_turn path.

[DESIGN-REVIEWED] 8c789d0

Description read

[DESCRIPTION-READ] e40fbad7c121565a6925ebeb1764ac078fa8a5fb1e0b1339c7bedd5e412a85a7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 8c789d0425979e7fb7100f6f1aadfe7d667a4b3a via the fork AI-review pipeline; updated in place on each push.

Review details

One advisory finding; nothing blocks the merge.

FINDING — src/kiro_crew/slack/gateway.py:9749 — bounded_chat_turn now re-raises TurnTimeoutError even when _run_chat has already absorbed the cancellation. The old asyncio.wait_for path returned normally in that case. So when a sub-agent injection turn hits the turn ceiling after the model has consumed the announce, if not t.cancelled() and t.exception(): calls notify_injection_failed. That sets _report_undelivered and appends an "Agent ❌ … ⚠️ Result delivery failed" card next to the timeout card, for a report that was actually delivered and settled → Fix: in _on_inject_done, skip notify_injection_failed when the exception is a TurnTimeoutError and _consumed[0] is True.

[OPUS-REVIEWED] 8c789d0

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 8c789d0425979e7fb7100f6f1aadfe7d667a4b3a via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8c789d0

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
iamwhatever pushed a commit that referenced this pull request Oct 9, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

Replaces #16301 by @junjiequ.

Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
iamwhatever pushed a commit that referenced this pull request Oct 9, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.

Replaces #16301 by @junjiequ.

Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks @junjiequ — your fix is re-authored as #18417 (rebased onto main so CI and the fork review lanes could run) and is green there. Closing this in favor of #18417, which keeps you as co-author. 🙏

@iamwhatever iamwhatever closed this Oct 9, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 9, 2026
iamwhatever pushed a commit that referenced this pull request Oct 10, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.

Replaces #16301 by @junjiequ.

Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
iamwhatever pushed a commit that referenced this pull request Oct 10, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.

Replaces #16301 by @junjiequ.

Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
iamwhatever pushed a commit that referenced this pull request Oct 10, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.

Replaces #16301 by @junjiequ.

Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
iamwhatever pushed a commit that referenced this pull request Oct 10, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.

Replaces #16301 by @junjiequ.

Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Oct 11, 2026
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.

Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.

The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.

Replaces #16301 by @junjiequ.

Co-authored-by: iamwhatever <26782336+cixuuz@users.noreply.github.com>
Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants