Repository navigation
fix(dashboard): cancel native turns before timeout unwind - #18417
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The description's "Generated UI index" section describes a commit that regenerates Watch
[DESIGN-REVIEWED] 899d312 Description read[DESCRIPTION-READ] d98e3eecc7e72e94a0788f2a5fedbcf74df54107a9c1eb4a35644d6d953699de That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: when a chat turn times out, actually stop the native agent's work before reporting it as stopped. This is a FIX (#18416).
[FIRST-PRINCIPLES-REVIEWED] 899d312 Description read[DESCRIPTION-READ] 207f226dc2d551dc3dc87ff76fedea349fe483dfe77b26253ddd1932c7402a24 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
d07886a to
2be6f34
Compare
|
Self-review (crew-pr-reviewer, read-only): RECOMMEND APPROVE This fixes the root cause of #18416. Before, the wall-clock deadline cancelled |
buluoray
left a comment
There was a problem hiding this comment.
Thanks, this fixes a real problem: at the wall-clock ceiling the native turn now gets session/cancel while ACP still marks the turn active, and the dashboard waits a bounded time for the ack before it unwinds. I checked the main claims against the source and they hold:
- The lease owner is the inner task
_bounded_turncreates (turn_owneris set fromcurrent_task()at acquire inside_run_chat). Nothing is awaited between the owner check andprovider.cancelincancel_owned_turn, so a stale deadline can't cancel a successor turn. - The deadline is a single
call_later, so it sends at most one native cancel. A cancel that raises, or that exceeds ack + 0.5s, becomeserror/timeout, and thefinallystill cancels the inner task, so the slot can't hang. - Turns that complete normally take the unchanged path:
shield(task), then the result, then the handle is cancelled. - No runtime is reset or killed on an unacknowledged outcome, which matches the non-goal and the runtime-ownership rules.
- Reverting each of these alone turns a named test red: the cancel-before-unwind ordering, the owner fence, the known-terminal skip, and replay arming.
Requesting changes for one spec-sync item. Auto-merge is on, so I'm holding owning-spec accuracy to the merge bar:
-
cancel_owned_turnsetsprev_turn_cancelled = Truebefore the provider cancel, for every attempted outcome (acked,timeout,error,no_turn). The comment explains this is fail-closed on purpose, and I agree with the behaviour. The specs don't match it yet:- The new paragraph in
learn-cron-dashboard.mdsays "ackedmarks the cancelled prompt and partial assistant output for reinjection", which reads as acked-only. session.mdownssession.py/session_lifecycle.py. Its Stop Orchestration step 1 saysprev_turn_cancelledis set only after the ack, and its Cancelled-turn context restore section says the flag is set on soft-cancel success. The newSessionManager.cancel_owned_turnisn't described there.
Please add a short
cancel_owned_turnsubsection tosession.mdcovering: the exact-owner fence, the known-terminal skip, replay armed before the first await on every non-skipped outcome, and no hard reset on an unacknowledged outcome (unlikestop_turn). Please also correct the learn-cron-dashboard sentence. This is docs only, no code change. - The new paragraph in
Non-blocking follow-ups:
- The ack bound isn't pinned by a test. Replacing the
wait_for(native_cancel(...), ack + 0.5)with a plain await keeps the suite green. A test wherecancel_owned_turnnever returns, asserting the turn unwinds with atimeoutoutcome, would lock in the "no hung slot" guarantee. - The loop in
_settle_deadline_cancelthat swallows cancellation isn't independently covered.test_second_cancellation_cannot_orphan_native_cancelcancels the wrapper while it is still inshield(task), so making the settle loop re-raise also stays green. - A Stop or shutdown that arrives during the deadline's ack window is held until the native cancel settles (about 10.5s by default) and ends as
TurnTimeoutErrorinstead ofCancelledError. It's bounded and seems intended; one sentence in the spec would make it explicit. - On the ordinary Stop path, a
_run_chatthat swallows the cancellation now still ends the wrapper withCancelledError, where before it returned the result. That looks more correct, but it isn't mentioned in the PR. - The "do not resume until idle" card gives the user no signal for when the session is idle. This probably belongs with #14058.
CI on 2be6f34: 89 success, 9 skipped, 0 failures, nothing pending. I ran the touched test files at this head (904 passed). I didn't run the full suite or the linters locally; for those I'm relying on CI.
2be6f34 to
a96b598
Compare
|
@buluoray thanks for the careful review. The spec-sync item is addressed in a96b598 (docs only, no code change); the branch is also rebased onto current main.
From the non-blocking list I also took the one-sentence spec note: a Stop or shutdown arriving during the ack window waits for it to settle (bounded at Local: comment-history gate and |
buluoray
left a comment
There was a problem hiding this comment.
Approving at a96b598. This supersedes my CHANGES_REQUESTED on 2be6f34: the spec now matches the code.
docs/system-specs/modules/session.md "Cancelled-turn context restore" now names the two writers of _Session.prev_turn_cancelled: stop_turn() sets it only on an acked soft cancel (session_lifecycle.py:3068), and cancel_owned_turn() sets it before the provider cancel starts, so every outcome it attempts leaves it set (session_lifecycle.py:3010). The new "Owner-fenced deadline cancel" section describes the same sequence the code runs: exact-owner fence, known-terminal skip, arm replay before the first await, then provider.cancel on the exact provider read in step 1, with no reset() or hard kill on any outcome. Arming replay on unacknowledged outcomes is fail-closed and the spec now says so.
CI on this head: 83 success, 10 skipped, 5 failure. All five (Build Wheel, Frontend Lint & Type Check, E2E, Build Desktop x2) fail on gen-ui-index: ui-index.generated.json is stale. That is not this diff: the PR touches no website/ file and no generated index, and the same check fails on its merge-base 6166905. A rebase onto current main should clear it.
I re-read the code and spec on this head; I did not re-run the Python tests.
62e2304 to
74b0096
Compare
When a guarded chat turn reaches its wall-clock ceiling, cancel the
native ACP turn against the exact session lease owner and wait for the
provider's bounded acknowledgement before unwinding the local Python
stream. This stops native work (and its built-in subagents) from
continuing to edit files or request tools after the dashboard has
reported the timeout, and prevents a stale deadline from cancelling a
successor turn.
Guarded and off-loop timeout dispatches now share the owner-fenced
cancellation sequence and one timeout-reporting path. A
provider-confirmed completed turn skips redundant cancellation and
replay; unacknowledged outcomes preserve partial output and warn that
resume is unsafe.
The virtual-time turn harness is given the healthy-session answer for
the new SessionManager.cancel_owned_turn call ("acked"), and
turn_dispatch keeps off the agent-sdk boundary: CancelOutcome is spelled
as a local Literal rather than imported from kiro_crew.providers, so the
shrink-only boundary gate stays at its baseline.
Replaces #16301 by @junjiequ.
Co-authored-by: JJ_Q <5853857+junjiequ@users.noreply.github.com>
74b0096 to
899d312
Compare
|
Self-review (crew-pr-reviewer, read-only): RECOMMEND APPROVE This fixes the root cause of #18416: at the wall-clock deadline |
bolichen97
left a comment
There was a problem hiding this comment.
Approved at head 899d312: PR Readiness is green, no open change requests, and the review lanes passed. (AI-assisted review at bolichen97's request.)
Problem / Motivation
Goal: Ensure a timed-out chat turn stops its native model work before the dashboard reports the timeout.
When a guarded chat turn reaches its wall-clock ceiling, Kiro Crew cancels the local Python stream task. That local unwind can mark the ACP turn inactive without sending
session/cancel, so the native turn and its built-in subagents can continue editing files or requesting tools after the dashboard has stopped listening.Why it matters
A timeout must stop the work it reports as stopped. Otherwise invisible native work can continue for minutes, mutate the workspace, strand queued results, and leave later Stop requests with no active turn to cancel.
Not a goal
What changed (motivation → approach → change)
Observed symptom: a timed-out dashboard stream stopped locally while its native ACP turn continued unattended.
Root cause: the timeout callback cancelled
_run_chatfirst. Stream teardown then cleared active-turn state before any native cancellation was sent, so later cancellation resolvedno_turneven though the underlying prompt was still running.Approach: use the session lease's existing owner task as the generation fence. At the deadline, request native cancellation only while that exact task still owns the session, wait for the provider's bounded acknowledgement, and only then unwind the local stream. This prevents a stale timeout from cancelling a successor turn.
Change: guarded and off-loop timeout dispatches now share the owner-fenced cancellation sequence and one timeout-reporting path. A provider-confirmed completed turn skips redundant cancellation and replay. Unfinished
no_turn, timeout, and error outcomes preserve partial output and show that resuming is unsafe because native cancellation was not acknowledged.Implementation map:
src/kiro_crew/dashboard/turn_dispatch.py: deadline ordering, acknowledgement handling, and shared timeout reporting.src/kiro_crew/session_lifecycle.pyandsrc/kiro_crew/session.py: exact-owner cancellation and provider-confirmed completion proof.src/kiro_crew/slack/gateway.pyandsrc/kiro_crew/apps/builtins/spec_builder/backend/runtime.py: pass session state and slot identity to the shared timeout wrapper and consume task exceptions.docs/system-specs/modules/learn-cron-dashboard.md: document the owner-fenced native cancellation contract..github/black-baseline.txt: remove the cancellation consistency test after this change makes it Black-clean.Backwards compatibility
Compatible: existing callers keep their timeout behavior and signatures remain compatible through optional cancellation context; production call sites now supply session state and slot identity for native cancellation.
Removes nothing: the change adds owner-fenced cancellation and more precise timeout reporting without deleting or rejecting an existing input, API, or recovery path.
Tests
test/test_turn_dispatch.py: deadline ordering, bounded acknowledgement, repeated cancellation, provider-confirmed completion, replay, and fail-closed timeout cards.test/test_session.py: exact-owner cancellation, wrong-owner refusal, completion proof, and pre-await replay marking.test/test_chat_turn_timeout_consistency.py: every timed dispatch uses a valid guarded form, supplies state and slot identity, and consumes task exceptions.test/test_spec_builder_routes_coverage.py: Spec Builder passes the exact state and slot into the shared timeout wrapper.git diff --checkpassed on the rebased head.Manual verification
N/A — the change is an async lifecycle and ownership repair with deterministic race coverage; it does not change a rendered UI surface.
Screenshots / video
N/A — no user-visible frontend files or layout behavior changed.
Related Issues
Fixes #18416.
Replaces #16301 by @junjiequ (original fork PR; re-authored onto the main repo so CI and the fork review lanes can run). #14058 also tracks stage-boundary, queued-message, and Stop recovery outside this focused native-cancellation repair.
Related but non-overlapping: #15918 reports an honest response when Stop finds an already-idle wedged slot; this PR prevents the native turn from being abandoned at the timeout boundary.
Pattern harvest
Rule candidate: review-prompt
Pattern: a timeout around a remote stream must stop and acknowledge the remote operation before cancelling the local consumer, fenced by the exact operation owner so a stale timeout cannot cancel successor work.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Generated UI index
Commit
74b0096d9regeneratessrc/kiro_crew/docs/ui-index.generated.jsonwithnpm run gen:ui.gen-ui-index --checkfails on current main, which reddens Frontend Lint, Build Wheel, Build Desktop and E2E on every PR. This is the same fix as #18753, regenerated on newer main (it also includes thenotifications.sound-category-dictationsetting main has since added). Once either lands, the other becomes a no-op on rebase.