Skip to content

fix(dashboard): drop a cancelled threshold request's pending commit - #36630

Merged
bolichen97 merged 1 commit into
mainfrom
fix/32569-autocompact-commit-cleanup
Nov 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/32569-autocompact-commit-cleanup

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A threshold or turn-limit request cancelled during its save leaves no pending commit behind for later saves of that transcript to write.

The compaction-threshold route (POST /api/chat/slots/{slot}/autocompact) and the turn-time-limit route (POST /api/chat/slots/{slot}/turn-timeout) record the value they committed in a process-wide dict (autocompact_commits / turn_timeout_commits) from the save's commit hook. Every save of that transcript writes the recorded value until the route publishes it to the slot and drops the record. Only the route's normal path dropped it. save_slot_off_loop shields its worker, so when the request task is cancelled during the save, the route unwinds at once while the worker runs on and may record the value after the route has gone. Nothing then publishes or drops it.

Why it matters

After such a cancellation, every later save of that transcript writes a value the tabs and the live session do not hold, until the next request for the transcript. A handler task is cancelled when the server's runner shuts down, and a test process builds many DashboardState instances that share the dict, so a stale record from one request can leak into another state's saves. The fix is also what keeps the commit window correct if handler cancellation on client disconnect is ever turned on.

Not a goal

  • Changing save_slot_off_loop, the save folds (autocompact_fold / turn_timeout_fold), or the normal-path publish and drop.
  • Changing test_a_queued_alias_save_writes_a_merges_commit. The intermittent failure the issue reports did not reproduce in 60 runs (see Manual verification), so that test stays as it is and the issue stays open for it.
  • Turning on aiohttp handler cancellation, or publishing a value whose request was cancelled.

What changed (motivation → approach → change)

A cancelled threshold or turn-limit request could leave its committed value recorded, and every later save of that transcript then wrote it.

The record is written by a hook that runs in the shielded save worker, which can run after the cancelled route has unwound. So a plain finally that drops the record is not enough: it runs before such a late write.

Both routes now await the save through one helper in slot_settings.py, _save_dropping_commit_on_cancel. It starts the save as its own task, eagerly, so the save still dispatches in the route's own tick, and awaits it shielded. On cancellation it waits for that save to settle, drops the record, and re-raises. The route still holds the transcript's transaction lock while it waits, so no later request's record can be the one dropped. chat_handlers.py re-exports the helper because the route functions run on that facade's globals. The spec paragraph that documents the commit record (docs/system-specs/modules/learn-cron-dashboard.md) states the cancellation rule for both routes.

Backwards compatibility

Compatible: responses and the normal path are unchanged; only a request that is already being cancelled now waits for its own save before it finishes unwinding.

Removes nothing: the only thing dropped is an in-memory record that nothing publishes once its request is gone.

Tests

  • test/test_session_autocompact_override.py::TestACancelledRequestLeavesNoCommit::test_a_cancelled_request_drops_its_commit[before-cancel|after-cancel]
  • test/test_session_turn_timeout.py::TestACancelledRequestLeavesNoCommit::test_a_cancelled_request_drops_its_commit[before-cancel|after-cancel]

Each runs the handler as a task it holds. The save stand-in follows the real contract, awaiting asyncio.shield(worker), so cancelling the route does not cancel the worker. The worker runs the commit hook either before the request is cancelled or after it has unwound. Each test asserts the dict holds no record for the transcript, the slot keeps its old value, and (threshold route) the live override was never set. No sleeps; every await is bounded.

Proof: PROVEN test/test_session_autocompact_override.py::TestACancelledRequestLeavesNoCommit::test_a_cancelled_request_drops_its_commit[before-cancel], test/test_session_autocompact_override.py::TestACancelledRequestLeavesNoCommit::test_a_cancelled_request_drops_its_commit[after-cancel], test/test_session_turn_timeout.py::TestACancelledRequestLeavesNoCommit::test_a_cancelled_request_drops_its_commit[before-cancel], test/test_session_turn_timeout.py::TestACancelledRequestLeavesNoCommit::test_a_cancelled_request_drops_its_commit[after-cancel] (head e93d146d682d)

prove.py ran on e93d146d682d, which has this head's parent and the same code and tests; only one spec sentence differs.

Manual verification

Run on Ubuntu Linux (x86_64, Python 3.12.3).

On the base (c22557cd8) with only the two test files added, all four cases fail for the reported reason:

AssertionError: assert 'dashboard:cancelled-threshold' not in {'dashboard:cancelled-threshold': (60.0,)}   (x2)
AssertionError: assert 'dashboard:cancelled-limit' not in {'dashboard:cancelled-limit': (7200,)}         (x2)
4 failed, 149 deselected

On this change:

-k cancelled_request, both files: 4 passed
the four new tests, 20 runs at -n0: 20/20 green
both files, 10 runs at -n 4 --dist loadgroup: 10/10 green (153 passed)
both files in two seeded shuffled orders (seeds 18731, 4242), -n0: 153 passed each
test_session_autocompact_override + test_session_turn_timeout + test_alias_slot_shared_project + test_transcript_binding_join_sites: 199 passed

Static gates (black, isort, flake8, mypy, comment history, brand name, file line cap, harness parity, sync I/O in async, loop-bound locks, docs lint, flake-pattern and wait ratchets, semgrep on the diff) all exit 0. scripts/local-gate.py's backend related set: 12951 passed, 2 failed in test/test_source_link_github_case.py::TestOneChipPerObject::test_case_variants_collapse_to_one_chip; the same two fail on the base when test/test_session_card_source_links_knob.py runs first on the worker, so they are not from this change.

The intermittent alias-save failure the issue reports was not reproduced. The issue's seven-file pytest command ran 60 times with every write to the transcript's metadata line and every set and drop of its commit record logged:

route workers CPU load runs failed
this change 4 8 busy processes 20 0
base 8 8 busy processes 20 0
base 4 none 20 0

Every fold in that test wrote 60.0, each record was set in the save worker before the queued alias save and dropped on the loop after it, and no other test touched that record. The host reached about 2x CPU oversubscription, against about 3.6x in the issue's run.

Related Issues

Part of #32569

Sibling sweep

Swept: git grep -nE "_commits\[[^]]*\] *=" -- src/
Sibling: src/kiro_crew/dashboard/chat_api/slot_settings.py:195 -- changed here
Sibling: src/kiro_crew/dashboard/chat_api/slot_settings.py:476 -- changed here
Sibling: src/kiro_crew/dashboard/guide_runs.py:946 -- not affected -- a token map written on the event loop, not from a shielded save worker's hook

Pattern harvest

Rule candidate: review-prompt
Pattern: state recorded from inside shielded or executor-run work and dropped only on the awaiting caller's normal path; on cancellation the caller must wait for that work to settle before it cleans up, because a plain finally runs before the worker's late write.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d1b9e62

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Backend Tests (3.12, 7) is red on this head (ff80c351a4) only because of the known wall-clock flake tracked in #28212, whose fix is open in #28293.

This PR changes only the dashboard slot settings and their tests, and the same node id is red on other unrelated heads today. Coverage Gate fails only because that shard failed. Rerunning the failed jobs once. The Design, First Principles, Opus and UX lanes stopped on a GitHub API rate limit before giving a verdict, so they are re-run too.

@iamwhatever
iamwhatever force-pushed the fix/32569-autocompact-commit-cleanup branch from ff80c35 to 64cce26 Compare November 18, 2026 20:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. The cleanup now waits for the shielded worker to finish before it drops the record, so the worker can't write the record after the drop. The transcript lock is held for that wait, so the drop can only remove this request's own record.

Notes

  • asyncio.wait([task]) has no time limit on the cancellation path, so a save that hangs during shutdown keeps the per-transcript lock until it finishes. It is worth one sentence in the spec saying this wait inherits the save's own bound.

[DESIGN-REVIEWED] d1b9e62

Description read

[DESCRIPTION-READ] 2dd3b3d1641a1ac6c4b3847027bc82b4d207074a95942a5f6915d1e57d94aab7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: both routes' cancellation leak is fixed in one shared helper that has two real callers, and four new tests fail on base for the reported reason.

What this change ships

Inventory (5 items) — 5 justified

Intent: cancelling a threshold or turn-limit save should not leave a stale value that every later save of that transcript writes. FIX.

  1. A cancelled compaction-threshold request no longer leaves its value recorded for later saves — justified
  2. A cancelled turn-time-limit request no longer leaves its value recorded for later saves — justified
  3. A request cancelled during its save now waits for that save to finish before it unwinds, still holding the transcript lock — justified
  4. A new private helper _save_dropping_commit_on_cancel, re-exported from the chat_handlers facade — justified
  5. The turn-time-limit spec paragraph now states the cancellation rule for both routes — justified

[FIRST-PRINCIPLES-REVIEWED] d1b9e62

Description read

[DESCRIPTION-READ] feeca0045a80e241dc999ca6a3f0a23f563bf7b7aa5a1efab95c649655468a17

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d1b9e62

Verdict parsed from the review's SHA-scoped output markers for commit d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d1b9e62d6e6d87ed787fc6f164aef6c25911bc9e false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased on main (64cce26); all reviews pass. The one red lane, Backend Tests (3.12, 8), is main's own failure: test_bare_session_handle_guard flags test/test_kas_token_refresh_recovery.py:245 (#39261). The fix is in #39262. This PR waits for that to merge, then gets rebased and re-run.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
The threshold and turn-time-limit routes record a committed value for
the transcript from the save's commit hook, and every later save of that
transcript writes it until the route publishes it and drops the record.
The save shields its executor future, so when the request is cancelled
during the save the route unwinds at once while the worker runs on and
may record the value after the route is gone. Nothing then publishes or
drops it, and every later save writes a value the tabs and the live
session do not hold.

Each route now runs the save as its own task and awaits it shielded. On
cancellation it waits for that save to settle, still inside the
transcript's transaction lock so no later request's record can be the
one dropped, drops the record and re-raises. One helper serves both
routes.

Part of #32569.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamwhatever
iamwhatever force-pushed the fix/32569-autocompact-commit-cleanup branch from 64cce26 to d1b9e62 Compare November 18, 2026 21:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head d1b9e62. Both the threshold route and the turn time limit route now await their save through one shared helper. If the request is cancelled, the helper waits for the shielded save to settle, still holding the transcript lock, then drops the pending commit record. A late commit therefore cannot outlive its request or remove a later request's record. The four new cancellation tests fail on the base for the reported reason and pass here, and both affected test files pass in full locally. I found no behaviour, crash or data-loss issue that blocks this change.

@bolichen97
bolichen97 merged commit 893c379 into main Nov 18, 2026
101 of 102 checks passed
@bolichen97
bolichen97 deleted the fix/32569-autocompact-commit-cleanup branch November 18, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants