Skip to content

fix(session): record the permit holder when the reacquire takes it - #19000

Merged
javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/session-permit-owner-recorded-on-acquire
Oct 13, 2026
Merged

javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/session-permit-owner-recorded-on-acquire

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a reset that lands while a woken claimant waits for the manager lock releases that claimant's turn permit exactly once, records the right holder, and does not make the queued message fail.

  • A message that finds its session busy parks on the session's turn permit in _reacquire_and_validate (src/kiro_crew/session_allocation.py). When the busy turn ends, its release hands the permit to the parked claimant, which then waits for the manager lock before it revalidates.
  • The claimant recorded itself as the permit's holder (turn_owner) only after that wait. A reset (from reset_conversation, or the teardown behind a force Stop) can hold the lock in the window. It reads turn_owner, which still names the turn that already released, records that task as the orphaned holder (_orphan_turn_holder), and releases the permit to wake parked claimants (_wake_turn_waiters). The permit it releases is the claimant's.
  • The claimant then finds its session popped and releases the permit a second time. asyncio.BoundedSemaphore raises ValueError: BoundedSemaphore released too many times, and nothing between _reacquire_and_validate and get_or_create handles it.

Why it matters

The message that waited behind the busy turn fails with a raw ValueError instead of being allocated a fresh session. When the reset is an ending fence, that ValueError also replaces the fence refusal that get_or_create turns into a wait. The wrong orphan record has a second effect: it names the turn that handed the permit over, so that task's next genuine release on the key, on a successor's permit, is absorbed by absorb_orphaned_release and the key stays busy.

Not a goal

  • Changing the lock order. The permit is still acquired with the manager lock released, as the session spec requires.
  • The per-key orphan record's shape. orphaned_holders keeps one entry per task per key, so it cannot tell two owed absorbs for one task apart, for example a turn that reset its own session (a replay) and loses a second session during its reacquire wait. The replay path's adopt_turn (session_allocation.py, after a successful claim) still clears that entry.
  • The acquire sites that do not record turn_owner: the compaction recycle in session_compaction.py (_recycle_unmanaged, also edited by a separate open change) and the background session's prompt in session_background.py. They release the permit directly on the session object, not through the key-based release.
  • AcpSessionProvider.shutdown under the turn permit. Measured against a backend that hangs and ignores SIGTERM, it returns in 5 to 10 s, because each of its awaits has its own bound.

What changed (motivation → approach → change)

  • Symptom: a reset in the window between a woken claimant's permit and its lock wait makes the claimant raise ValueError: BoundedSemaphore released too many times, and leaves the previous turn recorded as an orphaned holder.
  • Root cause: _reacquire_and_validate set session.turn_owner only after it had taken the lock and validated, and its refusal paths released the permit unconditionally. The reset's _orphan_turn_holder and _wake_turn_waiters assume that a locked permit belongs to turn_owner, which was false in that window.
  • Change, src/kiro_crew/session_allocation.py:
    • _reacquire_and_validate sets session.turn_owner = asyncio.current_task() as soon as session.semaphore.acquire() returns, before waiting for the lock. A reset in the window therefore records the claimant, the task that actually holds the permit.
    • The new _release_reacquired(key, session) gives the permit back on the refusal path (except BaseException, which covers the ending-fence refusal and cancellation) and on the not-valid path. When the session is no longer the registered occupant and absorb_orphaned_release(key) reports that a reset recorded this task, it skips the release, because the reset already released the permit. A session that is still registered is always released, so a stale-provider refusal releases as before.
  • Change, docs/system-specs/modules/session.md: the revalidation contract states that the holder is recorded at the acquire and that a reset-released permit is not released again.
  • Rejected: catching ValueError around the release. It hides the double release and leaves the wrong task recorded as the holder.
  • Rejected: absorbing whenever absorb_orphaned_release reports a record, without checking that the session was popped. A task that already owes an absorb for an earlier reset on the key would then skip releasing a permit it holds on a session that is still registered.
  • Rejected: taking the manager lock before the permit. The permit wait can last a whole turn, and holding the lock across it would stop allocation for every key.

Backwards compatibility

Compatible: a claim that succeeds behaves exactly as before. turn_owner is set slightly earlier, and every reader of the field treats it as "the task holding the permit", which is true from the acquire on. A refused or stale claim with no reset in the window releases its permit as before.
Security: no redaction, auth, sandbox, permission or deny code changes (none in this branch's diff); the lock order is unchanged.
Removes nothing: the change adds one helper method and moves one assignment.

Tests

Added to test/test_session_core_audit.py, beside test_reacquire_and_validate_releases_semaphore_on_cancellation, with shared helpers _until (a bounded yield loop, no wall clock), _live_session, _orphans, _cancel_and_drain (cancels a test's background task and awaits it, bounded, from a finally) and _hand_permit_into_reset_window. That last helper drives a turn T, a claimant W parked on its permit, and a reset queued on the manager lock, until T's release has handed the permit to W.

  • test_a_reset_while_a_woken_claimant_waits_for_the_lock_releases_the_permit_once: W returns False without raising. Neither T nor W is left in the orphan record, and the permit is free and acquirable.
  • test_a_fence_refusal_after_a_reset_in_the_reacquire_window_does_not_over_release: the same window refused at the ending fence surfaces SessionEndingError, not ValueError.
  • test_a_reset_in_the_reacquire_window_leaves_the_previous_turns_next_release_genuine: T then claims the successor and releases it, and the successor's permit is free afterwards instead of staying locked.
  • test_a_reset_while_a_claimant_is_still_parked_wakes_it_and_absorbs_the_holders_release: control for the parked-waiter path. The reset records T and wakes W, W returns False, and T's later release is absorbed without unlocking a successor that another task holds.

The tests order events through the asyncio primitives' waiter queues (Semaphore._waiters, Lock._waiters) rather than through timing.

Red on this branch's base, 843abe3, with only the changed test file copied in (pytest test/test_session_core_audit.py -k "reacquire_window or woken_claimant or still_parked" -n 0), the same in each of 3 serial runs. The two ValueErrors are the double release; the AssertionError is the wrong orphan record absorbing T's genuine release:

FAILED test/test_session_core_audit.py::test_a_reset_while_a_woken_claimant_waits_for_the_lock_releases_the_permit_once - ValueError: BoundedSemaphore released too many times
FAILED test/test_session_core_audit.py::test_a_fence_refusal_after_a_reset_in_the_reacquire_window_does_not_over_release - ValueError: BoundedSemaphore released too many times
FAILED test/test_session_core_audit.py::test_a_reset_in_the_reacquire_window_leaves_the_previous_turns_next_release_genuine - AssertionError: T's genuine release was absorbed; the key stays busy
3 failed, 1 passed, 8 deselected

After the fix: test/test_session_core_audit.py passes (12 passed, in each of 3 serial runs). The 44 test files that name _reacquire_and_validate, turn_owner, orphaned_holders, absorb_orphaned_release, _wake_turn_waiters, adopt_turn or session_allocation give 3929 passed, 6 skipped. mypy, flake8, isort and black are clean on session_allocation.py; the test file is in .github/black-baseline.txt and was not reformatted. scripts/check_black_formatting.py, scripts/docs-lint.sh and scripts/check_comment_history.py with COMMENT_HISTORY_BASE_REF=origin/main pass.

Manual verification

N/A: the window is a scheduling order between three tasks. The tests drive that exact order against a real SessionManager with a mock provider.

Related Issues

None found. Searched PRs and issues, open and merged, for _reacquire_and_validate, turn_owner, BoundedSemaphore released too many times, orphaned_holders, _wake_turn_waiters, reset waiter permit and stale turn owner. Related: PR #11243 added _wake_turn_waiters and the orphan record for a claimant still parked on the permit. Open PR #11813, PR #11831 and PR #11983 also edit _reacquire_and_validate, and none of their changed lines touches turn_owner, the orphan record or the wake.

Pattern harvest

Rule candidate: review-prompt
Pattern: an ownership record is written after an await that follows the acquire it describes, so a concurrent reader in that await reads the previous owner.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): docs/system-specs/modules/session.md
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 01:04
@javenciu
javenciu requested a review from CrysisDeu October 11, 2026 01:04
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 32d226505674530139ec31e54b5574678554407b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 32d226505674530139ec31e54b5574678554407b via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that cancel_owned_turn (session_lifecycle.py:3316) still behaves now that it sees a claimant as the turn owner before that claimant has revalidated.

What this change ships

Inventory (7 items) — 7 justified

Intent: a message waiting behind a busy turn should get a fresh session, not crash, when a reset lands while it waits. This is a FIX.

  1. A queued message no longer fails with ValueError: BoundedSemaphore released too many times when a reset lands in the reacquire window — justified
  2. An ending-fence refusal in that window now reaches the caller as SessionEndingError, which becomes a wait, instead of ValueError — justified
  3. The turn that handed over the permit no longer has its next release absorbed, so the key no longer stays busy — justified
  4. The permit holder (turn_owner) is recorded as soon as the permit is acquired, before the lock wait, instead of after validation — justified
  5. Refusal paths release the permit through _release_reacquired, which absorbs the orphan record once the session has been popped. This applies the same absorb-or-release step the 3 compaction sites already use (session_compaction.py:730,1046,1153) — justified
  6. The session spec's revalidation contract is updated to match — justified
  7. Four interleaving tests; the description says three of them fail on base — justified

[FIRST-PRINCIPLES-REVIEWED] 32d2265

Description read

[DESCRIPTION-READ] f161283174162f2bfe647f98bddc3e52e507ee1f8a45f4dd9cbedee3742643bc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 32d226505674530139ec31e54b5574678554407b via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check cancel_owned_turn (session_lifecycle.py:3316). While W waits for the lock, turn_owner now names W, so an owner-fenced cancel for W can reach provider.cancel before W starts a turn.

[DESIGN-REVIEWED] 32d2265

Description read

[DESCRIPTION-READ] 6a13fc910e4b4b836f770a3fc41e21c08ed64dbd36263b6ed3ffa3898565b981

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 32d226505674530139ec31e54b5574678554407b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 32d2265

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 32d226505674530139ec31e54b5574678554407b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 32d2265

@javenciu
javenciu force-pushed the fix/session-permit-owner-recorded-on-acquire branch from e1496f4 to 8a1f0f8 Compare October 11, 2026 07:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/session-permit-owner-recorded-on-acquire branch from 8a1f0f8 to a2beb34 Compare October 12, 2026 13:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
A claimant that waited on a busy session's turn permit is handed the
permit by the turn's release, then waits for the manager lock before
it revalidates. It recorded itself as the permit's holder
(turn_owner) only after that wait. A reset that popped the session in
the window read turn_owner, which still named the turn that had
already released, recorded that task as the orphaned holder, and
released the permit to wake parked claimants. The woken claimant then
found its session gone and released the permit again: the queued
message failed with "ValueError: BoundedSemaphore released too many
times", and the stale orphan record could later swallow the previous
turn's genuine release of a successor's permit, leaving the key busy.

_reacquire_and_validate now sets turn_owner as soon as the permit is
acquired. Its refusal and not-valid paths release through the new
_release_reacquired, which skips the release when the session is no
longer registered and absorb_orphaned_release reports that a reset
already released this task's permit. The lock order is unchanged. The
session spec's revalidation contract says so.
The tests for the permit-holder record awaited the reset, the claimant
and the turn through fixed 1 s and 5 s wait_for bounds, which
test_wait_discipline_ratchet.py counts as 16 new short fixed waits in
test_session_core_audit.py. A bound that short decides pass or fail on a
slow runner.

A task or future the test owns is awaited with
kiro_crew.testing.wait.await_until, which fails by name after half the
test's timeout. The remaining waits stay wait_for with a 30 s lost-run
ceiling: the event waits inside a task the test cancels while it is
parked there (await_until would leave its inner wait pending), and the
coroutine awaits (a free permit's acquire, the reset).
@javenciu
javenciu force-pushed the fix/session-permit-owner-recorded-on-acquire branch from a2beb34 to 32d2265 Compare October 12, 2026 20:29
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto main (no conflict) and replaced the 16 short wait_for bounds (1 s and 5 s) the new tests added to test/test_session_core_audit.py, which test/test_wait_discipline_ratchet.py counted as 5 -> 21: the reset, the woken claimant, the previous turn, the drain in _cancel_and_drain and the control test's release event are now awaited with kiro_crew.testing.wait.await_until (11 sites: lines 404, 458, 459, 494, 496, 516, 517, 522, 545, 555 and 561).
The 5 that stay wait_for (the turn's two event waits at 422 and 424, which the test cancels while they are parked, the two free-permit acquire() calls at 464 and 525, and the control test's sm.reset() at 553) take a 30 s ceiling, _LOST_RUN_CEILING_SECS at line 377; the file is back at its 5 counted waits and the ratchet file passes, with the 12 tests green in 3 of 3 runs.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu enabled auto-merge (squash) October 12, 2026 23:19

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes a real double release in _reacquire_and_validate (session_allocation.py). A woken claimant set turn_owner only after its lock wait, so a reset in that window recorded the previous turn as the orphaned holder and released the claimant's permit, and the claimant then released it again and raised ValueError: BoundedSemaphore released too many times. The fix sets turn_owner right after acquire(), and routes both refusal paths through _release_reacquired. That helper skips the release only when the session is no longer registered and absorb_orphaned_release finds this task's record, which matches how _orphan_turn_holder (it records only while the permit is locked) and _wake_turn_waiters behave on main. I tried the lane's open question, cancel_owned_turn now seeing W as owner during the lock wait: its only caller is the turn-deadline cancel in dashboard/turn_dispatch.py, which passes the turn's own task, and provider_turn_is_known_terminal guards it, so at worst W's own deadline cancels its own not-yet-started turn. A session that is still registered always releases, so a stale-provider refusal behaves as before. The four interleaving tests drive the order through the waiter queues rather than timing, three of them are red on base for the right reasons, and all lanes PASS on 32d2265. — crew-pr-reviewer (deep re-check)

@javenciu
javenciu merged commit 315affc into kirodotdev:main Oct 13, 2026
105 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants