Skip to content

fix(dashboard): keep the logout counter write off the event loop - #34371

Merged
bolichen97 merged 1 commit into
mainfrom
fix/34017-logout-revocation-off-loop
Nov 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/34017-logout-revocation-off-loop

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: kirocrew logout on a slow disk no longer freezes the gateway's event loop, so the loop watchdog cannot kill the gateway over the logout's revocation-counter write.

POST /api/logout (api_logout, which kirocrew logout calls) ran revoke_all_sessions() directly on the event loop. That bumps the persisted revocation counter (bump_revocation_gen), which writes a temp file and renames it onto token_revocation.gen while it holds _gen_lock. Every token validation takes the same lock through current_revocation_gen_or_none(). While the write is slow, for example while on-access antivirus scans the new temp file, no request is answered, and a stall past the watchdog's budget kills the gateway.

Why it matters

  • On a host with on-access scanning, one kirocrew logout can stop the whole gateway: HTTP, chats and channels all wait, and past the watchdog's budget the gateway is killed and every running turn is cut.
  • Moving the write to a worker thread alone would not help: the loop's validations would still wait on the lock the write holds.

Not a goal

What changed (motivation → approach → change)

A slow disk now delays only the kirocrew logout request that writes the counter, never the gateway's event loop. Nothing else about a logout changes for a user: until the write lands every session works as before, and once it lands every pre-logout session ends at once.

Path Root cause Change
POST /api/logout (handlers/core.py) api_logout ran revoke_all_sessions() on the event loop it awaits the new revoke_all_sessions_async() (token_auth_tokens.py)
Revocation counter (revocation_gen.py) bump_revocation_gen held _gen_lock, which every validation takes on the loop, across the temp-file write and the rename persist_next_revocation_gen() picks the base, writes and renames under _bump_lock alone, and remembers the value it wrote; publish_revocation_gen() makes the value live under _gen_lock, never lowers it, and says whether it advanced; bump_revocation_gen() runs the two in a row
revoke_all_sessions_async() new runs only persist_next_revocation_gen() on a worker thread; back on the loop, with no await between, it publishes the value and clears the link, pin and consumed state only when that publish advances the live value

The publish and the clear stay on the loop because loop handlers rely on the logout being one step. A link exchange checks a token's generation and then mints its replacement with no await between, and a peer pin protects a cookie only while it exists. A publish or a clear from another thread could land in that gap, or before the new value is live, and let a pre-logout link mint a session that survives the logout, or let a cookie pass its pin from another address while the write is slow.

The new value is still published only after the atomic rename succeeds, so a token is never minted with a generation that is not on disk. A write builds on the larger of the live value and the last value this process wrote, and reads the file only when neither is known, as main does before the first load: the gateway is the file's only writer, so an overlapping logout never writes at or below a value another one has already written, whether or not the file can be read. The state is cleared only when a logout advances the live value. A logout that reaches the loop after a newer one published clears nothing, so a pin bound since the newer logout stays in force. An unreadable base or a failed write still fails the logout with the same 500, and changes nothing: the live value and every pin stay as they were, so no cookie the failed logout left valid loses its pin. The synchronous revoke_all_sessions() follows the same rule: it clears the state only after its bump lands.

dashboard-token-auth.md states the counter's lock and ordering rules, and the facade's type-checking list in token_auth.py names the new functions.

Backwards compatibility

Compatible: no file format, file permission, API, config key or command changes. The same file is written with the same contents; only which thread writes it, which lock covers the write, and where the new value is published change.

Removes nothing: no path, key, record or public function is removed or renamed; bump_revocation_gen() and revoke_all_sessions() keep their signatures and results. bump_revocation_gen() builds on the larger of the live value and its own last write, which gives a different value from main only while another logout has written but not yet published. A logout whose bump fails keeps the in-memory link, pin and consumed state instead of dropping it, so a cookie the old generation still accepts keeps its pin.

Tests

The full backend suite runs in CI. On Ubuntu 24.04 (Python 3.12.3), with this exact patch on base 6a61c8465f, scripts/local-gate.py's related sets gave backend 29311 + 25155 passed and frontend 13653 + 5869 passed; the 6 backend reds there (4 doctor-config tests, 1 git-fixture test, and a boot-composition test that sees a built static/dist) fail the same way on the unfixed base on that host. mypy src/kiro_crew, flake8, isort, black, the flake-pattern ratchet (68 passed) and semgrep (0 findings) are clean. Determinism, on the identical patch at d2f735f7a7: test_revocation_gen.py passed 20 times at -n0, 10 times at -n 4 --dist loadgroup --max-worker-restart=2 beside test_refresh_tokens.py and test_windows_replace_retry_sweep.py, and once in shuffled order with test_token_auth.py (seed 7909, 182 tests).

  • test_validation_reads_the_counter_while_a_bump_write_is_stuck: while a write is held before its rename, a counter read on another thread answers at once with the old value; the new value is live and on disk once the rename lands.
  • test_a_pin_stays_in_force_while_the_logout_write_is_stuck: while the write of revoke_all_sessions_async() is held, the generation is unchanged and a pinned cookie is still refused from another address; afterwards that cookie is rejected as revoked.
  • test_only_the_logout_write_leaves_the_event_loop: the rename runs on a worker thread, and the publish and the state clear run on the loop's thread.
  • test_a_superseded_logout_keeps_the_pins_bound_after_the_newer_one: the logout that wrote 5 reaches the loop after the one that wrote 6 has published; a cookie minted and pinned in between still validates and is still refused from another address, and the live value stays 6.
  • test_a_bump_builds_on_a_value_persisted_but_not_yet_published: with 4 live, two bumps that have not published write 5 and then 6.
  • test_a_bump_never_writes_below_its_own_unpublished_write_when_the_file_cannot_be_read: with 5 and 6 written but 4 live, a bump whose file read fails writes 7, not 5.
  • test_a_failed_logout_write_raises_and_changes_nothing: a failed write raises OSError from both the async and the sync form, the live value stays the same, and a pinned cookie is still refused from another address.
  • The logout handler tests in test_dashboard_handlers_core_coverage_security_endpoints.py, test_cli_gateway_client.py and test_token_auth_session_lifecycle.py patch revoke_all_sessions_async; the facade's known-seam list (test_token_auth_facade.py) and the Windows replace sweep name the new functions.

Each new rule was broken on purpose and its test went red: running the whole synchronous revocation on a worker thread fails the pin test and the loop-thread test; publishing from the worker fails the loop-thread test; building only on the live value fails the overlap test; clearing on every publish fails the superseded-logout test; clearing on a failed write, in either form, fails the failed-logout test; falling back to the live value when the file cannot be read fails the unreadable-file test.

Proof: PROVEN test/test_revocation_gen.py::test_validation_reads_the_counter_while_a_bump_write_is_stuck, test/test_windows_replace_retry_sweep.py::test_converted_writer_renames_through_the_retry[dashboard/revocation_gen.py] (head 3ee714fd8a2e)

prove.py ran on 3ee714fd8a, this exact patch on base 6a61c8465f (tree fa8a7e5dfa).

Manual verification

On Ubuntu 24.04 (Python 3.12.3), with an isolated gateway (a temporary KIROCREW_HOME, a free loopback port), the counter's temp-file write slowed by 3 s, and a real kirocrew logout while /api/live and a token-authenticated /api/crons were polled:

Build Where the write ran Longest wait during the logout Old token
unfixed base 6a61c8465f the event loop's thread (MainThread) /api/live 2.997 s, authenticated request 2.980 s 403 once the write lands; requests sent during it wait for it
this head a worker thread /api/live 0.012 s, authenticated request 0.020 s 200 until the rename landed, 403 right after
this head, write fails (injected ENOSPC) a worker thread /api/live 0.018 s, authenticated request 0.019 s still 200; the counter file is unchanged

On the failed write, kirocrew logout exits 1 with HTTP 500: could not persist session revocation; logout not completed. On a normal disk the logout writes the counter, the pre-logout token gets 403 and a freshly minted token gets 200. All 7 tests in test/test_revocation_gen.py fail against the unfixed base's code.

Related Issues

Closes #34017

Sibling sweep

Swept: git grep -n "threading.Lock()" -- 'src/kiro_crew/dashboard/token_*.py' src/kiro_crew/dashboard/refresh_tokens.py src/kiro_crew/dashboard/revocation_gen.py
Sibling: src/kiro_crew/dashboard/revocation_gen.py:38 -- changed here
Sibling: src/kiro_crew/dashboard/token_auth_tokens.py:80 -- not affected -- #34031 on main holds this denylist lock for the dict only, never across the write
Sibling: src/kiro_crew/dashboard/refresh_tokens.py:214 -- not affected -- #34031 on main holds this refresh-store lock for the snapshot only, never across the write
Sibling: src/kiro_crew/dashboard/token_secret.py:830 -- not affected -- only the one-time key load holds it across disk I/O, and the boot runs that before the server accepts connections
Sibling: src/kiro_crew/dashboard/token_auth_app_scope.py:43 -- not affected -- the manifest read runs with the lock released

The search's other hits hold no write the loop waits on: revocation_gen.py:39 (_bump_lock) and refresh_tokens.py:226 (_write_sequence_lock) are taken only by writers, token_auth_tokens.py:81 (_io_lock) only by the denylist's writes, refresh_tokens.py:923 and token_auth_tokens.py:215 guard building each store once, token_auth_tokens.py:249 (TokenStateManager) is in memory only, and token_auth_tokens.py:1038 (_app_secret_move_lock) is taken only by replace_legacy_app_secret, never by a validation.

Pattern harvest

Rule candidate: review-prompt
Pattern: a threading.Lock that event-loop code takes is held across file I/O, so the loop waits on the disk even when the write itself runs in a worker thread; and moving a whole synchronous sequence onto a worker thread lets loop handlers run between its steps, so only the blocking I/O should leave the loop.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 10, 2026 06:35
@bolichen97
bolichen97 requested a review from CrysisDeu November 10, 2026 06:35
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 942c0f69abd13b1160359f61ef4db7e934920af2 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 942c0f69abd13b1160359f61ef4db7e934920af2 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix removes the cause: the slow write no longer runs under the lock that validation waits on. The one leftover is a sync logout path that production no longer calls.

Notes

  • Item 4 — rides along: the sync revoke_all_sessions() now clears state after the bump instead of before. After this diff it has 0 production callers: I grepped revoke_all_sessions\(\)|\.revoke_all_sessions\b under src/, and every hit except its definition is a comment. Only 6 test files call it. Its only production use of bump_revocation_gen() (token_auth_tokens.py:1278) goes with it. Subtraction: delete revoke_all_sessions() and bump_revocation_gen(), and point their tests at revoke_all_sessions_async() / persist_next_revocation_gen() + publish_revocation_gen(). This drops a second logout spelling that has to stay ordered the same way.

What this change ships

Inventory (5 items) — 4 justified

Intent: a kirocrew logout on a slow disk must not freeze the gateway or get it killed by the loop watchdog. This is a FIX.

  1. The logout's counter write runs on a worker thread, and validations no longer wait on it. — justified
  2. Overlapping logouts never write the same or a lower counter value (_written, _bump_lock). — justified
  3. A logout that a newer one already superseded leaves the newer pins alone. — justified
  4. The sync revoke_all_sessions() now keeps the pins in place when its bump fails. — rides along, 0 production callers
  5. New public names revoke_all_sessions_async, persist_next_revocation_gen and publish_revocation_gen, with the spec updated. — justified

[FIRST-PRINCIPLES-REVIEWED] 942c0f6

Description read

[DESCRIPTION-READ] bb9deb6d3e9b6631c16be144a8c10f08d5fd82218582b273d1e0d4b2a06f2383

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 942c0f69abd13b1160359f61ef4db7e934920af2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: splitting the write from the publish fixes the lock contention that caused the freeze, and moving the write to a thread alone would not have.

Notes

[DESIGN-REVIEWED] 942c0f6

Description read

[DESCRIPTION-READ] 268d20ab6d641d3b808508ddf0d5dc41e4553edc28f419b01b5060db0e305a62

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 942c0f69abd13b1160359f61ef4db7e934920af2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 942c0f6

Verdict parsed from the review's SHA-scoped output markers for commit 942c0f69abd13b1160359f61ef4db7e934920af2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 942c0f69abd13b1160359f61ef4db7e934920af2 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 942c0f69abd13b1160359f61ef4db7e934920af2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 942c0f6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 942c0f69abd13b1160359f61ef4db7e934920af2 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 10, 2026 06:48
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@bolichen97
bolichen97 force-pushed the fix/34017-logout-revocation-off-loop branch from b5cb42d to feb5052 Compare November 10, 2026 16:58
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/token_auth_tokens.py, src/kiro_crew/dashboard/handlers/core.py, src/kiro_crew/dashboard/token_auth.py, docs/system-specs/modules/dashboard-token-auth.md and test/test_cli_gateway_client.py since this head went green (69d3a9a -> 91cb4c4); CI re-running.
[operator: bolichen97]

@bolichen97

bolichen97 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: logout revocation moved onto a worker thread
resolution: narrowed -- only the counter's disk write leaves the event loop

  • span=324b234bfb47 Preserve peer pins while revocation is pending

fixed in feb5052: api_logout awaits revoke_all_sessions_async(), which runs only persist_next_revocation_gen() (base read, temp write, rename) on a worker thread.
The new generation is published and the pin/link/consumed state cleared back on the loop, one after the other with no await between, so no request sees cleared pins under the old generation.
While the write is in flight every pre-logout pin still refuses another address: test_a_pin_stays_in_force_while_the_logout_write_is_stuck; publish and clear run on the loop thread: test_only_the_logout_write_leaves_the_event_loop.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 10, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests shards (3.12 and Windows, shards 4 to 7) and the Coverage Gate on feb505229b are main's, not this PR's. The same node ids fail on main's push run https://github.com/kirodotdev/KiroCrew/actions/runs/42863269012 (ed80c1d0f2), and this diff touches none of those files.

This PR waits for those fixes. Once they merge, it rebases onto them and CI re-runs.

@bolichen97
bolichen97 force-pushed the fix/34017-logout-revocation-off-loop branch from feb5052 to baf13c3 Compare November 10, 2026 23:55
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in setup.cfg and the code-channel test double (#34646, the fix for the red Backend Tests shards linked above) since this head's CI ran (91cb4c4 -> c076070); CI re-running.
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The GPT 6.1 run on feb505229b hit a GitHub API rate limit before it could post, so no verdict for that head landed (run https://github.com/kirodotdev/KiroCrew/actions/runs/42860130419). Its draft raised one blocking finding on token_auth_tokens.py: when two logouts overlap and reach the loop out of order, the older one cleared a pin bound since the newer logout.

baf13c345a fixes it. publish_revocation_gen() now says whether it advanced the live value, and a logout that a newer one already published over leaves the state alone. The new test test_a_superseded_logout_keeps_the_pins_bound_after_the_newer_one fails on feb505229b and passes on baf13c345a. The lane re-runs on the new head.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 432 lines, more than the 400 it may approve on its own.

The PR moves only the revocation-counter write (persist_next_revocation_gen) to a worker thread, under a new _bump_lock and never under the _gen_lock that every validation takes. The new value is published and the link/pin/consumed state cleared on the loop thread, with no await between them, so peer pins stay in force until the new value is live. This fixes the replay window GPT flagged on an earlier head. Overlapping logouts build on max(live, file), and a logout a newer one has already published over clears nothing. The failure path matches main. I checked against the code: interleavings with validators, overlapping logouts, the counter not yet loaded, the facade patch mirroring, and the remaining sync callers (tests only). The new tests hold the real rename and would fail without the change. Cancellation mid-write is reachable only at shutdown, because the repo leaves aiohttp's handler cancellation at its default (off), and the value on disk loads on restart.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit baf13c345a · 432 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on baf13c3. Both earlier security findings are fixed and test-guarded at this head: pins stay in force while the logout write is in flight (b5cb42d finding), and a superseded logout no longer clears pins bound after the newer logout (feb5052 GPT 6.1 finding); 504 related tests pass and both new guards turn a named test red when broken. CI on this head is all rate-limit infra noise, so no AI lane and no backend test lane produced a result. Decisive lines: src/kiro_crew/dashboard/token_auth_tokens.py:1139-1146 (advanced = publish_revocation_gen(new_gen); _state.clear_all() only if advanced), src/kiro_crew/dashboard/revocation_gen.py:172-184 (publish_revocation_gen never lowers the live value and returns whether it advanced), src/kiro_crew/dashboard/revocation_gen.py:142-154 (write under _bump_lock only, base = max(live, file))

@bolichen97
bolichen97 force-pushed the fix/34017-logout-revocation-off-loop branch from baf13c3 to ece4a3b Compare November 11, 2026 17:05
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the logout's state clear was tied to the attempt, not to the generation advancing
resolution: removed -- the try/finally that cleared state on every outcome is gone; state clears only on an advancing publish

  • span=18da54385d98 Failed logout removes pins from sessions established during its write

fixed in ece4a3b: revoke_all_sessions_async() clears the link, pin and consumed state only when publish_revocation_gen() advances the live value, so a failed or superseded logout leaves every pin in place.
The sync revoke_all_sessions() bumps first and clears only after the bump lands, so neither form drops a pin while the old generation still accepts its cookie.
Pinned by test_a_failed_logout_write_raises_and_changes_nothing, red on baf13c3's code (the pin is gone after the failed write).
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the bump's base came from a file read while an overlapping write was unpublished
resolution: removed -- the per-bump file read is gone; the base is the live value or this process's last write, the file only when neither is known

  • span=e3ceb7b8feb9 Unreadable counter permits durable revocation rollback

fixed in ece4a3b: persist_next_revocation_gen() records the last value it wrote, under _bump_lock, and builds on the larger of it and the live value; it reads the file only when neither is known.
The gateway is the counter file's only writer, so no bump writes at or below a value this process already wrote, whether or not the file reads back.
Pinned by test_a_bump_never_writes_below_its_own_unpublished_write_when_the_file_cannot_be_read, red on baf13c3's code (the third bump wrote 5 over 6).
[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @buluoray thanks for checking baf13c345a. Once the lanes produced results there, CI was green apart from GPT 6.1, which raised two security-class findings on it. Both are fixed in ece4a3b40b, now the PR head:

  • A logout whose counter write failed still cleared the peer pins while the unchanged generation kept accepting the cookies they guard. The link, pin and consumed state is now cleared only when the logout advances the live generation, in the async and the sync form (test_a_failed_logout_write_raises_and_changes_nothing).
  • With overlapping logouts, a bump whose read of the counter file failed built on the stale live value and could write below a value already written. The bump now builds on the larger of the live value and the last value this process wrote, and reads the file only when neither is known (test_a_bump_never_writes_below_its_own_unpublished_write_when_the_file_cannot_be_read).

The two guards you named (pins stay in force during the write; a superseded logout keeps the newer pins) are unchanged. CI is re-running on the new head. Could you take another look once it is green?

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on ece4a3b. Every point behind our REQUEST_CHANGES on baf13c3 is fixed at the live head and each fix turns a named test red when reverted: a failed logout write no longer drops peer pins (GPT F1), and an unreadable counter file can no longer make an overlapping logout write a lower generation (GPT F2); the two earlier fixes (pins kept while the write is in flight, superseded logout keeps newer pins) still hold. GPT 6.1 and Opus 5.5 both have real "no blocking findings" results on this head, no [BLOCK-MERGE] exists, and every red check is noise. 8 backend test shards were still running when read. Decisive lines: src/kiro_crew/dashboard/token_auth_tokens.py:1140-1143 (async logout: persist on a worker thread, then _state.clear_all() only if publish_revocation_gen(new_gen); no try/finally, so a raise clears nothing), src/kiro_crew/dashboard/token_auth_tokens.py:1119-1120 (sync form: bump_revocation_gen() before _state.clear_all()), src/kiro_crew/dashboard/revocation_gen.py:151-156,176 (base = max(live, this process's last write); file read only when neither is known; _written set only after the rename succeeds), src/kiro_crew/dashboard/revocation_gen.py:189-190 (publish never lowers the live value)

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The one red test on ece4a3b40b is test/test_remove_slot_for_history_key_crew_log.py::TestSessionLedgerOnPermanentDelete::test_a_ledger_the_emitter_itself_opened_is_still_removed in Backend Tests (3.12, 6), and the Coverage Gate fails closed on that shard. It is the flaky test tracked in #34376, where this run is now recorded: this PR touches nothing under crew_log or the session ledger, and the same assertion fails on many unrelated PR heads. That test has too many reports this week to rerun again, so this PR waits for its fix and then re-syncs onto it. The two other red jobs on this head, CodeQL's actions analysis and the readiness signal, stopped on a GitHub API rate limit before producing a result. Every other check and every review lane is green here.

@bolichen97
bolichen97 force-pushed the fix/34017-logout-revocation-off-loop branch from ece4a3b to d2f735f Compare November 13, 2026 01:55
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/handlers/core.py, src/kiro_crew/dashboard/token_auth.py, src/kiro_crew/dashboard/token_auth_tokens.py, docs/system-specs/modules/dashboard-token-auth.md, test/test_cli_gateway_client.py and test/test_token_auth_session_lifecycle.py since this head went green (c076070 -> 9584db1); CI re-running.

The new base carries #35705, the fix for the flaky test_a_ledger_the_emitter_itself_opened_is_still_removed that failed Backend Tests (3.12, 6) on the old head. The PR's own added and removed lines are unchanged.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The five red checks on d2f735f7a7 do not come from this PR:

  • Backend Tests (3.12, 8) and Backend Tests (Windows) (8): test/test_reply_format_once.py::TestEveryMessageWhereTheBlockCannotBeTakenOut::test_the_table_names_every_selectable_harness. Main had just added the mimocode harness without its table row. fix(test): add mimocode to the compaction table every selectable harness needs #36290 fixed that on main at 03:19Z, after this run's merge commit was made.
  • Backend Tests (3.12, 7): test/test_denied_commands_security_cost.py::TestSelfModuleIndexIsLinear::test_the_scan_is_linear_not_quadratic took 0.507 s against its 0.5 s wall-clock bound. It is a known flaky timing test, in a file this PR does not touch.
  • Analyze (javascript-typescript): the CodeQL upload hit GitHub's API rate limit for the installation.
  • Coverage Gate: it fails closed on the shard reds above.

Rerunning the failed jobs would test the same merge commit again, so it cannot pick up #36290. Getting this green needs one re-sync onto main. This PR has already used its three re-syncs, so I am not rebasing it. A project member can grant one more by commenting Re-sync budget reset on this PR.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Smoke test of this head (d2f735f7a7), run against a real isolated gateway and the real kirocrew logout CLI, with the revocation-counter write slowed to 3 s.

Verified on Ubuntu 24.04 (Linux x86_64), Python 3.12.3.

  • Parent (unfixed), reproduced: isolated gateway (fresh KIROCREW_HOME, free loopback port), Path.write_text patched so the token_revocation.gen.tmp write takes 3 s, real kirocrew logout CLI. The write ran on the event-loop thread, and every /api/live and token-authenticated request during the logout waited ~3.0 s. The PR's six test files against the parent's code gave 14 failed: test_validation_reads_the_counter_while_a_bump_write_is_stuck on behaviour, the rest on missing new API.
  • Head, same harness: the write runs on a worker thread, and /api/live and token validation stayed at ≤14 ms throughout the 3 s write. The old token kept validating until the rename landed and was rejected right after (last 200 at the write end −18 ms, first 403 at the write end +32 ms). A normal logout bumps the counter (absent→1→2→3), rejects the pre-logout token with 403 and accepts a freshly minted one.
  • Failed write (ENOSPC injected after 2 s): the logout returns HTTP 500 revocation_persist_failed and the CLI exits 1. The counter on disk stays unchanged and the old token is still accepted.
  • Gates: black, subprocess-encoding, isort, flake8, mypy src/kiro_crew, comment-history, brand, file-line-cap, flake-pattern ratchet (68 passed) and semgrep --baseline-commit <parent> (0 findings) all pass. The PR's tests: 258 passed. local-gate.py: backend 29277 + 24219 passed, frontend 13855 + 5380 passed, with 2 backend reds that are pre-existing on the parent on that machine (test_up_to_date_diff_runs_no_textconv_planted_during_the_fetch, host git; test_the_dashboard_boot_keeps_its_chain_hooks_and_order, fails whenever website/dist is built, parent included).

Requests issued while the logout ran (from the probe log):

== parent 9584db1423 (unfixed), counter write slowed 3 s
  live: 46 requests during logout, max latency 2.978s, codes ['200']
  authed-old-token: 44 requests during logout, max latency 2.998s, codes ['200', '403']
after: gen on disk=1  old token /api/crons -> 403  fresh token -> 200
== head d2f735f7a7, counter write slowed 3 s
  live: 107 requests during logout, max latency 0.014s, codes ['200']
  authed-old-token: 105 requests during logout, max latency 0.011s, codes ['200', '403']
after: gen on disk=1  old token /api/crons -> 403  fresh token -> 200
== head d2f735f7a7, counter write slowed 2 s then fails (ENOSPC injected)
kirocrew logout: rc=1 err=['❌ Failed to revoke sessions: HTTP 500: could not persist session revocation; logout not completed']
  live: 88 requests during logout, max latency 0.009s, codes ['200']
after: gen on disk=3  old token /api/crons -> 200  fresh token -> 200

@bolichen97

Copy link
Copy Markdown
Contributor Author

Re-sync budget reset

Two of the red Backend Tests shards on d2f735f7a7 (3.12 shard 8 and Windows shard 8) fail test/test_reply_format_once.py::TestEveryMessageWhereTheBlockCannotBeTakenOut::test_the_table_names_every_selectable_harness, a main red whose fix (#36290) merged after this branch's base. A maintainer asked for this PR to be kept green through its watch window, so the re-sync count restarts here and the branch is rebased onto current main to pick that fix up. The PR's own change is not edited.

[operator: bolichen97]

kirocrew logout bumped the revocation counter on the gateway's event
loop, and the bump held the lock every token validation takes across
the temp-file write and rename. A slow disk (for example on-access
antivirus scanning the new temp file) therefore stalled every request
until the write finished, and past the loop watchdog's budget the
gateway was killed.

POST /api/logout now awaits revoke_all_sessions_async(), which runs
only the counter's write (persist_next_revocation_gen) on a worker
thread, under a lock that only writes take. Back on the loop, with no
await between, it publishes the new value and, only when that advances
the live value, clears the in-memory link, pin and consumed state, so
no request ever sees a half-done logout: until the write lands every
pre-logout pin still holds, and no handler can check an old token and
mint its replacement across the publish. A logout that reaches the
loop after a newer one has published, or whose write fails, leaves
that state alone, so no pin is dropped while the cookies it guards
still validate. The value is published only after the atomic rename
succeeds and never lowered, and a write builds on the larger of the
live value and the last value this process wrote, reading the file
only when neither is known, so an overlapping logout never writes
below one already written. The fail-closed rules are unchanged.

Closes #34017
@bolichen97
bolichen97 force-pushed the fix/34017-logout-revocation-off-loop branch from d2f735f to 942c0f6 Compare November 17, 2026 18:34
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/dashboard-token-auth.md, src/kiro_crew/dashboard/handlers/core.py, src/kiro_crew/dashboard/token_auth.py, test/test_cli_gateway_client.py, test/test_dashboard_handlers_core_coverage_security_endpoints.py and test/test_token_auth_session_lifecycle.py since this head's CI ran (9584db1 -> e72cf2e); CI re-running.

The new base carries #36290, the fix for test_the_table_names_every_selectable_harness that failed Backend Tests (3.12, 8) and Backend Tests (Windows) (8) on the old head, and #37451, #37530 and #37539, the fixes for the wall-clock bound in test_the_scan_is_linear_not_quadratic that failed Backend Tests (3.12, 7). The PR's own added and removed lines are unchanged.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 17, 2026
@bolichen97
bolichen97 merged commit 9ecbf3f into main Nov 17, 2026
100 checks passed
@bolichen97
bolichen97 deleted the fix/34017-logout-revocation-off-loop branch November 17, 2026 19:47
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway: kirocrew logout writes the revocation counter on the event loop, so a slow disk can stall it

3 participants