Repository navigation
fix(dashboard): keep the logout counter write off the event loop - #34371
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix removes the cause: the slow write no longer runs under the lock that validation waits on. The one leftover is a sync logout path that production no longer calls. Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: a
[FIRST-PRINCIPLES-REVIEWED] 942c0f6 Description read[DESCRIPTION-READ] bb9deb6d3e9b6631c16be144a8c10f08d5fd82218582b273d1e0d4b2a06f2383 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: splitting the write from the publish fixes the lock contention that caused the freeze, and moving the write to a thread alone would not have. Notes
[DESIGN-REVIEWED] 942c0f6 Description read[DESCRIPTION-READ] 268d20ab6d641d3b808508ddf0d5dc41e4553edc28f419b01b5060db0e305a62 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
b5cb42d to
feb5052
Compare
|
Rebased: main moved in src/kiro_crew/dashboard/token_auth_tokens.py, src/kiro_crew/dashboard/handlers/core.py, src/kiro_crew/dashboard/token_auth.py, docs/system-specs/modules/dashboard-token-auth.md and test/test_cli_gateway_client.py since this head went green (69d3a9a -> 91cb4c4); CI re-running. |
|
self-added: yes
|
|
[operator: bolichen97] The red Backend Tests shards (3.12 and Windows, shards 4 to 7) and the Coverage Gate on
This PR waits for those fixes. Once they merge, it rebases onto them and CI re-runs. |
feb5052 to
baf13c3
Compare
|
[operator: bolichen97] The GPT 6.1 run on
|
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 432 lines, more than the 400 it may approve on its own. The PR moves only the revocation-counter write (persist_next_revocation_gen) to a worker thread, under a new _bump_lock and never under the _gen_lock that every validation takes. The new value is published and the link/pin/consumed state cleared on the loop thread, with no await between them, so peer pins stay in force until the new value is live. This fixes the replay window GPT flagged on an earlier head. Overlapping logouts build on max(live, file), and a logout a newer one has already published over clears nothing. The failure path matches main. I checked against the code: interleavings with validators, overlapping logouts, the counter not yet loaded, the facade patch mirroring, and the remaining sync callers (tests only). The new tests hold the real rename and would fail without the change. Cancellation mid-write is reachable only at shutdown, because the repo leaves aiohttp's handler cancellation at its default (off), and the value on disk loads on restart. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
buluoray
left a comment
There was a problem hiding this comment.
Request changes on baf13c3. Both earlier security findings are fixed and test-guarded at this head: pins stay in force while the logout write is in flight (b5cb42d finding), and a superseded logout no longer clears pins bound after the newer logout (feb5052 GPT 6.1 finding); 504 related tests pass and both new guards turn a named test red when broken. CI on this head is all rate-limit infra noise, so no AI lane and no backend test lane produced a result. Decisive lines: src/kiro_crew/dashboard/token_auth_tokens.py:1139-1146 (advanced = publish_revocation_gen(new_gen); _state.clear_all() only if advanced), src/kiro_crew/dashboard/revocation_gen.py:172-184 (publish_revocation_gen never lowers the live value and returns whether it advanced), src/kiro_crew/dashboard/revocation_gen.py:142-154 (write under _bump_lock only, base = max(live, file))
baf13c3 to
ece4a3b
Compare
|
self-added: yes
|
|
self-added: yes
|
|
[operator: bolichen97] @buluoray thanks for checking
The two guards you named (pins stay in force during the write; a superseded logout keeps the newer pins) are unchanged. CI is re-running on the new head. Could you take another look once it is green? |
buluoray
left a comment
There was a problem hiding this comment.
Approve on ece4a3b. Every point behind our REQUEST_CHANGES on baf13c3 is fixed at the live head and each fix turns a named test red when reverted: a failed logout write no longer drops peer pins (GPT F1), and an unreadable counter file can no longer make an overlapping logout write a lower generation (GPT F2); the two earlier fixes (pins kept while the write is in flight, superseded logout keeps newer pins) still hold. GPT 6.1 and Opus 5.5 both have real "no blocking findings" results on this head, no [BLOCK-MERGE] exists, and every red check is noise. 8 backend test shards were still running when read. Decisive lines: src/kiro_crew/dashboard/token_auth_tokens.py:1140-1143 (async logout: persist on a worker thread, then _state.clear_all() only if publish_revocation_gen(new_gen); no try/finally, so a raise clears nothing), src/kiro_crew/dashboard/token_auth_tokens.py:1119-1120 (sync form: bump_revocation_gen() before _state.clear_all()), src/kiro_crew/dashboard/revocation_gen.py:151-156,176 (base = max(live, this process's last write); file read only when neither is known; _written set only after the rename succeeds), src/kiro_crew/dashboard/revocation_gen.py:189-190 (publish never lowers the live value)
|
[operator: bolichen97] The one red test on |
ece4a3b to
d2f735f
Compare
|
Rebased: main moved in src/kiro_crew/dashboard/handlers/core.py, src/kiro_crew/dashboard/token_auth.py, src/kiro_crew/dashboard/token_auth_tokens.py, docs/system-specs/modules/dashboard-token-auth.md, test/test_cli_gateway_client.py and test/test_token_auth_session_lifecycle.py since this head went green (c076070 -> 9584db1); CI re-running. The new base carries #35705, the fix for the flaky |
|
[operator: bolichen97] The five red checks on
Rerunning the failed jobs would test the same merge commit again, so it cannot pick up #36290. Getting this green needs one re-sync onto main. This PR has already used its three re-syncs, so I am not rebasing it. A project member can grant one more by commenting |
|
[operator: bolichen97] Smoke test of this head ( Verified on Ubuntu 24.04 (Linux x86_64), Python 3.12.3.
Requests issued while the logout ran (from the probe log): |
|
Re-sync budget reset Two of the red Backend Tests shards on [operator: bolichen97] |
kirocrew logout bumped the revocation counter on the gateway's event loop, and the bump held the lock every token validation takes across the temp-file write and rename. A slow disk (for example on-access antivirus scanning the new temp file) therefore stalled every request until the write finished, and past the loop watchdog's budget the gateway was killed. POST /api/logout now awaits revoke_all_sessions_async(), which runs only the counter's write (persist_next_revocation_gen) on a worker thread, under a lock that only writes take. Back on the loop, with no await between, it publishes the new value and, only when that advances the live value, clears the in-memory link, pin and consumed state, so no request ever sees a half-done logout: until the write lands every pre-logout pin still holds, and no handler can check an old token and mint its replacement across the publish. A logout that reaches the loop after a newer one has published, or whose write fails, leaves that state alone, so no pin is dropped while the cookies it guards still validate. The value is published only after the atomic rename succeeds and never lowered, and a write builds on the larger of the live value and the last value this process wrote, reading the file only when neither is known, so an overlapping logout never writes below one already written. The fail-closed rules are unchanged. Closes #34017
d2f735f to
942c0f6
Compare
|
Rebased: main moved in docs/system-specs/modules/dashboard-token-auth.md, src/kiro_crew/dashboard/handlers/core.py, src/kiro_crew/dashboard/token_auth.py, test/test_cli_gateway_client.py, test/test_dashboard_handlers_core_coverage_security_endpoints.py and test/test_token_auth_session_lifecycle.py since this head's CI ran (9584db1 -> e72cf2e); CI re-running. The new base carries #36290, the fix for |
Problem / Motivation
Goal:
kirocrew logouton a slow disk no longer freezes the gateway's event loop, so the loop watchdog cannot kill the gateway over the logout's revocation-counter write.POST /api/logout(api_logout, whichkirocrew logoutcalls) ranrevoke_all_sessions()directly on the event loop. That bumps the persisted revocation counter (bump_revocation_gen), which writes a temp file and renames it ontotoken_revocation.genwhile it holds_gen_lock. Every token validation takes the same lock throughcurrent_revocation_gen_or_none(). While the write is slow, for example while on-access antivirus scans the new temp file, no request is answered, and a stall past the watchdog's budget kills the gateway.Why it matters
kirocrew logoutcan stop the whole gateway: HTTP, chats and channels all wait, and past the watchdog's budget the gateway is killed and every running turn is cut.Not a goal
What changed (motivation → approach → change)
A slow disk now delays only the
kirocrew logoutrequest that writes the counter, never the gateway's event loop. Nothing else about a logout changes for a user: until the write lands every session works as before, and once it lands every pre-logout session ends at once.POST /api/logout(handlers/core.py)api_logoutranrevoke_all_sessions()on the event looprevoke_all_sessions_async()(token_auth_tokens.py)revocation_gen.py)bump_revocation_genheld_gen_lock, which every validation takes on the loop, across the temp-file write and the renamepersist_next_revocation_gen()picks the base, writes and renames under_bump_lockalone, and remembers the value it wrote;publish_revocation_gen()makes the value live under_gen_lock, never lowers it, and says whether it advanced;bump_revocation_gen()runs the two in a rowrevoke_all_sessions_async()persist_next_revocation_gen()on a worker thread; back on the loop, with no await between, it publishes the value and clears the link, pin and consumed state only when that publish advances the live valueThe publish and the clear stay on the loop because loop handlers rely on the logout being one step. A link exchange checks a token's generation and then mints its replacement with no await between, and a peer pin protects a cookie only while it exists. A publish or a clear from another thread could land in that gap, or before the new value is live, and let a pre-logout link mint a session that survives the logout, or let a cookie pass its pin from another address while the write is slow.
The new value is still published only after the atomic rename succeeds, so a token is never minted with a generation that is not on disk. A write builds on the larger of the live value and the last value this process wrote, and reads the file only when neither is known, as main does before the first load: the gateway is the file's only writer, so an overlapping logout never writes at or below a value another one has already written, whether or not the file can be read. The state is cleared only when a logout advances the live value. A logout that reaches the loop after a newer one published clears nothing, so a pin bound since the newer logout stays in force. An unreadable base or a failed write still fails the logout with the same 500, and changes nothing: the live value and every pin stay as they were, so no cookie the failed logout left valid loses its pin. The synchronous
revoke_all_sessions()follows the same rule: it clears the state only after its bump lands.dashboard-token-auth.mdstates the counter's lock and ordering rules, and the facade's type-checking list intoken_auth.pynames the new functions.Backwards compatibility
Compatible: no file format, file permission, API, config key or command changes. The same file is written with the same contents; only which thread writes it, which lock covers the write, and where the new value is published change.
Removes nothing: no path, key, record or public function is removed or renamed;
bump_revocation_gen()andrevoke_all_sessions()keep their signatures and results.bump_revocation_gen()builds on the larger of the live value and its own last write, which gives a different value from main only while another logout has written but not yet published. A logout whose bump fails keeps the in-memory link, pin and consumed state instead of dropping it, so a cookie the old generation still accepts keeps its pin.Tests
The full backend suite runs in CI. On Ubuntu 24.04 (Python 3.12.3), with this exact patch on base
6a61c8465f,scripts/local-gate.py's related sets gave backend 29311 + 25155 passed and frontend 13653 + 5869 passed; the 6 backend reds there (4 doctor-config tests, 1 git-fixture test, and a boot-composition test that sees a builtstatic/dist) fail the same way on the unfixed base on that host.mypy src/kiro_crew, flake8, isort, black, the flake-pattern ratchet (68 passed) and semgrep (0 findings) are clean. Determinism, on the identical patch atd2f735f7a7:test_revocation_gen.pypassed 20 times at-n0, 10 times at-n 4 --dist loadgroup --max-worker-restart=2besidetest_refresh_tokens.pyandtest_windows_replace_retry_sweep.py, and once in shuffled order withtest_token_auth.py(seed 7909, 182 tests).test_validation_reads_the_counter_while_a_bump_write_is_stuck: while a write is held before its rename, a counter read on another thread answers at once with the old value; the new value is live and on disk once the rename lands.test_a_pin_stays_in_force_while_the_logout_write_is_stuck: while the write ofrevoke_all_sessions_async()is held, the generation is unchanged and a pinned cookie is still refused from another address; afterwards that cookie is rejected as revoked.test_only_the_logout_write_leaves_the_event_loop: the rename runs on a worker thread, and the publish and the state clear run on the loop's thread.test_a_superseded_logout_keeps_the_pins_bound_after_the_newer_one: the logout that wrote 5 reaches the loop after the one that wrote 6 has published; a cookie minted and pinned in between still validates and is still refused from another address, and the live value stays 6.test_a_bump_builds_on_a_value_persisted_but_not_yet_published: with 4 live, two bumps that have not published write 5 and then 6.test_a_bump_never_writes_below_its_own_unpublished_write_when_the_file_cannot_be_read: with 5 and 6 written but 4 live, a bump whose file read fails writes 7, not 5.test_a_failed_logout_write_raises_and_changes_nothing: a failed write raisesOSErrorfrom both the async and the sync form, the live value stays the same, and a pinned cookie is still refused from another address.test_dashboard_handlers_core_coverage_security_endpoints.py,test_cli_gateway_client.pyandtest_token_auth_session_lifecycle.pypatchrevoke_all_sessions_async; the facade's known-seam list (test_token_auth_facade.py) and the Windows replace sweep name the new functions.Each new rule was broken on purpose and its test went red: running the whole synchronous revocation on a worker thread fails the pin test and the loop-thread test; publishing from the worker fails the loop-thread test; building only on the live value fails the overlap test; clearing on every publish fails the superseded-logout test; clearing on a failed write, in either form, fails the failed-logout test; falling back to the live value when the file cannot be read fails the unreadable-file test.
Proof: PROVEN test/test_revocation_gen.py::test_validation_reads_the_counter_while_a_bump_write_is_stuck, test/test_windows_replace_retry_sweep.py::test_converted_writer_renames_through_the_retry[dashboard/revocation_gen.py] (head 3ee714fd8a2e)
prove.pyran on3ee714fd8a, this exact patch on base6a61c8465f(treefa8a7e5dfa).Manual verification
On Ubuntu 24.04 (Python 3.12.3), with an isolated gateway (a temporary
KIROCREW_HOME, a free loopback port), the counter's temp-file write slowed by 3 s, and a realkirocrew logoutwhile/api/liveand a token-authenticated/api/cronswere polled:6a61c8465fMainThread)/api/live2.997 s, authenticated request 2.980 s/api/live0.012 s, authenticated request 0.020 sENOSPC)/api/live0.018 s, authenticated request 0.019 sOn the failed write,
kirocrew logoutexits 1 withHTTP 500: could not persist session revocation; logout not completed. On a normal disk the logout writes the counter, the pre-logout token gets 403 and a freshly minted token gets 200. All 7 tests intest/test_revocation_gen.pyfail against the unfixed base's code.Related Issues
Closes #34017
Sibling sweep
Swept: git grep -n "threading.Lock()" -- 'src/kiro_crew/dashboard/token_*.py' src/kiro_crew/dashboard/refresh_tokens.py src/kiro_crew/dashboard/revocation_gen.py
Sibling: src/kiro_crew/dashboard/revocation_gen.py:38 -- changed here
Sibling: src/kiro_crew/dashboard/token_auth_tokens.py:80 -- not affected -- #34031 on main holds this denylist lock for the dict only, never across the write
Sibling: src/kiro_crew/dashboard/refresh_tokens.py:214 -- not affected -- #34031 on main holds this refresh-store lock for the snapshot only, never across the write
Sibling: src/kiro_crew/dashboard/token_secret.py:830 -- not affected -- only the one-time key load holds it across disk I/O, and the boot runs that before the server accepts connections
Sibling: src/kiro_crew/dashboard/token_auth_app_scope.py:43 -- not affected -- the manifest read runs with the lock released
The search's other hits hold no write the loop waits on:
revocation_gen.py:39(_bump_lock) andrefresh_tokens.py:226(_write_sequence_lock) are taken only by writers,token_auth_tokens.py:81(_io_lock) only by the denylist's writes,refresh_tokens.py:923andtoken_auth_tokens.py:215guard building each store once,token_auth_tokens.py:249(TokenStateManager) is in memory only, andtoken_auth_tokens.py:1038(_app_secret_move_lock) is taken only byreplace_legacy_app_secret, never by a validation.Pattern harvest
Rule candidate: review-prompt
Pattern: a
threading.Lockthat event-loop code takes is held across file I/O, so the loop waits on the disk even when the write itself runs in a worker thread; and moving a whole synchronous sequence onto a worker thread lets loop handlers run between its steps, so only the blocking I/O should leave the loop.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)