Repository navigation
fix(dashboard): share slot list passes and stagger reconnects - #33825
Conversation
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The diff adds or changes no user-visible control, string or component. The website changes only affect reconnect timing and how slot data is fetched, so no screenshot is needed. [UX-REVIEWED] 57124fe Description read[DESCRIPTION-READ] 3e7e5a1dd95b011e7c6132e70913b7e4b6f0e6e27692429983db9db4f999b1ec That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the fix cuts the stall feedback loop on both sides (shared passes on the gateway, jitter and healthy-open reset in the client), and nothing renders new. Notes
[DESIGN-REVIEWED] 57124fe Description read[DESCRIPTION-READ] f76f0d8c1aad7f903660927d1ce0107f1e9820fd1b11ee5a0498e662f2e54513 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item addresses the storm in #33758. The one doubt is cost: sharing the connect pass is what forces the frame-hold machinery, and its benefit was never measured separately. Notes
What this change shipsInventory (8 items) — 7 justifiedIntent: stop one gateway loop stall from turning into a dashboard reconnect storm that makes the stall longer — FIX.
[FIRST-PRINCIPLES-REVIEWED] 57124fe Description read[DESCRIPTION-READ] 26576d884f4a3bb30ff3b421294f8d1209d95ed2964f7fc0150badb99fa6fb53 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Approving. Reconnect storms now share serialize_slots passes per audience, keyed by (loop, check_status, user), so one person never sees another person's slots (dashboard/slot_list_pass.py:92-111, chat_api/slot_detail.py:80-82). The PONG timer tolerates the gateway's own loop stall, and an unresponsive socket is still closed (dashboard/ws_heartbeat.py:31-44). Frames for a socket that is still connecting wait behind its snapshot (dashboard/ws_snapshot_hold.py:42-69, ws.py:828-834, 985-989). Client reconnects are jittered, with a backoff that resets only after a 10 s healthy open. No new listener, and no auth or origin change. Targeted tests pass, and three mutations turn named tests red.
|
Review notes from the #33769 side (I'm picking up #33769, the sibling sockets this PR lists under "Not a goal"). Plain comment, not a review. Scope vs #33769. At head Repro of the #33769 socket half. Locally (Python 3.12.15, aiohttp 3.14.5), this PR's Why the backend shards are red at this head. The failures come from this diff, not from main. I reproduced all of them locally on
The Windows shards (2, 3, 4, 7, 8) line up with the same Linux shard numbers, so they are most likely the same tests. Gaps (non-blocking). Not part of this PR's scope, noted for #33769: the sibling sweep in the body is accurate and I found no other |
7f0d5a5 to
98f35d8
Compare
…34167) * fix(dashboard): keep terminal and voice sockets through a loop stall The terminal and streaming dictation sockets used aiohttp's stock heartbeat, which closes a socket with 1006 when a loop stall makes its PONG timer run before the PONG already queued is read. Both now use LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws. The Mochi panel and Crew Companion sockets retried on a fixed doubling delay reset on every open, so every copy retried together after a stall and a gateway that drops new sockets was retried at the shortest delay. They now take a random part of up to half off each delay and reset the backoff only after a socket stayed open 10 s. Fixes #33769 * fix(dashboard): start app backoff over on a healthy watchdog replace A silence-watchdog replacement cleared the opened-at stamp without the healthy check the close path runs, so a long healthy socket replaced after a network change kept the grown delay. Run the same check there. Also reap the stall test's shell in a finally, and open each fake-timer try right after useFakeTimers.
98f35d8 to
dd69e8f
Compare
…34167) * fix(dashboard): keep terminal and voice sockets through a loop stall The terminal and streaming dictation sockets used aiohttp's stock heartbeat, which closes a socket with 1006 when a loop stall makes its PONG timer run before the PONG already queued is read. Both now use LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws. The Mochi panel and Crew Companion sockets retried on a fixed doubling delay reset on every open, so every copy retried together after a stall and a gateway that drops new sockets was retried at the shortest delay. They now take a random part of up to half off each delay and reset the backoff only after a socket stayed open 10 s. Fixes #33769 * fix(dashboard): start app backoff over on a healthy watchdog replace A silence-watchdog replacement cleared the opened-at stamp without the healthy check the close path runs, so a long healthy socket replaced after a network change kept the grown delay. Run the same check there. Also reap the stall test's shell in a finally, and open each fake-timer try right after useFakeTimers. Backport note: backend half only. The Mochi panel and Crew Companion reconnect changes (website/src/apps/**, website/src/utils/reconnectBackoff.ts and their tests) are app fixes and stay off this hotfix; the spec line keeps only the terminal and dictation socket sentence. (cherry picked from commit 1139925)
dd69e8f to
7867322
Compare
c3f06fc to
164aff4
Compare
164aff4 to
071c3c6
Compare
071c3c6 to
a0cb15f
Compare
|
self-added: yes
|
buluoray
left a comment
There was a problem hiding this comment.
Approve on a0cb15f. This PR stops a gateway stall from turning into a dashboard reconnect storm. Tabs that reconnect together share one slot-list pass, frames for a connecting tab wait behind its snapshot, and client retries are jittered. Both points of the earlier change request are fixed. The connect snapshot now takes its partial, held-keys and channel-hold labels from the same pass that took its rows (SlotPass, slot_list_pass.py:160-179, read at ws.py:989-995). A restore that finishes during the wait can no longer label an incomplete list complete, so unread dots are not drained. Reverting that makes test_a_rebuild_finished_after_the_pass_took_its_rows_keeps_the_snapshot_partial fail. approvalReconcile.test.tsx:156 now advances 10000 ms; putting back 1000 makes that file fail (1 failed, 21 passed). GPT's earlier hold-memory finding is also fixed: held frames count against WS_HARD_MAX_BUFFERED_BYTES and WS_HELD_MAX_FRAMES when they are held, and the release runs the stall check before each write. Removing either check reddens test_ws_snapshot_hold_ceiling.py (4 tests and 3 tests). Follow-up, not blocking: the new drop_superseded_slot_frames step (ws.py:1014-1016) has no test. Making it a no-op, or removing the fresh slots push after it, leaves every related test green. I read the code and it is correct: every slot frame builder writes type first, and the push gives the tab a full current list. A test that holds an older removal patch across a connect would guard it. Decisive lines: src/kiro_crew/dashboard/slot_list_pass.py:160-179 (_serialize takes rows and hold flags in one synchronous step), src/kiro_crew/dashboard/ws.py:918-995 (snapshot labels come from slot_pass, not live state), src/kiro_crew/dashboard/ws_snapshot_hold.py:126-144 (_admitted ceiling), src/kiro_crew/dashboard/ws_snapshot_hold.py:243-247 (_Release stall check), website/src/test/UseWebSocketCoverage.approvalReconcile.test.tsx:156 (10000), src/kiro_crew/dashboard/ws.py:1014-1016 (drop_superseded_slot_frames, untested: follow-up)
a0cb15f to
f5b3580
Compare
|
Coverage Gate is red at |
Dashboard reconnect storm: a stalled gateway loop dropped every dashboard socket at once and every tab reconnected together, each running two full slot list serializations on the event loop, which prolonged the stall. - Connect snapshots and GET /api/chat/slots now share serialize_slots passes per audience (slot_list_pass), so a reconnect burst costs one or two passes instead of one per tab. A state the pass table cannot key (no weak reference to it, or no hash) serializes for each caller at once, unshared. - A connecting socket still registers right after its handshake, so the catch-up reads its tab sends on open are answered with the socket registered; the hub holds that socket's frames until its connect snapshot is out (ws_snapshot_hold), so the snapshot stays its first frame. An awaited owner send to such a socket counts only once the send of its held frame completes. - What a socket holds while it waits for its snapshot counts against the hub's per-tab buffer ceiling (WS_HARD_MAX_BUFFERED_BYTES, and WS_HELD_MAX_FRAMES frames), and the release runs the stall check before it writes each held frame, so a tab that never reads is still dropped at that ceiling. - The dashboard socket restarts its PONG window instead of closing when the heartbeat timeout ran late or data is already queued, so a loop stall no longer drops healthy sockets (ws_heartbeat). - The client takes a random part of up to half off each reconnect delay, resets its backoff only after a socket stayed open 10 s, and the health probe no longer skips the backoff after an early drop. - A reconnect reads its slot list from the connect snapshot instead of a second GET; an empty snapshot, or none within 10 s, still reads it over HTTP.
buluoray
left a comment
There was a problem hiding this comment.
Approve on 57124fe. This head is a rebase of the head we approved (f5b3580) onto current main, with no change to the patch: the 48 files and every added and removed line are the same on both heads. The push dismissed our approval, so this renews it. The fixes still hold as reviewed: the connect snapshot takes its partial, held-keys and channel-hold labels from the pass that took its rows, so a restore that finishes during the wait cannot drain unread dots; held frames for a connecting tab stay under the byte and frame ceiling; the release checks for a stall before each write; and client reconnects are jittered. All checks on this head pass. The follow-up from our earlier review is still open and does not block: the drop_superseded_slot_frames step in ws.py has no test of its call site. Decisive lines: src/kiro_crew/dashboard/ws.py:987-996 (snapshot labels from slot_pass), src/kiro_crew/dashboard/ws_snapshot_hold.py:126-144 (_admitted ceiling), src/kiro_crew/dashboard/ws_snapshot_hold.py:243-247 (_Release stall check), website/src/hooks/websocket/connection.ts:195,219 (jittered delays), src/kiro_crew/dashboard/ws.py:1015-1017 (drop_superseded_slot_frames, still untested: follow-up)
Problem / Motivation
Goal: A gateway event loop stall no longer turns into a dashboard reconnect storm: sockets that answered the heartbeat stay open, tabs come back spread out, and tabs reconnecting together share slot list passes instead of costing two each.
When the gateway's event loop stalls past the dashboard socket heartbeat, every open tab loses its socket at the same moment (#33758). On main the way the tabs come back makes the stall longer: each reconnect runs two full slot list serializations on the loop, every tab retries on the same 1, 2, 4, 8 and 10 second schedule, and a gateway that accepts sockets and then drops them is retried about once a second by every tab.
Why it matters
Anyone with the dashboard open in several tabs or on several devices, on a gateway with many chat sessions, is hit. One long stall feeds itself: on main, with 12 tabs and 600 sessions, a gateway that accepted and then dropped sockets took 108
/api/wsupgrades in 30 seconds and spent about 10 of those seconds serializing slot lists on the event loop. Every chat, channel reply and cron on that gateway waits behind that work while the dashboard shows it as disconnected.Not a goal
serialize_slotsitself cheaper or moving it off the event loop.push_slots_update).What changed (motivation → approach → change)
After a gateway stall, dashboard tabs keep the sockets that answered in time, come back at spread-out moments, and share the slot list work, so one stall no longer feeds the next.
slot_list_pass.py,ws.py,chat_api/slot_detail.py)/api/wsconnect and everyGET /api/chat/slotsserializes the whole list on the loopws_snapshot_hold.py,websocket_hub.py)ws_heartbeat.py)connection.ts,useDashboardHealthProbe.ts)reconnectCatchUp.ts,useWebSocket.ts)GET /api/chat/slotsAfter a loop stall every tab reconnected at once, and the reconnects held the loop long enough to cause the next drop. Each reconnecting tab asked for the whole slot list twice (the connect snapshot and the catch-up GET), and the gateway serialized each request on its own, so N tabs cost 2N passes on the loop that had just stalled. aiohttp's PONG timer, which expired during the stall, ran before the PONG already waiting in the kernel was read, so sockets that answered in time were closed with code 1006. Every tab used the same delays and reset its backoff on every open, so all of them retried together, about once a second against a gateway that drops new sockets.
shared_slot_listnow serves connects and GETs per audience (include_check_status,dashboard_user). A caller after a quiet window serializes at once. Callers inside the window share one trailing pass, taken after all of them asked, so no caller gets an older list than before. The window is three times the last pass, between 20 ms and 2 s, so a cheap list makes nobody wait. A connecting socket still joins the registry right after its handshake, so no change after the reads its tab sends on open misses it. While the connect waits for a pass, the hub holds every frame for that socket (ws_snapshot_hold) and queues them in order right behind the snapshot, so the snapshot is still its first frame. An owner send that awaits delivery counts a held frame only once its send completes, withinWS_OWNER_SEND_MAX_S. The hold ends before the snapshot drains, so a tab that stops reading is still dropped by the stall check. The dashboard socket usesLagTolerantWebSocketResponse: a PONG timer that ran more than a second late, or that finds data already read, starts the window again; one that runs on time with nothing received still closes.In the dashboard client,
connection.tstakes a random part of up to half off each reconnect delay and starts the backoff over only after a socket stayed open 10 s (RECONNECT_HEALTHY_MS). The health probe brings a reconnect forward only when the last socket was refused at its handshake, as after a refused relay token renewal, and then within 1 s. The reconnect catch-up reads the list from the connect snapshot (createReconnectSnapshotWait), and the window that holds back unread marks and chimes runs from the open until that snapshot is applied. An empty snapshot, or none within 10 s, still reads the list over HTTP, because only a GET reply clears a list that went empty while the socket was down. Comments inslotResidue.tsanddashboardSlice.tsname that read;learn-cron-dashboard.mdandfrontend-conventions.mddescribe the shared pass and the snapshot, andtesting-conventions.mdsays a registered socket can still hold its frames.Backwards compatibility
Compatible: the
slotsframe and theGET /api/chat/slotsreply carry the same payload as on main, both callers get a list taken no earlier than they asked, and the heartbeat interval and close codes are unchanged. A frame for a socket whose connect waits for a shared pass goes out right after that socket's snapshot, at mostPASS_WINDOW_MAX_S(2 s) later than on main, in the order it was sent. An awaited owner send to such a socket returns once that frame's send completes, withinWS_OWNER_SEND_MAX_S(30 s), where on main it was sent at once. A waiting socket that would hold more thanWS_HARD_MAX_BUFFERED_BYTES(16 MiB) orWS_HELD_MAX_FRAMESframes, or whose release takes its buffer past that ceiling, is dropped, as main drops a tab with that much in its buffer, and an awaited owner send to it is not a delivery.Removes nothing: the reconnect catch-up no longer sends
GET /api/chat/slotswhen the connect snapshot already carries the list, but still sends it for an empty or missing snapshot, and the route and its other callers are unchanged.SocketConnection.resetBackoffbecomesnoteOpeninsidewebsite/src/hooks/websocket/, and its one caller,useWebSocket.ts, changes with it.Tests
test/test_slot_list_pass.py: eight concurrent GETs, or eight concurrent connects, take two passes; a GET after a quiet window takes its own pass at once; a caller inside the window gets a list taken after it asked; each audience keeps its own window; a connect that waits for the trailing pass is registered for the whole wait and a notification sent then reaches it right after its snapshot, before that snapshot drains; a connect whose pass failed still gets that notification; a failed pass raises for every caller; a cancelled caller is dropped; a state the table cannot key (no weak reference, or no hash) serializes for every caller.test/test_ws_snapshot_hold.py, through the hub's own send paths: broadcast, owner text and owner binary frames for a waiting socket go out after its snapshot, in order, and later frames go straight out; an owner send counts only once its held frame went out, and one that fails at the release is not a delivery (text and binary); a follow-up card for a tab that drops before its snapshot takes the no-client fallback; a held owner send is not dropped by the stall window while it waits, its stall window starts at the release, andWS_OWNER_SEND_MAX_Scounts its held time (never released, or released late); held frames reach the socket while the snapshot still waits to drain; a failed send is logged and the hold still ends.test/test_ws_heartbeat_loop_lag.py, over a raw TCP client: a PONG sent before a loop stall keeps the socket; the stock aiohttp socket drops that same peer with 1006 (a control that shows when aiohttp fixes this upstream); a silent peer is still dropped, and one that stays silent through a stall is dropped one window later.dashboard_ws.LagTolerantWebSocketResponseinstead ofweb.WebSocketResponse, andtest_dashboard_ws_stalled_client.pystarts measuring a stalled tab once neither tab holds frames for its snapshot, not once both are registered.test_members_eventlog_wiring_live_writers.pyfinds the snapshot send, started as a task, and checks that the connect awaits it before themembers_subscribedbaseline.reconnectCatchUp.snapshotWait.test.tspins the snapshot window (rows end it, an empty snapshot or no snapshot within the wait reads once, a closed socket disarms it, an earlier socket's read cannot end the next window).UseWebSocketCoverage.lifecycle.test.tsxanduseWebSocketReconnect.test.tspin the jitter bounds, the backoff kept after an early drop and reset after a healthy open, the health probe path, and the reconnect that reads no list over HTTP. The other changed frontend tests stop expecting the reconnect GET.Runs on a Linux development host, at this head (
62a43216de):-n0: 320 passed.test_dashboard_ws_stalled_client.py(a tab that reads nothing is dropped just over the limit),test_ws_owner_binary_delivery.py,test_slots_diff*.pyandtest_members_owner_gate.py,-n0: 219 passed.check_comment_history,check_brand_name,check_file_line_cap, the website build,tsc, eslint,i18n:renderandbench:render, plustest_flake_pattern_ratchet.py(68 passed) and the repository's semgrep rules on the diff (0 findings): all 64 jobs exit 0.Fixed after review on
98f35d8fa5(GPT 6.1: an owner frame held for a connect snapshot counted as delivered before its send completed, so a follow-up card for a tab that dropped first was reported as shown), and rebased onto main27cb69f8f6. The rebase resolved the overlap with main's relay quiet reconnect inconnection.tsanduseWebSocket.ts; two tests in main'suseWebSocket.relayQuietReconnect.test.tsnow let the health probe's reconnect take its at most 1 s, and a new one,the health probe brings a stopped session back after an earlier early drop, fails when the backoff is settled after the token-renewal stop instead of before it. On a DevSpace, atdd69e8fb11, each command exit 0 unless noted:pytest -n0 test/test_ws_snapshot_hold.py: 10 passed. With the two production files of the fix (websocket_hub.py,ws_snapshot_hold.py) at their previous text: exit 1, 8 failed, among themtest_a_tab_that_drops_before_its_snapshot_is_not_a_delivered_followup(the reply says "Follow-up card shown below the composer.")._await_owner_send, each exit 1 in the test that names it: the stall window measured from the call, the held wait unbounded, the cap measured from the release, and a held send counted at the release.pytest -n 6 --dist loadgroup --max-worker-restart=2on the 34 backend test files that drive owner sends, the snapshot hold, the shared pass or the heartbeat.test_flake_pattern_ratchet.py,check_black_formatting.py, black on the changed files, isort, flake8, mypy,check_subprocess_encoding.py,check_comment_history.py,check_brand_name.py,check_harness_parity.py,check_sync_io_in_async.py,check_file_line_cap.py,check_loop_bound_locks.py,check_testpaths_coverage.py,docs-lint.shand the repository's semgrep rules on the diff.tsc -p tsconfig.app.json, eslint on the changed website files, and vitest on the reconnect, health probe and websocket hook tests.dd69e8fb11: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 3 passes, 0.25 per tab).Rebased onto main after review; git range-diff shows the patch unchanged; related tests re-run on a DevSpace: at
7f0d5a5013on main8ec0a78355, the 13 changed backend test files 320 passed,test_ws_owner_binary_delivery.py,test_members_owner_gate.pyand the threetest_slots_diff*.pyfiles 48 passed, and the 11 changed website test files 246 passed. Wherever this body says "this head (62a43216de)", it means the reviewed commit;7f0d5a5013is that same patch on a newer main.Fixed after CI on
7f0d5a5013, which failed 13 tests on main's callers of this code: 12 that pass aSimpleNamespacestate toGET /api/chat/slots(test_member_chat_folders_gate.py,test_crew_mode_workers.py,test_chat_slots_crew_view.py;TypeError: cannot create weak reference), and the source pintest_members_eventlog_wiring_live_writers.py::test_the_connect_baseline_call_site_exists_in_the_ws_handler(ValueError: substring not found). On a DevSpace, at98f35d8fa5on main73e3275d98, each command exit 0 unless noted:pytest -n0on the three caller files: 12 failed before the fix, all pass after; the pin: 1 failed before, passes after.pytest -n0on the newtest_a_state_the_table_cannot_key_serializes_for_every_caller(2 cases).pytest -n 4 --dist loadgroup --max-worker-restart=2on the 14 changed backend test files (the pin file among them), the three caller files,test_ws_owner_binary_delivery.py,test_members_owner_gate.pyandtest_slots_diff*.py.test_flake_pattern_ratchet.py,check_black_formatting.py,check_subprocess_encoding.py, isort, flake8,mypy --platform linux,check_comment_history.py,check_brand_name.py,check_file_line_cap.py,docs-lint.sh,check_harness_parity.pyand the repository's semgrep rules on the diff.tsc -p tsconfig.app.json.1ece378503, each red again (exit 1):slot_list_pass.pywithout the fallback fails the 12 caller tests and the new test; the pin at its old text fails withValueError;ws.pywithawait snapshot_sendmoved after the baseline fails the pin by assertion.1ece378503.-n0, 10 of 10 at-n 4besidetest_slot_list_pass.py, and 5 shuffled orders of their 25 tests, at this patch on main1ece378503.Proof: not provable -- other: the tests import three new modules (slot_list_pass, ws_heartbeat, ws_snapshot_hold), so reverting every production hunk only errors at import (prove.py: INCONCLUSIVE at ab747b294a4a); the targeted reverts below fail by assertion
Targeted reverts instead, each in a throwaway worktree, each red by assertion in the test named (the first three bullets at
03c2b65167, which differs from this head only in the stalled-client test's wait):ws.pyandslot_list_pass.pyregistering the socket only when its pass takes the list:test_a_waiting_connect_is_registered_and_gets_its_snapshot_first.test_a_connect_whose_snapshot_failed_still_gets_the_frames_held_for_it.test_frames_held_for_a_connect_go_out_before_its_snapshot_drains. The release dropping the held frames, or owner sends not held:test_hub_frames_wait_for_the_snapshot_and_then_go_straight_out.ws.pyandslot_detail.pyback to serializing per request,slot_list_pass.pykept:test_eight_concurrent_gets_take_two_passesandtest_eight_concurrent_connects_take_two_passesfail (atbe55d8ac5a, an earlier build of this patch).LagTolerantWebSocketResponse._pong_not_receiveddeferring to aiohttp:test_a_pong_sent_in_time_keeps_the_socket_through_the_stallfails; the other 3 pass (atbe55d8ac5a).connection.ts,useWebSocket.ts,reconnectCatchUp.tsanduseDashboardHealthProbe.tsat the parent: 21 of the 246 frontend tests fail, among them the jitter, backoff and snapshot window tests above (atbe55d8ac5a; the frontend is unchanged since).Determinism, on the same host while other jobs loaded it:
test_slot_list_pass.py,test_ws_snapshot_hold.py,test_ws_heartbeat_loop_lag.pyandtest_dashboard_ws_stalled_client.py: 20 of 20 runs green at-n0, and 10 of 10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_dashboard_state_ws_unregister.pyandtest_slot_list_rebuild.py, at this head.--sequence.shuffle(seed 22102): 246 of 246 green (atbe55d8ac5a; the frontend is unchanged since).Fixed after review on
164aff4d2c(the connect snapshot labelled rows from a trailing shared pass with the liveslotsPartial,slotsHeldKeysandslotsHoldChannels, so a restore that finished during the wait made incomplete rows read as complete): the pass now carries the flags read in the same synchronous section that took its rows (SlotPass,shared_slot_pass), and the connect sends those. Newtest/test_ws_snapshot_hold_flags_with_shared_pass.py(2 tests); fourSimpleNamespacestate stubs in existing tests gained the two hold methods. Reproduce at071c3c636b:PYTHONPATH=src python -m pytest -n0 test/test_ws_snapshot_hold_flags_with_shared_pass.py: exit 0, 2 passed; with theslot_list_pass.pyandws.pyhunks reverted: exit 1, 2 failed ("incomplete rows were labelled complete"). The earlier DevSpace run of the six affected test files: 122 passed. GitHub CI runs the rest.Fixed after CI and review on
071c3c636b:slot_list_pass._serialize:test_slot_list_rebuild.py::TestConnectSnapshot::test_app_token_snapshot_omits_foreign_hold_state("Expected 'slot_list_held_keys' to not have been called") andtest_member_chat_folders_gate.py::TestSessionListMemberFilter::test_person_sees_every_sessionand::test_member_slots_omit_foreign_sessions(AttributeErroron aSimpleNamespacestate). The pass now reads the held keys and the channel hold only for the dashboard audience, as main'sws.pydid.WS_HARD_MAX_BUFFERED_BYTES, and the release then wrote all of it with no stall check. What a socket holds now counts against that ceiling andWS_HELD_MAX_FRAMES(one frame per KiB of it), a frame past either drops the socket, and the release runs the hub's stall check before it writes each held frame. Newtest/test_ws_snapshot_hold_ceiling.py(8 cases).Reproduce on a DevSpace, at this patch's tree (
21c39fefdd, heada0cb15fb94), withPYTHONPATH=src python -m pytest -n0:slot_list_pass.pyat071c3c636b: exit 1, 3 failed with the CI errors.test/test_ws_snapshot_hold_ceiling.py: exit 0, 8 passed. Withws_snapshot_hold.pyandwebsocket_hub.pyat071c3c636b, plus only theWS_HELD_MAX_FRAMESconstant the test imports: exit 1, 7 failed, all buttest_superseded_slot_frames_leave_the_held_total, which guards against still counting a discarded list. Only the admission check removed: exit 1, 4 failed. Only the release check removed: exit 1, 3 failed.Rebased onto main after review; patch unchanged (
git range-diffshows the one commit with only a context line moved intest/test_slots_diff_ws.py). Maine1b079ce1ecarries #35625, the fix for the two tree-scan tests #35088 broke on main. On a DevSpace, ata0cb15fb94merged into maine8283ceb85(the tree CI tested),test_wait_discipline_ratchet.py::test_the_scan_matches_known_sitesandtest_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkersexit 1 with the CI errors; atf5b3580235(this patch on maine1b079ce1e) both pass, and with #35625's two files reverted onf5b3580235both exit 1 again with the same errors. Related tests re-run there atf5b3580235, each exit 0:pytest -n 4 --dist loadgroup --max-worker-restart=2on the 20 changed backend test files, the 15 test files main changed since the old base that sendGET /api/chat/slots, open/api/wsor stub the dashboard socket,test_ws_owner_binary_delivery.py,test_members_owner_gate.py,test_member_chat_folders_gate.py,test_slots_diff.py,test_slots_diff_budget.py, both tree-scan files andtest_flake_pattern_ratchet.py: 1569 passed.Rebased onto main after review; patch unchanged (
git range-diffshows the one commit with only the import context insrc/kiro_crew/dashboard/ws.pychanged: main's #35732skill_read_aheadimport now sits beside this patch'sslot_list_passimport). Main1bccaa12f0carries #35725, the fix for the Coverage Gate per-file floor #35088 broke. On a DevSpace, at57124fee7a(this patch on main1bccaa12f0) unless noted, each exit 0:pytest -n0ontest_dashboard_skill_read_ahead.py(main's source pin over_serve_ws),test_members_eventlog_wiring_live_writers.py,test_slot_list_pass.py,test_ws_snapshot_hold.py,test_ws_snapshot_hold_ceiling.pyandtest_ws_snapshot_hold_flags_with_shared_pass.py: 75 passed.test_gateway_restart_readers.py,test_update_boot_apply.pyandtest_managed_launcher_restart.pyalone:gateway_restart.py53 of 53 statements,update_boot_apply.py150 of 151. With test(updates): cover gateway_restart and update_boot_apply to the 80% floor #35725's two test files reverted on8a3cd62cf7(this patch on mainb98b7cfaa3), the same run gives 24 of 53 and 113 of 151, below the 80% floor and the 113 the CI log reported atf5b3580235.8a3cd62cf7: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 3 passes, 0.25 per tab).Manual verification
A repro script ran a real gateway, its real dashboard bundle in headless Chromium and scripted browser-like clients on a Linux development host: at this head (
62a43216de) and at its parent (aa08664c56, main when that side ran), about 4 minutes per side.aa08664c56)62a43216de)The script exited 1 (bug present) on main and 0 (bug absent) at this head.
Screenshots / video
Why no screenshot: the frontend change is reconnect timing and which request carries the slot list, so no rendered surface changes.
Related Issues
Fixes #33758
Siblings of the same defects on other sockets: #33769. Related closed work: #22372.
Sibling sweep
Swept: git grep -nE "WebSocketResponse(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/terminal.py:665 -- tracked in #33769
Sibling: src/kiro_crew/dashboard/stt_stream.py:1642 -- tracked in #33769
Sibling: src/kiro_crew/dashboard/handlers/sso_login.py:36 -- not affected -- sends one error frame and closes, so it never waits on a PONG
Sibling: src/kiro_crew/browser_harness/relay.py:468 -- not affected -- no heartbeat is armed on this socket
The other hits in those two files, the terminal refusal sockets (
terminal.py:585,terminal.py:638) and the disabled-streaming reply (stt_stream.py:1630), close right after one error frame and never wait on a PONG.Swept: git grep -nE "serialize_slots(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/crew_workers.py:72 -- not affected -- the Crew page's own read, which a reconnect does not send
Sibling: src/kiro_crew/dashboard/state_parts/slots_push.py:325 -- not affected -- the slots broadcast already coalesces its passes on a leading and a trailing edge
Swept: git grep -nE "/api/ws?caps" -- website/src
Sibling: website/src/hooks/websocket/connection.ts -- changed here
Sibling: website/src/apps/mochi/panel/panelBridge.ts:598 -- tracked in #33769
Sibling: website/src/apps/crew-companion/sessionWatch.ts:451 -- tracked in #33769
Pattern harvest
Rule candidate: review-prompt
Pattern: a reconnect path that re-reads, over a second request, the full state its new socket's first frame already carries, or retries on a fixed schedule with no jitter
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Fixed after review on
515d56b39c(GPT 6.1: an app socket's connect snapshot used scopes read before the shared pass wait, so an app disabled during the wait still got another session's metadata and theyoloposture; the snapshot now re-judges the scopes witheffective_allowed_eventsafter the wait), and the flake-pattern ratchet ontest_unregistering_a_socket_ends_the_owner_sends_held_for_it(the 5 s bound is the assertion, now markedflake-ok). On a DevSpace, at92072a62a6, each command exit 0 unless noted:test/test_ws_app_scope_after_shared_pass.py: 2 passed. Withws.pyat its previous text the revoked-app case fails with "a disabled app was sent another session's metadata"; the control (an app that stays enabled keeps its scoped snapshot andyolo) passes.pytest -n0on that file,test_ws_snapshot_hold.py,test_flake_pattern_ratchet.py,test_slot_list_pass.py,test_ws_heartbeat_loop_lag.py,test_ws_event_scoping_live.py,test_chat_slots_crew_view.py,test_dashboard_ws_stalled_client.pyandtest_members_eventlog_wiring_live_writers.py: 195 passed.mypy --platform linux src/kiro_crew: clean.92072a62a6: exit 0, bug absent in all five scenarios.Fixed after CI on
92072a62a6, wheretest_slot_patch_frames.py::test_pin_reaches_a_capable_tab_as_a_patch_and_an_old_tab_as_a_listtimed out: the test patches only the connect-time scope loader, so the post-pass re-read hit the real reader, found the app not installed and the hub stopped sending it the pin's list. The test now patches the real reader too. On a DevSpace, at9dffe6fed6: that file andtest_flake_pattern_ratchet.py98 passed (exit 0); the eight test files that patchload_declared_events_for_connect210 passed with the one failure fixed here; black and flake8 clean.Rebased onto main
35dc109f0dafter review; patch unchanged (git range-diff shows=). The review lanes failed closed on the stale base ("could not read the merge-base"). Related tests re-run on a DevSpace ate25d406390: the changed backend test files plus the slot patch, slots diff, owner binary delivery, owner gate and chunk scope files, 274 passed.Fixed after CI on
e25d406390, where New-test stability timed outtest_slot_patch_frames.py::test_pin_reaches_a_capable_tab_as_a_patch_and_an_old_tab_as_a_listagain: the test now starts from an empty declaration cache and refresh set, so a reading another test cached for the app cannot reach the hub. I could not reproduce the CI timeout on a DevSpace (the 15 stability files, 3 passes,-n auto --timeout=120: 368 passed each pass, before and after this change), so this is a hermetic-isolation fix of the one state the test shared.