Skip to content

fix(dashboard): share slot list passes and stagger reconnects - #33825

Merged
bolichen97 merged 1 commit into
mainfrom
fix/sweep-g5-dashboard-reconnect-storm
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/sweep-g5-dashboard-reconnect-storm

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway event loop stall no longer turns into a dashboard reconnect storm: sockets that answered the heartbeat stay open, tabs come back spread out, and tabs reconnecting together share slot list passes instead of costing two each.

When the gateway's event loop stalls past the dashboard socket heartbeat, every open tab loses its socket at the same moment (#33758). On main the way the tabs come back makes the stall longer: each reconnect runs two full slot list serializations on the loop, every tab retries on the same 1, 2, 4, 8 and 10 second schedule, and a gateway that accepts sockets and then drops them is retried about once a second by every tab.

Why it matters

Anyone with the dashboard open in several tabs or on several devices, on a gateway with many chat sessions, is hit. One long stall feeds itself: on main, with 12 tabs and 600 sessions, a gateway that accepted and then dropped sockets took 108 /api/ws upgrades in 30 seconds and spent about 10 of those seconds serializing slot lists on the event loop. Every chat, channel reply and cron on that gateway waits behind that work while the dashboard shows it as disconnected.

Not a goal

What changed (motivation → approach → change)

After a gateway stall, dashboard tabs keep the sockets that answered in time, come back at spread-out moments, and share the slot list work, so one stall no longer feeds the next.

Part On main With this PR
Gateway slot list passes (slot_list_pass.py, ws.py, chat_api/slot_detail.py) every /api/ws connect and every GET /api/chat/slots serializes the whole list on the loop callers of one audience share passes: a lone caller is served at once, callers that arrive together share one trailing pass
Gateway connect frames (ws_snapshot_hold.py, websocket_hub.py) the snapshot is sent with no wait after registration, so nothing can come before it a connect that waits for a shared pass is registered first, and the hub holds its frames until the snapshot is queued; an owner send that waits for delivery counts a held frame only once its send completes; what a waiting socket holds counts against the per-tab buffer ceiling, and the release drops a socket that passes it
Gateway heartbeat (ws_heartbeat.py) aiohttp closes a socket whose PONG was queued behind a stall the PONG window starts again when the timer ran late or data is already waiting
Client backoff (connection.ts, useDashboardHealthProbe.ts) the same delays for every tab, reset on every open; the health probe skips the backoff a random part of up to half comes off each delay; the backoff resets only after a socket stayed open 10 s
Client catch-up (reconnectCatchUp.ts, useWebSocket.ts) sends its own GET /api/chat/slots uses the connect snapshot; reads over HTTP only when that is empty or missing

After a loop stall every tab reconnected at once, and the reconnects held the loop long enough to cause the next drop. Each reconnecting tab asked for the whole slot list twice (the connect snapshot and the catch-up GET), and the gateway serialized each request on its own, so N tabs cost 2N passes on the loop that had just stalled. aiohttp's PONG timer, which expired during the stall, ran before the PONG already waiting in the kernel was read, so sockets that answered in time were closed with code 1006. Every tab used the same delays and reset its backoff on every open, so all of them retried together, about once a second against a gateway that drops new sockets.

shared_slot_list now serves connects and GETs per audience (include_check_status, dashboard_user). A caller after a quiet window serializes at once. Callers inside the window share one trailing pass, taken after all of them asked, so no caller gets an older list than before. The window is three times the last pass, between 20 ms and 2 s, so a cheap list makes nobody wait. A connecting socket still joins the registry right after its handshake, so no change after the reads its tab sends on open misses it. While the connect waits for a pass, the hub holds every frame for that socket (ws_snapshot_hold) and queues them in order right behind the snapshot, so the snapshot is still its first frame. An owner send that awaits delivery counts a held frame only once its send completes, within WS_OWNER_SEND_MAX_S. The hold ends before the snapshot drains, so a tab that stops reading is still dropped by the stall check. The dashboard socket uses LagTolerantWebSocketResponse: a PONG timer that ran more than a second late, or that finds data already read, starts the window again; one that runs on time with nothing received still closes.

In the dashboard client, connection.ts takes a random part of up to half off each reconnect delay and starts the backoff over only after a socket stayed open 10 s (RECONNECT_HEALTHY_MS). The health probe brings a reconnect forward only when the last socket was refused at its handshake, as after a refused relay token renewal, and then within 1 s. The reconnect catch-up reads the list from the connect snapshot (createReconnectSnapshotWait), and the window that holds back unread marks and chimes runs from the open until that snapshot is applied. An empty snapshot, or none within 10 s, still reads the list over HTTP, because only a GET reply clears a list that went empty while the socket was down. Comments in slotResidue.ts and dashboardSlice.ts name that read; learn-cron-dashboard.md and frontend-conventions.md describe the shared pass and the snapshot, and testing-conventions.md says a registered socket can still hold its frames.

Backwards compatibility

Compatible: the slots frame and the GET /api/chat/slots reply carry the same payload as on main, both callers get a list taken no earlier than they asked, and the heartbeat interval and close codes are unchanged. A frame for a socket whose connect waits for a shared pass goes out right after that socket's snapshot, at most PASS_WINDOW_MAX_S (2 s) later than on main, in the order it was sent. An awaited owner send to such a socket returns once that frame's send completes, within WS_OWNER_SEND_MAX_S (30 s), where on main it was sent at once. A waiting socket that would hold more than WS_HARD_MAX_BUFFERED_BYTES (16 MiB) or WS_HELD_MAX_FRAMES frames, or whose release takes its buffer past that ceiling, is dropped, as main drops a tab with that much in its buffer, and an awaited owner send to it is not a delivery.

Removes nothing: the reconnect catch-up no longer sends GET /api/chat/slots when the connect snapshot already carries the list, but still sends it for an empty or missing snapshot, and the route and its other callers are unchanged. SocketConnection.resetBackoff becomes noteOpen inside website/src/hooks/websocket/, and its one caller, useWebSocket.ts, changes with it.

Tests

  • New test/test_slot_list_pass.py: eight concurrent GETs, or eight concurrent connects, take two passes; a GET after a quiet window takes its own pass at once; a caller inside the window gets a list taken after it asked; each audience keeps its own window; a connect that waits for the trailing pass is registered for the whole wait and a notification sent then reaches it right after its snapshot, before that snapshot drains; a connect whose pass failed still gets that notification; a failed pass raises for every caller; a cancelled caller is dropped; a state the table cannot key (no weak reference, or no hash) serializes for every caller.
  • New test/test_ws_snapshot_hold.py, through the hub's own send paths: broadcast, owner text and owner binary frames for a waiting socket go out after its snapshot, in order, and later frames go straight out; an owner send counts only once its held frame went out, and one that fails at the release is not a delivery (text and binary); a follow-up card for a tab that drops before its snapshot takes the no-client fallback; a held owner send is not dropped by the stall window while it waits, its stall window starts at the release, and WS_OWNER_SEND_MAX_S counts its held time (never released, or released late); held frames reach the socket while the snapshot still waits to drain; a failed send is logged and the hold still ends.
  • New test/test_ws_heartbeat_loop_lag.py, over a raw TCP client: a PONG sent before a loop stall keeps the socket; the stock aiohttp socket drops that same peer with 1006 (a control that shows when aiohttp fixes this upstream); a silent peer is still dropped, and one that stays silent through a stall is dropped one window later.
  • Nine backend test files point their socket stub at dashboard_ws.LagTolerantWebSocketResponse instead of web.WebSocketResponse, and test_dashboard_ws_stalled_client.py starts measuring a stalled tab once neither tab holds frames for its snapshot, not once both are registered.
  • test_members_eventlog_wiring_live_writers.py finds the snapshot send, started as a task, and checks that the connect awaits it before the members_subscribed baseline.
  • Frontend: reconnectCatchUp.snapshotWait.test.ts pins the snapshot window (rows end it, an empty snapshot or no snapshot within the wait reads once, a closed socket disarms it, an earlier socket's read cannot end the next window). UseWebSocketCoverage.lifecycle.test.tsx and useWebSocketReconnect.test.ts pin the jitter bounds, the backoff kept after an early drop and reset after a healthy open, the health probe path, and the reconnect that reads no list over HTTP. The other changed frontend tests stop expecting the reconnect GET.

Runs on a Linux development host, at this head (62a43216de):

  • pytest, all 13 changed backend test files, -n0: 320 passed.
  • pytest, 14 test files that drive the hub's send paths, among them test_dashboard_ws_stalled_client.py (a tab that reads nothing is dropped just over the limit), test_ws_owner_binary_delivery.py, test_slots_diff*.py and test_members_owner_gate.py, -n0: 219 passed.
  • vitest, all 11 changed website test files: 11 files and 246 tests passed.
  • Every other gate of the profile, among them black, isort, flake8, mypy, check_comment_history, check_brand_name, check_file_line_cap, the website build, tsc, eslint, i18n:render and bench:render, plus test_flake_pattern_ratchet.py (68 passed) and the repository's semgrep rules on the diff (0 findings): all 64 jobs exit 0.
  • The full backend and frontend suites and the Electron tests run in CI.

Fixed after review on 98f35d8fa5 (GPT 6.1: an owner frame held for a connect snapshot counted as delivered before its send completed, so a follow-up card for a tab that dropped first was reported as shown), and rebased onto main 27cb69f8f6. The rebase resolved the overlap with main's relay quiet reconnect in connection.ts and useWebSocket.ts; two tests in main's useWebSocket.relayQuietReconnect.test.ts now let the health probe's reconnect take its at most 1 s, and a new one, the health probe brings a stopped session back after an earlier early drop, fails when the backoff is settled after the token-renewal stop instead of before it. On a DevSpace, at dd69e8fb11, each command exit 0 unless noted:

  • pytest -n0 test/test_ws_snapshot_hold.py: 10 passed. With the two production files of the fix (websocket_hub.py, ws_snapshot_hold.py) at their previous text: exit 1, 8 failed, among them test_a_tab_that_drops_before_its_snapshot_is_not_a_delivered_followup (the reply says "Follow-up card shown below the composer.").
  • Four targeted mutations of _await_owner_send, each exit 1 in the test that names it: the stall window measured from the call, the held wait unbounded, the cap measured from the release, and a held send counted at the release.
  • pytest -n 6 --dist loadgroup --max-worker-restart=2 on the 34 backend test files that drive owner sends, the snapshot hold, the shared pass or the heartbeat.
  • test_flake_pattern_ratchet.py, check_black_formatting.py, black on the changed files, isort, flake8, mypy, check_subprocess_encoding.py, check_comment_history.py, check_brand_name.py, check_harness_parity.py, check_sync_io_in_async.py, check_file_line_cap.py, check_loop_bound_locks.py, check_testpaths_coverage.py, docs-lint.sh and the repository's semgrep rules on the diff.
  • tsc -p tsconfig.app.json, eslint on the changed website files, and vitest on the reconnect, health probe and websocket hook tests.
  • The repro script below, at dd69e8fb11: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 3 passes, 0.25 per tab).

Rebased onto main after review; git range-diff shows the patch unchanged; related tests re-run on a DevSpace: at 7f0d5a5013 on main 8ec0a78355, the 13 changed backend test files 320 passed, test_ws_owner_binary_delivery.py, test_members_owner_gate.py and the three test_slots_diff*.py files 48 passed, and the 11 changed website test files 246 passed. Wherever this body says "this head (62a43216de)", it means the reviewed commit; 7f0d5a5013 is that same patch on a newer main.

Fixed after CI on 7f0d5a5013, which failed 13 tests on main's callers of this code: 12 that pass a SimpleNamespace state to GET /api/chat/slots (test_member_chat_folders_gate.py, test_crew_mode_workers.py, test_chat_slots_crew_view.py; TypeError: cannot create weak reference), and the source pin test_members_eventlog_wiring_live_writers.py::test_the_connect_baseline_call_site_exists_in_the_ws_handler (ValueError: substring not found). On a DevSpace, at 98f35d8fa5 on main 73e3275d98, each command exit 0 unless noted:

  • pytest -n0 on the three caller files: 12 failed before the fix, all pass after; the pin: 1 failed before, passes after.
  • pytest -n0 on the new test_a_state_the_table_cannot_key_serializes_for_every_caller (2 cases).
  • pytest -n 4 --dist loadgroup --max-worker-restart=2 on the 14 changed backend test files (the pin file among them), the three caller files, test_ws_owner_binary_delivery.py, test_members_owner_gate.py and test_slots_diff*.py.
  • test_flake_pattern_ratchet.py, check_black_formatting.py, check_subprocess_encoding.py, isort, flake8, mypy --platform linux, check_comment_history.py, check_brand_name.py, check_file_line_cap.py, docs-lint.sh, check_harness_parity.py and the repository's semgrep rules on the diff.
  • vitest on the 11 changed website test files (246 passed) and tsc -p tsconfig.app.json.
  • Reverted in a throwaway worktree, at this patch on main 1ece378503, each red again (exit 1): slot_list_pass.py without the fallback fails the 12 caller tests and the new test; the pin at its old text fails with ValueError; ws.py with await snapshot_send moved after the baseline fails the pin by assertion.
  • The repro script below: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 2 passes, 0.17 per tab), here and at this patch on main 1ece378503.
  • The new test and the pin: 20 of 20 runs green at -n0, 10 of 10 at -n 4 beside test_slot_list_pass.py, and 5 shuffled orders of their 25 tests, at this patch on main 1ece378503.

Proof: not provable -- other: the tests import three new modules (slot_list_pass, ws_heartbeat, ws_snapshot_hold), so reverting every production hunk only errors at import (prove.py: INCONCLUSIVE at ab747b294a4a); the targeted reverts below fail by assertion

Targeted reverts instead, each in a throwaway worktree, each red by assertion in the test named (the first three bullets at 03c2b65167, which differs from this head only in the stalled-client test's wait):

  • ws.py and slot_list_pass.py registering the socket only when its pass takes the list: test_a_waiting_connect_is_registered_and_gets_its_snapshot_first.
  • No hold at registration: the same test. No release after a failed snapshot: test_a_connect_whose_snapshot_failed_still_gets_the_frames_held_for_it.
  • The held frames released only once the snapshot drained: test_frames_held_for_a_connect_go_out_before_its_snapshot_drains. The release dropping the held frames, or owner sends not held: test_hub_frames_wait_for_the_snapshot_and_then_go_straight_out.
  • ws.py and slot_detail.py back to serializing per request, slot_list_pass.py kept: test_eight_concurrent_gets_take_two_passes and test_eight_concurrent_connects_take_two_passes fail (at be55d8ac5a, an earlier build of this patch).
  • LagTolerantWebSocketResponse._pong_not_received deferring to aiohttp: test_a_pong_sent_in_time_keeps_the_socket_through_the_stall fails; the other 3 pass (at be55d8ac5a).
  • connection.ts, useWebSocket.ts, reconnectCatchUp.ts and useDashboardHealthProbe.ts at the parent: 21 of the 246 frontend tests fail, among them the jitter, backoff and snapshot window tests above (at be55d8ac5a; the frontend is unchanged since).

Determinism, on the same host while other jobs loaded it:

  • test_slot_list_pass.py, test_ws_snapshot_hold.py, test_ws_heartbeat_loop_lag.py and test_dashboard_ws_stalled_client.py: 20 of 20 runs green at -n0, and 10 of 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_dashboard_state_ws_unregister.py and test_slot_list_rebuild.py, at this head.
  • All 320 changed backend tests in a shuffled order (seed 31095): green, at this head.
  • The three new or rewritten frontend reconnect files: 10 of 10 runs green; all 11 changed frontend files with --sequence.shuffle (seed 22102): 246 of 246 green (at be55d8ac5a; the frontend is unchanged since).

Fixed after review on 164aff4d2c (the connect snapshot labelled rows from a trailing shared pass with the live slotsPartial, slotsHeldKeys and slotsHoldChannels, so a restore that finished during the wait made incomplete rows read as complete): the pass now carries the flags read in the same synchronous section that took its rows (SlotPass, shared_slot_pass), and the connect sends those. New test/test_ws_snapshot_hold_flags_with_shared_pass.py (2 tests); four SimpleNamespace state stubs in existing tests gained the two hold methods. Reproduce at 071c3c636b: PYTHONPATH=src python -m pytest -n0 test/test_ws_snapshot_hold_flags_with_shared_pass.py: exit 0, 2 passed; with the slot_list_pass.py and ws.py hunks reverted: exit 1, 2 failed ("incomplete rows were labelled complete"). The earlier DevSpace run of the six affected test files: 122 passed. GitHub CI runs the rest.

Fixed after CI and review on 071c3c636b:

  • CI (Backend Tests shard 2 on Linux and Windows, both New-test stability lanes) failed 3 tests at slot_list_pass._serialize: test_slot_list_rebuild.py::TestConnectSnapshot::test_app_token_snapshot_omits_foreign_hold_state ("Expected 'slot_list_held_keys' to not have been called") and test_member_chat_folders_gate.py::TestSessionListMemberFilter::test_person_sees_every_session and ::test_member_slots_omit_foreign_sessions (AttributeError on a SimpleNamespace state). The pass now reads the held keys and the channel hold only for the dashboard audience, as main's ws.py did.
  • GPT 6.1: frames held for a socket that waits for its connect snapshot were invisible to the slow-client ceiling, so a tab that reads nothing could hold more than WS_HARD_MAX_BUFFERED_BYTES, and the release then wrote all of it with no stall check. What a socket holds now counts against that ceiling and WS_HELD_MAX_FRAMES (one frame per KiB of it), a frame past either drops the socket, and the release runs the hub's stall check before it writes each held frame. New test/test_ws_snapshot_hold_ceiling.py (8 cases).

Reproduce on a DevSpace, at this patch's tree (21c39fefdd, head a0cb15fb94), with PYTHONPATH=src python -m pytest -n0:

  • The 3 CI tests: exit 0, 3 passed. With slot_list_pass.py at 071c3c636b: exit 1, 3 failed with the CI errors.
  • test/test_ws_snapshot_hold_ceiling.py: exit 0, 8 passed. With ws_snapshot_hold.py and websocket_hub.py at 071c3c636b, plus only the WS_HELD_MAX_FRAMES constant the test imports: exit 1, 7 failed, all but test_superseded_slot_frames_leave_the_held_total, which guards against still counting a discarded list. Only the admission check removed: exit 1, 4 failed. Only the release check removed: exit 1, 3 failed.
  • The repro script below, at this patch: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 3 passes, 0.25 per tab).
  • GitHub CI runs the rest.

Rebased onto main after review; patch unchanged (git range-diff shows the one commit with only a context line moved in test/test_slots_diff_ws.py). Main e1b079ce1e carries #35625, the fix for the two tree-scan tests #35088 broke on main. On a DevSpace, at a0cb15fb94 merged into main e8283ceb85 (the tree CI tested), test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites and test_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers exit 1 with the CI errors; at f5b3580235 (this patch on main e1b079ce1e) both pass, and with #35625's two files reverted on f5b3580235 both exit 1 again with the same errors. Related tests re-run there at f5b3580235, each exit 0:

  • pytest -n 4 --dist loadgroup --max-worker-restart=2 on the 20 changed backend test files, the 15 test files main changed since the old base that send GET /api/chat/slots, open /api/ws or stub the dashboard socket, test_ws_owner_binary_delivery.py, test_members_owner_gate.py, test_member_chat_folders_gate.py, test_slots_diff.py, test_slots_diff_budget.py, both tree-scan files and test_flake_pattern_ratchet.py: 1569 passed.
  • vitest on the 14 changed website test files: 287 passed.
  • The repro script below: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 2 passes, 0.17 per tab).

Rebased onto main after review; patch unchanged (git range-diff shows the one commit with only the import context in src/kiro_crew/dashboard/ws.py changed: main's #35732 skill_read_ahead import now sits beside this patch's slot_list_pass import). Main 1bccaa12f0 carries #35725, the fix for the Coverage Gate per-file floor #35088 broke. On a DevSpace, at 57124fee7a (this patch on main 1bccaa12f0) unless noted, each exit 0:

  • pytest -n0 on test_dashboard_skill_read_ahead.py (main's source pin over _serve_ws), test_members_eventlog_wiring_live_writers.py, test_slot_list_pass.py, test_ws_snapshot_hold.py, test_ws_snapshot_hold_ceiling.py and test_ws_snapshot_hold_flags_with_shared_pass.py: 75 passed.
  • Coverage of the two files the gate named, from test_gateway_restart_readers.py, test_update_boot_apply.py and test_managed_launcher_restart.py alone: gateway_restart.py 53 of 53 statements, update_boot_apply.py 150 of 151. With test(updates): cover gateway_restart and update_boot_apply to the 80% floor #35725's two test files reverted on 8a3cd62cf7 (this patch on main b98b7cfaa3), the same run gives 24 of 53 and 113 of 151, below the 80% floor and the 113 the CI log reported at f5b3580235.
  • The repro script below, at 8a3cd62cf7: exit 0, bug absent in all five scenarios (12 reconnecting tabs took 3 passes, 0.25 per tab).

Manual verification

A repro script ran a real gateway, its real dashboard bundle in headless Chromium and scripted browser-like clients on a Linux development host: at this head (62a43216de) and at its parent (aa08664c56, main when that side ran), about 4 minutes per side.

Scenario main (aa08664c56) this head (62a43216de)
12 tabs reconnecting at once to 600 sessions 24 slot list serializations (2.00 per tab) 2 (0.17 per tab)
a socket that answered its PING during a 17 s stall closed with 1006, 0.01 s after the loop resumed answered after 0.09 ms and kept open
6 tabs reconnecting after a 25 s stall 2.00 serializations per tab 0.33 per tab
reconnect delays across tabs every tab exactly 1000, 2000, 4000, 8000, 10000 ms spread 337 to 3599 ms per round (median 1316 ms)
a gateway that accepts and then drops sockets, 30 s 17 or 18 attempts per tab; 108 upgrades, 216 serializations, about 10 s of loop time 3 or 4 attempts per tab; 21 upgrades, 22 serializations, 1.1 s of loop time

The script exited 1 (bug present) on main and 0 (bug absent) at this head.

Screenshots / video

Why no screenshot: the frontend change is reconnect timing and which request carries the slot list, so no rendered surface changes.

Related Issues

Fixes #33758

Siblings of the same defects on other sockets: #33769. Related closed work: #22372.

Sibling sweep

Swept: git grep -nE "WebSocketResponse(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/terminal.py:665 -- tracked in #33769
Sibling: src/kiro_crew/dashboard/stt_stream.py:1642 -- tracked in #33769
Sibling: src/kiro_crew/dashboard/handlers/sso_login.py:36 -- not affected -- sends one error frame and closes, so it never waits on a PONG
Sibling: src/kiro_crew/browser_harness/relay.py:468 -- not affected -- no heartbeat is armed on this socket

The other hits in those two files, the terminal refusal sockets (terminal.py:585, terminal.py:638) and the disabled-streaming reply (stt_stream.py:1630), close right after one error frame and never wait on a PONG.
Swept: git grep -nE "serialize_slots(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/crew_workers.py:72 -- not affected -- the Crew page's own read, which a reconnect does not send
Sibling: src/kiro_crew/dashboard/state_parts/slots_push.py:325 -- not affected -- the slots broadcast already coalesces its passes on a leading and a trailing edge
Swept: git grep -nE "/api/ws?caps" -- website/src
Sibling: website/src/hooks/websocket/connection.ts -- changed here
Sibling: website/src/apps/mochi/panel/panelBridge.ts:598 -- tracked in #33769
Sibling: website/src/apps/crew-companion/sessionWatch.ts:451 -- tracked in #33769

Pattern harvest

Rule candidate: review-prompt
Pattern: a reconnect path that re-reads, over a second request, the full state its new socket's first frame already carries, or retries on a fixed schedule with no jitter

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Fixed after review on 515d56b39c (GPT 6.1: an app socket's connect snapshot used scopes read before the shared pass wait, so an app disabled during the wait still got another session's metadata and the yolo posture; the snapshot now re-judges the scopes with effective_allowed_events after the wait), and the flake-pattern ratchet on test_unregistering_a_socket_ends_the_owner_sends_held_for_it (the 5 s bound is the assertion, now marked flake-ok). On a DevSpace, at 92072a62a6, each command exit 0 unless noted:

  • New test/test_ws_app_scope_after_shared_pass.py: 2 passed. With ws.py at its previous text the revoked-app case fails with "a disabled app was sent another session's metadata"; the control (an app that stays enabled keeps its scoped snapshot and yolo) passes.
  • pytest -n0 on that file, test_ws_snapshot_hold.py, test_flake_pattern_ratchet.py, test_slot_list_pass.py, test_ws_heartbeat_loop_lag.py, test_ws_event_scoping_live.py, test_chat_slots_crew_view.py, test_dashboard_ws_stalled_client.py and test_members_eventlog_wiring_live_writers.py: 195 passed.
  • black, flake8, isort on the changed files and mypy --platform linux src/kiro_crew: clean.
  • The repro script below, at 92072a62a6: exit 0, bug absent in all five scenarios.

Fixed after CI on 92072a62a6, where test_slot_patch_frames.py::test_pin_reaches_a_capable_tab_as_a_patch_and_an_old_tab_as_a_list timed out: the test patches only the connect-time scope loader, so the post-pass re-read hit the real reader, found the app not installed and the hub stopped sending it the pin's list. The test now patches the real reader too. On a DevSpace, at 9dffe6fed6: that file and test_flake_pattern_ratchet.py 98 passed (exit 0); the eight test files that patch load_declared_events_for_connect 210 passed with the one failure fixed here; black and flake8 clean.

Rebased onto main 35dc109f0d after review; patch unchanged (git range-diff shows =). The review lanes failed closed on the stale base ("could not read the merge-base"). Related tests re-run on a DevSpace at e25d406390: the changed backend test files plus the slot patch, slots diff, owner binary delivery, owner gate and chunk scope files, 274 passed.

Fixed after CI on e25d406390, where New-test stability timed out test_slot_patch_frames.py::test_pin_reaches_a_capable_tab_as_a_patch_and_an_old_tab_as_a_list again: the test now starts from an empty declaration cache and refresh set, so a reading another test cached for the app cannot reach the hub. I could not reproduce the CI timeout on a DevSpace (the 15 stability files, 3 passes, -n auto --timeout=120: 368 passed each pass, before and after this change), so this is a hermetic-isolation fix of the one state the test shared.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 9, 2026 10:27
@bolichen97
bolichen97 requested a review from Zedmor November 9, 2026 10:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 57124fee7aa256f333c2e719b38ed1e86ab3ba08 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff adds or changes no user-visible control, string or component. The website changes only affect reconnect timing and how slot data is fetched, so no screenshot is needed.

[UX-REVIEWED] 57124fe

Description read

[DESCRIPTION-READ] 3e7e5a1dd95b011e7c6132e70913b7e4b6f0e6e27692429983db9db4f999b1ec

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 57124fee7aa256f333c2e719b38ed1e86ab3ba08 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix cuts the stall feedback loop on both sides (shared passes on the gateway, jitter and healthy-open reset in the client), and nothing renders new.

Notes

  • The heartbeat and backoff rows in the table describe ws_heartbeat.py and utils/reconnectBackoff.ts as if this PR adds them. Both are already on base; the diff only wires them in, so say that in the description.
  • ws_snapshot_hold keeps the snapshot as the first frame by holding, dropping and replaying frames, and that is now the most fragile part of the connect path. Sending a sequenced snapshot and letting the client drop older slot frames would remove the hold, as a later follow-up.

[DESIGN-REVIEWED] 57124fe

Description read

[DESCRIPTION-READ] f76f0d8c1aad7f903660927d1ce0107f1e9820fd1b11ee5a0498e662f2e54513

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 57124fee7aa256f333c2e719b38ed1e86ab3ba08 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item addresses the storm in #33758. The one doubt is cost: sharing the connect pass is what forces the frame-hold machinery, and its benefit was never measured separately.

Notes

  • Item 6 — oversized: connects only wait for a shared pass because of await shared_slot_pass( in ws.py, and that wait is what needs the frame hold (items 7–8). The cost is ws_snapshot_hold.py (274 lines), dropping superseded slot frames, a sequence reset and a forced push_slots_update(). The only measurement, "108 /api/ws upgrades in 30 seconds", was taken on main. That is before items 1–3 (no catch-up GET, jittered delays, backoff reset only after 10 s healthy) cut the passes to about one per tab and spread the tabs out. Subtraction: defer slot_list_pass.py and ws_snapshot_hold.py, and the websocket_hub.py hold and owner-send paths, until a measurement taken with items 1–5 shows the remaining per-connect passes still extend a stall.

What this change ships

Inventory (8 items) — 7 justified

Intent: stop one gateway loop stall from turning into a dashboard reconnect storm that makes the stall longer — FIX.

  1. A reconnecting tab takes its slot list from the connect snapshot; it reads over HTTP only when the snapshot is empty or missing after 10 s — justified
  2. Each reconnect delay loses a random part of up to half — justified
  3. The backoff starts over only after a socket stayed open 10 s — justified
  4. The health probe no longer skips the backoff after a socket that dropped young; after a refused handshake it reconnects within 1 s — justified
  5. A dashboard socket whose PONG was queued behind a stall is no longer closed with 1006 — justified
  6. Connects and slot GETs of one audience share a trailing pass, waiting up to 2 s — oversized: the gain after items 1–3 is unmeasured
  7. Unread marks and chimes stay held until the snapshot is applied, not until the GET reply — justified
  8. Frames to a connecting socket are held behind its snapshot, count against the buffer ceiling, and stale slot frames are dropped — justified

[FIRST-PRINCIPLES-REVIEWED] 57124fe

Description read

[DESCRIPTION-READ] 26576d884f4a3bb30ff3b421294f8d1209d95ed2964f7fc0150badb99fa6fb53

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 57124fee7aa256f333c2e719b38ed1e86ab3ba08 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 57124fe

Verdict parsed from the review's SHA-scoped output markers for commit 57124fee7aa256f333c2e719b38ed1e86ab3ba08.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 57124fee7aa256f333c2e719b38ed1e86ab3ba08 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 57124fee7aa256f333c2e719b38ed1e86ab3ba08 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 57124fe

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 57124fee7aa256f333c2e719b38ed1e86ab3ba08 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 9, 2026 10:47
buluoray
buluoray previously approved these changes Nov 9, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Reconnect storms now share serialize_slots passes per audience, keyed by (loop, check_status, user), so one person never sees another person's slots (dashboard/slot_list_pass.py:92-111, chat_api/slot_detail.py:80-82). The PONG timer tolerates the gateway's own loop stall, and an unresponsive socket is still closed (dashboard/ws_heartbeat.py:31-44). Frames for a socket that is still connecting wait behind its snapshot (dashboard/ws_snapshot_hold.py:42-69, ws.py:828-834, 985-989). Client reconnects are jittered, with a backoff that resets only after a 10 s healthy open. No new listener, and no auth or origin change. Targeted tests pass, and three mutations turn named tests red.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@JLargent42

Copy link
Copy Markdown
Contributor

Review notes from the #33769 side (I'm picking up #33769, the sibling sockets this PR lists under "Not a goal"). Plain comment, not a review.

Scope vs #33769. At head 7f0d5a5013, as the body says, the #33769 sites are unchanged: src/kiro_crew/dashboard/handlers/terminal.py:665 and src/kiro_crew/dashboard/stt_stream.py:1642 still build a stock web.WebSocketResponse(heartbeat=30), and website/src/apps/mochi/panel/panelBridge.ts:598,764 and website/src/apps/crew-companion/sessionWatch.ts:451,476-478 still reset on open and double with no jitter. So this PR fixes #33758 only, and #33769 needs its own change. That change wants to reuse LagTolerantWebSocketResponse and the jitter / healthy-open rule added here, so it is easiest after this lands.

Repro of the #33769 socket half. Locally (Python 3.12.15, aiohttp 3.14.5), this PR's test/test_ws_heartbeat_loop_lag.py passes 4/4, including test_the_stock_socket_drops_that_peer: the stock class that terminal and STT use on main closes a peer that answered its PING with 1006 after a loop stall. So the subclass works and the terminal and voice sockets have the bug on main.

Why the backend shards are red at this head. The failures come from this diff, not from main. I reproduced all of them locally on 7f0d5a5013, and the same 4 files pass 158/158 on main ddb535e187:

  1. TypeError: cannot create weak reference to 'types.SimpleNamespace' object in 12 tests: test/test_chat_slots_crew_view.py (7), test/test_crew_mode_workers.py (3), test/test_member_chat_folders_gate.py (2). Cause: src/kiro_crew/dashboard/slot_list_pass.py:85 keys _PASSES on a weakref.WeakKeyDictionary. api_chat_slots (chat_api/slot_detail.py:80) now goes through shared_slot_list, and those tests pass a SimpleNamespace state, which can't be weak-referenced. One possible fix that keeps production behaviour: keep the passes on the state object itself rather than in a module-level weak map, e.g.

    passes_by_audience = state.__dict__.setdefault("_slot_list_passes", {})
    passes = passes_by_audience.setdefault(audience, _AudiencePasses())

    with the same lookup in _take_trailing. That lifetime is the state's own, with no weakref. (Or, if the weak map is wanted, make the test stubs weak-referenceable. That touches 3 test files instead of 1 source file.)

  2. ValueError: substring not found at test/test_members_eventlog_wiring_live_writers.py:255: the test anchors on the literal await ws.send_str(snapshot_payload), which ws.py:985 now spells snapshot_send = asyncio.ensure_future(ws.send_str(snapshot_payload)). Point the anchor at ws.send_str(snapshot_payload) (the ordering assert against await state.send_members_subscribed(ws) at ws.py:1008 still holds).

  3. Backend Tests (3.12, 3): test_review_repair_routing_skill.py::...[kirocrew-worktree-dev-16000]. This one is inherited: the skill is 16004 bytes on this PR's base 8ec0a78355 and 15974 on main now, so a rebase clears it. Coverage Gate looks like a knock-on of the red shards.

The Windows shards (2, 3, 4, 7, 8) line up with the same Linux shard numbers, so they are most likely the same tests.

Gaps (non-blocking). Not part of this PR's scope, noted for #33769: the sibling sweep in the body is accurate and I found no other heartbeat= socket on the reconnect path.

@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from 7f0d5a5 to 98f35d8 Compare November 10, 2026 00:56
KiroCrewCaptain pushed a commit that referenced this pull request Nov 10, 2026
…34167)

* fix(dashboard): keep terminal and voice sockets through a loop stall

The terminal and streaming dictation sockets used aiohttp's stock
heartbeat, which closes a socket with 1006 when a loop stall makes its
PONG timer run before the PONG already queued is read. Both now use
LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws.

The Mochi panel and Crew Companion sockets retried on a fixed doubling
delay reset on every open, so every copy retried together after a stall
and a gateway that drops new sockets was retried at the shortest delay.
They now take a random part of up to half off each delay and reset the
backoff only after a socket stayed open 10 s.

Fixes #33769

* fix(dashboard): start app backoff over on a healthy watchdog replace

A silence-watchdog replacement cleared the opened-at stamp without the
healthy check the close path runs, so a long healthy socket replaced
after a network change kept the grown delay. Run the same check there.
Also reap the stall test's shell in a finally, and open each fake-timer
try right after useFakeTimers.
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 10, 2026
@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from 98f35d8 to dd69e8f Compare November 10, 2026 11:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 10, 2026
bolichen97 pushed a commit that referenced this pull request Nov 10, 2026
…34167)

* fix(dashboard): keep terminal and voice sockets through a loop stall

The terminal and streaming dictation sockets used aiohttp's stock
heartbeat, which closes a socket with 1006 when a loop stall makes its
PONG timer run before the PONG already queued is read. Both now use
LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws.

The Mochi panel and Crew Companion sockets retried on a fixed doubling
delay reset on every open, so every copy retried together after a stall
and a gateway that drops new sockets was retried at the shortest delay.
They now take a random part of up to half off each delay and reset the
backoff only after a socket stayed open 10 s.

Fixes #33769

* fix(dashboard): start app backoff over on a healthy watchdog replace

A silence-watchdog replacement cleared the opened-at stamp without the
healthy check the close path runs, so a long healthy socket replaced
after a network change kept the grown delay. Run the same check there.
Also reap the stall test's shell in a finally, and open each fake-timer
try right after useFakeTimers.

Backport note: backend half only. The Mochi panel and Crew Companion
reconnect changes (website/src/apps/**, website/src/utils/reconnectBackoff.ts
and their tests) are app fixes and stay off this hotfix; the spec line
keeps only the terminal and dictation socket sentence.

(cherry picked from commit 1139925)
@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from dd69e8f to 7867322 Compare November 10, 2026 18:06
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 10, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from c3f06fc to 164aff4 Compare November 11, 2026 05:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from 164aff4 to 071c3c6 Compare November 11, 2026 11:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from 071c3c6 to a0cb15f Compare November 11, 2026 22:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: frames held off the transport for a connect snapshot were invisible to the hub's buffer ceiling, and the release wrote them all with no stall check
resolution: narrowed -- the hold stays and gains the invariant it lacked: held bytes and frames count against WS_HARD_MAX_BUFFERED_BYTES and WS_HELD_MAX_FRAMES at admission, and each released frame passes the hub's stall check before it is written

  • Held frames bypass the slow-client memory ceiling span=3fe50c1e34c0 (src/kiro_crew/dashboard/ws_snapshot_hold.py:54)

fixed in a0cb15f: held and held_send refuse a frame that would take what the socket holds past the hub's Ceiling (bytes or frame count) and drop the socket the hub's way (one warning, _remove_ws, transport.abort()); an awaited owner send of that frame fails at once with ConnectionResetError and counts no delivery. release_held runs the hub's _drop_if_stalled right before it writes each held frame, so a release that passes the ceiling drops the socket and the held frames not yet written are discarded, their owner sends not delivered. New test/test_ws_snapshot_hold_ceiling.py (8 cases) passes at this head; 7 of them fail with ws_snapshot_hold.py and websocket_hub.py at 071c3c636b (the 40 x 512 KiB case reports 20973710 bytes held for one socket, ceiling 16777216).

buluoray
buluoray previously approved these changes Nov 11, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on a0cb15f. This PR stops a gateway stall from turning into a dashboard reconnect storm. Tabs that reconnect together share one slot-list pass, frames for a connecting tab wait behind its snapshot, and client retries are jittered. Both points of the earlier change request are fixed. The connect snapshot now takes its partial, held-keys and channel-hold labels from the same pass that took its rows (SlotPass, slot_list_pass.py:160-179, read at ws.py:989-995). A restore that finishes during the wait can no longer label an incomplete list complete, so unread dots are not drained. Reverting that makes test_a_rebuild_finished_after_the_pass_took_its_rows_keeps_the_snapshot_partial fail. approvalReconcile.test.tsx:156 now advances 10000 ms; putting back 1000 makes that file fail (1 failed, 21 passed). GPT's earlier hold-memory finding is also fixed: held frames count against WS_HARD_MAX_BUFFERED_BYTES and WS_HELD_MAX_FRAMES when they are held, and the release runs the stall check before each write. Removing either check reddens test_ws_snapshot_hold_ceiling.py (4 tests and 3 tests). Follow-up, not blocking: the new drop_superseded_slot_frames step (ws.py:1014-1016) has no test. Making it a no-op, or removing the fresh slots push after it, leaves every related test green. I read the code and it is correct: every slot frame builder writes type first, and the push gives the tab a full current list. A test that holds an older removal patch across a connect would guard it. Decisive lines: src/kiro_crew/dashboard/slot_list_pass.py:160-179 (_serialize takes rows and hold flags in one synchronous step), src/kiro_crew/dashboard/ws.py:918-995 (snapshot labels come from slot_pass, not live state), src/kiro_crew/dashboard/ws_snapshot_hold.py:126-144 (_admitted ceiling), src/kiro_crew/dashboard/ws_snapshot_hold.py:243-247 (_Release stall check), website/src/test/UseWebSocketCoverage.approvalReconcile.test.tsx:156 (10000), src/kiro_crew/dashboard/ws.py:1014-1016 (drop_superseded_slot_frames, untested: follow-up)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/sweep-g5-dashboard-reconnect-storm branch from a0cb15f to f5b3580 Compare November 12, 2026 03:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Coverage Gate is red at f5b3580235 for main's reason, not this patch's: the per-file floor reports src/kiro_crew/gateway_restart.py (58.5%, 31/53) and src/kiro_crew/update_boot_apply.py (74.8%, 113/151), the same two files and the same numbers as main's own CI run at e1b079ce1e (run 43140329527, job Coverage Gate). This PR changes neither file. #35725 adds the tests that bring both to the floor; this PR waits for it and rebases once it lands. Every backend and frontend test shard is green at this head.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 12, 2026
Dashboard reconnect storm: a stalled gateway loop dropped every
dashboard socket at once and every tab reconnected together, each
running two full slot list serializations on the event loop, which
prolonged the stall.

- Connect snapshots and GET /api/chat/slots now share serialize_slots
  passes per audience (slot_list_pass), so a reconnect burst costs one
  or two passes instead of one per tab. A state the pass table cannot
  key (no weak reference to it, or no hash) serializes for each caller
  at once, unshared.
- A connecting socket still registers right after its handshake, so
  the catch-up reads its tab sends on open are answered with the
  socket registered; the hub holds that socket's frames until its
  connect snapshot is out (ws_snapshot_hold), so the snapshot stays
  its first frame. An awaited owner send to such a socket counts only
  once the send of its held frame completes.
- What a socket holds while it waits for its snapshot counts against
  the hub's per-tab buffer ceiling (WS_HARD_MAX_BUFFERED_BYTES, and
  WS_HELD_MAX_FRAMES frames), and the release runs the stall check
  before it writes each held frame, so a tab that never reads is still
  dropped at that ceiling.
- The dashboard socket restarts its PONG window instead of closing when
  the heartbeat timeout ran late or data is already queued, so a loop
  stall no longer drops healthy sockets (ws_heartbeat).
- The client takes a random part of up to half off each reconnect
  delay, resets its backoff only after a socket stayed open 10 s, and
  the health probe no longer skips the backoff after an early drop.
- A reconnect reads its slot list from the connect snapshot instead of
  a second GET; an empty snapshot, or none within 10 s, still reads it
  over HTTP.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 57124fe. This head is a rebase of the head we approved (f5b3580) onto current main, with no change to the patch: the 48 files and every added and removed line are the same on both heads. The push dismissed our approval, so this renews it. The fixes still hold as reviewed: the connect snapshot takes its partial, held-keys and channel-hold labels from the pass that took its rows, so a restore that finishes during the wait cannot drain unread dots; held frames for a connecting tab stay under the byte and frame ceiling; the release checks for a stall before each write; and client reconnects are jittered. All checks on this head pass. The follow-up from our earlier review is still open and does not block: the drop_superseded_slot_frames step in ws.py has no test of its call site. Decisive lines: src/kiro_crew/dashboard/ws.py:987-996 (snapshot labels from slot_pass), src/kiro_crew/dashboard/ws_snapshot_hold.py:126-144 (_admitted ceiling), src/kiro_crew/dashboard/ws_snapshot_hold.py:243-247 (_Release stall check), website/src/hooks/websocket/connection.ts:195,219 (jittered delays), src/kiro_crew/dashboard/ws.py:1015-1017 (drop_superseded_slot_frames, still untested: follow-up)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard: a gateway loop stall drops every tab's socket and the tabs' reconnects make the stall longer

4 participants