Repository navigation
fix(dashboard): keep terminal and voice sockets through a loop stall - #34167
Conversation
The terminal and streaming dictation sockets used aiohttp's stock heartbeat, which closes a socket with 1006 when a loop stall makes its PONG timer run before the PONG already queued is read. Both now use LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws. The Mochi panel and Crew Companion sockets retried on a fixed doubling delay reset on every open, so every copy retried together after a stall and a gateway that drops new sockets was retried at the shortest delay. They now take a random part of up to half off each delay and reset the backoff only after a socket stayed open 10 s. Fixes #33769
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The diff changes no visible control, string or state. It only changes the timing of socket keep-alive and reconnects, so no screenshot is needed. [UX-REVIEWED] 587fcc7 Description read[DESCRIPTION-READ] 28bc5751d18afdad2165593daec99cfaeb45535b65615726043cad4516629ce3 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the fix targets the stall-ordering cause on both heartbeat sockets, and every change can be reverted. Notes
[DESIGN-REVIEWED] 587fcc7 Description read[DESCRIPTION-READ] f0b13b9189fc983a9abea83ff5998a0d18cc03b938eb940df509bc578c65581c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The one note is a helper module copied from #33825, which AGENTS.md says should have a single owner PR. Not checked: whether #33825 lands its own Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: when the gateway's event loop stalls, a terminal tab or dictation session whose client answered should stay connected, and the Mochi panel and Crew Companion should stop reconnecting in lockstep afterwards. This is a FIX (#33769).
[FIRST-PRINCIPLES-REVIEWED] 587fcc7 Description read[DESCRIPTION-READ] d4f9ebce6a8b938ea21bb25186a283a026ba707a670516e86e80cd4e82ef72e5 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)GPT 6.1 flagged blocking issues on This comment is updated in place on each push. Review detailsBLOCKING -- website/src/hooks/useChatNavigation.ts:98 -- Cached responses cannot restore evicted resource labels
Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1 — LOW. The failure needs more than 500 distinct bare-URL labels in one tab ( False positive or not applicable? A repository writer can comment: |
A silence-watchdog replacement cleared the opened-at stamp without the healthy check the close path runs, so a long healthy socket replaced after a network change kept the grown delay. Run the same check there. Also reap the stall test's shell in a finally, and open each fake-timer try right after useFakeTimers.
|
GPT 6.1 disposition (head 587fcc7): the downgraded finding at website/src/hooks/useChatNavigation.ts:98 is outside this diff. This PR does not touch that hook or the label cache, so no change here. Round 1 findings are fixed in 587fcc7: Opus (the watchdog replacement now runs the same healthy check as onclose, pinned by the two new silence-watchdog tests), GPT (the stall test reaps its shell in a finally), and Flake Pattern Ratchet J6. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Review on this head checked these points. LagTolerantWebSocketResponse keeps a socket only when the peer already answered, and it still closes a silent peer at most one window later. The terminal and STT constructor swaps keep their arguments, and the refusal sockets stay on the stock class. No test still patches the old terminal.web.WebSocketResponse seam. In both app sockets the backoff resets only after a socket proved healthy, on onclose and on the watchdog replacement path. A reviewer traced the targeted reverts in the body, and each one fails its test. The branch merges cleanly with main and with #33825.
Well done: the raw TCP control test shows stock aiohttp dropping a peer that answered in time. That pins the bug as well as the fix.
Optional:
- This PR and #33825 add the same
ws_heartbeat.pyand its test, and AGENTS.md asks for one owner PR for a shared helper. Please name the merge order and drop the copy once the other lands. On the same point, the header inreconnectBackoff.tssays the dashboard socket shares the rule, but nothing imports it there. - Every loop-lag test stalls past the tolerance. So removing either branch of
answered or lateinws_heartbeat.py:39keeps them green. A case where the stall ends inside the tolerance, plus an "still open after the stall" check, would pin both branches. - In
sessionWatch.socketSilence.test.tsandmochiSocketSilence.test.ts, theMath.randomspy is restored at the end of the test body. A try/finally like the sibling tests would stop it leaking after a failure.
…34167) * fix(dashboard): keep terminal and voice sockets through a loop stall The terminal and streaming dictation sockets used aiohttp's stock heartbeat, which closes a socket with 1006 when a loop stall makes its PONG timer run before the PONG already queued is read. Both now use LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws. The Mochi panel and Crew Companion sockets retried on a fixed doubling delay reset on every open, so every copy retried together after a stall and a gateway that drops new sockets was retried at the shortest delay. They now take a random part of up to half off each delay and reset the backoff only after a socket stayed open 10 s. Fixes #33769 * fix(dashboard): start app backoff over on a healthy watchdog replace A silence-watchdog replacement cleared the opened-at stamp without the healthy check the close path runs, so a long healthy socket replaced after a network change kept the grown delay. Run the same check there. Also reap the stall test's shell in a finally, and open each fake-timer try right after useFakeTimers. Backport note: backend half only. The Mochi panel and Crew Companion reconnect changes (website/src/apps/**, website/src/utils/reconnectBackoff.ts and their tests) are app fixes and stay off this hotfix; the spec line keeps only the terminal and dictation socket sentence. (cherry picked from commit 1139925)
Problem / Motivation
Goal: A gateway event loop stall no longer closes a terminal or dictation socket whose client answered in time, and the Mochi panel and Crew Companion no longer reconnect in lockstep afterwards.
The terminal socket (
/api/ws/terminal/<id>) and the streaming dictation socket (/api/ws/stt) use aiohttp's stockWebSocketResponse(heartbeat=30). When the loop stalls past the PONG deadline, the expired PONG timer runs before the PONG already waiting in the kernel is read, and the socket closes with 1006 although its client answered: the shell tab drops and dictation stops mid-sentence. The Mochi panel and Crew Companion open their own/api/wsand retry on a fixed 1, 2, 4 ... 30 s schedule that resets on every open, so every copy retries at the same moments, and a gateway that accepts and then drops sockets is retried at the shortest delay.Why it matters
Anyone with a terminal tab, voice dictation, the Mochi panel or the Crew Companion open while the gateway's loop stalls. A terminal or dictation session is lost for a socket that was healthy, and the app sockets add a synchronized reconnect burst to the loop that just stalled.
Not a goal
/api/wsand its client (hooks/websocket/connection.ts): Dashboard: a gateway loop stall drops every tab's socket and the tabs' reconnects make the stall longer #33758, in fix(dashboard): share slot list passes and stagger reconnects #33825.What changed (motivation → approach → change)
Symptom → cause → change, per socket:
handlers/terminal.py)web.WebSocketResponse(heartbeat=30, timeout=300)ws_heartbeat.LagTolerantWebSocketResponsewith the same argumentsstt_stream.py)web.WebSocketResponse(heartbeat=30, ...)LagTolerantWebSocketResponsewith the same argumentsws_heartbeat.pya1ff246558)apps/mochi/panel/panelBridge.ts) and Crew Companion (apps/crew-companion/sessionWatch.ts)utils/reconnectBackoff.tsjitteredandprovedHealthy, the one rule both app sockets use, same constants as #33825'sconnection.tsBoth app sockets run the same healthy check on every way a socket ends: its
onclose, and the silence watchdog's forced replacement (forceSilentReconnect/onDead), so a long healthy socket replaced after a network change retries at the short delay.LagTolerantWebSocketResponserestarts the PONG window when its timer ran more than a second late or finds data already read, and still closes a peer that sent nothing, so a dead terminal or mic socket is dropped at most one window later. Thews_heartbeat.pyandtest_ws_heartbeat_loop_lag.pyhere are identical copies of #33825's, so whichever PR lands second merges them as an identical add; I took the copy rather than stacking because #33825 is red on its own diff and this fix should not wait on it. The refusal sockets (terminal.pyrefusal paths, the disabled-streaming reply) close after one frame and keep the stock class.Terminal tests that stubbed
terminal.web.WebSocketResponsenow stubterminal.ws_heartbeat.LagTolerantWebSocketResponse(the constructor the shell path calls); the refusal paths still useweb.WebSocketResponse.Backwards compatibility
Compatible: heartbeat interval, timeouts, close codes and frames are unchanged; a socket is only kept open in the case where the peer had already answered, and the app reconnect delays stay within 1 to 30 s.
Removes nothing: no route, frame, export or option is removed; the two app sockets keep their own reconnect state and
retryConnect()still reconnects at once.Tests
test/test_ws_heartbeat_loop_lag.py(from fix(dashboard): share slot list passes and stagger reconnects #33825): over a raw TCP client, a PONG sent before a loop stall keeps the socket; the stock aiohttp socket drops that same peer with 1006 (the bug on main for terminal and STT); a silent peer is still dropped.test/test_terminal_handler_pty_integration.py::test_the_shell_socket_tolerates_a_loop_stall: the socket a real PTY shell binds to is the lag-tolerant one.test/test_stt_stream.py::TestAppleStreamingSession::test_the_session_socket_tolerates_a_loop_stall: the socket a dictation session streams over is the lag-tolerant one.CrewCompanionSessionWatch.test.tsandMochiPanelBridgeCoverage.test.tsx: each delay loses a random part (pinned bounds), the backoff keeps growing after an open-then-drop, and starts over after a socket stayed open 10 s.sessionWatch.socketSilence.test.tsandmochiSocketSilence.test.ts: a watchdog replacement of a socket that stayed healthy starts the backoff over (the next failed retry comes after 1000, not 2000).Proof: not provable -- other: the tests import the new ws_heartbeat module, so reverting every production hunk errors at import; the targeted reverts below fail by assertion
Targeted reverts:
terminal.pyandstt_stream.pyback toweb.WebSocketResponse: both socket-type tests fail (AssertionError: <class 'aiohttp.web_ws.WebSocketResponse'>).panelBridge.tsandsessionWatch.tsat main: both jitter/backoff tests fail (2 of 144 in those files).Runs on a Linux development host, Python 3.12.15, aiohttp 3.14.5, at this head:
-n 2, 10 terminal/STT/heartbeat files: 390 passed, 1 skipped;test_flake_pattern_ratchet.py: passed.check_comment_history,check_brand_name,check_file_line_cap, eslint on the touched files,npm run typecheck: pass.scripts/local-gate.pyrelated backend set: 25915 passed, 44 failed, all infile_explorer/tests/test_server.py,test_auto_improvement_members.py,test_browser_recording_skill.py,test_ci_file_shards.pyandtest_ci_pytest_progress.py, which fail the same way on mainddb535e187on this host (43 failed); none touches this diff.Manual verification
N/A — the stall is reproduced deterministically by
test_ws_heartbeat_loop_lag.py(a real aiohttp server and a raw socket, loop held by a busy wait), and the two socket-type tests pin that the terminal and dictation paths use that class; the app backoff is pinned with fake timers.Screenshots / video
Why no screenshot: the frontend change is reconnect timing in two app sockets, so no rendered surface changes.
Related Issues
Fixes #33769
Related: #33758 and its PR #33825, which adds
LagTolerantWebSocketResponsefor/api/ws.Sibling sweep
Swept: git grep -nE "WebSocketResponse(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/terminal.py -- changed here
Sibling: src/kiro_crew/dashboard/stt_stream.py -- changed here
Sibling: src/kiro_crew/dashboard/ws.py -- tracked in #33758
Sibling: src/kiro_crew/dashboard/handlers/sso_login.py -- not affected -- sends one error frame and closes, never waits on a PONG
Sibling: src/kiro_crew/browser_harness/relay.py -- not affected -- no heartbeat is armed on this socket
Swept: git grep -nE "/api/ws?caps" -- website/src
Sibling: website/src/apps/mochi/panel/panelBridge.ts -- changed here
Sibling: website/src/apps/crew-companion/sessionWatch.ts -- changed here
Sibling: website/src/hooks/websocket/connection.ts -- tracked in #33758
Pattern harvest
Rule candidate: review-prompt
Pattern: a socket with heartbeat= built from the stock web.WebSocketResponse, or a reconnect on a fixed doubling delay with no jitter that resets on open
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.