Skip to content

fix(dashboard): keep terminal and voice sockets through a loop stall - #34167

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
fix/33769-sibling-socket-stall
Nov 10, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
fix/33769-sibling-socket-stall

Conversation

@JLargent42

@JLargent42 JLargent42 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway event loop stall no longer closes a terminal or dictation socket whose client answered in time, and the Mochi panel and Crew Companion no longer reconnect in lockstep afterwards.

The terminal socket (/api/ws/terminal/<id>) and the streaming dictation socket (/api/ws/stt) use aiohttp's stock WebSocketResponse(heartbeat=30). When the loop stalls past the PONG deadline, the expired PONG timer runs before the PONG already waiting in the kernel is read, and the socket closes with 1006 although its client answered: the shell tab drops and dictation stops mid-sentence. The Mochi panel and Crew Companion open their own /api/ws and retry on a fixed 1, 2, 4 ... 30 s schedule that resets on every open, so every copy retries at the same moments, and a gateway that accepts and then drops sockets is retried at the shortest delay.

Why it matters

Anyone with a terminal tab, voice dictation, the Mochi panel or the Crew Companion open while the gateway's loop stalls. A terminal or dictation session is lost for a socket that was healthy, and the app sockets add a synchronized reconnect burst to the loop that just stalled.

Not a goal

What changed (motivation → approach → change)

Symptom → cause → change, per socket:

Part On main With this PR
Terminal socket (handlers/terminal.py) stock web.WebSocketResponse(heartbeat=30, timeout=300) ws_heartbeat.LagTolerantWebSocketResponse with the same arguments
Dictation socket (stt_stream.py) stock web.WebSocketResponse(heartbeat=30, ...) LagTolerantWebSocketResponse with the same arguments
ws_heartbeat.py absent the subclass from #33825, byte-identical (blob a1ff246558)
Mochi panel (apps/mochi/panel/panelBridge.ts) and Crew Companion (apps/crew-companion/sessionWatch.ts) fixed doubling delay, reset on every open a random part of up to half comes off each delay; the backoff resets only after a socket stayed open 10 s
utils/reconnectBackoff.ts absent jittered and provedHealthy, the one rule both app sockets use, same constants as #33825's connection.ts

Both app sockets run the same healthy check on every way a socket ends: its onclose, and the silence watchdog's forced replacement (forceSilentReconnect / onDead), so a long healthy socket replaced after a network change retries at the short delay.

LagTolerantWebSocketResponse restarts the PONG window when its timer ran more than a second late or finds data already read, and still closes a peer that sent nothing, so a dead terminal or mic socket is dropped at most one window later. The ws_heartbeat.py and test_ws_heartbeat_loop_lag.py here are identical copies of #33825's, so whichever PR lands second merges them as an identical add; I took the copy rather than stacking because #33825 is red on its own diff and this fix should not wait on it. The refusal sockets (terminal.py refusal paths, the disabled-streaming reply) close after one frame and keep the stock class.

Terminal tests that stubbed terminal.web.WebSocketResponse now stub terminal.ws_heartbeat.LagTolerantWebSocketResponse (the constructor the shell path calls); the refusal paths still use web.WebSocketResponse.

Backwards compatibility

Compatible: heartbeat interval, timeouts, close codes and frames are unchanged; a socket is only kept open in the case where the peer had already answered, and the app reconnect delays stay within 1 to 30 s.

Removes nothing: no route, frame, export or option is removed; the two app sockets keep their own reconnect state and retryConnect() still reconnects at once.

Tests

  • test/test_ws_heartbeat_loop_lag.py (from fix(dashboard): share slot list passes and stagger reconnects #33825): over a raw TCP client, a PONG sent before a loop stall keeps the socket; the stock aiohttp socket drops that same peer with 1006 (the bug on main for terminal and STT); a silent peer is still dropped.

  • test/test_terminal_handler_pty_integration.py::test_the_shell_socket_tolerates_a_loop_stall: the socket a real PTY shell binds to is the lag-tolerant one.

  • test/test_stt_stream.py::TestAppleStreamingSession::test_the_session_socket_tolerates_a_loop_stall: the socket a dictation session streams over is the lag-tolerant one.

  • CrewCompanionSessionWatch.test.ts and MochiPanelBridgeCoverage.test.tsx: each delay loses a random part (pinned bounds), the backoff keeps growing after an open-then-drop, and starts over after a socket stayed open 10 s.

  • sessionWatch.socketSilence.test.ts and mochiSocketSilence.test.ts: a watchdog replacement of a socket that stayed healthy starts the backoff over (the next failed retry comes after 1000, not 2000).

Proof: not provable -- other: the tests import the new ws_heartbeat module, so reverting every production hunk errors at import; the targeted reverts below fail by assertion

Targeted reverts:

  • terminal.py and stt_stream.py back to web.WebSocketResponse: both socket-type tests fail (AssertionError: <class 'aiohttp.web_ws.WebSocketResponse'>).
  • panelBridge.ts and sessionWatch.ts at main: both jitter/backoff tests fail (2 of 144 in those files).
  • The watchdog-path reset removed (first commit's code): both replacement tests fail (2 of 22 in the silence files).

Runs on a Linux development host, Python 3.12.15, aiohttp 3.14.5, at this head:

  • pytest -n 2, 10 terminal/STT/heartbeat files: 390 passed, 1 skipped; test_flake_pattern_ratchet.py: passed.
  • vitest, 5 Mochi/Companion files: 227 passed.
  • black and subprocess-encoding gates, isort, flake8, check_comment_history, check_brand_name, check_file_line_cap, eslint on the touched files, npm run typecheck: pass.
  • scripts/local-gate.py related backend set: 25915 passed, 44 failed, all in file_explorer/tests/test_server.py, test_auto_improvement_members.py, test_browser_recording_skill.py, test_ci_file_shards.py and test_ci_pytest_progress.py, which fail the same way on main ddb535e187 on this host (43 failed); none touches this diff.

Manual verification

N/A — the stall is reproduced deterministically by test_ws_heartbeat_loop_lag.py (a real aiohttp server and a raw socket, loop held by a busy wait), and the two socket-type tests pin that the terminal and dictation paths use that class; the app backoff is pinned with fake timers.

Screenshots / video

Why no screenshot: the frontend change is reconnect timing in two app sockets, so no rendered surface changes.

Related Issues

Fixes #33769

Related: #33758 and its PR #33825, which adds LagTolerantWebSocketResponse for /api/ws.

Sibling sweep

Swept: git grep -nE "WebSocketResponse(" -- src/
Sibling: src/kiro_crew/dashboard/handlers/terminal.py -- changed here
Sibling: src/kiro_crew/dashboard/stt_stream.py -- changed here
Sibling: src/kiro_crew/dashboard/ws.py -- tracked in #33758
Sibling: src/kiro_crew/dashboard/handlers/sso_login.py -- not affected -- sends one error frame and closes, never waits on a PONG
Sibling: src/kiro_crew/browser_harness/relay.py -- not affected -- no heartbeat is armed on this socket
Swept: git grep -nE "/api/ws?caps" -- website/src
Sibling: website/src/apps/mochi/panel/panelBridge.ts -- changed here
Sibling: website/src/apps/crew-companion/sessionWatch.ts -- changed here
Sibling: website/src/hooks/websocket/connection.ts -- tracked in #33758

Pattern harvest

Rule candidate: review-prompt
Pattern: a socket with heartbeat= built from the stock web.WebSocketResponse, or a reconnect on a fixed doubling delay with no jitter that resets on open

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

The terminal and streaming dictation sockets used aiohttp's stock
heartbeat, which closes a socket with 1006 when a loop stall makes its
PONG timer run before the PONG already queued is read. Both now use
LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws.

The Mochi panel and Crew Companion sockets retried on a fixed doubling
delay reset on every open, so every copy retried together after a stall
and a gateway that drops new sockets was retried at the shortest delay.
They now take a random part of up to half off each delay and reset the
backoff only after a socket stayed open 10 s.

Fixes #33769
@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners November 9, 2026 22:40
@JLargent42 JLargent42 self-assigned this Nov 9, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff changes no visible control, string or state. It only changes the timing of socket keep-alive and reconnects, so no screenshot is needed.

[UX-REVIEWED] 587fcc7

Description read

[DESCRIPTION-READ] 28bc5751d18afdad2165593daec99cfaeb45535b65615726043cad4516629ce3

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix targets the stall-ordering cause on both heartbeat sockets, and every change can be reverted.

Notes

  • ws_heartbeat.py is a copy of fix(dashboard): share slot list passes and stagger reconnects #33825's module. AGENTS.md says PRs that need the same new helper share one owner PR, and the others import it and name the merge order. Two identical adds merge cleanly today, but the two copies drift if either PR changes its copy before both land.
  • The subclass relies on private aiohttp fields (_pong_response_cb, _pong_heartbeat, _loop), but setup.cfg allows any aiohttp>=3.9,<4. Only _need_heartbeat_reset has a fallback. If a release renames another field, the timer raises inside its loop callback, and a dead peer is never dropped. A test that checks those fields exist would catch this at upgrade time.

[DESIGN-REVIEWED] 587fcc7

Description read

[DESCRIPTION-READ] f0b13b9189fc983a9abea83ff5998a0d18cc03b938eb940df509bc578c65581c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The one note is a helper module copied from #33825, which AGENTS.md says should have a single owner PR.

Not checked: whether #33825 lands its own ws_heartbeat.py byte-identical to this copy, as the description claims.

Notes

What this change ships

Inventory (7 items) — 6 justified

Intent: when the gateway's event loop stalls, a terminal tab or dictation session whose client answered should stay connected, and the Mochi panel and Crew Companion should stop reconnecting in lockstep afterwards. This is a FIX (#33769).

  1. A terminal tab no longer drops when a loop stall hits a socket that already answered the PING — justified
  2. Dictation no longer stops mid-sentence in the same case — justified
  3. New lag-tolerant socket class ws_heartbeat.py, copied from fix(dashboard): share slot list passes and stagger reconnects #33825 — inherited, copy of fix(dashboard): share slot list passes and stagger reconnects #33825's module
  4. Mochi panel and Crew Companion reconnect delays lose a random part of up to half — justified
  5. Their backoff no longer resets on every open, only after a socket stays open 10 s — justified
  6. A watchdog replacement of a socket that stayed healthy starts the backoff over — justified
  7. A spec paragraph describing both changes in learn-cron-dashboard.md — justified

[FIRST-PRINCIPLES-REVIEWED] 587fcc7

Description read

[DESCRIPTION-READ] d4f9ebce6a8b938ea21bb25186a283a026ba707a670516e86e80cd4e82ef72e5

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 587fcc7

Verdict parsed from the review's SHA-scoped output markers for commit 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- website/src/hooks/useChatNavigation.ts:98 -- Cached responses cannot restore evicted resource labels

const { isLoading: resolving } = useQuery({
Trigger: A transcript contains 520 distinct bare URLs whose batches resolve before the earlier query-cache entries expire. rememberLabel evicts the first 20 labels, and useChatNavigation selects that first batch’s cached query again.
Outcome: staleTime: Infinity prevents its query function from rerunning, while the cached summaries are ignored. Resolved resource labels revert to hostname fallbacks, and these unresolved misses prevent subsequent links from being enriched.
Anchor: defect
Fix: Use one authoritative label cache for miss detection and rendering, so cached summaries remain usable after the separate map evicts their labels.

[BLOCK-MERGE-DOWNGRADED] 587fcc7
[GPT-REVIEWED] 587fcc7

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — LOW. The failure needs more than 500 distinct bare-URL labels in one tab (useChatNavigation.ts:56, eviction at :69). The first batch then has to become the miss set again (:95) and hit a cached query that staleTime: Infinity won't refetch (:112). Labels already shown have no data stake: the worst case is that sidebar link labels fall back to hostnames, which the user can see, and a reload clears the tab-local store (:54). This PR doesn't touch the file: git diff HEAD^1 HEAD has no hunk in it, and its last change came from #33963. The fix means reworking the label-cache design in a module this PR never changes, which costs more than the harm it prevents.

[ADJUDICATION] 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 total=1 uphold=0 downgrade=1
DOWNGRADE F1 website/src/hooks/useChatNavigation.ts:98 reason=disproportionate-remedy
[GPT-ADJUDICATED] 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 587fcc7f4b2b0989b3f11637e90cf80f13ffcd37 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
A silence-watchdog replacement cleared the opened-at stamp without the
healthy check the close path runs, so a long healthy socket replaced
after a network change kept the grown delay. Run the same check there.
Also reap the stall test's shell in a finally, and open each fake-timer
try right after useFakeTimers.
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 10, 2026
@JLargent42

Copy link
Copy Markdown
Contributor Author

GPT 6.1 disposition (head 587fcc7): the downgraded finding at website/src/hooks/useChatNavigation.ts:98 is outside this diff. This PR does not touch that hook or the label cache, so no change here. Round 1 findings are fixed in 587fcc7: Opus (the watchdog replacement now runs the same healthy check as onclose, pinned by the two new silence-watchdog tests), GPT (the stall test reaps its shell in a finally), and Flake Pattern Ratchet J6.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 10, 2026 01:40

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review on this head checked these points. LagTolerantWebSocketResponse keeps a socket only when the peer already answered, and it still closes a silent peer at most one window later. The terminal and STT constructor swaps keep their arguments, and the refusal sockets stay on the stock class. No test still patches the old terminal.web.WebSocketResponse seam. In both app sockets the backoff resets only after a socket proved healthy, on onclose and on the watchdog replacement path. A reviewer traced the targeted reverts in the body, and each one fails its test. The branch merges cleanly with main and with #33825.
Well done: the raw TCP control test shows stock aiohttp dropping a peer that answered in time. That pins the bug as well as the fix.
Optional:

  • This PR and #33825 add the same ws_heartbeat.py and its test, and AGENTS.md asks for one owner PR for a shared helper. Please name the merge order and drop the copy once the other lands. On the same point, the header in reconnectBackoff.ts says the dashboard socket shares the rule, but nothing imports it there.
  • Every loop-lag test stalls past the tolerance. So removing either branch of answered or late in ws_heartbeat.py:39 keeps them green. A case where the stall ends inside the tolerance, plus an "still open after the stall" check, would pin both branches.
  • In sessionWatch.socketSilence.test.ts and mochiSocketSilence.test.ts, the Math.random spy is restored at the end of the test body. A try/finally like the sibling tests would stop it leaking after a failure.

@KiroCrewCaptain
KiroCrewCaptain merged commit 1139925 into main Nov 10, 2026
108 of 113 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/33769-sibling-socket-stall branch November 10, 2026 01:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 10, 2026
bolichen97 pushed a commit that referenced this pull request Nov 10, 2026
…34167)

* fix(dashboard): keep terminal and voice sockets through a loop stall

The terminal and streaming dictation sockets used aiohttp's stock
heartbeat, which closes a socket with 1006 when a loop stall makes its
PONG timer run before the PONG already queued is read. Both now use
LagTolerantWebSocketResponse, the subclass #33825 adds for /api/ws.

The Mochi panel and Crew Companion sockets retried on a fixed doubling
delay reset on every open, so every copy retried together after a stall
and a gateway that drops new sockets was retried at the shortest delay.
They now take a random part of up to half off each delay and reset the
backoff only after a socket stayed open 10 s.

Fixes #33769

* fix(dashboard): start app backoff over on a healthy watchdog replace

A silence-watchdog replacement cleared the opened-at stamp without the
healthy check the close path runs, so a long healthy socket replaced
after a network change kept the grown delay. Run the same check there.
Also reap the stall test's shell in a finally, and open each fake-timer
try right after useFakeTimers.

Backport note: backend half only. The Mochi panel and Crew Companion
reconnect changes (website/src/apps/**, website/src/utils/reconnectBackoff.ts
and their tests) are app fixes and stay off this hotfix; the spec line
keeps only the terminal and dictation socket sentence.

(cherry picked from commit 1139925)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Terminal and voice sockets drop on a loop stall, and the Mochi panel and Crew Companion reconnect in lockstep

2 participants