Skip to content

fix(chat): stop the crash-recovery reload loop on a busy long chat - #16017

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/chat-crash-loop-recovery
Oct 2, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/chat-crash-loop-recovery

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A long chat that is still running no longer freezes the dashboard on open, and a crash no longer reopens the same chat in a loop.

Users report the app window freezing over and over (#12907). The terminal shows Renderer process killed, then the window comes back frozen again. The web dashboard freezes the same way. Users cannot switch chats at all.

Why it matters

Anyone with a long chat that is busy (a long task, a monitor loop) can be locked out of the whole dashboard. Restarting does not help, because the app reopens that same chat. The only way out today is to clear browser storage by hand.

Not a goal

What changed (motivation → approach → change)

Two fixes. One breaks the loop. One removes the freeze.

Fix Where Effect
Safe reload website/electron/renderer-recovery.js, website/electron/window-lifecycle.js The crash-recovery reload adds safe=1 to the dashboard URL. If that load gets a 403, the token retry keeps safe=1
website/src/lib/safeReload.ts, website/src/main.tsx, website/src/lib/staleShellHeal.ts The page reads safe=1 once at startup, then strips it from the address bar. Every reload the page makes for itself during boot (restored UI prefs, the stale-shell heal, a stale-chunk reload) puts safe=1 back, so the load that renders is still safe
website/src/pages/chat/useChatPageSessionController.ts On a safe reload, no chat is opened automatically; the user picks one
Bounded open website/src/components/ChatPane.tsx A running chat now loads the last PANE_HYDRATE_LIMIT rows, like an idle one

The freeze: ChatPane dropped its row limit whenever the chat was running. A busy long chat then loaded its whole history at once. The renderer hung, and the watchdog killed it.

The loop: the recovery reload restored the last open chat. That was the same busy chat, so it froze again.

The old reason for loading everything while running no longer holds. The server collapses streaming chunks before it counts and slices rows, so a bounded page is correct mid-stream too. This is the change #10026 proposed, redone on current main.

The remembered chat is not erased. A normal launch still returns to it. A manual refresh is a normal load, because safe=1 is already gone from the URL.

Known residual

ChatPane.tsx still reads the whole transcript for a crewmate pane whose bounded window holds no speech (crewmateQuietUnproven). That path is not changed here. It only applies to crewmate panes, and bounding it needs its own design for the quiet-hint read. It is left for a follow-up.

Backwards compatibility

Compatible: nothing that works today is rejected. safe=1 is a new optional URL flag, and normal loads behave as before.

Tests

  • website/electron/test/renderer-recovery.test.js: the recovery reload URL carries safe=1, with or without a credential.
  • website/src/lib/safeReload.test.ts: safe=1 is detected, stripped from the URL, and other params stay.
  • website/src/lib/safeReload.test.ts: a boot-time reload of a safe load restores safe=1; on a normal load it is a plain reload.
  • website/src/test/staleShellHeal.test.ts, website/src/test/staleChunkReloadKeepsSafe.test.ts: the stale-shell heal and the stale-chunk handler keep safe=1 on a safe load. Both fail without the fix.
  • website/src/test/ChatPage.persist.integration.test.tsx: a safe reload opens no chat; once the user opens one, closing it moves to another chat as usual.
  • website/src/test/ChatPage.persist.integration.test.tsx: a safe reload opens no chat, and a normal load still restores the remembered one.
  • website/src/test/ChatPane.hydrateBound.test.tsx: a running chat requests a bounded page. Both tests fail with the ChatPane change reverted.

Manual verification

Still required: open a running long chat in the desktop app built from this branch (make desktop), and confirm it opens without freezing. Then force a renderer crash and confirm the window comes back with no chat open.

Screenshots / video

Why no screenshot: No layout or style changes; the diff only changes which chat opens after a crash and how many rows a running chat loads.

Related Issues

Supersedes #10026, which was closed as covered by #13203; #13203 bounded only the background warm-up, not this ChatPane path.

Fixes #12907

Pattern harvest

Rule candidate: review-prompt
Pattern: "a crash-recovery path that restores the same state that caused the crash"

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team October 1, 2026 22:19
@iamwhatever
iamwhatever requested a review from a team as a code owner October 1, 2026 22:19
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — 🟡 CONCERNS

UX-level review of 60bda97d66054bc81732483fcccbcdd2384765d3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

Crash recovery lands the user on the welcome screen with no explanation of why their chat did not reopen.

Watch

  • Silent safe-reload landing: captureSafeReload() only flips a module flag and useChatPageSessionController skips the auto-open ("A crash-recovery reload opens nothing… The user picks."), so after a renderer crash the window returns showing the welcome hero instead of the chat the user was watching, with zero notice — at the moment they just saw the window die, that reads as a lost session until they spot the chat in the tab strip. Low frequency (crash only) × moderate confusion × every recovery. Smallest fix: when isSafeReload() is true, show one dismissible line on the welcome surface — "Recovered after a crash — your last chat wasn't reopened automatically; open it from the list."

[UX-REVIEWED] 60bda97

Description read

[DESCRIPTION-READ] 86a3908b5b4d4b33f4f64a54b762c0d4602625235276ed53bbc50032259d190d

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 60bda97d66054bc81732483fcccbcdd2384765d3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The safe-flag survives four self-reloads but not all six: bundleReload.ts:97 and reconnectCatchUp.ts:92 still drop safe=1 and can resume the loop.

Not justified as shipped

  • Item 5 — symptom-level: safe=1 preservation is wired per call site; grep of window.location.reload() in website/src finds 6 automatic self-reload paths, 4 routed through reloadKeepingSafe, 2 not — src/hooks/websocket/bundleReload.ts:97 (fires on the first status frame after a gateway update, plausibly during a safe boot) and src/hooks/websocket/reconnectCatchUp.ts:92 (fires on reconnect after an update restart). Either reopens the remembered chat and the loop resumes; the "Known residual" section names only the crewmate path.
    Subtraction: replace those two window.location.reload() calls with the existing reloadKeepingSafe from src/lib/safeReload.ts.
    Clears when: both siblings route through reloadKeepingSafe, or the description's Known residual names them as deliberately left out.

What this change ships

Inventory (7 items) — 6 justified

Intent: stop a busy long chat from freezing the dashboard on open and from being reopened in a loop after the crash (#12907) — a FIX.

  1. A running chat now opens with the last bounded page of rows, not its whole history — justified
  2. After a renderer crash or hang, the dashboard reopens with no chat; the user picks — justified
  3. The recovery reload carries a new safe=1 URL flag — justified
  4. A 403 token retry keeps safe=1, so the retried load stays safe — justified
  5. Boot-time self-reloads (UI prefs, durable keys, stale chunk, stale shell) keep safe=1 — symptom-level: two sibling automatic reloads still drop it
  6. safe=1 is stripped from the address bar at once; a manual refresh is normal — justified
  7. The remembered chat is kept; a normal launch still returns to it — justified

The deleted "unbounded while streaming is deliberate" pin (comment + test) is declared in the description and reversed with evidence: linked issue #12907 and prior proposal #10026.

[FIRST-PRINCIPLES-REVIEWED] 60bda97

Description read

[DESCRIPTION-READ] e3cb8f4a56e951364a01e0b91ffaac1947e7fae101e892e9d5ec1e4a82f79dbc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 60bda97d66054bc81732483fcccbcdd2384765d3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 60bda97

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 60bda97d66054bc81732483fcccbcdd2384765d3: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 60bda97d66054bc81732483fcccbcdd2384765d3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Desktop crash-recovery path is declared unverified — the description's own "Manual verification: Still required" must be done before merge.

Watch

  • The description says the desktop run ("open a running long chat… force a renderer crash") is still required. The Electron recovery → loadURL → token-retry chain only runs in the packaged shell CI never exercises; if safe=1 is dropped on a path the unit tests don't model, the loop survives for exactly the Dashboard crash loop: long chat session freezes renderer, recovery reloads into the same slot #12907 users. Mitigated: the CI-tested ChatPane bound removes the freeze cause independently, so the consequence is a degraded guard, not a dead dashboard.
    Clears when: the author performs the manual desktop verification named in the description and records the result on the PR.

  • safe=1 survival is a cooperation convention: every current and future self-initiated boot reload must remember reloadKeepingSafe(). The three known sites are patched and pinned by tests, but a new window.location.reload() in boot code silently drops the flag with no gate catching it.
    Clears when: a lint/test gate forbids bare location.reload() in boot modules, or the flag moves to a store that survives reloads without per-site opt-in (see Suggestions).

Suggestions

  • Capture the flag once into a sessionStorage one-shot consumed at the first auto-open decision; any boot reload then preserves it with no reloadKeepingSafe cooperation, deleting the convention and two of the bespoke keep-safe tests.

[DESIGN-REVIEWED] 60bda97

Description read

[DESCRIPTION-READ] 34f298f225d35fe786ea710608b6a99e0f22674d4746a7f40e66e596095ec519

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 60bda97d66054bc81732483fcccbcdd2384765d3 — this comment is updated in place on each push.

Review details

Nothing blocks merge; one advisory — the unbounded whole-transcript hydrate this change removes for running slots is still routinely reachable on the crewmate pane.

FINDING — website/src/components/ChatPane.tsx:739 — the surviving crewmateQuietUnproven latch still sets limitRef.current = undefined, and api.chatSlotDetail(slot, undefined) omits ?limit= so api_chat_slot_detail returns the FULL chained history from disk; a Members-page crewmate pane on a long patrol session (rows present, warmHasMore === true, filterCrewmateChat yields no speech in the 50-row window — the case the latch exists for) therefore performs exactly the whole-transcript read the new comment says "froze the renderer", and safe=1 does not cover it because that pane is opened from Members, not by the useChatPageSessionController auto-open → Fix: cap that upgrade at a bounded ceiling (SLOT_DETAIL_MAX_LIMIT) instead of undefined, or decide the never-spoken hint from the page's total/has_more — needs a value outside this diff's lines, so it cannot gate this change.

[OPUS-REVIEWED] 60bda97

Verdict parsed from the review's SHA-scoped output markers for commit 60bda97d66054bc81732483fcccbcdd2384765d3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 60bda97d66054bc81732483fcccbcdd2384765d3: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the fix/chat-crash-loop-recovery branch from 4e42340 to ed50e00 Compare October 1, 2026 23:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: reloadKeepingSafe -- boot-time pref reloads re-add safe=1

  • span=16d5f4349e78: fixed in ed50e00

Legit: the cold-hydrate / roster-gap reload in main.tsx dropped safe=1 before the render load.
Fix: both restored > 0 reloads in main.tsx call reloadKeepingSafe(), which re-adds safe=1 when the current load is safe and is a plain reload otherwise.
Test: safeReload.test.ts covers both branches.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=0a88f4feb16b: fixed in ed50e00

Legit: the module-level flag kept skipping auto-select after the user opened then closed a chat, so no sibling opened.
Fix: a mount-scoped hadActiveSlotRef limits the skip to the first auto-open; after any active slot, the normal sibling pick runs.
Test: ChatPage.persist.integration.test.tsx fails without the ref and passes with it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=d79ed9afe1bd: accepted, named as a known residual

The crewmateQuietUnproven full read stays as is; it applies only to crewmate panes whose bounded window holds no speech.
The PR body's Known residual section names it; bounding it needs its own design for the quiet-hint read, outside this crash-loop fix.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=a63a2a31cb6d: needs-a-decision (manual run by the maintainer)

Unit tests cover the recovery URL, the safe flag capture and the no-auto-open path; CI has no Electron crash harness.
The end-to-end run (open a running long chat; force a renderer crash; confirm no chat reopens) is pending the maintainer's local build and will be posted here.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=86195c1eb1ee: rebutted as out of goal for this PR

A post-crash notice is a new user-visible surface with new i18n strings; the goal here is breaking the freeze loop.
The empty chat surface already shows the sidebar list to pick from. A notice can follow as its own UX change.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=676d24fd04a3: rebutted (disproportional)

The landing is the normal new-chat page with the full chat list in the sidebar, so every chat is one click away.
The page only appears after the renderer was killed by a crash, which is already a visible event to the user.
A "reloaded after a crash" notice is a new UI surface; this PR's goal is breaking the freeze loop, not explaining recovery.
The same notice was declined as out of scope on the prior head; this item is that request in a new place.

@iamwhatever
iamwhatever force-pushed the fix/chat-crash-loop-recovery branch from ed50e00 to 786fce3 Compare October 2, 2026 00:28
@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: main-window did-navigate records whether the navigated URL carried safe=1; the 403 token retry reloads through withSafeReload when it did

  • span=4224e4e09b96: fixed in 786fce3

The main-window 403 retry now rebuilds the URL with safe=1 when the failed load carried it (hasSafeReload(url) in did-navigate, withSafeReload in the retry).
A load without safe=1 retries exactly as before.
Regression tests in website/electron/test/renderer-recovery.test.js cover hasSafeReload on safe, unsafe and malformed URLs.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=b4ce56ad5abf: fixed in 786fce3

The safeReload.ts doc no longer claims the tab set is skipped; it says only the auto-open is skipped.
Restoring the tab strip is harmless here: SessionTabStrip renders labels only, and a tab mounts no ChatPane until the user opens it, so no history loads.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@iamwhatever
iamwhatever force-pushed the fix/chat-crash-loop-recovery branch from 786fce3 to 60bda97 Compare October 2, 2026 05:39
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=e582a7df246b: fixed

fixed in 60bda97: the stale-shell heal (staleShellHeal.ts) and the stale-chunk handler (main.tsx vite:preloadError) now reload through reloadKeepingSafe, so a safe boot that hits either keeps safe=1.
New tests staleShellHeal.test.ts and staleChunkReloadKeepsSafe.test.ts fail without the fix and pass with it.
bundleReload.ts and reconnectCatchUp.ts stay as-is: both need a gateway update event during the page's life (a second status frame with a new version, or an armed update latch plus a socket reopen), so neither fires on the boot path before the user picks a chat.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=b2e9ca88c738: rebutted

A "Kiro restarted, pick a chat" notice is a new on-screen element, which is outside this PR's goal of stopping the crash loop.
The safe load is rare (it only follows a renderer crash), and the page still shows the full chat list to pick from.
Same answer as the earlier UX note on this PR; a notice can follow as its own UI change.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 2, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=6e0dd19da46a: needs-a-decision

needs-a-decision: the live crash -> withSafeReload -> captureSafeReload path needs a hand test on a build of this branch; unit tests pin each step, not the live chain.
Maintainer: please run make desktop on this branch, open a long chat while it is running (should open without freezing), then force a renderer crash (should return with no chat open), and report the result here.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=e4ea71a8008f: needs-a-decision

needs-a-decision: yes, this deliberately reverses the old "unbounded while running" pin. The trade is full history on open vs. not freezing the renderer, and the freeze is the #12907 crash loop.
Older history still loads on scroll-up, the same as for an idle chat.
Maintainer: please confirm the bounded tail for running chats is the call you want.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
@chenmingwei23
chenmingwei23 merged commit a2abb0b into main Oct 2, 2026
98 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/chat-crash-loop-recovery branch October 2, 2026 15:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
bolichen97 added a commit that referenced this pull request Oct 3, 2026
…16017) (#16391)

(cherry picked from commit a2abb0b)

Co-authored-by: Joe Guo <12429733+iamwhatever@users.noreply.github.com>
Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard crash loop: long chat session freezes renderer, recovery reloads into the same slot

2 participants