Skip to content

fix(dashboard): stop cautious boot slowing the recovery it exists for - #16954

Merged
chenmingwei23 merged 2 commits into
mainfrom
replace/cautious-boot-recovery-11288
Oct 6, 2026
Merged

chenmingwei23 merged 2 commits into
mainfrom
replace/cautious-boot-recovery-11288

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: After a loop-stall crash dump, let a gateway that had actually been serving boot back at full speed instead of being staggered as if it had wedged mid-startup.

Cautious boot staggers the startup battery when a recent loop-stall dump says the last instance wedged. It reads the dump's age and the host's current memory posture, but not whether the instance that wrote the dump had ever finished starting up. So a gateway that served for hours and then stalled got the same stagger as one that wedged mid-boot. The boot immediately after a stall is therefore the slowest one, on exactly the host that most needs to come back — and on a host already near the stall threshold that stagger can trip the watchdog again and write another dump. Observed directly: two-minute lifetimes while the stagger held, then 7.4h / 9h / 15.5h once the dumps were moved aside by hand.

Why it matters

The stagger exists to protect the startup battery. When the previous instance had already reached a serving state, the battery is demonstrably not what wedged it, so staggering the recovery boot only slows the recovery it was meant to help — and can re-trip the watchdog on a pressured host, turning a single stall into a restart loop. Users on hosts with no supervisor (e.g. Windows, #6590) are left with a gateway that recovers slowest exactly when recovery matters most.

Not a goal

What changed (motivation → approach → change)

Observed symptom: the first boot after a loop-stall dump is the slowest, even when the stalled instance had served for hours. Root cause: _evaluate in cautious boot scaled the delay by dump age and current posture only — it had no signal for whether the stalled instance had ever reached a serving state. Change:

  • DashboardState.ready is the codebase's own definition of "finished starting up". Both sites that publish it (start_dashboard, start_api_server) now record this process's identity triple (pid, pid_domain, start_id) — the same triple a dump header carries — via record_healthy_boot.
  • The write is dispatched as a tracked background task, never awaited: the coroutine's return publishes KIROCREW_READY, so awaiting a marker write on network storage could hold readiness open forever and make a supervisor respawn into the same hang.
  • The next boot asks dump_owner_reached_healthy(dump): true only when the marker names the SAME process as the dump header (same pid, same domain, start identity present and equal). The match is deliberately strict — a false "reached healthy" removes the stagger and is how a startup-wedge re-wedges, while a false negative only costs today's slower boot. So a missing marker, an absent start identity on either side, a recycled pid, a marker from another host, and the legacy identity-less header all answer no.
  • When the owner reached healthy: no stagger on a calm host, a mild stagger (not maximum) on a still-pressured one — a downgrade, not an off-switch.
  • The marker read uses O_NOFOLLOW / O_NONBLOCK where available and an S_ISREG check everywhere, and is byte-bounded, so a symlink or FIFO planted at the agent-writable marker path reads as absent rather than hanging the boot path. The write goes through atomic_write for its O_EXCL unique temp file for the same reason.

This is a bug fix restoring intended behaviour (not a product-shape change): the stagger's purpose is unchanged, only a missing input is added. Everything fails open — an unreadable dump store, config error, or probe failure yields a normal un-staggered boot.

Backwards compatibility

Compatible: adds a new best-effort marker and a new, stricter condition that can only make a boot FASTER than today; it rejects, renames or removes nothing. The change is inert on any platform without a start identity (the identity-less path answers "not healthy", i.e. today's behaviour), and leaves every existing cautious-boot decision path untouched.

Tests

test/test_cautious_boot.py extended (38 cases pass locally):

  • a dump owner that reached healthy skips the stagger on a calm host and keeps a mild (not maximum) stagger on a pressured host;
  • the strict identity match: missing marker, missing/- start identity, recycled pid, foreign-host marker, and legacy identity-less header all keep the full stagger;
  • the marker read is bounded and treats a non-regular file (symlink/FIFO) as absent;
  • record_healthy_boot never raises on a write failure.

Manual verification

N/A — unit coverage sufficient; the behaviour is fully exercised by the identity-match and fail-open unit tests, with no UI or external-service surface.

Screenshots / video

N/A — no user-visible frontend surface changed (dashboard/API boot path only).

Related Issues

Fixes #6590

Pattern harvest

Rule candidate: work that belongs to the NEXT boot (a hint, a marker) must never be awaited on the readiness path — dispatch it off-loop so a stalled data home cannot hold KIROCREW_READY open and make a supervisor respawn into the same hang.

Further patterns surfaced by this fix:

  • A file at an agent-writable path on the boot path must be opened with O_NOFOLLOW/O_NONBLOCK + S_ISREG, so a planted symlink or FIFO reads as absent instead of hanging a path with no recovery.
  • Asymmetric-cost decisions want asymmetric defaults: when a false positive re-wedges a host and a false negative only costs a slower boot, require strict positive evidence and treat every ambiguity as negative.

Checklist

  • Tests added/updated and passing locally (test/test_cautious_boot.py, 38 passed).
  • black --check clean on all changed files.
  • No workflow or security-scope files touched.
  • Fails open on every error path.

Contribution License Agreement

I confirm this contribution is made under the terms of the project's Contribution License Agreement.


Supersedes #11288 (fork PR by @GoZippy). Original author @GoZippy; both commits retain their original authorship. The change continues here as a same-repo PR so the full CI and review lanes can run.

@iamwhatever
iamwhatever requested a review from a team as a code owner October 5, 2026 07:45
@iamwhatever
iamwhatever requested a review from CrysisDeu October 5, 2026 07:45
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 73e4eb68293722380801d271c006ee2c8c8a1e28 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The marker is written the moment state.ready flips, while much of the startup battery still runs after it, so a stall right after ready drops the stagger, which is the false-positive direction.

Watch

  • "Reached ready" does not mean the battery finished. The diff says _dispatch_healthy_boot_marker(state) runs right after state.ready = True, with the comment "this instance got the whole startup battery away". But in slack/gateway.py, work still runs after that point: _ensure_subagent_coordinator, _wait_for_memory_preparation, the restore/open worker, the crewmate prune, channel-agent resume and cron arming. The kiro-cli children that session restore spawned also outlive it. A host that wedges seconds after ready under that leftover load will have its marker match its dump, and the next boot loses the full stagger. The description itself calls that outcome "how a startup-wedge re-wedges". The incident it cites (7–15h lifetimes) would be fixed just as well by a stricter signal.
    Clears when: the marker is written only after the post-ready startup settles (after the memory barrier and cron arm in GatewayOrchestrator.run), or after a minimum uptime, or the PR shows that a stall between ready and that point cannot be caused by the staggered groups.

Suggestions

  • Write the marker only after N minutes of uptime, or record the ready timestamp in it and require some uptime before the dump. The second option keeps today's dispatch site, and either one makes "served before stalling" literally true.

[DESIGN-REVIEWED] 73e4eb6

Description read

[DESCRIPTION-READ] e98e7ed0b1d96219df7d1c0941058f980e4ad60adf6feb981986e294c5869c8d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 73e4eb68293722380801d271c006ee2c8c8a1e28 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

Not justified as shipped

  • Item 5 — undeclared: the diff takes three files off the black baseline and reformats lines unrelated to the fix (pause_before log call, pid_identity_alive condition, win32 blank lines). The description says only "black --check clean on all changed files."

What this change ships

Inventory (5 items) — 4 justified

Intent: after a loop-stall crash, a gateway that had already been serving restarts at full speed instead of being slowed down as if it had hung during startup. FIX (#6590).

  1. After a stall, the next boot skips the stagger on a calm host if the stalled instance had finished starting up — justified
  2. The same case on a tight or critical host gets the mild stagger instead of the maximum — justified
  3. Each process that finishes starting up (dashboard and API server) writes a last-healthy-boot marker in the dumps directory, in the background — justified
  4. The marker is read strictly: a symlink, FIFO, oversized file, missing start identity or pid mismatch all count as "not healthy" — justified
  5. Black-baseline removal and reformatting of untouched lines — undeclared, rides along, no harm

[FIRST-PRINCIPLES-REVIEWED] 73e4eb6

Description read

[DESCRIPTION-READ] 95fd25e4de54864faec940d7bc93e152e18bf77b33db714adf1af347221a1bad

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 73e4eb68293722380801d271c006ee2c8c8a1e28 — this comment is updated in place on each push.

Review details

No findings that block the merge. One advisory finding.

FINDING — src/kiro_crew/dashboard/crash_dump_store.py:1004 — an in-app restart or update runs os.execv (reexec_python_module / reexec_launcher, called from updates.py:1811 and gateway.py:11317). On Linux and macOS that keeps the PID, and procfs field 22 / the libproc start time stay the same. So the post-exec image opens a new dump with the same # PID: n @ domain start=s header. If that image wedges during its startup battery, it never writes a marker of its own, so the pre-exec image's marker still passes m_pid == str(pid) and m_domain == domain and m_start == start_id. The next boot then treats a startup wedge as "reached a serving state": the stagger drops from MAX to MILD on a pressured host, and to none on an ample one → Fix: tie the marker to one dump, not one PID. Have record_healthy_boot also write the name of the active dump file (_active_dump_file.path.name), and have dump_owner_reached_healthy require that name to equal dump_path.name.

[OPUS-REVIEWED] 73e4eb6

Verdict parsed from the review's SHA-scoped output markers for commit 73e4eb68293722380801d271c006ee2c8c8a1e28.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 73e4eb68293722380801d271c006ee2c8c8a1e28: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 73e4eb68293722380801d271c006ee2c8c8a1e28 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 73e4eb6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 73e4eb68293722380801d271c006ee2c8c8a1e28: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 5, 2026
@iamwhatever
iamwhatever force-pushed the replace/cautious-boot-recovery-11288 branch from a3c2e57 to ca8bb8c Compare October 5, 2026 16:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 5, 2026
@iamwhatever
iamwhatever force-pushed the replace/cautious-boot-recovery-11288 branch from ca8bb8c to 091cc8c Compare October 5, 2026 16:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 5, 2026
Cautious boot staggers the startup battery when a recent loop-stall dump
says the last instance wedged. It reads the dump's age and the host's
current memory posture, but not whether the instance that wrote the dump
had ever finished starting up -- so a gateway that served for nine hours
and then stalled got the same treatment as one that wedged mid-boot. The
start immediately after a stall is therefore the slowest one, on exactly
the host that most needs to come back, and on a host already near the
stall threshold that stagger can trip the watchdog again and write
another dump. Reported upstream as #6590, with the
cycle observed directly: two-minute lifetimes while it held, and 7.4h,
9h and 15.5h once the dumps were moved aside by hand.

The stagger exists to protect the startup BATTERY. An instance that
reached a serving state got the whole battery away before it stalled, so
the battery is not what wedged it. DashboardState.ready is the codebase's
own definition of that point, so both sites that publish it now record
the instance's identity triple -- the same (pid, domain, start_id) a dump
header carries -- and the next boot asks whether the dump it found was
written by a process that had reached it.

A downgrade, not an off-switch: a host still reading tight or critical
keeps a mild stagger, because current pressure is its own signal and is
not answered by what the previous instance managed to do.

The match is deliberately strict, because the two errors do not cost the
same. A false "reached healthy" removes the stagger and is how a host
that wedges during startup re-wedges; a false negative only costs the
slower boot that happens today. So a missing marker, a start identity
absent on either side, a recycled pid, a marker from another host and
the legacy header that carries no identity at all all answer no. That
also makes the change inert on any platform without a start identity,
and leaves every existing decision path untouched.

The marker is written off-loop: it is a small write, but the data home
can be on network storage, and this runs on the gateway loop.

Recording it is best-effort and never raises -- a data home that cannot
take the write must not fail a gateway that is otherwise healthy. A
missing marker reads as "did not reach healthy", which is the
conservative answer.
Three ways the marker could cost the boot it exists to speed up.

The write was awaited. `await asyncio.to_thread(...)` keeps the event loop
responsive but not the coroutine, and this coroutine's RETURN is what
publishes KIROCREW_READY -- so a data home on a stalled network share held
readiness open indefinitely, and a supervisor waiting on that line
respawns into the same hang. Nothing about the marker belongs to this
boot, so it is dispatched as a tracked background task instead.

The read followed and blocked. The marker sits in the dumps directory,
which the agent can write to, and `read_text` follows a symlink planted
at that name and blocks opening a FIFO. The enclosing except is fail-OPEN
and cannot catch a hang. The read now uses O_NOFOLLOW and O_NONBLOCK
where the platform has them, checks S_ISREG everywhere, and is bounded to
one short line, so anything that is not a plain file reads as absent --
the same conservative answer a missing marker already gives.

The write followed too, which is worse than reading wrong. Its temp name
was derived from the PID, and PID 1 in a container is deterministic
across restarts, so a symlink can be waiting at that name before the
process that writes through it exists -- and a plain write follows it and
truncates the target. It now publishes through atomic_write, whose temp
file is unique and O_EXCL.

Also drops an issue number from a test docstring, and prunes three
entries that this change made black-clean from the formatting baseline.
@iamwhatever
iamwhatever force-pushed the replace/cautious-boot-recovery-11288 branch from 091cc8c to 73e4eb6 Compare October 5, 2026 19:56
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): RECOMMEND APPROVE

Reviewed head 73e4eb6. The new last-healthy-boot marker is written once, when the dashboard reports ready, and never raises. It holds the same (pid, domain, start_id) identity a dump header carries. dump_owner_reached_healthy is True only on an exact match with a start identity on both sides, so every unknown case keeps today's stagger. The marker read refuses symlinks (O_NOFOLLOW), FIFOs (O_NONBLOCK) and non-regular files, and is capped at 256 bytes, so a planted file cannot hang the boot path. The worst a planted marker can do is skip the stagger on a calm host, and a pressured host still gets the mild delay. Some lines are only reformatted, and .github/black-baseline.txt changes with them, which adds diff noise but no behaviour.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 6, 2026 06:12
@chenmingwei23
chenmingwei23 merged commit 9d8d18e into main Oct 6, 2026
151 of 154 checks passed
@chenmingwei23
chenmingwei23 deleted the replace/cautious-boot-recovery-11288 branch October 6, 2026 06:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

After a loop-stall termination the gateway cannot recover unattended on Windows: no supervisor, and a recent dump slows every restart

3 participants