Repository navigation
fix(dashboard): stop cautious boot slowing the recovery it exists for - #16954
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The marker is written the moment Watch
Suggestions
[DESIGN-REVIEWED] 73e4eb6 Description read[DESCRIPTION-READ] e98e7ed0b1d96219df7d1c0941058f980e4ad60adf6feb981986e294c5869c8d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. Not justified as shipped
What this change shipsInventory (5 items) — 4 justifiedIntent: after a loop-stall crash, a gateway that had already been serving restarts at full speed instead of being slowed down as if it had hung during startup. FIX (#6590).
[FIRST-PRINCIPLES-REVIEWED] 73e4eb6 Description read[DESCRIPTION-READ] 95fd25e4de54864faec940d7bc93e152e18bf77b33db714adf1af347221a1bad That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings that block the merge. One advisory finding. FINDING — src/kiro_crew/dashboard/crash_dump_store.py:1004 — an in-app restart or update runs [OPUS-REVIEWED] 73e4eb6 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
a3c2e57 to
ca8bb8c
Compare
ca8bb8c to
091cc8c
Compare
Cautious boot staggers the startup battery when a recent loop-stall dump says the last instance wedged. It reads the dump's age and the host's current memory posture, but not whether the instance that wrote the dump had ever finished starting up -- so a gateway that served for nine hours and then stalled got the same treatment as one that wedged mid-boot. The start immediately after a stall is therefore the slowest one, on exactly the host that most needs to come back, and on a host already near the stall threshold that stagger can trip the watchdog again and write another dump. Reported upstream as #6590, with the cycle observed directly: two-minute lifetimes while it held, and 7.4h, 9h and 15.5h once the dumps were moved aside by hand. The stagger exists to protect the startup BATTERY. An instance that reached a serving state got the whole battery away before it stalled, so the battery is not what wedged it. DashboardState.ready is the codebase's own definition of that point, so both sites that publish it now record the instance's identity triple -- the same (pid, domain, start_id) a dump header carries -- and the next boot asks whether the dump it found was written by a process that had reached it. A downgrade, not an off-switch: a host still reading tight or critical keeps a mild stagger, because current pressure is its own signal and is not answered by what the previous instance managed to do. The match is deliberately strict, because the two errors do not cost the same. A false "reached healthy" removes the stagger and is how a host that wedges during startup re-wedges; a false negative only costs the slower boot that happens today. So a missing marker, a start identity absent on either side, a recycled pid, a marker from another host and the legacy header that carries no identity at all all answer no. That also makes the change inert on any platform without a start identity, and leaves every existing decision path untouched. The marker is written off-loop: it is a small write, but the data home can be on network storage, and this runs on the gateway loop. Recording it is best-effort and never raises -- a data home that cannot take the write must not fail a gateway that is otherwise healthy. A missing marker reads as "did not reach healthy", which is the conservative answer.
Three ways the marker could cost the boot it exists to speed up. The write was awaited. `await asyncio.to_thread(...)` keeps the event loop responsive but not the coroutine, and this coroutine's RETURN is what publishes KIROCREW_READY -- so a data home on a stalled network share held readiness open indefinitely, and a supervisor waiting on that line respawns into the same hang. Nothing about the marker belongs to this boot, so it is dispatched as a tracked background task instead. The read followed and blocked. The marker sits in the dumps directory, which the agent can write to, and `read_text` follows a symlink planted at that name and blocks opening a FIFO. The enclosing except is fail-OPEN and cannot catch a hang. The read now uses O_NOFOLLOW and O_NONBLOCK where the platform has them, checks S_ISREG everywhere, and is bounded to one short line, so anything that is not a plain file reads as absent -- the same conservative answer a missing marker already gives. The write followed too, which is worse than reading wrong. Its temp name was derived from the PID, and PID 1 in a container is deterministic across restarts, so a symlink can be waiting at that name before the process that writes through it exists -- and a plain write follows it and truncates the target. It now publishes through atomic_write, whose temp file is unique and O_EXCL. Also drops an issue number from a test docstring, and prunes three entries that this change made black-clean from the formatting baseline.
091cc8c to
73e4eb6
Compare
|
Self-review (crew-pr-reviewer, read-only): RECOMMEND APPROVE Reviewed head |
Problem / Motivation
Goal: After a loop-stall crash dump, let a gateway that had actually been serving boot back at full speed instead of being staggered as if it had wedged mid-startup.
Cautious boot staggers the startup battery when a recent loop-stall dump says the last instance wedged. It reads the dump's age and the host's current memory posture, but not whether the instance that wrote the dump had ever finished starting up. So a gateway that served for hours and then stalled got the same stagger as one that wedged mid-boot. The boot immediately after a stall is therefore the slowest one, on exactly the host that most needs to come back — and on a host already near the stall threshold that stagger can trip the watchdog again and write another dump. Observed directly: two-minute lifetimes while the stagger held, then 7.4h / 9h / 15.5h once the dumps were moved aside by hand.
Why it matters
The stagger exists to protect the startup battery. When the previous instance had already reached a serving state, the battery is demonstrably not what wedged it, so staggering the recovery boot only slows the recovery it was meant to help — and can re-trip the watchdog on a pressured host, turning a single stall into a restart loop. Users on hosts with no supervisor (e.g. Windows, #6590) are left with a gateway that recovers slowest exactly when recovery matters most.
Not a goal
tight/criticalkeeps a mild stagger, because current pressure is its own signal.What changed (motivation → approach → change)
Observed symptom: the first boot after a loop-stall dump is the slowest, even when the stalled instance had served for hours. Root cause:
_evaluatein cautious boot scaled the delay by dump age and current posture only — it had no signal for whether the stalled instance had ever reached a serving state. Change:DashboardState.readyis the codebase's own definition of "finished starting up". Both sites that publish it (start_dashboard,start_api_server) now record this process's identity triple(pid, pid_domain, start_id)— the same triple a dump header carries — viarecord_healthy_boot.KIROCREW_READY, so awaiting a marker write on network storage could hold readiness open forever and make a supervisor respawn into the same hang.dump_owner_reached_healthy(dump): true only when the marker names the SAME process as the dump header (same pid, same domain, start identity present and equal). The match is deliberately strict — a false "reached healthy" removes the stagger and is how a startup-wedge re-wedges, while a false negative only costs today's slower boot. So a missing marker, an absent start identity on either side, a recycled pid, a marker from another host, and the legacy identity-less header all answer no.O_NOFOLLOW/O_NONBLOCKwhere available and anS_ISREGcheck everywhere, and is byte-bounded, so a symlink or FIFO planted at the agent-writable marker path reads as absent rather than hanging the boot path. The write goes throughatomic_writefor itsO_EXCLunique temp file for the same reason.This is a bug fix restoring intended behaviour (not a product-shape change): the stagger's purpose is unchanged, only a missing input is added. Everything fails open — an unreadable dump store, config error, or probe failure yields a normal un-staggered boot.
Backwards compatibility
Compatible: adds a new best-effort marker and a new, stricter condition that can only make a boot FASTER than today; it rejects, renames or removes nothing. The change is inert on any platform without a start identity (the identity-less path answers "not healthy", i.e. today's behaviour), and leaves every existing cautious-boot decision path untouched.
Tests
test/test_cautious_boot.pyextended (38 cases pass locally):-start identity, recycled pid, foreign-host marker, and legacy identity-less header all keep the full stagger;record_healthy_bootnever raises on a write failure.Manual verification
N/A — unit coverage sufficient; the behaviour is fully exercised by the identity-match and fail-open unit tests, with no UI or external-service surface.
Screenshots / video
N/A — no user-visible frontend surface changed (dashboard/API boot path only).
Related Issues
Fixes #6590
Pattern harvest
Rule candidate: work that belongs to the NEXT boot (a hint, a marker) must never be awaited on the readiness path — dispatch it off-loop so a stalled data home cannot hold
KIROCREW_READYopen and make a supervisor respawn into the same hang.Further patterns surfaced by this fix:
O_NOFOLLOW/O_NONBLOCK+S_ISREG, so a planted symlink or FIFO reads as absent instead of hanging a path with no recovery.Checklist
test/test_cautious_boot.py, 38 passed).black --checkclean on all changed files.Contribution License Agreement
I confirm this contribution is made under the terms of the project's Contribution License Agreement.
Supersedes #11288 (fork PR by @GoZippy). Original author @GoZippy; both commits retain their original authorship. The change continues here as a same-repo PR so the full CI and review lanes can run.