Skip to content

feat(doctor): list the gateway boots the retained crash dumps record - #19945

Closed
javenciu wants to merge 1 commit into
mainfrom
feat/gateway-boot-ledger
Closed

javenciu wants to merge 1 commit into
mainfrom
feat/gateway-boot-ledger

Conversation

@javenciu

@javenciu javenciu commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: kirocrew doctor lists the retained crash-dump files as gateway starts, with when each began and how it went, without a new record and without changing when doctor reports a problem.

Every gateway start pre-creates a crash-dump file so faulthandler has a stable descriptor (open_dump_file, src/kiro_crew/dashboard/crash_dump_store.py:605), and a file with thread stacks is a start that wedged. The directory holds what is left after two steps the gateway runs before it opens its own file (src/kiro_crew/dashboard/server_runtime/heartbeat.py:65 to :67): the sweep removes the header-only file of a session that is gone (sweep_stale_dumps, crash_dump_store.py:482), and rotation keeps at most ten files and removes header-only ones first (rotate_dumps, :536; _DEFAULT_MAX_DUMPS, :56). So the retained files are the stalls on record plus the newest start, each one a gateway start, and not a complete boot history. Doctor does not read the directory that way. The Loop-stall Crash Dumps section (_doctor_crash_dumps, src/kiro_crew/doctor_checks/workload.py:406) reports the newest dump with stacks, how long ago it was and whether a later session superseded it. To see how often the gateway wedged and came back, and when each start began, an operator lists the dump directory by hand.

Why it matters

#9559 reports a gateway that stalled again after every relaunch while kirocrew doctor reported the backend healthy. An operator who is deciding whether to restart once more has no single view of the recent stalls and when each start began. The data is on disk; it is read one file at a time, by file name (#15077 reconstructs a recurrence that way).

Not a goal

  • A verdict or any change to doctor's exit status. The recent loop-stall crash dump issue (workload.py:480) stays current while two or more retained dumps hold stacks (_stalls_on_record < 2, workload.py:440, added with fix(dashboard): report a loop-stall dump once, not every boot #10747), so a run of recent stalls already turns doctor red there and a count of stalls here would only restate it. The new section takes no issues list.
  • Any new persistence. There is no new file, marker or write on the boot path; the section reads files the gateway already writes. That is why it is not a complete boot history: a start that never wedged leaves its header-only file only until the next start sweeps it, and the section prints a line saying so.
  • Positive evidence of an orderly shutdown (crash_dump_store: header-only loop-stall dumps are still classified as clean exits, so a non-watchdog gateway death leaves no surfaced stacks #14724 proposes a marker). not running says the writer of the newest file is gone, not how it ended; running says it is alive, not that it is serving.
  • Any change to how dumps are classified, swept or rotated.

What changed (motivation → approach → change)

Goal → approach. The dump directory already holds each stall and the newest boot, so the change is one reader over it, one keyword on the owner test it needs, and one doctor section that prints what the reader returns.

  • last_boots (crash_dump_store.py:764) returns the newest _BOOTS_LISTED (5, :739) retained dumps as (started, state) pairs, newest first. It reuses the module's own readers: _list_dumps (:151) for the files, _is_header_only (:241) for the stacks, dump_superseded (:691) for the successor test and _owner_alive (:400), the owner test the sweep (:514) and the rotation (:570) use, for the newest file's writer. That last one is the only reader this change edits: _owner_alive gains a keyword-only identity_required (:404, default False), and last_boots is the only caller that passes it (:800). With it set, the own-PID shortcut is off (:445), and a header with no start identity, or one that cannot be compared with the live process's, answers None instead of falling back to plain PID liveness (:451 to :457); last_boots reads None as unknown. The sweep and the rotation keep the default, which is the code main had; test_owner_alive_without_recorded_start_id_trusts_liveness (test/test_crash_dump_store.py:1074) pins the answer the keyword changes. The rotation needs the default: it spares a file only when the answer is True (:570), so a live gateway whose header has no comparable start identity would answer None under the keyword and could be removed. The states are:
    • stalled: the dump holds stacks, the test dumps_with_stacks (:673) counts, so the number of stalled rows is read the same way as the repeated-stall issue.
    • running / not running: the newest file, no stacks, and its writer is alive with the recorded start identity / gone. _owner_alive matches the header's pid, PID domain and start id against the live process, so a recycled pid does not read as the owner, and takes the probe as a parameter (is_pid_alive, default platform_compat.pid_exists, src/kiro_crew/platform_compat.py:6438), which the tests inject. not running is a start that exited without wedging and has not been swept yet; the next start sweeps it.
    • clean so far: an older file with no stacks whose successor, per dump_superseded, is a later local boot that did not wedge. The sweep removes such a file once its owner is gone, so one shows only while the owner cannot be shown gone.
    • unknown: the dump cannot be read; the newest file's owner cannot be probed from here (no # PID: line, another PID domain, or a start identity that is missing or cannot be compared with the live process's); or an older file with no stacks and no such successor. A file the reader cannot open is unknown, never clean so far or running.
  • _boot_started (crash_dump_store.py:745) gives the start time: the opened stamp in the header (open_dump_file writes it), because a stalled dump's mtime is the stall and can be hours after the start; the file's mtime when the header has no usable stamp; and no time when the platform cannot convert the mtime. The stamp pattern (_OPENED_RE, crash_dump_store.py:742) accepts only digits, so no byte of a file reaches the printed row.
  • _doctor_last_boots (workload.py:492) prints the section: a boots: row, one wrapped line saying the newest five retained crash-dump files are shown, one per gateway start, and that they are the retained stalls and newest start within that slice and not a complete boot history, and one row per file. It is called once after the crash-dump rows (src/kiro_crew/cli_doctor.py:2302; last_boots is bound on the facade at :139). It is silent when no dump is retained, prints one warning row (without the error text) when the directory cannot be read, and takes no issues list, so it cannot add an issue.
  • Docs: doctor item 24 in docs/system-specs/modules/cli.md:1308.

Rejected. A separate boot ledger: a file under logs/ that every start appends to, with its own reader and a crash-loop verdict. The verdict counted the starts that followed a stall, which are the dumps with stacks the existing issue already counts, so it could not fire while that issue was silent. What a ledger would add beyond the dumps is a line for each start that never wedged, and no reader needs that count today; recording it is new persistence and belongs with the change that first reads it. It would also put another file in an agent-writable tree that needs its own bounded reads, link-safe appends and a rotation that does not follow a linked directory. Also rejected: labelling the newest row from the doctor's own gateway: probe (services._doctor_gateway_reachability), which asks whether something answers on the configured port, not whether this file's writer is alive, and runs in a later section.

Backwards compatibility

Compatible: doctor gains one informational section after Loop-stall Crash Dumps. Every existing row and the exit status are unchanged, and the section prints nothing when no dump is retained. _owner_alive keeps its call shape for existing callers: the new keyword is keyword-only and its default is the old behaviour.

Removes nothing: no key, row, path, record or validator is removed, renamed or tightened.

Tests

In short: every new test fails on main because the reader and the section do not exist, and passes here in three runs; each deliberate break of the reader, the section or the sweep-order test fails only its own cases; the doctor, crash-dump and neighbouring suites and the gates pass; and a real kirocrew doctor run over a scratch data home prints the section and the same closing issue line as main.

test/test_doctor_last_boots.py (24 tests, no sleeps, every dump on tmp_path with explicit UTC stamps and os.utime; the one liveness probe is either this test process's own pid or an injected is_pid_alive; size as a fact: 131 added source lines over three files, crash_dump_store.py 96 (and 3 removed), workload.py 31 and cli_doctor.py 4, plus 399 lines of tests and 1 line of docs):

  • Reader: the newest five dumps read as boots in order with the right state each, the sixth older dump left out, and the stalled count equal to dumps_with_stacks over the same directory; the newest header-only file is running when its header names this process with the start identity the process has now, not running when it names a pid no process holds, when an injected probe says the pid is gone, or when this process's recorded start identity differs from its current one, and unknown when the header names another PID domain, has no # PID: line, or names a pid the probe reports alive (this process's own included) with no start identity to compare; only the newest file is probed; a dump the bounded reader refuses (a directory at a dump name) is unknown, and the header-only dump before it stays clean so far because a later local boot exists; a header-only dump whose successor is stalled, or belongs to another PID domain, is unknown and not clean so far; the start time is the header stamp, then the mtime when the stamp is not a date or is missing; a time the platform cannot convert is no time; the header the real open_dump_file writes reads back as the start even with a day-old mtime, and as running, because its writer is the test process; no dump, or a directory that does not exist, is no boot.
  • The sweep: with the gateway's own order (sweep_stale_dumps, rotate_dumps, then open_dump_file) the header-only dump of a gone session is removed and the reader lists the new file as running and the earlier stall as stalled. This pins the premise of the heading line, that a start which never wedged is not listed.
  • Section: the exact heading, caveat line and five rows in order, with not running for a newest writer that is gone and running for one that is alive; an unreadable dump prints as unknown; silent with no dump; a reader that raises prints one warning row and none of its message; five stalled dumps raise the existing recent loop-stall crash dump issue through _doctor_crash_dumps, the section prints five stalled rows, issues is unchanged by it and the function has no parameters; _doctor calls the section after _doctor_crash_dumps and hands it nothing.

Red on main (the new file overlaid on a detached origin/main, f9031e841f, three runs): 24 failed each, with AttributeError: module 'kiro_crew.dashboard.crash_dump_store' has no attribute 'last_boots' (17 tests), AttributeError: module 'kiro_crew.doctor_checks.workload' has no attribute '_doctor_last_boots' (5), AttributeError: <module 'kiro_crew.cli_doctor'> has no attribute 'last_boots' (1) and AssertionError: assert ['_doctor_crash_dumps'] == ['_doctor_cra...r_last_boots'] (1).

Green here, on this commit with a clean working tree:

  • 24 passed at -n0 in three runs.
  • Deliberate breaks, each alone on a throwaway checkout and each failing only its own cases: the newest dump always running; an owner that cannot be probed read as running; the injected probe ignored; any later boot counted as a successor; an unreadable dump read as clean so far; the header stamp ignored; the five-row limit removed; stacks never detected; the section printing when empty; the caveat line dropped; and the sweep left out of the sweep-order test.
  • Several tests patch module attributes, so the new file ran first in one process at -n0, followed by test/test_taskrunner_atomic_persistence.py, test/test_taskrunner_execution_capture_async.py, test/test_unattended_slot_guardrails.py and test/test_wheel_apply.py: green.
  • All the test/test_cli_doctor*.py and test/test_doctor_*.py files plus test/test_crash_dump_store.py at -n0, three runs: 1065 passed each time. The files that name the dump store (test_cautious_boot.py, test_loop_stall_surfacing.py, test_cron_loop_stall_breaker.py, test_diag_threads.py, test_loop_watchdog.py, test_snapshot.py, test_stall_attribution.py) and test_transcript_derivation_seam.py: green.
  • The flake-pattern and pid-reader ratchets, and the gates for black, isort, flake8, mypy over src/kiro_crew/, docs-lint, comment history, brand name, agent-sdk boundary, sync-io-in-async, loop-bound locks, testpaths, cwd-relative reads, subprocess encoding, changelog, decisions, focus-cue, feature-map and harness-parity: exit 0.

Manual verification

Ran python -m kiro_crew doctor from this branch and from origin/main (f9031e841f), each with HOME and KIROCREW_HOME under a scratch directory and KIROCREW_PORT set to a free port, over one dump directory (a copy for each run): three dumps with stacks, one header-only dump written by hand, and the newest dump written by the real open_dump_file. The branch prints, directly after Loop-stall Crash Dumps:

Gateway Boots
  boots:       newest first, one per retained crash dump
               The newest five retained crash-dump files are shown, one per gateway start. They
               are the retained stalls and newest start within that slice, not a complete boot
               history.
    2026-10-16T15:01:36Z  not running
    2026-10-16T13:01:36Z  stalled
    2026-10-16T12:36:36Z  clean so far
    2026-10-16T12:16:36Z  stalled
    2026-10-16T11:31:36Z  stalled

The closing line is the same ❌ Fix these issues: list on both and ends with recent loop-stall crash dump (2h ago), and both exit 1. A diff of the two outputs differs only in the new block and in environment lines (timestamps, the scratch and checkout paths, the port, free space and inodes, the branch line). The newest row reads not running in the same report whose Connectivity section prints gateway: not running: that scratch dump's writer had exited. One row is an artifact of the staging: the clean so far row is a header-only file written by hand, which a gateway start would have swept.

Screenshots / video

N/A — no user-visible frontend surface changes. The doctor output is text and is shown above.

Related Issues

Refs #9559
Refs #14724
Refs #15077

These are context, not work this change claims. #9559 is a public report of a gateway that stalled after every relaunch while doctor read healthy. #14724 proposes an orderly-shutdown marker, which this change does not add. #15077 reconstructs a recurrence of stall dumps by listing dump file names by hand.

Pattern harvest

Not generalizable: this is a new diagnostic, not a fixed defect. It reads an agent-writable directory through the dump store's existing bounded readers and prints only digits it matched and a fixed set of words.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 13, 2026 21:03
@javenciu
javenciu requested a review from mbriones98 October 13, 2026 21:03
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening right after opening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 13, 2026
@javenciu javenciu reopened this Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision dc69f18dead77f32e00fb729f32ff01bc570339a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of dc69f18dead77f32e00fb729f32ff01bc570339a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check by hand that a live gateway's newest header-only dump prints running on macOS and Windows. Tests inject the liveness probe, so CI never exercises the real start-identity probe.

[DESIGN-REVIEWED] dc69f18

Description read

[DESCRIPTION-READ] 7eae5760aa6e05465e5578254c111cab960a01fe4982b68abd580d7c1e899f37

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of dc69f18dead77f32e00fb729f32ff01bc570339a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The writer-liveness label on the newest row adds an identity_required keyword to the owner test the sweep and rotation share. The Goal is how each start went, and stalled versus not answers that.

Not justified as shipped

  • Item 3 — oversized: the description gives Windows: new sessions missing all core/cron MCP tools — dashboard gateway loop-stalls on startup every relaunch #9559's harm as "no single view of the recent stalls and when each start began". A stall is read from the file alone. The running/not running/unknown label for the newest file needs a new mode on _owner_alive (identity_required, one caller at crash_dump_store.py:800). The sweep (:514) and the rotation (:570) both run through that function. The doctor already reports whether a gateway is up in its gateway: row.
    Subtraction: delete identity_required and its branch in _owner_alive. Label every header-only row by its file, the same way older rows are labelled.
    Clears when: the keyword is removed, or a reported case shows an operator needed this file's writer liveness that the gateway: row could not give.
  • Item 4 — oversized: three labels for a header-only file that is not the newest is more than the Goal needs. clean so far shows "only while the owner cannot be shown gone", which the description admits, and the alternative is unknown. Stalled versus not stalled is what removes the named harm.
    Subtraction: merge clean so far and unknown for header-only files into one "no stall" state, and drop the dump_superseded call in last_boots.
    Clears when: the states are merged, or a reader is named who acts differently on clean so far than on unknown.

What this change ships

Inventory (7 items) — 5 justified

Intent: let an operator see in kirocrew doctor how often the gateway wedged across recent starts, and when each start began. ADDITION.

  1. doctor gains a Gateway Boots section listing up to five retained dumps, newest first — justified
  2. each row shows the start time from the header's opened stamp, falling back to the file's mtime — justified
  3. the newest row says running / not running / unknown based on its writer's identity — oversized
  4. older stall-free rows read clean so far or unknown — oversized
  5. one wrapped line warns the list is not a complete boot history — justified
  6. one warning row when the dump directory cannot be read — justified
  7. the cli.md spec gains doctor item 24 — justified

[FIRST-PRINCIPLES-REVIEWED] dc69f18

Description read

[DESCRIPTION-READ] e28b1729fde5bb1f1a8a013f035d72139adc2bc492b6e8e6d4bb04ee4fffda56

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed dc69f18dead77f32e00fb729f32ff01bc570339a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] dc69f18

Verdict parsed from the review's SHA-scoped output markers for commit dc69f18dead77f32e00fb729f32ff01bc570339a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable dc69f18dead77f32e00fb729f32ff01bc570339a: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of dc69f18dead77f32e00fb729f32ff01bc570339a and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/dashboard/crash_dump_store.py:774 -- An overlapping live gateway’s header-only dump survives sweep and rotation and appears as clean so far, contradicting “not listed once a later one has begun” -> Fix: qualify that disappearance requires its owner to be confirmed dead and its dump swept.
[GPT-REVIEWED] dc69f18

False positive or not applicable? A repository writer can comment:
/ai-review override gpt dc69f18dead77f32e00fb729f32ff01bc570339a: <one-sentence reason>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 13, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Review round on 65c277ac8b. Every point below is fixed in f36c1c71fe, a second commit on the branch. Each new test fails on 65c277ac8b and passes on f36c1c71fe, three runs each at -n0.

  1. GPT, src/kiro_crew/recovery/boot_ledger.py:165: the append follows a link planted in logs/. Fixed. record_boot now appends through platform_log_append.append_line (boot_ledger.py:160), the confined append the decision log already uses. It opens logs/ relative to the data home with O_NOFOLLOW, opens the file with O_NOFOLLOW (on Windows, without following a reparse point), and refuses anything that is not a regular file with one name. A symlink or a hard link at the ledger name, or a symlink at logs/, now makes the append fail. record_boot returns None for that boot and writes nothing through the link. Tests: test_a_symlink_at_the_ledger_name_is_not_written_through, test_a_hard_link_at_the_ledger_name_is_not_written_through and test_a_symlinked_logs_directory_is_not_written_through (test/test_boot_ledger.py:322-350).

  2. GPT, boot_ledger.py:186: a ts of 10**400 raises OverflowError out of math.isfinite. Fixed. _parse checks 0 <= ts <= _TS_MAX before anything converts the value (boot_ledger.py:181). An int compares exactly at any size. NaN and the infinities fail the same comparison, so the isfinite call is gone. Tests: test_a_timestamp_too_large_for_a_float_costs_its_line_not_the_read (10**400, -(10**400), 1e400 and NaN each cost only their own line) and test_an_oversized_integer_stamp_does_not_stop_doctor, which drives _doctor_gateway_boots over the forged line.

  3. GPT, src/kiro_crew/dashboard/crash_dump_store.py:1085: an unreadable predecessor dump is recorded as clean. Fixed. prior_session_exit now reads each listed dump with the raising _read_dump_head before the ownership check (crash_dump_store.py:1088). A dump nobody can read may belong to the predecessor, so the failure reaches the existing fallback and the answer is unknown. A read failure never yields clean. Test: test_a_dump_whose_header_cannot_be_read_is_unknown_not_clean puts a directory at a dump name, which the header read refuses on every platform.

  4. Opus, src/kiro_crew/doctor_checks/gateway_boots.py:58: on Windows a ts past the year 3000 crashes doctor's row format. Fixed both ways the review named. _TS_MAX is 32503680000, 3000-01-01 (boot_ledger.py:99), so the read drops a line past it. The row also catches (OverflowError, OSError, ValueError) around the conversion and prints the raw number (gateway_boots.py:60). Tests: test_a_timestamp_past_the_year_3000_is_not_a_boot (the ceiling converts to 3000-01-01, the edge is kept, and 40000000000 is dropped) and test_a_stamp_the_platform_cannot_convert_prints_as_the_number (a record at 1e20 reaches the row and prints as 1e+20; on Linux the conversion raises OverflowError: timestamp out of range for platform time_t on the old head).

  5. First Principles: the gateway-restart skill line told the agent not to restart, which contradicts Not a goal. Fixed by making the line read-only. skills/gateway-restart/SKILL.md:80 now says to include the Gateway Boots rows in what the agent tells the user, and that the procedure below is the same whatever they say. The "do not restart" and "minutes apart" clauses are gone. I kept the line rather than dropping the hunk, because the reason in Why it matters is that the agent deciding on a restart has nothing to read. The spec (docs/system-specs/modules/session.md, Boot ledger) and docs/system-specs/modules/cli.md row 24 now say the same thing.

  6. First Principles item 7: the per-boot id had no consumer. Deleted. The line is {"prior_exit", "ts"}, BootRecord has those two fields, and the doctor row shows the time and the prior exit. The ledger is new in this PR, so no format is owed to anyone. Not a goal now says a boot id returns with its first correlating reader.

  7. First Principles item 8: crash_loop and recent(limit) took parameters nobody varies. Deleted crash_loop; the doctor compares unclean_boots with CRASH_LOOP_BOOTS itself (src/kiro_crew/doctor_checks/gateway_boots.py:43). recent takes no count and returns the newest RECENT_MAX boots (boot_ledger.py:210, :114), a constant, so what a read hands back stays bounded whatever a forged or runaway ledger holds. test_recent_never_returns_more_than_its_ceiling pins the bound and the signature.

The spec also states what the append refuses, the ts bound and the raw-number fallback.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=0aa45b823d1d

fixed f36c1c7: record_boot appends through platform_log_append.append_line (src/kiro_crew/recovery/boot_ledger.py:160). It opens logs/ and the file with O_NOFOLLOW (on Windows, without following a reparse point) and refuses anything but a regular file with one name, so a planted symlink or hard link fails the append and nothing is written through it.
test_a_symlink_at_the_ledger_name_is_not_written_through, test_a_hard_link_at_the_ledger_name_is_not_written_through and test_a_symlinked_logs_directory_is_not_written_through (test/test_boot_ledger.py:322-350) fail on 65c277ac8b, where the record is written through the link, and pass on f36c1c71fe, three runs each.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=0aa45b823d1d

fixed f36c1c7: _parse checks 0 <= ts <= _TS_MAX before anything converts the value (src/kiro_crew/recovery/boot_ledger.py:181). An int compares exactly at any size, and NaN and the infinities fail the same comparison, so the math.isfinite call is gone.
test_a_timestamp_too_large_for_a_float_costs_its_line_not_the_read (10**400, -(10**400), 1e400, NaN) and test_an_oversized_integer_stamp_does_not_stop_doctor fail on 65c277ac8b with OverflowError: int too large to convert to float and pass on f36c1c71fe, three runs each.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=d7c4e484d708

fixed f36c1c7: prior_session_exit reads each listed dump with the raising _read_dump_head before the ownership check (src/kiro_crew/dashboard/crash_dump_store.py:1088). A failure reaches the existing fallback and the answer is unknown; a read failure never yields clean.
test_a_dump_whose_header_cannot_be_read_is_unknown_not_clean puts a directory at a dump name, which the header read refuses on every platform. It fails on 65c277ac8b ('clean' == 'unknown') and passes on f36c1c71fe, three runs each.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=4727ae55bd64

fixed f36c1c7, both ways the review named. _TS_MAX is 32503680000, 3000-01-01 (src/kiro_crew/recovery/boot_ledger.py:99), so a line past it is dropped on read. The row also catches (OverflowError, OSError, ValueError) around the conversion and prints the raw number (src/kiro_crew/doctor_checks/gateway_boots.py:60).
test_a_timestamp_past_the_year_3000_is_not_a_boot (the ceiling converts to 3000-01-01, the edge is kept, 40000000000 is dropped) and test_a_stamp_the_platform_cannot_convert_prints_as_the_number (a 1e20 record prints as 1e+20; on 65c277ac8b the row raises OverflowError: timestamp out of range for platform time_t) fail on 65c277ac8b and pass on f36c1c71fe, three runs each.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=fed04d1f04fe

fixed f36c1c7: the instruction not to restart is removed, and so is the "minutes apart" clause. skills/gateway-restart/SKILL.md:80 now reads: include the Gateway Boots rows in what you tell the user; the rows are information only, and the procedure below is the same whatever they say.
I kept the line rather than dropping the hunk. Why it matters gives the reason for it: an agent deciding whether to restart has nothing to read. A read-only line meets that need and matches Not a goal ("Nothing restarts, pauses, backs off or blocks on it"). The spec (docs/system-specs/modules/session.md, Boot ledger), docs/system-specs/modules/cli.md row 24, the module docstring and the body's What changed now say the same.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=c0a9559a64df

fixed f36c1c7: the instruction not to restart is removed, and so is the "minutes apart" clause. skills/gateway-restart/SKILL.md:80 now reads: include the Gateway Boots rows in what you tell the user; the rows are information only, and the procedure below is the same whatever they say.
I kept the line rather than dropping the hunk. Why it matters gives the reason for it: an agent deciding whether to restart has nothing to read. A read-only line meets that need and matches Not a goal ("Nothing restarts, pauses, backs off or blocks on it"). The spec (docs/system-specs/modules/session.md, Boot ledger), docs/system-specs/modules/cli.md row 24, the module docstring and the body's What changed now say the same.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=6fe152636392

fixed f36c1c7: accepted the subtraction. The per-boot id is gone from the line ({"prior_exit", "ts"}), from BootRecord, from the doctor row (src/kiro_crew/doctor_checks/gateway_boots.py:63 prints the time and the prior exit) and from the spec. uuid is no longer imported and BOOT_ID_MAX_CHARS no longer exists. The ledger is new in this PR, so no format is owed.
Not a goal now carries the line: a per-boot id returns with its first correlating reader. test_a_boot_is_one_json_line_with_two_fields pins the line; test_a_value_read_off_disk_is_printed_inert pins that no byte of the file reaches the row.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=ed980a2e65e2

fixed f36c1c7: accepted the subtraction. crash_loop is deleted; the doctor compares unclean_boots with CRASH_LOOP_BOOTS itself (src/kiro_crew/doctor_checks/gateway_boots.py:43). recent takes no count (src/kiro_crew/recovery/boot_ledger.py:210) and returns the newest RECENT_MAX boots (:114).
The bound stays as a constant rather than a parameter, so what a read hands back stays bounded whatever a forged or runaway ledger holds; the bytes read are bounded separately by READ_MAX_BYTES. test_recent_never_returns_more_than_its_ceiling pins the bound and the signature; test_the_defaults_are_fifteen_minutes_and_five_boots pins the latch and the window.

@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 13, 2026
@javenciu javenciu reopened this Oct 13, 2026
@javenciu
javenciu force-pushed the feat/gateway-boot-ledger branch from f36c1c7 to fa9c2a6 Compare October 14, 2026 01:35
@javenciu javenciu changed the title feat(doctor): record gateway boots and report a crash-loop verdict feat(doctor): list the gateway boots the retained crash dumps record Oct 14, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author
  1. The First Principles BLOCK on f36c1c71fe is accepted. The boot ledger, its crash-loop verdict, prior_session_exit and every export without a consumer are removed, and the branch is one commit, fa9c2a6f1a. The description is rewritten to match, so its Goal and Not a goal changed with the scope: the verdict could never fire while the existing issue was silent (src/kiro_crew/doctor_checks/workload.py:333, added with fix(dashboard): report a loop-stall dump once, not every boot #10747), so it has no place here.
  2. What remains is the Gateway Boots section, read from the retained crash dumps through last_boots (src/kiro_crew/dashboard/crash_dump_store.py:717). It is information only: it takes no issues list and does not change doctor's exit status. Its tests fail on main and pass here, three runs each, and each deliberate break fails only its own cases.
  3. The Design and UX points on f36c1c71fe were about the window, the crash loop row and the prior exit values, none of which is printed any more. One correction to the earlier description: the dump directory is not a log of every restart, because the gateway removes the header-only file of a session that is gone before it opens its own (src/kiro_crew/dashboard/server_runtime/heartbeat.py:65). The rows are the files left after that sweep and the rotation, each one a gateway start, so the stalls on record and the newest start; the section says so under its heading, and a test runs the gateway's own sweep and open order. The newest header-only file is labelled running or not running by the owner test the sweep itself uses, and unknown when the owner cannot be probed from here.
  4. The GPT BLOCK on f36c1c71fe (src/kiro_crew/recovery/boot_ledger.py:156, rotation following a linked logs/ directory before the append refuses it) is answered by removal: the ledger, its rotation and every write under logs/ are gone, and the section only reads the dump files. On this head record_boot( matches nothing in src/, test/ or docs/, and the callers of rotate_jsonl_at( are exactly the ones main already has.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=2b41ff860c58

fixed fa9c2a6: accepted the subtraction. recovery/boot_ledger.py, prior_session_exit and PRIOR_EXIT_*, the ledger branch and ledger_base in record_healthy_boot, and doctor_checks/gateway_boots.py are not in the change, and the crash-loop verdict went with them. Doctor's exit status is untouched: src/kiro_crew/doctor_checks/workload.py:333 is still the one place a run of stalls turns it red.
The rate the review suggested reading from the stacked dumps is what remains. last_boots (src/kiro_crew/dashboard/crash_dump_store.py:717) lists the retained dumps with the start time from each header, and _doctor_last_boots (workload.py:385) prints them. It takes no issues list; test_five_stalled_dumps_raise_the_existing_issue_and_the_section_adds_none stages five stalled dumps and asserts the one existing issue and an unchanged issues list.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=9577db943dad

fixed fa9c2a6: accepted the subtraction. PRIOR_EXITS, READ_MAX_BYTES, RECORD_MAX_BYTES, BOOTS_FILE, boots_path, record_boot's return value, ledger_base and unclean_boots(window_s) all lived in the ledger, which is not in the change, so none of them exists.
The change adds one public function, last_boots (src/kiro_crew/dashboard/crash_dump_store.py:717), which src/kiro_crew/cli_doctor.py:138 binds and which _doctor_last_boots (src/kiro_crew/doctor_checks/workload.py:385) calls. _BOOTS_LISTED, _OPENED_RE and _boot_started are private to the module.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=4631e99c1ef6

fixed fa9c2a6: accepted. The ledger and its verdict are removed, so there is no second spelling of the repeated-stall issue: _stalls_on_record < 2 (src/kiro_crew/doctor_checks/workload.py:333) decides it alone.
The section that remains reads the dumps that issue already counts, and a stalled row is exactly a dump dumps_with_stacks counts (test_the_last_five_dumps_read_as_boots_newest_first asserts the two counts are equal over one directory). It cannot add an issue because _doctor_last_boots takes no issues parameter, and _doctor calls it with no arguments (test_doctor_runs_the_section_after_the_crash_dump_section_and_hands_it_nothing).

@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 16, 2026
@javenciu
javenciu enabled auto-merge (squash) October 16, 2026 11:41
@javenciu

Copy link
Copy Markdown
Contributor Author

span=602df76413b9

  • The skill change exists only in the description. fixed: the description no longer promises a skills/gateway-restart/SKILL.md line or names an agent following that skill as a reader. The bullet is gone from What changed, "Why it matters" names the operator only, and the test-size line no longer counts skill text. The diff changes five files and none is a skill, and skills/gateway-restart/SKILL.md has no doctor step at this head, as the last sentence of docs/system-specs/modules/cli.md:1270 says.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=2e1da17b5f04

  • The description says _owner_alive is reused, but the diff changes it. fixed: What changed now names the keyword and why cleanup keeps its default. _owner_alive gains a keyword-only identity_required, default False (src/kiro_crew/dashboard/crash_dump_store.py:404), and last_boots is the only caller that passes it (:800). With it set, the own-PID shortcut is off (:445) and a missing or incomparable start identity answers None (:451-457). The default path is the code main had, and test/test_crash_dump_store.py:1074 pins the answer the keyword changes. The sweep (:514) and the rotation (:570) keep the default: the rotation spares a file only on True, so a live gateway whose header has no comparable start identity would answer None under the keyword and could be removed. Backwards compatibility now says the keyword is keyword-only and defaults to the old behaviour.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=ce0d3cce587d

  • _doctor_last_boots prints the same caveat twice. kept: the lines overlap only in "one per ...". The boots: row says what a row is and the order (src/kiro_crew/doctor_checks/workload.py:513); the wrapped line adds the count and that the list is not a complete boot history (:514-517). The suggested line, "last 5 gateway starts on record ... (starts that never stalled are not kept)", would be wrong on two rows: the newest start is listed whether or not it stalled (running or not running, src/kiro_crew/dashboard/crash_dump_store.py:799-801), and clean so far is a start that never stalled (:802-803). The five rows are the newest retained files, not the last five starts (:795). Both lines are pinned by test/test_doctor_last_boots.py:35-41.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=b8308f279c13

  • On an older row, clean so far reads as if that start might still be running. kept: it might be. The reader probes only the newest file's writer (src/kiro_crew/dashboard/crash_dump_store.py:799-800), so for an older row it knows the dump has no stacks and a later local boot followed, not that the start has ended; the sweep that leaves the file in place removes it only once its owner is gone (:514). exited cleanly would claim an exit the reader never observes, and the description lists shutdown evidence as a non-goal (crash_dump_store: header-only loop-stall dumps are still classified as clean exits, so a non-watchdog gateway death leaves no surfaced stacks #14724 proposes the marker). no stall drops the "so far" that says the file can still gain stacks. running and not running stay distinct because they label only the newest file (:799-801). On the summary's ask for a gloss on unknown and clean so far: both words are defined in docs/system-specs/modules/cli.md:1270 and in last_boots (:778-788). A legend under the heading would lengthen the lines the first suggestion asks to shorten, so the printed section keeps one word per row.

@javenciu

Copy link
Copy Markdown
Contributor Author
  • The description says gateway-restart/SKILL.md gains a line, but the diff changes no skill. fixed: the description no longer claims a skill change or an agent reader. The SKILL.md bullet is removed from What changed and "Why it matters" names the operator who reads kirocrew doctor. The diff changes five files and none is a skill. The _owner_alive change that this review's inventory lists as undeclared (item 4) is declared in What changed now, with the reason cleanup keeps the default.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 16, 2026
@javenciu
javenciu force-pushed the feat/gateway-boot-ledger branch from 9ae4b6e to dc69f18 Compare October 17, 2026 00:38
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto current main as one commit. The code diff is unchanged, and the description's line numbers for src/kiro_crew/cli_doctor.py and docs/system-specs/modules/cli.md are updated to where main moved them.

The GPT 6.1 finding at website/src/components/chat/crewmateBubbles.ts:126 is on a line that came from main (#21142); this PR changes no file under website/.

It cites docs/decisions/2026-10-09-crewmate-chat-draws-messages-only.md as standing, but main now has docs/decisions/2026-10-16-crewmate-chat-draws-the-spawn-run-card.md (#21173), which supersedes it and was not in the base that review used.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 17, 2026
auto-merge was automatically disabled October 17, 2026 01:26

Pull request was closed

@javenciu javenciu reopened this Oct 17, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@javenciu
javenciu enabled auto-merge (squash) October 17, 2026 08:56
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at dc69f18. The PID-reuse fix we asked for is in: with identity_required, _owner_alive skips the own-PID shortcut and compares start identities (L451). The default path that sweep and rotation use is unchanged. These findings on this head have no reply:

  • GPT 6.1: the docstring at L774 says a start that never wedged is not listed once a later one begins, but an overlapping live gateway's file is listed as clean so far. Rewording the sentence would clear it.
  • First Principles items 3 (identity_required) and 4 (merge clean so far/unknown): post a head=dc69f18 disposition for each. The earlier Design/UX replies do not answer these subtractions.
  • Design punchline: say whether running was checked by hand on macOS and Windows.
    CI is green; Sensitive Change Review is still queued. Our older change request can be dismissed once these are answered. I did not run tests or start processes. — crew-pr-reviewer (r109)

@javenciu
javenciu disabled auto-merge October 18, 2026 05:31
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing without merging, to keep my open PRs to the ones close to landing.

The Oct 17 hold on this head (dc69f18) needs the docstring at crash_dump_store.py L774 reworded, since an overlapping live gateway's file is listed as clean so far. It also needs replies to First Principles items 3 and 4, and to the Design question about checking running by hand on macOS and Windows.

last_boots isn't on main as of 2e50054. I've deleted the branch; it can be restored from this page at dc69f18.

@javenciu javenciu closed this Oct 18, 2026
@javenciu
javenciu deleted the feat/gateway-boot-ledger branch October 18, 2026 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants