Repository navigation
Conversation
|
Closing and reopening right after opening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check by hand that a live gateway's newest header-only dump prints [DESIGN-REVIEWED] dc69f18 Description read[DESCRIPTION-READ] 7eae5760aa6e05465e5578254c111cab960a01fe4982b68abd580d7c1e899f37 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The writer-liveness label on the newest row adds an Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: let an operator see in
[FIRST-PRINCIPLES-REVIEWED] dc69f18 Description read[DESCRIPTION-READ] e28b1729fde5bb1f1a8a013f035d72139adc2bc492b6e8e6d4bb04ee4fffda56 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/dashboard/crash_dump_store.py:774 -- An overlapping live gateway’s header-only dump survives sweep and rotation and appears as False positive or not applicable? A repository writer can comment: |
|
Review round on
The spec also states what the append refuses, the |
|
span=0aa45b823d1d
|
|
span=0aa45b823d1d
|
|
span=d7c4e484d708
|
|
span=4727ae55bd64
|
|
span=fed04d1f04fe
|
|
span=c0a9559a64df
|
|
span=6fe152636392
|
|
span=ed980a2e65e2
|
|
Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
f36c1c7 to
fa9c2a6
Compare
|
|
span=2b41ff860c58
|
|
span=9577db943dad
|
|
span=4631e99c1ef6
|
|
span=602df76413b9
|
|
span=2e1da17b5f04
|
|
span=ce0d3cce587d
|
|
span=b8308f279c13
|
|
9ae4b6e to
dc69f18
Compare
|
Rebased onto current The GPT 6.1 finding at It cites |
|
Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
Pull request was closed
|
HOLD at dc69f18. The PID-reuse fix we asked for is in: with
|
|
Closing without merging, to keep my open PRs to the ones close to landing. The Oct 17 hold on this head (dc69f18) needs the docstring at
|
Problem / Motivation
Goal:
kirocrew doctorlists the retained crash-dump files as gateway starts, with when each began and how it went, without a new record and without changing when doctor reports a problem.Every gateway start pre-creates a crash-dump file so faulthandler has a stable descriptor (
open_dump_file,src/kiro_crew/dashboard/crash_dump_store.py:605), and a file with thread stacks is a start that wedged. The directory holds what is left after two steps the gateway runs before it opens its own file (src/kiro_crew/dashboard/server_runtime/heartbeat.py:65to:67): the sweep removes the header-only file of a session that is gone (sweep_stale_dumps,crash_dump_store.py:482), and rotation keeps at most ten files and removes header-only ones first (rotate_dumps,:536;_DEFAULT_MAX_DUMPS,:56). So the retained files are the stalls on record plus the newest start, each one a gateway start, and not a complete boot history. Doctor does not read the directory that way. TheLoop-stall Crash Dumpssection (_doctor_crash_dumps,src/kiro_crew/doctor_checks/workload.py:406) reports the newest dump with stacks, how long ago it was and whether a later session superseded it. To see how often the gateway wedged and came back, and when each start began, an operator lists the dump directory by hand.Why it matters
#9559 reports a gateway that stalled again after every relaunch while
kirocrew doctorreported the backend healthy. An operator who is deciding whether to restart once more has no single view of the recent stalls and when each start began. The data is on disk; it is read one file at a time, by file name (#15077 reconstructs a recurrence that way).Not a goal
recent loop-stall crash dumpissue (workload.py:480) stays current while two or more retained dumps hold stacks (_stalls_on_record < 2,workload.py:440, added with fix(dashboard): report a loop-stall dump once, not every boot #10747), so a run of recent stalls already turns doctor red there and a count of stalls here would only restate it. The new section takes no issues list.not runningsays the writer of the newest file is gone, not how it ended;runningsays it is alive, not that it is serving.What changed (motivation → approach → change)
Goal → approach. The dump directory already holds each stall and the newest boot, so the change is one reader over it, one keyword on the owner test it needs, and one doctor section that prints what the reader returns.
last_boots(crash_dump_store.py:764) returns the newest_BOOTS_LISTED(5,:739) retained dumps as(started, state)pairs, newest first. It reuses the module's own readers:_list_dumps(:151) for the files,_is_header_only(:241) for the stacks,dump_superseded(:691) for the successor test and_owner_alive(:400), the owner test the sweep (:514) and the rotation (:570) use, for the newest file's writer. That last one is the only reader this change edits:_owner_alivegains a keyword-onlyidentity_required(:404, defaultFalse), andlast_bootsis the only caller that passes it (:800). With it set, the own-PID shortcut is off (:445), and a header with no start identity, or one that cannot be compared with the live process's, answersNoneinstead of falling back to plain PID liveness (:451to:457);last_bootsreadsNoneasunknown. The sweep and the rotation keep the default, which is the codemainhad;test_owner_alive_without_recorded_start_id_trusts_liveness(test/test_crash_dump_store.py:1074) pins the answer the keyword changes. The rotation needs the default: it spares a file only when the answer isTrue(:570), so a live gateway whose header has no comparable start identity would answerNoneunder the keyword and could be removed. The states are:stalled: the dump holds stacks, the testdumps_with_stacks(:673) counts, so the number ofstalledrows is read the same way as the repeated-stall issue.running/not running: the newest file, no stacks, and its writer is alive with the recorded start identity / gone._owner_alivematches the header's pid, PID domain and start id against the live process, so a recycled pid does not read as the owner, and takes the probe as a parameter (is_pid_alive, defaultplatform_compat.pid_exists,src/kiro_crew/platform_compat.py:6438), which the tests inject.not runningis a start that exited without wedging and has not been swept yet; the next start sweeps it.clean so far: an older file with no stacks whose successor, perdump_superseded, is a later local boot that did not wedge. The sweep removes such a file once its owner is gone, so one shows only while the owner cannot be shown gone.unknown: the dump cannot be read; the newest file's owner cannot be probed from here (no# PID:line, another PID domain, or a start identity that is missing or cannot be compared with the live process's); or an older file with no stacks and no such successor. A file the reader cannot open isunknown, neverclean so farorrunning._boot_started(crash_dump_store.py:745) gives the start time: theopenedstamp in the header (open_dump_filewrites it), because a stalled dump's mtime is the stall and can be hours after the start; the file's mtime when the header has no usable stamp; and no time when the platform cannot convert the mtime. The stamp pattern (_OPENED_RE,crash_dump_store.py:742) accepts only digits, so no byte of a file reaches the printed row._doctor_last_boots(workload.py:492) prints the section: aboots:row, one wrapped line saying the newest five retained crash-dump files are shown, one per gateway start, and that they are the retained stalls and newest start within that slice and not a complete boot history, and one row per file. It is called once after the crash-dump rows (src/kiro_crew/cli_doctor.py:2302;last_bootsis bound on the facade at:139). It is silent when no dump is retained, prints one warning row (without the error text) when the directory cannot be read, and takes no issues list, so it cannot add an issue.docs/system-specs/modules/cli.md:1308.Rejected. A separate boot ledger: a file under
logs/that every start appends to, with its own reader and a crash-loop verdict. The verdict counted the starts that followed a stall, which are the dumps with stacks the existing issue already counts, so it could not fire while that issue was silent. What a ledger would add beyond the dumps is a line for each start that never wedged, and no reader needs that count today; recording it is new persistence and belongs with the change that first reads it. It would also put another file in an agent-writable tree that needs its own bounded reads, link-safe appends and a rotation that does not follow a linked directory. Also rejected: labelling the newest row from the doctor's owngateway:probe (services._doctor_gateway_reachability), which asks whether something answers on the configured port, not whether this file's writer is alive, and runs in a later section.Backwards compatibility
Compatible: doctor gains one informational section after
Loop-stall Crash Dumps. Every existing row and the exit status are unchanged, and the section prints nothing when no dump is retained._owner_alivekeeps its call shape for existing callers: the new keyword is keyword-only and its default is the old behaviour.Removes nothing: no key, row, path, record or validator is removed, renamed or tightened.
Tests
In short: every new test fails on
mainbecause the reader and the section do not exist, and passes here in three runs; each deliberate break of the reader, the section or the sweep-order test fails only its own cases; the doctor, crash-dump and neighbouring suites and the gates pass; and a realkirocrew doctorrun over a scratch data home prints the section and the same closing issue line asmain.test/test_doctor_last_boots.py(24 tests, no sleeps, every dump ontmp_pathwith explicit UTC stamps andos.utime; the one liveness probe is either this test process's own pid or an injectedis_pid_alive; size as a fact: 131 added source lines over three files,crash_dump_store.py96 (and 3 removed),workload.py31 andcli_doctor.py4, plus 399 lines of tests and 1 line of docs):stalledcount equal todumps_with_stacksover the same directory; the newest header-only file isrunningwhen its header names this process with the start identity the process has now,not runningwhen it names a pid no process holds, when an injected probe says the pid is gone, or when this process's recorded start identity differs from its current one, andunknownwhen the header names another PID domain, has no# PID:line, or names a pid the probe reports alive (this process's own included) with no start identity to compare; only the newest file is probed; a dump the bounded reader refuses (a directory at a dump name) isunknown, and the header-only dump before it staysclean so farbecause a later local boot exists; a header-only dump whose successor is stalled, or belongs to another PID domain, isunknownand notclean so far; the start time is the header stamp, then the mtime when the stamp is not a date or is missing; a time the platform cannot convert is no time; the header the realopen_dump_filewrites reads back as the start even with a day-old mtime, and asrunning, because its writer is the test process; no dump, or a directory that does not exist, is no boot.sweep_stale_dumps,rotate_dumps, thenopen_dump_file) the header-only dump of a gone session is removed and the reader lists the new file asrunningand the earlier stall asstalled. This pins the premise of the heading line, that a start which never wedged is not listed.not runningfor a newest writer that is gone andrunningfor one that is alive; an unreadable dump prints asunknown; silent with no dump; a reader that raises prints one warning row and none of its message; five stalled dumps raise the existingrecent loop-stall crash dumpissue through_doctor_crash_dumps, the section prints fivestalledrows,issuesis unchanged by it and the function has no parameters;_doctorcalls the section after_doctor_crash_dumpsand hands it nothing.Red on
main(the new file overlaid on a detachedorigin/main,f9031e841f, three runs):24 failedeach, withAttributeError: module 'kiro_crew.dashboard.crash_dump_store' has no attribute 'last_boots'(17 tests),AttributeError: module 'kiro_crew.doctor_checks.workload' has no attribute '_doctor_last_boots'(5),AttributeError: <module 'kiro_crew.cli_doctor'> has no attribute 'last_boots'(1) andAssertionError: assert ['_doctor_crash_dumps'] == ['_doctor_cra...r_last_boots'](1).Green here, on this commit with a clean working tree:
24 passedat-n0in three runs.running; an owner that cannot be probed read asrunning; the injected probe ignored; any later boot counted as a successor; an unreadable dump read asclean so far; the header stamp ignored; the five-row limit removed; stacks never detected; the section printing when empty; the caveat line dropped; and the sweep left out of the sweep-order test.-n0, followed bytest/test_taskrunner_atomic_persistence.py,test/test_taskrunner_execution_capture_async.py,test/test_unattended_slot_guardrails.pyandtest/test_wheel_apply.py: green.test/test_cli_doctor*.pyandtest/test_doctor_*.pyfiles plustest/test_crash_dump_store.pyat-n0, three runs:1065 passedeach time. The files that name the dump store (test_cautious_boot.py,test_loop_stall_surfacing.py,test_cron_loop_stall_breaker.py,test_diag_threads.py,test_loop_watchdog.py,test_snapshot.py,test_stall_attribution.py) andtest_transcript_derivation_seam.py: green.src/kiro_crew/, docs-lint, comment history, brand name, agent-sdk boundary, sync-io-in-async, loop-bound locks, testpaths, cwd-relative reads, subprocess encoding, changelog, decisions, focus-cue, feature-map and harness-parity: exit 0.Manual verification
Ran
python -m kiro_crew doctorfrom this branch and fromorigin/main(f9031e841f), each withHOMEandKIROCREW_HOMEunder a scratch directory andKIROCREW_PORTset to a free port, over one dump directory (a copy for each run): three dumps with stacks, one header-only dump written by hand, and the newest dump written by the realopen_dump_file. The branch prints, directly afterLoop-stall Crash Dumps:The closing line is the same
❌ Fix these issues:list on both and ends withrecent loop-stall crash dump (2h ago), and both exit 1. A diff of the two outputs differs only in the new block and in environment lines (timestamps, the scratch and checkout paths, the port, free space and inodes, the branch line). The newest row readsnot runningin the same report whoseConnectivitysection printsgateway: not running: that scratch dump's writer had exited. One row is an artifact of the staging: theclean so farrow is a header-only file written by hand, which a gateway start would have swept.Screenshots / video
N/A — no user-visible frontend surface changes. The doctor output is text and is shown above.
Related Issues
Refs #9559
Refs #14724
Refs #15077
These are context, not work this change claims. #9559 is a public report of a gateway that stalled after every relaunch while doctor read healthy. #14724 proposes an orderly-shutdown marker, which this change does not add. #15077 reconstructs a recurrence of stall dumps by listing dump file names by hand.
Pattern harvest
Not generalizable: this is a new diagnostic, not a fixed defect. It reads an agent-writable directory through the dump store's existing bounded readers and prints only digits it matched and a fixed set of words.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.