Repository navigation
fix: stop core dumps from delaying a crashed gateway's restart - #31607
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each unit's line goes after the cause the description names: no unit this repo writes sets What this change shipsInventory (6 items) — 6 justifiedIntent: A Linux operator whose gateway crashes gets it back within seconds instead of after about five minutes. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 2350793 Description read[DESCRIPTION-READ] f9503d2cf0506333379e4ddb02402e6d9edd78d2d3843b8ba2d5385f3a4c955c That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the fix removes the root cause, an inherited unlimited core limit, from every unit this repo writes, and the existing required-directive check re-renders installed pod units without new code. Notes
[DESIGN-REVIEWED] 2350793 Description read[DESCRIPTION-READ] cce76ba44804db4f6303e2a206c1341b249308c621586351cacf175345509592 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
5d0a548 to
8d68560
Compare
|
[operator: bolichen97] The shard-2 red on the previous head ( |
Every gateway unit this repo writes left RLIMIT_CORE at systemd's default, which is unlimited wherever core_pattern pipes to systemd-coredump. A fatal signal then streams the whole multi-GB gateway into systemd-coredump, the process cannot finish exiting until that is done, and systemd-coredump stops at its 5-minute runtime cap, so the dashboard and sessions stay down for about five minutes. Set LimitCORE=0 in the unit `kirocrew service install` renders (user and system scope), the pod template unit (and require it, so pod up re-renders an installed pod unit without it), the EC2 bootstrap unit and the two documented hand-written units. The faulthandler stack dump stays the crash record; processes the gateway starts inherit the limit, and a drop-in restores core files. Closes #26092
8d68560 to
2350793
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved for the on-call. Review lanes pass on 2350793, no conflict, no open change request.
Every gateway unit this repo writes left RLIMIT_CORE at systemd's default, which is unlimited wherever core_pattern pipes to systemd-coredump. A fatal signal then streams the whole multi-GB gateway into systemd-coredump, the process cannot finish exiting until that is done, and systemd-coredump stops at its 5-minute runtime cap, so the dashboard and sessions stay down for about five minutes. Set LimitCORE=0 in the unit `kirocrew service install` renders (user and system scope), the pod template unit (and require it, so pod up re-renders an installed pod unit without it), the EC2 bootstrap unit and the two documented hand-written units. The faulthandler stack dump stays the crash record; processes the gateway starts inherit the limit, and a drop-in restores core files. Closes #26092 (cherry picked from commit c40e8b7) Original author: Bolin Chen (#31607). test/test_pod_systemd.py docstring hand-resolved: kept main's docstring, which names the LimitCORE check.
Problem / Motivation
Goal: After a fatal signal, a gateway run by a systemd unit exits and is restarted within seconds instead of about five minutes.
When the gateway dies from a fatal signal (a
SIGBUShere), the dashboard and every session stay down for about five minutes before systemd restarts it. The unit sets no core limit, so the gateway runs with core dumps unlimited. On hosts whosecore_patternpipes tosystemd-coredump(Amazon Linux 2023, Fedora, Arch), the dying multi-GB process cannot finish exiting until its core has streamed, andsystemd-coredumpstops at its own 5-minute runtime cap. The reporter measured 305 s with cores on and 4 s with them off.Why it matters
Every Linux operator whose distro pipes cores to
systemd-coredumploses the dashboard, Slack and running sessions for about five minutes on each crash, on top of the crash itself. The core is not even kept:systemd-coredumpis stopped before it records it.Not a goal
kirocrew service install(orpod upfor a pod) re-renders it.LimitCORE=infinityin a drop-in.What changed (motivation → approach → change)
A crashed gateway now exits at once, so systemd restarts it after its usual 10 seconds.
Root cause: no unit this repo writes sets
LimitCORE, so the gateway inherits systemd's default, which is unlimited whereversystemd-coredumphandles cores.Every gateway unit now sets
LimitCORE=0under[Service]. The faulthandler stack dump stays the crash record, and the limit also applies to the processes the gateway starts.kirocrew service install, system and user scopeservice/linux_unit.pypod/unit.py, plusLimitCORE=in_REQUIRED_DIRECTIVESsopod upre-renders an installed pod unit that lacks itkirocrew cloudEC2 bootstrap unitcloud/templates/kirocrew-ec2.yamldocs/guides/assets/kirocrew.service,docs/guides/remote-and-mobile.mdLimitCORE=0rather than0:infinity: it only lowers a limit, so it can never raise a hard limit an administrator set host-wide, and it parses on every supported systemd (thesoft:hardform needs systemd 229). The owning specs (docs/system-specs/modules/cli.md§ Service Management,docs/system-specs/modules/dev-fleet.md§ Pod Unit Self-Heal) say what the limit does, that child processes inherit it, and how to opt back in.Backwards compatibility
Compatible: one resource limit is added to each unit; nothing that loads or starts today stops doing so, and
LimitCORE=0is valid on every systemd the units support.Removes nothing: no key, path, record or public function changes. The gateway and the processes it starts no longer write core files, which nothing in Kiro Crew reads; an operator restores them with a
LimitCORE=infinitydrop-in.Tests
test_service.py::TestLinuxUnitRendering::test_render_unit_turns_core_dumps_off[False|True]: the rendered unit carriesLimitCORE=0inside[Service]for both scopes.test_pod_systemd.py::...::test_a_unit_missing_a_required_directive_is_stale[LimitCORE=]: an installed pod unit withoutLimitCORE=is stale, sopod upre-renders it (the existingRestartPreventExitStatus=case is kept as the other parameter).test_the_rendered_template_satisfies_its_own_required_directivesholds the template to it.test_cloud_ec2.py::TestTemplate::test_gateway_unit_turns_core_dumps_off: the EC2 bootstrap's gateway unit carriesLimitCORE=0in[Service].All are pure text checks on rendered units: no clock, no host state, no ordering.
Proof: PROVEN test/test_cloud_ec2.py::TestTemplate::test_gateway_unit_turns_core_dumps_off, test/test_pod_systemd.py::TestTheUnitFileNeverOutlivesAFailedLoad::test_a_unit_missing_a_required_directive_is_stale[LimitCORE=], test/test_service.py::TestLinuxUnitRendering::test_render_unit_turns_core_dumps_off[False], test/test_service.py::TestLinuxUnitRendering::test_render_unit_turns_core_dumps_off[True] (head 8d68560)
Manual verification
On an Amazon Linux 2023 host with systemd 252 and
core_patternpiped tosystemd-coredump, a transient user service run with-p LimitCORE=0that filled 512 MB and sent itselfSIGBUSexited 2 s after start;systemd-coredumplogged "terminated abnormally without generating a coredump".Screenshots / video
N/A: no user-visible frontend change.
Related Issues
Closes #26092
Sibling sweep
Swept: git grep -n "[Service]" -- src/ docs/
Sibling: src/kiro_crew/pod/unit.py -- changed here
Sibling: src/kiro_crew/cloud/templates/kirocrew-ec2.yaml -- changed here
Sibling: docs/guides/assets/kirocrew.service -- changed here
Sibling: docs/guides/remote-and-mobile.md -- changed here
Sibling: src/kiro_crew/apps/builtins/dev_fleet/live_service.py -- not affected -- a drop-in on the live unit, which carries the limit
Pattern harvest
Not generalizable: one systemd directive missing from hand-written unit texts; there is no code shape a lint rule could match.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)