Skip to content

fix: stop core dumps from delaying a crashed gateway's restart - #31607

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/26092-unit-limit-core
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/26092-unit-limit-core

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: After a fatal signal, a gateway run by a systemd unit exits and is restarted within seconds instead of about five minutes.

When the gateway dies from a fatal signal (a SIGBUS here), the dashboard and every session stay down for about five minutes before systemd restarts it. The unit sets no core limit, so the gateway runs with core dumps unlimited. On hosts whose core_pattern pipes to systemd-coredump (Amazon Linux 2023, Fedora, Arch), the dying multi-GB process cannot finish exiting until its core has streamed, and systemd-coredump stops at its own 5-minute runtime cap. The reporter measured 305 s with cores on and 4 s with them off.

Why it matters

Every Linux operator whose distro pipes cores to systemd-coredump loses the dashboard, Slack and running sessions for about five minutes on each crash, on top of the crash itself. The core is not even kept: systemd-coredump is stopped before it records it.

Not a goal

  • Finding why the gateway crashed (that is the SIGBUS report this one links to).
  • Re-rendering units that are already installed: as today, a unit changes when kirocrew service install (or pod up for a pod) re-renders it.
  • A setting to keep core files: an operator who wants them sets LimitCORE=infinity in a drop-in.

What changed (motivation → approach → change)

A crashed gateway now exits at once, so systemd restarts it after its usual 10 seconds.

Root cause: no unit this repo writes sets LimitCORE, so the gateway inherits systemd's default, which is unlimited wherever systemd-coredump handles cores.

Every gateway unit now sets LimitCORE=0 under [Service]. The faulthandler stack dump stays the crash record, and the limit also applies to the processes the gateway starts.

Unit Where
kirocrew service install, system and user scope service/linux_unit.py
Dev Fleet pod template pod/unit.py, plus LimitCORE= in _REQUIRED_DIRECTIVES so pod up re-renders an installed pod unit that lacks it
kirocrew cloud EC2 bootstrap unit cloud/templates/kirocrew-ec2.yaml
The documented user unit and hand-written system unit docs/guides/assets/kirocrew.service, docs/guides/remote-and-mobile.md

LimitCORE=0 rather than 0:infinity: it only lowers a limit, so it can never raise a hard limit an administrator set host-wide, and it parses on every supported systemd (the soft:hard form needs systemd 229). The owning specs (docs/system-specs/modules/cli.md § Service Management, docs/system-specs/modules/dev-fleet.md § Pod Unit Self-Heal) say what the limit does, that child processes inherit it, and how to opt back in.

Backwards compatibility

Compatible: one resource limit is added to each unit; nothing that loads or starts today stops doing so, and LimitCORE=0 is valid on every systemd the units support.

Removes nothing: no key, path, record or public function changes. The gateway and the processes it starts no longer write core files, which nothing in Kiro Crew reads; an operator restores them with a LimitCORE=infinity drop-in.

Tests

  • test_service.py::TestLinuxUnitRendering::test_render_unit_turns_core_dumps_off[False|True]: the rendered unit carries LimitCORE=0 inside [Service] for both scopes.
  • test_pod_systemd.py::...::test_a_unit_missing_a_required_directive_is_stale[LimitCORE=]: an installed pod unit without LimitCORE= is stale, so pod up re-renders it (the existing RestartPreventExitStatus= case is kept as the other parameter). test_the_rendered_template_satisfies_its_own_required_directives holds the template to it.
  • test_cloud_ec2.py::TestTemplate::test_gateway_unit_turns_core_dumps_off: the EC2 bootstrap's gateway unit carries LimitCORE=0 in [Service].

All are pure text checks on rendered units: no clock, no host state, no ordering.

Proof: PROVEN test/test_cloud_ec2.py::TestTemplate::test_gateway_unit_turns_core_dumps_off, test/test_pod_systemd.py::TestTheUnitFileNeverOutlivesAFailedLoad::test_a_unit_missing_a_required_directive_is_stale[LimitCORE=], test/test_service.py::TestLinuxUnitRendering::test_render_unit_turns_core_dumps_off[False], test/test_service.py::TestLinuxUnitRendering::test_render_unit_turns_core_dumps_off[True] (head 8d68560)

Manual verification

On an Amazon Linux 2023 host with systemd 252 and core_pattern piped to systemd-coredump, a transient user service run with -p LimitCORE=0 that filled 512 MB and sent itself SIGBUS exited 2 s after start; systemd-coredump logged "terminated abnormally without generating a coredump".

Screenshots / video

N/A: no user-visible frontend change.

Related Issues

Closes #26092

Sibling sweep

Swept: git grep -n "[Service]" -- src/ docs/
Sibling: src/kiro_crew/pod/unit.py -- changed here
Sibling: src/kiro_crew/cloud/templates/kirocrew-ec2.yaml -- changed here
Sibling: docs/guides/assets/kirocrew.service -- changed here
Sibling: docs/guides/remote-and-mobile.md -- changed here
Sibling: src/kiro_crew/apps/builtins/dev_fleet/live_service.py -- not affected -- a drop-in on the live unit, which carries the limit

Pattern harvest

Not generalizable: one systemd directive missing from hand-written unit texts; there is no code shape a lint rule could match.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 17:52
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 17:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 18:13
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 23507939b45fd2c790265e4a7ca975009eb93760 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 23507939b45fd2c790265e4a7ca975009eb93760 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each unit's line goes after the cause the description names: no unit this repo writes sets LimitCORE. The defect cites #26092 and a measured reproduction.

What this change ships

Inventory (6 items) — 6 justified

Intent: A Linux operator whose gateway crashes gets it back within seconds instead of after about five minutes. This is a FIX.

  1. A gateway installed with kirocrew service install, in system or user scope, no longer writes core files, so a crash restarts it promptly — justified
  2. A Dev Fleet pod gateway no longer writes core files — justified
  3. On pod up, an installed pod unit without LimitCORE= is treated as stale and re-rendered — justified
  4. The gateway on a kirocrew cloud EC2 instance no longer writes core files — justified
  5. The documented user unit and the hand-written system unit in the guides carry LimitCORE=0 — justified
  6. The specs and the guide say how to get core files back with a LimitCORE=infinity drop-in — justified

[FIRST-PRINCIPLES-REVIEWED] 2350793

Description read

[DESCRIPTION-READ] f9503d2cf0506333379e4ddb02402e6d9edd78d2d3843b8ba2d5385f3a4c955c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 23507939b45fd2c790265e4a7ca975009eb93760 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix removes the root cause, an inherited unlimited core limit, from every unit this repo writes, and the existing required-directive check re-renders installed pod units without new code.

Notes

  • An installed kirocrew service unit only picks up the limit when kirocrew service install runs again, unlike pod units. Users who never reinstall keep waiting about five minutes on each crash. A line in the release notes telling them to re-run install would close that.

[DESIGN-REVIEWED] 2350793

Description read

[DESCRIPTION-READ] cce76ba44804db4f6303e2a206c1341b249308c621586351cacf175345509592

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 23507939b45fd2c790265e4a7ca975009eb93760 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2350793

Verdict parsed from the review's SHA-scoped output markers for commit 23507939b45fd2c790265e4a7ca975009eb93760.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 23507939b45fd2c790265e4a7ca975009eb93760 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 23507939b45fd2c790265e4a7ca975009eb93760 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2350793

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 23507939b45fd2c790265e4a7ca975009eb93760 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/26092-unit-limit-core branch from 5d0a548 to 8d68560 Compare November 6, 2026 20:38
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The shard-2 red on the previous head (test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget) was main's, not this change's: this PR does not touch that package, and #31625 raised the budget on main. Rebased onto it (5d0a548 -> 8d68560, diff unchanged); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026
Every gateway unit this repo writes left RLIMIT_CORE at systemd's
default, which is unlimited wherever core_pattern pipes to
systemd-coredump. A fatal signal then streams the whole multi-GB
gateway into systemd-coredump, the process cannot finish exiting until
that is done, and systemd-coredump stops at its 5-minute runtime cap,
so the dashboard and sessions stay down for about five minutes.

Set LimitCORE=0 in the unit `kirocrew service install` renders (user
and system scope), the pod template unit (and require it, so pod up
re-renders an installed pod unit without it), the EC2 bootstrap unit
and the two documented hand-written units. The faulthandler stack
dump stays the crash record; processes the gateway starts inherit
the limit, and a drop-in restores core files.

Closes #26092
@bolichen97
bolichen97 force-pushed the fix/26092-unit-limit-core branch from 8d68560 to 2350793 Compare November 7, 2026 02:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 2350793, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit c40e8b7 into main Nov 7, 2026
97 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/26092-unit-limit-core branch November 7, 2026 03:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
buluoray added a commit that referenced this pull request Nov 7, 2026
Every gateway unit this repo writes left RLIMIT_CORE at systemd's
default, which is unlimited wherever core_pattern pipes to
systemd-coredump. A fatal signal then streams the whole multi-GB
gateway into systemd-coredump, the process cannot finish exiting until
that is done, and systemd-coredump stops at its 5-minute runtime cap,
so the dashboard and sessions stay down for about five minutes.

Set LimitCORE=0 in the unit `kirocrew service install` renders (user
and system scope), the pod template unit (and require it, so pod up
re-renders an installed pod unit without it), the EC2 bootstrap unit
and the two documented hand-written units. The faulthandler stack
dump stays the crash record; processes the gateway starts inherit
the limit, and a drop-in restores core files.

Closes #26092

(cherry picked from commit c40e8b7)

Original author: Bolin Chen (#31607). test/test_pod_systemd.py docstring hand-resolved: kept main's docstring, which names the LimitCORE check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

After a fatal signal the gateway takes 5 minutes to exit, because its systemd unit leaves core dumps unlimited

2 participants