Skip to content

feat(cli): relay a dead replacement gateway.log tail on restart - #20055

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/restart-relay-gateway-log-tail
Oct 14, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/restart-relay-gateway-log-tail

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When kirocrew restart's detached replacement gateway exits at once, the restart terminal shows why.

Today the restart prints "Replacement gateway (pid N) died immediately (exit status S)" and a pointer to kirocrew logs -f. The gateway wrote its reason to gateway.log, which the restart opened for it as stdout and stderr. At least four boot refusals in slack/gateway.py end this way: the persistence preflight, the signing-key preflight and its 30 s deadline, the exit when no worker thread is available, and the truncated token_signing.key refusal. #19829 relays the repair steps for the last one only, and its First Principles disposition promised this general relay.

Why it matters

The person who ran the restart is looking at that terminal, and the restart already knows the boot failed. One relay covers every refusal that writes to the log, including ones added later, with no per-refusal code.

Not a goal

  • The service-unit path. A unit writes to the journal or the platform log, and the RESTART_NOT_UP report already names where to look.
  • The "did not become ready" branch. That replacement is still running, so its log is not yet a cause.
  • Any change to when the gateway refuses.

What changed (motivation → approach → change)

_restart records which file gateway.log is and how long it is, as (st_dev, st_ino, size), just before _spawn_detached_gateway, through _gateway_log_mark(). The replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs its refusal into a fresh file, so a size alone is no offset into what it wrote. On the _READY_DIED verdict the restart prints "Its last lines in gateway.log:" followed by _gateway_log_lines_since(mark). If gateway.log is still the measured file, that helper reads only the bytes past the recorded size. If it was rotated, it reads what the replacement wrote before the rotation, from gateway.log.prev past the recorded size and only when .prev is the measured file, then the whole fresh gateway.log. It keeps at most the final 16 KiB and the last 20 non-blank lines. It drops a first line the byte cap cut in half, keeps only printable characters in each line, and caps each line at 400 characters, because log text can quote values another process chose. An unreadable log gives an empty list, so the message falls back to today's text and the kirocrew logs -f pointer stays.

Backwards compatibility

Additive. The "died immediately" message gains lines only when the replacement wrote to the log. Exit codes and audit records are unchanged.

Tests

  • test_replacement_that_dies_relays_only_its_own_log_lines fails on origin/main and passes here. It rotates an old, longer gateway.log the way the booting replacement does, and checks that the refusal written to the fresh file is shown and the old lines are not.
  • test_replacement_that_never_becomes_ready_relays_no_log_lines covers the still-running branch.
  • TestGatewayLogRelay covers an unknown mark, a log that did not grow, a missing log, a log created after the mark, a rotated log shorter and longer than the old one, output written before the rotation and read from .prev, a .prev left by an earlier boot that is not relayed, the 20-line cap with blank lines, terminal control sequences, a line cut by the byte cap, and the per-line cap.
  • test_cli.py gives 507 passed and 2 failed. The two failures are TestDoctorStt tests that fail the same way on origin/main in this environment. flake8 and the scoped black gate pass.

Manual verification

N/A. The change is one print path in _restart, and the tests drive _restart through it.

Related Issues

Follow-up to #19829, from its First Principles review.

Pattern harvest

Generalizable: a launcher that starts a process and reports its failure should show the reason the process gave, not only point at where it wrote it. This relays the dead replacement's own log tail instead of adding one hint per refusal.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Fixes #20054

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners October 14, 2026 01:43
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 55979df2c00a8ba7892100f68b7bdf15f7738a82 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 55979df2c00a8ba7892100f68b7bdf15f7738a82 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check on Windows that the parent's gateway.log handle doesn't block the boot rename. If it does, the replacement keeps writing to the same file, and the mark's same-file branch should then relay its new lines.

[DESIGN-REVIEWED] 55979df

Description read

[DESCRIPTION-READ] 53b6ed7abc5afa94080fe05c9d34564625bb3ce8caa9332cb552dd7b795384f5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 55979df2c00a8ba7892100f68b7bdf15f7738a82 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (4 items) — 4 justified

Intent: when a restarted gateway dies on boot, the person at the restart terminal sees the reason it gave there, not just a pointer to the logs. FIX (#20054; test_replacement_that_dies_relays_only_its_own_log_lines fails on base).

  1. The "died immediately" restart report now prints up to 20 of the replacement's own last gateway.log lines. — justified
  2. The relay follows the boot rotation to gateway.log.prev by file identity, so an older gateway's lines are never shown as this one's (rotation confirmed at cli.py:1142). — justified
  3. Relayed lines keep only printable characters and stop at 400 characters each, because log text is untrusted content headed for the operator's terminal. — justified
  4. The relay reads at most 16 KiB and drops the first line if that cut splits it; on a read error the old text and the kirocrew logs -f pointer stay. — justified

[FIRST-PRINCIPLES-REVIEWED] 55979df

Description read

[DESCRIPTION-READ] 48ffb27249247f6968530f986a7951e5531d6bd6bef32e0f03fa02f1bc446f57

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 55979df2c00a8ba7892100f68b7bdf15f7738a82 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 55979df

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 55979df2c00a8ba7892100f68b7bdf15f7738a82 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 55979df

When kirocrew restart's detached replacement exits at once, print the
last 20 non-blank lines it appended to gateway.log, so every boot
refusal (persistence preflight, signing-key preflight and its deadline,
no worker thread, truncated key) shows its reason in the restart
terminal. Only bytes written after the spawn are read, capped at 16 KiB,
and each line is reduced to printable characters.
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/restart-relay-gateway-log-tail branch from 27c4397 to 55979df Compare October 14, 2026 02:45
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Cause: the relay records a size on one file and reads from another

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Mechanism: the replacement rotates gateway.log to .prev at boot

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • What happens: a shorter fresh log relays nothing, a longer one skips its start

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Consequence: refusals logged after the rotation print nothing new

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Why the tests pass: the size was mocked and the file never rotated

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Item 1: the offset belongs to the old file

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • cli_server.py: log_offset is the old gateway.log's size

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • cli_server.py: end <= offset suppresses the diagnosis after rotation

issue (blocking): Fixed in 55979df2c0. The relay measured one file and read another: the replacement rotates a non-empty gateway.log to gateway.log.prev at boot (cli._setup_cli_logging) and logs every slack/gateway.py refusal into a fresh file, so the old size was no offset into it. _gateway_log_size() is now _gateway_log_mark(), which records (st_dev, st_ino, size). _gateway_log_lines_since(mark) reads past the size only while gateway.log is still the measured file. After a rotation it reads what the replacement wrote before the rotation from gateway.log.prev, past the size and only when .prev is the measured file, then the whole fresh gateway.log. test_replacement_that_dies_relays_only_its_own_log_lines now rotates an old log 50 lines long the same way the booting replacement does and checks the refusal is shown, and TestGatewayLogRelay adds rotated-shorter, rotated-longer, pre-rotation output from .prev, a stale .prev from an earlier boot, and a log created after the mark.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 14, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readiness sweep: full diff read, PR Readiness success on this head, no live change requests or blocking threads. No regression found.

@bolichen97
bolichen97 merged commit 8e95a33 into kirodotdev:main Oct 14, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kirocrew restart should show why a replacement gateway died

2 participants