Skip to content

fix(pod): retire stale Windows boot logs before each start - #21580

Closed
LuckVd wants to merge 2 commits into
kirodotdev:mainfrom
LuckVd:fix/21571-clear-stale-pod-logs
Closed

LuckVd wants to merge 2 commits into
kirodotdev:mainfrom
LuckVd:fix/21571-clear-stale-pod-logs

Conversation

@LuckVd

@LuckVd LuckVd commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Windows pod up/pod down refusals report the current boot's outcome, never stale log lines from a previous boot.

On Windows, the Task Scheduler pod wrapper appends boot output to pod.out.log/pod.err.log (src/kiro_crew/pod/windows.py:443-444), and the pod down/pod up fail-closed refusal tails those files via recent_journal (src/kiro_crew/pod/cli.py:679, src/kiro_crew/pod/windows.py:1281). Because start() never retired the previous boot's logs, a refusal that carried no fault of its own still printed a stale line such as FATAL: no built dist from an earlier failed start.

Why it matters

Windows pod operators get a misleading failure message: the fail-closed refusal surfaces an error left over from a previous boot, so the operator debugs a problem that no longer exists — or masks the real cause. The journal is the only evidence pod down/pod up print, so stale entries directly corrupt the diagnosis of every refusal.

Not a goal

  • Changing what the wrapper writes to the logs or how recent_journal tails them.
  • Rotating, sizing, or otherwise managing the log files beyond retiring the previous boot's leftovers at start.
  • Any non-Windows pod path.

What changed (motivation → approach → change)

  • Observed symptom: a pod down/pod up refusal printed FATAL: no built dist left over from an earlier failed start.
  • Underlying root cause: start() retired the previous boot's result_path but left the previous boot's pod.out.log/pod.err.log in place, and recent_journal tails both files.
  • The specific change: start() now unlinks the previous boot's pod.out.log/pod.err.log (via log_paths) before schtasks /Create, alongside the existing result_path unlink; unlink(missing_ok=True) keeps first-ever starts safe, and the call sits after write_task_script, which already creates the artifacts directory. So the journal only ever contains the current boot's output.

Backwards compatibility

Compatible: nothing that works today stops working. The only behavior delta is that the journal no longer contains lines from boots before the current one; the wrapper recreates both log files on every boot, and no key, field, validator, type, or public function is renamed, removed, or narrowed.

Removes nothing: the change only deletes transient log files that the next boot recreates; no path, record, key, or public symbol is taken away.

Tests

  • New: test/test_pod_windows.py::test_start_logs_only_the_current_boot_output — a fake schtasks appends new output to pre-seeded logs containing previous stdout / FATAL: no built dist, then recent_journal must show only the current boot's lines.
  • Red-test check: with only src/kiro_crew/pod/windows.py reverted to origin/main, the new test fails (AssertionError: assert 'previous stdout' not in ...); restored, it passes.
  • .venv/bin/pytest test/test_pod_windows.py -n0 — 52 passed, 2 skipped (both skips are win32-only real-schtasks.exe tests, skipped on Linux by design).
  • .venv/bin/pytest test/test_pod_windows.py test/test_pod_windows_boot.py test/test_pod_windows_boot_failures.py test/test_pod.py -n0 — 614 passed, 6 skipped (same win32-only skips).
  • Not run locally: the full ~56k-test suite — left to CI per repo scale; the diff touches only pod/windows.py::start and its test file.

Manual verification

N/A — unit coverage sufficient: the wrapper's log writes and the journal tail are both exercised by the new test and the existing pod suites; no UI, external service, or integration path is involved.

Screenshots / video

N/A — the diff touches no user-visible frontend surface (backend pod lifecycle only).

Related Issues

Pattern harvest

Rule candidate: review-prompt
Pattern: a fail-closed path that tails a shared log file must ensure the producer retires (not appends across runs to) that file at the start of each run, or stale output from a previous run is reported as the current failure.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A, no documentation surface in this diff
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@LuckVd
LuckVd requested review from a team and KiroCrewCaptain as code owners October 17, 2026 09:45
@LuckVd
LuckVd requested a review from bolichen97 October 17, 2026 09:45
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 17, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close to ready: the fix is correct and the test is a real regression pin. One should-fix remains.

Head reviewed: 677cefb. The same-repo AI lanes do not apply to fork PRs, and no fork lane comment is on this head yet. Readiness is action required.

What I verified:

  • The root cause matches the code on main: the wrapper appends with >> and 2>>, start() only retired .winresult, and recent_journal tails both log files.
  • The new test passes on this head (test/test_pod_windows.py: 52 passed, 2 skipped on Linux). It fails when only pod/windows.py is reverted to the base, so it pins the bug.
  • log_paths parent exists at that point, because write_task_script creates it.

🟡 1 should-fix (inline): the bare unlink on a log file can fail with a sharing violation on Windows. Before this change that turned into no error. After it, pod up rolls back and fails.

CI notes, not caused by this diff:

  • PR Hygiene failed against an earlier version of the PR body; the re-run on the edited body was cancelled. Edit the body again or push to re-trigger it.
  • Backend Tests (Windows) (1) failed on test_ci_ipv6_routing.py (a collection subprocess timeout) and (4) on test_extras.py (a requirements pin mismatch). Neither touches pod code. The branch is 63 commits behind main, so a rebase may clear the pin mismatch.

# The wrapper appends to these files, so retire the previous boot's output
# before scheduling this one; otherwise a refusal can report a stale FATAL.
for log_path in log_paths(cfg, name):
log_path.unlink(missing_ok=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 A sharing violation on a log file now aborts pod up

The wrapper is not the only process that can hold these logs open. This module already retries ERROR_SHARING_VIOLATION on the task wrapper (_unlink_waiting_out_sharing), because an indexer, antivirus, or Task Scheduler can hold a just-used file for a moment. A freshly written log file is a typical scan target. The gateway child also inherits these handles, so any leftover descendant keeps them open. If unlink raises PermissionError (WinError 32), the except OSError below runs _rollback_start, and pod up fails. Before this change, the same state started normally and only appended to the old log.

        for log_path in log_paths(cfg, name):
            log_path.unlink(missing_ok=True)

Why it matters: a cosmetic problem (a stale line in the journal) becomes a failed start on a timing condition the operator cannot see.

Suggestion: use the existing helper, so a brief hold is waited out and a long hold still fails closed:

        for log_path in log_paths(cfg, name):
            _unlink_waiting_out_sharing(log_path)

Add one test that makes the first unlink raise PermissionError with winerror=32 and asserts that start() still succeeds. If you prefer that a held log never blocks a start, you can catch the error, and truncate or skip the retirement instead. Either is fine. Please choose one deliberately.
Rule: AGENTS.md "Cross-platform"; precedent in _unlink_waiting_out_sharing (#16124).

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1348e46abf3cbe934a659ed9bf7aacf811693fb4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🔴 BLOCK (blocking)

Design-level review of 1348e46abf3cbe934a659ed9bf7aacf811693fb4 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: BLOCK

The PR deletes the previous boots' logs to fix a stale-FATAL problem that journal_mark already solves on the only path that calls start(), and that deletion breaks the documented pod logs contract.

Blockers

Fix targets an already-marked path and deletes history the docs say is kept

  • Cause: the patch adds for log_path in log_paths(cfg, name): log_path.unlink(missing_ok=True) inside windows.start().
  • The problem is already handled where start() runs: the only CLI path that calls start() is _up. It takes journal_since = rt.journal_mark(cfg, name) at cli.py:650, before start_pod, and then tails with recent_journal(..., since=journal_since) at cli.py:731. So a failed start already shows only the current boot's lines. windows.journal_mark's docstring says this mark exists for exactly that purpose.
  • The new test doesn't use that path: it calls recent_journal(cfg, "smoke") without since, which is not what the CLI does after a start.
  • The path that does tail without a mark is untouched: the already-active branch (cli.py:570-572, "reports the tail as it stands") never calls start(). Any stale line it prints survives this patch.
  • Consequence: src/kiro_crew/pod/README.md:409-414 and the runtime.journal_mark docstring both say "Every backend keeps earlier starts' output… An earlier start's FATAL stays in pod logs." After this patch, pod logs on Windows (cli.py:1593) loses all earlier boots, so an operator can no longer see why the previous start failed. Windows also drifts from launchd, which keeps its history. The description says "Removes nothing", which is wrong: the history is taken away, and neither doc is updated.
  • Fix: drop the unlink. Reproduce issue pod down/up refusal on Windows (no identity record) shows a stale FATAL line from the previous start #21571 through the real CLI flow, find the read that has no since mark (most likely the already-active or pod down path), and pass a mark there.

Clears when: the PR either reproduces the stale line through a CLI path that runs start() with journal_since set and shows the mark failing, or it replaces the unlink with a since-mark on the unmarked reader and keeps README.md:409-414 true.

[DESIGN-REVIEWED] 1348e46

Description read

[DESCRIPTION-READ] b5a35c86b073a7f11c468c6e594681c61c233a4ef6e830e158ba2ca3f110b0e8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🔴 BLOCK

Premise-level review of 1348e46abf3cbe934a659ed9bf7aacf811693fb4 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

Deleting the previous boots' logs repeats what journal_mark/since already does for the pod up refusal, and it erases the pod logs history the base promises to keep.

Not justified as shipped

  • Item 1 — duplicate of src/kiro_crew/pod/windows.py:1279 (journal_mark): the refusal already reads rt.recent_journal(cfg, name, 30, since=journal_since) (cli.py:731), with the mark taken before start_pod (cli.py:650). On base, pod down never calls recent_journal; the only two callers are cli.py:731 and cli.py:1593.
    Subtraction: delete the for log_path in log_paths(...) unlink loop. The new test calls recent_journal with no since, so it is testing pod logs, not the refusal path.
    Clears when: the loop is removed, or a failing reproduction on base goes through since=journal_since.
  • Item 2 — undeclared: the description says "Removes nothing". The diff removes the earlier-boot history that base text points users to: "Earlier starts' output is in kirocrew pod logs {name}" (windows.py:1322).
    Clears when: Item 1's loop is removed.
  • Item 3 — undeclared: a log file that cannot be unlinked now fails pod up before /Create. The other Windows cleanups use _unlink_waiting_out_sharing for this case (windows.py:150).

What this change ships

Inventory (3 items) — 0 justified

Intent: Windows pod failures should show only the current boot's cause. FIX.

  1. Each Windows pod up deletes the previous boots' pod.out.log/pod.err.log — duplicate of windows.py:1279 journal_mark
  2. kirocrew pod logs on Windows loses all output from earlier boots — undeclared
  3. pod up can now fail if it cannot delete an old log — undeclared

Blockers

  • Second spelling of the journal mark, framed as a fix it does not make. The stated defect is that the refusal shows stale lines. That path is already bounded by since=journal_since (cli.py:650, cli.py:731). What actually changes is pod logs (cli.py:1593), the one surface the base docstring keeps whole on purpose (windows.py:1321-1323).
    Subtraction: delete the unlink loop and the test, which only exercises the since=None path.
    Clears when: the loop is removed, or a repro shows a stale line in a refusal printed with since=journal_since.

[FIRST-PRINCIPLES-REVIEWED] 1348e46

Description read

[DESCRIPTION-READ] f6d5d3547a01ea51b733b03a2dbe1dd914319a564c12b7c0a6ce3a985cac7adb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 1348e46abf3cbe934a659ed9bf7aacf811693fb4 via the fork AI-review pipeline; updated in place on each push.

1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/pod/windows.py:705 -- Reset journal offsets when recreating the logs

log_path.unlink(missing_ok=True)
Repeated Windows boot failure with equal-length output -> _up captures the previous sizes before start() deletes the logs -> recent_journal(..., since=journal_since) skips all current output and falsely reports “no pod log”; longer output loses its leading bytes.
Anchor: residual/crash-data-loss-corruption
Fix: Make Windows journal marks start at zero for the logs recreated by start(), preserving the other backends’ offset behavior.
[BLOCK-MERGE] 1348e46
[GPT-REVIEWED] 1348e46

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

There are no adjudicable findings. The one fenced finding, F1, stands as written: its trigger is common, so the record a FLAG needs can't be completed.

F1 (fenced). The defect is confirmed in code. _up takes journal_since = rt.journal_mark(cfg, name) at src/kiro_crew/pod/cli.py:650, before rt.start_pod runs at cli.py:652. With the patch, start() then deletes both logs (the diff's lines at windows.py:705). The mark comes from log_mark.mark, which records each file's size before the delete (src/kiro_crew/pod/log_mark.py:22). Later, read_since seeks to that old offset whenever the new file is at least as long (log_mark.py:35-36). It returns the whole file only when the new file is shorter.

  • Equal length: the read comes back empty, so recent_journal prints "no pod log from this start" (src/kiro_crew/pod/windows.py:1319-1323).
  • Longer: the first bytes of the current boot's output are dropped, which is the start of the failure output that cli.py:731 prints.

Why the trigger is common: the same boot failure repeated on Windows, such as the same FATAL message twice, produces output of identical length. That is an ordinary situation, not an extreme one.

Recovery: the full current log is still on disk and readable with kirocrew pod logs. What goes wrong is the diagnosis printed at that moment: it is silent and misleading. No stored data is corrupted.

[ADJUDICATION] 1348e46 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 1348e46

[ADJUDICATION-FENCED] 1348e46 fenced=1 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/pod/windows.py:705 -- The pre-start size mark (cli.py:650, log_mark.py:22) outlives the unlink, so read_since (log_mark.py:35-36) skips current output whenever a repeated boot failure writes at least as many bytes, which is a common case, not a rare one.
[GPT-ADJUDICATED-FENCED] 1348e46

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1348e46abf3cbe934a659ed9bf7aacf811693fb4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings blocking merge.

FINDING — src/kiro_crew/pod/windows.py:711 — cli._up takes journal_mark (the old log sizes) before start_pod, but start now runs log_path.unlink(missing_ok=True). A failed boot whose new pod.err.log grows to at least the old size is then read by log_mark.read_since from the stale offset, so the start of this boot's own stderr, including the real cause, is cut from the failure report. When the boot writes nothing, the report still says "Earlier starts' output is in kirocrew pod logs", and that output has just been deleted. → Fix: drop the unlink, because the existing mark/since read already excludes the previous boot's FATAL. If the logs must reset each boot, delete them in cli._up before journal_mark is taken (an untouched file) and update the empty-log message.

[OPUS-REVIEWED] 1348e46

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 18, 2026
@LuckVd
LuckVd force-pushed the fix/21571-clear-stale-pod-logs branch from dde47de to 1348e46 Compare October 18, 2026 17:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Thank you for the fix and for the clear regression test.

Main has since solved the same issue in a different way: #21729 (merged, and it closed #21571). On main, pod up records the size of both Windows log files just before start_pod and reads only what was written after that mark, so a failed start no longer shows an earlier start's FATAL, and the earlier output stays available in kirocrew pod logs.

With that change on main, this PR is no longer needed, and merging it would now cause a problem. _up takes the size mark before start(), and this PR deletes both logs inside start(). The new boot then writes into fresh, shorter files, so the read from the old offset skips the start of this boot's own output (or all of it), which is the part that holds the real failure reason. It would also delete the history that the empty-log message points users to.

I am closing this as superseded by #21729. If you still see a stale line on Windows with current main, please open a new issue with the output and we will take a look. Thanks again for the contribution.

@bolichen97 bolichen97 closed this Oct 19, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pod down/up refusal on Windows (no identity record) shows a stale FATAL line from the previous start

2 participants