Skip to content

fix(subagent): retry the session record write that owns a run's cwd - #17484

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/native-chain-cwd-windows-15921
Oct 7, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/native-chain-cwd-windows-15921

Conversation

@buluoray

@buluoray buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A subagent run keeps its recorded project directory (cwd) and session_id even when the first write of that record fails, and a record that is still lost shows up in the log.

On a Windows CI shard, test/test_member_continuation_identity.py::test_native_chained_continuations_record_the_project failed with KeyError: 'cwd': the run finished and read_state returned a record, but the record had no cwd. That field comes from exactly one write, the "session record" _write_state_off_loop in subagent_manager/run.py. That write had no retry, ignored the False that update_state returns when it skips a merge on an unreadable state, and swallowed any exception at debug level, so nothing in the job log said the field was lost.

Why it matters

The recorded cwd is what a continuation (spawn_continue, REST continue, follow-ups after a gateway restart) reads back through recorded_cwd to run in the same project. A lost record silently sends the next continuation to the pool default directory, and the same gap reddens the Windows shard for unrelated PRs (#13109 was the one hit).

Not a goal

  • Changing update_state or the atomic write itself. On current main _atomic_write already goes through atomic_write -> replace_with_retry, and update_state already reads through read_bytes_with_retry (fix(subagent): claim a whole answer's ending and keep its result whole #16185), which covers the Windows rename/read sharing-violation half of the triage comment.
  • Retrying the other best-effort state writes (PID record, cleanup identity sidecar).

What changed (motivation → approach → change)

  • Symptom: a finished run's state.json without cwd, no visible error.
  • Root cause: the session-record write is the single owner of cwd/session_id on the spawn path, but unlike the model-provenance write directly above it, it made one attempt and treated a skipped merge as success.
  • Change: run.py gives the session-record write the same ONE bounded retry as the provenance write (an exception or a False return gets a second attempt; only a reported write ends the loop; CancelledError still ends it without a second writer). If both attempts fail, it logs at WARNING and the run continues, so persistence still never blocks a spawn. The subagent spec's continuation-cwd paragraph records this contract.

Backwards compatibility

Compatible: the same fields are written by the same drained helper; the only differences are a second attempt after a failure and a WARNING when both fail.

Removes nothing: the diff adds a retry and a log line; no field, record, key or function is removed or narrowed.

Tests

In test/test_subagent_provenance_write.py:

  • test_session_record_write_retries_once[raises|skips]: the first session-record attempt raises PermissionError / returns False; asserts exactly two attempts and that cwd lands. Red on origin/main 9bf943b (one attempt).
  • test_session_record_write_that_never_lands_is_a_warning: both attempts fail; asserts the retry is bounded at two, the run still finishes without error, and one WARNING names the run. Red on origin/main.

Mutation matrix (each applied to run.py, then restored): no retry -> 3 tests red; skip counted as success -> [skips] red; WARNING dropped -> warning test red; three attempts -> warning test red (bound). The existing test_pid_and_session_records_are_written_off_loop (exactly one session-record write on success) still passes. Whole file: 23 passed with -n0.

Manual verification

N/A — the original failure only reproduces on the Windows runner; this host is Linux, so the failure was simulated by making update_state raise/skip on the session-record call. The Windows backend lanes in CI are the live check.

Related Issues

Closes #15921

Pattern harvest

Rule candidate: review-prompt
Pattern: a best-effort persistence call that is the sole writer of a field others read back must check the writer's reported result and retry/log above debug, not just catch exceptions at debug.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@buluoray
buluoray requested a review from a team as a code owner October 6, 2026 18:48
@buluoray
buluoray requested a review from chenmingwei23 October 6, 2026 18:48
@buluoray
buluoray force-pushed the fix/native-chain-cwd-windows-15921 branch from c916f38 to e04f508 Compare October 6, 2026 18:59
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of e04f5089b55c0ab91e93bd36c8afb090d5d8b663 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The root cause is a guess: KeyError: 'cwd' also happens when the write succeeds but _session_cwd is empty, and this PR neither retries nor logs that case.

Watch

  • The diagnosis has no evidence behind it. The run builds the record with if cleanup_cwd: state_update["cwd"] = cleanup_cwd. So if the Windows client reports an empty cwd, the write succeeds, the record has no cwd key, and the test hits the same KeyError: 'cwd'. No retry runs and no WARNING fires, so tests: test_native_chained_continuations_record_the_project loses the session record's cwd on a Windows shard #15921 would stay flaky with nothing new in the log. The description says the failure was only simulated, and nothing shows it was a failed write rather than an empty value.
    Clears when: the record also logs a WARNING when cwd is missing from state_update, or a Windows CI log shows the new failed-write WARNING on a red run.

[DESIGN-REVIEWED] e04f508

Description read

[DESCRIPTION-READ] bc6729e1cb1a414fae8fff3ef07813b5c7ee971e70d3acafaac27b25f8f275e8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of e04f5089b55c0ab91e93bd36c8afb090d5d8b663 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The retry wraps a write whose read and rename already retry, and nobody knows why it failed on Windows, so "Closes #15921" claims more than the fix shows.

Not justified as shipped

  • Item 1 — symptom-level: the description says the root cause is that the write "made one attempt and treated a skipped merge as success". But update_state already reads through read_bytes_with_retry, and its write goes through replace_with_retry (the _atomic_write → atomic_write path). The description itself says that "covers the Windows rename/read sharing-violation half". Nothing in this change shows which failure on Windows still gets through those retries. The tests fake the failure (raise PermissionError / return False) and do not reproduce it, so the Windows cause is still unnamed.
    Subtraction: keep Item 2's WARNING and defer the retry loop until that WARNING names the real failure on a Windows shard. Or narrow Closes #15921 to Refs #15921.
    Clears when: a Windows CI log or a test that fails on base shows a failure read_bytes_with_retry/replace_with_retry does not absorb and a second update_state call does.

What this change ships

Inventory (3 items) — 2 justified

Intent: a subagent continuation keeps running in the project it started in, even after the first record write fails. FIX.

  1. A failed or skipped session-record write gets one more attempt — symptom-level
  2. A session record that is lost after both attempts now shows up as a WARNING in the log — justified
  3. The subagent spec now describes the session-record retry and the warning — justified

[FIRST-PRINCIPLES-REVIEWED] e04f508

Description read

[DESCRIPTION-READ] bb98536e5909c90651360aba8786edb6db0acfa5c1168723bebdc288dee62ffd

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 6, 2026
@buluoray

buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Disposition: Design Review CONCERNS ("cwd can also be missing when _session_cwd is empty") — checked, not reachable for #15921, no change.

cleanup_cwd is client.cwd when it is a non-empty string, else info.cwd (src/kiro_crew/subagent_manager/run.py ~L2342-2345). The failing test, test/test_member_continuation_identity.py::test_native_chained_continuations_record_the_project, sets provider.cwd = str(project) on every iteration (L182) before the run starts, so cleanup_cwd is always the non-empty tmp_path/chess path and state_update["cwd"] is always populated. The platform has no part in that value: it is a literal the test assigns to a mock. For this test, the empty-value path the review describes cannot produce KeyError: 'cwd'. The only way cwd can be missing is the record write not landing, and that is the path this PR retries and logs.

A missing cwd from an empty provider value in production is a separate question about which providers report no cwd. It is out of scope for this issue, and adding a WARNING there would also fire for runs that legitimately have no project.

@buluoray

buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Disposition: First Principles CONCERNS ("the retry wraps a write whose read and rename already retry") — partly accepted, kept as shipped.

What read_bytes_with_retry / replace_with_retry absorb is a sharing violation on the read or the rename. What they do not absorb, and a second update_state call does, is every path where update_state returns False or raises outside those two wrappers:

  • the read retries run out and update_state reports a skip (False). Before this PR the caller treated that as success, with nothing logged;
  • the bytes are read but fail json.loads (ValueError) or are not an object, which update_state also reports as a skip (src/kiro_crew/subagent_persistence.py ~L1056-1064);
  • an exception from _atomic_write's own temp-file open/write/fsync before the rename, which propagates out of update_state rather than being retried.

On base, all of these left the record without cwd/session_id, and nothing above DEBUG showed it. The model-provenance write in the same function already uses this exact one-retry contract for the same reason, so this PR does not add a new mechanism.

Agreed: no Windows log names which of these hit the flaky shard. The tests fake the failure at the _write_state_off_loop boundary (exception, and False), because that boundary is the one this change owns. That is why the WARNING is part of the fix: if #15921 comes back, the shard log names it (Session record (session_id/cwd) was not persisted ... after 2 attempts) instead of only showing the downstream KeyError. All 8 Windows backend shards on e04f5089b5 passed, which is consistent with the fix but does not prove it for an intermittent failure. I'm keeping Closes #15921, with the WARNING as the tripwire that reopens it.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e04f5089b55c0ab91e93bd36c8afb090d5d8b663 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e04f508

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e04f5089b55c0ab91e93bd36c8afb090d5d8b663: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e04f5089b55c0ab91e93bd36c8afb090d5d8b663 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e04f508

Verdict parsed from the review's SHA-scoped output markers for commit e04f5089b55c0ab91e93bd36c8afb090d5d8b663.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e04f5089b55c0ab91e93bd36c8afb090d5d8b663: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@chenmingwei23
chenmingwei23 merged commit 9845e0f into main Oct 7, 2026
102 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/native-chain-cwd-windows-15921 branch October 7, 2026 04:15
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: test_native_chained_continuations_record_the_project loses the session record's cwd on a Windows shard

2 participants