Repository navigation
fix(subagent): retry the session record write that owns a run's cwd - #17484
Conversation
c916f38 to
e04f508
Compare
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The root cause is a guess: Watch
[DESIGN-REVIEWED] e04f508 Description read[DESCRIPTION-READ] bc6729e1cb1a414fae8fff3ef07813b5c7ee971e70d3acafaac27b25f8f275e8 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The retry wraps a write whose read and rename already retry, and nobody knows why it failed on Windows, so "Closes #15921" claims more than the fix shows. Not justified as shipped
What this change shipsInventory (3 items) — 2 justifiedIntent: a subagent continuation keeps running in the project it started in, even after the first record write fails. FIX.
[FIRST-PRINCIPLES-REVIEWED] e04f508 Description read[DESCRIPTION-READ] bb98536e5909c90651360aba8786edb6db0acfa5c1168723bebdc288dee62ffd That sha256 names the PR description this verdict read. Recompute it with |
|
Disposition: Design Review CONCERNS ("
A missing |
|
Disposition: First Principles CONCERNS ("the retry wraps a write whose read and rename already retry") — partly accepted, kept as shipped. What
On base, all of these left the record without Agreed: no Windows log names which of these hit the flaky shard. The tests fake the failure at the |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal: A subagent run keeps its recorded project directory (
cwd) andsession_ideven when the first write of that record fails, and a record that is still lost shows up in the log.On a Windows CI shard,
test/test_member_continuation_identity.py::test_native_chained_continuations_record_the_projectfailed withKeyError: 'cwd': the run finished andread_statereturned a record, but the record had nocwd. That field comes from exactly one write, the "session record"_write_state_off_loopinsubagent_manager/run.py. That write had no retry, ignored theFalsethatupdate_statereturns when it skips a merge on an unreadable state, and swallowed any exception at debug level, so nothing in the job log said the field was lost.Why it matters
The recorded
cwdis what a continuation (spawn_continue, REST continue, follow-ups after a gateway restart) reads back throughrecorded_cwdto run in the same project. A lost record silently sends the next continuation to the pool default directory, and the same gap reddens the Windows shard for unrelated PRs (#13109 was the one hit).Not a goal
update_stateor the atomic write itself. On current main_atomic_writealready goes throughatomic_write->replace_with_retry, andupdate_statealready reads throughread_bytes_with_retry(fix(subagent): claim a whole answer's ending and keep its result whole #16185), which covers the Windows rename/read sharing-violation half of the triage comment.What changed (motivation → approach → change)
state.jsonwithoutcwd, no visible error.cwd/session_idon the spawn path, but unlike the model-provenance write directly above it, it made one attempt and treated a skipped merge as success.run.pygives the session-record write the same ONE bounded retry as the provenance write (an exception or aFalsereturn gets a second attempt; only a reported write ends the loop;CancelledErrorstill ends it without a second writer). If both attempts fail, it logs at WARNING and the run continues, so persistence still never blocks a spawn. The subagent spec's continuation-cwd paragraph records this contract.Backwards compatibility
Compatible: the same fields are written by the same drained helper; the only differences are a second attempt after a failure and a WARNING when both fail.
Removes nothing: the diff adds a retry and a log line; no field, record, key or function is removed or narrowed.
Tests
In
test/test_subagent_provenance_write.py:test_session_record_write_retries_once[raises|skips]: the first session-record attempt raisesPermissionError/ returnsFalse; asserts exactly two attempts and thatcwdlands. Red on origin/main 9bf943b (one attempt).test_session_record_write_that_never_lands_is_a_warning: both attempts fail; asserts the retry is bounded at two, the run still finishes without error, and one WARNING names the run. Red on origin/main.Mutation matrix (each applied to
run.py, then restored): no retry -> 3 tests red; skip counted as success ->[skips]red; WARNING dropped -> warning test red; three attempts -> warning test red (bound). The existingtest_pid_and_session_records_are_written_off_loop(exactly one session-record write on success) still passes. Whole file: 23 passed with-n0.Manual verification
N/A — the original failure only reproduces on the Windows runner; this host is Linux, so the failure was simulated by making
update_stateraise/skip on the session-record call. The Windows backend lanes in CI are the live check.Related Issues
Closes #15921
Pattern harvest
Rule candidate: review-prompt
Pattern: a best-effort persistence call that is the sole writer of a field others read back must check the writer's reported result and retry/log above debug, not just catch exceptions at debug.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.