Skip to content

test: absorb the Windows post-rewrite metadata-read transient in revive tests - #15396

Merged
iamwhatever merged 1 commit into
mainfrom
fix/win-revive-metadata-read-transient-13110
Sep 30, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/win-revive-metadata-read-transient-13110

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make test/test_session_control_revive.py deterministic on Windows so it stops failing the Backend Tests (Windows) shard for reasons unrelated to what each test exercises.

Commit 2138c2c3fc (feat(session-control): session_revive, PR #13110) landed on main and its new test file test/test_session_control_revive.py began failing on the Backend Tests (Windows) shard 2 of 8 across many open PRs. The Windows shard did not run on the merge commit itself, so it landed unchecked. Linux passes the same heads. The failing set overlaps but is not identical run to run and differs across the two Windows runner images — the fingerprint of a filesystem-timing dependency, not a logic bug.

The failures surface as a mix of resume_conflict refusals ("this session changed while resuming"), a fixture reporting its clear "did not land", and a hook that ran once instead of twice — all downstream of one cause.

Why it matters

While red, this single test file blocks the Backend Tests (Windows) check on ~10 unrelated open PRs, so nothing merges through that lane until it is fixed. It is currently the single biggest merge blocker in the queue.

Not a goal

  • No production behavior change. The revive/metadata code is Windows-correct on both runner images and is not touched.
  • Not a broad test-infra change: only the four-plus tests in this one file that read metadata synchronously on the event loop are affected.

What changed (motivation → approach → change)

Symptom → root cause. These tests drive a real on-disk ConversationLog through metadata rewrites (close_target → clear_closed / update_metadata_if) and then read the metadata line straight back — all synchronously on the event loop via asyncio.run. On Windows a just-rewritten file is briefly unopenable while an indexer or AV scanner holds it (ERROR_SHARING_VIOLATION → PermissionError, milliseconds long). Production tolerates this: _read_metadata_state retries _METADATA_READ_ATTEMPTS times, and every real caller either runs the read off the loop (where _pause_for_transient_retry sleeps between attempts) or accepts the fail-closed resume_conflict and retries. But that read-retry pause is a deliberate no-op on the loop (so it never blocks the event loop). A synchronous on-loop test read therefore spins through its 3 attempts in microseconds and reports the sharing hold as a transient → resume_conflict. That is a failure decided by the OS timing, not by the condition the test sets up.

Approach. Confirmed the mechanism at the read seam without a Windows box: modelling a 50 ms sharing window on the metadata open(), the production on-loop path (3 rapid attempts, no sleep) reads the line as unreadable/transient, while the same read with the fix reads it as readable. The alternative — mocking get_metadata_status in every test the way sibling suites already do (test_dashboard_sessions_clear.py, test_slot_close_recreation_race.py, test_dashboard_chat.py) — was rejected because it would stop these tests exercising the real read path they were written to cover; the point of this file is that revive is asserted against real slot objects and real on-disk metadata.

Change. One autouse fixture, _absorb_metadata_read_transient, that lets the same real read wait the hold out on the loop: it forces _pause_for_transient_retry to sleep on the loop (mirroring the off-loop production path) and raises the read-attempt budget so any runner image's hold clears well within it. It touches only the read-retry seam, so every injected condition these tests rely on is untouched — the facade-level stubs (get_metadata_status / clear_closed / update_metadata_if replaced on the log) bypass _read_metadata_state entirely, and the write-path OSError injections are on a different path. This asserts what is true on both Windows runner images (the read succeeds once the hold clears; the outcome is decided by each test's own condition) rather than pinning one image's timing.

Backwards compatibility

Compatible: test-only change (one file under test/), no production code, no contract touched. Nothing that works today stops working.

Tests

  • test/test_session_control_revive.py: adds the autouse _absorb_metadata_read_transient fixture. It locks in that the metadata read on the event loop waits out a transient sharing hold, so the revive outcome each test asserts is decided by that test's injected condition, not by OS file-sharing timing.
  • Verified locally on Linux: the full file is 97 passed with the fixture in place; the board-named tests (test_a_history_click_that_loses_the_race..., test_a_marker_restore_that_fails_once_recovers_on_the_retry, test_with_a_hook_a_gone_folder_still_drops..., test_a_clean_hook_publishes_once..., test_a_cancellation_during_the_filing_rolls_back..., test_the_filing_runs_under_the_slot_metadata_txn_lock, test_revive_accepts_every_key_spelling..., and the original four) all pass.
  • Read-seam validation (transient modelled at the open()): production on-loop defaults read transient/unreadable → resume_conflict; with the fixture the same read is readable.

Manual verification

N/A — cannot run the Windows runner locally; validated by modelling the sharing window at the read seam and by the mechanism being confined to the read-retry path. CI's Backend Tests (Windows) shard is the authoritative check.

Related Issues

Fixes the Windows shard breakage introduced by 2138c2c3fc (feat(session-control): session_revive, #13110).

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that drives a real on-disk store and then reads it back synchronously on the event loop inherits the platform's file-sharing timing (Windows post-rewrite ERROR_SHARING_VIOLATION), because the production read-retry pause is a deliberate no-op on the loop. Such a test must either read off the loop, mock the read, or raise/force the on-loop retry — never depend on the immediate on-loop read succeeding.

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 30, 2026 11:28
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 9cf8cd57d76dc55165738945cf3c1f7bf713832a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Description documents a superseded read-side fix; the shipped fixture settles the write side instead, so the PR's stated validation describes a different revision.

Watch

  • The description's "Change"/"Tests"/"Read-seam validation"/"Pattern harvest" sections describe _absorb_metadata_read_transient ("forces _pause_for_transient_retry to sleep… raises the read-attempt budget") while the diff ships _settle_metadata_rewrite_before_read, whose docstring says the opposite ("no attempt count… raised and the retry pause is not patched"). A future engineer debugging a recurrence of this flake will search for a fixture and mechanism that don't exist, and the "97 passed" / read-seam-modelling claims cannot be attributed to this revision.
    Clears when: the description's approach/change/tests/pattern-harvest sections are rewritten to describe the write-side settle actually shipped, with the local-run and modelling evidence re-stated for this fixture.
  • The settle silently gives up after its 2 s deadline and returns, reverting to today's flake with no signal that the budget was the reason. Acceptable fallback, but worth knowing if the Windows shard reds recur.
    Clears when: a green Backend Tests (Windows) shard 2 run on this head (CI is the authoritative check the PR itself names).

[DESIGN-REVIEWED] 9cf8cd5

Description read

[DESCRIPTION-READ] 652e4cf9f7b81e629c317e8bae8f9ee33cedc29355a4223ea7b49f0d5df7100c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 9cf8cd57d76dc55165738945cf3c1f7bf713832a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9cf8cd5

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9cf8cd57d76dc55165738945cf3c1f7bf713832a: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 9cf8cd57d76dc55165738945cf3c1f7bf713832a — this comment is updated in place on each push.

Review details

No blocking issues; one advisory finding.

FINDING — test/test_session_control_revive.py:65 — the docstring asserts the destination is "always settled — the test awaits the rewrite before reading, deterministically", but the probe below it returns silently at if time.monotonic() >= deadline: return (line 97) and on except FileNotFoundError: return, and closes its handle before returning, so a Windows hold outlasting the 2.0 s budget (or re-taken after the probe) still reaches the deliberately single-attempt on-loop read at src/kiro_crew/dashboard/chat_handlers.py:13849 and re-produces the same resume_conflict the fixture claims it removes → Fix: state what the code does ("best-effort: waits up to 2 s for the destination to become openable, then proceeds"), or make deadline expiry fail loudly so a non-transient hold is diagnosed as itself rather than as resume_conflict.

[OPUS-REVIEWED] 9cf8cd5

Verdict parsed from the review's SHA-scoped output markers for commit 9cf8cd57d76dc55165738945cf3c1f7bf713832a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9cf8cd57d76dc55165738945cf3c1f7bf713832a: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/win-revive-metadata-read-transient-13110 branch 2 times, most recently from 07ccdfc to a9a3cc1 Compare September 30, 2026 12:11
… Windows

The revive tests drive a real on-disk ConversationLog through metadata
rewrites and then read the line back synchronously on the event loop.
Every rewrite lands atomically through replace_with_retry (temp file
renamed over the target). On Windows the freshly-renamed file is briefly
unopenable while an indexer/AV scanner holds it (ERROR_SHARING_VIOLATION).
Production tolerates this off the loop, where the read retry sleeps; on
the loop that pause is a deliberate no-op (a live server must not block
the loop inside a read), so a synchronous on-loop test read races the
hold and fails closed with resume_conflict -- a failure decided by OS
file-sharing timing, not by what the test set up.

Leave the production READ path exactly as it is: do not raise any attempt
count, timeout, or poll budget and do not patch the retry pause, since
that hides the race rather than resolving it. Instead synchronize the
filesystem transition on the WRITE side -- after each metadata-line rename
lands, confirm the destination is openable again before the writer
returns. Every metadata rewrite funnels through replace_with_retry and
every real caller reads only after its write returns, so the read always
opens a settled file: the test awaits the rewrite before reading,
deterministically. The read still returns the true current line, so a
line another writer changes concurrently (a link written during the read)
is still seen -- nothing is served from a stale snapshot.

This supersedes earlier attempts in this branch: raising the read budget
(flagged by review as hiding the race), wrapping only the two metadata
writers (missed reads that follow a different write path, e.g. the
case-variant live-key test), and forcing the on-loop read pause to sleep
(again flagged as touching the read budget). Wrapping the shared rename
primitive covers every write path without touching the read at all.

Fixes the Backend Tests (Windows) shard-2 breakage introduced by
2138c2c (feat(session-control): session_revive), which landed
unchecked because the Windows shard did not run on the merge commit.
@chenmingwei23
chenmingwei23 force-pushed the fix/win-revive-metadata-read-transient-13110 branch from a9a3cc1 to 9cf8cd5 Compare September 30, 2026 12:36
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants