Skip to content

test: backdate closes before resumes so windows clock ties cannot 409 - #17145

Merged
iamwhatever merged 1 commit into
mainfrom
refactor/df-p3-resume-windows-transient
Oct 6, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
refactor/df-p3-resume-windows-transient

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Backend Tests (Windows) (2) stops failing on unrelated PRs with a 409 resume_conflict from tests that close a session and resume it straight away.

Since #16435, four nodes in Windows shard 2 fail intermittently with {"error": "this session changed while resuming; open it again", "code": "resume_conflict"}:

  • test_dashboard_chat.py::TestFolderCRUD::test_resume_session_unhides_folder
  • test_slot_ownership_checkpoint.py::TestTranscriptOwnershipWithoutALiveSlot::test_an_app_resumes_its_own_closed_session
  • test_slot_ownership_checkpoint.py::TestTranscriptOwnershipWithoutALiveSlot::test_no_key_or_the_bare_name_means_the_slots_own_transcript[no-key|bare-key]

None of the affected PRs touches resume.

Why it matters

The shard is red on most open PRs, so every one of them spends a rerun, or a human has to rule that the red is unrelated, before it can merge. A rerun cannot be trusted either: the same class failed again on a rerun with a different node.

Not a goal

  • No production change. The resume's compare-and-clear deliberately keeps a closed marker stamped in the same clock reading as the resume's start; test(session-control): pin the revive's same-clock-reading refusal #15444 pins that as a maintainer decision, and this PR leaves it, and its pin test_a_revive_in_the_close_clock_reading_keeps_the_marker, untouched.
  • No retry or wider read budget on the resume's identity re-reads. The issue's first guess (a transient sharing violation on that re-read) is not the cause; see below.
  • No sweep of tests that do not close and then resume.

What changed (motivation → approach → change)

Symptom → cause. The 409 is a clock tie, not an unreadable file. A resume clears closed only when closed_at is strictly earlier than the moment the resume started (clear_closed(only_if_closed_before=resume_started_at)), and a tie keeps the marker on purpose (#15444). Both stamps come from time.time(), which moves every ~15.6 ms on Windows CPython 3.12. These tests close a session and resume it within one reading, so on Windows the clear is skipped and the resume's verification ("closed" in _after) refuses with the body above. Before #16435 a resume through the HTTP endpoint never verified the clear, so a tie published silently; since then every closed resume verifies it.

Approach. State the precondition the tests mean, "closed BEFORE the resume begins", instead of leaving it to clock resolution. test_session_control_revive._archive already does exactly this for its own file.

Change.

File Change
test/chat_test_helpers.py New close_before_resume(log, key): asserts the session is closed and moves its closed_at 60 s back, beside the existing move_transcript_past stamp helper.
test/test_slot_ownership_checkpoint.py _persist_and_close calls it, so every close→resume in the file starts after its close.
test/test_dashboard_chat.py test_resume_session_unhides_folder calls it after its closed save. test_resume_close_resume_no_duplicate_history calls it before its second resume and now asserts both resumes return 200.

The last test was vacuous on a tie: its second resume 409'd unasserted, so the "resume again and close without changes" cycle it exists for never ran, and it still passed.

Diagnosis evidence (corrected diagnosis also posted on #17117):

Probe (Linux, BASE c6bfd72db) Result
ONE PermissionError injected on the post-clear metadata re-read (the issue's hypothesis) 4/4 green: get_metadata_status already retries 3x and absorbs it
Read failing 3x in a row (exhausted retry) reds, and every failure prints Captured log call: WARNING kiro_crew.history ... could not read metadata ... after 3 attempts; the CI failure reports (jobs 111925846311, 111920436730) have no captured-log section, so no exhausted read happened there
coarse_clock (Windows tick, 1/64 s), probe on clear_closed 17 reds in 10x4 runs; all 17 are closed_at >= boundary with the marker kept; zero unreadable reads
coarse_clock sweep, 3x at -n 4 over all 31 resume-exercising test files (3,803 tests) tie-exposed: exactly the 5 nodes changed here, plus the deliberate #15444 pin

Backwards compatibility

Compatible: test-only; no production module, wire format, or public name changes.

Removes nothing: the diff adds a test helper and stronger assertions; no test, assertion, or capability is dropped.

Tests

De-flake verification protocol (repeat at -n0 and -n 4, forced condition red on the parent and green on the fix, mutation proof), over the 5 changed nodes unless stated:

Run Parent (BASE) This PR
KC_FORCE=coarse_clock, -n0, 20x 18/20 runs red, 28 node failures (unhides_folder 7, an_app_resumes 2, [no-key] 11, [bare-key] 8); the dedupe node passes vacuously 20/20 green (5 passed, ~2-3 s each)
plain, -n0, 20x — 20/20 green
coarse_clock, -n 4 --dist loadgroup --max-worker-restart=2, 10x, whole test_dashboard_chat.py + test_slot_ownership_checkpoint.py + co-tenants test_resume_publishes_hydrated_slot.py, test_resume_refuses_an_in_flight_delete.py — 10/10 green, 1052 passed each (~27-48 s)
Mutation: close_before_resume made a no-op, coarse_clock, -n0, 10x — 9/10 runs red; the dedupe node reds BY NAME on its new resp.status == 200 assertion, so the vacuity fix is not vacuous itself

Related backend set (run_scoped_tests.py --surface backend, 200 files, -n 4): 10,445 passed, 1 skipped, 4 xfailed, 0 failed. local-gate.py --dry-run: 200 backend / 322 frontend (cross-surface) related files; the frontend set was not run locally because the diff touches no website/** file, and CI runs it.

Local review: the GPT- and Opus-lane contracts (both run on Opus 5.5; no GPT model was available locally) reported no findings, and a completeness pass found no other tie-exposed close→resume test.

Static: black, isort, flake8, mypy (2183 files clean), check_black_formatting.py, check_subprocess_encoding.py, and CI's semgrep rule set in --baseline-commit mode (242 rules, 3 files, 0 new findings).

Manual verification

N/A — test-only change; the protocol runs above are the verification. The Windows half is proven only by this PR's exact-SHA Backend Tests (Windows) shards; Linux forced-clock runs model the tick, they do not replace Windows evidence.

Related Issues

Fixes #17117
Part of #16536

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that closes a session and resumes it with no stated "closed BEFORE the resume begins" precondition ties on a coarse clock; set the stamp (close_before_resume), never sleep to separate it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 5, 2026 21:26
@bolichen97
bolichen97 requested a review from iamwhatever October 5, 2026 21:26
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of dc7d8d34f78fff3975ccf060dab3a6086a3c39ea — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that this exact SHA's Backend Tests (Windows) (2) shard is green. The Linux coarse-clock runs only simulate the Windows tick and don't prove the fix there.

[DESIGN-REVIEWED] dc7d8d3

Description read

[DESCRIPTION-READ] 9f761f7444746268d4398c6c6ae5dca067aa016d38c941239dbb1085f4f0bdd9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of dc7d8d34f78fff3975ccf060dab3a6086a3c39ea and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] dc7d8d3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt dc7d8d34f78fff3975ccf060dab3a6086a3c39ea: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed dc7d8d34f78fff3975ccf060dab3a6086a3c39ea — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] dc7d8d3

Verdict parsed from the review's SHA-scoped output markers for commit dc7d8d34f78fff3975ccf060dab3a6086a3c39ea.

False positive or not applicable? A repository writer can comment:
/ai-review override fable dc7d8d34f78fff3975ccf060dab3a6086a3c39ea: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
Backend Tests (Windows) (2) fails on unrelated PRs with 409
resume_conflict ("this session changed while resuming"). The cause is
not an unreadable identity re-read. A resume clears `closed` only when
closed_at is strictly earlier than the resume's start, and a tie keeps
the marker on purpose (#15444). Both stamps come from time.time(),
which moves every ~15.6 ms on Windows CPython 3.12, so a test that
closes a session and resumes it at once ties there, the clear is
skipped, and the post-clear verification refuses. Since #16435 every
closed resume verifies that clear.

Add close_before_resume() to chat_test_helpers: it states the
precondition "closed BEFORE the resume begins" by moving closed_at
60 s back, as test_session_control_revive._archive already does. Call
it from test_slot_ownership_checkpoint's _persist_and_close and from
the two close-then-resume tests in test_dashboard_chat.
test_resume_close_resume_no_duplicate_history also asserts that both
of its resumes return 200; on a tie its second resume was refused
unasserted and the test passed without running the cycle it covers.

Fixes #17117
@bolichen97
bolichen97 force-pushed the refactor/df-p3-resume-windows-transient branch from f9db227 to dc7d8d3 Compare October 6, 2026 00:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 6, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 6, 2026 02:07

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change: backdates session closes before resumes so Windows clock ties cannot produce a 409.

@iamwhatever
iamwhatever merged commit 4421fd8 into main Oct 6, 2026
105 checks passed
@iamwhatever
iamwhatever deleted the refactor/df-p3-resume-windows-transient branch October 6, 2026 02:07
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resume: a transient Windows sharing violation on the post-hook identity re-read refuses the resume as resume_conflict

2 participants