Skip to content

test(crewmate-prune): release the lock holder on the second expired wait - #23389

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/prune-wait-deterministic
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/prune-wait-deterministic

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_holder_that_outlasts_the_wait_is_waited_for_not_given_up_on passes or fails on what the prune lock wait does, not on how fast the CI runner is.

The test failed on Windows shard 7/8 of #23334 with assert 1 >= 2 (#23363). It held the prune lock in a thread, released it on threading.Timer(0.45, release.set) and asserted report.lock_waits >= 2 with PRUNE_LOCK_WAIT_S = 0.1. That counts expired waits against wall-clock time: on a loaded runner the time before the first wait begins can use up most of the 0.45 s, so only one wait expires.

Why it matters

The test guards the rule that a contending prune pass never gives up while another process holds the lock, which keeps a session from binding a crewmate the holder has not judged yet. A red it produces on unrelated PRs costs a rerun each time, and it teaches people to ignore that test.

Not a goal

What changed (motivation → approach → change)

  • Symptom: assert report.lock_waits >= 2 sees 1 on a loaded Windows runner.
  • Root cause: the release was timed by a wall clock that starts before the contender's first wait, so the number of expired waits inside the window depended on scheduling.
  • Change: the test now wraps mig.logger.warning and counts the crewmate prune still waiting WARNING, which the loop logs once per expired wait, right after report.lock_waits += 1. On the second one it sets release, so the holder lets go only after two waits have expired, whatever the runner speed. The threading.Timer is gone. The test also asserts the holder thread finished. Every other assertion is unchanged.

Backwards compatibility

Compatible: test-only change. No production code is touched.

Removes nothing: the test keeps every assertion it had.

Tests

Proof: not provable -- test-only change

  • If the pass gave up after its first expired wait, the WARNING would be logged only once. release would then never be set, the contender would return with lock_waits == 1, and assert report.lock_waits >= 2 would fail. So the test still checks the no-give-up rule, and the count it asserts now follows from the release condition instead of from timing.
  • Local (Windows, Python 3.13): the test passed 3 runs in a row. test_crewmate_prune_migration.py plus test_flake_pattern_ratchet.py: 149 passed, 2 skipped, 1 failed. The failure is TestKeptOnDoubt::test_a_link_at_a_thread_path_is_no_record, with OSError: [WinError 1314]: this host may not create symlinks. That test is untouched. black, isort, flake8 and the comment-history gate are clean.

Manual verification

N/A — unit coverage sufficient: this is a test-only change, and the test runs the real lock-contention path.

Related Issues

Fixes #23363

Sibling sweep

Swept: git grep -n "threading.Timer(" -- test/test_crewmate_prune_migration.py
Sibling: test/test_crewmate_prune_migration.py:1573 -- not affected -- keeps the default 120 s wait and asserts lock_waits == 0, so its 0.2 s Timer only ends the hold and counts nothing against the clock

Pattern harvest

Not generalizable: the shape "count expirations of a short timeout inside a fixed wall-clock window" is already a named flake class in testing-conventions.md, and this was one instance of it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): N/A, test-only
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 23, 2026 08:42
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 2175a179d2fc2d95987129a7f1e3829d95c1b959 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 2175a179d2fc2d95987129a7f1e3829d95c1b959 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 2175a179d2fc2d95987129a7f1e3829d95c1b959 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The fix makes the release depend on how many lock waits have expired, not on wall-clock time, which removes the flake's cause; it changes test code only.

Notes

  • If the warning is ever not logged, the holder still lets go after its release.wait(5) fallback, and lock_waits would then be well above 2. So the >= 2 assertion holds either way, and the test cannot pin "released on exactly the second wait". That is acceptable, because the property under test is "does not give up".

[DESIGN-REVIEWED] 2175a17

Description read

[DESCRIPTION-READ] 6f3b98005c7cde970523186cc3b851840254056593acd2bf82220869a175de43

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 2175a179d2fc2d95987129a7f1e3829d95c1b959 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2175a17

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 2175a179d2fc2d95987129a7f1e3829d95c1b959 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2175a17

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@leonlaiyc

leonlaiyc commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

On the red Readiness: none of the cancelled Backend Tests (3.12) shards (1, 3, 6, 7) failed a test. Each was stopped at the 60-minute job limit. Shard 7's log shows the leaf corpus gate group alone at 35469 passed in 3537s. Before running the files this PR changes, every leaf-mode job runs all ~694 corpus gate files. That's #23450, and #23461 fixes it (its runs are waiting on maintainer approval). Coverage Gate fails as a consequence: backend-test=cancelled -- failing closed. The AI reviews passed on 59b2551a0. Rerunning the same matrix before #23461 lands will probably hit the same limit, so I'm leaving this head as it is.

@iamwhatever
iamwhatever force-pushed the claude/prune-wait-deterministic branch from 59b2551 to e3f44d8 Compare October 29, 2026 07:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 29, 2026
@iamwhatever
iamwhatever force-pushed the claude/prune-wait-deterministic branch from e3f44d8 to c713f37 Compare October 29, 2026 10:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 29, 2026
test_a_holder_that_outlasts_the_wait_is_waited_for_not_given_up_on
released the holder on a 0.45 s timer and asserted at least two expired
0.1 s waits. On a loaded runner the time before the first wait began
used up most of that window, so only one wait expired. Release from
inside the wait loop instead, when the second "still waiting" WARNING
is logged, so the count no longer depends on the wall clock.

Fixes kirodotdev#23363

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamwhatever
iamwhatever force-pushed the claude/prune-wait-deterministic branch from c713f37 to 2175a17 Compare November 3, 2026 21:22
@iamwhatever iamwhatever added the ci-full-run Force the full backend test matrix instead of the reduced scope; read on the next push label Nov 3, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Thanks for this PR. On head 2175a17 the red is Backend Tests (3.12, 5): the job hit its 1h10m time limit and was cancelled, and Coverage Gate fails after it. That job has already been re-run several times on the same merge base with the same result, and this diff does not touch the tests in that shard.

Main has changed since these runs, including the setup of the end-to-end decisions tests that were failing in shard 5 on other PRs the same day (#29623). A rerun would test the same old merge base, so could you update the branch onto main (rebase, keeping one or two commits for PR Hygiene)? CI will then run against the current base.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks @leonlaiyc for this fix! The fork CI test shards for this PR kept hitting the 70-minute runner limit (#23450), so I opened #30818 with your exact commit (authorship kept) from an in-repo branch, where CI runs on the larger runners. This PR stays open; review and CI continue in #30818.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you, @leonlaiyc! Your change landed on main through the in-repo copy #30818 (your commit and authorship kept), which ran CI on the larger runners. Closing this one as merged-via-copy.

@iamwhatever iamwhatever closed this Nov 6, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-full-run Force the full backend test matrix instead of the reduced scope; read on the next push fork Pull request from a fork (external contributor)

Projects

None yet

3 participants