Skip to content

test(skills): wait for the coalesced follow-up prune before asserting - #24142

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/skill-view-coalesced-prune-join
Oct 26, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/skill-view-coalesced-prune-join

Conversation

@IngridMorstrad

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_two_concurrent_releases_each_prune_their_own_view stops failing on loaded CI runners, without weakening what it checks.

Since #23913 the test fails intermittently on Backend Tests shard 2, on Linux and Windows, on PRs that do not touch skill projection. It reports either the first release's view lingered or the second concurrent release skipped its prune on the busy in-process lock and its view lingered. #23965 collects about 40 occurrence reports.

Why it matters

It is the most frequent flake in the repository right now: 51+ PRs failed on it in a day, re-runs often fail again, and every hit costs a shard re-run on an unrelated PR.

Not a goal

  • Changing how end prunes coalesce (_coalesced_end_prune). The product behaviour is correct (see below); only the test's wait is wrong.
  • Other skill-projection tests, which wait on events or on every live prune thread and are not affected (Sibling sweep).

What changed (motivation → approach → change)

Root cause: a test synchronization gap, not a product race. Since #23913, an end that arrives while another end's prune pass is running only records a request in _END_PRUNE_REQUESTED and returns. When the running pass finishes, that end consumes the request and starts the follow-up pass on a new kirocrew-skill-view-prune thread, just before its own thread exits. The test built its join list once, right after the two release() calls, so it never joined that third thread. On a loaded runner it asserted while the follow-up pass was still walking. Which view lingered depends on which release took the first pass, which is why CI reports both messages.

The product does not leak the view. Every end is still followed by a pass that started after its lease was released, and the forced-order test below checks that the owed view is gone once the follow-up finishes.

  • _settle_end_prunes joins the release threads, then waits with kiro_crew.testing.wait.wait_until (the bounded wait the Determinism contract asks for, no sleep) until no end prune is running or requested for the test's agents directory. A pending request always has its key in _END_PRUNE_RUNNING until the last follow-up pass finishes, so an empty state means every owed pass has run.
  • test_two_concurrent_releases_each_prune_their_own_view uses it in place of the join over a thread list captured once. Its assertions are unchanged.
  • test_a_release_handed_to_the_follow_up_pass_is_waited_for[first|second] forces the failing order. One end's pass walks while the other projection still holds its lease. The other end then records a request, and the follow-up pass is held until the settle wait is polling. It runs once with each release leading.

Backwards compatibility

Compatible: test-only change; no product code is touched.

Removes nothing: the diff replaces one test's wait and adds a test; no key, route, record or public function is removed.

Tests

Red/green for the forced order, test_a_release_handed_to_the_follow_up_pass_is_waited_for, 25 iterations of each parametrization:

Wait Result
origin/main's join-only sync (the wait_until line swapped for return) 50 of 50 fail, the view handed to the follow-up pass lingered
this PR 50 of 50 pass

Stress on the original test, -n 32, with "load" meaning 32 extra CPU-spinning processes on a 32-core host:

Tree Load Failures
origin/main 8edd26d none 150 of 400
origin/main 8edd26d load 215 of 400, then 533 of 1000 (both CI messages seen)
this PR none 0 of 400
this PR load 0 of 400, then 0 of 1000
this PR, both tests together load 0 of 3000 (1000 original + 2000 forced-order)

Proof: not provable -- test-only change

Manual verification

N/A: the change is a test wait, covered by the red/green and stress runs above. Local gates were run on the commit: the scoped backend set (152 passed), the flake-pattern and wait-discipline ratchets (113 passed), black, isort, flake8, comment-history, file-line-cap, sync-io, harness parity, feature map, brand, loop-bound locks and semgrep scan --config semgrep/ 1.78.0 (0 findings). The scoped set had to run with TMPDIR outside ~/.kiro/crew. On this host the default TMPDIR sits under the crew home, and test_native_skill_projection.py::test_launch_record_resources_match_the_alias_file_kiro_cli_loads then refuses the tmp project as managed state. That test passes with any other TMPDIR, and its file is not in this diff.

Related Issues

Closes #23965

Sibling sweep

Swept: git grep -n "kirocrew-skill-view-prune" -- test/
Sibling: test/test_skill_view_relaunch_harness.py:310 -- not affected -- it waits until no prune thread is alive, and a follow-up thread starts before its parent exits
Sibling: test/test_native_skill_projection_rewrites.py:1063 -- not affected -- a single end, so no coalesced follow-up pass

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that joins a list of worker threads captured once, when the code under test can start more workers from those threads (a hand-off or follow-up pass).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

test_two_concurrent_releases_each_prune_their_own_view joined only the
prune threads it saw right after both releases. Since kirodotdev#23913 an end that
arrives while the other end's pass runs records a request and returns, and
the running end starts the follow-up pass on a new prune thread just before
it exits. The test never joined that thread, so on a loaded runner it
asserted while the follow-up pass was still walking and reported whichever
view that pass owed as lingering.

The test now joins the release threads and then waits, under the bounded
wait_until, until no end prune is running or requested for its agents
directory. A new test forces the order (one end walks while the other still
holds its lease, and the follow-up pass is held until the wait is polling)
for each release leading, and fails with the old join-only wait.

Closes kirodotdev#23965
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 9017c17bd5d5942d2a062a13e90c1cb7dce7e389 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 9017c17bd5d5942d2a062a13e90c1cb7dce7e389 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only fix for a test that stopped waiting too early. The settle wait reads the same _END_PRUNE_RUNNING/_END_PRUNE_REQUESTED state that _coalesced_end_prune writes, and the key shape matches (key[1] is directory.absolute().as_posix()).

[DESIGN-REVIEWED] 9017c17

Description read

[DESCRIPTION-READ] 184e9a88b2a0d8d3515acf51bab6a309fce4848b81dccc50912cb23bf3f91b53

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 9017c17bd5d5942d2a062a13e90c1cb7dce7e389 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9017c17

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 9017c17bd5d5942d2a062a13e90c1cb7dce7e389 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9017c17

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 9017c17. Looks good to ship. Test-only, one file, closes a tier:T1 flake issue.

Checked against _coalesced_end_prune on main: a request is only ever recorded while its key is in _END_PRUNE_RUNNING, and the key stays there until the last follow-up pass finds no further request. So waiting until neither set holds a key for this agents directory means every owed pass has run. The key shape matches (key[1] is directory.absolute().as_posix()). The forced-order test holds the follow-up pass until the settle wait is polling, so it fails deterministically under the old join-once wait, and it runs with each release leading. The settle wait uses the bounded wait_until helper, with no sleeps.

Security checklist: no auth, secrets, network exposure, CI trust path, approval logic or untrusted execution is touched. No findings.

Not approving yet: the Backend Tests shards (3.12, 1 to 8) are still queued and readiness is still checking on this head. Approval can follow once readiness reports passed.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 9017c17bd5d5942d2a062a13e90c1cb7dce7e389 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 26, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at head 9017c17. Checked: readiness passed on this head, all checks green, AI review lanes report no blocking findings, no other changes requested, base is main, mergeable, one test file only.

@bolichen97
bolichen97 merged commit 5d98031 into kirodotdev:main Oct 26, 2026
103 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 26, 2026
bolichen97 added a commit that referenced this pull request Oct 29, 2026
The v0.11.0-rc.2 release run failed two Test Release Candidate shards on
tests main has already fixed:

- test(skills): wait for the coalesced follow-up prune before asserting (#24142)
  (cherry picked from commit 5d98031)
- test(skills): freeze the lease-scan clock in the legacy-lock test (#24257)
  (cherry picked from commit 3ff45cb)
- test: pin the prune clock in the seam pruning test (#24191)
  (cherry picked from commit 33fd421)

#24191's test seeds a 2026-09-25 history file and prunes at 30 days, so it
fails on every run after 2026-10-25. #24142 follows the end-prune
coalescing that #23913 brought to this branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_two_concurrent_releases_each_prune_their_own_view sees a lingering view since #23913

2 participants