Skip to content

test(crew-log): wait for the eager seal before sweeping a destroyed log - #36623

Merged
bolichen97 merged 1 commit into
mainfrom
fix/35553-sweep-after-destroy-flake
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/35553-sweep-after-destroy-flake

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_destroyed_sessions_log_is_then_collectable_by_the_sweep passes on every run, instead of sometimes failing with (0, 0).

In Backend Tests the test fails at random with assert (0, 0) == (1, 0) at its sweep line (test/test_crew_log_retention.py:1141), on main and on pull requests that do not touch the crew log (#35553). It passes when rerun.

Why it matters

Every unrelated pull request that draws this red pays a rerun and a flake report, and a sweep test that fails at random trains people to ignore it when it fails for a real reason.

Not a goal

  • Changing the retention sweep, remove_unit or the eager folder. A sweep that meets a held lease skips the unit and a later pass collects it, by design; in production a unit is collected only once its close is older than the retention window, long after the seal its close woke.
  • The other tests in this file.

What changed (motivation → approach → change)

The test now waits for the crew log's background folding to finish before it runs the sweep, so the sweep always finds the closed log free to remove.

destroy writes session/closed, and that entry also wakes the eager folder, which seals the unit's savepoint under a shared hold of the unit's lease. The test waited only for the crew log's writer (assert_drained()), so it could age the close and call sweep_expired while that seal still held the lease. The sweep's sole claim then answered owned, nothing was removed, and the sweep returned (0, 0). Running the sweep off the loop did not cover this: it waits for the folder only around the unlinks, after the lease claim.

After destroy, the test asserts eager.drain(timeout=FALLBACK_TIMEOUT_SECS) before it ages the close: the same wait the history-delete path takes since #35705.

Backwards compatibility

Compatible: only one test changes; no product code does.

Removes nothing: the change adds one wait to a test and takes no assertion, record or path away.

Tests

test/test_crew_log_retention.py::test_a_destroyed_sessions_log_is_then_collectable_by_the_sweep waits for the eager folder after destroy, then still asserts the sweep answers (1, 0) and the unit is gone.

Determinism proof (testing-conventions, Proving a determinism fix), on Ubuntu 24.04.5 LTS (Linux x86_64), Python 3.12.3, parent = c22557cd83:

Run Parent c22557cd83 This head
Forced condition (below), -n0, 5 runs 5 red: assert (0, 0) == (1, 0), remove_unit('session', 'acp-aged') -> 'owned' 5 green: seal hold seen: True, remove_unit('session', 'acp-aged') -> 'removed'
Unforced, -n0 30 runs: 30 green (a race; CI hits it, a quiet host rarely does) 20 runs: 20 green
-n 4 --dist loadgroup --max-worker-restart=2 with test/test_crew_log_fold_eager.py and test/test_crew_log_lease.py, 10 runs 10 green (116 passed each)
test/test_crew_log_retention.py shuffled, seed 20848 (62 ids) 62 passed
Call time of the changed test (--durations) 0.06 s 0.06 s

The forced condition is a throwaway -p plugin, not committed. It wraps the lease acquire inside checkpoint.save (the close-time seal): on the eager worker thread, once the acquire for the unit returns, it sleeps 1.5 s while still holding the lease, and the sweep's remove_unit for that unit first waits (bounded) until that hold exists, which is the order CI hit. The plugin logs both steps, so a run where the seal never held the lease cannot count. On this head the seal still held the lease on every run and the test waited it out; the asserts after the sweep ((1, 0), the unit gone) ran each time.

The flake-pattern ratchet and semgrep scan --config semgrep/ --error against the parent report nothing; prove.py answers NOTHING_TO_PROVE (test-only change).

Manual verification

N/A — test-only change; the forced condition above drives exactly the interleaving CI hit, run on Linux (Ubuntu 24.04).

Related Issues

Closes #35553

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

test_a_destroyed_sessions_log_is_then_collectable_by_the_sweep failed
intermittently with (0, 0). The session/closed that destroy writes
wakes the eager folder, which seals the unit's savepoint under a shared
hold of the unit's lease. When the sweep ran while that seal was still
in flight, its sole claim on the lease answered owned and nothing was
removed.

The test now waits for the eager folder to settle before it ages the
close, the same wait the history-delete path takes. The sweep itself is
unchanged: it collects only units whose close is older than the
retention window, and a unit it finds owned is collected on a later
pass.

Closes #35553
@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d3d49727091b29042ba2f7f58c266a98a22089e8 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d3d4972

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d3d49727091b29042ba2f7f58c266a98a22089e8 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision d3d49727091b29042ba2f7f58c266a98a22089e8 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d3d49727091b29042ba2f7f58c266a98a22089e8 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d3d49727091b29042ba2f7f58c266a98a22089e8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is test-only and waits on the real signal (the eager seal's lease hold) through the existing eager.drain, so it fixes the race at its cause. Product code is untouched.

[DESIGN-REVIEWED] d3d4972

Description read

[DESCRIPTION-READ] b8636191ead6f87236eefbdca60a5fcec06afe6f7421a73e71f43bd2e884c5b4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d3d49727091b29042ba2f7f58c266a98a22089e8 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d3d4972

Verdict parsed from the review's SHA-scoped output markers for commit d3d49727091b29042ba2f7f58c266a98a22089e8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d3d49727091b29042ba2f7f58c266a98a22089e8 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 14, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on d3d4972. This change fixes a flaky retention test. After a session is destroyed, the test now waits for the crew log's background folder to finish sealing the unit before it runs the sweep. The sweep assertions are unchanged: the sweep must still remove exactly one unit and the unit must be gone. The wait is on a real signal. It uses the eager folder's own counters, has a 60 second bound from the shared test helper, and gives a failure message. I reproduced the race on purpose. I held the seal's lease while the sweep claimed it. Without the new wait, the test failed three times out of three with the same (0, 0) seen in CI. With the wait, it passed three times out of three. The same race in production only delays collection: the sweep skips a unit whose lease is held, and the next hourly pass removes it. The history-delete path already waits the same way. Only this one test file changes. Non-blocking: the failure message could include the elapsed time and the folder's queued and settled counts, as the testing conventions ask. Decisive lines: test/test_crew_log_retention.py:1122-1124, test/test_crew_log_retention.py:1144-1145, src/kiro_crew/crew_log/eager.py:1160-1191, src/kiro_crew/crew_log/store_removal.py:103-106, src/kiro_crew/crew_log/store_removal.py:166-199, src/kiro_crew/dashboard/handlers/sessions_parts/slot_teardown.py:383-390

@bolichen97
bolichen97 merged commit 8b73a99 into main Nov 14, 2026
99 of 107 checks passed
@bolichen97
bolichen97 deleted the fix/35553-sweep-after-destroy-flake branch November 14, 2026 23:37
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_destroyed_sessions_log_is_then_collectable_by_the_sweep in test_crew_log_retention.py

2 participants