Skip to content

test: remove crew-log units with the eager folder held in test_crew_log_emit - #16459

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/crew-log-deferral-teardown
Oct 5, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/crew-log-deferral-teardown

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_predecessor_collected_during_the_deferral_costs_no_dropped_write stops failing with Directory not empty when the eager folder is still folding the unit the test removes.

The test fails intermittently on Backend Tests (3.12, 4) with OSError: [Errno 39] Directory not empty from its own shutil.rmtree of a crew-log session folder. It has turned shards red on several PRs that touch no crew-log code (#15596, #16387, #16034, #16039, #14457, #15957). Details and the reproduction are in #16458.

Why it matters

A red backend shard blocks readiness on whatever unrelated PR it lands on, and Coverage Gate fails closed behind it. Each hit costs a rerun and an investigation.

Not a goal

  • Changing product code. Retention removes a unit holding its lease sole and inside eager.paused(), so the eager folder cannot be mid-batch while the unit's files go. Only the test's bare removal skips that.
  • Reworking crew-log tests in other files.

What changed (motivation → approach → change)

Symptom the test's shutil.rmtree of the unit folder fails with Directory not empty
Cause the eager folder lists the unit's segments, then store._open_lock recreates .lock inside the folder the rmtree is emptying
Change the test removes the unit the way retention does: drain the eager folder, then remove inside eager.paused()

Each committed entry wakes the eager folder, and it folds on its own thread after emit.flush() returns. The fold opens the unit through CrewLog.open, which lists the segments and then takes the append lock. Taking the lock creates the folder and .lock if they are missing. A fold that lists the segments before the rmtree and takes the lock during it puts .lock back, and the last rmdir fails.

A new _remove_as_retention_does() helper in test/test_crew_log_emit.py calls eager.drain() and runs the removal inside eager.paused(), asserting both. It is the pattern #16130 uses. All three bare removals in the file go through it: the failing shutil.rmtree, and two log.jsonl unlinks in test_a_resume_of_the_same_store_writes_no_previous_edge and test_a_store_whose_front_retention_removed_reports_no_edge. Those two race the same thread and hit WinError 32 on Windows. The testing-conventions rule from #16130 now names shutil.rmtree and the POSIX symptom too.

Backwards compatibility

Compatible: test-only change plus one documentation sentence. No product code is touched.

Removes nothing: the diff adds one test helper and routes three existing removals through it.

Tests

  • The three tests keep every assertion. Each now removes the unit's files with the eager folder settled and held.

Proof that the race is the cause and the hold removes it: a local harness forces the interleaving. It holds the eager thread at its first lock creation for the unit, starts the test's rmtree only once the fold is held, and releases the fold when the rmtree reaches the unit folder's rmdir. On main f16fa23 it fails the test with the CI error 3 of 3 runs. On this branch the same harness passes 3 of 3, because the drain makes the held fold finish before the removal starts.

Also run on this branch:

  • pytest test/test_crew_log_emit.py on the head rebased onto main af1eb61: 316 passed
  • the three changed tests 5 times in a row, single process: all passed
  • black, isort and flake8 on the changed test file: clean
  • mypy on the changed test file: the same 6 errors as main, none on changed lines
  • scripts/check_comment_history.py against origin/main and scripts/docs_lint.py: passed

Manual verification

N/A: test-only change, covered by the runs above.

Related Issues

Closes #16458

Pattern harvest

Rule candidate: review-prompt
Pattern: a test removes crew-log files with a bare unlink() or shutil.rmtree instead of draining the eager folder and holding eager.paused().

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested a review from a team as a code owner October 3, 2026 09:50
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 56664096cf6d98f8ced4c3c421ccdb62eab5f7e1 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 5666409

Description read

[DESCRIPTION-READ] a809e0e1a8a7eea9cc6860e3aa984b1c35df8f02f596e5e1cadda973c8296366

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 56664096cf6d98f8ced4c3c421ccdb62eab5f7e1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5666409

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 56664096cf6d98f8ced4c3c421ccdb62eab5f7e1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5666409

@jeeshofone
jeeshofone force-pushed the fix/crew-log-deferral-teardown branch from 1c3e9ba to e67ff8b Compare October 5, 2026 09:27
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 5, 2026
…og_emit

test_a_predecessor_collected_during_the_deferral_costs_no_dropped_write removed a
crew-log unit folder with a bare shutil.rmtree while the eager folder could still
be folding it on its own thread. A fold that listed the segments before the rmtree
recreates the unit's .lock inside the folder, so the final rmdir failed with
Directory not empty. Retention removes a unit inside eager.paused(); the three bare
removals in this file now drain the folder and hold it the same way.

Closes kirodotdev#16458
@jeeshofone
jeeshofone force-pushed the fix/crew-log-deferral-teardown branch from e67ff8b to 5666409 Compare October 5, 2026 10:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 5, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a test-only change and it makes the tests stricter. Each of the three removals now runs the way retention does: wait for the eager folder to settle, hold it, then remove. That closes the race where a fold recreates .lock inside the folder being deleted (ENOTEMPTY on POSIX, WinError 32 on Windows). The tests' own assertions are unchanged. The helper adds checks that the folder settled and the hold was granted, so a busy host shows up as a clear failure, not a hidden race. The testing-conventions doc now names the rmtree case too. All checks are green. Approving.

For all three, I read the head SHA's source through gh api, the full diff and every existing review comment. I posted or changed nothing. All required checks pass or are skipped. Each PR has all-PASS AI verdicts and still needs a human review. The @kirocrew-computer MCP server is declared in the agent spec but not configured, so it was unavailable; this review didn't need it.

@iamwhatever
iamwhatever merged commit 6a31a63 into kirodotdev:main Oct 5, 2026
101 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_crew_log_emit: a bare rmtree of a crew-log unit races the eager folder (Directory not empty)

2 participants