Skip to content

test(crew-log): wait by progress in the write-path and exhaustion tests - #32339

Merged
bolichen97 merged 1 commit into
mainfrom
fix/23740-crew-log-write-path-drain
Nov 7, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/23740-crew-log-write-path-drain

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The crew-log write-path and exhaustion tests wait for their writes by writer progress, so a slow disk can no longer fail them.

61 read-back waits in test/test_crew_log_emit_write_path.py and test/test_crew_log_edge_exhaustion.py still give the writer a fixed emit.flush() budget to drain before the test reads the log: 5 s by default, 10 s or 20 s where set. Each append is an fsync under a cross-process lock, so any fixed budget is a rate assertion on the host's disk. On a loaded runner the entries can still be on their way when it runs out, and the test fails with nothing wrong.

The 11-05 status comment on #23740 left these two files to #27515. That PR merged but moved only its three held-burst waits, so these are the last plain read-back waits the issue covers.

Why it matters

A red from a fixed flush budget costs a rerun and hides real failures among host-speed ones. #23639 was one of these.

Not a goal

  • The two waits in test/test_crew_log_emit_write_path.py whose message names what the test checks: "the writer claim outlived the pass it was made for" (flush must answer True once the claim is released) and "the writer never finished giving up on the entry" (flush must end against a store that keeps refusing). Those keep calling flush(), as the issue asks.
  • The assert not emit.flush(timeout=0.5) checks in both files: they assert a batch is retained, which a progress barrier cannot say.
  • The sites left on purpose elsewhere (named in the 11-05 comment): test_crew_log_edge_concurrency.py's flush(timeout=0.0), test_crew_log_order.py's flake-ok wait, fixtures/crew_log_crash_child.py, the ratchet's own sample text, and the waits whose message names their subject.
  • Any change to emit.flush(), test/crew_log_drain.py, or production code.

What changed (motivation → approach → change)

Each of the 61 waits now calls crew_log_drain.assert_drained(), which these files already import. It fails only when the writer lands nothing in a session log for a whole 15 s window, or at its 60 s ceiling under the suite's 120 s timeout.

Old budget Waits
5 s (the default) 42
10 s 1
20 s 18

git grep -cE 'assert [a-z_]*emit\.flush\(' -- test/ goes from 74 to 13: 53 to 2 in the write-path file and 10 to 0 in the exhaustion file.

Every converted site waits for the writer before the test reads the file, a counter or a pin; none tests flush() itself. Waits inside a coroutine call the barrier directly on the loop, holding it as flush() did, as the three held-burst waits #27515 moved already do.

One trade, stated plainly: at the 18 former 20 s sites, a single append that stalls for 15 to 20 s with nothing else landing now fails, where it used to pass. A drain that keeps landing entries now gets up to 60 s instead of 20 s. test/crew_log_drain.py sizes that 15 s window at about 13 times the measured mean append on a loaded Windows shard.

Backwards compatibility

Compatible: test-only. No production code, fixture or helper changes.

Removes nothing: only the barrier call inside existing tests changes; every test and assertion stays.

Tests

The changed files are the tests. Runs on head 516be60fc2cb, the same patch as this head (patch-id db3237d64de2; the rebase onto newer main touches none of these files, test/crew_log_drain.py, test/crew_log_emit_helpers.py or src/kiro_crew/crew_log/). Linux, on a shared host under heavy disk load (IO pressure some avg300 70-75% throughout):

Run Result
Shuffled node ids at -n0, seeds 23740 and 4127 2/2, 53 passed each
20 x -n 4 --dist loadgroup --max-worker-restart=2 on both files, two batches of 10 18/20. Run 1: two tests on two different workers, test_every_terminal_path_releases_its_pin_after_a_definitive_drop[completed] and test_overflow_while_the_creating_record_is_queued_still_creates_the_log, both failed landed nothing for 15.0s in the same 80 s, while inline writes stalled too (test_a_write_that_finishes_late_sheds_nothing took 38 s against 20 s in every other run). Run 19: test_a_retained_batch_is_written_at_shutdown_not_abandoned failed at assert emit.drain_for_shutdown(timeout=20.0), a line this diff does not change: one entry did not land in 20 s. So this host's disk stalls, past 20 s at times; run 1's two reds were former 20 s sites, the trade named above
20 x -n0 on both files 20/20, 53 passed each
Both files at origin/main, 10 x -n 3, same host 10/10
Forced condition: a throwaway plugin, not committed, sleeps 6 s before every CrewLog.append / append_many; 7 tests origin/main: 4 failed, 3 at assert emit.flush() and 1 reading entries an earlier test on its worker left queued when its budget ran out. This head: 7/7
The 4 related files that name these two in their docstrings 260 passed
test/test_flake_pattern_ratchet.py 68 passed
semgrep scan --config semgrep/ on the two files 1 finding, on an untouched line (test-autouse-shared-monkeypatch, the exhaustion file's fixture at line 30); none on a changed line

The forced tests: test_an_append_never_runs_on_the_event_loop_thread, test_a_turn_boundary_crossing_a_full_queue_cannot_mis_attribute_an_entry, test_a_failed_append_is_retained_and_lands_on_the_next_pass, test_a_dropped_turn_start_leaves_its_turn_unthreaded_not_folded_into_the_last, test_a_later_write_during_retention_queues_behind_the_retained_batch, and both test_cleared_error_lands_entries_in_order_with_contiguous_seq cases.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: the change is the test barrier itself, run as above.

Screenshots / video

N/A — test-only; no frontend surface is touched.

Related Issues

Closes #23740

Sibling sweep

Swept: git grep -nE "assert [a-z_]*emit.flush(" -- test/
Sibling: test/test_crew_log_emit_write_path.py -- changed here
Sibling: test/test_crew_log_edge_exhaustion.py -- changed here
Sibling: test/test_crew_log_edge_concurrency.py -- not affected -- its flush(timeout=0.0) is the subject of the test
Sibling: test/test_crew_log_order.py -- not affected -- its flake-ok tag records a chosen 20 s wait
Sibling: test/fixtures/crew_log_crash_child.py -- not affected -- runs in a child process with its own message
Sibling: test/test_crew_log_emit.py -- not affected -- its message says the walk terminating is the check
Sibling: test/test_crew_log_slot_index_listing_cache.py -- not affected -- its message names the board append it checks
Sibling: test/test_crew_log_writer_group_commit.py -- not affected -- waits default_timeout(), not a fixed budget
Sibling: test/test_flake_pattern_ratchet.py -- not affected -- the ratchet's own sample text
Sibling: test/crew_log_drain.py -- not affected -- a docstring naming the pattern
Sibling: test/test_change_cards_settlement.py -- not affected -- a docstring naming the pattern

Pattern harvest

Not generalizable: the K13 ratchet in test/test_flake_pattern_ratchet.py already counts fixed flush ceilings and lets the count only fall.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 7, 2026 13:02
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ab622ee14a2ecfdb5eae8234e01117b341168688 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ab622ee

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ab622ee14a2ecfdb5eae8234e01117b341168688 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 7, 2026 13:14
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 7, 2026
The read-back waits in these two files gave the writer a fixed flush
budget (5 s by default, 10 s or 20 s where set) to drain before the test
read the log. Each append is an fsync under a cross-process lock, so a
fixed budget asserts the host's disk speed. assert_drained() fails only
when the writer lands nothing for a whole window, or at its 60 s
ceiling. The two waits whose message names what the test checks keep
calling flush() directly.
@bolichen97
bolichen97 force-pushed the fix/23740-crew-log-write-path-drain branch from bd31ccf to ab622ee Compare November 7, 2026 15:58
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The Backend Tests (3.12, 7) red on bd31ccf was test_platform_collect_ignore.py::test_the_windows_list_ignores_exactly_the_listed_test_files, which failed on every leaf-test run on main until #32209 (closes #32199) merged at 13:07Z, four minutes after this head's run started. Rebased onto main past that fix as ab622ee; the diff is unchanged (same patch id), and CI, including the review lanes that failed closed on the earlier API rate limit, is re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ab622ee14a2ecfdb5eae8234e01117b341168688 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ab622ee14a2ecfdb5eae8234e01117b341168688 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ab622ee14a2ecfdb5eae8234e01117b341168688 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is test-only and moves 61 fixed flush budgets onto the existing progress barrier. The one tradeoff, where the 18 former 20 s sites now fail on a single stall of 15–20 s, is stated and can be reverted.

Notes

  • Most converted sites still pay the full fsync on every append. The suite's own unsync_appends helper takes it out and would shrink both the time spent and the 15–20 s stall risk at the tests that check records rather than durability. That would be a follow-up PR, not part of this one.

[DESIGN-REVIEWED] ab622ee

Description read

[DESCRIPTION-READ] 34072e8d10f8a41a3763ef5e73b51211fb5f4bbf18b88ea427640c81d8b360c4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ab622ee14a2ecfdb5eae8234e01117b341168688 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ab622ee

Verdict parsed from the review's SHA-scoped output markers for commit ab622ee14a2ecfdb5eae8234e01117b341168688.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ab622ee14a2ecfdb5eae8234e01117b341168688 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: crew-log write-path and exhaustion tests wait on progress instead of fixed timing; test files only.

@bolichen97
bolichen97 merged commit c0a4fd9 into main Nov 7, 2026
98 checks passed
@bolichen97
bolichen97 deleted the fix/23740-crew-log-write-path-drain branch November 7, 2026 17:09
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests: move crew-log read-back barriers off the fixed emit.flush() budget

2 participants