Repository navigation
fix(test): make the on_done wait causal in test_workflows_service - #11759
Conversation
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
test_start_launches_run_and_injects_on_done polls RAM-terminal status and then covers the durable flush and result-to-chat callback with a 3-second wall clock. The driver's terminal path runs the RAM transition, then an fsync-bound flush on a worker thread, then the on_done callback, so a loaded Windows disk stretches the flush past any fixed budget and the test reads an empty callback list. Wait on the causal signal instead: _wait_durable_terminal awaits the driver task, which settles only after the callback has fired. Raise that helper's internal cap from a 3-second clock to the file's 30-second _HANG_GUARD_SECS: the cap is a hang guard, not the assertion, and stays well under the 120-second pytest-timeout. Every assertion is kept, and the callback ordering is now asserted directly via notified.is_set(). Fixes kirodotdev#10964
4cdcfcb to
26791c0
Compare
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify #10964 actually goes quiet on Windows shards post-merge — one green run cannot prove an intermittent flake is gone. I confirmed the causal claim in the base source: the driver task settles only after [DESIGN-REVIEWED] 26791c0 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change that replaces a wall-clock sleep with a causal wait on the driver's terminal path, so the Windows-shard flake in test_workflows_service can no longer assert before the on_done callback has run. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
…rodotdev#11759) test_start_launches_run_and_injects_on_done polls RAM-terminal status and then covers the durable flush and result-to-chat callback with a 3-second wall clock. The driver's terminal path runs the RAM transition, then an fsync-bound flush on a worker thread, then the on_done callback, so a loaded Windows disk stretches the flush past any fixed budget and the test reads an empty callback list. Wait on the causal signal instead: _wait_durable_terminal awaits the driver task, which settles only after the callback has fired. Raise that helper's internal cap from a 3-second clock to the file's 30-second _HANG_GUARD_SECS: the cap is a hang guard, not the assertion, and stays well under the 120-second pytest-timeout. Every assertion is kept, and the callback ordering is now asserted directly via notified.is_set(). Fixes kirodotdev#10964 Co-authored-by: Wael Dimassi <ydimawae@amazon.com>
Problem / Motivation
test/test_workflows_service.py::test_start_launches_run_and_injects_on_donefails intermittently on Windows backend shards withassert ([])— theon_donecallback list is empty when the test asserts on it. It fails across PRs whose diffs cannot reach the workflows service (#10955, #10793 sightings in #10964), while the same shards' Linux and macOS siblings pass. It is the last member of #10964's tracked set still exposed on main: the two registration/cancellation tests were fixed by #11115, and the busy-slot sibling by #11456.Why it matters
Every author whose PR lands on the affected Windows shard reads a red that their change cannot cause, and either burns a rerun or starts debugging code they never touched. #10964 documents this happening repeatedly across unrelated PRs.
What changed (motivation → approach → change)
Observed symptom:
doneis empty at the assert, only on loaded Windows runners.Root cause: the driver's terminal path (
RunRegistry.mark_terminal_async) runs in causal order — RAM terminal transition, thenpersist_async(a full-record serialize plus fsync-bound atomic write on a worker thread), then_notify_done(theon_donecallback). The test observes only the first step (_wait_terminalpolls RAM status) and covers the remaining two with a wall clock: originallyasyncio.sleep(0.02), laterwait_for(notified.wait(), 3.0)(#10841). A loaded Windows disk stretches the flush past any fixed budget, so the test asserts before the callback fires. This is the same wall-clock-under-load shape #11115 removed from the registration tests.Change: wait on the causal signal instead.
_wait_durable_terminalawaits the driver task itself, which settles only after the callback has fired — semantics already pinned bytest_durable_terminal_wait_includes_the_completion_callback(added in #11456). Also raise that helper's internal cap from a 3-second clock to the file's existing 30-second_HANG_GUARD_SECS: the cap is a hang guard, not the assertion (per the constant's own documentation), it stays well under the 120-secondpytest-timeout, and #10964's dataset shows a single fsync chain exhausting a 1-second budget on a loaded runner — a whole run plus terminal flush under 3 seconds is the same exposure.Every existing assertion is kept, and the test now asserts the callback ordering directly (
notified.is_set()after the durable wait) — strictly stronger than the wall-clock version.Backwards compatibility
Compatible: test-only change; no production code or contract is touched. The helper's larger cap changes behavior only for a wait that would otherwise have timed out, and no caller depends on the timeout firing.
Tests
This PR changes tests.
test_start_launches_run_and_injects_on_donenow locks in that the driver task settles only after the durable flush and theon_donecallback, with no wall-clock dependency. The callback-before-settle ordering it relies on is already locked in bytest_durable_terminal_wait_includes_the_completion_callback.Manual verification
Verified by reading the driver path (
RunRegistry.mark_terminal_async→persist_async→_notify_done) and the helper it reuses; the file parses (ast.parse) and is black-clean (--target-version py310, line length 100). My sandbox has no Python ≥3.12 and no Windows runtime, so I could not execute the test locally — CI is the executable verification, and because the failure is intermittent a green run should be read together with the mechanism argument above, not as standalone proof.Related Issues
Fixes #10964
Pattern harvest
Rule candidate: review-prompt
Pattern: test observes an intermediate state (RAM-terminal) and covers the remaining causal chain (flush, callback) with a fixed wall-clock wait — wait on the terminal causal signal (task settle, event set by the last step) instead.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement