Skip to content

fix(test): make the on_done wait causal in test_workflows_service - #11759

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
dimwael:fix/workflows-service-windows-flakes
Sep 18, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
dimwael:fix/workflows-service-windows-flakes

Conversation

@dimwael

@dimwael dimwael commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

test/test_workflows_service.py::test_start_launches_run_and_injects_on_done fails intermittently on Windows backend shards with assert ([]) — the on_done callback list is empty when the test asserts on it. It fails across PRs whose diffs cannot reach the workflows service (#10955, #10793 sightings in #10964), while the same shards' Linux and macOS siblings pass. It is the last member of #10964's tracked set still exposed on main: the two registration/cancellation tests were fixed by #11115, and the busy-slot sibling by #11456.

Why it matters

Every author whose PR lands on the affected Windows shard reads a red that their change cannot cause, and either burns a rerun or starts debugging code they never touched. #10964 documents this happening repeatedly across unrelated PRs.

What changed (motivation → approach → change)

Observed symptom: done is empty at the assert, only on loaded Windows runners.

Root cause: the driver's terminal path (RunRegistry.mark_terminal_async) runs in causal order — RAM terminal transition, then persist_async (a full-record serialize plus fsync-bound atomic write on a worker thread), then _notify_done (the on_done callback). The test observes only the first step (_wait_terminal polls RAM status) and covers the remaining two with a wall clock: originally asyncio.sleep(0.02), later wait_for(notified.wait(), 3.0) (#10841). A loaded Windows disk stretches the flush past any fixed budget, so the test asserts before the callback fires. This is the same wall-clock-under-load shape #11115 removed from the registration tests.

Change: wait on the causal signal instead. _wait_durable_terminal awaits the driver task itself, which settles only after the callback has fired — semantics already pinned by test_durable_terminal_wait_includes_the_completion_callback (added in #11456). Also raise that helper's internal cap from a 3-second clock to the file's existing 30-second _HANG_GUARD_SECS: the cap is a hang guard, not the assertion (per the constant's own documentation), it stays well under the 120-second pytest-timeout, and #10964's dataset shows a single fsync chain exhausting a 1-second budget on a loaded runner — a whole run plus terminal flush under 3 seconds is the same exposure.

Every existing assertion is kept, and the test now asserts the callback ordering directly (notified.is_set() after the durable wait) — strictly stronger than the wall-clock version.

Backwards compatibility

Compatible: test-only change; no production code or contract is touched. The helper's larger cap changes behavior only for a wait that would otherwise have timed out, and no caller depends on the timeout firing.

Tests

This PR changes tests. test_start_launches_run_and_injects_on_done now locks in that the driver task settles only after the durable flush and the on_done callback, with no wall-clock dependency. The callback-before-settle ordering it relies on is already locked in by test_durable_terminal_wait_includes_the_completion_callback.

Manual verification

Verified by reading the driver path (RunRegistry.mark_terminal_async → persist_async → _notify_done) and the helper it reuses; the file parses (ast.parse) and is black-clean (--target-version py310, line length 100). My sandbox has no Python ≥3.12 and no Windows runtime, so I could not execute the test locally — CI is the executable verification, and because the failure is intermittent a green run should be read together with the mechanism argument above, not as standalone proof.

Related Issues

Fixes #10964

Pattern harvest

Rule candidate: review-prompt
Pattern: test observes an intermediate state (RAM-terminal) and covers the remaining causal chain (flush, callback) with a fixed wall-clock wait — wait on the terminal causal signal (task settle, event set by the last step) instead.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@dimwael
dimwael requested review from a team and buluoray September 18, 2026 09:40
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 26791c0fa855c4dffcad8d7a1abc5bc4197ab113 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 26791c0

test_start_launches_run_and_injects_on_done polls RAM-terminal status
and then covers the durable flush and result-to-chat callback with a
3-second wall clock. The driver's terminal path runs the RAM transition,
then an fsync-bound flush on a worker thread, then the on_done callback,
so a loaded Windows disk stretches the flush past any fixed budget and
the test reads an empty callback list.

Wait on the causal signal instead: _wait_durable_terminal awaits the
driver task, which settles only after the callback has fired. Raise that
helper's internal cap from a 3-second clock to the file's 30-second
_HANG_GUARD_SECS: the cap is a hang guard, not the assertion, and stays
well under the 120-second pytest-timeout. Every assertion is kept, and
the callback ordering is now asserted directly via notified.is_set().

Fixes kirodotdev#10964
@dimwael
dimwael force-pushed the fix/workflows-service-windows-flakes branch from 4cdcfcb to 26791c0 Compare September 18, 2026 09:45
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 26791c0fa855c4dffcad8d7a1abc5bc4197ab113 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify #10964 actually goes quiet on Windows shards post-merge — one green run cannot prove an intermittent flake is gone.

I confirmed the causal claim in the base source: the driver task settles only after mark_terminal_async runs persist_async then _notify_done synchronously (registry.py:493-508), so assert notified.is_set() after _wait_durable_terminal is causal, not racy. The replaced wait was the last wall-clock cover in the file's tracked flake set, and the new assertion is strictly stronger than the one it replaces.

[DESIGN-REVIEWED] 26791c0

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 26791c0fa855c4dffcad8d7a1abc5bc4197ab113 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 26791c0

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 18, 2026 11:07

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change that replaces a wall-clock sleep with a causal wait on the driver's terminal path, so the Windows-shard flake in test_workflows_service can no longer assert before the on_done callback has run. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.

@iamwhatever
iamwhatever merged commit 87bf836 into kirodotdev:main Sep 18, 2026
82 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…rodotdev#11759)

test_start_launches_run_and_injects_on_done polls RAM-terminal status
and then covers the durable flush and result-to-chat callback with a
3-second wall clock. The driver's terminal path runs the RAM transition,
then an fsync-bound flush on a worker thread, then the on_done callback,
so a loaded Windows disk stretches the flush past any fixed budget and
the test reads an empty callback list.

Wait on the causal signal instead: _wait_durable_terminal awaits the
driver task, which settles only after the callback has fired. Raise that
helper's internal cap from a 3-second clock to the file's 30-second
_HANG_GUARD_SECS: the cap is a hang guard, not the assertion, and stays
well under the 120-second pytest-timeout. Every assertion is kept, and
the callback ordering is now asserted directly via notified.is_set().

Fixes kirodotdev#10964

Co-authored-by: Wael Dimassi <ydimawae@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_workflows_service.py fails on Windows shards across unrelated PRs

2 participants