Skip to content

test(dashboard): wait on signals, not clocks -- dynamic_dashboard_cards, dashboard_updates_coverage, dashboard_chat - #19489

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-dashboard
Oct 13, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-dashboard

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The three dashboard test files no longer bet on the runner's speed: every fixed short wait in them now waits on the signal the code emits.

test/test_dynamic_dashboard_cards.py, test/test_dashboard_updates_coverage.py and test/test_dashboard_chat.py held 97 short fixed waits between them -- asyncio.wait_for(..., timeout=2..10) on an Event, a task or a bare coroutine, and polls of 10 to 200 short asyncio.sleep turns. Each one passes only while the CI runner is as fast as the machine the test was written on. #19248 landed the helpers (kiro_crew.testing.wait) and the per-file ratchet; this is the burn-down for batch E (dashboard).

Why it matters

A 5 s wait_for that loses on a slow Windows or macOS runner reds an unrelated pull request, and #19093, #19144, #19177 and #19205 were each one of these. test_dynamic_dashboard_cards.py alone was one of the larger holders of such waits under test/.

Not a goal

  • Changing any product code or src/kiro_crew/testing/wait.py.
  • Touching any test file outside the three named above, or any other line of test/wait_discipline_baseline.txt.
  • Removing the three asyncio.wait_for(finished.wait(), 60) sites in test_dashboard_chat.py: a 60 s ceiling is already at or above the 30 s lost-run floor, so the ratchet does not count them and they are left unchanged.

What changed (motivation → approach → change)

The ratchet counts a site when the wait's ceiling decides pass versus fail. The fix is to wait on what the test was actually waiting for and let the helper's lost-run bound (half the test's timeout) only name a hang.

Per file, counted sites before → after:

File Before After
test/test_dynamic_dashboard_cards.py 52 0
test/test_dashboard_updates_coverage.py 23 0
test/test_dashboard_chat.py 22 0

Each kind of wait, and the signal it now rides on:

  • asyncio.wait_for(event.wait(), N) → await await_until(event, ...): the Event the fake or test sets (entered / started / cancelling / cancelled / release in cards; restore_entered / swap_entered / seen in chat; the leader started in updates).
  • asyncio.wait_for(task_or_future, N) → await await_until(task, ...): the task or future the test created (service.worker, slot.task, gather(*state._background_tasks), gather(task, return_exceptions=True), old_worker / restarted, and restore_task / swap_task / pick_task / replay). Its result or exception propagates as before (the assert await ... == "fable-5" and candidate = await ... sites keep their value); pytest.raises(CancelledError) blocks are unchanged.
  • asyncio.wait_for(coroutine(), N) → await await_until(asyncio.ensure_future(coroutine()), ...): the helper takes an Event or future by contract, so shutdown(app), openai_compat._stream_response, openai_compat._blocking_response and updates._check_git_checkout are wrapped in a task first.
  • A poll (for _ in range(N): if cond: break; await asyncio.sleep(0.0x)) → await async_wait_until(predicate, describe=...) on the state the loop watched: the SSE/log writes growing, the _QueueLogHandler attach, the : keepalive frame, register_sse being called, ws.send_str being awaited, the racing queue's probe count, and -- with off_loop=True, as they read process state via platform_compat -- the git helper and git process exiting by recorded identity.
  • A poll on a lock's waiter queue → await until_parked(state._folders_lock, count=2): the two folder reparents parked on the store lock, a LoopBoundLock that until_parked reads directly (the hand-rolled _both_queued() is gone).

Two fakes park a generation/turn open until the code cancels it (wait_for(asyncio.Event().wait(), N)); these become a local _parked() that awaits the Event unbounded on purpose, because each such test already awaits the cancellation it expects (cancelling / cancelled / the worker) under the bound, so a never-cancelled park fails by name there.

_ShortGitWaits in test_dashboard_updates_coverage.py (which cut every git wait to a fixed 0.5 s) is renamed _TimedOutGitWaits and times the fetch out on a signal: it waits (off the loop) for the stand-in git to record its helper, or one loop turn when there is none, then cancels communicate() and raises TimeoutError exactly as the real wait_for does. The class's two process-exit polls move to async_wait_until(off_loop=True) on the recorded identity.

Remaining counted sites: none. No # flake-ok was added. Baseline diff: the three lines 52 test_dynamic_dashboard_cards.py, 23 test_dashboard_updates_coverage.py, 22 test_dashboard_chat.py are deleted from test/wait_discipline_baseline.txt; no other line moves.

Main drift on the ratchet gate (not this PR's sites): rebased onto origin/main after the F821 fixes merged (#19478, #19485). The ratchet's only remaining red is on two untouched files -- test/test_surface_deny_notice_sites.py (0 -> 1) and test/test_webhook_permission_requests.py (2 -> 3), owned elsewhere; it surfaces as the tree-scan inside Backend Tests (3.12, 2) (which cascades to Coverage Gate). git diff --name-only origin/main...HEAD lists only the three test files and the baseline.

Backwards compatibility

Compatible: test-only change; no product code, public function or configuration is touched.

Removes nothing user-facing. The only removals/renames are test-internal, each with no reader outside its own file:

  • Reader: test/test_dashboard_updates_coverage.py:_TimedOutGitWaits -- test-only -- renamed from _ShortGitWaits; grep of src/ and test/ and of every open PR diff finds no other user of the old name.
  • Reader: test/test_dashboard_chat.py -- test-only -- the local _both_queued() helper and two unused import asyncio as _asyncio statements are removed; both were used only by the waits this PR converted, and nothing else in the file references them.

Merge-order note

The one rename (_ShortGitWaits → _TimedOutGitWaits) and the removed _both_queued are both test-internal to a single file. Every currently open PR's diff was grepped for both names and none references either, so no alias is needed and no in-flight PR is affected.

Tests

Changed tests (all three files, 97 sites) keep every assertion; what each one locks in is unchanged. Determinism check on this host:

  • The three files together: 1178 passed, run three times, no failure.
  • python test/test_wait_discipline_ratchet.py reports 0 for the three files; --write-baseline touches only their three lines and, re-run after rebase, changes nothing further.
  • test/test_flake_pattern_ratchet.py (merge-base comparison): 53 passed.
  • Static gates on the touched files: black --check, isort --check-only and flake8 all exit 0.

Not a flaky-test FIX in the D12 sense (no forced-condition red is claimed): these waits had not yet lost on CI; the change removes the bet before they do.

Manual verification

N/A — unit coverage sufficient: the diff is the tests themselves.

Related Issues

Part of #19246

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 18:15
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 899d4ef18bc62769e716df8960b85a142cc701d3 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 899d4ef18bc62769e716df8960b85a142cc701d3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 899d4ef18bc62769e716df8960b85a142cc701d3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 899d4ef

Description read

[DESCRIPTION-READ] 4623fe9dbe9b0b9c5b06b798f054543797d934eaecd3d01610271e679b51f1fc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 899d4ef18bc62769e716df8960b85a142cc701d3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 899d4ef

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 899d4ef18bc62769e716df8960b85a142cc701d3: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 899d4ef18bc62769e716df8960b85a142cc701d3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 899d4ef

Verdict parsed from the review's SHA-scoped output markers for commit 899d4ef18bc62769e716df8960b85a142cc701d3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 899d4ef18bc62769e716df8960b85a142cc701d3: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-dashboard branch from 6dfa5c2 to 4ea76b2 Compare October 12, 2026 18:54
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-dashboard branch from 4ea76b2 to 3515290 Compare October 12, 2026 19:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
buluoray
buluoray previously approved these changes Oct 12, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The fixed-time waits in test_dashboard_chat.py, test_dashboard_updates_coverage.py and test_dynamic_dashboard_cards.py now wait on what the code under test emits: await_until on events and tasks, async_wait_until in place of the hand-rolled sleep/poll loops, until_parked for the folder-lock waiter count, and a probe-count predicate instead of sleep(0.05). Every new wait is bounded by default_timeout(); the intentionally unbounded _parked() is always ended by the code under test's own cancellation, which is then awaited under a bound. No assertion was weakened, and the helper-record wait in the git-timeout tests now fails by name instead of quietly continuing after 5 s. No product code changed; the baseline only drops the three touched files.

Verified on this head: the ratchet passes (39) and the three files pass (1178) under -n 4 --dist loadgroup. Removing the helper-record signal in _TimedOutGitWaits reddens test_a_fetch_timeout_returns_while_a_helper_holds_stderr; dropping the entered wait reddens both cases of test_one_inflight_latest_event_survives_and_removal_wins.

Non-blocking nits:

  • The _TimedOutGitWaits docstring says the no-helper path times out "once communicate() has started", but one sleep(0) does not guarantee that.
  • _parked() relies on every caller awaiting its cancellation under a bound, and only its docstring says so.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 12, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-dashboard branch from 3515290 to e075249 Compare October 13, 2026 00:48
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebase-only update (head e0752496fa): the force-push resolves a conflict in test/wait_discipline_baseline.txt after main moved. No test logic changed — the only conflict was that baseline file, and its diff vs origin/main still removes exactly my three lines (test_dashboard_chat.py, test_dashboard_updates_coverage.py, test_dynamic_dashboard_cards.py); no other file's count was re-added or lowered. Ratchet gate green locally, and all three files pass (1178 tests). Re-requesting review since the push auto-dismissed the prior approval.

@CrysisDeu
CrysisDeu requested a review from buluoray October 13, 2026 01:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 13, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR swaps 97 fixed short waits in three dashboard test files for waits on the signal the code under test emits. The cards and chat conversions are faithful, and no product code changes. One conversion makes a test unable to prove what it exists to prove, and it is still unanswered: the current-head Opus finding at test/test_dashboard_updates_coverage.py:405 has no disposition. I checked it against the head. setup.cfg sets --timeout=120, so default_timeout() is 60 s. The stand-in git and its helper in _HELPER_GIT (lines 276 and 279) both run time.sleep(60) and start before the wait does. So if the timeout or cancel path stopped killing the process group, the _gone() waits in test_a_fetch_timeout_returns_while_a_helper_holds_stderr and test_a_cancelled_fetch_ends_git_and_its_helper would still see both processes exit on their own before the deadline, and helper_exited / git_exited would pass. The old 5 s bound caught that regression. Suggested fix: make the stand-in sleeps much longer than the lost-run bound (for example time.sleep(600); the finally blocks already kill survivors by identity), so only a real kill can satisfy the wait, then check it with a mutant that removes the group kill. Separately, the no-helper _TimedOutGitWaits branch's one sleep(0) does not guarantee communicate() has started, so fix the docstring or wait on a real signal. — crew-pr-reviewer (deep re-check)

…ds, dashboard_updates_coverage, dashboard_chat
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-dashboard branch from e075249 to 899d4ef Compare October 13, 2026 02:55
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Addressed the blocking review on head 899d4ef18b (test-only; no product code).

What changed (both in test/test_dashboard_updates_coverage.py):

  1. _HELPER_GIT: the stand-in git and its spawned helper now time.sleep(600) instead of time.sleep(60). setup.cfg sets --timeout=120, so default_timeout() is 60 s; a 60 s sleep let both processes self-exit right at the lost-run bound, so test_a_fetch_timeout_returns_while_a_helper_holds_stderr and test_a_cancelled_fetch_ends_git_and_its_helper could pass even if the kill path stopped killing the group. At 600 s only a real group-kill can satisfy the _gone() / _both_gone() waits within the bound. The finally blocks already SIGKILL any survivor by its recorded identity, so nothing leaks.
  2. _TimedOutGitWaits no longer has a sleep(0) branch. It now always waits on a real start signal — the record the stand-in git writes ({pidfile} for its helper, else {pidfile}.git for git's own pid), read off the loop — before it cancels communicate() and raises TimeoutError. git writes that record only once it is running and communicate() is reading, which is exactly what the old sleep(0) could not guarantee. Docstring updated to match.

Mutant check (as requested): I replaced the fetch timeout/cancel await platform_compat.kill_and_reap(proc) with a no-op in src/kiro_crew/dashboard/handlers/updates.py and ran the two tests — both go red (fail by name via async_wait_until, after the 60 s bound), then both pass once the kill is restored. Reverted; updates.py is unchanged in this PR.

Disposition — ai-review (Opus) finding @ test/test_dashboard_updates_coverage.py:405: addressed by changes (1) and (2) above. The conversion no longer weakens the regression the test exists to prove: the stand-in outlives the lost-run bound, so the _gone() waits can only be satisfied by a real process-group kill, and the mutant above demonstrates the test reddens when that kill is removed.

Local: test_wait_discipline_ratchet.py::test_no_file_gains_a_short_fixed_wait passes; the three files pass (1178). Baseline still removes only the three lines. Re-requesting review.

@CrysisDeu
CrysisDeu requested a review from iamwhatever October 13, 2026 02:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 13, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE at 899d4ef18b

What it fixes

Fixed short sleeps in three dashboard test files flaked under load. This PR replaces them with waits on what the code under test emits. Every wait is bounded by default_timeout(). Only tests and the wait-discipline baseline change; no product code does.

What I verified

Since my approval on 35152906e2, the only change apart from a rebase is in test/test_dashboard_updates_coverage.py. It answers both of iamwhatever's points on e0752496fa:

  1. A slow exit could satisfy the kill wait — fixed. In _HELPER_GIT, the stand-in git and its helper now sleep for 600 s (test/test_dashboard_updates_coverage.py:276, :279). That is ten times the 60 s default_timeout() that --timeout=120 gives (setup.cfg:450). The _gone() waits at :407 and :475 can now pass only if the processes are actually killed. The finally blocks still kill any survivor by its pinned start-time identity (:418, :484).
  2. The no-helper branch relied on one sleep(0) — fixed. _TimedOutGitWaits now always waits on a real record: the helper's pid when there is a helper, otherwise git's own .git record (:311-325, :349). ensure_future(aw) schedules communicate() before that wait, so the cancel always hits a started read.

Mutation: I replaced the tree kill in kill_and_reap (src/kiro_crew/platform_compat.py:7299, await kill_process_tree_async(...)) with pass. That reddens test_a_fetch_timeout_returns_while_a_helper_holds_stderr, test_a_cancelled_fetch_ends_git_and_its_helper and test_the_timeout_tests_cleanup_signals_no_helper_it_saw_exit. No stand-in process survived the run, and the tree was clean afterwards.

Suites at this head:

  • The three touched files: 1178 passed with -n 4 --dist loadgroup.
  • The wait-discipline and flake-pattern ratchets: 92 passed.

Blocking

None.

Non-blocking follow-ups

  • The _TimedOutGitWaits docstring says the stand-in writes its record "only after … communicate() is reading". The record doesn't guarantee that. The ordering comes from ensure_future scheduling communicate() before the wait yields. A wording fix only.
  • _parked() still depends on every caller awaiting its cancellation under a bound, and only its docstring says so (carried over from my previous review).

Not verified

  • I did not re-run iamwhatever's counterfactual, where the old 60 s sleep plus a missing group kill stays green. The point rests on the timing arithmetic above.
  • I did not re-check the rest of the rebased diff line by line. I relied on git range-diff showing no changes outside this file, apart from context.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR replaces 97 fixed short waits in three dashboard test files with waits on the signal the code under test emits (await_until on events and tasks, async_wait_until for polls, until_parked for the folder lock), and drops those three files from the wait-discipline baseline; no product code changes. I re-read the full diff at 899d4ef against both points of our earlier change request on e075249. Both are fixed: the stand-in git and its helper in _HELPER_GIT now sleep 600 s (test/test_dashboard_updates_coverage.py:276/279), ten times the 60 s default_timeout, so the _gone() waits can only pass if the timeout/cancel path really kills the process group. _TimedOutGitWaits also now always waits on a real record (the helper pid, or git's own .git record when there is no helper) instead of one sleep(0). I checked that every later wait_for in updates._check_git_checkout sits after the first fetch timeout returns, so the fake raising TimeoutError on every call does not mask a different path. The cards and chat conversions keep every assertion and every result propagation, and the intentionally unbounded _parked() is always ended by a cancellation the test awaits under a bound. Checks are green and the base is main, so the only remaining block was our own review. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever merged commit 68fbf44 into main Oct 13, 2026
148 of 149 checks passed
@iamwhatever
iamwhatever deleted the unflaky/wait-migrate-dashboard branch October 13, 2026 17:19
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants