Repository navigation
test(dashboard): wait on signals, not clocks -- dynamic_dashboard_cards, dashboard_updates_coverage, dashboard_chat - #19489
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 899d4ef Description read[DESCRIPTION-READ] 4623fe9dbe9b0b9c5b06b798f054543797d934eaecd3d01610271e679b51f1fc That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
6dfa5c2 to
4ea76b2
Compare
4ea76b2 to
3515290
Compare
buluoray
left a comment
There was a problem hiding this comment.
Approving. The fixed-time waits in test_dashboard_chat.py, test_dashboard_updates_coverage.py and test_dynamic_dashboard_cards.py now wait on what the code under test emits: await_until on events and tasks, async_wait_until in place of the hand-rolled sleep/poll loops, until_parked for the folder-lock waiter count, and a probe-count predicate instead of sleep(0.05). Every new wait is bounded by default_timeout(); the intentionally unbounded _parked() is always ended by the code under test's own cancellation, which is then awaited under a bound. No assertion was weakened, and the helper-record wait in the git-timeout tests now fails by name instead of quietly continuing after 5 s. No product code changed; the baseline only drops the three touched files.
Verified on this head: the ratchet passes (39) and the three files pass (1178) under -n 4 --dist loadgroup. Removing the helper-record signal in _TimedOutGitWaits reddens test_a_fetch_timeout_returns_while_a_helper_holds_stderr; dropping the entered wait reddens both cases of test_one_inflight_latest_event_survives_and_removal_wins.
Non-blocking nits:
- The
_TimedOutGitWaitsdocstring says the no-helper path times out "oncecommunicate()has started", but onesleep(0)does not guarantee that. _parked()relies on every caller awaiting its cancellation under a bound, and only its docstring says so.
3515290 to
e075249
Compare
|
Rebase-only update (head |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR swaps 97 fixed short waits in three dashboard test files for waits on the signal the code under test emits. The cards and chat conversions are faithful, and no product code changes. One conversion makes a test unable to prove what it exists to prove, and it is still unanswered: the current-head Opus finding at test/test_dashboard_updates_coverage.py:405 has no disposition. I checked it against the head. setup.cfg sets --timeout=120, so default_timeout() is 60 s. The stand-in git and its helper in _HELPER_GIT (lines 276 and 279) both run time.sleep(60) and start before the wait does. So if the timeout or cancel path stopped killing the process group, the _gone() waits in test_a_fetch_timeout_returns_while_a_helper_holds_stderr and test_a_cancelled_fetch_ends_git_and_its_helper would still see both processes exit on their own before the deadline, and helper_exited / git_exited would pass. The old 5 s bound caught that regression. Suggested fix: make the stand-in sleeps much longer than the lost-run bound (for example time.sleep(600); the finally blocks already kill survivors by identity), so only a real kill can satisfy the wait, then check it with a mutant that removes the group kill. Separately, the no-helper _TimedOutGitWaits branch's one sleep(0) does not guarantee communicate() has started, so fix the docstring or wait on a real signal. — crew-pr-reviewer (deep re-check)
…ds, dashboard_updates_coverage, dashboard_chat
e075249 to
899d4ef
Compare
|
Addressed the blocking review on head What changed (both in
Mutant check (as requested): I replaced the fetch timeout/cancel Disposition — ai-review (Opus) finding @ Local: |
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE at 899d4ef18b
What it fixes
Fixed short sleeps in three dashboard test files flaked under load. This PR replaces them with waits on what the code under test emits. Every wait is bounded by default_timeout(). Only tests and the wait-discipline baseline change; no product code does.
What I verified
Since my approval on 35152906e2, the only change apart from a rebase is in test/test_dashboard_updates_coverage.py. It answers both of iamwhatever's points on e0752496fa:
- A slow exit could satisfy the kill wait — fixed. In
_HELPER_GIT, the stand-in git and its helper now sleep for 600 s (test/test_dashboard_updates_coverage.py:276,:279). That is ten times the 60 sdefault_timeout()that--timeout=120gives (setup.cfg:450). The_gone()waits at:407and:475can now pass only if the processes are actually killed. Thefinallyblocks still kill any survivor by its pinned start-time identity (:418,:484). - The no-helper branch relied on one
sleep(0)— fixed._TimedOutGitWaitsnow always waits on a real record: the helper's pid when there is a helper, otherwise git's own.gitrecord (:311-325,:349).ensure_future(aw)schedulescommunicate()before that wait, so the cancel always hits a started read.
Mutation: I replaced the tree kill in kill_and_reap (src/kiro_crew/platform_compat.py:7299, await kill_process_tree_async(...)) with pass. That reddens test_a_fetch_timeout_returns_while_a_helper_holds_stderr, test_a_cancelled_fetch_ends_git_and_its_helper and test_the_timeout_tests_cleanup_signals_no_helper_it_saw_exit. No stand-in process survived the run, and the tree was clean afterwards.
Suites at this head:
- The three touched files: 1178 passed with
-n 4 --dist loadgroup. - The wait-discipline and flake-pattern ratchets: 92 passed.
Blocking
None.
Non-blocking follow-ups
- The
_TimedOutGitWaitsdocstring says the stand-in writes its record "only after …communicate()is reading". The record doesn't guarantee that. The ordering comes fromensure_futureschedulingcommunicate()before the wait yields. A wording fix only. _parked()still depends on every caller awaiting its cancellation under a bound, and only its docstring says so (carried over from my previous review).
Not verified
- I did not re-run iamwhatever's counterfactual, where the old 60 s sleep plus a missing group kill stays green. The point rests on the timing arithmetic above.
- I did not re-check the rest of the rebased diff line by line. I relied on
git range-diffshowing no changes outside this file, apart from context.
iamwhatever
left a comment
There was a problem hiding this comment.
This PR replaces 97 fixed short waits in three dashboard test files with waits on the signal the code under test emits (await_until on events and tasks, async_wait_until for polls, until_parked for the folder lock), and drops those three files from the wait-discipline baseline; no product code changes. I re-read the full diff at 899d4ef against both points of our earlier change request on e075249. Both are fixed: the stand-in git and its helper in _HELPER_GIT now sleep 600 s (test/test_dashboard_updates_coverage.py:276/279), ten times the 60 s default_timeout, so the _gone() waits can only pass if the timeout/cancel path really kills the process group. _TimedOutGitWaits also now always waits on a real record (the helper pid, or git's own .git record when there is no helper) instead of one sleep(0). I checked that every later wait_for in updates._check_git_checkout sits after the first fetch timeout returns, so the fake raising TimeoutError on every call does not mask a different path. The cards and chat conversions keep every assertion and every result propagation, and the intentionally unbounded _parked() is always ended by a cancellation the test awaits under a bound. Checks are green and the base is main, so the only remaining block was our own review. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: The three dashboard test files no longer bet on the runner's speed: every fixed short wait in them now waits on the signal the code emits.
test/test_dynamic_dashboard_cards.py,test/test_dashboard_updates_coverage.pyandtest/test_dashboard_chat.pyheld 97 short fixed waits between them --asyncio.wait_for(..., timeout=2..10)on an Event, a task or a bare coroutine, and polls of 10 to 200 shortasyncio.sleepturns. Each one passes only while the CI runner is as fast as the machine the test was written on. #19248 landed the helpers (kiro_crew.testing.wait) and the per-file ratchet; this is the burn-down for batch E (dashboard).Why it matters
A 5 s
wait_forthat loses on a slow Windows or macOS runner reds an unrelated pull request, and #19093, #19144, #19177 and #19205 were each one of these.test_dynamic_dashboard_cards.pyalone was one of the larger holders of such waits undertest/.Not a goal
src/kiro_crew/testing/wait.py.test/wait_discipline_baseline.txt.asyncio.wait_for(finished.wait(), 60)sites intest_dashboard_chat.py: a 60 s ceiling is already at or above the 30 s lost-run floor, so the ratchet does not count them and they are left unchanged.What changed (motivation → approach → change)
The ratchet counts a site when the wait's ceiling decides pass versus fail. The fix is to wait on what the test was actually waiting for and let the helper's lost-run bound (half the test's timeout) only name a hang.
Per file, counted sites before → after:
test/test_dynamic_dashboard_cards.pytest/test_dashboard_updates_coverage.pytest/test_dashboard_chat.pyEach kind of wait, and the signal it now rides on:
asyncio.wait_for(event.wait(), N)→await await_until(event, ...): the Event the fake or test sets (entered/started/cancelling/cancelled/releasein cards;restore_entered/swap_entered/seenin chat; the leaderstartedin updates).asyncio.wait_for(task_or_future, N)→await await_until(task, ...): the task or future the test created (service.worker,slot.task,gather(*state._background_tasks),gather(task, return_exceptions=True),old_worker/restarted, andrestore_task/swap_task/pick_task/replay). Its result or exception propagates as before (theassert await ... == "fable-5"andcandidate = await ...sites keep their value);pytest.raises(CancelledError)blocks are unchanged.asyncio.wait_for(coroutine(), N)→await await_until(asyncio.ensure_future(coroutine()), ...): the helper takes an Event or future by contract, soshutdown(app),openai_compat._stream_response,openai_compat._blocking_responseandupdates._check_git_checkoutare wrapped in a task first.for _ in range(N): if cond: break; await asyncio.sleep(0.0x)) →await async_wait_until(predicate, describe=...)on the state the loop watched: the SSE/logwritesgrowing, the_QueueLogHandlerattach, the: keepaliveframe,register_ssebeing called,ws.send_strbeing awaited, the racing queue's probe count, and -- withoff_loop=True, as they read process state viaplatform_compat-- the git helper and git process exiting by recorded identity.await until_parked(state._folders_lock, count=2): the two folder reparents parked on the store lock, aLoopBoundLockthatuntil_parkedreads directly (the hand-rolled_both_queued()is gone).Two fakes park a generation/turn open until the code cancels it (
wait_for(asyncio.Event().wait(), N)); these become a local_parked()that awaits the Event unbounded on purpose, because each such test already awaits the cancellation it expects (cancelling/cancelled/ the worker) under the bound, so a never-cancelled park fails by name there._ShortGitWaitsintest_dashboard_updates_coverage.py(which cut every git wait to a fixed 0.5 s) is renamed_TimedOutGitWaitsand times the fetch out on a signal: it waits (off the loop) for the stand-in git to record its helper, or one loop turn when there is none, then cancelscommunicate()and raisesTimeoutErrorexactly as the realwait_fordoes. The class's two process-exit polls move toasync_wait_until(off_loop=True)on the recorded identity.Remaining counted sites: none. No
# flake-okwas added. Baseline diff: the three lines52 test_dynamic_dashboard_cards.py,23 test_dashboard_updates_coverage.py,22 test_dashboard_chat.pyare deleted fromtest/wait_discipline_baseline.txt; no other line moves.Main drift on the ratchet gate (not this PR's sites): rebased onto
origin/mainafter the F821 fixes merged (#19478, #19485). The ratchet's only remaining red is on two untouched files --test/test_surface_deny_notice_sites.py(0 -> 1) andtest/test_webhook_permission_requests.py(2 -> 3), owned elsewhere; it surfaces as the tree-scan insideBackend Tests (3.12, 2)(which cascades toCoverage Gate).git diff --name-only origin/main...HEADlists only the three test files and the baseline.Backwards compatibility
Compatible: test-only change; no product code, public function or configuration is touched.
Removes nothing user-facing. The only removals/renames are test-internal, each with no reader outside its own file:
test/test_dashboard_updates_coverage.py:_TimedOutGitWaits-- test-only -- renamed from_ShortGitWaits; grep ofsrc/andtest/and of every open PR diff finds no other user of the old name.test/test_dashboard_chat.py-- test-only -- the local_both_queued()helper and two unusedimport asyncio as _asynciostatements are removed; both were used only by the waits this PR converted, and nothing else in the file references them.Merge-order note
The one rename (
_ShortGitWaits→_TimedOutGitWaits) and the removed_both_queuedare both test-internal to a single file. Every currently open PR's diff was grepped for both names and none references either, so no alias is needed and no in-flight PR is affected.Tests
Changed tests (all three files, 97 sites) keep every assertion; what each one locks in is unchanged. Determinism check on this host:
python test/test_wait_discipline_ratchet.pyreports 0 for the three files;--write-baselinetouches only their three lines and, re-run after rebase, changes nothing further.test/test_flake_pattern_ratchet.py(merge-base comparison): 53 passed.black --check,isort --check-onlyandflake8all exit 0.Not a flaky-test FIX in the D12 sense (no forced-condition red is claimed): these waits had not yet lost on CI; the change removes the bet before they do.
Manual verification
N/A — unit coverage sufficient: the diff is the tests themselves.
Related Issues
Part of #19246