Skip to content

test(singles-4): replace fixed waits with signal waits - #22884

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/wait-w2-r
Oct 22, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/wait-w2-r

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Replace the 50 short fixed waits in 50 single-site test files with waits on the signal each test is actually about, so a slow CI runner cannot red them.

Why it matters

A wait_for(..., timeout=5), a .result(timeout=5) or a 200-turn sleep(0.01) poll bets that the runner is as fast as the machine the test was written on. That bet is the single commonest cause of a flaky backend test here, and every lost bet reds an unrelated pull request.

Not a goal

  • Migrating test files outside this batch.
  • Changing production code.
  • Reformatting the eight touched files that .github/black-baseline.txt still lists (they stay listed; only the migrated lines change).
  • Removing a short wait that is a test's own assertion or measurement (those keep a tight bound on the helper, or a # flake-ok reason).

What changed (motivation → approach → change)

Every counted site in these 50 files (one per file) is replaced with the matching kiro_crew.testing.wait helper:

  • asyncio.wait_for(event.wait(), N) / wait_for(task, N) / wait_for(gather(...), N) → await_until(event | future, what=...).
  • asyncio.wait_for(coro, N) → finished(coro, what).
  • Future.result(timeout=5) on a thread-pool or run_coroutine_threadsafe future → .result(timeout=default_timeout()).
  • Hand-rolled for _ in range(N): if cond: break; sleep(x) / while time.monotonic() < deadline polls → wait_until(cond, describe=...) or async_wait_until(cond, describe=...). The poll helpers _await_pass_state, _settle (x3), _queued_on, _await_detached, _await_until and _await keep their signatures and delegate (_await_detached and the not-found _settle drop their unused tries / rounds; the side-steer _settle drops its unused timeout).
  • test_session_control_reload.py: _queued_on is exactly until_parked(lock).
  • test_session_autocompact_override.py: the first request's persist is held until the second request is parked on the slot's transaction lock (or, absent serialization, has committed), instead of for a fixed 0.5 s.
  • test_session_start_queued_behind.py: the terminate responder wakes on each frame the runtime writes (a side_effect on the mocked stdin.write sets an event) instead of polling every 10 ms.
  • test_session_transfer_outbound.py: the stalled peer awaits a never-set event; only the lapsed no-progress deadline can end it, which is what the test asserts.
  • test_memory_admission_queue.py: _settle first waits for the preparation task's done-callbacks to have fired (the drain is armed by one), then for the slot to go idle with no background task left.
  • test_session_not_found_surfaces.py: _settle waits on the state between one task ending and its respawn registering, instead of sleeping across that gap.
  • test_shutdown_event.py: the "wait_for times out on the never-set event" test asserts its timeout with pytest.raises(asyncio.TimeoutError).
  • test_opencode_session_mcp.py, test_pod_e2e_video_guard.py, test_push_guard_squash.py: the wait ends on the process / pid / mark state, and the original failure branches run after it.
  • Several files carried a # flake-ok: comment on the line BEFORE the call, where the ratchet does not read it; those sites are migrated rather than re-marked.

Kept on purpose:

  • test_resource_pressure_notify.py: finished(..., timeout=2.0) -- the bound IS the assertion (the sample must be forfeited at PROBE_TIMEOUT_SECS, before the probe's own 5 s release).
  • test_session_usage_scrape.py: finished(..., timeout=10) -- the patched 0.2 s deadline must end the refresh before the hung API's 30 s release.
  • test_native_subagent_boundary.py: finished(..., timeout=_FRAME_CEILING_SECS) keeps the module's documented 20 s ceiling under its timeout(30) mark.
  • test_messaging_attachments.py: a ticker sleep(0.005) measures free loop time during a 150 ms blocking read; it waits on no signal, so it keeps a # flake-ok reason on its line.

test/wait_discipline_baseline.txt: the 50 lines go to zero and are deleted.

Files:
test_mcp_gateway_recaller.py, test_mcp_gateway_session_inject.py, test_mcp_launch_approval.py, test_mcp_preflight.py, test_meetings_audio_import.py, test_members_eventlog_wiring_append_ordering.py, test_memory_admission_queue.py, test_messaging_approval.py, test_messaging_attachments.py, test_messaging_auto_title.py, test_mochi_activity.py, test_mochi_autonomous_unfreeze.py, test_model_access_fallback_interactive.py, test_native_subagent_boundary.py, test_opencode_session_mcp.py, test_papyrus_latex.py, test_pid_lifecycle_process_sweeps.py, test_pod_e2e_video_guard.py, test_pptx_maker_routes.py, test_proc_non_utf8_kill_paths.py, test_provider_spawn_off_loop_18172.py, test_push_guard_squash.py, test_queue_boundary_finalize.py, test_registry_store_art_prewarm.py, test_registry_store_art_prewarm_rewarm.py, test_remote_crew_chaining.py, test_resource_pressure_notify.py, test_runtime_start_gate_headroom.py, test_sandbox_argv.py, test_scenarios_v2_logic.py, test_session_autocompact_override.py, test_session_card_source_links_knob.py, test_session_control_reload.py, test_session_control_revive.py, test_session_map_mirror.py, test_session_more_coverage.py, test_session_not_found_surfaces.py, test_session_pulse_counter.py, test_session_scope_reap_seams.py, test_session_start_queued_behind.py, test_session_storage_api.py, test_session_terminated_reap.py, test_session_transfer_outbound.py, test_session_tree_push.py, test_session_usage_refresh_routes.py, test_session_usage_scrape.py, test_session_work_dir.py, test_shutdown_event.py, test_side_steer_queue.py, test_skill_catalog_snapshot.py

Part of wave-2 fixed-wait migration.

Backwards compatibility

Removes nothing: test-only changes; no production code, API or schema touched.

Tests

  • test/test_wait_discipline_ratchet.py and test/test_flake_pattern_ratchet.py: 113 passed (the 50 files now scan at zero sites; baseline canonical; no changed file gains a flaky shape).
  • The 50 touched files: 2294 passed, 10 skipped (-n 8).
  • isort, flake8 and the scoped black gate (scripts/check_black_formatting.py) clean on the touched files.

Pre-existing failures on main (not from this PR)

Manual verification

N/A — unit coverage sufficient; the two ratchets gate regression.

Related Issues

Part of wave-2 fixed-wait migration.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 21, 2026 19:51
@CrysisDeu
CrysisDeu requested a review from buluoray October 21, 2026 19:51
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c712596f468aeec09dd005e581a3494c487ec4f3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision c712596f468aeec09dd005e581a3494c487ec4f3 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c712596f468aeec09dd005e581a3494c487ec4f3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only migration. Each removed assertion is still enforced, because wait_until / async_wait_until raise on timeout. The bounds that are themselves the assertion are kept on purpose.

[DESIGN-REVIEWED] c712596

Description read

[DESCRIPTION-READ] 49e1d31479b20d2c147c8db2790c637a2ec73e451d6a62930e5ecfaf44862e11

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c712596f468aeec09dd005e581a3494c487ec4f3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c712596

Verdict parsed from the review's SHA-scoped output markers for commit c712596f468aeec09dd005e581a3494c487ec4f3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c712596f468aeec09dd005e581a3494c487ec4f3: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c712596f468aeec09dd005e581a3494c487ec4f3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c712596

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c712596f468aeec09dd005e581a3494c487ec4f3: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 21, 2026 21:14
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD -- reviewed 3ba032f.

The migration is mechanical and reads correctly: each poll or fixed wait_for becomes the matching kiro_crew.testing.wait helper with the same predicate, the bounds that are themselves assertions are kept, and CI is green. The test_sandbox_argv.py and test_proc_non_utf8_kill_paths.py hunks only change how the test waits, not what the sandbox or kill path does.

What blocks approval: the Design Review's two notes have no reply.

  • test_session_transfer_outbound.py#L419-L421: await asyncio.Event().wait() has no bound of its own. If the no-progress deadline regresses, the test now hangs until pytest-timeout instead of failing by name, which is the shape this wave is trying to remove. Wrapping the stalled upload in finished(..., timeout=<generous>) would keep a regression a red test.
  • test_session_autocompact_override.py#L610-L614 reads txn_lock._waiters directly; until_parked already does that behind a capability check, so a reply or a switch to that helper would settle it.

To clear: fix or rebut each note on the PR.

— crew-pr-reviewer (r118)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 21, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 21, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 22, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 22, 2026
Fifty test files each held one short fixed wait: a wait_for or
Future.result ceiling of a few seconds, or a sleep inside a poll loop.
Each site now waits on the signal the code under test emits, through
kiro_crew.testing.wait (await_until, finished, async_wait_until,
wait_until, until_parked), under the lost-run bound only.

The waits that ARE the assertion keep a tight bound on the helper:
the stalled resource probe (2 s), the hung usage refresh (10 s) and
the frame ceiling of the subagent boundary tests. The shutdown-event
timeout test asserts its timeout through pytest.raises. One loop
ticker that measures free loop time keeps its sleep with a flake-ok
reason. The transfer stall waits on a never-set event instead of a
1 s sleep, since only the lapsed deadline can end it.

Each file's line leaves test/wait_discipline_baseline.txt.
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 22, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE -- reviewed c712596.

This head is a rebase of 3ba032f (range-diff shows only baseline context lines). Every migrated site keeps its predicate, the bounds that are themselves assertions are kept, and the sandbox/kill-path test files change only how the test waits.

The two r118 notes, re-checked: the stalled-peer await asyncio.Event().wait() (test_session_transfer_outbound.py#L419-L421) still fails by name under the suite's --timeout=120, so a regression stays a red test. The txn_lock._waiters read (test_session_autocompact_override.py#L610-L614) is on an asyncio.Lock that the first request holds, and until_parked cannot express the 'or the second committed' half. Neither is a defect.

Design PASS, Opus and GPT no findings on this head; all checks green.

— crew-pr-reviewer (r119)

@iamwhatever
iamwhatever merged commit f6a8f24 into main Oct 22, 2026
105 checks passed
@iamwhatever
iamwhatever deleted the unflaky/wait-w2-r branch October 22, 2026 06:33
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants