Repository navigation
test(slack-gateway): fix flaky wedged-write graceful-bound test - #29155
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change, and stubbing teardown removes the step that raced the shared 0.3 s deadline. Notes
[DESIGN-REVIEWED] 8c7f279 Description read[DESCRIPTION-READ] fdba130b4e7a816da1dd20790ac86410d0beb6cec0f7ecf7e6b0fcd81585fce5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
9152372 to
098150d
Compare
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
…bound On a loaded shard _stop_services_and_sessions can take long enough that the outer wait_for(GRACEFUL_SHUTDOWN_SECS) fires and cancels _shutdown() before settle_admitted_writes is reached, so the "at the graceful shutdown deadline" WARNING is never logged and the [True] parametrization fails intermittently. Fix: replace _stop_services_and_sessions with an AsyncMock no-op in the wedged-write test. Once settle_admitted_writes is reached the warning fires unconditionally -- either at the internal graceful deadline (remaining <= 0) or via the CancelledError path when the outer wait_for fires while inside it. The real teardown is not under test here. #9052 (rubencu) rewrites the autonudge shutdown drain and also touches test_slack_gateway.py; this change is minimal and local to the one failing parametrization to rebase cleanly against it.
098150d to
8c7f279
Compare
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed 8c7f279. This is test only: one line stubs orch._stop_services_and_sessions with an AsyncMock in test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound. With the real teardown out of the way, settle_admitted_writes is always reached before the 0.3 s bound, so on a loaded shard the warning assertion no longer depends on how long teardown takes.
One non-blocking note: the stub applies to both parametrizations, and it means the test no longer covers the real teardown eating the whole graceful bound before the pending write is named. The description says that production gap is tracked in #29158. With that issue open, narrowing this test to the warning path is fine.
The red entries at this head are older cancelled duplicates, and only PR Readiness is pending. Approving.
Problem / Motivation
Goal: Fix flaky
test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound[True]so it passes deterministically on every CI shard.The
[True]parametrization fails intermittently on loaded shards (two hits in 3 h on main and onfeat/ultra-mode-feedback-5; run https://github.com/kirodotdev/KiroCrew/actions/runs/41627822676 job 125893226305 and https://github.com/kirodotdev/KiroCrew/actions/runs/41612059493, "Backend Tests (3.12, 2)").Exact assertion:
Test added by #18648 (merged 2026-10-23, commit 9c9d87e).
Why it matters
When this flakes it turns main red and blocks unrelated PRs.
Not a goal
[False]parametrization (it never flakes).test_a_write_slower_than_the_drain_cut_lands_before_shutdown_returns.What changed (motivation → approach → change)
Root cause: For
under_graceful_bound=True,_shutdown()is wrapped inasyncio.wait_for(shutdown, timeout=gw.GRACEFUL_SHUTDOWN_SECS=0.3s). Inside,graceful_deadline = now + GRACEFUL_SHUTDOWN_SECS(exit_path.py line 55 comment: "this is that bound's deadline"). Both timers point at the same wall-clock moment. The execution path after the drain is cut is:_stop_services_and_sessionsruns (exit_path.py:250)settle_admitted_writes(graceful_deadline)→ logs"at the graceful shutdown deadline"warningOn a loaded shard step 2 takes long enough that the outer
wait_for(0.3 s)fires and cancels_shutdown()before step 3 is reached.settle_admitted_writesis never called, the warning never logged, the assertion fails. Oncesettle_admitted_writesIS reached the warning fires unconditionally — viaremaining <= 0at the deadline, or viaexcept asyncio.CancelledErrorinside it.The same ordering gap exists in production (
_shutdown_and_exitline 610), but it is a log-only gap (no durability impact); tracked in #29158. exit_path.py is not changed here because #9052 is actively rewriting the same block and any change there now would conflict.Fix: Replace
orch._stop_services_and_sessionswithAsyncMock(return_value=None)before calling_shutdown(). Step 2 becomes instant, ensuring step 3 is always reached within the 0.3 s budget. The real teardown is not under test here (the test verifies write-naming, not session/cron cleanup); the sibling testtest_a_wedged_autonudge_drain_is_cut_so_the_teardown_still_runsuses the same pattern (lines 1052–1053).PR #9052 (rubencu, since Sep 6) rewrites the autonudge shutdown drain and also touches
test_slack_gateway.py; this change is minimal and local to one parametrization to rebase cleanly against it.Backwards compatibility
Removes nothing: test-only change.
Tests
Locked in:
test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound[True]— now deterministic because_stop_services_and_sessionsis mocked to return instantly, guaranteeingsettle_admitted_writesis always reached before the 0.3 s graceful deadline fires.CI evidence: (to be updated with green shard job URL once CI completes on this head)
Related Issues
Flaky test: run 41627822676 job 125893226305 (main, 2026-11-03T11:28Z).
Production log gap: #29158.
Sibling sweep
Swept:
grep -n "_cut_drain_over_a_slow_write\|under_graceful_bound" test/test_slack_gateway.pySiblings: none —
_cut_drain_over_a_slow_writeis only called in the two tests inTestShutdown; only the[True]parametrization has the ordering race.Pattern harvest
Not generalizable: the race is specific to two timers pointing at the same wall-clock moment with a slow step between them. The production fix approach is in #29158.
Checklist