Skip to content

test(slack-gateway): fix flaky wedged-write graceful-bound test - #29155

Merged
bolichen97 merged 1 commit into
mainfrom
fix/flaky-slack-gateway-wedged-write-bound
Nov 3, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/flaky-slack-gateway-wedged-write-bound

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Fix flaky test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound[True] so it passes deterministically on every CI shard.

The [True] parametrization fails intermittently on loaded shards (two hits in 3 h on main and on feat/ultra-mode-feedback-5; run https://github.com/kirodotdev/KiroCrew/actions/runs/41627822676 job 125893226305 and https://github.com/kirodotdev/KiroCrew/actions/runs/41612059493, "Backend Tests (3.12, 2)").

Exact assertion:

AssertionError: shutdown ended without naming the admitted write it left pending

Test added by #18648 (merged 2026-10-23, commit 9c9d87e).

Why it matters

When this flakes it turns main red and blocks unrelated PRs.

Not a goal

What changed (motivation → approach → change)

Root cause: For under_graceful_bound=True, _shutdown() is wrapped in asyncio.wait_for(shutdown, timeout=gw.GRACEFUL_SHUTDOWN_SECS=0.3s). Inside, graceful_deadline = now + GRACEFUL_SHUTDOWN_SECS (exit_path.py line 55 comment: "this is that bound's deadline"). Both timers point at the same wall-clock moment. The execution path after the drain is cut is:

  1. drain_task runs ~0.1 s (drain_budget=0.05 s + grace=0.05 s)
  2. _stop_services_and_sessions runs (exit_path.py:250)
  3. settle_admitted_writes(graceful_deadline) → logs "at the graceful shutdown deadline" warning

On a loaded shard step 2 takes long enough that the outer wait_for(0.3 s) fires and cancels _shutdown() before step 3 is reached. settle_admitted_writes is never called, the warning never logged, the assertion fails. Once settle_admitted_writes IS reached the warning fires unconditionally — via remaining <= 0 at the deadline, or via except asyncio.CancelledError inside it.

The same ordering gap exists in production (_shutdown_and_exit line 610), but it is a log-only gap (no durability impact); tracked in #29158. exit_path.py is not changed here because #9052 is actively rewriting the same block and any change there now would conflict.

Fix: Replace orch._stop_services_and_sessions with AsyncMock(return_value=None) before calling _shutdown(). Step 2 becomes instant, ensuring step 3 is always reached within the 0.3 s budget. The real teardown is not under test here (the test verifies write-naming, not session/cron cleanup); the sibling test test_a_wedged_autonudge_drain_is_cut_so_the_teardown_still_runs uses the same pattern (lines 1052–1053).

PR #9052 (rubencu, since Sep 6) rewrites the autonudge shutdown drain and also touches test_slack_gateway.py; this change is minimal and local to one parametrization to rebase cleanly against it.

Backwards compatibility

Removes nothing: test-only change.

Tests

Locked in: test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound[True] — now deterministic because _stop_services_and_sessions is mocked to return instantly, guaranteeing settle_admitted_writes is always reached before the 0.3 s graceful deadline fires.

CI evidence: (to be updated with green shard job URL once CI completes on this head)

Related Issues

Flaky test: run 41627822676 job 125893226305 (main, 2026-11-03T11:28Z).
Production log gap: #29158.

Sibling sweep

Swept: grep -n "_cut_drain_over_a_slow_write\|under_graceful_bound" test/test_slack_gateway.py

Siblings: none — _cut_drain_over_a_slow_write is only called in the two tests in TestShutdown; only the [True] parametrization has the ordering race.

Pattern harvest

Not generalizable: the race is specific to two timers pointing at the same wall-clock moment with a slow step between them. The production fix approach is in #29158.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners November 3, 2026 12:18
@CrysisDeu
CrysisDeu requested a review from bolichen97 November 3, 2026 12:18
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change, and stubbing teardown removes the step that raced the shared 0.3 s deadline.

Notes

  • The stub is set before the if under_graceful_bound branch, so it covers [False] too, but the description says the change is "local to one parametrization". Either move the stub inside the branch or correct the description.
  • The real fix is in production: the outer wait_for can cancel shutdown before settle_admitted_writes logs anything. That gap stays open until fix: shutdown can cancel before settle_admitted_writes names the admitted write #29158 lands.

[DESIGN-REVIEWED] 8c7f279

Description read

[DESCRIPTION-READ] fdba130b4e7a816da1dd20790ac86410d0beb6cec0f7ecf7e6b0fcd81585fce5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@CrysisDeu
CrysisDeu force-pushed the fix/flaky-slack-gateway-wedged-write-bound branch from 9152372 to 098150d Compare November 3, 2026 12:20
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8c7f279

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 false-positive: <reason, 20 to 500 characters>

…bound

On a loaded shard _stop_services_and_sessions can take long enough
that the outer wait_for(GRACEFUL_SHUTDOWN_SECS) fires and cancels
_shutdown() before settle_admitted_writes is reached, so the
"at the graceful shutdown deadline" WARNING is never logged and
the [True] parametrization fails intermittently.

Fix: replace _stop_services_and_sessions with an AsyncMock no-op in
the wedged-write test.  Once settle_admitted_writes is reached the
warning fires unconditionally -- either at the internal graceful
deadline (remaining <= 0) or via the CancelledError path when the
outer wait_for fires while inside it.  The real teardown is not
under test here.

#9052 (rubencu) rewrites the autonudge shutdown drain and also
touches test_slack_gateway.py; this change is minimal and local to
the one failing parametrization to rebase cleanly against it.
@CrysisDeu
CrysisDeu force-pushed the fix/flaky-slack-gateway-wedged-write-bound branch from 098150d to 8c7f279 Compare November 3, 2026 12:21
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8c7f279

Verdict parsed from the review's SHA-scoped output markers for commit 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8c7f2792ea17b4fa2029a8dfce1a0dd98e02d4d4 false-positive: <reason, 20 to 500 characters>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 8c7f279. This is test only: one line stubs orch._stop_services_and_sessions with an AsyncMock in test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound. With the real teardown out of the way, settle_admitted_writes is always reached before the 0.3 s bound, so on a loaded shard the warning assertion no longer depends on how long teardown takes.

One non-blocking note: the stub applies to both parametrizations, and it means the test no longer covers the real teardown eating the whole graceful bound before the pending write is named. The description says that production gap is tracked in #29158. With that issue open, narrowing this test to the warning path is fine.

The red entries at this head are older cancelled duplicates, and only PR Readiness is pending. Approving.

@bolichen97
bolichen97 enabled auto-merge (squash) November 3, 2026 13:13
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 3, 2026
@bolichen97
bolichen97 merged commit fc0120a into main Nov 3, 2026
98 of 102 checks passed
@bolichen97
bolichen97 deleted the fix/flaky-slack-gateway-wedged-write-bound branch November 3, 2026 13:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants