Skip to content

fix(test): cut the wedged-write shutdown inside the wait after the drain - #33704

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/bug-resolve/33609
Nov 9, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/bug-resolve/33609

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound[True] test the path it names on every runner, so it stops failing on loaded Windows CI.

It failed on Backend Tests (Windows) shard 2/8 on a head that touches no Slack code: test\test_slack_gateway.py:1178: AssertionError: shutdown ended without naming the admitted write it left pending.

Why it matters

A flaky red in a backend shard blocks unrelated PRs and costs a rerun each time it hits.

Not a goal

  • No change to shutdown behaviour or budgets in exit_path.py or AutoNudge.

What changed (motivation → approach → change)

  • Symptom: the [True] case sometimes finds no "at the graceful shutdown deadline" warning on Windows.
  • Root cause: the case wrapped _shutdown() in asyncio.wait_for(timeout=0.3). _shutdown spends about 0.1 s of that same 0.3 s in the AutoNudge drain cut: the PERSISTENCE_DRAIN_GRACE_SECS budget, then the drain's own post-cancellation window. On a loaded runner the outer timer fired while the gateway was still in await asyncio.wait({drain_task}) after the cut. That cancellation sends _shutdown down its interrupted path (src/kiro_crew/slack/gateway_runtime/exit_path.py:243-295 on main), which never calls settle_admitted_writes. The removed test comment said settle "is always reached before the graceful deadline, regardless of shard load". The drain phase made that false.
  • Change (test only): the [True] case now runs _shutdown as a task and waits until settle_admitted_writes has been entered, recorded through a wrapper. Then it cancels the task. That is what the graceful bound does at expiry, and it now always lands inside the wait after the drain cut. In this case _shutdown's own deadline is the lost-run bound, so only that cancellation ends the wait. The [False] case keeps _shutdown's own 0.3 s deadline. It has no outer timer, so it cannot take the interrupted path. If the drain runs past 0.3 s, settle sees the deadline already passed and still warns. Both cases also assert that settle was entered. The unused contextlib import is dropped.
  • Alternatives rejected: (1) Accept the drain's own warning as well. That warning fires in every run of this test, so the assertion would stop proving anything about settle. (2) Raise the 0.3 s timer. That is still a race against runner load. (3) Also call settle from the interrupted path, which is a product change. Nothing goes unnamed on that path today, and it would log the same writes twice.

Intended? / Repro

  • Intended? No. The test docstring and fix: name in-flight writes when shutdown is cut during teardown #31508 say the write the drain cut leaves is named when the graceful bound cuts the wait after the teardown. The test meant to put the bound inside that wait. Its fixed 0.3 s timer could land earlier, during the drain cut. The product's interrupted path is separate and deliberate (exit_path.py:276-295). There the drain's own warning already names the pending writes ("AutoNudge shutdown stopped waiting on N owned task(s)/write(s) still pending ..."). So the product is right and the test's ordering is wrong.
  • Repro (CI log, no local run): run 42549002034 / job 128942141726, PR fix(platform): five browser harness, apparmor and scratch sweep fixes #33386 head a027ea5. The captured log has exit_path.py:240 AutoNudge drain still pending after 0.1s; cancelled it and autonudge.py:1072 AutoNudge shutdown stopped waiting on 2 owned task(s)/write(s) still pending 0.1s after cancellation, and no settle warning. So the outer 0.3 s cancellation landed during the drain phase, before settle was reached.

Backwards compatibility

Compatible: test-only change.

Removes nothing: only one test's ordering changes; no product code, key or record is touched.

Tests

  • test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound[True]: the graceful-bound cancellation landing inside the wait after the drain cut still names the pending write at WARNING.
  • [False]: _shutdown's own deadline ends that wait and names the write.

Proof: not provable -- test-only change

Per host rule, no tests were run locally. CI runs them on all platforms. Static gates run: black, flake8, isort, comment-history.

Manual verification

N/A — test-only change, covered by CI.

Related Issues

Fixes #33609

Sibling sweep

Swept: git grep -n "wait_for(.*GRACEFUL_SHUTDOWN_SECS" -- test/
Sibling: test/test_slack_gateway.py:1064 -- not affected -- its bound is FALLBACK_TIMEOUT_SECS since #30653, so it is a safety net and not a race

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that bounds a coroutine with a short fixed timer to reach one inner branch races the earlier phases on a slow runner; deliver the cancellation after an event marks that branch instead

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners November 9, 2026 05:05
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision c808844d2fc0dd556cdf5c7347904238b417f652 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c808844d2fc0dd556cdf5c7347904238b417f652 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c808844d2fc0dd556cdf5c7347904238b417f652 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change. It waits on an event that marks the settle branch instead of racing a fixed timer, which removes the flake at its cause.

Notes

  • The [True] case now cancels the task by hand instead of going through _shutdown_and_exit's real wait_for bound. If that bound's value or how it is applied changes, this test will no longer catch it.

[DESIGN-REVIEWED] c808844

Description read

[DESCRIPTION-READ] 2668aa159ae127df679481ad7cd21dd215645717c604dbaf1a52265f423e32dd

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c808844d2fc0dd556cdf5c7347904238b417f652 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c808844

Verdict parsed from the review's SHA-scoped output markers for commit c808844d2fc0dd556cdf5c7347904238b417f652.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c808844d2fc0dd556cdf5c7347904238b417f652 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c808844d2fc0dd556cdf5c7347904238b417f652 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c808844

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c808844d2fc0dd556cdf5c7347904238b417f652 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
The under_graceful_bound case wrapped _shutdown in a 0.3 s wait_for
timer. On a loaded Windows runner that timer fired while the drain cut
was still waiting out its own window, so _shutdown took its interrupted
path, which never reaches settle_admitted_writes, and the deadline
warning the test asserts was never logged.

The case now delivers the bound as the cancellation it is at expiry,
after settle_admitted_writes has started, so it always tests the path
it names. The other case keeps _shutdown's own 0.3 s deadline, which
has no outer timer to race.

Fixes #33609
@CrysisDeu
CrysisDeu force-pushed the crew/bug-resolve/33609 branch from df3e27e to c808844 Compare November 9, 2026 06:37
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 9, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 9, 2026 09:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; the wedged-write shutdown test now cancels after the wait past the drain cut has started, instead of racing a timer against the cut.

@KiroCrewCaptain
KiroCrewCaptain merged commit e4d1553 into main Nov 9, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/bug-resolve/33609 branch November 9, 2026 09:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound fails on Windows CI

2 participants