Skip to content

fix: name in-flight writes when shutdown is cut during teardown - #31508

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/29158-settle-on-cancel
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/29158-settle-on-cancel

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When the graceful shutdown bound cuts the gateway teardown, the log still names any AutoNudge write left in flight.

_shutdown() waits for admitted AutoNudge writes only after the gateway teardown returns. If the teardown is still running when the outer wait_for(..., GRACEFUL_SHUTDOWN_SECS) in _shutdown_and_exit fires, the cancellation lands on the teardown await and the wait is never entered. The warning AutoNudge still had N admitted mutation(s)/write(s) in flight at the graceful shutdown deadline is skipped, so the log says nothing about a write the hard exit then abandons.

Why it matters

An operator reading gateway.log after a slow shutdown cannot tell whether an AutoNudge change made it to disk. The store write is atomic (the file keeps the new state or the previous one), so nothing is corrupted; what is lost is the one line that explains a missing change after the restart.

Not a goal

  • Changing the graceful budget, the drain cut, or the interrupted-drain branch of _shutdown().
  • Making the abandoned write durable: past the deadline it is still left to run on its thread.

What changed (motivation → approach → change)

A slow teardown could swallow the one warning that names an unfinished write, because the wait that logs it ran only after the teardown returned.

In gateway_runtime/exit_path.py, the teardown await is now wrapped in try/finally, and the finally calls settle_admitted_writes(graceful_deadline). A teardown that is cancelled or raises still reaches that wait. It is bounded by the same deadline as the outer bound, so when the graceful bound is what cancelled the teardown it logs the pending count and returns at once, and the cancellation then propagates as before. The owning gateway spec says the same.

Backwards compatibility

Compatible: the normal path is unchanged; a cancelled or failed teardown now also runs the existing deadline-bounded wait before its exception propagates.

Removes nothing: the change only adds a wait on the error path; no key, route or guard is taken away.

Tests

  • test/test_slack_gateway.py::TestShutdown::test_a_cancel_during_the_teardown_still_names_the_wedged_write: a real AutoNudge service with a write wedged past the drain cut, a teardown that never returns, and a cancellation delivered once the teardown has started. It asserts the shutdown re-raises the cancellation, the write is still pending, and the "at the graceful shutdown deadline" warning was logged. The cancel is driven by the teardown-entered event and the graceful budget is 0 (already expired, as it is when the bound fires), so no clock race decides the result.

Proof: PROVEN test/test_slack_gateway.py::TestShutdown::test_a_cancel_during_the_teardown_still_names_the_wedged_write (head e609b1b)

Determinism: 20/20 green at -n0; 10/10 green at -n 4 --dist loadgroup --max-worker-restart=2 with TestShutdown and test/test_monitor_turn_completion.py as co-tenants; TestShutdown (23 tests) green in a shuffled order, seed 29295.

Manual verification

N/A — unit coverage sufficient: the test drives the real _shutdown() and a real AutoNudge service; only the teardown is replaced by one that never returns.

Screenshots / video

N/A — backend only, no user-visible UI change.

Related Issues

Closes #29158

Sibling sweep

Swept: git grep -nE "settle_admitted_writes(" -- src/
Siblings: none -- exit_path.py is the only caller of settle_admitted_writes, and the interrupted-drain branch runs its teardown as an owned task bounded by the deadline

Pattern harvest

Not generalizable: the defect is one ordering in one shutdown method, a log step placed after an await the outer bound can cancel; there is no shared shape a rule could match without flagging every intended post-await step.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 15:36
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 15:36
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the finally reuses the existing wait, which is bounded by the deadline, so a cut teardown still logs the pending writes.

Notes

  • When a cancellation arrives before the graceful deadline (any source other than the outer bound), the finally now holds it until graceful_deadline. It no longer propagates at once. The wait is still bounded, but the spec could say so.

[DESIGN-REVIEWED] 1d48141

Description read

[DESCRIPTION-READ] 780c6bd7243acd8196da6b581e056006bb4f1b71378ee2a834e7a01f06e18b8c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the fix moves the one existing wait into a finally, which is where the cause is (a log step placed after an await the outer bound can cancel), and issue #29158 records the defect.

What this change ships

Inventory (4 items) — 4 justified

Intent: when a slow shutdown is cut, the gateway log still names any AutoNudge write left unfinished. This is a FIX.

  1. A teardown cut by the graceful bound still logs the "admitted write(s) in flight at the graceful shutdown deadline" warning. — justified
  2. A teardown that raises now waits for admitted writes, up to the same deadline, before the error propagates. — justified
  3. The slack-gateway spec now says the wait also runs after a cancelled teardown. — justified
  4. A regression test covers a teardown cancelled while a write is still wedged. — justified

[FIRST-PRINCIPLES-REVIEWED] 1d48141

Description read

[DESCRIPTION-READ] 076631ae7e44bdc9d29af79624eab00f4e5ed81ec6c7908b898a865301d71707

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1d48141

Verdict parsed from the review's SHA-scoped output markers for commit 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1d48141

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1d4814163b4a8eff47bb3e8c7b2c5dbfff43fd1c false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 15:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests (3.12, 2), Backend Tests (Windows) (2) and Coverage Gate come from one test, test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget. It also fails on main since #30774, which this diff does not touch. #31625 fixes it (tracked in #31648). This PR waits for that fix to merge, then rebases onto it and re-runs CI.

When the graceful shutdown bound cancels the gateway teardown, the
wait for admitted AutoNudge writes that follows it never ran, so the
warning naming a write left in flight at the deadline was skipped.
Run that wait in a finally around the teardown, so a cancelled or
failed teardown still waits out the graceful budget and logs what
is left.
@bolichen97
bolichen97 force-pushed the fix/29158-settle-on-cancel branch from e609b1b to 1d48141 Compare November 7, 2026 00:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 1d48141, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit 7878189 into main Nov 7, 2026
97 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/29158-settle-on-cancel branch November 7, 2026 01:56
buluoray pushed a commit that referenced this pull request Nov 8, 2026
When the graceful shutdown bound cancels the gateway teardown, the
wait for admitted AutoNudge writes that follows it never ran, so the
warning naming a write left in flight at the deadline was skipped.
Run that wait in a finally around the teardown, so a cancelled or
failed teardown still waits out the graceful budget and logs what
is left.

(cherry picked from commit 7878189)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: shutdown can cancel before settle_admitted_writes names the admitted write

2 participants