Repository navigation
fix: name in-flight writes when shutdown is cut during teardown - #31508
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the Notes
[DESIGN-REVIEWED] 1d48141 Description read[DESCRIPTION-READ] 780c6bd7243acd8196da6b581e056006bb4f1b71378ee2a834e7a01f06e18b8c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the fix moves the one existing wait into a What this change shipsInventory (4 items) — 4 justifiedIntent: when a slow shutdown is cut, the gateway log still names any AutoNudge write left unfinished. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 1d48141 Description read[DESCRIPTION-READ] 076631ae7e44bdc9d29af79624eab00f4e5ed81ec6c7908b898a865301d71707 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
[operator: bolichen97] The red |
When the graceful shutdown bound cancels the gateway teardown, the wait for admitted AutoNudge writes that follows it never ran, so the warning naming a write left in flight at the deadline was skipped. Run that wait in a finally around the teardown, so a cancelled or failed teardown still waits out the graceful budget and logs what is left.
e609b1b to
1d48141
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved for the on-call. Review lanes pass on 1d48141, no conflict, no open change request.
When the graceful shutdown bound cancels the gateway teardown, the wait for admitted AutoNudge writes that follows it never ran, so the warning naming a write left in flight at the deadline was skipped. Run that wait in a finally around the teardown, so a cancelled or failed teardown still waits out the graceful budget and logs what is left. (cherry picked from commit 7878189)
Problem / Motivation
Goal: When the graceful shutdown bound cuts the gateway teardown, the log still names any AutoNudge write left in flight.
_shutdown()waits for admitted AutoNudge writes only after the gateway teardown returns. If the teardown is still running when the outerwait_for(..., GRACEFUL_SHUTDOWN_SECS)in_shutdown_and_exitfires, the cancellation lands on the teardown await and the wait is never entered. The warningAutoNudge still had N admitted mutation(s)/write(s) in flight at the graceful shutdown deadlineis skipped, so the log says nothing about a write the hard exit then abandons.Why it matters
An operator reading
gateway.logafter a slow shutdown cannot tell whether an AutoNudge change made it to disk. The store write is atomic (the file keeps the new state or the previous one), so nothing is corrupted; what is lost is the one line that explains a missing change after the restart.Not a goal
_shutdown().What changed (motivation → approach → change)
A slow teardown could swallow the one warning that names an unfinished write, because the wait that logs it ran only after the teardown returned.
In
gateway_runtime/exit_path.py, the teardown await is now wrapped intry/finally, and thefinallycallssettle_admitted_writes(graceful_deadline). A teardown that is cancelled or raises still reaches that wait. It is bounded by the same deadline as the outer bound, so when the graceful bound is what cancelled the teardown it logs the pending count and returns at once, and the cancellation then propagates as before. The owning gateway spec says the same.Backwards compatibility
Compatible: the normal path is unchanged; a cancelled or failed teardown now also runs the existing deadline-bounded wait before its exception propagates.
Removes nothing: the change only adds a wait on the error path; no key, route or guard is taken away.
Tests
test/test_slack_gateway.py::TestShutdown::test_a_cancel_during_the_teardown_still_names_the_wedged_write: a real AutoNudge service with a write wedged past the drain cut, a teardown that never returns, and a cancellation delivered once the teardown has started. It asserts the shutdown re-raises the cancellation, the write is still pending, and the "at the graceful shutdown deadline" warning was logged. The cancel is driven by the teardown-entered event and the graceful budget is0(already expired, as it is when the bound fires), so no clock race decides the result.Proof: PROVEN test/test_slack_gateway.py::TestShutdown::test_a_cancel_during_the_teardown_still_names_the_wedged_write (head e609b1b)
Determinism: 20/20 green at
-n0; 10/10 green at-n 4 --dist loadgroup --max-worker-restart=2withTestShutdownandtest/test_monitor_turn_completion.pyas co-tenants;TestShutdown(23 tests) green in a shuffled order, seed 29295.Manual verification
N/A — unit coverage sufficient: the test drives the real
_shutdown()and a real AutoNudge service; only the teardown is replaced by one that never returns.Screenshots / video
N/A — backend only, no user-visible UI change.
Related Issues
Closes #29158
Sibling sweep
Swept: git grep -nE "settle_admitted_writes(" -- src/
Siblings: none -- exit_path.py is the only caller of settle_admitted_writes, and the interrupted-drain branch runs its teardown as an owned task bounded by the deadline
Pattern harvest
Not generalizable: the defect is one ordering in one shutdown method, a log step placed after an await the outer bound can cancel; there is no shared shape a rule could match without flagging every intended post-await step.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)