Repository navigation
Conversation
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
UX Review (Opus 5.5, fork) — ✅ PASSUX-level review of UX-Verdict: PASS The only new user-visible element is the restart message, and attachment-01 shows it. No first-time reader has looked at it, because fork reviews skip the blind read. Pushing the branch to this repository would get it one. [UX-REVIEWED] a2b4e7e Description read[DESCRIPTION-READ] c87d3c9712a8ea17d6a23f4c5cda383bfabe4dd5aafc57f8d3c61604cd1d340e That sha256 names everything this verdict read: the PR description, plus the 5 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: shutdown waits only on admitted work and stops at the graceful deadline, all ten readers of the new refusal are listed, and Notes
[DESIGN-REVIEWED] a2b4e7e Description read[DESCRIPTION-READ] 907eccae9cba7421a5212cfe9aefd9bf83074b76a69cef7eb5438a293e946647 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The defect is real on the base: Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: when the gateway restarts, finish every goal-loop and monitor update it already accepted instead of losing it or racing it. FIX.
[FIRST-PRINCIPLES-REVIEWED] a2b4e7e Description read[DESCRIPTION-READ] e129c9058b4eff3336d2a8feb54b8a258c91625107b870f1c8a4c8c6847e4743 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
The approval-stall restart predicate now also requires
|
|
A maintainer attempted a rebase onto main One conflict, in Your call: keep #8993's blanket label, narrow its test to resumable stops, or add a third label. Everything else rebased cleanly. |
|
Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. |
6cfd444 to
36c1240
Compare
36c1240 to
7b4e562
Compare
|
|
|
Round push:
Validation: the 34 changed test modules and every |
|
|
|
|
|
|
|
|
|
This round's review of
|
|
|
|
|
|
|
Rebased onto Main's #22217 counts each monitor tick's outcome in Verified on
This push only resolves the conflict. The |
|
HOLD at 79a98a7. GPT and Opus have no findings on this head, UX passes, and the branch merges cleanly. What blocks approval:
The disclosed re-probe residual is tracked by the stacked PR #18648. — crew-pr-reviewer (r117) |
|
|
|
Rebased onto Main's #22837 split that module by behaviour and moved the stop-race tests this PR edits into The push also carries First Principles item 3, accepted in 6321510908. The manual-fire route's duplicate approval-hold refusal arm is gone: the shutdown refusal is a branch inside the existing Verified on
The |
This is the round's single push, made after every workflow run on Checked locally on this head: the owning test modules and every |
|
HOLD at a2b4e7e. Opus, GPT 6.1, Design, First Principles and UX are clean on this head, the r117 First Principles items have dispositions, and every check is now green. What still blocks approval:
Non-blocking: I read the description, the credential and shutdown code and the lane bodies, not every test module. I did not run tests. — crew-pr-reviewer (r119) |
bolichen97
left a comment
There was a problem hiding this comment.
[operator: bolichen97] My earlier change request is addressed on a2b4e7e. Approving to clear it; other open gates on this PR still apply.
|
This change has already shipped in full. #18648 was stacked on this branch and carried this PR's commit ( Two items are not in this PR and are tracked as a separate follow-up:
|
|
Thanks @rubencu for this work. Closing as superseded, as you suggested: #18648 was stacked on this branch and its squash merge 9c9d87e (merged 2026-10-23, now on main) carries this PR's full change. Main already has test/test_autonudge_shutdown_drain.py, test_autonudge_shutdown_mutations.py, the settle_admitted_writes drain in slack/gateway_runtime/exit_path.py, and nudge_channel_fire.py. A rebase would leave an empty commit. The two follow-ups you listed (one helper for both delivered-marker clears, splitting autonudge_service/gate.py) can go in a new PR. |
Problem / Motivation
Goal: Gateway shutdown finishes every goal-loop and monitor write it already accepted, so a restart never loses or races an accepted update, cycle or turn charge.
Why it matters
The goal indicator must reflect real autonomous work. A stale restart or an abandoned persistence write can mislead the operator or lose lifecycle state.
Not a goal
campaign/package.What changed (motivation → approach → change)
Symptom:
_shutdown()never shut AutoNudge down. It closed sessions, channels and storage while a loop update, a timer callback's bookkeeping or a monitor's turn charge could still be waiting on the service lock or an executor write.Root cause: AutoNudge had no admission boundary and no drain. Every write called
run_in_executordirectly, so once its caller was cancelled nothing knew the write was pending.Why a bounded wait on in-flight writes is not enough: each store write is atomic, so the file is never torn; what shutdown loses is an accepted change. Two races lose one even under a bounded drain. (1) A caller
_shutdown()cancels while it waits for the service lock has not submitted its write yet, so a drain over submitted futures never sees it. (2) A handler or timer callback still running after the drain's snapshot admits a new write that lands after storage closed. The admission lease closes (2); running each admitted transaction in a registered child that outlives its caller's cancellation closes (1). Admission covers only the bookkeeping a delivery already earned, so shutdown never waits on an inline channel turn.shutdown()closes admission synchronously. Each public mutation takes a generation lease before its first await; a timer callback holds a multi-write lease (owner-task contextvar) only inside bookkeeping its delivery earned. A later caller getsNudgeAdmissionRefusedwith a typed reason: every mutation route answers an audited 503 "The gateway is restarting, so this was not applied. Try again once it is back."; session-change refusals stay 409._start_persistence, registers every executor write.remove,clear_terminal_monitor,stop_monitor,update_monitorandrelease_approval_holdrun their admitted transaction in a registered, shielded child; a cancelled caller is held until its own write settles, as on main.shutdown()cancels timers outside drained bookkeeping, then drains owners and raw writes through one_DurabilityDrain. It bounds the waiter (PERSISTENCE_DRAIN_GRACE_SECS, 2 s after a cancellation), never the write. A timer arm after closure is refused._shutdown()drains AutoNudge before sessions, channels and storage close, keeps the closed service published through dashboard cleanup, and bounds the rest of teardown byGRACEFUL_SHUTDOWN_SECS. A Slack nudge's post-reply bookkeeping joins the drain through the publicAutoNudgeService.drain_section(). When it cuts that drain so its teardown can start, it then waits on the admitted work still in flight through the publicAutoNudgeService.settle_admitted_writes(), until its graceful deadline, so the hard exit cannot abandon an admitted write.terminal_deliveredbefore its re-probe, so a restart in that window settles instead of repeating the turn, and a retarget clears it; shutdown now cancels a re-probe in flight, which without the marker repeats the turn. Main repeats it on any restart in that window, so it is also a fix of its own: whether it ships here is the open maintainer decision (needs-author-decision). State version stays 1.add, which removes it after the new row's write and before its timer arms, inside the admitted transaction, so a closed or racing shutdown keeps the old loop's stop signal.ensure_patrollogs the new shutdown refusal at DEBUG, not WARNING: a bind racing a gateway stop meets it on every such stop, and the bind still commits. Other 503s stay WARNING.Docs:
learn-cron-dashboard.md,messaging.md,slack-gateway.md,monitor-architecture.md. Callback scope, rollback safety, context isolation and the reconciliation with #16993 and #17069 are under Change detail, after the tests.Backwards compatibility
Compatible: a running service behaves as before; new keyword arguments default to the old behaviour, and only calls that arrive after
shutdown()began are refused. Monitor records stay at state version 1; a gateway that predatesterminal_deliveredkeeps it inextra_fieldsand writes it back.get_instance(),Noneonce shutdown finishes, and logs instead of raising during the drain.test_shutdown_drains_autonudge_before_session_storage_close).terminal_deliveredinextra_fields(test_a_rollback_keeps_a_delivered_but_unsettled_monitor_running_and_delivers_nothing)./goal clearcallsremovewith no handler, so a refusal reaches that turn; nothing is removed.errorin the inlineauto-nudge-errornotice, so a 503 reads the restart sentence (main's "a refusal lands inline" test pins a 503's text there).Exceptionand logs;_remove_terminating_loop(catchesOSErroronly) passes it to_settle_campaign_from_watchdog._prepare_loop_launchit passes the refusal to its caller.revoke_crew_execution,_wake_crew_bodyandsuspend_crewscatchException;watchdog_cyclepasses it tosweep_repo.execution._releasecatchesExceptionaround it.Tests
Current validation on
a2b4e7e606(one commit directly on4533b1eb5b). Opus 5.5's review of7fa6681592found thatmark_monitor_turn_acceptedput the whole accepted dashboard turn task in AutoNudge's shutdown drain (_inflight_adds), so a gateway stop waited on a running provider turn: up to the gateway's 2s drain cut plus the 2s post-cancellation window, after whichsettle_admitted_writeswaited on it again. The accepted task now stays out of_inflight_adds, and shutdown no longer clears that task's captured completion authority; the entry ends with its owner task. A completion the turn emits while the gateway closes its sessions still persists under its lease, andsettle_admitted_writesdrains that write, so shutdown still never discards an accepted wake's completion.test_MUTATION_accepted_dashboard_completion_after_shutdown_is_durablepins both halves:shutdown()returns while the accepted turn is still running, and the turn's later completion is on disk after a reload. With only that test applied it fails on7fa6681592, where shutdown is still waiting at the test's 30s lost-run ceiling. Main's #22837 splittest_monitor_directive_apply.pyby behaviour, so this change's edit to the stop-race tests there (the fake lock-acquire helpers accept and forward keyword arguments) moved with them intotest_monitor_directive_apply_stop_races.py, the same 14 lines. The manual-fire route's two approval-hold refusal arms are one now: the shutdown refusal is a branch inside the existingexcept Exception(First Principles item 3). Main's #22217firing.pyconflict stays resolved as before: its tick count runs just before this change's_settle_fire_cycle. Every other per-file patch ID is unchanged from7fa6681592. The owning test modules and everytest_slack_gateway*.pymodule pass (1,727 + 1,087 tests), as do the three directive modules after the split, the manual-fire tests and main's monitor-loop counters (187 tests), and the flake-pattern and wait-discipline ratchets. Main's owner splits since52a302b914moved three of this change's hunks without changing them: the monitor-update hunk ofsession_directive_apply.pynow sits insession_directive_apply_monitor.py(#22138), theautonudge.pyfacade docstring lists_DRAIN_CANCEL_GRACE_SECSamong main's current seams (#22116), and the spec paragraph keeps main's renames; every other hunk replays line for line. Main's #21464 moved most oftest_slack_gateway.pyinto topic modules; this change'sTestShutdowntests stay intest_slack_gateway.py, which now imports what they use. Main's #21552 stops speculative session starts before_shutdown's first await and awaits them with the session closes; since that teardown now runs in_stop_services_and_sessions,_shutdownhands it the stopped tasks through the keyword-onlyeager_spawnsargument (the composition contract records the new signature). The shutdown-drain tests remain split across four collected topic modules plus helpers, and every newly added test module stays below the 2,000-line cap.gateway_shutdown_budget.py, which this change formats, is black-clean, so it leaves.github/black-baseline.txt; the baselined Black gate refuses a clean file that is still listed. Baselined Black, isort, Flake8, mypy, the comment-history, brand-name, subprocess-encoding, documentation-lint, line-cap and harness-parity gates, and the single-commit push guard pass on this head.test/test_autonudge_persistence_submission_point.py: walksautonudge.pyand everyautonudge_servicemodule and fails if_write_stateor_saveis handed torun_in_executor,to_threadorsubmitanywhere but_start_persistence, the single submission point the drain relies on; a planted submission proves the scan finds one.SERVICE_SHUTTING_DOWN_MESSAGE); the tests that pinned its old wording now pin this one.monitorFixtures.ts,automationNormalizer.test.tsanduseWebSocketGoalLoopSeed.test.tsread the monitor state version frommonitoring/contract.jsoninstead of a literal, so a fixture cannot drift from the contract version. No user-facing change.test/test_autonudge_shutdown_drain.py: shutdown waits for a blocking executor write; a cancelled shutdown still drains it; a wedged write is abandoned after the grace window without being cancelled; an uncancelled shutdown waits past the window; repeated cancellations stay inside one window; one drain spans both phases; a started timer callback drains through its bookkeeping; a dormant timer is cancelled before its callback; started channel callbacks cannot rearm; admitted opposite mutations inside a callback commit; admission does not leak to a detached child; the boundary refuses late add/update/remove/monitor calls; an admitted mutation commits after closure; a cancelled admitted add is drained; a turn hook after shutdown arms nothing; shutdown never awaits its own task; a terminal re-probe in flight at shutdown is cancelled, and the next start settles it once without a second turn.test/test_autonudge_shutdown_drain.pyandtest/test_autonudge_settlement.py(delivered marker): a landed owed turn moves intoterminal_deliveredbefore its re-probe and a failed first write keeps it delivered for the cycle persist; restart settles a delivered terminal without refiring and clears a reopened one; a record with onlyterminal_pendingstill delivers once; the settlement runs before the gate opt-out; failedgoneandholdswrites leave memory, store and restart agreed; aholdssettlement retires a timer re-armed during its write andfire_nowrefuses during it; main'stest_an_unobservable_subject_does_not_get_settledpasses unmodified.test_a_rollback_keeps_a_delivered_but_unsettled_monitor_running_and_delivers_nothingwrites the delivered-but-unsettled record through real persistence, asserts the stored row is at version 1 withterminal_pendingempty, loads it with a decoder and encoder that predateterminal_delivered, and asserts the loop loads active and re-armed, not blocked or quarantined, a quiet tick delivers nothing, and the key survives that build's write; the upgraded gateway then settles once without a turn.test/test_monitor_turn_completion.py: a started completion drains before session close; caller cancellation propagates only after the durable write; a wedged completion write releases the caller after the window while the owner stays registered; a completion that never started is refused after closure; shutdown returns while an accepted dashboard turn is still running, and that turn's later completion persists under its captured lease; cancellation wins over a failed completion write.test/test_autonudge_slack_terminal_completion.py:test_a_slack_wake_that_ends_its_monitor_still_posts_its_replydrives the real timer, tick and dispatch path for aslack:monitor whose wake runs inside the timer task and charges its completion there. For a spentmax_agent_turnsbudget, an approval stall recorded during the turn and a user stop during the turn, the wake still reaches its turn-row, reply-post and replay-save steps, the timer task finishes uncancelled, and the loop and the store both hold the terminal outcome. All three cases fail on the published head6de2c8d5.test/test_slack_gateway.py: gateway shutdown awaits the AutoNudge drain, drains it before session storage closes, keeps dashboard accounting durable before session close, and continues cleanup after a cancelled drain. After the gateway's own drain cut,test_a_write_slower_than_the_drain_cut_lands_before_shutdown_returnslands a write that outlasts the cut before_shutdown()returns, andtest_a_wedged_write_past_the_drain_cut_still_ends_shutdown_at_the_graceful_bound(both the method's own deadline and the outer bound) ends shutdown with the write pending and its count at WARNING.test/test_autonudge_shutdown_terminal_reprobe.py:test_a_retarget_drops_the_previous_subjects_delivered_terminal_marker: a retarget clears the previous subject's delivered marker, in memory and on disk.test/test_slack_gateway_composition_contract.py: neither the Slack facade nor an owner imports anautonudge_servicemodule, so a nudge's post-reply bookkeeping reaches the drain only throughAutoNudgeService.drain_section()(re-adding the private_drain_registered_sectionimport turns it red);test_shutdown_drains_the_reply_bookkeeping_instead_of_cutting_itstill drains the reply through that section, and the refactor contract records the new member and its binding.test/test_slack_gateway.py,test/test_slot_close_nudge_race.py, andtest/test_autonudge_manual_fire.py: normal and owned interrupted teardown keep the closed singleton visible through dashboard cleanup and withdraw it afterward; a slot close in that window fails withnudge_retire_failedwhile its open tab and active memory/disk loop row agree; a closed service's manual fire arms nothing and maps to literal 503shutting_downwith a denied audit.test/test_autonudge_approval_stall.py: an approval hold admitted before shutdown lands and one after closure is dropped; the same for a release; a release that lands after closure arms nothing; a held loop with a delivered terminal marker neither fires nor re-probes on the hold tick, and after the release settles once with no second turn while the marker stays untouched by the release.test/test_autonudge_shutdown_drain.py: shutdown during an open-ledger extension lands the raise as one write or drops it, with store and memory agreeing and nothing armed after closure.test/test_autonudge_shutdown_drain.py(fix(work-ledger): arm a work-ledger patrol on a conductor's bind when it has none #17069): shutdown during a displaced default patrol's wake cancels the tick mid-delivery, drains it inside its delivery record, or finds it finished, and in each case the removed row stays out of the store, is not announced as fired and is not re-armed; a failed displacement gives no tick back after closure, whether that tick was cancelled or is draining; a completion for the displaced row resolves through shutdown without a charge or write, and one after closure is refused.test/test_conductor_patrol.py: a bind after closure is refused by the authorizer beforeadd, with the shutdown reason, logged at DEBUG, and still commits; a closed service's refusal outranks an unavailable audit log; an audit outage that races closure keeps its own 503 at WARNING (test_an_audit_outage_that_races_closure_still_logs_its_refusal_at_warning). The four shutdown-test stub waits are bounded by_LOST_RUN_SECS.test/test_autonudge_authz_cov80.py: a closed service's POST arm preserves the active loop, its store bytes and its stop sentinel; an add racing closure receives the stale path and keeps the existing default sentinel; a stale default sentinel is still present when the new row is written and gone beforeaddreturns, with the timer armed; a new goal after a stop-file finish is not killed by the stale file;test_a_caller_cancelled_during_the_add_leaves_no_stale_sentinel_and_the_goal_runscancels the caller while the replacement's write is held and asserts the row committed, the stop file is gone and the first tick runs the goal (fails on the published headb342f186: the file remains and the first tick finishes the loop); a shutdown that follows the cancelled caller drains the removal, so the next start loads the new goal active with no stop file; a replacement whose write fails leaves the existing loop and its stop file in place; a stale entry that cannot be removed (a directory) saves the goal finished with no timer, persisted, withaddedthenupdated, the warning naming the path, and runs once the entry is gone; a failed finished-save leaves the active row to its first tick;test_a_stale_sentinel_finish_is_published_only_after_its_write_commitsholds the finished-save write and asserts the live row still reads active while the payload being written already carries the finish (fails on5b2373c5df, which set the live row before the write); a path under a regular file arms the goal as usual.remove,stop_monitor,clear_terminal_monitor,update_monitorandrelease_approval_hold, a caller cancelled mid-write is held until the write settles, and a closure cancel during that hold is absorbed; a prior write on a callback's multi-write lease, a detached_persist_soon, and an observer's task cannot hold a cancelled caller or count toward its transaction; main'stest/test_autonudge_remove_offload_425.pyand the spec-buildertest_cancelled_persistence_cannot_be_overtakenpass unmodified. A removal cannot land between the credential liveness check and the trust write. A cancelled guard exit still releases its user count. Tasks an observer creates see no transaction owner, no submission token and no monitor update post-commit handle (test_an_observer_task_cannot_reach_a_monitor_updates_post_commit_handle).test/test_perf_boot_path.pyfollows_shutdowninto_stop_services_and_sessionswith its three update-check assertions unchanged;test/test_slack_gateway_more_coverage.pyandtest/test_autonudge_stop_auth.pyadd a delivered-marker case beside each owed-marker case; the fault injectors intest_a_failed_settlement_write_leaves_the_terminal_turn_owedandtest_a_retarget_during_lock_acquisition_does_not_crash_the_settlementselect the staged settlement write and the delivered marker, with their assertions unchanged apart from the delivered field and an added injector-fired assertion. The frontend fixtures read the contract's monitor state version, which stays 1. Seam fakes in the monitor, wake-judge, cycle-failure, provider-trust, directive-apply and spec-builder tests accept the newadmissionkeyword; the refactor contract manifest records the new members, signatures, bindings and seam.threading.Event: the shutdown-drain, ledger-patrol, mutation, authz and turn-completion tests poll it withasync_wait_until(<event>.is_set). They do not park<event>.waiton the default executor, which the flake-pattern ratchet refuses (K15, test: bound held-side event waits by the lost-run bound and ratchet to_thread waits #20418). Thetimeout=keyword this change adds to_within_lost_runlives intest/slack_gateway_helpers.py, where main's Slack gateway test split put that helper.terminal_pendingturns the rollback test and the restart-settlement tests red; letting observers keep the post-commit handle turns its observer test red; dropping the hold's or the release's lease, arming the release past the guard, or writing the ledger extension outsideupdateeach turns its new test red; re-arming the removed row, giving the tick back after closure, writing the removed row's completion, cancelling the firing default instead of detaching it, or logging the shutdown refusal at WARNING each turns its fix(work-ledger): arm a work-ledger patrol on a conductor's bind when it has none #17069 test red; leaving the completion child without its caller as owner turns all three cases oftest_a_slack_wake_that_ends_its_monitor_still_posts_its_replyred; moving the stale sentinel unlink back aftersvc.addreturns turnstest_a_caller_cancelled_during_the_add_leaves_no_stale_sentinel_and_the_goal_runsred by name; dropping thesettle_admitted_writescall turns the two new shutdown tests (three cases) andtest_shutdown_awaits_autonudge_drainred; dropping the retarget reset ofterminal_deliveredturns the retarget test red.Change detail
shutdown()closes admission synchronously.add/update/remove/add_monitor/update_monitor/stop_monitor/record_monitor_turn_completioncapture a single-write generation lease before their first await; a timer callback that crosses its post-sleep boundary holds a multi-write lease (contextvar, owner-task scoped, never inherited by a detached child) for the writes its own task makes, handed explicitly to the shielded judge-state and debounced persists. New callers after closure getNudgeAdmissionRefused._drained_bookkeeping: the fire cycle's new_settle_fire_cycle, a structured monitor's dispatched/busy/failure record) and the caller-owned public mutations register the callback for the drain; the fire cycle refuses to start a delivery after closure, and a cancelled delivery's exit bookkeeping still commits under the callback's lease. A running-callback registry reaches a callback whose own loop was removed mid-callback.terminal_pendinginto a newterminal_deliveredfield and persists that before its provider re-probe, so a restart in that window settles without repeating the turn. A failed first marker write keeps the delivered fact in memory for the cycle persist instead of downgrading it to owed. Restart recognizes a setterminal_deliveredas settlement-only, without invoking_on_fire. The re-probe answersholdsorgone:holdssettles, and every other reading (the subject came back, or the re-probe could not read it) clears the marker as main does. Both branches stage their change, andholdsstages its outcome and deactivation with it, so each publishes to the live monitor only after the write lands: a failed write costs one extra re-probe instead of diverging from restart, and a failedholdswrite announces nothing. Publishing a successfulholdssettlement immediately retires the loop's timer, including one re-armed by turn completion during the staged write.fire_nowrefuses with 409 during that write instead of arming a timer the publish would cancel. A record with onlyterminal_pendingkeeps its owed-turn behavior. A retarget clearsterminal_deliveredwith the rest of the observation baseline, so the previous subject's delivered turn cannot settle the new subject without observing it. The re-probe runs between two drained sections, never inside one. Shutdown cancels a re-probe already in flight with its callback instead of waiting on it, a callback drained through the delivery record starts none, and either way the delivered marker stays owed for the next start.terminal_pendingkeeps main's meaning (the owed turn, empty when none is owed) andMONITOR_STATE_VERSIONstays 1, so no record is rewritten at a new version. A gateway rolled back past this change filesterminal_deliveredunderextra_fields, writes it back untouched, reads nothing owed and keeps every structured monitor running; the upgraded gateway later finds the marker and settles once. The expiry notice and themonitor_updateapplier readterminal_deliveredbesideterminal_pending, so a delivered turn whose cap won the race is still worded and refused revival as terminal.NudgeAdmissionRefusedcarries a typedNudgeAdmissionReason. Shutdown refusals return an audited 503 from every mutation authorizer and the legacy DELETE route; session-change add refusals remain 409._start_persistence, which registers the raw executor future.remove(),clear_terminal_monitor(),stop_monitor(),update_monitor()andrelease_approval_hold()run their admitted transaction in a registered child. A caller cancelled while its mutation is being written is held until the write settles and then receivesCancelledError, as on main (a wedged write releases it after the drain's post-cancellation window); the hold is decided by the transaction's own submission, counted on a token carried in the detached child's context, so a detached persist under the same callback lease never holds it._persist_soonclears that token for only the task it creates while preserving lease inheritance, so the release-path deadline persist cannot increment the child transaction's count. A caller cancelled while it still waits for admission or a lock is released at once, and the admitted mutation completes in the registered child, whichshutdown()drains. Completion accounting runs in a shielded owned task._emitclears the transaction owner, the submission token and a monitor update's post-commit handle for the observers it calls, so a task an observer creates can neither pass for the owner's callback, count toward the transaction's writes, nor reach the update's credential continuation.record_monitor_turn_completion's owned task carries its caller's task as_admitted_transaction_owner, set beforecreate_taskand reset after, as_run_admitted_transactiondoes. A Slack monitor's wake runs its turn inside the timer task, and a completion that ends the loop (spent turn budget, unanswered approval, user stop during the turn) retires that timer from the child;_cancel_timermust read that as self-cancellation, or it cancels the timer task awaiting the child and the wake never writes its turn row, posts its reply or saves it for replay._schedule_release's failure warning keeps its traceback.shutdown()cancels every timer outside drained bookkeeping and the reconciler, then drains control tasks, owners and raw writes through one_DurabilityDrain. The drain bounds the waiter, never the write: after the first cancellation it keeps waiting forPERSISTENCE_DRAIN_GRACE_SECS(2 s, insideSIGNAL_MARGIN_SECS) and then leaves; the write keeps running and is atomic._arm_timer/_arm_from_deadlinerefuse after closure, so drained work and late turn hooks cannot rearm. The runtime claim sets are released only once the drain settled; an unsettled drain withdraws only the singleton and its crew-log subscriptions._shutdown()cancels handler tasks, drainsautonudge_svc.shutdown()before awaiting those handlers, then gathers the handlers as the first bounded step before cron/heartbeat and session/channel teardown. A handler still running after admission closes receives the intended 503 shutdown refusal at its next mutation write. A Slack nudge whose provider already returned holds one section, opened throughAutoNudgeService.drain_section(), from that return through its completion report, usage row, session release, thread post and replay save, so a stop landing there drains the reply instead of cutting it. When_shutdown()cuts the drain after onePERSISTENCE_DRAIN_GRACE_SECSwindow so its teardown can start, that cut is not durability: after the teardown it awaitsAutoNudgeService.settle_admitted_writes(graceful_deadline), which waits on every admitted mutation and executor write still in flight, re-snapshotting each pass, until the graceful deadline. Past it, or when the graceful bound cancels the wait, the pending count is logged at WARNING and shutdown ends; the write keeps running on its thread and the store write is atomic. Main waited on a slow write until the supervisor's kill; the drain cut alone would have hard-exited over a write slower than about 4 s._stop_services_and_sessions()finishes dashboard cleanup on both the normal and owned interrupted paths, then the gateway withdraws the singleton and its crew-log subscriptions. A slot close in that window reaches closed admission and aborts instead of silently persisting a closed tab beside an active loop.fire_nowlikewise answers an audited literal 503shutting_downbefore its guarded timer arm, rather than reporting 200 over a no-op._shutdown()(moved into_stop_services_and_sessions, with the handler gather first) runs in a separate owned task._shutdown()waits withasyncio.wait(..., timeout=remaining)only until the graceful deadline measured from its own start; at expiry it cancels the owned task without awaiting it, so a handler removal or another teardown step that swallows cancellation cannot extend the waiter, and_shutdown()raises a freshasyncio.CancelledError().gateway_shutdown_budget.pyandslack-gateway.mdstep 3 state the same bound.autonudge_service/model.py, drain inmaintenance.py, owner registration inmutations.py/monitor_records.py, callback admission infiring.py, arm guards intimers.py, lifecycle and persistence entry points inautonudge.py._DRAIN_CANCEL_GRACE_SECSis a declared facade seam._record_approval_hold) and the release (release_approval_hold) are admitted and drained.notify_approval_stalledtakes the hold lease synchronously, registers the recorder task in_inflight_addsbefore its first step, and shutdown's re-snapshot loop waits for it._schedule_releasetakes the release lease where the person's action arrives, a direct release is admitted byrelease_approval_holditself, and its body runs in a registered child. One that arrives after closure is refused. The hooks drop it at DEBUG without writing or announcing, and a direct release of a still-held loop raises the shutdown refusal. A release that lands after closure arms nothing through the existing arm guards._extend_for_open_ledgerneeds no separate path because itsupdateruns under the tick's lease, and the running-callback registry still reaches a tick detached from the timer table. Shutdown therefore lands the raise as one write or drops it before the write starts.release_approval_holdgains anadmissionkeyword recorded in the refactor contract signature. The catch-all release warning on the manual-fire route keeps its traceback.detach_firing_default_timertakes out of the timer table stays in the running-callback registry, so shutdown cancels it while it delivers and drains only its registered bookkeeping sections. Those snapshot the live table, which no longer holds the removed row, so the row is never written back, announced as fired or re-armed. The give-back on a failed replacement returns the tick to the timer table only while admission is open. A turn completion for the removed row is dropped at DEBUG without a charge or write, and its drained transaction resolves as soon as it holds the lock, without a write. The default patrol arm on a bind is an ordinary admittedadd, refused after closure, andensure_patrollogs that refusal at DEBUG while the bind still commits. Main'sdetach_firing_default_timeris unchanged. The merged_add_locked/_add_unserializedcarry bothdefault_patrolandadmission(refactor contract signature updated).addasretire_stale_stop_sentinel, and the transaction removes the file after the new row's shielded write and before its timer is armed, so an already-closed service, a closure racingaddand a failed write all preserve the existing loop's stop signal, and a caller cancelled mid-add cannot leave the file for the new loop's first tick (main unlinked before the add and did not shield; the published head unlinked afteraddreturned). A stale entry that cannot be removed and that the timer's ownexistsstill sees is logged with its path and the new loop is saved finished understop_sentinel, timerless, in a second write under the add's lease (only an add carrying a stale path takes a multi-write lease), emittingaddedthenupdated; the finished state is written from a staged copy and the live row changes only once that write is durable (a concurrent reader never sees a stop the store does not hold), and if the write fails the committed active row stands unchanged and its first tick finishes it, as before. A held-loop manual fire maps shutdown to literal 503shutting_downwithout a warning traceback, and the shutdown spec distinguishes settled slot-close refusal from an unsettled writer's maintenance-lock wait.learn-cron-dashboard.md,messaging.md,slack-gateway.mdandmonitor-architecture.mddescribe admission, the drain, the delivered marker and the shutdown order; a stale comment and spec sentence claiming the goal popover sendsactive: trueon every edit of a running loop now say what main does.Disclosed residual, closed by the stacked follow-up:
fix/autonudge-delivered-reprobe-retentionretains the marker across two such readings before clearing.Design questions
Admission is scoped to bookkeeping, not to the callback: a timer callback is cancelled at shutdown like any timer, and is drained only while it sits inside bookkeeping its delivery already earned, because the alternative (draining the whole callback) made every gateway stop wait on an inline channel turn that only session close could end. The carrier stays the owner-scoped contextvar lease plus caller-owner registration rather than explicit leases on owned inner tasks, because the bookkeeping a cancelled delivery writes on its way out (the controller's busy record, the gateway's completion report) runs after closure on the callback's own task and is admitted only through that lease, and an inner task would break the self-re-arm the settlement performs from the timer task.
The delivered marker is a new field, not a new value of
terminal_pending: a value change needs a state-version bump to keep an older gateway from reading the marker as an owed turn, and an older gateway retires every record at a version it does not know, so a rollback would stop every live structured monitor for good. A field an older decoder does not know lands inextra_fields, which that decoder keeps and writes back.Advisories, not implemented in this change: the one-turn exit-bookkeeping grace is a heuristic (~3 loop iterations; registration after 3+ extra awaits is missed and its write lands after shutdown() returned because the lease generation is never bumped -- bumping
_mutation_generationonce the drain settles would turn that into a refusal); when the drain is interrupted by the bound itselfremaining ~= 0so a drain deadline belowgraceful_deadlinenow matters for every gateway, not only--no-crons; two sets instead of mixed_inflight_addsmembership; guardowner.cancel()on "no other active section for this owner" (re-entrancy).An orphaned
irq.pollexecutor thread still occupies a default-executor worker untilghreturns; nothing waits on it, but drained_write_statecalls share that pool.One more from the final local review round, also not implemented here:
_settle_delivered_terminalholds the service lock across its re-probe, where the ordinary terminal branch probes first and re-checks under the lock afterwards -- the same shape would stop a settlement tick from blocking public mutations whileghis slow.Manual verification
Screenshots / video
The goal popover's inline notice showing the two refusals this change returns, rendered from the built dashboard with stubbed API answers and synthetic data.
Related Issues
No linked issue; the behavior was reported directly from a live dashboard session.
Pattern harvest
Rule candidate: review-prompt
Pattern: a service whose writes are executor futures awaited by cancellable callers needs one registered submission point and a shutdown drain that bounds the waiter after cancellation, never the write.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement