Repository navigation
fix: make a stale-wake stop a compare-and-remove under the lock - #14535
Conversation
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify Suggestions
[DESIGN-REVIEWED] c225556 |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The lock-ownership registry ships zero observable change — "every in-tree caller already held the lock" — while the precondition re-check alone removes the defect. Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: a wake-delivered stop must never delete or overwrite a loop a person paused, whatever interleaving occurs — a FIX (issue #14533, regression tests failing on main).
[FIRST-PRINCIPLES-REVIEWED] c225556 |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed |
|
|
|
|
|
74bb624 to
0eade8e
Compare
|
Pushed 0eade8e. The PR is still two commits; the second commit was amended. This is the final round agreed for this PR.
|
|
|
|
Rebased onto main note: the two red shards on
|
b53343c to
338a45d
Compare
|
note: Rebased onto main The two Windows reds on
|
338a45d to
8f6dd25
Compare
|
note: Rebased onto main The two shard-4 reds on |
The retention and identity check that protects a person-paused loop now runs as a precondition inside the same locked section as the removal, so a pause that lands while the stop waits for the lock is kept instead of deleted. The unserialized removal path raises unless its caller holds the service mutation lock.
On a research-owned slot the stale-wake stop's update takes the same lock-held precondition as removal, so a concurrent pause is kept. A stop whose row is already gone stays an idempotent success; only a recorded guard refusal refuses. The structured-monitor plumbing is reverted (that path was already safe), the refusal side channel becomes a small guard class, tests pin the lock guards, and the keepalive test scopes its fake clock to mcp_core so a background thread cannot consume it.
8f6dd25 to
c225556
Compare
|
Rebased onto main note: The only test failure on |
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed on behalf of the maintainers (readiness sweep 2026-09-29): readiness passed, the required PR Readiness check is green on this head, no live CHANGES_REQUESTED, no open blocking review threads; where the PR's files also moved on main since its merge-base, the PR's own tests were run on the main+PR merge tree and passed. Approving and squash-merging.
Problem / Motivation
Goal: A monitor's own wake can never delete a loop a person paused, however the pause and the stop interleave.
A wake-delivered
monitor_stoporautonudge_stopchecks, without the service lock, that the loop is still the session's monitor and was not paused by a person. It then callssvc.remove, which yields while taking the lock and re-checks nothing. A dashboard pause that lands in that window is deleted, with no row left to recover. Details are in #14533.Why it matters
A person's pause is a deliberate stop that the system promises to keep as retained evidence. Losing it deletes the loop's message, cadence, cycle count and deadline. A new user request is the only way back, and the person may not notice.
Not a goal
USER_STOPrather than removing a row.What changed (motivation → approach → change)
_StaleWakeStopGuard) tosvc.remove. Under the mutation lock, it re-checks that the row is still the wake's loop bound to this session and is not an inactive, non-replaceable (person-paused) row. A failed precondition changes nothing and returns the existing stop refusal. The lock-free check stays only as an early answer.svc.update.updatetakes the same optional precondition, evaluated on the live row under the lock, so a pause that lands while the stop waits is kept and the stop is refused. Other update callers are unchanged.on_absentinside the same lock hold. If the session's slot holds no loop, the row is gone (a user delete landed first), so the stop keeps its success wording and is audited as allowed. If the slot holds a different loop, a concurrent arm replaced it: the stop is refused with the "no longer this session's monitor" wording, audited as denied, and the replacement keeps running. A write that never took the lock (maintenance quiesce) is reported as "Monitor NOT stopped" with a retry hint, because the row may still be live.on_absent, which checks the session's slot under the lock just as the guard does. A missing row with an empty slot is success. A missing row with a different loop in the slot means a concurrent arm replaced it, so the stop answers "Monitor NOT stopped" and says the new loop is still running. A write that never took the lock (maintenance) answers "Monitor NOT stopped" with the retry hint._refuse_stale_wake_stopand the "Monitor NOT stopped" refusal wording. Both are kept: the pre-check answers early, the guard is the authoritative re-check, and the guard's refusals now use the same stop wording.RuntimeErrorunless the current task owns the service's maintenance lock. It uses explicit checks, notassert, so the guarantee holds underpython -O.remove,remove_by_slot,clear_terminal_monitorand the maintenance view pass the lock they hold.The spec sentence in
babysit-pr-watch.mdnow says the stop decision is taken under the lock.Backwards compatibility
None breaking. Callers of the public
removeAPIs are unchanged. Only the internal unserialized path gained a lock-ownership requirement, and every in-tree caller already held the lock.Tests
test_stale_wake_stop_preserves_pause_landing_before_remove[monitor_stop|autonudge_stop]uses a real AutoNudgeService. It forces a person's pause to land after the directive's check and beforeremovetakes the lock, then asserts that the paused row survives with its message, budget and reason. Both cases failed on main (the row was deleted) and pass with the fix.test_remove_unserialized_requires_mutation_lock: the unserialized removal refuses to run without the lock. This failed on main.test_stale_wake_stop_preserves_pause_landing_before_research_update: on a research-owned slot, where the stop sets the loop inactive instead of removing it, a person's pause that lands before the update takes the lock staysmanualand the stop is refused. Failed before this change (the row becameautonudge_stop, which auto-research reads as a finished campaign).test_stale_wake_stop_of_row_replaced_before_write_is_refused[...]: a concurrent arm replaces the wake's loop before the stop takes the lock, onmonitor_stop,autonudge_stopand the research-update path. The stop is refused and audited as denied, and the replacement stays active. All three failed on the previous head, which reported "No further nudges will fire".test_stale_wake_stop_that_never_took_the_lock_is_not_reported_stopped: with the mutation lock unavailable, the live row is untouched and the stop answers "Monitor NOT stopped". Failed on the previous head, which reported success.test_user_stop_that_never_took_the_lock_is_not_reported_stopped[...]: the same for a person's ownmonitor_stop/autonudge_stopwith no wake id, including the research-update path. All three failed on08bf64cd2c.test_user_stop_of_row_replaced_before_write_is_refused[...]: a concurrent arm replaces the loop after a person's stop resolved it and before the write locks. The stop is refused and audited as denied, and the replacement stays active. All three cases failed onb06e56e393.test_session_keepalive.pyscopes its fake clock tokiro_crew.mcp_core.time. It previously replacedtime.monotonicglobally, and the path-resolver pool's background reaper thread (subprocess_pool/executor.py) consumed the fake values, which failed the test on Windows in shards that included the new race tests.Targeted suites (every
test_autonudge*,test_monitor*,test_session_keepaliveandtest_auto_research_onloop*file plustest_driver_session_directives.py, on current main): 1661 passed; every other test file that callsapply_session_directive: 788 passed. flake8, black, mypy on touched sources and the comment-history gate are clean.Manual verification
N/A: a server-side race, covered by the deterministic regression test above.
Screenshots / video
Why no screenshot: No UI change.
Related Issues
Fixes #14533. Found in the review of #14514; the race is on main via #14371.
Pattern harvest
Rule candidate: review-prompt
Pattern: A decision taken before an
awaitthat acquires a lock must be re-taken under that lock (a compare-and-mutate precondition), or the mutation acts on state the decision never saw.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)