Skip to content

fix: make a stale-wake stop a compare-and-remove under the lock - #14535

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
Pearcekieser:fix/stale-wake-stop-race
Sep 29, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
Pearcekieser:fix/stale-wake-stop-race

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A monitor's own wake can never delete a loop a person paused, however the pause and the stop interleave.

A wake-delivered monitor_stop or autonudge_stop checks, without the service lock, that the loop is still the session's monitor and was not paused by a person. It then calls svc.remove, which yields while taking the lock and re-checks nothing. A dashboard pause that lands in that window is deleted, with no row left to recover. Details are in #14533.

Why it matters

A person's pause is a deliberate stop that the system promises to keep as retained evidence. Losing it deletes the loop's message, cadence, cycle count and deadline. A new user request is the only way back, and the person may not notice.

Not a goal

What changed (motivation → approach → change)

Behavior Change
The stop decision runs in the same locked section as the removal The stale-wake stop passes a precondition (_StaleWakeStopGuard) to svc.remove. Under the mutation lock, it re-checks that the row is still the wake's loop bound to this session and is not an inactive, non-replaceable (person-paused) row. A failed precondition changes nothing and returns the existing stop refusal. The lock-free check stays only as an early answer.
The research-slot stop has the same guarantee On a research-owned slot the stale-wake stop marks the loop inactive through svc.update. update takes the same optional precondition, evaluated on the live row under the lock, so a pause that lands while the stop waits is kept and the stop is refused. Other update callers are unchanged.
A stop whose row is missing tells gone from replaced When the lock-held write finds the wake's row missing, the service calls the guard's on_absent inside the same lock hold. If the session's slot holds no loop, the row is gone (a user delete landed first), so the stop keeps its success wording and is audited as allowed. If the slot holds a different loop, a concurrent arm replaced it: the stop is refused with the "no longer this session's monitor" wording, audited as denied, and the replacement keeps running. A write that never took the lock (maintenance quiesce) is reported as "Monitor NOT stopped" with a retry hint, because the row may still be live.
A person's own stop is not reported when nothing was written A stop with no wake id has no guard, but it still passes on_absent, which checks the session's slot under the lock just as the guard does. A missing row with an empty slot is success. A missing row with a different loop in the slot means a concurrent arm replaced it, so the stop answers "Monitor NOT stopped" and says the new loop is still running. A write that never took the lock (maintenance) answers "Monitor NOT stopped" with the retry hint.
Rebased on #14514 #14514 added the lock-free stop pre-check _refuse_stale_wake_stop and the "Monitor NOT stopped" refusal wording. Both are kept: the pre-check answers early, the guard is the authoritative re-check, and the guard's refusals now use the same stop wording.
Removal cannot run without the lock The unserialized removal path raises RuntimeError unless the current task owns the service's maintenance lock. It uses explicit checks, not assert, so the guarantee holds under python -O. remove, remove_by_slot, clear_terminal_monitor and the maintenance view pass the lock they hold.

The spec sentence in babysit-pr-watch.md now says the stop decision is taken under the lock.

Backwards compatibility

None breaking. Callers of the public remove APIs are unchanged. Only the internal unserialized path gained a lock-ownership requirement, and every in-tree caller already held the lock.

Tests

  • test_stale_wake_stop_preserves_pause_landing_before_remove[monitor_stop|autonudge_stop] uses a real AutoNudgeService. It forces a person's pause to land after the directive's check and before remove takes the lock, then asserts that the paused row survives with its message, budget and reason. Both cases failed on main (the row was deleted) and pass with the fix.
  • test_remove_unserialized_requires_mutation_lock: the unserialized removal refuses to run without the lock. This failed on main.
  • test_stale_wake_stop_preserves_pause_landing_before_research_update: on a research-owned slot, where the stop sets the loop inactive instead of removing it, a person's pause that lands before the update takes the lock stays manual and the stop is refused. Failed before this change (the row became autonudge_stop, which auto-research reads as a finished campaign).
  • Row already gone: a delete landing between the pre-check and the stop's lock leaves the stop reporting success and audited as allowed, on the remove and the research-update paths.
  • test_stale_wake_stop_of_row_replaced_before_write_is_refused[...]: a concurrent arm replaces the wake's loop before the stop takes the lock, on monitor_stop, autonudge_stop and the research-update path. The stop is refused and audited as denied, and the replacement stays active. All three failed on the previous head, which reported "No further nudges will fire".
  • test_stale_wake_stop_that_never_took_the_lock_is_not_reported_stopped: with the mutation lock unavailable, the live row is untouched and the stop answers "Monitor NOT stopped". Failed on the previous head, which reported success.
  • test_user_stop_that_never_took_the_lock_is_not_reported_stopped[...]: the same for a person's own monitor_stop/autonudge_stop with no wake id, including the research-update path. All three failed on 08bf64cd2c.
  • test_user_stop_of_row_replaced_before_write_is_refused[...]: a concurrent arm replaces the loop after a person's stop resolved it and before the write locks. The stop is refused and audited as denied, and the replacement stays active. All three cases failed on b06e56e393.
  • test_session_keepalive.py scopes its fake clock to kiro_crew.mcp_core.time. It previously replaced time.monotonic globally, and the path-resolver pool's background reaper thread (subprocess_pool/executor.py) consumed the fake values, which failed the test on Windows in shards that included the new race tests.
  • Tests for the lock guard: a task that does not own the lock and a lock that is not the service lock are both refused, and the row survives.
  • Existing tests still cover a live wake stopping its own active loop and a user stop removing.

Targeted suites (every test_autonudge*, test_monitor*, test_session_keepalive and test_auto_research_onloop* file plus test_driver_session_directives.py, on current main): 1661 passed; every other test file that calls apply_session_directive: 788 passed. flake8, black, mypy on touched sources and the comment-history gate are clean.

Manual verification

N/A: a server-side race, covered by the deterministic regression test above.

Screenshots / video

Why no screenshot: No UI change.

Related Issues

Fixes #14533. Found in the review of #14514; the race is on main via #14371.

Pattern harvest

Rule candidate: review-prompt
Pattern: A decision taken before an await that acquires a lock must be re-taken under that lock (a compare-and-mutate precondition), or the mutation acts on state the decision never saw.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested a review from a team as a code owner September 27, 2026 22:01
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention fork Pull request from a fork (external contributor) labels Sep 27, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed c2255562266ea2433693acc654de88c996155c3c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c225556

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of c2255562266ea2433693acc654de88c996155c3c via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify _claim_mutation_lock raising inside _acquire_mutation_lock cannot leave the lock held — a stale owner entry would wedge every later mutation.

Suggestions

  • In the _acquire_mutation_lock hunk, _claim_mutation_lock(lock) runs after lock.acquire() with no release on raise; if an owner entry ever leaks (a future bare lock.release() after acquire), the next acquire raises with the lock held and all autonudge mutations for that data home block until restart. Release on claim failure, or fold claim/release into a context manager so the pair cannot be mismatched at a future call site.

[DESIGN-REVIEWED] c225556

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of c2255562266ea2433693acc654de88c996155c3c via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The lock-ownership registry ships zero observable change — "every in-tree caller already held the lock" — while the precondition re-check alone removes the defect.

Not justified as shipped

  • Item 6 — rides along: the fix is the guard re-taken under self._lock; the _MUTATION_LOCK_OWNERS registry polices a different lock (_maintenance_lock), whose 5 in-tree _remove_unserialized callers (grepped) all already hold it, and the description concedes "every in-tree caller already held the lock" — so its zero option changes nothing a user or the system would notice, and its only witness is the test that ships with it.
    Subtraction: delete _MUTATION_LOCK_OWNERS, _claim/_assert/_unclaim/_release_mutation_lock, and the threaded mutation_lock parameter; keep the precondition/on_absent re-check, which is the whole fix.
    Clears when: a caller that reaches _remove_unserialized without the maintenance lock is named (a repro or defect), or the deletion is made.
  • Item 7 — rides along: test_session_keepalive clock scoping is a declared test-infrastructure fix for a Windows shard failure the new tests exposed.

What this change ships

Inventory (8 items) — 6 justified

Intent: a wake-delivered stop must never delete or overwrite a loop a person paused, whatever interleaving occurs — a FIX (issue #14533, regression tests failing on main).

  1. A pause landing while a wake's stop awaits the lock now survives; the stop is refused — justified
  2. The same guarantee on a research-owned slot, via the update precondition — justified
  3. A stop whose row vanished succeeds only if nothing replaced it; a replacement refuses — justified
  4. A stop that never took the lock answers "Monitor NOT stopped", with a retry hint — justified
  5. A person's own stop of a replaced row is refused; the replacement keeps running — justified
  6. Direct unlocked removal now raises RuntimeError (ownership registry) — rides along: enforces a discipline all 5 counted callers already satisfy
  7. test_session_keepalive scopes its fake clock to mcp_core.time — rides along: declared Windows-shard fix
  8. babysit-pr-watch.md documents the stop decision as lock-held — justified

[FIRST-PRINCIPLES-REVIEWED] c225556

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed c2255562266ea2433693acc654de88c996155c3c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c225556

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Research-slot stop keeps a concurrent pause

Disposition: fixed in 74bb624.
On a research-owned slot the stale-wake stop marks the loop inactive through svc.update. update / _update_unserialized now take the same optional precondition as removal, evaluated on the live row under the lock; a refused precondition writes nothing, and the stop raises the stale-wake refusal. test_stale_wake_stop_preserves_pause_landing_before_research_update failed before this change (the row became autonudge_stop) and now keeps manual. Other update callers pass no precondition and are unchanged.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Structured-monitor plumbing removed

Disposition: fixed in 74bb624.
Checked against the code: the structured stop_monitor already runs under _lock, looks the row up by the wake's own id, and leaves a row with an existing outcome untouched; structured rows cannot be paused through update. So the precondition added to stop_monitor and authorize_and_stop_monitor is removed along with its test; autonudge_authz.py is identical to main again, and "Not a goal" holds.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • The lock-owner check enforces the fix's invariant

Disposition: rebutted.
The fix only holds if every removal decides under the lock. The owner registry makes that invariant fail loudly: the unserialized removal raises unless the current task owns the service lock, so a future caller cannot reintroduce the check-then-remove race by calling it bare. It is declared in the description, raises explicitly (not assert, which vanishes under python -O), and is pinned by tests for a missing lock, a non-owner task and a wrong lock.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Refusal side channel simplified

Disposition: fixed in 74bb624.
The closure-plus-dict is replaced by a small _StaleWakeStopGuard object with a typed refusal attribute and raise_refusal(). The call on a None row went away with the structured plumbing. It still records the reason on itself because the service precondition contract is Callable[[NudgeLoop], bool], shared with clear_terminal_monitor.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Lock guards pinned

Disposition: fixed in 74bb624.
New tests: a task that does not own the claimed lock is refused with "mutation lock must be held by the caller", and a locked lock that is not the service maintenance lock is refused with "mutation lock must be the service maintenance lock"; the row survives in both. Each test was confirmed to fail with its guard temporarily removed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/stale-wake-stop-race branch from 74bb624 to 0eade8e Compare September 28, 2026 00:12
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed 0eade8e. The PR is still two commits; the second commit was amended. This is the final round agreed for this PR.

  • Round-2 Opus blocker fixed: a stop whose row is already gone now reports success and is audited as allowed. Only a refusal recorded by the lock-held guard refuses.
  • Windows backend shard 7 (test_session_keepalive): root cause found. The test replaced time.monotonic globally, and the path-resolver pool's background reaper thread (subprocess_pool/executor.py) also calls time.monotonic, so it consumed the fake values. That happened in shards that included this PR's new race tests. The test now scopes its fake clock to kiro_crew.mcp_core.time. It is test-only, the assertion is unchanged, and it passes in one process after the race tests.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • A stop whose row is already gone stays an idempotent success (session_directive_apply.py:1194 and :1182)

Disposition: fixed in 0eade8e.
_stop_resolved_loop raises the stale-wake refusal only when the lock-held guard recorded a refusal. When the remove or research update applied nothing and no refusal was recorded, the row was already gone (a user delete or a maintenance quiesce landed first), so the stop returns its existing success wording and is audited as allowed.
New tests cover a delete landing between the pre-check and the stop's lock, on both the remove path and the research-update path. Both failed on 74bb624 (the stop was refused with "Monitor NOT changed"). The real-refusal tests (a person's pause) still refuse.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Lock-ownership registry on the unserialized removal

Disposition: needs-a-decision.
Every in-tree caller already holds the lock today, as this item says. The registry keeps the fix from regressing: the race in #14533 was a check-then-remove, and the ownership check makes any future caller of the unserialized removal fail loudly unless it decides under the lock. It is declared in the description, raises explicitly, and is pinned by tests for a missing lock, a non-owner task and a wrong lock.
Maintainers: keep it as an enforced invariant, or drop it and rely on the precondition alone? I will follow your ruling.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased onto main 6ec57f60f0 as b53343c8b4. The rebase applied cleanly and changes no behaviour; 1714 targeted tests pass.

note: the two red shards on 7c059d7a8c were unrelated to this PR.

  • Windows shard 6 failed the wall-clock check in test_fence_marker_separators.py::test_forwarded_fence_long_separator_run_is_linear again, at 0.543 s against a 0.5 s limit.
  • Backend 3.12 shard 7 failed test_channel_connect_row.py::test_a_slack_relink_cannot_land_between_the_compare_and_the_clear. That test passes 38 of 38 locally and covers Slack channel linking, which this PR does not touch.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/stale-wake-stop-race branch from b53343c to 338a45d Compare September 28, 2026 17:00
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note: Rebased onto main b5dcaaa6cd as 338a45d24f with no behaviour change. 1,338 targeted tests pass locally.

The two Windows reds on b53343c8b4 are in tests this PR doesn't touch, and both pass locally on the rebased tree:

  • shard 1: test_history_index_worker.py::test_a_contended_write_waits_instead_of_failing ("the write did not actually contend"), a timing-dependent contention test.
  • shard 6: test_fence_marker_separators.py::test_forwarded_fence_long_separator_run_is_linear, 0.547 s against its 0.5 s wall-clock limit. This is the third time it has gone over on a Windows runner for this PR, so the limit looks too tight for those runners.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/stale-wake-stop-race branch from 338a45d to 8f6dd25 Compare September 28, 2026 18:50
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note: Rebased onto main e88e807786 as 8f6dd25e87, with no behaviour change. 1,154 targeted tests pass locally.

The two shard-4 reds on 338a45d24f (3.12 and Windows) were test_pid_reader_ratchet.py::test_pid_reader_sites_match_the_recorded_count_exactly. It fails on this PR's old base b5dcaaa6cd without any of this PR's changes, and #14778 on main corrected the recorded count. It passes on the rebased tree. Coverage Gate failed only because those shards did.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
The retention and identity check that protects a person-paused loop now runs as a precondition inside the same locked section as the removal, so a pause that lands while the stop waits for the lock is kept instead of deleted. The unserialized removal path raises unless its caller holds the service mutation lock.
On a research-owned slot the stale-wake stop's update takes the same lock-held precondition as removal, so a concurrent pause is kept. A stop whose row is already gone stays an idempotent success; only a recorded guard refusal refuses. The structured-monitor plumbing is reverted (that path was already safe), the refusal side channel becomes a small guard class, tests pin the lock guards, and the keepalive test scopes its fake clock to mcp_core so a background thread cannot consume it.
@Pearcekieser
Pearcekieser force-pushed the fix/stale-wake-stop-race branch from 8f6dd25 to c225556 Compare September 28, 2026 21:45
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased onto main 15184342e0 as c225556226. No behaviour change; 1,154 targeted tests pass.

note: The only test failure on 8f6dd25e87 was Windows backend shard 7, test_spawn_approval_channel_governance_13491.py::TestTheGovernanceCeilingGatesChannelDelivery::test_a_permitted_channel_still_prompts_and_resolves_the_press (assert False is True). This PR doesn't touch spawn approval or channel governance, and the file passes 5 of 5 locally, so this push re-runs it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 28, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 29, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on behalf of the maintainers (readiness sweep 2026-09-29): readiness passed, the required PR Readiness check is green on this head, no live CHANGES_REQUESTED, no open blocking review threads; where the PR's files also moved on main since its merge-base, the PR's own tests were run on the main+PR merge tree and passed. Approving and squash-merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A monitor's own wake can delete a loop a person paused (stop races a pause)

2 participants