Skip to content

fix(autonudge): check the stop sentinel off the event loop - #39103

Merged
javenciu merged 1 commit into
mainfrom
fix/sync-io-autonudge-firing
Nov 18, 2026
Merged

javenciu merged 1 commit into
mainfrom
fix/sync-io-autonudge-firing

Conversation

@javenciu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The autonudge timer tick stops checking its stop-sentinel file on the event loop, so a slow or stuck mount under that path cannot freeze every session's turn.

src/kiro_crew/autonudge_service/firing.py is listed in .github/sync-io-in-async-fs-baseline.txt. In _run_timer_callback, every tick of every loop with a stop-sentinel path called Path(loop.stop_sentinel_path).exists() directly inside the async def.

Why it matters

A stat on a mount that stops answering blocks the loop for as long as the kernel takes. Past the loop-stall limit the watchdog restarts the gateway and drops every in-flight turn. The check runs on every tick of every armed loop, so it is hit often.

Not a goal

  • The other baseline files are untouched.
  • No change to what the tick does when the sentinel is or is not there.

What changed (motivation → approach → change)

  • The tick now reads await asyncio.to_thread(Path(loop.stop_sentinel_path).exists), the idiom mutations.py in the same package already uses for the same kind of file. An empty sentinel path still skips the check. The result is held in sentinel_found rather than joined to the path test with and, because mypy reads the and form as a str result.
  • The firing.py line is removed from the baseline with python3 scripts/check_sync_io_in_async.py --update-baseline.

Backwards compatibility

Compatible: same result, same log line, same stop reason, same order of effects; only the thread that runs the stat changes.

Removes nothing: no function, key or behaviour is removed.

Tests

  • test_sentinel_check_runs_off_the_event_loop records the thread that calls Path.exists for the sentinel and asserts it is not the loop thread, then checks the loop still stops with the sentinel reason.
  • RED on origin/main source: the test fails with "sentinel stat ran on the loop thread". The sync-io gate with the baseline line removed also fails: firing.py:200, "1 new offender(s)".
  • GREEN on this head: the test passes, the gate passes with the line removed, and 2,534 tests in the autonudge files, the flake and wait ratchets, the gate-wiring files and the loop-counter tests pass.

Proof: PROVEN test_sentinel_check_runs_off_the_event_loop (head 0c5c8d9)

Related Issues

None.

Sibling sweep

Swept: python3 scripts/check_sync_io_in_async.py --report
Sibling: the other files in the baseline -- not affected -- each has its own row and a separate change

Pattern harvest

Not generalizable: the sync-io gate already catches this class; the fix is the one listed call.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The autonudge tick looked for the stop-sentinel file with a plain
Path.exists() inside an async function. That is a blocking filesystem
call on the loop that serves every session. Run it in a worker thread,
as the neighbouring code in mutations.py already does, and drop the
file from the sync-io baseline.
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 18, 2026 13:36
@javenciu
javenciu requested a review from iamwhatever November 18, 2026 13:36
@javenciu
javenciu enabled auto-merge (squash) November 18, 2026 13:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. This is a one-call mechanism fix: the stat moves off the loop, and the sync-io gate keeps the class from coming back.

What this change ships

Inventory (3 items) — 3 justified

Intent: stop a slow or stuck mount under the stop-sentinel path from freezing every session's turn during an autonudge tick. FIX.

  1. The autonudge tick now checks for the stop-sentinel file on a worker thread instead of the event loop — justified
  2. firing.py comes off the sync-io-in-async baseline, so the gate now catches a new offender there — justified
  3. A new test fails if the sentinel stat runs on the loop thread — justified

[FIRST-PRINCIPLES-REVIEWED] 0c5c8d9

Description read

[DESCRIPTION-READ] e84a2652d1db480685c671f87e570141eab4457937d729417a09eb5a3486c5b4

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Moving the sentinel stat off the loop fixes the cause of the stall; the code after it already awaits, so the new suspension point adds no new interleaving.

[DESIGN-REVIEWED] 0c5c8d9

Description read

[DESCRIPTION-READ] a008580d5796aaa45612837654f193764572e5c4d77208a3202710fcbf0388e9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0c5c8d9

Verdict parsed from the review's SHA-scoped output markers for commit 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0c5c8d9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0c5c8d9bd5b3d037b6ac50dbb626ddc26f8b312e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier pending: being reviewed.

This is a one-call fix, and I'd merge it as it stands. In _run_timer_callback, the stop-sentinel check now runs Path.exists through await asyncio.to_thread(...), the same way mutations.py already does. An empty sentinel path still skips the check. The log line, the stop reason and the early return are all unchanged.

The change adds one new pause point (await) before the sentinel check. I checked what can happen to the loop's state during that pause:

  • A pause or delete that arrives then calls _cancel_timer, which cancels the tick task cleanly. _timer's finally block then clears the running-callback state and the mutation lease.
  • update() already handles a loop row that has been deleted, so a stop landing after a removal is harmless.
  • The push-tick markers are set before the check, so that ordering does not change.

The firing.py line comes off the sync-io baseline, and that check passed in CI under Source Gates. The new test records which thread calls exists for the sentinel. It would fail on main, and it also checks that the loop still stops with the sentinel reason. All CI checks and review lanes passed.

Tier: pending: being reviewed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 0c5c8d9bd5 · 38 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head, the stop sentinel's exists() check in firing.py now runs through asyncio.to_thread. It still sits behind the empty-path guard, and the log line, stop reason and early return are the same as before. A stop or re-arm can cancel the tick during the new await. That leaves the timer in the same state as a cancel at the awaits that already existed. The firing.py row is gone from the sync-io baseline, so the ratchet only got stricter. With main's firing.py restored, the new test fails with "sentinel stat ran on the loop thread". With this head's version it passes.

Done well: the test records which thread actually runs the stat and also asserts the stop reason, so it cannot pass if the check never runs.

Optional:

  • _stop_file_lifted in autonudge_service/mutations.py (line 92) runs the same exists() on the event loop. It runs during resume, while self._lock is held. The gate cannot see it because the call sits inside a sync helper. A follow-up could compute that result with asyncio.to_thread before taking the lock.
  • Open PR #11397 rewrites the same if in firing.py, so whichever of the two lands second will need a small rebase.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 18, 2026
@javenciu
javenciu merged commit a7b8a77 into main Nov 18, 2026
99 checks passed
@javenciu
javenciu deleted the fix/sync-io-autonudge-firing branch November 18, 2026 15:32
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants