Skip to content

fix(cron): read a reaped run's failure_recorded off the run, not the store - #23097

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/reaper-reads-run-failure-recorded
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/reaper-reads-run-failure-recorded

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A cron run that already counted and alerted its own failure is not counted or alerted a second time when the reaper later kills it after a store reload.

_force_reap decides whether a reaped run counts toward auto-pause, and whether the user is alerted, with not job.failure_recorded. job is the registry's copy. failure_recorded is runtime-only, and a store reload during the run (another process writing crons.json) replaces that object with one where the flag is False. So a run whose handler already called record_failure() and sent "Run failed", and which then hung past the reaper deadline after a reload, is counted a second time (count_failure=True) and alerted a second time (_alert_timeout). The dedup does not catch the second alert, because its "Reaped after …" text differs. The Opus review on #22094 named this.

Why it matters

The user gets two "Run failed" notifications (bell and Slack/channel) for one failed run, and the job reaches its auto-pause limit one run early. A job with the default limit of 5 can pause after four real failures.

Not a goal

  • The other runtime-only field the reaper reads off job, result_produced. A reaped run's result is never merged, so which object it is read from has no visible effect established here.
  • cancel()'s terminal merge, which fix(cron): fence scheduled terminal settlement #9038 is rewriting.
  • How the reaper or the wait_for arm decides a run is stuck.

What changed (motivation → approach → change)

  • Symptom: after a store reload during a run, the reaper alerts and counts a failure the run already alerted and counted.
  • Cause: counted in _force_reap reads failure_recorded off the registry's job instead of the run's own object.
  • Change: read it off ran (_RunClaim.job, falling back to job when the claim has none), as the next line already does for run_never_started. The comment above counted now says both markers are runtime-only and read off ran.
  • The learn-cron-dashboard.md bullet that explains why the reaper reads run_never_started off _RunClaim.job now names failure_recorded too.

Backwards compatibility

Compatible: without a reload, ran and job are the same object and nothing changes. After one, the reaper now reads the flag the run actually set.

Removes nothing: a duplicate count and a duplicate alert stop happening; no path, field, key or validator is removed.

Tests

  • test_a_reload_during_the_run_does_not_count_or_alert_a_recorded_failure_again (new, test/test_cron_timeout_alert.py) gives _force_reap a claim whose job has failure_recorded=True, with a separate reloaded copy of the same job in svc._jobs. It asserts the reaped run is still recorded (Reaped after …) and that the reloaded copy's consecutive_failures stays 0 and no timeout alert is sent.
  • Red before: with origin/main's run_teardown.py the new test fails: assert 1 == 0 on consecutive_failures.
  • Local runs: test_cron_timeout_alert.py, every test_cron_reaper*.py, test_cron_oneshot_end.py and test_cron_auto_pause*.py (245 passed); test_flake_pattern_ratchet.py on the committed tree (60 passed). black, isort, flake8, mypy on the changed module, check_black_formatting.py, check_comment_history.py and check_brand_name.py are clean.

Manual verification

N/A — unit coverage sufficient: the test drives the real _force_reap and merge with the two-object state a reload leaves, which is the whole condition.

Related Issues

Closes #23096

Follows the Opus review finding on #22094.

Pattern harvest

Rule candidate: review-prompt
Pattern: a runtime-only CronJob field read off the registry's job (self._jobs) after an await, instead of off the run's own object (_RunClaim.job); a store reload resets it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

…store

_force_reap decides whether a reaped run counts toward auto-pause, and
whether the user is told, from failure_recorded. It read the flag off
the registry's job, but the flag is runtime-only and a store reload
during the run replaces that object with one where it is False. A run
that already recorded and alerted its own failure, then hung past the
deadline, was counted a second time and sent a second "Run failed"
alert.

Read it off the run's own object (_RunClaim.job), as the next line
already does for run_never_started.

Closes kirodotdev#23096

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 22, 2026 13:29
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 437b3b8f2d1de1ba5cf0dd82916077bd758d4a26 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 437b3b8f2d1de1ba5cf0dd82916077bd758d4a26 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a one-line root-cause fix that reads the run-scoped flag off the run's own object, the same way the next line already reads run_never_started. The test reproduces the two-object state that a store reload leaves.

[DESIGN-REVIEWED] 437b3b8

Description read

[DESCRIPTION-READ] fbd15b1f33c419cc07219ef1463db5994e8e7077ccbdde62aee64dc888bfc33e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 437b3b8f2d1de1ba5cf0dd82916077bd758d4a26 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (3 items) — 3 justified

Intent: when a cron run has already counted and alerted its own failure, the reaper does not count or alert it again after a store reload. This is a FIX.

  1. A reaped run that already recorded its failure is no longer counted twice toward auto-pause, and no second "Run failed" alert is sent, after a store reload — justified
  2. The learn-cron-dashboard.md reaper bullet now says failure_recorded is also read off _RunClaim.job — justified
  3. A new regression test that fails on base (consecutive_failures reads 1 where 0 is expected) — justified

[FIRST-PRINCIPLES-REVIEWED] 437b3b8

Description read

[DESCRIPTION-READ] 4a44ee826fe96009a4f8b332eab47c2f1f2df091786024ce85d280860d192678

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 437b3b8f2d1de1ba5cf0dd82916077bd758d4a26 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 437b3b8

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 437b3b8f2d1de1ba5cf0dd82916077bd758d4a26 via the fork AI-review pipeline; updated in place on each push.

1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/cron_service/service/run_teardown.py:303 -- Suppressing the count also loses an unpersisted failure

and not ran.failure_recorded
Trigger: A recurring command cron with four persisted failures calls record_failure() for its fifth failure, then remains in awaited failure-alert delivery. An external store update is reloaded before _reaper_loop() calls _force_reap(), replacing the registry job with its four-failure disk copy.
Outcome: count_failure=False makes _merge_terminal_state_locked() persist four failures without the run’s auto-pause. _run_job_isolated() skips its result merge because the claim is reaped, permanently losing the fifth failure and leaving the job enabled.
Anchor: residual/crash-data-loss-corruption
Fix: Separate alert suppression from durable accounting; reconcile the run’s unpersisted failure and auto-pause in the locked terminal merge, preserving the existing generation and resume fences.

[BLOCK-MERGE] 437b3b8
[GPT-REVIEWED] 437b3b8

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1: the bug is real and the harm is small, but a store reload while a run is stuck is a plausible combination, not an extreme one. I could not show from the code that it is rare enough to FLAG.

  • Conditions, all confirmed in code:

    • The run's own record_failure() bumps the count and sets the auto-pause only on the object the run executes on (cron_service/model.py:532-539).
    • The reaped run's finalizer skips its result merge (cron_service/service/run_execution.py:205,219).
    • With count_failure=False, the terminal merge never calls record_failure() on the disk copy (run_merge.py:438). The patch's change to run_teardown.py:299-303 makes that happen whenever the run already recorded its failure, so the reloaded four-failure copy is what gets saved.
  • Harm and recovery: the failure counter is off by one and the auto-pause comes one run late. The next failed run counts normally and pauses the job. That is closer to LOW than to data corruption.

  • Why not FLAG: I did not confirm how often the reaper's deadline is reached without the wait_for at run_execution.py:376 ending the run first, so the rarity argument is incomplete.

    [ADJUDICATION] 437b3b8 total=0 uphold=0 downgrade=0
    [GPT-ADJUDICATED] 437b3b8

    [ADJUDICATION-FENCED] 437b3b8 fenced=1 flagged=0
    UPHOLD-FENCED F1 src/kiro_crew/cron_service/service/run_teardown.py:303 -- When a run that already recorded its failure is reaped after a store reload, the skipped result merge plus count_failure=False persists the reloaded copy without that failure or its auto-pause, and a reload during a stuck run is plausible enough that the rarity case for FLAG is not shown.
    [GPT-ADJUDICATED-FENCED] 437b3b8

@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Closing this one. The GPT 6.1 finding on 437b3b8f2d is correct, and it shows the premise I took from the #22094 review does not hold.

CronJob.record_failure() only increments in memory. A run's count reaches the store through its own result merge, which _run_job_isolated skips for a reaped run. In the command path, record_failure() runs before await self._alert_cron_failure(...) (slack/gateway_runtime/cron_exec.py). So a run the reaper kills while it is stuck in alert delivery has failure_recorded=True, but its failure is not persisted and its alert may never have gone out.

After a store reload, the registry's copy is the disk copy, without that failure. Main's not job.failure_recorded (False there) makes the reaper count the failure once and alert once, which is right. Reading the flag off ran, as this PR did, would drop the only count and the only alert for that run, which is the loss GPT describes. Without a reload, job is ran, and both versions behave the same.

There is no double count to fix here. Whether the reaper could ever send a duplicate alert depends on whether the handler's alert was actually delivered, and the reaper cannot see that, so that is not a one-line change either. Closing #23096 with the same explanation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cron reaper alerts and counts a failure twice when the store reloaded during the run

1 participant