Skip to content

fix(cron): fence scheduled terminal settlement - #9038

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
rubencu:fix/cron-stale-runtime
Oct 29, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
rubencu:fix/cron-stale-runtime

Conversation

@rubencu

@rubencu rubencu commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Keep a scheduled cron occurrence from being admitted again after its cancel or reap terminal save fails.

A scheduled cancel or reap releases its run claim before the terminal store write returns. A failed write can leave the stored occurrence due again.

The fence is process memory: an abrupt restart before a retry lands reads the stale store and can admit the occurrence again, so this narrows the window rather than closing it.

Why it matters

Duplicate unattended work can repeat side effects after a cancellation or timeout even though the prior run already started.

Not a goal

  • Rebuild the stale branch's owed-fire scheduler or delivery outbox.
  • Change script, sandbox, gateway, history, or scheduler structure beyond current main's terminal settlement seam.
  • Change manual-run terminal persistence, which has no scheduled occurrence to fence.

What changed (motivation → approach → change)

A scheduled cancel or reap freezes one terminal record and holds it until a save lands it. Keeping the claim until the write lands was rejected: under a failing store a manual Run would answer 409 and cancel, reap and shutdown would loop; the held record blocks only scheduled admission.

This reverses a documented cancel rule: an unlanded park of a cancelled one-shot was never retried, so it could not overwrite a later choice, though a resume made in the gap was overridden. Now a scheduled run's park is retried, and user_pause_generation carries the old rule's reason: a pause or resume made after the freeze keeps its whole pause state. The pin "a later write parked the job" now expects the park on disk.

  • cron_service/settlement.py owns the record. freeze_terminal_record writes the cancel's park on it and reads the runtime-only denied and never-started flags off the claim's job. For a dispatched delete_after_run one-shot it applies auto_pause_failed_one_shot to the claim's job, as the finally does, and copies the pause, audited once. It counts a reap's failure; the stored limit decides the auto-pause, even after a schedule edit, audited once by the landing save. apply_settlement_record re-applies the hold after every reload.
  • The record goes to the held-record owner as a settlement before the first write, so a tick that wins the lock and reloads still finds the job held. The immediate write releases it; after a failed write or busy store, each tick makes one attempt until a save returns. Meanwhile the job stays off schedule with the persisted user_paused. Only the held save writes the park, and the listed job then takes it. A run claimed after the record owns the job's pause and count, so no retry or merge writes the record's over them.
  • Ordinary held result records keep main's rule: the live job's pause state wins.
  • enable_job increments user_pause_generation for a request that changes the pause state, and for any one-shot resume, since another process can hold an unsaved park for it. Other requests record nothing and keep the store bytes.
  • A scheduled cancel reports a later pause or resume its save kept, a busy store met after a resume, or a pause saved before it: a resume raises CronPauseNotSaved, a pause returns success.
  • A manual cancel or reap freezes no record. Its one attempt writes main's fields onto the synced copy (status, park, a reap's failure count), and a failed delete_after_run one-shot's auto-pause comes from the run's own teardown.
  • A failed terminal save no longer restores a provisional hold on a job made recurring.

Code: cron_service/settlement.py, execution.py (shared auto-pause), and service/run_merge.py, run_teardown.py and run_execution.py. cron.py keeps _enable_job_locked, the settlement map, the cancel wording and the new binding. cron_service/model.py splits record_failure's count from its limit check, and with store.py persists the generation, omitted at zero. learn-cron-dashboard.md records the reversal and the mixed-version limit.

Backwards compatibility

Compatible: user_pause_generation is additive, defaults to zero, and is written only after a pause transition or a one-shot resume. A store with neither keeps its bytes. Public cron APIs are unchanged. A manual cancel or reap writes the fields main writes. A one-shot resume another process saves during a manual run now moves the counter, so that store write reloads the job and the run's in-memory auto-pause does not land. The cancel rule above is an intentional behavior change for scheduled runs only.

Mixed versions: a build without the counter never moves it. Its pause is kept, since the park is a pause too. A resume it makes while a record is held does not count as newer, so the park lands over it (tested, and in the spec).

Removes nothing: no supported path is removed.

Tests

Every test named here passes on this head.

  • test_cancel_save_failure_cannot_readmit_occurrence_after_reload fails on main (a second scheduled claim is admitted after a reload).
  • test_a_tick_that_wins_the_lock_before_the_first_merge_keeps_the_park, test_only_a_scheduled_settlement_yields_to_a_resume_after_the_freeze and test_a_newer_held_run_is_not_reported_as_the_cancel_s_settlement failed on an earlier head. In test_cron_cancel.py, reloaded_then_busy now reports the job paused (the tick in the gap lands the park) and resumed_here_then_busy the resume, not a newer run (fails on 3ebebe9).
  • test_a_manual_cancel_or_reap_freezes_no_terminal_record fails on 058a0a8 (a manual cancel or reap froze a record). test_a_resume_saved_during_a_manual_run_survives_its_terminal_write and test_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shot now also pin that the manual merge gets no record; they failed on 0e7cc94 (a stale auto-pause over a resume saved during the run). test_a_manual_failure_at_the_limit_audits_one_auto_pause fails on 0e7cc94 (two cron_auto_pause rows).
  • test_a_scheduled_reap_keeps_a_limit_saved_during_the_run fails on 28024bb (a reap auto-paused at the old limit after another process saved 0 or 10). test_a_scheduled_reap_at_the_stored_limit_audits_one_auto_pause pins one cron_auto_pause row, from the save that lands the pause.
  • Fail on 50d126f: paused_before_cancel_then_busy (false CronPauseNotSaved), test_a_held_reap_at_the_stored_limit_pauses_a_job_whose_schedule_changed (job left enabled), test_a_reloaded_copy_does_not_audit_a_failed_one_shot_s_pause_twice (two rows).
  • test_a_held_reap_retried_during_a_newer_run_keeps_that_run_s_count and test_a_terminal_merge_keeps_the_count_of_a_run_claimed_before_it fail on 0fcf044 (a newer run's count reset: 2 → 1, 0 → 1).
  • Other regressions cover equal-generation reloads, undispatched one-shots, the fire-time gate, list saves after a landed park, newer resumes, older builds and user_pause_generation transitions.
  • Cron, mcp_cron, runtime, integration, metrics and Slack-gateway tests pass, also reversed and shuffled.

Manual verification

N/A: deterministic store-fault and concurrency tests cover the changed paths.

Screenshots / video

N/A: no user-visible frontend change.

Related Issues

No linked issue.

Pattern harvest

Rule candidate: concurrency
Pattern: a deferred terminal write must carry one immutable run record, yield to a pause or resume ordered after it, and release only after durable storage returns.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@rubencu
rubencu requested review from a team and patrigao September 6, 2026 13:58
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 6, 2026
@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 76d0a6168359a7a3be7a7dbc986807abc9026313 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Main's in-memory hold does get lost on a reload, and this PR's held settlement record is applied again after each reload. Nothing is removed, the new persisted field is additive, and the remaining restart window is disclosed.

Notes

  • The PR rewrites the pin "a later write parked the job" (test_cron_cancel.py:1314). That test recorded a decision: nothing would retry the park, so no later write could overwrite a choice made since the cancel. The PR does call this a reversal and keeps the old reason alive through user_pause_generation. Still, it reverses a recorded rule without a maintainer saying so, so get a human ack on the PR.
  • The fix is a lot of machinery to narrow a window rather than close it: settlement.py, newer_run_owns, a generation counter on every pause transition, and duplicated audit logic. If more holes turn up, a durable pending-settlement marker in crons.json would close the restart gap and replace most of the in-memory ordering rules. That belongs in a follow-up PR, not this one.

[DESIGN-REVIEWED] 76d0a61

Description read

[DESCRIPTION-READ] 46efbcbd392d443beef5f0427d1f43123d6859d6a98361209ea2596fa858c5cf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings (all downgraded on adjudication)

Reviewed 76d0a6168359a7a3be7a7dbc986807abc9026313 via the fork AI-review pipeline; updated in place on each push.

Opus 5.5 adjudication downgraded every blocking finding. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

Review details

BLOCKING -- src/kiro_crew/cron.py:1145 -- A failed resume discharges the scheduled cancellation fence

job.user_pause_generation += 1
Trigger: A scheduled one-shot has a held cancellation settlement after a failed terminal save. Dashboard resume reaches _enable_job_locked, but its _save() also fails before committing. With the store unchanged, the next healthy tick retains the unsaved generation and reaches _save_held_run_records_locked → apply_settlement_record.
Outcome: The retry treats the failed resume as a newer committed choice, skips the cancellation park, saves the job enabled, and releases the settlement. The cancelled occurrence runs again although resume returned an error.
Anchor: persist-before-you-publish
Fix: Restore the generation and all modified pause-state fields when the resume write fails before committing, so an uncommitted request cannot supersede the settlement.

[BLOCK-MERGE-DOWNGRADED] 76d0a61
[GPT-REVIEWED] 76d0a61

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — harm rung: LOW. Every condition this needs is confirmed in the code:

  • Conditions. A scheduled settlement is held after its terminal save fails (patch run_merge.py hunk at L742–751, which raises TerminalSettlementPending). The resume's own save then fails (base src/kiro_crew/cron.py:1115), after the PR's generation bump at patch L1145. The store is unchanged, so nothing reloads. Finally apply_settlement_record sees the newer user_pause_generation and keeps the target's pause state (patch settlement.py L1082–1085).

  • Outcome. The base code already leaves a failed resume applied in memory. src/kiro_crew/cron.py:1099–1115 sets user_paused and enabled before _save() and never rolls them back, so the job runs again in base too. The result also matches what the user explicitly asked for, a resume, and the save failure is reported to them. Nothing data-related is lost: a later pause or delete fixes it.

  • Cost of the real fix. It means adding rollback-on-failed-save to _enable_job_locked, which has none on any path in base. That is new state-restore semantics on a mutator shared by every caller, added for the rare case of two write failures in a row whose result is the one the user requested.

    [ADJUDICATION] 76d0a61 total=1 uphold=0 downgrade=1
    DOWNGRADE F1 src/kiro_crew/cron.py:1145 reason=disproportionate-remedy
    [GPT-ADJUDICATED] 76d0a61

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 76d0a6168359a7a3be7a7dbc986807abc9026313 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 76d0a61

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 76d0a6168359a7a3be7a7dbc986807abc9026313 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix reuses the existing held-record owner. It reverses a recorded rule, "Nothing saves it later", and says so openly, backed by a test that fails on main and shows the occurrence admitted a second time.

What this change ships

Inventory (9 items) — 8 justified

Intent: stop a cancelled or timed-out scheduled cron occurrence from running again when its terminal store write fails. This is a FIX.

  1. When a scheduled cancel or reap fails to save, the occurrence stays off the schedule and each timer tick retries the save until one lands — justified
  2. A cancelled one-shot's park that failed to save is now written later instead of never; the old pin "a later write parked the job" is reversed and the description says so — justified
  3. A new user_pause_generation key in crons.json, bumped by pause/resume and one-shot resume (4 consumers in diff: settlement apply, terminal merge, held save, cancel report) — justified
  4. A cancel that meets a resume made in the gap now answers 409 "the job was resumed meanwhile" instead of re-holding the job — justified
  5. The cancel message for a held settlement now says the job may run again only after a restart — justified
  6. A scheduled reap's failure is checked against the stored auto-pause limit when the save lands, so a limit edited during the run wins — justified
  7. The failed one-shot auto-pause becomes one function, applied at freeze and in the finally, and audited once — justified
  8. A failed terminal save no longer puts the hold back on a job that was made recurring — justified
  9. A user cancel now passes the run's auto_paused_at_start into the merge — undeclared

[FIRST-PRINCIPLES-REVIEWED] 76d0a61

Description read

[DESCRIPTION-READ] 969e368223c8a8b1862e53e5dfdf3d29dbed0eaeb51ff2f544bb2a1d116d0ba6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@rubencu

rubencu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the GPT 5.6 blocking finding (F1, pruned-runtime skips consuming scheduled executions) in 9cd287e: all three pruned-launch sites (script cron, single-agent, agent-sequence acquisition) now book through a shared _record_pruned_launch_skip() helper that mirrors the overlap-skip retention pattern — last_status = "error" keeps CronScheduler._execute from recording a success, run_never_started = True is the retention marker that stops _merge_job_result deleting a due one-shot delete_after_run job, and record_failure() is still not called so the update handoff costs no auto-pause strike. Regression tests updated to the never-started contract plus a dedicated test for the one-shot retention shape (test_pruned_install_retains_a_due_one_shot); verified the updated tests fail against the previous commit's behavior. 318 tests pass, lint/type gates clean.

@rubencu
rubencu force-pushed the fix/cron-stale-runtime branch from 9cd287e to 805fed0 Compare September 6, 2026 15:26
@rubencu

rubencu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Round-2 findings addressed in 805fed0 (amended into the review-fix commit to stay within the two-commit gate):

F1 (at-job refire loop): _record_pruned_launch_skip now disables a pruned-skipped at-job IN MEMORY only. _merge_job_result's at-job enabled propagation is gated on not delete_after_run or fire_time_denied, so for the retained delete_after_run shape the flag never reaches disk — the drained gateway stops the zero-delay refire loop while the replacement gateway still sees an enabled job to retry. Regression: test_pruned_at_job_is_quiesced_in_memory + test_pruned_recurring_job_stays_enabled.

F2 (partial sequences): the sequence-loop acquisition now checks _prompt_dispatched; when a prior agent already ran, it records a normal failed run (strike + explanatory last_error, no retention marker) so the one-shot cannot replay completed side effects. Regression: test_mid_sequence_pruning_records_a_failure_not_never_started. The single-agent path acquires before any dispatch and its transient retry is gated on not _prompt_dispatched, so it needs no equivalent guard.

F3 (ENOENT scope): both handlers bind the exception and consult _enoent_names_this_install() — the excuse only applies when the missing path is inside this install's own tree (or exec-level ENOENT with no path); a deleted user script or provider binary stays a real failure. Regression: test_unrelated_missing_path_still_fails_even_when_pruned.

322 targeted tests pass; flake8/isort/mypy clean.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 6, 2026
@rubencu
rubencu force-pushed the fix/cron-stale-runtime branch from 805fed0 to 39723b0 Compare September 6, 2026 16:21
@rubencu

rubencu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Round-3 findings addressed in 39723b0:

F1 (never-started launches still consume their schedule): new CronJob.keep_overdue marker (in-memory only, reset at the start of every run, never merged). When set by _record_pruned_launch_skip, _execute returns before advancing last_run_ts and before the at-job fired/parked disable, and _merge_job_result skips its at-job enabled propagation — so on disk the job remains exactly as due/enabled as before the drained process touched it, and the replacement gateway retries immediately. The drained process itself is still quiesced by the in-memory enabled = False. This deliberately does NOT change run_never_started semantics for the overlap/starvation paths, where the tick was genuinely spent. Scheduler-level regressions: test_execute_keep_overdue_leaves_the_schedule_owed, test_execute_keep_overdue_does_not_park_a_plain_at_job.

F2 (pathless ENOENT): _enoent_names_this_install now rejects pathless ENOENT (a resolve_script_path()-style missing user script stays a real failure). On containment: limiting to Path(sys.prefix) alone would miss the package tree, which lives beside the prefix in versioned installs — so containment is the union of two concrete roots, sys.prefix and this module's own package tree (Path(__file__).resolve().parents[2]), never their common parent (which degrades to / on non-versioned layouts). Regression: test_pathless_enoent_is_a_real_failure_even_when_pruned.

Also brought the touched files to black formatting (the Backend Lint failure). The Backend Tests shard-4 test_snapshot.py failures are timing-sensitive tests untouched by this diff and pass locally (26/26); flagging as suspected base/host flake pending this head's re-run.

325 targeted tests pass; black/flake8/isort/mypy clean.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 6, 2026
@rubencu

rubencu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Round-4 finding addressed in 99431b4: the outer _run_job_isolated drift-correction (last_run_ts = started_at for 'every' jobs) is now guarded with not job.keep_overdue, so a pruned-install skip survives outer finalization and the replacement gateway retries the owed run immediately. Regression: test_run_job_isolated_keeps_a_pruned_every_job_overdue (mutation-verified: un-guarding the assignment fails it). The reaper/timeout stamp sites are untouched — they only apply to runs that genuinely dispatched, which a pruned launch never does. 326 targeted tests pass; black/flake8/isort/mypy clean.

@rubencu
rubencu force-pushed the fix/cron-stale-runtime branch from 39723b0 to 99431b4 Compare September 6, 2026 16:35
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 6, 2026
@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Head 0e7cc94 (one commit on main 4c9d96a) answers the review of 6d86f70 by bolichen97 and the AI lanes. The rebase clears the conflict in docs/system-specs/modules/learn-cron-dashboard.md, keeping the auto-pause limit wording from #15524 beside the text of this PR.

Fixed:

  • Must-fix (also the Opus blocking finding): a manual cancel or reap no longer copies the frozen auto_paused and failure count over the synced store copy. On the manual path the merge counts the failure with record_failure() on the synced target under the lock, and applies only the auto-pause of a failed dispatched delete_after_run one-shot, as main does. test_a_resume_saved_during_a_manual_run_survives_its_terminal_write[False-0|True-1] resumes the job through a second CronService between the freeze and the merge, and asserts the store keeps enabled=True, auto_paused=False. Both cases fail on 6d86f70 ((False, True) == (True, False)) and pass here. The test drives the manual terminal merge directly rather than through cancel().
  • Should-fix (First Principles item 6): TerminalSettlementPending.record and the terminal_record argument at its raise are gone.

Not changed, for a maintainer:

  • The reversal of the cancel park rule still needs an explicit maintainer decision, as the review and the Design lane say. If you would like it recorded, I can add a docs/decisions entry in this PR.
  • Design, user_pause_generation writer enforcement and the alternative of fencing only the occurrence: both follow from that same decision, so they wait with it.
  • A narrow race with feat(cron): per-job auto-pause limit, including never #15524: a scheduled settlement counts its failure on the frozen copy, so a per-job auto-pause limit that another process saves between the freeze and the landing save is not read for that one failure. Re-counting on the store copy would change the frozen-record design, so it is left as is unless you want it changed.

CI on 6d86f70: the shard 7 timeout in test_session_allocation_bindings.py is fixed on main by #21640, which this head carries. 5,256 cron tests (6 host-state collection errors in test_app_script_cron_sibling_import.py) and the static gates passed on 3d6cda4; the PR tests and the 2,000-line cap passed again on 4c9d96a.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ready yet. The fixes from my review of 6d86f70 hold on this head:

  • The manual cancel no longer writes the frozen auto_paused and failure count over a resume another process saved. test_a_resume_saved_during_a_manual_run_survives_its_terminal_write pins it: I reverted the manual branch of _merge_terminal_state_locked to the 6d86f70 form and both parametrized cases fail with (False, True) == (True, False).
  • TerminalSettlementPending.record is gone.
  • The conflict in docs/system-specs/modules/learn-cron-dashboard.md is resolved. GitHub reports the branch mergeable, and a local merge against main is clean.

Head reviewed: 0e7cc94. The four cron test files in this PR pass locally (397 passed).

🔴 2 must-fix. Both are the Opus and GPT lane blocks on this head. I reproduced each one at this head with a small test and checked the same scenario on main, where neither happens.

  • 🔴 A manual run of an auto-paused delete_after_run one-shot, resumed from another process before the merge, ends up auto-paused and disabled again. Main keeps the resume.
  • 🔴 A manual reap that reaches the failure limit writes two auto_paused audit rows, one from the frozen copy and one from the store copy. Main writes one.

Details are inline.

Internal Content Scan reports no verdict (the scan did not finish), and several backend test shards were still running when I read the checks. Neither is caused by the diff as far as I can see.

and terminal_record.last_status == "error"
and not target.auto_paused
):
target.auto_paused = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The manual branch can write a stale auto-pause over a resume another process saved.

This condition cannot tell "the freeze applied the failed one-shot auto-pause" from "the job was already auto-paused when the run started". auto_pause_failed_one_shot returns early for a job that is already auto-paused, so the frozen record keeps the old True. If another process resumes the job between the freeze and this merge, the synced target has auto_paused=False, the guard not target.auto_paused passes, and the merge sets it back.

            if (
                dispatched
                and terminal_record.auto_paused
                and target.delete_after_run
                and terminal_record.last_status == "error"
                and not target.auto_paused
            ):
                target.auto_paused = True

Reproduced at this head: an auto-paused delete_after_run one-shot, _merge_terminal_state_with_retry(scheduled=False, dispatched=True), with a second CronService calling enable_job inside the merge wrapper as your new test does. After the resume the store reads enabled=True, auto_paused=False; after the merge it reads enabled=False, auto_paused=True. On main the same resume survives, because this path never writes auto_paused.

Why it matters: the comment above says the manual branch writes only what this run changed, so the store's resume should win. Here it does not, and the user's resume is undone silently.

Suggestion: apply the auto-pause only when this run made the transition, for example record in the freeze whether auto_pause_failed_one_shot changed the job, or compare user_pause_generation as the scheduled branch does. Add a case to test_a_resume_saved_during_a_manual_run_survives_its_terminal_write with an auto-paused delete_after_run one-shot and dispatched=True.
Rule: the manual branch contract in the comment above; PR "Not a goal" (manual run terminal persistence).

record.enabled = False
record.user_paused = True
if count_failure:
record.record_failure()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 A manual reap at the failure limit writes the auto_paused audit twice.

The freeze counts the failure on copy.copy(job), and record_failure emits _audit_pause_change("auto_paused") when the copy reaches the limit. The new manual branch in _merge_terminal_state_locked then calls target.record_failure() on the store copy, which audits again.

    if count_failure:
        record.record_failure()

Reproduced at this head: a job at 4 consecutive failures (default limit 5), _merge_terminal_state_with_retry(scheduled=False, count_failure=True), with CronJob._audit_pause_change wrapped. Two auto_paused rows come from two different objects. On main, _merge_terminal_state_locked with count_failure=True writes one.

Why it matters: the first row is built from the in-memory count, not the store's. If the store's count is lower, the log records a pause the store never takes, and every manual reap at the limit doubles the security event.

Suggestion: for a manual caller, do not count the failure on the frozen copy (the merge counts it on the synced copy), or count it on the copy without the audit. Emit the audit only from the write that persists the pause. A test that wraps _audit_pause_change and asserts one row would pin it.
Rule: one audit per pause transition (auto_pause_failed_one_shot docstring: "the transition is audited once").

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Head f106f68 (one commit on main aad21da) answers the AI review of 0e7cc94. Both fixes are in cron_service/service/run_merge.py, each with a regression in test/test_cron_terminal_settlement.py that fails on 0e7cc94:

  • GPT (blocking), a manual one-shot teardown overwrote a newer resume: the manual path now applies the one-shot auto-pause only when this run caused it (the run did not start auto-paused) and no newer pause or resume generation is in the store. test_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shot[cancel-this build|reap-this build|reap-older build] drives the real cancel() and _force_reap with a manual claim and resumes through a second CronService between the freeze and the merge.
  • Opus (blocking), a reaped manual run at the failure limit logged two auto-pause SEL rows: a manual record no longer counts the failure on the frozen copy, so the one audit row comes from the write that persists the pause. test_a_manual_failure_at_the_limit_audits_one_auto_pause[reap|direct merge] asserts exactly one cron_auto_pause row, matching the store. The scheduled settlement still counts on its frozen record.

Known limit, unchanged: cancel() passes no starting pause state, so on a manual cancel only the generation check protects a resume, and a resume from an older build between the freeze and the merge is not covered. That is the same mixed-version limit the spec documents for the scheduled path.

The First Principles rider about manual-run persistence (pass): the manual path now counts the failure on the synced target, as main does. The Internal Content Scan on 0e7cc94 was cancelled before it reached a verdict. The reversal of the cancel park rule still waits on the maintainer decision noted above.

5,261 cron tests (8 host-state collection errors) and the static gates passed on 4c9d96a; 4,914 cron and Slack cron-exec tests and the 2,000-line cap passed again on aad21da.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both blockers from my review of 0e7cc94 are fixed on this head, f106f68:

  • A manual cancel or reap of an auto-paused delete_after_run one-shot now keeps a resume saved between the freeze and the merge. I removed the two new guards (not auto_paused_at_start and the user_pause_generation comparison) and all three cases of test_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shot fail with (False, True, False) == (True, False, False).
  • A manual reap at the failure limit now writes one cron_auto_pause row. I put back count_failure=count_failure in both places and both cases of test_a_manual_failure_at_the_limit_audits_one_auto_pause fail with two rows.

The four cron test files in this PR pass locally at this head (402 passed). Design, First Principles, Opus and GPT report no blocking finding on this head. I am dismissing my earlier change request.

One 🟡 inline: I reproduced the finding the GPT lane downgraded, and it differs from main. Not a blocker from me, but worth a look, because main now lets the limit be edited from the form.

Still open before merge, none caused by the code in this diff:

  • The branch conflicts with main in docs/system-specs/modules/learn-cron-dashboard.md, after the auto-pause limit edit landed there. A rebase is needed.
  • Internal Content Scan reports no verdict (the scan did not finish), and several backend shards were still running when I read the checks.
  • Main CI is red on its own (the line cap audit on a test file in main), which blocks approval until main is green.

record.enabled = False
record.user_paused = True
if count_failure:
record.record_failure()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 A scheduled reap applies the failure limit from the in-memory job, not the stored one.

freeze_terminal_record calls record.record_failure() on a copy of the job the run started with, so the auto-pause decision uses that copy's auto_pause_after_failures. The scheduled branch then copies the frozen auto_paused onto the synced target. A limit change saved by another process during the run is kept on disk, but the pause it should have prevented is written anyway. user_pause_generation does not move for a limit-only update, so nothing fences it.

    if count_failure:
        record.record_failure()

Reproduced at this head: a recurring job with 4 failures and the default limit, the stored limit rewritten to 0 by another writer, then a scheduled _force_reap. The store ends with auto_pause_after_failures=0, consecutive_failures=5, enabled=False, auto_paused=True. The same test on main ends with enabled=True, auto_paused=False, because main counts the failure on the synced copy.

Why it matters: the user asked the job never to pause and it stops anyway. The pause is audited and a resume clears it, so it is recoverable, but it is a behavior change from main in a path this PR did not mean to change.

Suggestion: keep the frozen failure increment, but decide the threshold pause on the synced target with its current limit (for example, apply the count on the target in the merge and let record_failure there emit the pause and its audit). A test that rewrites the stored limit before a scheduled reap would pin it.
Rule: a stored setting saved during a run wins over the run's stale copy, as the manual branch comment states.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 058a0a8.

  • The frozen record still counts the failure of this run, so it keeps the occurrence count, the retries and the generation fence. It no longer decides the pause.
  • Wherever the record lands on the job the store hands back (the immediate merge, a held retry, a list save), pause_at_stored_limit decides the threshold with the stored limit of that job.
  • The cron_auto_pause row is written once, after the save that lands the pause.
  • record_failure is split into count_failure and failure_limit_reached. Every other caller is unchanged.

Tests in test/test_cron_terminal_settlement.py:

  • test_a_scheduled_reap_keeps_a_limit_saved_during_the_run[0|10] is your repro. The job has 4 failures, another writer rewrites the stored limit to 0 (or 10) during the run, then a scheduled _force_reap runs. The store ends enabled=True, auto_paused=False, consecutive_failures=5, with no audit row. It fails on 87bfa01 and on 28024bb.
  • test_a_scheduled_reap_at_the_stored_limit_audits_one_auto_pause[...] covers a lowered limit and an unchanged one: one pause and exactly one row. That holds when the first save fails and a later save lands the pause, too.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Head 87bfa01 (one commit on main d7240d0) is a rebase with no code change. It clears the conflict in docs/system-specs/modules/learn-cron-dashboard.md with #22875 (the auto-pause limit on the edit form) and keeps the sentences of both sides; the changes this PR makes to that file are identical, sentence for sentence, to f106f68. 5,276 cron tests and the static gates passed on 6bb453a, and the PR, spec-reading and cron-script tests and the 2,000-line cap passed again on d7240d0.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 28024bb, rebased onto a70e93f (no conflict).

GPT finding on 87bfa01 (upheld): fixed. A manual cancel now passes auto_paused_at_start from the run object to the terminal merge, as the reaper already does, so a resume saved by a build without the pause counter is no longer overwritten by the auto-pause the run started from. New case test_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shot[cancel-older build] fails on 87bfa01 and passes here.

5,268 cron tests passed (8 known host-state collection errors in test_app_script_cron_sibling_import.py); static gates and the 2,000-line cap passed.

The inline on settlement.py:74 (a scheduled reap reads the failure limit from the frozen copy) is not changed in this push.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 28024bb. The only change since the last review in the cron code is the cancel fix: cancel() in run_teardown.py now passes auto_paused_at_start read off the object the run executed on, which is the one keyword the earlier blocking finding asked for.

What I verified on this head:

  • The cron cancel and terminal settlement suites pass (93 passed).
  • With those five lines reverted, test_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shot[cancel-older build] fails, so the fix is pinned by a test.
  • No new finding in the delta. The earlier yellow on the stale failure limit stands as already raised.

The code looks good to ship from my side. Not approving yet because the remaining gates are open: five CI workflows are awaiting maintainer approval and have not run, PR Readiness is red for that reason, and the GPT and Opus lanes have not produced a verdict on this head. A maintainer also needs to confirm this does not overlap the open P0 on cron jobs vanishing from the store after a run, since this PR changes the terminal store writes.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 058a0a8. The change since the last review is the fix for the stale failure limit finding: a scheduled reap's frozen record now only counts the failure (count_failure), and the job the store hands back decides the auto pause with its stored auto_pause_after_failures (pause_at_stored_limit). The single cron_auto_pause row is written by the save that lands the pause (audit_landed_pause), with a marker on the record so a held retry does not write a second one.

What I verified on this head:

  • The cron terminal settlement, cancel, bindings, malformed entry, busy store and reload suites pass (438 passed).
  • With the delta in settlement.py, model.py and run_merge.py reverted, test_a_scheduled_reap_keeps_a_limit_saved_during_the_run fails for both a stored limit of 0 and of 10, and test_a_scheduled_reap_at_the_stored_limit_audits_one_auto_pause fails in three of four cases. Both are pinned by tests.
  • I traced the manual and held save paths: record_failure is now count_failure plus failure_limit_reached, so the manual branch keeps its old behavior and its one audit row.
  • The GPT and Opus lanes report no findings on this head; Design and First Principles pass. The First Principles note on the manual branch scope matches the earlier manual branch findings, which are fixed; I have nothing new to add there.

No new finding in the delta. The earlier yellow is resolved.

Not approving yet: PR Readiness and the backend test shards are still running on this head, the Design lane asks for an explicit maintainer sign off on reversing the documented cancel rule, and a maintainer needs to confirm this does not overlap the open P0 on recurring script cron jobs vanishing from the store after a successful run. This PR touches the same store decode and held record code that the P0 report suspects, though not the successful run result merge or any removal path.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD: waiting on a maintainer decision. Reviewed at 058a0a8.

The code looks ready. All of bolichen97's earlier must-fix items are fixed and pinned by tests that fail on the previous heads. GPT 6.1 and Opus 5.5 report no findings on this head, and CI is green.

Open before approval:

  1. The PR reverses the documented cancel park rule for scheduled runs. Design Review asks for a maintainer's explicit sign-off on the PR, and bolichen97 also asks a maintainer to confirm this does not overlap the open P0 on recurring script cron jobs vanishing from the store. Neither has been given.
  2. First Principles item 6 on this head, with no reply: the manual branch now writes a frozen one-shot auto-pause (run_merge.py#L626-L637), which the Not a goal excludes. Either restrict the frozen record to scheduled runs, or have a human amend the Not a goal.
  3. Design Review notes 1 and 2 on this head: say in the description why keeping the run claim until the write lands was rejected, and that the in-memory fence does not cover an abrupt restart.

The maintainer's ruling plus replies to items 2 and 3 clear this hold.

— crew-pr-reviewer (r120)

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 3ebebe9, rebased onto main bf47e1598b without conflicts. It answers items 2 and 3 of the crew-pr-reviewer hold (comment 6346285077).

  • Item 2 (First Principles item 6): the frozen terminal record is now limited to scheduled runs. freeze_terminal_record runs only for a scheduled cancel or reap, and the manual branch of run_merge.py is main's code again, so a manual run no longer writes a frozen one-shot auto-pause. test_a_manual_cancel_or_reap_freezes_no_terminal_record (recurring and delete_after_run one-shot, cancel and reap) fails on 058a0a8.
  • Manual runs now match main in 15 of 16 probed cases, audit rows included. The 16th is a resume of a not-paused delete_after_run one-shot during its run, then a cancel: main auto-pauses it, and this head leaves it enabled. A one-shot resume bumps user_pause_generation, a scheduled-path protection (another process can hold an unsaved park for that one-shot) that four tests and the spec pin. That store write exposes a defect main already has: the manual path audits an auto-pause that the next _sync() reload loses. Main shows it when a failed delete_after_run one-shot is cancelled or reaped while another writer saves. That is main's own issue, separate from this PR, which does not fix it; the description says so.
  • Item 3 (Design notes 1 and 2): the description now says why keeping the run claim until the write lands was rejected (a failing store would answer manual Runs with 409 and loop cancel, reap and shutdown), and that the in-memory fence narrows, but does not close, the window for an abrupt restart.
  • Item 1 is still a maintainer call: the explicit sign-off on reversing the documented cancel park rule for scheduled runs, and the confirmation bolichen97 asked for that this does not overlap P0 Recurring --script cron jobs vanish from the store after a successful run (agent crons persist) #22485. Nothing in this push changes either.

The Goal, Why it matters and Not a goal sections are unchanged. On bf47e1598b, the PR's tests and every test that imports the cron modules pass (12,300; the only failures also fail on main on this host), as do the composition contracts and the static gates.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Correction to the note above: the 12,300 cron tests ran on 4efafcf60f, the base before the final re-sync, not on bf47e1598b.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 3ebebe9.

The change since 058a0a8 limits the frozen terminal record to scheduled runs: a manual cancel or reap no longer calls freeze_terminal_record, and the manual branch of _merge_terminal_state_locked is main's code again. That resolves the earlier scope note about the manual branch writing a frozen one-shot auto-pause.

What I verified on this head:

  • The terminal settlement, cancel, bindings, malformed entry, busy store and reload suites pass (442 passed).
  • The new test file run against the 058a0a8 sources fails 10 cases, including every test_a_manual_cancel_or_reap_freezes_no_terminal_record case, so the delta is pinned.
  • On the design question: I am fine with reversing the documented cancel park rule for scheduled runs. The reason the old rule existed (a late park must not overwrite a later choice) is carried by user_pause_generation, the new key is additive and omitted at zero, and the restart window is stated in the description.
  • On the open P0 about recurring script jobs vanishing after a successful run: this PR does not touch the successful run result merge, record decoding beyond the one new key, field caps, or any removal path, and the held record reapply it extends only acts on held records. I do not see an overlap.

One finding, inline below: the Opus lane block on this head reproduces as a wrong reason in the 409 text when a user resumes a scheduled one-shot while it is being cancelled and the store is busy. The job state is right; the message is not. I rate it non-blocking on its own, but the lane block holds readiness, and the fix is small.

Not approving yet: the Opus lane blocks this head, PR Readiness is still checking, and the backend test shards are still running.

and self._held_settlements.get(job_id) is held
and held.run_generation == generation
and live is not None
and live.user_pause_generation <= held.user_pause_generation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A resume during a busy cancel is reported as "a newer run of the job was recorded meanwhile".

When a scheduled at one-shot is cancelled and the user resumes it while the merge is awaited, enable_job moves user_pause_generation past the held record's. If the store lock is then busy, the merge never runs, so _merge_terminal_state_with_retry returns the busy reason instead of _PARK_NEWER_CHOICE. Here settlement_pending is False because the live generation is newer, and _cancelled_park_report sees an enabled at job, so it returns _ON_SCHEDULE_MESSAGE. The CronPauseNotSaved text then tells the user a newer run was recorded, which did not happen; the user's resume put the job back on its schedule, and _RESUMED_MESSAGE was added for exactly that case.

                    and live.user_pause_generation <= held.user_pause_generation
                )
            ...
            note, message = seams._cancelled_park_report(
                live, park_failure, saved, held=settlement_pending
            )

Suggestion: when the held record is this cancel's settlement (held.run_generation == generation) and live.user_pause_generation > held.user_pause_generation, pass _PARK_NEWER_CHOICE as the reason so the report says the job was resumed (or paused) meanwhile. Update the resumed_here_then_busy expectation in test/test_cron_cancel.py, which currently pins the wrong text.

Rule: a user-facing error reason must describe what actually happened.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 50d126f, rebased onto main 6f4cdf1f8b without conflicts. It fixes the busy-resume reason from bolichen97's inline comment and the Opus lane block (run_teardown.py:578).

  • When the held record is this cancel's own settlement and the user resumed the job meanwhile (the live user_pause_generation is newer and the job is enabled), a busy store no longer reports "a newer run of the job was recorded meanwhile". The park failure is reported with _PARK_NEWER_CHOICE, so the 409 carries _RESUMED_MESSAGE and the history row says the job was resumed and is back on its schedule.
  • The cancel reports a resume, but not a pause, this way. Under a busy store a pause may not have been saved, and reporting "paused meanwhile" would claim a pause that never landed, the same false-report class as this finding. A pause still goes through the saved-store check: a saved pause returns success and says the job is paused, and an unsaved one still says the one-shot may run again. If you want the pause path covered too, it can report "paused meanwhile" only when the pause write actually lands.
  • resumed_here_then_busy in test_cron_cancel.py now expects the resume note and fails on 3ebebe9. paused_here_then_busy and pause_unsaved_then_busy pin both pause outcomes. The manual path is unchanged.
  • On bf47e1598b, the PR's tests and every test that imports the cron modules pass (12,302; the only failures also fail on main on this host). After the clean re-sync to 6f4cdf1f8b, the PR's tests, the tests main changed since, the composition contracts and the static gates pass again.

Thanks for the sign-off on the scheduled-run cancel rule and the P0 #22485 check.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 50d126f (range-diff against the previous head: one commit, rebased, plus the resume fix and three new busy cases).

The point from my last review is fixed: a one-shot resumed after the freeze, then cancelled into a busy store, now reports the resume instead of a newer run, and resumed_here_then_busy pins the right text. Thanks.

Two items remain before this can go in. Both lanes that block on this head are about them, so readiness will stay red until they are addressed.

  1. run_teardown.py line 580: a pause that was already saved before the cancel still produces a false 409. I reproduced it locally: start a scheduled plain one-shot, call enable_job(id, False) while it runs, then cancel with the store lock busy. The cancel raises CronPauseNotSaved with "may run again only if the gateway restarts first", and history says "may run again after a restart", while the store on disk already holds enabled False and user_paused True. Cause: the frozen record copies the paused job, so the held record's user_pause_generation equals the live one, settlement_pending is True, and the saved-store read is skipped. Suggestion: when live.user_paused is already True (or the record's user_pause_generation was bumped by a saved pause), still read _saved_job_state before choosing the held wording, and add a case like paused_before_cancel_then_busy next to paused_here_then_busy.

  2. settlement.py line 148: in apply_settlement_record, the branch for a changed schedule returns before pause_at_stored_limit runs. A held reap that counted the failure reaching the limit, landing after another process edited the schedule, saves the threshold count with auto_paused False, so the job gets one more unattended run. The fix the GPT lane suggests (evaluate the stored limit in that branch and re-derive enabled) is small and keeps the newer-choice fence.

Design: I am fine with the approach. Keeping the terminal record in memory and retrying the write on each tick, for scheduled runs only, reuses the existing held-record owner and adds no new long-term debt. The in-memory limit across restarts and the two copies of the limit rules are reasonable follow-ups.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: changed-schedule settlement re-checks the counted failure against the stored limit

  • span=4e1bba793510 settlement.py:148: Schedule edits bypass the held failure's auto-pause limit

fixed in 0fcf044: a held reap that lands after another process changed the schedule now applies the record's failure count and calls pause_at_stored_limit, behind the same paused-at-start guard as the main path. The newer-user-choice check still runs first, and only the old schedule's park is dropped. test_a_held_reap_at_the_stored_limit_pauses_a_job_whose_schedule_changed fails on 50d126f.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the one-shot auto-pause is applied once, on the run's own job object, and copied onto the frozen record

  • span=4e1bba793510 settlement.py:63: Reloaded jobs emit duplicate auto-pause audit records

fixed in 0fcf044: the freeze now applies auto_pause_failed_one_shot to the claim's job, the same object the run's finally pauses, so the finally finds it already paused and writes no second cron_auto_pause row. When a reload replaced the listed job, the frozen record carries the pause and the settlement rules decide whether it lands. test_a_reloaded_copy_does_not_audit_a_failed_one_shot_s_pause_twice fails on 50d126f.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a paused live job reads the saved store even while its own settlement is pending

  • span=a440cd777a58 run_teardown.py:600: a pause saved before the cancel is reported as unsaved

fixed in 0fcf044: the cancel now reads the saved store whenever the live job is user- or auto-paused, even while its own settlement is pending, and a saved pause returns success as on main. A pause that was never saved still gets the held message. paused_before_cancel_then_busy in test_cron_cancel.py fails on 50d126f.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 0fcf044, rebased onto main 3cdcfe893c without conflicts. It fixes the three defects from bolichen97's review (items 1 and 2) and the GPT and Opus lanes on 50d126f.

  • Item 1, run_teardown.py (also the Opus lane). A pause saved before the cancel no longer gets a false 409. The cancel now reads the saved store whenever the live job is paused, even while its own settlement is pending, so a saved pause returns success as on main. A pause that was never saved still says the one-shot may run again. paused_before_cancel_then_busy in test_cron_cancel.py fails on 50d126f.
  • Item 2, settlement.py:148 (also GPT). A held reap that lands after another process changed the schedule now checks its counted failure against the stored limit and pauses the job when the limit is reached. The newer-user-choice check still runs first, and only the old schedule's park is dropped. test_a_held_reap_at_the_stored_limit_pauses_a_job_whose_schedule_changed fails on 50d126f.
  • GPT, settlement.py:63. A reloaded copy no longer writes a second cron_auto_pause audit row. The freeze pauses the run's own job object, which the finalizer then finds already paused, and copies that pause onto the frozen record. test_a_reloaded_copy_does_not_audit_a_failed_one_shot_s_pause_twice fails on 50d126f.
  • The manual-run path, the resume-only busy-cancel rule and the description's Goal, Why it matters and Not a goal sections are unchanged.
  • The PR's tests, every test that imports the cron modules and the tests main changed since the last published base 6f4cdf1f8b pass (12,758 on 9fb1d4aa52); the only failures also fail on main on this host. The flake ratchet, the composition contracts, the security line budget and the static gates pass. After the final re-sync onto 3cdcfe893c, main's newly changed test file and the composition contracts pass again.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 0fcf044 (range-diff against 50d126f: one commit, rebased, plus the fixes below). Both items from my previous review are fixed:

  1. run_teardown.py: the cancel now reads the saved store even while its own settlement is pending, so a pause saved before the cancel froze its record answers success instead of a 409. Pinned by the new paused_before_cancel_then_busy case.
  2. settlement.py apply_settlement_record: when the schedule changed, the counted failure still lands and pause_at_stored_limit decides the pause; only the old schedule's park is dropped. Pinned by test_a_held_reap_at_the_stored_limit_pauses_a_job_whose_schedule_changed.

The new paused_here path in freeze_terminal_record pauses the run's own object once and carries the pause on the frozen record, so a reload no longer audits a failed one-shot twice. That also looks right to me.

Verified locally at this head: test/test_cron_terminal_settlement.py and test/test_cron_cancel.py pass (108 tests), with the PR source on the path.

No new findings. I will approve once readiness passes on this head: four backend shards and the Opus lane are still running.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

I will approve once readiness passes on this head: four backend shards and the Opus lane are still running.

Readiness on 0fcf044e now waits only on the Opus 5.5 Review. Its fork run (run 40857855235) completed without posting a review for this head: the step that looks up the PR could not reach api.github.com (connect: connection refused on all three attempts), so the lane failed closed before opening its check-run. I re-ran it once at 12:34Z and it failed the same way on another runner. Could a maintainer re-run that lane?

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up on the Opus 5.5 lane for 0fcf044e. Re-running the old run (run 40857855235) does not work: it failed its PR lookup on all three attempts, the last one after the runners recovered, so the request in my earlier note can be dropped. Re-running Fast Gate started fresh lane runs on the same head instead, with no new commit, and the Opus 5.5 review now reports no blocking findings (run 40889384148). Readiness is still not green: the fresh GPT 6.1 review raises a new blocking finding at settlement.py:177, where a held settlement retried by a tick can reset a concurrent manual run's failure count.

A scheduled cancel or reap releases its claim before the terminal
store write returns, so a failed write left the occurrence due again.
The frozen terminal record now goes to the held-record owner as a
settlement and is retried once per tick until a save returns.

The settlement freeze and apply rules live in cron_service/settlement.py.
The record carries the cancel's own park, reads the denied and
never-started flags off the object the run executed on, and re-applies
its hold at its own generation after a reload. Ordinary held result
records keep their previous save rule. A failed manual terminal save
no longer restores a recurring job's provisional hold.
@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a run claimed after a held settlement owns the job's pause and failure count, so neither a retry nor the terminal merge copies the record's over them

  • span=4e1bba793510 src/kiro_crew/cron_service/settlement.py:177: Settlement retry erases a newer manual run’s failure

fixed in 76d0a61: a held settlement no longer writes its pause state or failure count onto a job object that a run claimed after the record is executing on (newer_run_owns in settlement.py: that claim runs on the same object and its generation is higher than the record's). The order is by recency, not by size. The newer run builds its count on top of the record's, so its failure stays counted, and a newer success's 0 stays 0. The same check covers the changed-schedule copy, the hold, every reload re-apply, the held save (decided on the live job, not its copy) and _merge_terminal_state_locked. Both regressions fail on 0fcf044: test_a_held_reap_retried_during_a_newer_run_keeps_that_run_s_count (2 becomes 1, 0 becomes 1) and test_a_terminal_merge_keeps_the_count_of_a_run_claimed_before_it.

@rubencu

rubencu commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97, here is what changed since your review of 0fcf044e. The head is now 76d0a616.

  • Fix: GPT found one new case on 0fcf044e: a held scheduled reap's retry reset the failure count of a Run Now in flight on the same job (settlement.py:177). A run claimed after the held record now owns the job's pause state and failure count, so no retry or merge copies the record's over them. The check is newer_run_owns in cron_service/settlement.py, called from four sites in cron_service/service/run_merge.py, about 60 lines in all.
  • New tests in test/test_cron_terminal_settlement.py, both failing on 0fcf044e (about 85 lines): test_a_held_reap_retried_during_a_newer_run_keeps_that_run_s_count and test_a_terminal_merge_keeps_the_count_of_a_run_claimed_before_it. There is also one signature pin in test/test_cron_service_bindings.py and one sentence in the cron spec.
  • Rebase: onto main ae1bb2dd. Main's fix(cron): park a cancelled delete_after_run one-shot #25130 now parks every cancelled one-shot, so one existing test now expects a cancelled delete_after_run one-shot to be parked, and the spec's cancel line takes main's wording.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 76d0a61. Good to ship; I will approve once readiness passes on this head.

What changed since my review of 0fcf044 (range-diff against the rebase onto main ae1bb2d):

  • The GPT 6.1 blocking finding on 0fcf044 (settlement.py:177, a held scheduled reap retried by a tick reset the failure count of a Run Now in flight) is fixed. newer_run_owns checks that the live claim executes on the same job object and has a higher generation than the held record, and all four sites that land a settlement record (_hold_run_record, _apply_held_run_records_to, the held save, and the immediate terminal merge) now skip the record's auto_paused and failure count in that case. The park and enabled rules are unchanged.
  • I ran test_cron_terminal_settlement.py, test_cron_cancel.py, test_cron_service_bindings.py and test_cron_load_malformed_entry.py at this head: 426 passed. With settlement.py and run_merge.py restored to their 0fcf044 versions, the two new regressions fail, so they pin the fix.
  • The other range-diff changes are the rebase: the spec line and one test now expect a cancelled one-shot to be parked, following main.

Non-blocking, no change needed: the GPT 6.1 finding on this head (cron.py:1145, a resume whose save fails still bumps the pause generation) was downgraded on adjudication. I agree: main also keeps a failed resume applied in memory, and the result is what the user asked for.

Lanes on this head: Design, First Principles, Opus 5.5 and GPT 6.1 have no blocking findings; UX skipped. Backend tests, E2E, integration and readiness are still running.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 76d0a61. Gates checked: readiness passed on this head, all checks green, no AI lane BLOCK (GPT 6.1 finding at cron.py:1145 downgraded on adjudication and agreed non-blocking), 13 files, no merge conflict, design sign-off given earlier. The GPT finding on 0fcf044 (held reap retry reset a newer run's count) is fixed by newer_run_owns; the two new regressions fail on 0fcf044 code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants