Repository navigation
fix(cron): fence scheduled terminal settlement - #9038
Conversation
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Main's in-memory hold does get lost on a reload, and this PR's held settlement record is applied again after each reload. Nothing is removed, the new persisted field is additive, and the remaining restart window is disclosed. Notes
[DESIGN-REVIEWED] 76d0a61 Description read[DESCRIPTION-READ] 46efbcbd392d443beef5f0427d1f43123d6859d6a98361209ea2596fa858c5cf That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findings (all downgraded on adjudication)Reviewed Opus 5.5 adjudication downgraded every blocking finding. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents. Review detailsBLOCKING --
Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1 — harm rung: LOW. Every condition this needs is confirmed in the code:
|
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix reuses the existing held-record owner. It reverses a recorded rule, "Nothing saves it later", and says so openly, backed by a test that fails on main and shows the occurrence admitted a second time. What this change shipsInventory (9 items) — 8 justifiedIntent: stop a cancelled or timed-out scheduled cron occurrence from running again when its terminal store write fails. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 76d0a61 Description read[DESCRIPTION-READ] 969e368223c8a8b1862e53e5dfdf3d29dbed0eaeb51ff2f544bb2a1d116d0ba6 That sha256 names the PR description this verdict read. Recompute it with |
|
Addressed the GPT 5.6 blocking finding (F1, pruned-runtime skips consuming scheduled executions) in 9cd287e: all three pruned-launch sites (script cron, single-agent, agent-sequence acquisition) now book through a shared |
9cd287e to
805fed0
Compare
|
Round-2 findings addressed in 805fed0 (amended into the review-fix commit to stay within the two-commit gate): F1 (at-job refire loop): F2 (partial sequences): the sequence-loop acquisition now checks F3 (ENOENT scope): both handlers bind the exception and consult 322 targeted tests pass; flake8/isort/mypy clean. |
805fed0 to
39723b0
Compare
|
Round-3 findings addressed in 39723b0: F1 (never-started launches still consume their schedule): new F2 (pathless ENOENT): Also brought the touched files to black formatting (the Backend Lint failure). The Backend Tests shard-4 325 targeted tests pass; black/flake8/isort/mypy clean. |
|
Round-4 finding addressed in 99431b4: the outer |
39723b0 to
99431b4
Compare
|
Head 0e7cc94 (one commit on main 4c9d96a) answers the review of 6d86f70 by bolichen97 and the AI lanes. The rebase clears the conflict in Fixed:
Not changed, for a maintainer:
CI on 6d86f70: the shard 7 timeout in |
bolichen97
left a comment
There was a problem hiding this comment.
Not ready yet. The fixes from my review of 6d86f70 hold on this head:
- The manual cancel no longer writes the frozen
auto_pausedand failure count over a resume another process saved.test_a_resume_saved_during_a_manual_run_survives_its_terminal_writepins it: I reverted the manual branch of_merge_terminal_state_lockedto the 6d86f70 form and both parametrized cases fail with(False, True) == (True, False). TerminalSettlementPending.recordis gone.- The conflict in
docs/system-specs/modules/learn-cron-dashboard.mdis resolved. GitHub reports the branch mergeable, and a local merge against main is clean.
Head reviewed: 0e7cc94. The four cron test files in this PR pass locally (397 passed).
🔴 2 must-fix. Both are the Opus and GPT lane blocks on this head. I reproduced each one at this head with a small test and checked the same scenario on main, where neither happens.
- 🔴 A manual run of an auto-paused
delete_after_runone-shot, resumed from another process before the merge, ends up auto-paused and disabled again. Main keeps the resume. - 🔴 A manual reap that reaches the failure limit writes two
auto_pausedaudit rows, one from the frozen copy and one from the store copy. Main writes one.
Details are inline.
Internal Content Scan reports no verdict (the scan did not finish), and several backend test shards were still running when I read the checks. Neither is caused by the diff as far as I can see.
| and terminal_record.last_status == "error" | ||
| and not target.auto_paused | ||
| ): | ||
| target.auto_paused = True |
There was a problem hiding this comment.
🔴 The manual branch can write a stale auto-pause over a resume another process saved.
This condition cannot tell "the freeze applied the failed one-shot auto-pause" from "the job was already auto-paused when the run started". auto_pause_failed_one_shot returns early for a job that is already auto-paused, so the frozen record keeps the old True. If another process resumes the job between the freeze and this merge, the synced target has auto_paused=False, the guard not target.auto_paused passes, and the merge sets it back.
if (
dispatched
and terminal_record.auto_paused
and target.delete_after_run
and terminal_record.last_status == "error"
and not target.auto_paused
):
target.auto_paused = TrueReproduced at this head: an auto-paused delete_after_run one-shot, _merge_terminal_state_with_retry(scheduled=False, dispatched=True), with a second CronService calling enable_job inside the merge wrapper as your new test does. After the resume the store reads enabled=True, auto_paused=False; after the merge it reads enabled=False, auto_paused=True. On main the same resume survives, because this path never writes auto_paused.
Why it matters: the comment above says the manual branch writes only what this run changed, so the store's resume should win. Here it does not, and the user's resume is undone silently.
Suggestion: apply the auto-pause only when this run made the transition, for example record in the freeze whether auto_pause_failed_one_shot changed the job, or compare user_pause_generation as the scheduled branch does. Add a case to test_a_resume_saved_during_a_manual_run_survives_its_terminal_write with an auto-paused delete_after_run one-shot and dispatched=True.
Rule: the manual branch contract in the comment above; PR "Not a goal" (manual run terminal persistence).
| record.enabled = False | ||
| record.user_paused = True | ||
| if count_failure: | ||
| record.record_failure() |
There was a problem hiding this comment.
🔴 A manual reap at the failure limit writes the auto_paused audit twice.
The freeze counts the failure on copy.copy(job), and record_failure emits _audit_pause_change("auto_paused") when the copy reaches the limit. The new manual branch in _merge_terminal_state_locked then calls target.record_failure() on the store copy, which audits again.
if count_failure:
record.record_failure()Reproduced at this head: a job at 4 consecutive failures (default limit 5), _merge_terminal_state_with_retry(scheduled=False, count_failure=True), with CronJob._audit_pause_change wrapped. Two auto_paused rows come from two different objects. On main, _merge_terminal_state_locked with count_failure=True writes one.
Why it matters: the first row is built from the in-memory count, not the store's. If the store's count is lower, the log records a pause the store never takes, and every manual reap at the limit doubles the security event.
Suggestion: for a manual caller, do not count the failure on the frozen copy (the merge counts it on the synced copy), or count it on the copy without the audit. Emit the audit only from the write that persists the pause. A test that wraps _audit_pause_change and asserts one row would pin it.
Rule: one audit per pause transition (auto_pause_failed_one_shot docstring: "the transition is audited once").
|
Head f106f68 (one commit on main aad21da) answers the AI review of 0e7cc94. Both fixes are in
Known limit, unchanged: The First Principles rider about manual-run persistence (pass): the manual path now counts the failure on the synced target, as main does. The Internal Content Scan on 0e7cc94 was cancelled before it reached a verdict. The reversal of the cancel park rule still waits on the maintainer decision noted above. 5,261 cron tests (8 host-state collection errors) and the static gates passed on 4c9d96a; 4,914 cron and Slack cron-exec tests and the 2,000-line cap passed again on aad21da. |
bolichen97
left a comment
There was a problem hiding this comment.
Both blockers from my review of 0e7cc94 are fixed on this head, f106f68:
- A manual cancel or reap of an auto-paused
delete_after_runone-shot now keeps a resume saved between the freeze and the merge. I removed the two new guards (not auto_paused_at_startand theuser_pause_generationcomparison) and all three cases oftest_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shotfail with(False, True, False) == (True, False, False). - A manual reap at the failure limit now writes one
cron_auto_pauserow. I put backcount_failure=count_failurein both places and both cases oftest_a_manual_failure_at_the_limit_audits_one_auto_pausefail with two rows.
The four cron test files in this PR pass locally at this head (402 passed). Design, First Principles, Opus and GPT report no blocking finding on this head. I am dismissing my earlier change request.
One 🟡 inline: I reproduced the finding the GPT lane downgraded, and it differs from main. Not a blocker from me, but worth a look, because main now lets the limit be edited from the form.
Still open before merge, none caused by the code in this diff:
- The branch conflicts with main in
docs/system-specs/modules/learn-cron-dashboard.md, after the auto-pause limit edit landed there. A rebase is needed. Internal Content Scanreports no verdict (the scan did not finish), and several backend shards were still running when I read the checks.- Main CI is red on its own (the line cap audit on a test file in main), which blocks approval until main is green.
| record.enabled = False | ||
| record.user_paused = True | ||
| if count_failure: | ||
| record.record_failure() |
There was a problem hiding this comment.
🟡 A scheduled reap applies the failure limit from the in-memory job, not the stored one.
freeze_terminal_record calls record.record_failure() on a copy of the job the run started with, so the auto-pause decision uses that copy's auto_pause_after_failures. The scheduled branch then copies the frozen auto_paused onto the synced target. A limit change saved by another process during the run is kept on disk, but the pause it should have prevented is written anyway. user_pause_generation does not move for a limit-only update, so nothing fences it.
if count_failure:
record.record_failure()Reproduced at this head: a recurring job with 4 failures and the default limit, the stored limit rewritten to 0 by another writer, then a scheduled _force_reap. The store ends with auto_pause_after_failures=0, consecutive_failures=5, enabled=False, auto_paused=True. The same test on main ends with enabled=True, auto_paused=False, because main counts the failure on the synced copy.
Why it matters: the user asked the job never to pause and it stops anyway. The pause is audited and a resume clears it, so it is recoverable, but it is a behavior change from main in a path this PR did not mean to change.
Suggestion: keep the frozen failure increment, but decide the threshold pause on the synced target with its current limit (for example, apply the count on the target in the merge and let record_failure there emit the pause and its audit). A test that rewrites the stored limit before a scheduled reap would pin it.
Rule: a stored setting saved during a run wins over the run's stale copy, as the manual branch comment states.
There was a problem hiding this comment.
Fixed in 058a0a8.
- The frozen record still counts the failure of this run, so it keeps the occurrence count, the retries and the generation fence. It no longer decides the pause.
- Wherever the record lands on the job the store hands back (the immediate merge, a held retry, a list save),
pause_at_stored_limitdecides the threshold with the stored limit of that job. - The
cron_auto_pauserow is written once, after the save that lands the pause. record_failureis split intocount_failureandfailure_limit_reached. Every other caller is unchanged.
Tests in test/test_cron_terminal_settlement.py:
test_a_scheduled_reap_keeps_a_limit_saved_during_the_run[0|10]is your repro. The job has 4 failures, another writer rewrites the stored limit to 0 (or 10) during the run, then a scheduled_force_reapruns. The store endsenabled=True, auto_paused=False, consecutive_failures=5, with no audit row. It fails on 87bfa01 and on 28024bb.test_a_scheduled_reap_at_the_stored_limit_audits_one_auto_pause[...]covers a lowered limit and an unchanged one: one pause and exactly one row. That holds when the first save fails and a later save lands the pause, too.
|
Head 87bfa01 (one commit on main d7240d0) is a rebase with no code change. It clears the conflict in |
|
Pushed 28024bb, rebased onto a70e93f (no conflict). GPT finding on 87bfa01 (upheld): fixed. A manual cancel now passes 5,268 cron tests passed (8 known host-state collection errors in The inline on |
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 28024bb. The only change since the last review in the cron code is the cancel fix: cancel() in run_teardown.py now passes auto_paused_at_start read off the object the run executed on, which is the one keyword the earlier blocking finding asked for.
What I verified on this head:
- The cron cancel and terminal settlement suites pass (93 passed).
- With those five lines reverted,
test_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shot[cancel-older build]fails, so the fix is pinned by a test. - No new finding in the delta. The earlier yellow on the stale failure limit stands as already raised.
The code looks good to ship from my side. Not approving yet because the remaining gates are open: five CI workflows are awaiting maintainer approval and have not run, PR Readiness is red for that reason, and the GPT and Opus lanes have not produced a verdict on this head. A maintainer also needs to confirm this does not overlap the open P0 on cron jobs vanishing from the store after a run, since this PR changes the terminal store writes.
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 058a0a8. The change since the last review is the fix for the stale failure limit finding: a scheduled reap's frozen record now only counts the failure (count_failure), and the job the store hands back decides the auto pause with its stored auto_pause_after_failures (pause_at_stored_limit). The single cron_auto_pause row is written by the save that lands the pause (audit_landed_pause), with a marker on the record so a held retry does not write a second one.
What I verified on this head:
- The cron terminal settlement, cancel, bindings, malformed entry, busy store and reload suites pass (438 passed).
- With the delta in
settlement.py,model.pyandrun_merge.pyreverted,test_a_scheduled_reap_keeps_a_limit_saved_during_the_runfails for both a stored limit of 0 and of 10, andtest_a_scheduled_reap_at_the_stored_limit_audits_one_auto_pausefails in three of four cases. Both are pinned by tests. - I traced the manual and held save paths:
record_failureis nowcount_failureplusfailure_limit_reached, so the manual branch keeps its old behavior and its one audit row. - The GPT and Opus lanes report no findings on this head; Design and First Principles pass. The First Principles note on the manual branch scope matches the earlier manual branch findings, which are fixed; I have nothing new to add there.
No new finding in the delta. The earlier yellow is resolved.
Not approving yet: PR Readiness and the backend test shards are still running on this head, the Design lane asks for an explicit maintainer sign off on reversing the documented cancel rule, and a maintainer needs to confirm this does not overlap the open P0 on recurring script cron jobs vanishing from the store after a successful run. This PR touches the same store decode and held record code that the P0 report suspects, though not the successful run result merge or any removal path.
|
HOLD: waiting on a maintainer decision. Reviewed at 058a0a8. The code looks ready. All of bolichen97's earlier must-fix items are fixed and pinned by tests that fail on the previous heads. GPT 6.1 and Opus 5.5 report no findings on this head, and CI is green. Open before approval:
The maintainer's ruling plus replies to items 2 and 3 clear this hold. — crew-pr-reviewer (r120) |
|
Pushed 3ebebe9, rebased onto main
The Goal, Why it matters and Not a goal sections are unchanged. On |
|
Correction to the note above: the 12,300 cron tests ran on |
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 3ebebe9.
The change since 058a0a8 limits the frozen terminal record to scheduled runs: a manual cancel or reap no longer calls freeze_terminal_record, and the manual branch of _merge_terminal_state_locked is main's code again. That resolves the earlier scope note about the manual branch writing a frozen one-shot auto-pause.
What I verified on this head:
- The terminal settlement, cancel, bindings, malformed entry, busy store and reload suites pass (442 passed).
- The new test file run against the 058a0a8 sources fails 10 cases, including every
test_a_manual_cancel_or_reap_freezes_no_terminal_recordcase, so the delta is pinned. - On the design question: I am fine with reversing the documented cancel park rule for scheduled runs. The reason the old rule existed (a late park must not overwrite a later choice) is carried by
user_pause_generation, the new key is additive and omitted at zero, and the restart window is stated in the description. - On the open P0 about recurring script jobs vanishing after a successful run: this PR does not touch the successful run result merge, record decoding beyond the one new key, field caps, or any removal path, and the held record reapply it extends only acts on held records. I do not see an overlap.
One finding, inline below: the Opus lane block on this head reproduces as a wrong reason in the 409 text when a user resumes a scheduled one-shot while it is being cancelled and the store is busy. The job state is right; the message is not. I rate it non-blocking on its own, but the lane block holds readiness, and the fix is small.
Not approving yet: the Opus lane blocks this head, PR Readiness is still checking, and the backend test shards are still running.
| and self._held_settlements.get(job_id) is held | ||
| and held.run_generation == generation | ||
| and live is not None | ||
| and live.user_pause_generation <= held.user_pause_generation |
There was a problem hiding this comment.
A resume during a busy cancel is reported as "a newer run of the job was recorded meanwhile".
When a scheduled at one-shot is cancelled and the user resumes it while the merge is awaited, enable_job moves user_pause_generation past the held record's. If the store lock is then busy, the merge never runs, so _merge_terminal_state_with_retry returns the busy reason instead of _PARK_NEWER_CHOICE. Here settlement_pending is False because the live generation is newer, and _cancelled_park_report sees an enabled at job, so it returns _ON_SCHEDULE_MESSAGE. The CronPauseNotSaved text then tells the user a newer run was recorded, which did not happen; the user's resume put the job back on its schedule, and _RESUMED_MESSAGE was added for exactly that case.
and live.user_pause_generation <= held.user_pause_generation
)
...
note, message = seams._cancelled_park_report(
live, park_failure, saved, held=settlement_pending
)Suggestion: when the held record is this cancel's settlement (held.run_generation == generation) and live.user_pause_generation > held.user_pause_generation, pass _PARK_NEWER_CHOICE as the reason so the report says the job was resumed (or paused) meanwhile. Update the resumed_here_then_busy expectation in test/test_cron_cancel.py, which currently pins the wrong text.
Rule: a user-facing error reason must describe what actually happened.
|
Pushed 50d126f, rebased onto main
Thanks for the sign-off on the scheduled-run cancel rule and the P0 #22485 check. |
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 50d126f (range-diff against the previous head: one commit, rebased, plus the resume fix and three new busy cases).
The point from my last review is fixed: a one-shot resumed after the freeze, then cancelled into a busy store, now reports the resume instead of a newer run, and resumed_here_then_busy pins the right text. Thanks.
Two items remain before this can go in. Both lanes that block on this head are about them, so readiness will stay red until they are addressed.
-
run_teardown.py line 580: a pause that was already saved before the cancel still produces a false 409. I reproduced it locally: start a scheduled plain one-shot, call enable_job(id, False) while it runs, then cancel with the store lock busy. The cancel raises CronPauseNotSaved with "may run again only if the gateway restarts first", and history says "may run again after a restart", while the store on disk already holds enabled False and user_paused True. Cause: the frozen record copies the paused job, so the held record's user_pause_generation equals the live one, settlement_pending is True, and the saved-store read is skipped. Suggestion: when live.user_paused is already True (or the record's user_pause_generation was bumped by a saved pause), still read _saved_job_state before choosing the held wording, and add a case like paused_before_cancel_then_busy next to paused_here_then_busy.
-
settlement.py line 148: in apply_settlement_record, the branch for a changed schedule returns before pause_at_stored_limit runs. A held reap that counted the failure reaching the limit, landing after another process edited the schedule, saves the threshold count with auto_paused False, so the job gets one more unattended run. The fix the GPT lane suggests (evaluate the stored limit in that branch and re-derive enabled) is small and keeps the newer-choice fence.
Design: I am fine with the approach. Keeping the terminal record in memory and retrying the write on each tick, for scheduled runs only, reuses the existing held-record owner and adds no new long-term debt. The in-memory limit across restarts and the two copies of the limit rules are reasonable follow-ups.
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
Pushed 0fcf044, rebased onto main
|
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 0fcf044 (range-diff against 50d126f: one commit, rebased, plus the fixes below). Both items from my previous review are fixed:
run_teardown.py: the cancel now reads the saved store even while its own settlement is pending, so a pause saved before the cancel froze its record answers success instead of a 409. Pinned by the newpaused_before_cancel_then_busycase.settlement.pyapply_settlement_record: when the schedule changed, the counted failure still lands andpause_at_stored_limitdecides the pause; only the old schedule's park is dropped. Pinned bytest_a_held_reap_at_the_stored_limit_pauses_a_job_whose_schedule_changed.
The new paused_here path in freeze_terminal_record pauses the run's own object once and carries the pause on the frozen record, so a reload no longer audits a failed one-shot twice. That also looks right to me.
Verified locally at this head: test/test_cron_terminal_settlement.py and test/test_cron_cancel.py pass (108 tests), with the PR source on the path.
No new findings. I will approve once readiness passes on this head: four backend shards and the Opus lane are still running.
Readiness on |
|
Follow-up on the Opus 5.5 lane for |
A scheduled cancel or reap releases its claim before the terminal store write returns, so a failed write left the occurrence due again. The frozen terminal record now goes to the held-record owner as a settlement and is retried once per tick until a save returns. The settlement freeze and apply rules live in cron_service/settlement.py. The record carries the cancel's own park, reads the denied and never-started flags off the object the run executed on, and re-applies its hold at its own generation after a reload. Ordinary held result records keep their previous save rule. A failed manual terminal save no longer restores a recurring job's provisional hold.
|
self-added: yes
|
|
@bolichen97, here is what changed since your review of
|
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed at 76d0a61. Good to ship; I will approve once readiness passes on this head.
What changed since my review of 0fcf044 (range-diff against the rebase onto main ae1bb2d):
- The GPT 6.1 blocking finding on 0fcf044 (
settlement.py:177, a held scheduled reap retried by a tick reset the failure count of a Run Now in flight) is fixed.newer_run_ownschecks that the live claim executes on the same job object and has a higher generation than the held record, and all four sites that land a settlement record (_hold_run_record,_apply_held_run_records_to, the held save, and the immediate terminal merge) now skip the record'sauto_pausedand failure count in that case. The park andenabledrules are unchanged. - I ran
test_cron_terminal_settlement.py,test_cron_cancel.py,test_cron_service_bindings.pyandtest_cron_load_malformed_entry.pyat this head: 426 passed. Withsettlement.pyandrun_merge.pyrestored to their 0fcf044 versions, the two new regressions fail, so they pin the fix. - The other range-diff changes are the rebase: the spec line and one test now expect a cancelled one-shot to be parked, following main.
Non-blocking, no change needed: the GPT 6.1 finding on this head (cron.py:1145, a resume whose save fails still bumps the pause generation) was downgraded on adjudication. I agree: main also keeps a failed resume applied in memory, and the result is what the user asked for.
Lanes on this head: Design, First Principles, Opus 5.5 and GPT 6.1 have no blocking findings; UX skipped. Backend tests, E2E, integration and readiness are still running.
bolichen97
left a comment
There was a problem hiding this comment.
Approved at 76d0a61. Gates checked: readiness passed on this head, all checks green, no AI lane BLOCK (GPT 6.1 finding at cron.py:1145 downgraded on adjudication and agreed non-blocking), 13 files, no merge conflict, design sign-off given earlier. The GPT finding on 0fcf044 (held reap retry reset a newer run's count) is fixed by newer_run_owns; the two new regressions fail on 0fcf044 code.
Problem / Motivation
Goal: Keep a scheduled cron occurrence from being admitted again after its cancel or reap terminal save fails.
A scheduled cancel or reap releases its run claim before the terminal store write returns. A failed write can leave the stored occurrence due again.
The fence is process memory: an abrupt restart before a retry lands reads the stale store and can admit the occurrence again, so this narrows the window rather than closing it.
Why it matters
Duplicate unattended work can repeat side effects after a cancellation or timeout even though the prior run already started.
Not a goal
What changed (motivation → approach → change)
A scheduled cancel or reap freezes one terminal record and holds it until a save lands it. Keeping the claim until the write lands was rejected: under a failing store a manual Run would answer 409 and cancel, reap and shutdown would loop; the held record blocks only scheduled admission.
This reverses a documented cancel rule: an unlanded park of a cancelled one-shot was never retried, so it could not overwrite a later choice, though a resume made in the gap was overridden. Now a scheduled run's park is retried, and
user_pause_generationcarries the old rule's reason: a pause or resume made after the freeze keeps its whole pause state. The pin"a later write parked the job"now expects the park on disk.cron_service/settlement.pyowns the record.freeze_terminal_recordwrites the cancel's park on it and reads the runtime-only denied and never-started flags off the claim's job. For a dispatcheddelete_after_runone-shot it appliesauto_pause_failed_one_shotto the claim's job, as the finally does, and copies the pause, audited once. It counts a reap's failure; the stored limit decides the auto-pause, even after a schedule edit, audited once by the landing save.apply_settlement_recordre-applies the hold after every reload.user_paused. Only the held save writes the park, and the listed job then takes it. A run claimed after the record owns the job's pause and count, so no retry or merge writes the record's over them.enable_jobincrementsuser_pause_generationfor a request that changes the pause state, and for any one-shot resume, since another process can hold an unsaved park for it. Other requests record nothing and keep the store bytes.CronPauseNotSaved, a pause returns success.delete_after_runone-shot's auto-pause comes from the run's own teardown.Code:
cron_service/settlement.py,execution.py(shared auto-pause), andservice/run_merge.py,run_teardown.pyandrun_execution.py.cron.pykeeps_enable_job_locked, the settlement map, the cancel wording and the new binding.cron_service/model.pysplitsrecord_failure's count from its limit check, and withstore.pypersists the generation, omitted at zero.learn-cron-dashboard.mdrecords the reversal and the mixed-version limit.Backwards compatibility
Compatible:
user_pause_generationis additive, defaults to zero, and is written only after a pause transition or a one-shot resume. A store with neither keeps its bytes. Public cron APIs are unchanged. A manual cancel or reap writes the fields main writes. A one-shot resume another process saves during a manual run now moves the counter, so that store write reloads the job and the run's in-memory auto-pause does not land. The cancel rule above is an intentional behavior change for scheduled runs only.Mixed versions: a build without the counter never moves it. Its pause is kept, since the park is a pause too. A resume it makes while a record is held does not count as newer, so the park lands over it (tested, and in the spec).
Removes nothing: no supported path is removed.
Tests
Every test named here passes on this head.
test_cancel_save_failure_cannot_readmit_occurrence_after_reloadfails on main (a second scheduled claim is admitted after a reload).test_a_tick_that_wins_the_lock_before_the_first_merge_keeps_the_park,test_only_a_scheduled_settlement_yields_to_a_resume_after_the_freezeandtest_a_newer_held_run_is_not_reported_as_the_cancel_s_settlementfailed on an earlier head. Intest_cron_cancel.py,reloaded_then_busynow reports the job paused (the tick in the gap lands the park) andresumed_here_then_busythe resume, not a newer run (fails on 3ebebe9).test_a_manual_cancel_or_reap_freezes_no_terminal_recordfails on 058a0a8 (a manual cancel or reap froze a record).test_a_resume_saved_during_a_manual_run_survives_its_terminal_writeandtest_a_manual_run_keeps_a_resume_of_its_auto_paused_one_shotnow also pin that the manual merge gets no record; they failed on 0e7cc94 (a stale auto-pause over a resume saved during the run).test_a_manual_failure_at_the_limit_audits_one_auto_pausefails on 0e7cc94 (twocron_auto_pauserows).test_a_scheduled_reap_keeps_a_limit_saved_during_the_runfails on 28024bb (a reap auto-paused at the old limit after another process saved 0 or 10).test_a_scheduled_reap_at_the_stored_limit_audits_one_auto_pausepins onecron_auto_pauserow, from the save that lands the pause.paused_before_cancel_then_busy(falseCronPauseNotSaved),test_a_held_reap_at_the_stored_limit_pauses_a_job_whose_schedule_changed(job left enabled),test_a_reloaded_copy_does_not_audit_a_failed_one_shot_s_pause_twice(two rows).test_a_held_reap_retried_during_a_newer_run_keeps_that_run_s_countandtest_a_terminal_merge_keeps_the_count_of_a_run_claimed_before_itfail on 0fcf044 (a newer run's count reset: 2 → 1, 0 → 1).user_pause_generationtransitions.Manual verification
N/A: deterministic store-fault and concurrency tests cover the changed paths.
Screenshots / video
N/A: no user-visible frontend change.
Related Issues
No linked issue.
Pattern harvest
Rule candidate: concurrency
Pattern: a deferred terminal write must carry one immutable run record, yield to a pause or resume ordered after it, and release only after durable storage returns.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.