Repository navigation
feat(cron): per-job auto-pause limit, including never - #15524
Conversation
|
Intent: Let each cron job set how many consecutive failures auto-pause it, with 0 meaning never, so a job that must keep running through host overload is not silently stopped after five bad wakes. |
2630901 to
960e6c4
Compare
|
Rebased: main moved since this head's CI started (c1a02ce -> 3ecea02), and the Windows backend shards on
|
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that a downgraded build keeps an unknown [DESIGN-REVIEWED] a55aad4 Description read[DESCRIPTION-READ] d5d77ddb462bc72c17b2e6a044fb926e49d343ac105ba27b55ade982e5677dd3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The new Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: let a cron job's owner decide how many failures in a row pause it, including never, so a recovery-path watcher survives host overload. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] a55aad4 Description read[DESCRIPTION-READ] 4499e88930bf317e1cde63ae7d4bb45d95be20aa1693c21d9ef2397b4e022683 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
960e6c4 to
ed01a89
Compare
|
self-added: yes
|
|
self-added: no
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
ed01a89 to
d301a94
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
d301a94 to
6269d85
Compare
|
self-added: yes
|
6269d85 to
e4ebd31
Compare
|
chore (non-blocking): identical-tree retrigger, no code change. |
|
|
|
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at 5217236.
The code is sound. Every pause decision goes through auto_pause_limit() (model.py#L536). A job with limit 0 or above 5 backs off from its 5th failure in a row: 5 minutes, doubling to an hour (model.py#L442-L470). That bounds the spend I asked about on 10-02, and the SEL row records who set the limit. Your reasoning for dropping the owner-only check on 0 holds: a limit of 10000 hits the same backoff, and crons.json is writable from the sandbox. The First Principles and Design items on this head have replies that hold up, and CI is green.
What still blocks: three deferred items have no linked issue.
- A Schedule page control for the setting (Not a goal says it belongs in a follow-up PR).
- A Schedule page note explaining why the next fire time moved during a backoff (your Design reply).
- Applying a limit changed while a run is in flight (your Opus reply). If you now treat this as intended behaviour and not a follow-up, say so on the PR and no issue is needed.
#18050 is closed and covers a different race. To clear this, open one issue per remaining item and link each one from the description.
— crew-pr-reviewer (r111)
|
@iamwhatever todo (non-blocking): each deferred item now has its own linked issue.
They are on the fork because the upstream repo only accepts an issue opened together with its PR. Each one moves upstream with its PR when that work starts. #18050 is not a tracker for any of these; it is the separate, closed resume race. No code changed on |
|
chore (non-blocking): rebased on main, no behaviour change. |
|
note (non-blocking): Backend Tests (3.12, 2) on |
|
chore (non-blocking): identical-tree retrigger, no code change. |
|
A cron job auto-paused after a hard-coded 5 consecutive failures, with no way to change it per job. auto_pause_after_failures (0..10000, default 5, 0 = never) is now settable through cron_add/cron_update, kirocrew cron add/update --auto-pause-after, and the dashboard create/update API.
|
HOLD at 4ac12bc: the branch now conflicts with main. The code is still sound. Every pause decision goes through To clear:
The new head needs another review after the rebase. I read the source and CI; I did not run tests. — crew-pr-reviewer (r112) |
|
chore (non-blocking): rebased on main, no behaviour change. |
|
@iamwhatever todo (non-blocking): both HOLD items are done. The branch is rebased onto main as |
|
|
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE at a55aad4.
Both r112 items are cleared. The branch merges cleanly onto main, and the description now links Pearcekieser#39, #40 and #41 for the Schedule control, the backoff note and the in-flight limit change. Each issue covers its item. The rebase from 4ac12bc changes only the facade imports that main's mcp_cron.py split requires.
The logic is unchanged from what r111 found sound: every pause decision reads auto_pause_limit(), and a job set to 0 or above 5 backs off from its 5th failure, from 5 minutes doubling to an hour (model.py L427-L470). The First Principles item on the SEL row has a reply for this head that holds, since CLI writes never pass through call_tool_with_logging. The other lanes have no findings, and all checks on this head are green.
I read the source and CI; I did not run tests.
— crew-pr-reviewer (r113)
Problem / Motivation
Goal: Let each cron job choose how many failures in a row pause it, including never, with failing runs backed off from the 5th failure so a raised or never limit still bounds spend.
A cron job pauses itself after 5 failed runs in a row. The 5 is the constant
_AUTO_PAUSE_THRESHOLDinsrc/kiro_crew/cron.py, and no job field or config key overrides it. A watcher job that must keep running through a burst of host overload gets paused after five bad wakes, and then does nothing until someone runskirocrew cron resume.Why it matters
The auto-pause protects against a broken job firing forever, but some jobs are the recovery path themselves: a heartbeat or review-watch poll that fails while the host is saturated and would succeed a few minutes later. For those, the fixed 5 turns a short overload into a silent outage, and the only workaround today is a second job that resumes the first.
The gateway already leaves four kinds of infrastructure failure out of the count: starvation, a run that never started, a transient ACP retry, and a shared-runtime death. What still counts under overload is a run that did start and then failed: a script or command that overran its timeout, or an agent run that ended with a non-ok status and no output (
record_failure()inslack/gateway.py, the started-overrun arm and the no-output arm). Those cannot be classified as weather from the run alone, because a slow host and a broken job produce the same timeout, so leaving them uncounted would also stop a genuinely broken job from ever pausing. The per-job limit lets the job's owner say which of the two a burst of them means for that job.Not a goal
kirocrew cron resumefrom a run already in flight. A run judges the limit from its own copy of the job, so a mid-run change takes effect on the next run, and the worst case is one extra pause that a resume clears. The resume-versus-in-flight-run race already exists on main without any per-job limit, so it belongs in its own issue and PR.What changed (motivation → approach → change)
Each job gets its own limit,
auto_pause_after_failures, so a job that has to keep running through a burst of failures no longer stops after the fixed 5. The limit is an integer from 0 to 10000, default 5, where 0 means the job never auto-pauses on consecutive failures. The separate loop-stall breaker (_pause_for_loop_stall) does not read this limit and still applies.cron_add/cron_updateargumentauto_pause_after_failures, 0 to 10000mcp_cron.py, schema bounds invalidation.pykirocrew cron add/update --auto-pause-after N, 0 to 10000cli.py,cli_commands.pyThe value is validated once, in the store (
validate_auto_pause_after_failuresincron_service/fields.py), so every surface refuses the same bad input. Every comparison goes throughCronJob.auto_pause_limit(), which returnsNonefor 0 and falls back to the default for a stored value no writer can produce (not an integer, or outside 0..10000).record_failurereads it, so a job set to 0 is never paused. The gateway's shared-runtime-death bound inslack/gateway.pyis unchanged from main: deaths a job did not cause hand over to its counter at the default of 5 for every job, so a tight limit such as 2 governs only failures the job owns, and a job set to 0 or above 5 gets the 5 charges that start its backoff without being paused.A run keeps its own copy of the job, so a limit changed while it runs applies from the next run; the run-record merge is unchanged from main.
Spend safety for 0 and raised limits
A job that never pauses keeps firing however often it fails, and an agent job pays for a model turn on every run. A large limit does the same thing for longer: limit 10000 on an
every=60job is about 7 days of failing runs. The backoff is the one control that bounds that, and it applies the same way to 0 and to any limit above 5.Any job still enabled at its 5th failure in a row backs off. It waits 5 minutes after
last_run_ts, doubling per further failure up to an hour (CronJob.failure_backoff_until()incron_service/model.py). For aneveryjoblast_run_tsis the last run's start; for a cron-expression job it is the last run's completion.is_due,next_wake_secsandcompute_next_run_tsincron_service/schedule.pyall honour it, so a failingevery=60job runs 5 minutes after its 5th failure, 10 after its 6th, and from its 9th on at most once an hour. A success resets the counter and the backoff with it. A job with a limit of 5 or less pauses before it would back off.The SEL audit bounds no spend; it records who turned the pause off or raised it. Every write whose old or new limit is 0 or above the default emits a
cron_auto_pause_limitevent with the job id, its creator, the session that made the write and the old and new limits (CronJob.audit_auto_pause_limit_change, outcomesnever_pause_set,never_pause_clearedandlimit_changed). The writer is the creating session for an add and thecron_updatecaller for an update, as the row'ssession_keyandactor_session_key, and falls back tocron:<id>for a CLI write. The SEL log is served to the dashboard over/api/sel/events, which is where an owner finds which session set a job to never pause.The trade-off is that a job with a raised limit, including the watcher this PR is for, is no longer retried at its own interval after the 5th failure in a row. It keeps running, but on the backoff above until one run succeeds.
Agents can set every limit, 0 included, through the chat tools and the CLI. An earlier revision made 0 owner-only with an MCP refusal and a denied-command rule. That was dropped: 0 and 10000 hit the same backoff from the 5th failure, so the refusal bounded nothing that the backoff does not already bound, and
crons.jsonis sandbox-writable, so a check on who calls could not hold. The PR no longer touchessrc/kiro_crew/security/.crons.jsononly gets the key when it differs from 5, so existing stores are byte-identical. The cron and CLI specs indocs/system-specs/modules/describe the field.Backwards compatibility
Compatible: the field is optional and defaults to today's 5, a record without it loads as 5, and a default-valued job serializes exactly as before.
Tests
test/test_cron_auto_pause_limit.pycovers:cron_addandcron_updateaccepting 0 through the validated_call_tool_locallypath and writing thenever_pause_setandnever_pause_clearedSEL rowscron_auto_pause_limitevent when either side of a write is 0 or above the default, and no event for changes within 1..5is_due,next_wake_secsandcompute_next_run_tsforeveryandcronscheduleslearn-cron-dashboard.mdand compared with_AUTO_PAUSE_MAXand_AUTO_PAUSE_THRESHOLDkirocrew cron add/...row incli.mdnaming--auto-pause-after N,0= never, and the flag oncron updatecron_addandcron_updateMCP descriptions stating the 0..10000 range, the default, 0 = never, and the backoff as 5 minutes doubling to one hourtest/test_cron_refactor_contract.py,test/test_cli.pyandtest/test_cron_load_malformed_entry.pyadd the field to their fixtures: the full-record round trip, the exactadd_jobcall the CLI makes, and the malformed-numeric-field matrix (a poisoned value loads as 5).test/test_cli.pyalso reads the realcron add --helpandcron update --helpoutput and compares the--auto-pause-aftertext with a string built from_AUTO_PAUSE_THRESHOLDand_AUTO_PAUSE_MAX, the same constants the CLI help, thecron_add/cron_updatetool descriptions and the schema bounds invalidation.pyare built from.test/test_runtime_death_is_a_process_event.pydrives shared runtime deaths through the gateway's real cron callback: a job with limit 2 survives four deaths it did not cause and pauses on the fifth, and a job with limit 0 or above 5 is charged at the default so its backoff starts, without pausing it.Manual verification
N/A: unit coverage exercises the store and MCP write paths and the run-record merge, and the change has no UI. The CLI flags forward to the same validated store call.
Related Issues
no linked issue: raised in a dashboard session about a review-watch cron job that kept auto-pausing during host memory pressure.
Deferred follow-ups, each tracked on its own issue:
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)