Skip to content

feat(cron): per-job auto-pause limit, including never - #15524

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/cron-max-consecutive-failures
Oct 20, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/cron-max-consecutive-failures

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let each cron job choose how many failures in a row pause it, including never, with failing runs backed off from the 5th failure so a raised or never limit still bounds spend.

A cron job pauses itself after 5 failed runs in a row. The 5 is the constant _AUTO_PAUSE_THRESHOLD in src/kiro_crew/cron.py, and no job field or config key overrides it. A watcher job that must keep running through a burst of host overload gets paused after five bad wakes, and then does nothing until someone runs kirocrew cron resume.

Why it matters

The auto-pause protects against a broken job firing forever, but some jobs are the recovery path themselves: a heartbeat or review-watch poll that fails while the host is saturated and would succeed a few minutes later. For those, the fixed 5 turns a short overload into a silent outage, and the only workaround today is a second job that resumes the first.

The gateway already leaves four kinds of infrastructure failure out of the count: starvation, a run that never started, a transient ACP retry, and a shared-runtime death. What still counts under overload is a run that did start and then failed: a script or command that overran its timeout, or an agent run that ended with a non-ok status and no output (record_failure() in slack/gateway.py, the started-overrun arm and the no-output arm). Those cannot be classified as weather from the run alone, because a slow host and a broken job produce the same timeout, so leaving them uncounted would also stop a genuinely broken job from ever pausing. The per-job limit lets the job's owner say which of the two a burst of them means for that job.

Not a goal

  • A Schedule page control for the setting. That needs a UI placement decision and screenshots, so it belongs in a follow-up PR.
  • Changing the default. A job that never sets the field still pauses after 5 failures.
  • Changing what a pause does or how a job resumes.
  • Re-checking a run against a limit changed while that run is in flight, or protecting a kirocrew cron resume from a run already in flight. A run judges the limit from its own copy of the job, so a mid-run change takes effect on the next run, and the worst case is one extra pause that a resume clears. The resume-versus-in-flight-run race already exists on main without any per-job limit, so it belongs in its own issue and PR.

What changed (motivation → approach → change)

Each job gets its own limit, auto_pause_after_failures, so a job that has to keep running through a burst of failures no longer stops after the fixed 5. The limit is an integer from 0 to 10000, default 5, where 0 means the job never auto-pauses on consecutive failures. The separate loop-stall breaker (_pause_for_loop_stall) does not read this limit and still applies.

Where you set it How Code
Chat tools cron_add / cron_update argument auto_pause_after_failures, 0 to 10000 mcp_cron.py, schema bounds in validation.py
CLI kirocrew cron add/update --auto-pause-after N, 0 to 10000 cli.py, cli_commands.py

The value is validated once, in the store (validate_auto_pause_after_failures in cron_service/fields.py), so every surface refuses the same bad input. Every comparison goes through CronJob.auto_pause_limit(), which returns None for 0 and falls back to the default for a stored value no writer can produce (not an integer, or outside 0..10000). record_failure reads it, so a job set to 0 is never paused. The gateway's shared-runtime-death bound in slack/gateway.py is unchanged from main: deaths a job did not cause hand over to its counter at the default of 5 for every job, so a tight limit such as 2 governs only failures the job owns, and a job set to 0 or above 5 gets the 5 charges that start its backoff without being paused.

A run keeps its own copy of the job, so a limit changed while it runs applies from the next run; the run-record merge is unchanged from main.

Spend safety for 0 and raised limits

A job that never pauses keeps firing however often it fails, and an agent job pays for a model turn on every run. A large limit does the same thing for longer: limit 10000 on an every=60 job is about 7 days of failing runs. The backoff is the one control that bounds that, and it applies the same way to 0 and to any limit above 5.

Any job still enabled at its 5th failure in a row backs off. It waits 5 minutes after last_run_ts, doubling per further failure up to an hour (CronJob.failure_backoff_until() in cron_service/model.py). For an every job last_run_ts is the last run's start; for a cron-expression job it is the last run's completion. is_due, next_wake_secs and compute_next_run_ts in cron_service/schedule.py all honour it, so a failing every=60 job runs 5 minutes after its 5th failure, 10 after its 6th, and from its 9th on at most once an hour. A success resets the counter and the backoff with it. A job with a limit of 5 or less pauses before it would back off.

The SEL audit bounds no spend; it records who turned the pause off or raised it. Every write whose old or new limit is 0 or above the default emits a cron_auto_pause_limit event with the job id, its creator, the session that made the write and the old and new limits (CronJob.audit_auto_pause_limit_change, outcomes never_pause_set, never_pause_cleared and limit_changed). The writer is the creating session for an add and the cron_update caller for an update, as the row's session_key and actor_session_key, and falls back to cron:<id> for a CLI write. The SEL log is served to the dashboard over /api/sel/events, which is where an owner finds which session set a job to never pause.

The trade-off is that a job with a raised limit, including the watcher this PR is for, is no longer retried at its own interval after the 5th failure in a row. It keeps running, but on the backoff above until one run succeeds.

Agents can set every limit, 0 included, through the chat tools and the CLI. An earlier revision made 0 owner-only with an MCP refusal and a denied-command rule. That was dropped: 0 and 10000 hit the same backoff from the 5th failure, so the refusal bounded nothing that the backoff does not already bound, and crons.json is sandbox-writable, so a check on who calls could not hold. The PR no longer touches src/kiro_crew/security/.

crons.json only gets the key when it differs from 5, so existing stores are byte-identical. The cron and CLI specs in docs/system-specs/modules/ describe the field.

Backwards compatibility

Compatible: the field is optional and defaults to today's 5, a record without it loads as 5, and a default-valued job serializes exactly as before.

Tests

test/test_cron_auto_pause_limit.py covers:

  • the default, a custom limit, a limit above 5, and 0 never pausing
  • invalid or out-of-range stored values falling back to the default
  • the value surviving a save and reload, and legacy records loading as 5
  • validation and range errors on the store and MCP tool surfaces, and a new limit not lifting an existing pause
  • cron_add and cron_update accepting 0 through the validated _call_tool_locally path and writing the never_pause_set and never_pause_cleared SEL rows
  • the SEL cron_auto_pause_limit event when either side of a write is 0 or above the default, and no event for changes within 1..5
  • the backoff: none below 5 failures, 5 minutes at the 5th, doubling, capped at an hour, reset by a success, the same for limits 6, 50 and 10000 as for 0, absent for limits 1 and 5, and applied by is_due, next_wake_secs and compute_next_run_ts for every and cron schedules
  • the cron spec paragraph stating the range, default and 0 = never, read from learn-cron-dashboard.md and compared with _AUTO_PAUSE_MAX and _AUTO_PAUSE_THRESHOLD
  • the kirocrew cron add/... row in cli.md naming --auto-pause-after N, 0 = never, and the flag on cron update
  • the published cron_add and cron_update MCP descriptions stating the 0..10000 range, the default, 0 = never, and the backoff as 5 minutes doubling to one hour

test/test_cron_refactor_contract.py, test/test_cli.py and test/test_cron_load_malformed_entry.py add the field to their fixtures: the full-record round trip, the exact add_job call the CLI makes, and the malformed-numeric-field matrix (a poisoned value loads as 5). test/test_cli.py also reads the real cron add --help and cron update --help output and compares the --auto-pause-after text with a string built from _AUTO_PAUSE_THRESHOLD and _AUTO_PAUSE_MAX, the same constants the CLI help, the cron_add/cron_update tool descriptions and the schema bounds in validation.py are built from.

test/test_runtime_death_is_a_process_event.py drives shared runtime deaths through the gateway's real cron callback: a job with limit 2 survives four deaths it did not cause and pauses on the fifth, and a job with limit 0 or above 5 is charged at the default so its backoff starts, without pausing it.

Manual verification

N/A: unit coverage exercises the store and MCP write paths and the run-record merge, and the change has no UI. The CLI flags forward to the same validated store call.

Related Issues

no linked issue: raised in a dashboard session about a review-watch cron job that kept auto-pausing during host memory pressure.

Deferred follow-ups, each tracked on its own issue:

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested a review from a team as a code owner September 30, 2026 18:15
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: Let each cron job set how many consecutive failures auto-pause it, with 0 meaning never, so a job that must keep running through host overload is not silently stopped after five bad wakes.
Not a goal: A Schedule page control, a new default, or any change to what a pause does or how a job resumes.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 30, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/cron-max-consecutive-failures branch from 2630901 to 960e6c4 Compare September 30, 2026 20:00
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased: main moved since this head's CI started (c1a02ce -> 3ecea02), and the Windows backend shards on 2630901150 failed on three fixtures that pin the full field set and did not yet name auto_pause_after_failures:

  • test/test_cli.py::TestCronCli (6 tests): the exact add_job kwargs the CLI forwards
  • test/test_cron_load_malformed_entry.py::test_numeric_field_consumer_matrix_pin: the numeric-field classification (the field loads a poisoned value as 5)
  • test/test_dashboard_cron_approval.py::TestCronListFields: the /api/crons job stub

960e6c44b8 adds the field to each fixture; no production code changed. The 201 test files that touch the cron surfaces pass locally (16118 passed). Shard 2's test_auto_improvement_members.py timeouts are unrelated to this diff.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of a55aad4a69ec40d30fa3ba92d6833d706af801cf via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a downgraded build keeps an unknown auto_pause_after_failures key without reading it, so a job set to 0 falls back to pausing at 5 instead of breaking the load.

[DESIGN-REVIEWED] a55aad4

Description read

[DESCRIPTION-READ] d5d77ddb462bc72c17b2e6a044fb926e49d343ac105ba27b55ade982e5677dd3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of a55aad4a69ec40d30fa3ba92d6833d706af801cf via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new cron_auto_pause_limit SEL event mostly repeats the per-call audit row that MCP cron writes already get. For CLI writes it records no actor at all.

Not justified as shipped

  • Item 5 — duplicate of mcp_shared_calls.call_tool_with_logging (src/kiro_crew/mcp_shared_calls.py:500): the description says the event "records who turned the pause off or raised it". That wrapper already writes a row for every cron_add and cron_update call, keyed by the caller's session_key, with the arguments in resources. On the CLI path, which that row does not cover, the new event falls back to cron:<id>, so it doesn't name who made the change either. What it adds is previous_limit, plus the limit staying visible when a long message pushes the arguments past the 500-character resources cap.
    Subtraction: delete audit_auto_pause_limit_change, _lifts_default_pause, the audit_actor kwarg popped in cron.py, the kwargs["audit_actor"] = _authz_session_key() line in mcp_cron.py, and both job_writes.py call sites.
    Clears when: the event is removed, or the description names a reader who needs previous_limit or the over-500-character case and cannot get it from the existing cron_add/cron_update invocation rows.

What this change ships

Inventory (8 items) — 7 justified

Intent: let a cron job's owner decide how many failures in a row pause it, including never, so a recovery-path watcher survives host overload. ADDITION.

  1. cron_add/cron_update take a per-job auto-pause limit (0..10000, default 5) — justified
  2. kirocrew cron add/update --auto-pause-after N — justified
  3. A limit of 0 means the job never auto-pauses on consecutive failures — justified
  4. Jobs with a limit of 0 or above 5 back off from their 5th failure, starting at 5 minutes and doubling to 1 hour — justified
  5. A new SEL cron_auto_pause_limit event with actor and previous/new limit — duplicate of mcp_shared_calls.call_tool_with_logging
  6. crons.json carries the key only when it differs from the default — justified
  7. The shared-runtime-death hand-over stays at the default of 5 for every job — justified
  8. The cron and CLI specs describe the field and the backoff — justified

[FIRST-PRINCIPLES-REVIEWED] a55aad4

Description read

[DESCRIPTION-READ] 4499e88930bf317e1cde63ae7d4bb45d95be20aa1693c21d9ef2397b4e022683

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed a55aad4a69ec40d30fa3ba92d6833d706af801cf via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a55aad4

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed a55aad4a69ec40d30fa3ba92d6833d706af801cf via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a55aad4

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/cron-max-consecutive-failures branch from 960e6c4 to ed01a89 Compare September 30, 2026 22:30
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: apply_run_record re-enables a recurring job when the merged verdict clears a pause its run tripped

fixed in ed01a89. Verified: record_failure sets enabled=False on the live job, and when _sync() does not reload, the merge target is that same object, so clearing auto_paused alone left the job stopped in memory and enabled on disk.
apply_run_record now sets enabled=True when run.auto_pause_tripped is set, the merged verdict is not paused, the job is not user-paused, and it is not an at-job (the at-job branch keeps owning enabled).
Four new TestInFlightLimitChange tests; the two with the merge target as the run's own object failed before the fix and pass after. A user-paused job stays disabled.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: no

fixed in ed01a89. The learn-cron-dashboard.md record_failure row and the cli.md auto-pause sentence now name the job's auto_pause_limit() (default 5, 0 = never). The line-482 paragraph already described the per-job field and was left as is.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in ed01a89. CronJob.auto_pause_limit() now falls back to the default for any stored value outside 0..10000, matching the writers' validation and its own docstring. The malformed-value test adds 10001 and 10**9; both failed before the fix.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in ed01a89. test_cli.py runs the real kirocrew cron add --help and cron update --help through main() and asserts the --auto-pause-after help names the default 5 and that 0 means never.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in ed01a89. The auto_pause_after_failures key is removed from the api_crons GET payload; it had no consumer, and the Schedule-page control that would read it is out of this PR's scope. The field is still set through the POST/PATCH bodies the description lists.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/cron-max-consecutive-failures branch from ed01a89 to d301a94 Compare October 1, 2026 00:15
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in d301a94. The --auto-pause-after help (cli.py), the cron_add/cron_update tool descriptions (mcp_cron.py) and both schema max_val bounds (validation.py) are now built from _AUTO_PAUSE_THRESHOLD and _AUTO_PAUSE_MAX, so a change to either constant changes every rendered copy.
The help test compares the real cron add --help / cron update --help output with a string built from the same constants and checks that a job built without the field holds _AUTO_PAUSE_THRESHOLD.
The markdown specs keep the literal values; a spec cannot interpolate, and it names _AUTO_PAUSE_THRESHOLD next to the number.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in d301a94. test_runtime_death_is_a_process_event.py drives shared runtime deaths through the gateway's real cron callback: a job with limit 2 pauses on the second death and a job with limit 0 stays enabled after 20. Both cases fail when the gateway line is reverted to the fixed _AUTO_PAUSE_THRESHOLD.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in d301a94. TestInFlightLimitChange covers a fired one-shot at-job (retained, fire-time-denied, delete-after-run) whose run tripped a pause the merge overrules: it stays disabled. Removing the run.schedule.kind != "at" guard makes the case fail.

@Pearcekieser
Pearcekieser force-pushed the feat/cron-max-consecutive-failures branch from d301a94 to 6269d85 Compare October 1, 2026 01:00
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes

fixed in 6269d85. test_cron_auto_pause_limit.py::test_the_module_spec_states_the_current_limit_contract reads docs/system-specs/modules/learn-cron-dashboard.md and asserts the paragraph states 0..{_AUTO_PAUSE_MAX}, `_AUTO_PAUSE_THRESHOLD` = {_AUTO_PAUSE_THRESHOLD} and "0 = never auto-pause", the same spec-against-constant pattern test_cron_history_summary_truncation.py uses for this file.
Changing the doc's 10000 to 9999 makes the test fail. cli.md states neither the default nor the range as a number, so it has nothing to pin.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@Pearcekieser
Pearcekieser force-pushed the feat/cron-max-consecutive-failures branch from 6269d85 to e4ebd31 Compare October 1, 2026 02:00
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

chore (non-blocking): identical-tree retrigger, no code change. 5217236b26 has the same tree as 2d76fbdb42; only the commit date moved. It re-runs Backend Tests (3.12, 7), Backend Tests (3.12, 8) and Coverage Gate, which failed on a tree-scan timeout in test_session_allocation_bindings.py and a FileNotFoundError in test_agent_scratch.py, both outside this diff and both passing locally. If either fails again on this head I will treat it as real.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

rebutted. The range has a use beyond 0. A limit of 1 to 4 pauses an expensive agent job sooner than the default of 5, which neither 0 nor the default can express. A limit above 5 keeps a job retrying on the backoff through a bad stretch but still gives it a stopping point, so an owner does not have to choose between pausing at 5 and never pausing. The setting is one integer field validated against _AUTO_PAUSE_MAX. A never-pause flag would be a second field next to the same counter, and the issue this PR closes asks for a per-job limit, not a flag. The owner kept the integer range when the owner-only refusal was dropped.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

rebutted. The description is right that the audit row bounds no spend, and it does not claim to. The row has a reader: the dashboard serves the security event log over /api/sel/events, and that is where an owner finds which session set a job to never pause or raised its limit. session_key and created_by name the job's creator, not the session that later changed the limit through cron_update, which is why audit_actor carries the caller. The row also records kirocrew cron writes made from an agent's shell, which the chat-tool call log never sees. The owner asked to keep it when the owner-only refusal was dropped.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

intended. The watcher this PR is for can live with that delay. The backoff starts only at the 5th failure in a row, waits 5 minutes and doubles per failure, so it reaches the one-hour cap at the 9th, and the first success resets the count and the interval. A host that stays saturated long enough to fail 9 runs in a row gets one attempt an hour instead of one per interval, which is the spend bound the backoff exists for. The PR description states this trade-off.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES at 5217236.

The code is sound. Every pause decision goes through auto_pause_limit() (model.py#L536). A job with limit 0 or above 5 backs off from its 5th failure in a row: 5 minutes, doubling to an hour (model.py#L442-L470). That bounds the spend I asked about on 10-02, and the SEL row records who set the limit. Your reasoning for dropping the owner-only check on 0 holds: a limit of 10000 hits the same backoff, and crons.json is writable from the sandbox. The First Principles and Design items on this head have replies that hold up, and CI is green.

What still blocks: three deferred items have no linked issue.

  1. A Schedule page control for the setting (Not a goal says it belongs in a follow-up PR).
  2. A Schedule page note explaining why the next fire time moved during a backoff (your Design reply).
  3. Applying a limit changed while a run is in flight (your Opus reply). If you now treat this as intended behaviour and not a follow-up, say so on the PR and no issue is needed.

#18050 is closed and covers a different race. To clear this, open one issue per remaining item and link each one from the description.

— crew-pr-reviewer (r111)

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

@iamwhatever todo (non-blocking): each deferred item now has its own linked issue.

  1. A Schedule page control for the per-job limit is Pearcekieser/KiroCrew#39.
  2. A Schedule page note saying a job is in failure backoff and why its next run moved is Pearcekieser/KiroCrew#40.
  3. Applying a limit changed while a run is in flight is Pearcekieser/KiroCrew#41.

They are on the fork because the upstream repo only accepts an issue opened together with its PR. Each one moves upstream with its PR when that work starts. #18050 is not a tracker for any of these; it is the separate, closed resume race. No code changed on 5217236b26.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

chore (non-blocking): rebased on main, no behaviour change. b9451472a3 replaces 5217236b26. Main #21951 split cli_commands.py into per-command modules, so this PR's two hunks moved to src/kiro_crew/cli_commands_cron.py: _cron_add passes auto_pause_after_failures, and the cron update field loop includes it. 4,769 cron, cron chat-tool, CLI, split-facade, validation and runtime-death tests pass locally.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): Backend Tests (3.12, 2) on b9451472a3 failed three tree-scan tests in test/test_facade_mirror_guards.py on the 120 s timeout, and Coverage Gate fails because that shard did. The same timeout hit three other branches in the same hour: fix/21617-temporary-briefing-withheld (run 39196983039), feat/promote-ephemeral-session (39196852832) and rfc-remote-computer-use (39196616985). Locally all 61 tests in that file pass on this head, the slowest at 53 s. This is runner load, not this diff. I will push one identical-tree retrigger in a CI slot, with no code change.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

chore (non-blocking): identical-tree retrigger, no code change. 4ac12bcf7f has the same tree as b9451472a3; only the commit date moved. It re-runs Backend Tests (3.12, 2) and Coverage Gate, which failed on the 120 s timeout in three test_facade_mirror_guards.py tree scans that hit other branches the same hour. If that timeout repeats on this head while other branches pass, I will treat it as real.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

rebutted. The row does not claim to catch every write, and the description does not offer it as a control. It records the writes that go through cron_add, cron_update and kirocrew cron, which is how agents and owners set the limit in practice, and it names the session that made the change. A direct edit of crons.json is outside every audit in the cron store, not only this one. The row has a reader: the dashboard serves the security event log over /api/sel/events, where an owner can see which session set a job to never pause. The maintainer review of 10-19 03:12Z accepted the row on that basis, and the owner asked to keep it.

A cron job auto-paused after a hard-coded 5 consecutive failures, with no
way to change it per job. auto_pause_after_failures (0..10000, default 5,
0 = never) is now settable through cron_add/cron_update, kirocrew cron
add/update --auto-pause-after, and the dashboard create/update API.
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 4ac12bc: the branch now conflicts with main.

The code is still sound. Every pause decision goes through auto_pause_limit() (model.py#L427-L440), and a job set to 0 or above 5 backs off from its 5th failure (model.py#L442-L470). The current-head First Principles item on the SEL row has a reply that holds, the other lanes have no findings, and CI is green.

To clear:

  1. Rebase onto main to resolve the conflict.
  2. Link the three follow-up issues from the PR description, not only from a comment. cron: Schedule page control for the per-job auto-pause limit Pearcekieser/KiroCrew#39, ci: scope workflow permissions to publishing jobs #40 and fix(wechat): guard WeCom frames against malformed payloads #41 cover the Schedule control, the backoff note and the in-flight limit change; the description's Related Issues section still says there is no linked issue. Our r111 change request asked for links in the description.

The new head needs another review after the rebase. I read the source and CI; I did not run tests.

— crew-pr-reviewer (r112)

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

chore (non-blocking): rebased on main, no behaviour change. a55aad4a69 replaces 4ac12bcf7f. Main #21598 split mcp_cron.py into owner modules. The cron_add and cron_update argument handling and the audit_actor pass stay in mcp_cron.py; _AUTO_PAUSE_AFTER_DESCRIPTION and both tool schema properties moved to mcp_cron_tool_list.py. The facade's TYPE_CHECKING blocks now name _AUTO_PAUSE_MAX, _AUTO_PAUSE_THRESHOLD and _AUTO_PAUSE_AFTER_DESCRIPTION, which test_mcp_cron_facade.py and test_facade_mirror_guards.py require. 4,762 cron, cron chat-tool, CLI, facade, validation and runtime-death tests pass locally.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

@iamwhatever todo (non-blocking): both HOLD items are done. The branch is rebased onto main as a55aad4a69, which merges cleanly. The Related Issues section of the description now links Pearcekieser#39 (Schedule page control), #40 (backoff note on the Schedule page) and #41 (limit changed while a run is in flight). No code changed beyond the rebase.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

intended. A build from before this PR already handles the key that way. cron_service/store.py on main keeps any top-level job key outside _KNOWN_JOB_KEYS as a key from a newer build and writes it back unchanged on the next save (the comment at line 41 and the cap at line 651). That build never reads auto_pause_after_failures, so its own fixed threshold of 5 applies: a job set to 0 pauses at its 5th failure in a row, and the load does not break. On upgrade again, the preserved key comes back into effect. This PR adds auto_pause_after_failures to this build's _KNOWN_JOB_KEYS probe, which test_reset_to_default_after_reload_sticks covers.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

rebutted. The call_tool_with_logging row covers only the chat tools. kirocrew cron add and kirocrew cron update from a shell never pass through it, so for those writes the new event is the only record of the change; it names cron:<id> as the actor when no session is known, which is what the store knows at that point. The row also carries previous_limit next to the new value, so an owner reading /api/sel/events sees a job go from 5 to 0 without diffing two tool-call payloads. The maintainer review of 10-19 03:12Z accepted the event on that basis, and the owner asked to keep it.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at a55aad4.

Both r112 items are cleared. The branch merges cleanly onto main, and the description now links Pearcekieser#39, #40 and #41 for the Schedule control, the backoff note and the in-flight limit change. Each issue covers its item. The rebase from 4ac12bc changes only the facade imports that main's mcp_cron.py split requires.

The logic is unchanged from what r111 found sound: every pause decision reads auto_pause_limit(), and a job set to 0 or above 5 backs off from its 5th failure, from 5 minutes doubling to an hour (model.py L427-L470). The First Principles item on the SEL row has a reply for this head that holds, since CLI writes never pass through call_tool_with_logging. The other lanes have no findings, and all checks on this head are green.

I read the source and CI; I did not run tests.

— crew-pr-reviewer (r113)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants