Skip to content

fix(cron): record error when no tool call of a cron run was approved - #37604

Merged
bolichen97 merged 1 commit into
mainfrom
fix/35101-cron-unapproved-run-error
Nov 19, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/35101-cron-unapproved-run-error

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a cron run in which no tool call ran, because every call was refused by a governance profile, by the agent spec's own PreToolUse hook, or left unapproved, shows as Error and names those calls, instead of reading ok.

A cron agent run records last_status: ok when no tool call was approved, none was blocked by the security gate, and at least one was refused another way: a governance denial (policy_deny), the spec hook's refusal (spec_hook), or an approval that was rejected or timed out. The Schedule page then shows a run that did nothing as a success.

Why it matters

In #34927 a daily job ran an agent whose spec carries PreToolUse hooks, under a profile that turns hooks off. Every tool call was refused and each run still showed ok, so nothing told the user the job had stopped working.

Not a goal

  • Moving the failure counter or auto-pausing a recurring job for these runs: a policy state or an absent approver is not evidence the job is broken (fix(cron): record failure when the gate blocks every tool call #2666).
  • Naming the unapproved calls of a run that also had an approved call. That run stays ok.
  • Changing how a tool-free run, an all-security-blocked run or a partly security-blocked (Partial) run is recorded.
  • Changing the on_tool_gate callback shape.

What changed (motivation → approach → change)

A run whose tool calls were all refused, none of them by the security gate, now reads Error and says which calls were refused.

The run's tool calls last_status before last_status now Failure counter
none approved, none security-blocked, at least one refused another way ok error, the reason names up to three calls, then and N more does not move
tool-free, all or some security-blocked, or at least one approved unchanged unchanged unchanged

The cause was the run's tally in cron_verdict.py: it kept a refusal's title and reason only for a security block, and kept every other refusal as a bare count, so the verdict in runtime_charge.py had nothing to report and called the run a success. The tally now keeps such calls too, and the verdict records the run as error without counting a failure or a success. For every kind of refused call the tally keeps the exact count but only the first three by name, each title and reason redacted and then cut to the 500-character line cap, so a run that notes many long titles holds a bounded amount of text; the reason ends with and N more when it names fewer calls than it counts. The reason reads none of N tool call(s) ran, all refused or not approved: <title> (<refusal text>), with not approved for an approval that recorded no text. It is the same line in last_error and in front of the delivered result, and it does not use the Partial wording, so the Schedule page shows Error. A delete_after_run one-shot is kept and paused, as any run ending in error is. The gate-verdict table in learn-cron-dashboard.md gains this arm.

Backwards compatibility

Compatible: the on_tool_gate shape, the job's fields and the run record are unchanged; a run in which no tool call ran, which read ok, now reads error, and a reason that names three of more than three refused calls ends with and N more.
Removes nothing: _GateTally is private to cron_verdict.py, and slack/gateway.py only re-exports it; it folds the parallel refused_rules list into refused_calls, makes refused a count, and stops storing call names past the three its summary shows, which nothing read. The two test files that read its fields, test_cron_refusal_status.py and test_llm_helpers_batch_guard_deferral.py, assert the count and the kept names.

Tests

Proof: PROVEN test/test_cron_refusal_status.py::test_an_unapproved_run_is_not_a_failure, test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[unapproved], test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[policy-deny], test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[spec-hook], test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[mixed] (head 674724b)

Determinism: the two adapted test_llm_helpers_batch_guard_deferral.py tests passed 20 of 20 runs at -n0, test_cron_refusal_status.py 10 of 10 at -n 4 --dist loadgroup --max-worker-restart=2, and both files once in a shuffled order (seed 35101, 54 of 54). With the source changes reverted, prove.py finds 14 of the new and changed tests failing, the two adapted batch-guard tests among them.

  • test_an_unapproved_run_is_not_a_failure: a timed-out approval run records error and still never auto-pauses after five runs.
  • TestUnapprovedRun::test_a_run_with_no_approved_call_records_error: an unapproved, a policy_deny, a spec_hook and a mixed run each record error with no failure counted.
  • TestUnapprovedRun::test_it_spends_no_failure_budget and test_it_does_not_reset_a_failure_streak: five such runs leave the job enabled and unpaused, and one after two blocked runs keeps consecutive_failures at 2.
  • TestUnapprovedRun::test_the_reason_names_each_call_with_its_refusal_text and test_the_reason_names_at_most_three: each call is named with the first line of its reason or not approved, at most three, within 500 characters.
  • TestUnapprovedRun::test_the_schedule_page_shows_it_as_error_not_partial: the reason does not match the Partial pattern in backendPhrases.json.
  • TestUnapprovedRun::test_the_notification_names_the_refused_calls and test_it_reaches_the_sequential_path: the delivered body opens with the same reason as last_error on the single-agent and the agent_sequence paths.
  • TestUnapprovedRun::test_one_approved_call_keeps_the_run_ok and test_a_tool_free_run_stays_ok: the unchanged arms stay ok.
  • TestSchedulerContract::test_an_unapproved_one_shot_is_kept_and_auto_paused: a delete_after_run one-shot with only a policy denial is kept, paused and disabled, with no failure counted.
  • test_the_tally_counts_every_call_but_keeps_three_bounded_names (security-blocked and unresolved): ten calls with 100,000-character titles and reasons leave an exact count of 10, three kept names, each field within 500 characters, and a reason within 500 characters that ends with and 7 more.
  • test_llm_helpers_batch_guard_deferral.py::test_a_deferral_retried_next_step_reports_no_security_block and test_a_transport_floor_refusal_is_still_a_security_block: a batch-guard deferral retried on the next step leaves a security-block count of 0 and no kept name, and a transport-floor refusal counts 1 and keeps its title.

Manual verification

Run on Linux (x86_64), from the PR's commit and from its base, in an isolated gateway (throwaway data home, ephemeral port on 127.0.0.1) with the packaged fake agent backend and crons on. A cron agent job created through POST /api/crons with a [[GATED]] message makes the fake raise a real permission request for fake demo tool; the job was fired once through POST /api/crons/<id>/run. Without a profile the unattended cron approved that call, so a governance profile bound to the cron surface that allows only read was written into the isolated home. The audit then recorded the call as denied, mechanism policy_deny.

Case Path last_status base → fix last_error base → fix consecutive_failures base → fix
the call denied by the cron-bound profile (policy_deny) gateway ok → error None → none of 1 tool call(s) ran, all refused or not approved: fake demo tool (Blocked by governance policy: …) 0 → 0
same run, Schedule page Status dashboard OK → Error
a governance denial, a spec-hook refusal and an approval with no recorded answer verdict ok → error None → none of 3 tool call(s) ran, …: Write /vault/a.md (Denied by governance profile: no writes), Bash ls /vault (Refused by the agent spec's PreToolUse hook), Read /vault/b.md (not approved) 0 → 0
a tool call with no permission request ([[TOOL]]) gateway ok → ok None → None 0 → 0
one approved and one unapproved call verdict ok → ok None → None 0 → 0
one security-blocked call verdict error → error all 1 tool call(s) blocked by the security gate: … both 1 → 1
no tool call verdict ok → ok None → None 0 → 0

auto_paused stayed false in every case. On the fix the gateway run's last_result and its notification start with Blocked: and the same reason, and its history row reads failure with that reason. The gateway case was re-run on this head with the same result, and its reason carries no and N more.

The bounded tally, against a tally that kept every call:

Check a tally that kept every call this code
100 calls of one kind, each with a 100,000-character title and reason 100 entries, 20,002,090 characters kept a count of 100; 3 entries, 3,000 characters kept; the 500-character reason ends , and 97 more
reason for three or fewer calls (three unapproved; one security-blocked, alone or beside an approved call; three security-blocked) byte-identical in all four cases
a title carrying a 300-character bearer token and a 40-character secret from index 480 no 12-character piece of either in the reason the same

Screenshots / video

The Schedule page for the same job shape and refusal, without the fix and on this head:

Schedule row reads OK on the base although the run's only tool call was denied

Schedule row reads Error on the fix, and a job whose tool call needed no permission reads OK

Related Issues

Closes #35101

Builds on #35337 by @rohnkum.

Split out of #34927. #28603 also touches how a cron run names a lost call; whichever lands second folds onto the other.

Sibling sweep

Swept: git grep -n -e '_GateTally' -e '_apply_gate_verdict' -e 'refusal_summary' -e '_annotate_partial_block' -e 'auto_pause_failed_one_shot' -- src website/src
Sibling: src/kiro_crew/slack/gateway_runtime/cron_verdict.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/runtime_charge.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/cron_turn.py -- not affected -- both cron agent paths call the shared _annotate_partial_block and _apply_gate_verdict, so they take the new arm unchanged
Sibling: src/kiro_crew/slack/gateway_runtime/cron_failure.py -- not affected -- it alerts after a counted failure and the new arm counts none
Sibling: src/kiro_crew/cron_service/execution.py -- not affected -- auto_pause_failed_one_shot already keeps and pauses a delete_after_run run whose status is error
Sibling: src/kiro_crew/cron_service/service/run_execution.py -- not affected -- it calls auto_pause_failed_one_shot with the run's last_status, so the new error arm is kept and paused like any other
Sibling: src/kiro_crew/cron_service/settlement.py -- not affected -- a cancelled or reaped run's settlement applies the same auto_pause_failed_one_shot rule
Sibling: src/kiro_crew/slack/gateway.py -- not affected -- it re-exports _GateTally, _annotate_partial_block and _apply_gate_verdict unchanged
Sibling: website/src/utils/cronPartlyBlocked.ts -- not affected -- the new reason does not match its Partial pattern, so the run shows as Error

Pattern harvest

Not generalizable: the gap is one missing arm in a single verdict function whose arms are all listed in one spec table.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 674724b7be7444cc21dd093a43932d33a2818ce1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 674724b7be7444cc21dd093a43932d33a2818ce1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix is aimed at its cause, the _GateTally that dropped non-security refusals. One rider, which bounds and suffixes the security-block names, goes past the Goal.

Notes

  • Item 4 — rides along: the Goal covers only runs where nothing ran, yet _keep and _more now also cap and suffix the existing all-blocked and partial reasons (all {self.refused} tool call(s)… + more). ## Not a goal excludes "Changing how … an all-security-blocked run or a partly security-blocked (Partial) run is recorded". Subtraction: keep refused as the base list and apply the three-name cap and the and N more suffix only on the none_approved branch.

What this change ships

Inventory (5 items) — 4 justified

Intent: a user whose cron job's tool calls were all refused by policy, by a spec hook or for lack of approval sees the run as failed instead of ok. FIX (#34927).

  1. A run with no approved call and no security block now reads Error on the Schedule page instead of OK — justified
  2. That run's last_error and delivered result open with "none of N tool call(s) ran…", naming up to three calls with their refusal text — justified
  3. That run moves neither the failure counter nor the success counter, and a one-shot run is kept and paused — justified
  4. All-blocked and Partial reasons now end with "and N more" past three calls, and each name is cut to 500 characters — rides along, outside ## Not a goal
  5. The test_an_unapproved_run_is_not_a_failure pin flips from != "error" to == "error". Its docstring's reason, keeping the failure counter still, is kept, and the description declares the flip — justified

[FIRST-PRINCIPLES-REVIEWED] 674724b

Description read

[DESCRIPTION-READ] 9b35d8f096d3fe9861bec831c132d643965395aacecaf25961016d484dbb5179

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 674724b7be7444cc21dd093a43932d33a2818ce1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this reuses the existing Partial arm's "error, neither counter moves" verdict, and nothing outside cron_verdict.py reads the reshaped _GateTally fields.

[DESIGN-REVIEWED] 674724b

Description read

[DESCRIPTION-READ] b0fce6bcb7a76f5f16d7f8a76171e73daf52cc5ed10d6d9aeac0745cdd797df9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 674724b7be7444cc21dd093a43932d33a2818ce1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 674724b

Verdict parsed from the review's SHA-scoped output markers for commit 674724b7be7444cc21dd093a43932d33a2818ce1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 674724b7be7444cc21dd093a43932d33a2818ce1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 674724b7be7444cc21dd093a43932d33a2818ce1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 674724b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 674724b7be7444cc21dd093a43932d33a2818ce1 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Adds a fourth verdict arm, none_approved: no call approved, none security-blocked, at least one refused another way. Such a run records last_status=error, names up to three refused calls with their refusal text, and moves neither the failure nor the success counter. Verified: tools that ran without a permission request are still reported as approved (turn_stream.py:690-696), so the arm only fires when nothing executed. The scheduler's _execute skips record_success() on error, so a failure streak is kept. Both the single-agent and sequential cron paths share the annotate/verdict functions. auto_pause_failed_one_shot keeps and pauses delete_after_run one-shots, as the description and spec say. The design matches the existing fire-time governance-deny handling. The spec doc table is updated, the tests fail without the change, and no existing test depends on the old ok behaviour for an all-unapproved run.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit f586517acf · 250 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 16, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 16, 2026 07:19
@bolichen97

bolichen97 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Superseding decisions require a maintainer override span=32e74e8d8d8c

Rebutted: not in this PR. git diff d15505809e1d...f586517acf76 changes only docs/system-specs/modules/learn-cron-dashboard.md, cron_verdict.py, runtime_charge.py and test/test_cron_refusal_status.py; it touches neither src/kiro_crew/crew_mode.py nor any file under docs/decisions/.
DEFAULT_CREW_MODE_AGENT reads TEAM_LEAD_AGENT_FILENAME at this PR's base and CONDUCTOR_AGENT_FILENAME on main: the change and the superseding entries the finding cites landed on main, so recorded-decisions-are-not-regressions has no Supersedes: entry in this diff to fire on.
The lane's own adjudication reads it the same way (FLAG F1: the cited default and entries come from base main). The new head 9001be7 still carries only the same four files.
[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/35101-cron-unapproved-run-error branch from f586517 to 9001be7 Compare November 18, 2026 01:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in gateway_runtime/cron_verdict.py, gateway_runtime/runtime_charge.py, test/test_cron_refusal_status.py, test/test_llm_helpers_batch_guard_deferral.py and docs/system-specs/modules/learn-cron-dashboard.md since this head went green (d155058 -> cba2110); CI re-running.

The previous head failed the two tests main added in test/test_llm_helpers_batch_guard_deferral.py, which read the tally's security-blocked calls as a list of titles; they now read the count and the kept names. Four assertions in this PR's own tests follow main's Blocked: banner wording. The third red, test_an_unchanged_folder_is_not_walked_again, is fixed on main (#38980, #39010).

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
@bolichen97
bolichen97 force-pushed the fix/35101-cron-unapproved-run-error branch from 862fa58 to 3081d75 Compare November 19, 2026 20:20
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/learn-cron-dashboard.md since this head went green (cba2110 -> 5c9bf25); CI re-running.

The code patch is unchanged (same patch-id). The previous head's reds were all main's: test_every_new_module_level_cache_has_a_per_test_reset failed on a cache main added in #38992 and fixed in #39479, and the Windows-only test_the_bar_is_resent_only_when_its_items_change (#39566) and test_pr_watchers.py watcher waits (#39485) are flakes main has worked on since (#39505, #39516). The new base carries all three.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
A cron run in which no tool call was approved and none was blocked by
the security gate recorded last_status ok, even when every call was
refused by a governance profile, by the agent spec's PreToolUse hook,
or by a rejected or timed-out approval. _GateTally kept only a count of
those calls, and _apply_gate_verdict called record_success() for them.

_GateTally now keeps the first three such calls' titles and first
reason lines, and a new none_approved property marks a run with no
approved call, no security block and at least one other refusal.
_apply_gate_verdict records that run as error without moving either
counter, so a delete_after_run one-shot is kept and auto-paused.
refusal_summary() names the kept calls with their refusal text, or
"not approved", and _annotate_partial_block puts the same reason in
front of the delivered result. The reason does not use the
partial-block wording, so the Schedule page shows the run as Error.

The tally counts every refused and unresolved call but keeps only the
three it names of each kind, each title and reason cut to the summary
line's cap, and the summary ends with "and N more" when it names fewer
than it counts, so a run that notes many long titles holds a bounded
amount of text.

Co-authored-by: Rohan Kumar <rohnkum@amazon.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/learn-cron-dashboard.md since this head was last judged (5c9bf25 -> 8f478cc); CI re-running.

The rebase was clean and the change itself is identical (same patch id); main's edits to that spec sit in other sections of the file. No code fix was needed: the only red on the previous head was the GPT 6.1 Review check, whose review passes did not complete because the review provider answered with rate-limit errors (HTTP 429), so it published no verdict. It re-runs on this push. On the rebased commit the related backend tests, the type check, lint and the diff's static rules pass, and the description's Proof line now names this head.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/35101-cron-unapproved-run-error branch from 3081d75 to 674724b Compare November 19, 2026 21:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 674724b and checked that the new arm only fires when no tool call ran. Tools that run without a permission request still count as approved, and runs that are security-blocked or partly blocked keep their existing verdicts. Both the single-agent and sequential cron paths use the shared verdict, and neither counter moves, so this cannot auto-pause a recurring job. The reshaped tally has no remaining readers of its old list fields. Each kept title and reason goes through the outbound display sink before it is capped, and the related cron verdict, dedup, placeholder and composition tests pass on this head. I found no code defect, so nothing blocks.

@bolichen97
bolichen97 merged commit 82ee8d2 into main Nov 19, 2026
99 checks passed
@bolichen97
bolichen97 deleted the fix/35101-cron-unapproved-run-error branch November 19, 2026 22:50
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cron: a run whose every tool call was policy-denied, hook-refused or unapproved records ok

2 participants