Repository navigation
fix(cron): record error when no tool call of a cron run was approved - #37604
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix is aimed at its cause, the Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: a user whose cron job's tool calls were all refused by policy, by a spec hook or for lack of approval sees the run as failed instead of
[FIRST-PRINCIPLES-REVIEWED] 674724b Description read[DESCRIPTION-READ] 9b35d8f096d3fe9861bec831c132d643965395aacecaf25961016d484dbb5179 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this reuses the existing Partial arm's "error, neither counter moves" verdict, and nothing outside [DESIGN-REVIEWED] 674724b Description read[DESCRIPTION-READ] b0fce6bcb7a76f5f16d7f8a76171e73daf52cc5ed10d6d9aeac0745cdd797df9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Adds a fourth verdict arm, none_approved: no call approved, none security-blocked, at least one refused another way. Such a run records last_status=error, names up to three refused calls with their refusal text, and moves neither the failure nor the success counter. Verified: tools that ran without a permission request are still reported as approved (turn_stream.py:690-696), so the arm only fires when nothing executed. The scheduler's _execute skips record_success() on error, so a failure streak is kept. Both the single-agent and sequential cron paths share the annotate/verdict functions. auto_pause_failed_one_shot keeps and pauses delete_after_run one-shots, as the description and spec say. The design matches the existing fire-time governance-deny handling. The spec doc table is updated, the tests fail without the change, and no existing test depends on the old ok behaviour for an all-unapproved run. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
f586517 to
9001be7
Compare
9001be7 to
862fa58
Compare
|
Rebased: main moved in gateway_runtime/cron_verdict.py, gateway_runtime/runtime_charge.py, test/test_cron_refusal_status.py, test/test_llm_helpers_batch_guard_deferral.py and docs/system-specs/modules/learn-cron-dashboard.md since this head went green (d155058 -> cba2110); CI re-running. The previous head failed the two tests main added in [operator: bolichen97] |
862fa58 to
3081d75
Compare
|
Rebased: main moved in docs/system-specs/modules/learn-cron-dashboard.md since this head went green (cba2110 -> 5c9bf25); CI re-running. The code patch is unchanged (same patch-id). The previous head's reds were all main's: [operator: bolichen97] |
A cron run in which no tool call was approved and none was blocked by the security gate recorded last_status ok, even when every call was refused by a governance profile, by the agent spec's PreToolUse hook, or by a rejected or timed-out approval. _GateTally kept only a count of those calls, and _apply_gate_verdict called record_success() for them. _GateTally now keeps the first three such calls' titles and first reason lines, and a new none_approved property marks a run with no approved call, no security block and at least one other refusal. _apply_gate_verdict records that run as error without moving either counter, so a delete_after_run one-shot is kept and auto-paused. refusal_summary() names the kept calls with their refusal text, or "not approved", and _annotate_partial_block puts the same reason in front of the delivered result. The reason does not use the partial-block wording, so the Schedule page shows the run as Error. The tally counts every refused and unresolved call but keeps only the three it names of each kind, each title and reason cut to the summary line's cap, and the summary ends with "and N more" when it names fewer than it counts, so a run that notes many long titles holds a bounded amount of text. Co-authored-by: Rohan Kumar <rohnkum@amazon.com>
|
Rebased: main moved in docs/system-specs/modules/learn-cron-dashboard.md since this head was last judged (5c9bf25 -> 8f478cc); CI re-running. The rebase was clean and the change itself is identical (same patch id); main's edits to that spec sit in other sections of the file. No code fix was needed: the only red on the previous head was the GPT 6.1 Review check, whose review passes did not complete because the review provider answered with rate-limit errors (HTTP 429), so it published no verdict. It re-runs on this push. On the rebased commit the related backend tests, the type check, lint and the diff's static rules pass, and the description's Proof line now names this head. [operator: bolichen97] |
3081d75 to
674724b
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head 674724b and checked that the new arm only fires when no tool call ran. Tools that run without a permission request still count as approved, and runs that are security-blocked or partly blocked keep their existing verdicts. Both the single-agent and sequential cron paths use the shared verdict, and neither counter moves, so this cannot auto-pause a recurring job. The reshaped tally has no remaining readers of its old list fields. Each kept title and reason goes through the outbound display sink before it is capped, and the related cron verdict, dedup, placeholder and composition tests pass on this head. I found no code defect, so nothing blocks.
Problem / Motivation
Goal: a cron run in which no tool call ran, because every call was refused by a governance profile, by the agent spec's own PreToolUse hook, or left unapproved, shows as Error and names those calls, instead of reading
ok.A cron agent run records
last_status: okwhen no tool call was approved, none was blocked by the security gate, and at least one was refused another way: a governance denial (policy_deny), the spec hook's refusal (spec_hook), or an approval that was rejected or timed out. The Schedule page then shows a run that did nothing as a success.Why it matters
In #34927 a daily job ran an agent whose spec carries PreToolUse hooks, under a profile that turns hooks off. Every tool call was refused and each run still showed
ok, so nothing told the user the job had stopped working.Not a goal
ok.on_tool_gatecallback shape.What changed (motivation → approach → change)
A run whose tool calls were all refused, none of them by the security gate, now reads Error and says which calls were refused.
last_statusbeforelast_statusnowokerror, the reason names up to three calls, thenand N moreThe cause was the run's tally in
cron_verdict.py: it kept a refusal's title and reason only for a security block, and kept every other refusal as a bare count, so the verdict inruntime_charge.pyhad nothing to report and called the run a success. The tally now keeps such calls too, and the verdict records the run aserrorwithout counting a failure or a success. For every kind of refused call the tally keeps the exact count but only the first three by name, each title and reason redacted and then cut to the 500-character line cap, so a run that notes many long titles holds a bounded amount of text; the reason ends withand N morewhen it names fewer calls than it counts. The reason readsnone of N tool call(s) ran, all refused or not approved: <title> (<refusal text>), withnot approvedfor an approval that recorded no text. It is the same line inlast_errorand in front of the delivered result, and it does not use the Partial wording, so the Schedule page shows Error. Adelete_after_runone-shot is kept and paused, as any run ending inerroris. The gate-verdict table inlearn-cron-dashboard.mdgains this arm.Backwards compatibility
Compatible: the
on_tool_gateshape, the job's fields and the run record are unchanged; a run in which no tool call ran, which readok, now readserror, and a reason that names three of more than three refused calls ends withand N more.Removes nothing:
_GateTallyis private tocron_verdict.py, andslack/gateway.pyonly re-exports it; it folds the parallelrefused_ruleslist intorefused_calls, makesrefuseda count, and stops storing call names past the three its summary shows, which nothing read. The two test files that read its fields,test_cron_refusal_status.pyandtest_llm_helpers_batch_guard_deferral.py, assert the count and the kept names.Tests
Proof: PROVEN test/test_cron_refusal_status.py::test_an_unapproved_run_is_not_a_failure, test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[unapproved], test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[policy-deny], test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[spec-hook], test/test_cron_refusal_status.py::TestUnapprovedRun::test_a_run_with_no_approved_call_records_error[mixed] (head 674724b)
Determinism: the two adapted
test_llm_helpers_batch_guard_deferral.pytests passed 20 of 20 runs at-n0,test_cron_refusal_status.py10 of 10 at-n 4 --dist loadgroup --max-worker-restart=2, and both files once in a shuffled order (seed 35101, 54 of 54). With the source changes reverted,prove.pyfinds 14 of the new and changed tests failing, the two adapted batch-guard tests among them.test_an_unapproved_run_is_not_a_failure: a timed-out approval run recordserrorand still never auto-pauses after five runs.TestUnapprovedRun::test_a_run_with_no_approved_call_records_error: an unapproved, apolicy_deny, aspec_hookand a mixed run each recorderrorwith no failure counted.TestUnapprovedRun::test_it_spends_no_failure_budgetandtest_it_does_not_reset_a_failure_streak: five such runs leave the job enabled and unpaused, and one after two blocked runs keepsconsecutive_failuresat 2.TestUnapprovedRun::test_the_reason_names_each_call_with_its_refusal_textandtest_the_reason_names_at_most_three: each call is named with the first line of its reason ornot approved, at most three, within 500 characters.TestUnapprovedRun::test_the_schedule_page_shows_it_as_error_not_partial: the reason does not match the Partial pattern inbackendPhrases.json.TestUnapprovedRun::test_the_notification_names_the_refused_callsandtest_it_reaches_the_sequential_path: the delivered body opens with the same reason aslast_erroron the single-agent and theagent_sequencepaths.TestUnapprovedRun::test_one_approved_call_keeps_the_run_okandtest_a_tool_free_run_stays_ok: the unchanged arms stayok.TestSchedulerContract::test_an_unapproved_one_shot_is_kept_and_auto_paused: adelete_after_runone-shot with only a policy denial is kept, paused and disabled, with no failure counted.test_the_tally_counts_every_call_but_keeps_three_bounded_names(security-blocked and unresolved): ten calls with 100,000-character titles and reasons leave an exact count of 10, three kept names, each field within 500 characters, and a reason within 500 characters that ends withand 7 more.test_llm_helpers_batch_guard_deferral.py::test_a_deferral_retried_next_step_reports_no_security_blockandtest_a_transport_floor_refusal_is_still_a_security_block: a batch-guard deferral retried on the next step leaves a security-block count of 0 and no kept name, and a transport-floor refusal counts 1 and keeps its title.Manual verification
Run on Linux (x86_64), from the PR's commit and from its base, in an isolated gateway (throwaway data home, ephemeral port on 127.0.0.1) with the packaged fake agent backend and crons on. A cron agent job created through
POST /api/cronswith a[[GATED]]message makes the fake raise a real permission request forfake demo tool; the job was fired once throughPOST /api/crons/<id>/run. Without a profile the unattended cron approved that call, so a governance profile bound to thecronsurface that allows onlyreadwas written into the isolated home. The audit then recorded the call asdenied, mechanismpolicy_deny.last_statusbase → fixlast_errorbase → fixconsecutive_failuresbase → fixpolicy_deny)ok→errorNone→none of 1 tool call(s) ran, all refused or not approved: fake demo tool (Blocked by governance policy: …)ok→errorNone→none of 3 tool call(s) ran, …: Write /vault/a.md (Denied by governance profile: no writes), Bash ls /vault (Refused by the agent spec's PreToolUse hook), Read /vault/b.md (not approved)[[TOOL]])ok→okNone→Noneok→okNone→Noneerror→errorall 1 tool call(s) blocked by the security gate: …bothok→okNone→Noneauto_pausedstayed false in every case. On the fix the gateway run'slast_resultand its notification start withBlocked:and the same reason, and its history row readsfailurewith that reason. The gateway case was re-run on this head with the same result, and its reason carries noand N more.The bounded tally, against a tally that kept every call:
, and 97 moreScreenshots / video
The Schedule page for the same job shape and refusal, without the fix and on this head:
Related Issues
Closes #35101
Builds on #35337 by @rohnkum.
Split out of #34927. #28603 also touches how a cron run names a lost call; whichever lands second folds onto the other.
Sibling sweep
Swept: git grep -n -e '_GateTally' -e '_apply_gate_verdict' -e 'refusal_summary' -e '_annotate_partial_block' -e 'auto_pause_failed_one_shot' -- src website/src
Sibling: src/kiro_crew/slack/gateway_runtime/cron_verdict.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/runtime_charge.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/cron_turn.py -- not affected -- both cron agent paths call the shared
_annotate_partial_blockand_apply_gate_verdict, so they take the new arm unchangedSibling: src/kiro_crew/slack/gateway_runtime/cron_failure.py -- not affected -- it alerts after a counted failure and the new arm counts none
Sibling: src/kiro_crew/cron_service/execution.py -- not affected --
auto_pause_failed_one_shotalready keeps and pauses adelete_after_runrun whose status iserrorSibling: src/kiro_crew/cron_service/service/run_execution.py -- not affected -- it calls
auto_pause_failed_one_shotwith the run'slast_status, so the newerrorarm is kept and paused like any otherSibling: src/kiro_crew/cron_service/settlement.py -- not affected -- a cancelled or reaped run's settlement applies the same
auto_pause_failed_one_shotruleSibling: src/kiro_crew/slack/gateway.py -- not affected -- it re-exports
_GateTally,_annotate_partial_blockand_apply_gate_verdictunchangedSibling: website/src/utils/cronPartlyBlocked.ts -- not affected -- the new reason does not match its Partial pattern, so the run shows as Error
Pattern harvest
Not generalizable: the gap is one missing arm in a single verdict function whose arms are all listed in one spec table.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)