Repository navigation
fix(cron): report a security-gate refusal when other calls ran - #11573
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the intended escalation story: a job whose one write is refused every run now errors forever, banner each delivery, never reaching auto-pause. [DESIGN-REVIEWED] 6dcb9e4 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Blocked-then-adapted jobs — the old pin's "normal, healthy path" — now read What this change shipsInventory (7 items) — 6 justifiedIntent: make a cron run that lost some tool calls to the security gate say so, instead of reporting a clean
[FIRST-PRINCIPLES-REVIEWED] 6dcb9e4 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/slack/gateway.py:938 -- id-less approved calls are tallied twice, so False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
CI status: the remaining red shards (Windows 5/7, Ubuntu 5/6) are the 6 |
A cron run whose tool calls were mostly approved and one refused by the always-enforced deny gate recorded last_status ok, notified nobody, and left the refusal only in the SEL audit row. The model's prose is its own account of the run, so a refused final document write still read as done and the document was silently lost. _GateTally gains a partial-block arm: at least one call security-blocked while the run was not a total block. Such a run keeps last_status error with a reason naming the refused call(s) and how many of the run's calls they were, spends nothing from the auto-pause failure budget (the approved calls prove the job can work), and does not record a success (a lost call is not a clean run, and a success would reset a failure streak and the failure-alert dedup). The same reason is prefixed to the run's result before it is stored and hashed, so the dashboard bell and slot, the channel leg and Slack all name the refusal beside the prose. One spelling of the reason serves both the row and the delivery. The tally does not separate the gate's title tier from its input tier; either refusal means the work behind the call did not happen. sim: https://t.corp.amazon.com/P502167825
07e36bb to
6dcb9e4
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: a cron run where only SOME tool calls were security-refused recorded as a success, so the lost work never surfaced -- adds partially_blocked plus a shared refusal_summary() so the cron row and the delivered result name the refused call, and neither the failure nor the success counter moves. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/learn-cron-dashboard.md.
…otdev#11573) A cron run whose tool calls were mostly approved and one refused by the always-enforced deny gate recorded last_status ok, notified nobody, and left the refusal only in the SEL audit row. The model's prose is its own account of the run, so a refused final document write still read as done and the document was silently lost. _GateTally gains a partial-block arm: at least one call security-blocked while the run was not a total block. Such a run keeps last_status error with a reason naming the refused call(s) and how many of the run's calls they were, spends nothing from the auto-pause failure budget (the approved calls prove the job can work), and does not record a success (a lost call is not a clean run, and a success would reset a failure streak and the failure-alert dedup). The same reason is prefixed to the run's result before it is stored and hashed, so the dashboard bell and slot, the channel leg and Slack all name the refusal beside the prose. One spelling of the reason serves both the row and the delivery. The tally does not separate the gate's title tier from its input tier; either refusal means the work behind the call did not happen. sim: https://t.corp.amazon.com/P502167825
Summary
A cron run whose tool calls were partly refused by the
always_deny_inputsecurity gate ended withlast_status = okand no notification. The refused write was silently lost while the model's prose still claimed the work was done._GateTally.all_blockedrequiredapproved == 0 and unresolved == 0; anything else fell through tojob.record_success(). So "3 calls approved, 1 refused" read exactly like a clean run.Ticket: SIM-T P502167825.
Already fixed on main (not redone here)
What this PR changes
One mechanism in
src/kiro_crew/slack/gateway.py:_GateTally.partially_blocked: at least one security refusal while the run was not a total block._GateTally.refusal_summary(): one redacted, capped line naming up to three refused tools, shared bylast_errorand the delivered banner so the cron row and the notification cannot disagree. It goes through the shareddisplay_safesink, so an LLM-authored tool title such as<!channel>cannot page a channel when the run reports it (render_for_slackredacts but does not defang mentions)._annotate_partial_block(): prefixes the delivered result with⛔ <summary> — that work did not happen.so the refusal lands beside the prose on every delivery leg (dashboard, channel, Slack) and in the dedup hash._apply_gate_verdict()now has three outcomes:error,record_failure()(unchanged).last_status = "error",last_error = refusal_summary(), counted in neither direction. The approved calls prove the job can work, so it must not march toward auto-pause; the lost call means it did not succeed, so it must not read asokor reset a failure streak.record_success()(unchanged).Both cron agent paths share the verdict helper, so they cannot drift.
Not in this PR (follow-up)
Field-scoping the body scan for MCP write tools via a per-tool allow list (
llm_helpers.py_extract_tool_input_strings) was considered and left out: it needs a per-tool schema contract for third-party MCP servers, which is more than a small mechanism. Tracked as follow-up under the same ticket.Pattern harvest
Rule candidate: review-prompt
Pattern: an allowed/denied tally reduced to a two-way
all_denied/successverdict;denied > 0 and allowed > 0needs its own outcome.Tests
test/test_cron_refusal_status.py(31 tests, all passing with-n 0):last_status == "error",last_errorand the delivered body name the refused tool,consecutive_failuresunchanged, no auto-pause.ok, unchanged.<!channel>as a refused title is defanged on bothlast_errorand the delivered body.Docs:
docs/system-specs/modules/learn-cron-dashboard.mddescribes the three verdicts..github/black-baseline.txt: the test file is now formatted and removed from the baseline.Local review
<!channel>in the banner) — fixed viadisplay_safe, pinned by a test.sim: https://t.corp.amazon.com/P502167825