Skip to content

fix(cron): report a security-gate refusal when other calls ran - #11573

Merged
bolichen97 merged 1 commit into
mainfrom
fix/cron-partial-deny-verdict
Sep 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/cron-partial-deny-verdict

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A cron run whose tool calls were partly refused by the always_deny_input security gate ended with last_status = ok and no notification. The refused write was silently lost while the model's prose still claimed the work was done.

_GateTally.all_blocked required approved == 0 and unresolved == 0; anything else fell through to job.record_success(). So "3 calls approved, 1 refused" read exactly like a clean run.

Ticket: SIM-T P502167825.

Already fixed on main (not redone here)

What this PR changes

One mechanism in src/kiro_crew/slack/gateway.py:

  • _GateTally.partially_blocked: at least one security refusal while the run was not a total block.
  • _GateTally.refusal_summary(): one redacted, capped line naming up to three refused tools, shared by last_error and the delivered banner so the cron row and the notification cannot disagree. It goes through the shared display_safe sink, so an LLM-authored tool title such as <!channel> cannot page a channel when the run reports it (render_for_slack redacts but does not defang mentions).
  • _annotate_partial_block(): prefixes the delivered result with ⛔ <summary> — that work did not happen. so the refusal lands beside the prose on every delivery leg (dashboard, channel, Slack) and in the dedup hash.
  • _apply_gate_verdict() now has three outcomes:
    • all blocked: error, record_failure() (unchanged).
    • partially blocked: last_status = "error", last_error = refusal_summary(), counted in neither direction. The approved calls prove the job can work, so it must not march toward auto-pause; the lost call means it did not succeed, so it must not read as ok or reset a failure streak.
    • no refusal: record_success() (unchanged).

Both cron agent paths share the verdict helper, so they cannot drift.

Not in this PR (follow-up)

Field-scoping the body scan for MCP write tools via a per-tool allow list (llm_helpers.py _extract_tool_input_strings) was considered and left out: it needs a per-tool schema contract for third-party MCP servers, which is more than a small mechanism. Tracked as follow-up under the same ticket.

Pattern harvest

Rule candidate: review-prompt
Pattern: an allowed/denied tally reduced to a two-way all_denied / success verdict; denied > 0 and allowed > 0 needs its own outcome.

Tests

test/test_cron_refusal_status.py (31 tests, all passing with -n 0):

  • partial block: last_status == "error", last_error and the delivered body name the refused tool, consecutive_failures unchanged, no auto-pause.
  • all approved: ok, unchanged.
  • all blocked: unchanged failure path and auto-pause after the threshold.
  • <!channel> as a refused title is defanged on both last_error and the delivered body.
  • multi-agent sequence tallies across the whole run.

Docs: docs/system-specs/modules/learn-cron-dashboard.md describes the three verdicts. .github/black-baseline.txt: the test file is now formatted and removed from the baseline.

Local review

  • claude-opus-4.8 lane: 0 findings.
  • gpt-5.6-sol lane: 1 blocking finding (live <!channel> in the banner) — fixed via display_safe, pinned by a test.

sim: https://t.corp.amazon.com/P502167825

@bolichen97
bolichen97 requested review from a team and pepmach September 17, 2026 19:03
@bolichen97
bolichen97 enabled auto-merge (squash) September 17, 2026 19:04
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the intended escalation story: a job whose one write is refused every run now errors forever, banner each delivery, never reaching auto-pause.

[DESIGN-REVIEWED] 6dcb9e4

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Blocked-then-adapted jobs — the old pin's "normal, healthy path" — now read error every run; verify that permanent red row is intended for jobs that recover.

What this change ships

Inventory (7 items) — 6 justified

Intent: make a cron run that lost some tool calls to the security gate say so, instead of reporting a clean ok — a FIX.

  1. A partially blocked run's cron row now shows error naming the refused calls (was silent ok) — justified
  2. Delivered result gains a ⛔ banner naming the refusal on every leg and in the dedup hash — justified
  3. A partially blocked run no longer resets a failure streak or failure-alert dedup — justified
  4. All-blocked reason line now shares the defanged refusal_summary() spelling — justified
  5. A blocked+unresolved run (no approvals) now also reports error — justified
  6. Spec's verdict section rewritten to the three-arm table, same commit — justified
  7. Test file removed from the black baseline — rides along

[FIRST-PRINCIPLES-REVIEWED] 6dcb9e4

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/slack/gateway.py:938 -- id-less approved calls are tallied twice, so "of {total} tool call(s)" reports a false denominator -> Fix: omit the denominator.
[GPT-REVIEWED] 6dcb9e4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6dcb9e4

Verdict parsed from the review's SHA-scoped output markers for commit 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6dcb9e47ae6c67bf3630235f16bf9a2016fb2f08: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

CI status: the remaining red shards (Windows 5/7, Ubuntu 5/6) are the 6 test_session_control.py / test_member_session_control.py failures introduced on main by #11438; they fail identically on a clean origin/main and are not touched by this diff. The fix is open as #11593. Once it merges I will rebase onto main and re-push; no reruns until then.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
A cron run whose tool calls were mostly approved and one refused by the
always-enforced deny gate recorded last_status ok, notified nobody, and
left the refusal only in the SEL audit row. The model's prose is its own
account of the run, so a refused final document write still read as
done and the document was silently lost.

_GateTally gains a partial-block arm: at least one call security-blocked
while the run was not a total block. Such a run keeps last_status error
with a reason naming the refused call(s) and how many of the run's calls
they were, spends nothing from the auto-pause failure budget (the
approved calls prove the job can work), and does not record a success
(a lost call is not a clean run, and a success would reset a failure
streak and the failure-alert dedup). The same reason is prefixed to the
run's result before it is stored and hashed, so the dashboard bell and
slot, the channel leg and Slack all name the refusal beside the prose.
One spelling of the reason serves both the row and the delivery.

The tally does not separate the gate's title tier from its input tier;
either refusal means the work behind the call did not happen.

sim: https://t.corp.amazon.com/P502167825
@bolichen97
bolichen97 force-pushed the fix/cron-partial-deny-verdict branch from 07e36bb to 6dcb9e4 Compare September 17, 2026 22:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: a cron run where only SOME tool calls were security-refused recorded as a success, so the lost work never surfaced -- adds partially_blocked plus a shared refusal_summary() so the cron row and the delivered result name the refused call, and neither the failure nor the success counter moves. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/learn-cron-dashboard.md.

@bolichen97
bolichen97 merged commit 0d6675c into main Sep 18, 2026
86 checks passed
@bolichen97
bolichen97 deleted the fix/cron-partial-deny-verdict branch September 18, 2026 01:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…otdev#11573)

A cron run whose tool calls were mostly approved and one refused by the
always-enforced deny gate recorded last_status ok, notified nobody, and
left the refusal only in the SEL audit row. The model's prose is its own
account of the run, so a refused final document write still read as
done and the document was silently lost.

_GateTally gains a partial-block arm: at least one call security-blocked
while the run was not a total block. Such a run keeps last_status error
with a reason naming the refused call(s) and how many of the run's calls
they were, spends nothing from the auto-pause failure budget (the
approved calls prove the job can work), and does not record a success
(a lost call is not a clean run, and a success would reset a failure
streak and the failure-alert dedup). The same reason is prefixed to the
run's result before it is stored and hashed, so the dashboard bell and
slot, the channel leg and Slack all name the refusal beside the prose.
One spelling of the reason serves both the row and the delivery.

The tally does not separate the gate's title tier from its input tier;
either refusal means the work behind the call did not happen.

sim: https://t.corp.amazon.com/P502167825
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants