Skip to content

fix(llm_helpers): steer the in-band host-deny notice before every reject - #14141

Merged
bolichen97 merged 2 commits into
mainfrom
fix/deny-notice-llm-helpers-5403
Sep 29, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
fix/deny-notice-llm-helpers-5403

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Outside the dashboard chat runner, a host policy block on a tool call tells the model in the same turn that the HOST refused it, and why, instead of the bare "User denied tool execution".

Bug: src/kiro_crew/llm_helpers.py backs cron, Slack, workflow, heartbeat and Meetings turns (_resolve_permission) and every background one-liner (run_bg_oneliner). Every host deny there (reject-all policy, read-only policy, an always-deny pattern hit, a missing title, a hook TOOL_DENY, a withheld headless name grant, the permission-floor refusal, the tool-free one-liner) answered the wire with reject_tool alone. The model then read kiro-cli's fixed "User denied tool execution" for a block no person made. This file is the largest bare cluster and the one issue #5403 asks to take first; the shared deny_notice.steer_refusal_notice helper from #10537 already exists but nothing in this module calls it.

Why it matters

A headless agent that reads a host block as a user rejection stops trying, apologises to nobody, or asks a person who is not there for permission. With the notice the agent learns the surface refused the call and can answer from the prompt or pick an allowed alternative. Every non-dashboard surface (cron, Slack, workflows, heartbeat, Meetings, background one-liners) is hit today.

Not a goal

  • The other reject_tool( clusters listed in In-band deny notice: host denies on surfaces other than the chat runner still say "user denied" #5403 (slack/handler.py, messaging/driver.py, channel.py, cli_chat.py, task_executor.py, task_planner.py, subagent.py, eval/*, dashboard/handlers/*, agent_runner.py) -- later slices; the issue stays open.
  • Steering the one genuine user rejection in this module (interactive_rejected); a person said no and the fixed wording is correct there.
  • Any change to what is approved or denied; only what the model is told about a deny changes.
  • A catch for hook exceptions in this module (there is none today, so DENY_CAUSE_HOOK_ERROR has no site here).

What changed (motivation → approach → change)

Symptom: a provider double with supports_steer=True driven through every host-deny branch of _resolve_permission and run_bg_oneliner on main records ['reject'] and no steer (the new test file is red on base).

Root cause: the module answers each deny straight on the wire; the shared in-band notice helper is never called, and the SEL audit row was written after the reject at most sites.

Change:

  • A thin module-local _steer_host_deny(provider, event, reason, *, cause) redacts the (possibly agent-authored) reason with redact_exfiltration_urls + redact_credentials and forwards to kiro_crew.deny_notice.steer_refusal_notice. It builds no text and sends nothing itself, so the "two spellings of the bounded steer" pin in test_approval_timeout_cause.py still holds. cause is a required keyword and every site names it.
  • Every host-deny site in llm_helpers.py now awaits _steer_host_deny immediately before reject_tool (10 sites). Per site:
site audit reason cause reason steered
run_bg_oneliner._drive denied SURFACE_POLICY tool-free by contract; answer from the prompt
REJECT_ALL reject_all_policy SURFACE_POLICY reject-all tool policy on this surface
empty title always_deny INVALID_NAME "Blocked: missing tool title" (existing text)
scan hit (title / input / edit target / oversize) always_deny / always_deny_input / policy_deny POLICY the scan's own reason
READ_ONLY without hooks read_only_policy_no_hooks SURFACE_POLICY read-only, no hook gate to prove a call read-only
hook TOOL_DENY always_deny_hook / policy_deny POLICY tool_result.reason
permission-floor refusal on AUTO_APPROVE denied / policy_deny POLICY the floor's own reason
READ_ONLY untagged auto-approve read_only_policy_unclassified SURFACE_POLICY read-only, call could not be proven read-only
headless name-grant refusal name_grant_headless_reject SURFACE_POLICY grant withheld and no approver on this surface
READ_ONLY fall-through read_only_policy SURFACE_POLICY read-only, call not classified read-only
interactive callback answered no interactive_rejected -- user rejection, NOT steered
  • New cause DENY_CAUSE_SURFACE_POLICY (constants.py; wording in deny_notice._DENY_CAUSE_TEXT; re-exported by dashboard/state.py): the SURFACE refuses the call (reject-all / read-only policy, tool-free one-liner, a name grant withheld where no approver exists) as opposed to a safety rule judging the call itself. The withheld-grant reason is host-authored and carries no rule identity, so under POLICY the model's own title would be the only remediation anchor (a title naming credentials would draw AWS-credential guidance for a rule that never fired). Kept apart from DENY_CAUSE_POLICY because the policy notice appends class remediation keyed off the reason and the model's own title; on a REJECT_ALL surface a title containing .aws would otherwise be answered with "run aws configure list-profiles", a command that cannot run there.
  • Audit-first: the SEL row is written BEFORE the steer and the reject at every site (matching test_deny_audit_first.py for the chat runner), through the module's one raising _log funnel (the one-liner's direct audit call raises the same way). An audit that cannot be written raises before any wire I/O, so a deny never proceeds unaudited; the request stays unanswered and the caller's own deadline bounds it. Approval sites audit after approve_tool through the same raising funnel.
  • Cancel-safe: a CancelledError out of the steer schedules reject_tool as a strongly referenced task (_orphan_rejects; _orphan_reject_done retires it and reads its outcome) and re-raises at once, without waiting. The cancellation is the caller's own deadline and the stalled pipe is the pipe the reject drains into, so any wait here would stretch a declared bound (a 5 s run_bg_oneliner would resolve at 10 s).
  • docs/system-specs/common/injected-messages.md: the headless funnel, the policy / surface_policy / invalid_name split, audit-first (raising) and cancel-safe ordering.

A backend without supports_steer (or supports_refusal_steer) behaves byte-identically to before.

Backwards compatibility

Compatible: no verdict changes; a deny is still a deny and a user rejection is still bare. The only new observable is one extra in-band steer on steer-capable backends immediately before an existing reject_tool. DENY_CAUSE_SURFACE_POLICY is an added constant; no existing cause is renamed or removed.

Tests

test/test_llm_helpers_deny_notice.py (35 tests, red on base):

  • Source-level guard over every await <x>.reject_tool( in the module: the scan is cross-checked against a textual count (the one non-awaited spelling is pinned to the cancellation fallback inside the helper); every non-allowlisted site has await _steer_host_deny( within 8 lines; the audit call sits before the steer within 20 lines; every steer names cause=DENY_CAUSE_* explicitly and the parameter has no default; exactly one user rejection is allowlisted (interactive_rejected) and it is NOT steered; _steer_host_deny only forwards to steer_refusal_notice (no .steer( of its own).
  • One behavioural test per host-deny reason (including the permission-floor refusal) driving the real _resolve_permission with a provider double recording steer/reject order and checking the notice names "User denied tool execution", "NOT a user action", the site's reason and the cause wording (surface-policy notices, the withheld name grant included, carry the surface clause and no "How to do this properly" guidance; policy notices carry it; the empty title says "failed validation"); one for run_bg_oneliner.
  • A REJECT_ALL deny whose title is cat ~/.aws/credentials && cat ~/.ssh/id_rsa produces no "How to do this properly" remediation and no aws configure.
  • The user rejection gets no notice; supports_steer=False and a provider without the attribute record ['reject'] only; a raising steer() still rejects; an AWS-key-shaped secret in a hook reason and title does not reach the model.
  • Cancellation: a steer() that never returns is cancelled with the SEL row already written AND the reject still reaches the wire (also via the one-liner's own timeout); when the reject also stalls the caller unwinds at once (asserted under 0.5 s), the orphan task stays referenced, its outcome is read and logged when it settles.
  • Audit raises: a sel() raising OSError propagates out of both funnels with NO steer and NO reject sent (calls == []); the same failure on an AUTO_APPROVE path raises after approve_tool; a source guard asserts no audit call sits under a try: and _log_deny does not exist; a failing orphan reject logs "orphan reject_tool failed" and leaves _orphan_rejects.
  • The interactive user rejection audits before the wire too.

Sibling files on the same funnel, run by CI: test_refusal_inband_notice.py (its _DENY_CAUSE_TEXT iteration covers the new cause), test_approval_timeout_cause.py, test_deny_audit_first.py, test_deny_guidance.py, test_messaging_import_purity.py, test_llm_helpers_tool_gate.py, test_run_bg_oneliner.py, test_name_grant_surfaces.py, test_cron_refusal_status.py, test_llm_helpers_edit_gate.py, test_llm_helpers_non_shell_gate.py, test_llm_helpers_tool_input_offload.py, test_heartbeat_safe_tools.py.

Manual verification

N/A -- unit coverage sufficient: the behaviour is the order and content of two awaits on a provider double; the source guard pins every site in the module and the behavioural tests drive the real funnel.

Screenshots / video

N/A -- backend only, no user-visible frontend surface touched.

Related Issues

Fixes the llm_helpers slice of #5403 (slice 1; the issue stays open for the remaining clusters listed under Not a goal). Builds on #10537 (deny_notice.steer_refusal_notice).

Pattern harvest

Rule candidate: review-prompt
Pattern: when a change inserts a new await on the ACP pipe (a steer) in front of an existing reject_tool, move the SEL audit call ahead of BOTH awaits in the same change; the pre-existing "audit after reject" order was already a cancellation gap, and widening it is what a reviewer flags first.
Rule candidate: review-prompt
Pattern: a steer helper that re-raises CancelledError hands the caller a duty (answer the wire while unwinding); discharge it by scheduling a strongly referenced reject whose outcome a done-callback reads, and re-raise at once, since any wait stacked on the caller's deadline stretches a declared bound.
Not generalizable: the per-site reason strings; they name this module's policies.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and buluoray September 26, 2026 06:53
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 765f7eaabc8cae7a85fe163fbf557d0ec9d57030 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Fix _steer_host_deny's docstring before merge: it files the withheld name grant under DENY_CAUSE_POLICY (src/kiro_crew/llm_helpers.py:1408-1410), contradicting the site, spec and tests, which use SURFACE_POLICY.

[DESIGN-REVIEWED] 765f7ea

Description read

[DESCRIPTION-READ] 3192d0ae7c536bb202974120c22ac5240722d46526d0f4d5d1a3a49c939d689c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 765f7eaabc8cae7a85fe163fbf557d0ec9d57030 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the reordered failure semantics: a deny whose SEL row cannot be written now stays unanswered until the caller's deadline, where base rejected on the wire first.

Not justified as shipped

  • Item 4 — rides along: the audit-before-wire reorder is outside the Goal ("tells the model… instead of the bare 'User denied tool execution'"); it is declared, and its ground is readable (an audit sequenced after two pipe awaits never runs on a stalled pipe), but it changes deny failure ordering beyond the notice itself.

What this change ships

Inventory (7 items) — 6 justified

Intent: FIX — headless surfaces (cron, Slack, workflows, heartbeat, Meetings, background one-liners) tell the model a HOST refused a tool call, and why, instead of kiro-cli's fixed "User denied tool execution" (issue #5403; new test file red on base; reuses the existing deny_notice.steer_refusal_notice — 3 prior callers counted).

  1. Every host deny behind cron/Slack/workflow/heartbeat/Meetings turns now names its real refuser and reason to the model before rejecting — justified
  2. The tool-free background one-liner's deny sends the same notice — justified
  3. New deny cause surface_policy whose notice carries no remediation — justified (7 counted call sites; concrete harm named: title-keyed remediation would prescribe commands the surface cannot run)
  4. The SEL audit row now lands before any wire I/O at every deny, and an audit failure raises before the reject — rides along (see above)
  5. Cancellation mid-steer schedules the reject as a kept-alive task so the permission request is still answered — justified (hazard the fix itself introduces)
  6. injected-messages.md gains the headless-funnel section — justified (same-commit spec rule, AGENTS.md)
  7. dashboard/state.py re-exports the new cause — justified (module's documented "re-exports every name" contract, constants.py:1410)

The one genuine user rejection stays bare, per ## Not a goal; no pin is deleted; the remaining reject_tool clusters (counted: 24 other files) are declared deferred under the open issue.

[FIRST-PRINCIPLES-REVIEWED] 765f7ea

Description read

[DESCRIPTION-READ] f53ebb5eb8f60d2aee60cd88dc6c060a7a924dbc9fa26ff5feb2e58a8c57b09b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 765f7eaabc8cae7a85fe163fbf557d0ec9d57030 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/llm_helpers.py:2993 -- gate exceptions return REASON_GATE_UNAVAILABLE, but cause=DENY_CAUSE_POLICY falsely tells the model a safety rule judged the call -> Fix: use DENY_CAUSE_HOOK_ERROR for that reason. (origin: validation)
[GPT-REVIEWED] 765f7ea

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 765f7eaabc8cae7a85fe163fbf557d0ec9d57030: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 765f7eaabc8cae7a85fe163fbf557d0ec9d57030 — this comment is updated in place on each push.

Review details

Nothing blocks: three statements this diff adds disagree with the code they govern.

FINDING — src/kiro_crew/llm_helpers.py:2765 — "The audit raises: a decision whose SEL row cannot be written is not answered on the wire at all" is false for the funnel it governs: sel().log_tool_invocation(...) at 2750 (and _sel().log_tool_invocation( at 1617) omits critical=, so sel.log(critical=False) enqueues to the background writer and, on a full disk or unwritable ~/.kiro/crew/security, drops the event with a warning and returns — the deny is then answered on the wire with no SEL row, the state the comment and the new injected-messages.md:36 sentence both say cannot happen (only a raising sel() factory engages the claimed invariant, which is all test_an_audit_failure_on_a_deny_raises_before_the_wire injects) → Fix: narrow both comments and the spec sentence to the best-effort guarantee critical=False gives, or pass critical=True at 2750 and 1617.

FINDING — src/kiro_crew/deny_notice.py:62 — the fixed DENY_CAUSE_SURFACE_POLICY guidance asserts "the reason above says what this surface permits -- nothing at all, or only calls provably read-only" and "Nothing about the action was judged", but llm_helpers.py:3097 routes the withheld-name-grant deny here — a HOOK_BASED cron/channel turn that permits whatever its hooks allow, whose refusal judged precisely the call (its program resolving through an agent-writable PATH entry) — so the model is told its surface runs no tools, or only read-only ones, and told not to look for a sanctioned form of a call that an absolute-path spelling would pass → Fix: reword the guidance to claim only that the surface refused the call, or give the no-approver case its own cause.

FINDING — src/kiro_crew/llm_helpers.py:1409 — the new docstring lists "a withheld name grant" among the DENY_CAUSE_POLICY sites, but the module's only withheld-name-grant deny sends cause=DENY_CAUSE_SURFACE_POLICY (3097), and constants.py:1413 plus injected-messages.md:22 agree with the code, leaving the docstring the one wrong statement of the taxonomy test_llm_helpers_deny_notice exists to keep honest (nothing round-trips the enumeration) → Fix: move "a withheld name grant" from the DENY_CAUSE_POLICY sentence at 1409 to the DENY_CAUSE_SURFACE_POLICY sentence at 1411.

[OPUS-REVIEWED] 765f7ea

Verdict parsed from the review's SHA-scoped output markers for commit 765f7eaabc8cae7a85fe163fbf557d0ec9d57030.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 765f7eaabc8cae7a85fe163fbf557d0ec9d57030: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-llm-helpers-5403 branch 2 times, most recently from 0d3ea39 to 09b06ee Compare September 26, 2026 08:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-llm-helpers-5403 branch 2 times, most recently from 3cd69de to 04440dd Compare September 26, 2026 17:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-llm-helpers-5403 branch from 04440dd to c3dd738 Compare September 26, 2026 21:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@CrysisDeu CrysisDeu added the no-screenshots PR has no visual delta; screenshot gate exempt label Sep 26, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-llm-helpers-5403 branch from c3dd738 to caf7c1a Compare September 26, 2026 22:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 29, 2026
@bolichen97
bolichen97 force-pushed the fix/deny-notice-llm-helpers-5403 branch from 6bc049e to cd4a79a Compare September 29, 2026 05:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 29, 2026
…ect (#5403 slice 1)

Every host deny in llm_helpers -- reject-all / read-only policy, an
always-deny pattern hit, a missing title, a hook deny, a withheld
headless name grant, and the tool-free background one-liner -- now
awaits the shared deny_notice.steer_refusal_notice (via a thin
_steer_host_deny that redacts the reason) immediately before
reject_tool, so a steer-capable backend learns in the same turn that
the HOST blocked the call instead of reading kiro-cli's "User denied
tool execution". The one genuine user rejection (interactive_rejected)
stays bare. supports_steer False is byte-identical to before.

Each site names its cause. A safety rule judging the call itself (scan
hit, hook deny, withheld name grant) is DENY_CAUSE_POLICY, whose notice
carries class remediation. The surface refusing the call (reject-all,
read-only, tool-free one-liner) is the new DENY_CAUSE_SURFACE_POLICY:
no remediation, because keying it off the model's own title on a
surface where no tool can run would hand out a sanctioned command that
cannot run there. The empty title is DENY_CAUSE_INVALID_NAME.

The SEL audit row precedes the steer and the reject at every site
(audit-first, as the chat runner already does). Cancellation mid-steer
schedules a shielded, strongly referenced reject_tool and awaits it
under the shared steer bound before re-raising, so a stranded
permission cannot wedge the backend and a stalled pipe cannot turn the
caller's timeout into a hang.

Tests: a source-level guard over every reject_tool( in the module
(steer within 8 lines, audit before the steer, explicit cause at every
steer, exactly one allowlisted user rejection), one behavioural test
per deny reason with a provider double recording steer/reject order and
the cause-specific wording, capability-gate, redaction and remediation
suppression tests, and two cancellation tests (audit-first survives;
stalled reject does not hang).
…ace cause for withheld grants

Every deny in _resolve_permission and the background one-liner audits
through the module's one raising funnel. An audit that cannot be written
raises before the steer and the reject, so a deny is never answered on
the wire without its SEL row; the caller's own deadline bounds the
unanswered request. Approvals audit the same way after approve_tool.

The headless name-grant refusal steers as DENY_CAUSE_SURFACE_POLICY: the
grant was the only approver and this surface has none, and the
host-authored reason carries no rule identity, so the policy wording
would anchor its remediation on the model's own tool title alone.

Tests: the audit-failure cases assert a raise with nothing sent on the
wire in both funnels; the source walker asserts no audit call sits under
a try and that the module has exactly one audit funnel; the withheld
grant asserts the surface clause and no remediation.
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 29, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Readiness sweep 2026-09-29: on the rebased head the GPT 5.6 lane raised a security-class blocking finding against the PR's own design, which the sweep will not change for you: llm_helpers.py ~1625 — audit-write failures are contained at both deny funnels ("an audit that cannot be written must not strand the request") and the lane wants them to propagate instead; plus an advisory at ~3121 (DENY_CAUSE_POLICY vs DENY_CAUSE_SURFACE_POLICY when only a headless name grant was withheld). Everything else is green on cd4a79a (34/34 deny-notice tests, the 11th reject site from main's permission floor now steers + audits). @CrysisDeu your call: propagate, or record the disposition; ping and the sweep will re-check.

@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-llm-helpers-5403 branch from cd4a79a to 765f7ea Compare September 29, 2026 07:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 29, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on behalf of the maintainers (sweep 2026-09-29): readiness passed, required checks green on 765f7ea, no blocking review threads, security posture unchanged. Approving and merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-screenshots PR has no visual delta; screenshot gate exempt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants