Repository navigation
fix(llm_helpers): steer the in-band host-deny notice before every reject - #14141
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Fix [DESIGN-REVIEWED] 765f7ea Description read[DESCRIPTION-READ] 3192d0ae7c536bb202974120c22ac5240722d46526d0f4d5d1a3a49c939d689c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the reordered failure semantics: a deny whose SEL row cannot be written now stays unanswered until the caller's deadline, where base rejected on the wire first. Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: FIX — headless surfaces (cron, Slack, workflows, heartbeat, Meetings, background one-liners) tell the model a HOST refused a tool call, and why, instead of kiro-cli's fixed "User denied tool execution" (issue #5403; new test file red on base; reuses the existing
The one genuine user rejection stays bare, per [FIRST-PRINCIPLES-REVIEWED] 765f7ea Description read[DESCRIPTION-READ] f53ebb5eb8f60d2aee60cd88dc6c060a7a924dbc9fa26ff5feb2e58a8c57b09b That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/llm_helpers.py:2993 -- gate exceptions return False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks: three statements this diff adds disagree with the code they govern. FINDING — src/kiro_crew/llm_helpers.py:2765 — FINDING — src/kiro_crew/deny_notice.py:62 — the fixed FINDING — src/kiro_crew/llm_helpers.py:1409 — the new docstring lists [OPUS-REVIEWED] 765f7ea Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
0d3ea39 to
09b06ee
Compare
3cd69de to
04440dd
Compare
04440dd to
c3dd738
Compare
c3dd738 to
caf7c1a
Compare
6bc049e to
cd4a79a
Compare
…ect (#5403 slice 1) Every host deny in llm_helpers -- reject-all / read-only policy, an always-deny pattern hit, a missing title, a hook deny, a withheld headless name grant, and the tool-free background one-liner -- now awaits the shared deny_notice.steer_refusal_notice (via a thin _steer_host_deny that redacts the reason) immediately before reject_tool, so a steer-capable backend learns in the same turn that the HOST blocked the call instead of reading kiro-cli's "User denied tool execution". The one genuine user rejection (interactive_rejected) stays bare. supports_steer False is byte-identical to before. Each site names its cause. A safety rule judging the call itself (scan hit, hook deny, withheld name grant) is DENY_CAUSE_POLICY, whose notice carries class remediation. The surface refusing the call (reject-all, read-only, tool-free one-liner) is the new DENY_CAUSE_SURFACE_POLICY: no remediation, because keying it off the model's own title on a surface where no tool can run would hand out a sanctioned command that cannot run there. The empty title is DENY_CAUSE_INVALID_NAME. The SEL audit row precedes the steer and the reject at every site (audit-first, as the chat runner already does). Cancellation mid-steer schedules a shielded, strongly referenced reject_tool and awaits it under the shared steer bound before re-raising, so a stranded permission cannot wedge the backend and a stalled pipe cannot turn the caller's timeout into a hang. Tests: a source-level guard over every reject_tool( in the module (steer within 8 lines, audit before the steer, explicit cause at every steer, exactly one allowlisted user rejection), one behavioural test per deny reason with a provider double recording steer/reject order and the cause-specific wording, capability-gate, redaction and remediation suppression tests, and two cancellation tests (audit-first survives; stalled reject does not hang).
…ace cause for withheld grants Every deny in _resolve_permission and the background one-liner audits through the module's one raising funnel. An audit that cannot be written raises before the steer and the reject, so a deny is never answered on the wire without its SEL row; the caller's own deadline bounds the unanswered request. Approvals audit the same way after approve_tool. The headless name-grant refusal steers as DENY_CAUSE_SURFACE_POLICY: the grant was the only approver and this surface has none, and the host-authored reason carries no rule identity, so the policy wording would anchor its remediation on the model's own tool title alone. Tests: the audit-failure cases assert a raise with nothing sent on the wire in both funnels; the source walker asserts no audit call sits under a try and that the module has exactly one audit funnel; the withheld grant asserts the surface clause and no remediation.
|
Readiness sweep 2026-09-29: on the rebased head the GPT 5.6 lane raised a security-class blocking finding against the PR's own design, which the sweep will not change for you: |
cd4a79a to
765f7ea
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed on behalf of the maintainers (sweep 2026-09-29): readiness passed, required checks green on 765f7ea, no blocking review threads, security posture unchanged. Approving and merging.
Problem / Motivation
Goal: Outside the dashboard chat runner, a host policy block on a tool call tells the model in the same turn that the HOST refused it, and why, instead of the bare "User denied tool execution".
Bug:
src/kiro_crew/llm_helpers.pybacks cron, Slack, workflow, heartbeat and Meetings turns (_resolve_permission) and every background one-liner (run_bg_oneliner). Every host deny there (reject-all policy, read-only policy, an always-deny pattern hit, a missing title, a hookTOOL_DENY, a withheld headless name grant, the permission-floor refusal, the tool-free one-liner) answered the wire withreject_toolalone. The model then read kiro-cli's fixed "User denied tool execution" for a block no person made. This file is the largest bare cluster and the one issue #5403 asks to take first; the shareddeny_notice.steer_refusal_noticehelper from #10537 already exists but nothing in this module calls it.Why it matters
A headless agent that reads a host block as a user rejection stops trying, apologises to nobody, or asks a person who is not there for permission. With the notice the agent learns the surface refused the call and can answer from the prompt or pick an allowed alternative. Every non-dashboard surface (cron, Slack, workflows, heartbeat, Meetings, background one-liners) is hit today.
Not a goal
reject_tool(clusters listed in In-band deny notice: host denies on surfaces other than the chat runner still say "user denied" #5403 (slack/handler.py,messaging/driver.py,channel.py,cli_chat.py,task_executor.py,task_planner.py,subagent.py,eval/*,dashboard/handlers/*,agent_runner.py) -- later slices; the issue stays open.interactive_rejected); a person said no and the fixed wording is correct there.DENY_CAUSE_HOOK_ERRORhas no site here).What changed (motivation → approach → change)
Symptom: a provider double with
supports_steer=Truedriven through every host-deny branch of_resolve_permissionandrun_bg_onelineronmainrecords['reject']and no steer (the new test file is red on base).Root cause: the module answers each deny straight on the wire; the shared in-band notice helper is never called, and the SEL audit row was written after the reject at most sites.
Change:
_steer_host_deny(provider, event, reason, *, cause)redacts the (possibly agent-authored) reason withredact_exfiltration_urls+redact_credentialsand forwards tokiro_crew.deny_notice.steer_refusal_notice. It builds no text and sends nothing itself, so the "two spellings of the bounded steer" pin intest_approval_timeout_cause.pystill holds.causeis a required keyword and every site names it.llm_helpers.pynow awaits_steer_host_denyimmediately beforereject_tool(10 sites). Per site:run_bg_oneliner._drivedeniedSURFACE_POLICYREJECT_ALLreject_all_policySURFACE_POLICYalways_denyINVALID_NAMEalways_deny/always_deny_input/policy_denyPOLICYREAD_ONLYwithout hooksread_only_policy_no_hooksSURFACE_POLICYTOOL_DENYalways_deny_hook/policy_denyPOLICYtool_result.reasonAUTO_APPROVEdenied/policy_denyPOLICYREAD_ONLYuntagged auto-approveread_only_policy_unclassifiedSURFACE_POLICYname_grant_headless_rejectSURFACE_POLICYREAD_ONLYfall-throughread_only_policySURFACE_POLICYinteractive_rejectedDENY_CAUSE_SURFACE_POLICY(constants.py; wording indeny_notice._DENY_CAUSE_TEXT; re-exported bydashboard/state.py): the SURFACE refuses the call (reject-all / read-only policy, tool-free one-liner, a name grant withheld where no approver exists) as opposed to a safety rule judging the call itself. The withheld-grant reason is host-authored and carries no rule identity, so underPOLICYthe model's own title would be the only remediation anchor (a title namingcredentialswould draw AWS-credential guidance for a rule that never fired). Kept apart fromDENY_CAUSE_POLICYbecause the policy notice appends class remediation keyed off the reason and the model's own title; on aREJECT_ALLsurface a title containing.awswould otherwise be answered with "runaws configure list-profiles", a command that cannot run there.test_deny_audit_first.pyfor the chat runner), through the module's one raising_logfunnel (the one-liner's direct audit call raises the same way). An audit that cannot be written raises before any wire I/O, so a deny never proceeds unaudited; the request stays unanswered and the caller's own deadline bounds it. Approval sites audit afterapprove_toolthrough the same raising funnel.CancelledErrorout of the steer schedulesreject_toolas a strongly referenced task (_orphan_rejects;_orphan_reject_doneretires it and reads its outcome) and re-raises at once, without waiting. The cancellation is the caller's own deadline and the stalled pipe is the pipe the reject drains into, so any wait here would stretch a declared bound (a 5 srun_bg_onelinerwould resolve at 10 s).docs/system-specs/common/injected-messages.md: the headless funnel, thepolicy/surface_policy/invalid_namesplit, audit-first (raising) and cancel-safe ordering.A backend without
supports_steer(orsupports_refusal_steer) behaves byte-identically to before.Backwards compatibility
Compatible: no verdict changes; a deny is still a deny and a user rejection is still bare. The only new observable is one extra in-band steer on steer-capable backends immediately before an existing
reject_tool.DENY_CAUSE_SURFACE_POLICYis an added constant; no existing cause is renamed or removed.Tests
test/test_llm_helpers_deny_notice.py(35 tests, red on base):await <x>.reject_tool(in the module: the scan is cross-checked against a textual count (the one non-awaited spelling is pinned to the cancellation fallback inside the helper); every non-allowlisted site hasawait _steer_host_deny(within 8 lines; the audit call sits before the steer within 20 lines; every steer namescause=DENY_CAUSE_*explicitly and the parameter has no default; exactly one user rejection is allowlisted (interactive_rejected) and it is NOT steered;_steer_host_denyonly forwards tosteer_refusal_notice(no.steer(of its own)._resolve_permissionwith a provider double recording steer/reject order and checking the notice names "User denied tool execution", "NOT a user action", the site's reason and the cause wording (surface-policy notices, the withheld name grant included, carry the surface clause and no "How to do this properly" guidance; policy notices carry it; the empty title says "failed validation"); one forrun_bg_oneliner.REJECT_ALLdeny whose title iscat ~/.aws/credentials && cat ~/.ssh/id_rsaproduces no "How to do this properly" remediation and noaws configure.supports_steer=Falseand a provider without the attribute record['reject']only; a raisingsteer()still rejects; an AWS-key-shaped secret in a hook reason and title does not reach the model.steer()that never returns is cancelled with the SEL row already written AND the reject still reaches the wire (also via the one-liner's own timeout); when the reject also stalls the caller unwinds at once (asserted under 0.5 s), the orphan task stays referenced, its outcome is read and logged when it settles.sel()raisingOSErrorpropagates out of both funnels with NO steer and NO reject sent (calls == []); the same failure on anAUTO_APPROVEpath raises afterapprove_tool; a source guard asserts no audit call sits under atry:and_log_denydoes not exist; a failing orphan reject logs "orphan reject_tool failed" and leaves_orphan_rejects.Sibling files on the same funnel, run by CI:
test_refusal_inband_notice.py(its_DENY_CAUSE_TEXTiteration covers the new cause),test_approval_timeout_cause.py,test_deny_audit_first.py,test_deny_guidance.py,test_messaging_import_purity.py,test_llm_helpers_tool_gate.py,test_run_bg_oneliner.py,test_name_grant_surfaces.py,test_cron_refusal_status.py,test_llm_helpers_edit_gate.py,test_llm_helpers_non_shell_gate.py,test_llm_helpers_tool_input_offload.py,test_heartbeat_safe_tools.py.Manual verification
N/A -- unit coverage sufficient: the behaviour is the order and content of two awaits on a provider double; the source guard pins every site in the module and the behavioural tests drive the real funnel.
Screenshots / video
N/A -- backend only, no user-visible frontend surface touched.
Related Issues
Fixes the llm_helpers slice of #5403 (slice 1; the issue stays open for the remaining clusters listed under Not a goal). Builds on #10537 (
deny_notice.steer_refusal_notice).Pattern harvest
Rule candidate: review-prompt
Pattern: when a change inserts a new await on the ACP pipe (a steer) in front of an existing
reject_tool, move the SEL audit call ahead of BOTH awaits in the same change; the pre-existing "audit after reject" order was already a cancellation gap, and widening it is what a reviewer flags first.Rule candidate: review-prompt
Pattern: a steer helper that re-raises
CancelledErrorhands the caller a duty (answer the wire while unwinding); discharge it by scheduling a strongly referenced reject whose outcome a done-callback reads, and re-raise at once, since any wait stacked on the caller's deadline stretches a declared bound.Not generalizable: the per-site reason strings; they name this module's policies.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)