Repository navigation
fix(eval-subagent): steer the in-band host-deny notice before every reject - #16357
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check one change the visible description doesn't mention: [DESIGN-REVIEWED] 2601024 Description read[DESCRIPTION-READ] 22f57fc18a8005459975b1ba8dd404b7149466381b49cb4f3f126e9d2e3f02d6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The fix is complete on its own; reordering the subagent audit ahead of the wire reject relies on symmetry and makes SEL audit failures best-effort without saying so. Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: on eval and subagent runs, the model should be told when the host refused a tool call, not that a user denied it. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 2601024 Description read[DESCRIPTION-READ] 1ae406c2b90b099769c3dd7b668b180d2a2c8bf37d39e33a7861fe1ca6e85cda That sha256 names the PR description this verdict read. Recompute it with |
4e6b3c1 to
77a36d3
Compare
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
…eject Outside the chat runner a rejected permission reaches the model as kiro-cli's fixed "User denied tool execution", so on the eval harness and the subagent surface a host verdict read as a person cancelling. Every host deny on eval/runner.py, eval/judge.py and the subagent funnel now steers the shared deny notice (llm_helpers._steer_host_deny) while the permission request is still unanswered, then rejects; the subagent funnel takes a REQUIRED cause= so each of its nine call sites in subagent_manager/run.py carries a per-site verdict (policy / surface_policy / None for approver rejections and limit bails). Audit rows stay first at every site. The subagent funnel's SEL audit is best-effort: a failure to write it (an unloadable SEL trust root is permanent per process) no longer skips the steer and the reject, which would leave the permission request unanswered and hang the turn. Answering the model is the critical path. Fixes the eval/subagent slice of #5403. Co-authored-by: iamwhatever <12429733+iamwhatever@users.noreply.github.com>
77a36d3 to
2601024
Compare
buluoray
left a comment
There was a problem hiding this comment.
Summary
Head 2601024f7585d2cfffa2a2ffefeb94ff96ea2793. This adds the in-band host-deny notice (llm_helpers._steer_host_deny) before each host reject_tool on three surfaces: the eval runner (3 sites), the eval judge (1 site), and the subagent reject funnel SubagentManager._reject_and_log (9 call sites in subagent_manager/run.py, now with a required cause=). No path in the diff can turn a deny into an allow. The diff adds no approve_tool call, and every touched branch still ends in reject_tool. A failure on the new steer path still answers the request with a reject:
- Steer
Exceptionis swallowed insidesteer_refusal_notice(deny_notice.py). The reject then runs. - Steer
CancelledErrorschedules an orphanreject_toolbefore it re-raises (llm_helpers.py:1544-1548). - An SEL write failure in the funnel is caught, and the code still steers and rejects (
subagent.py:4093-4112). emit_counternever raises (metrics/events.py:22-29).
Every new error path fails closed.
Verified
- Read
docs/system-specs/modules/security.mdfirst. Nothing in it constrains this path beyond the existing rule that deny decisions are audited and stay denies. Both still hold. - Read every changed site at the head SHA:
eval/runner.py:510-617: gate refusal, then path check (sensitive / no path), then unsafe tool.eval/judge.py:104-113: the steer sits inside the existingif event.request_id:guard.subagent.py:4033-4112: the funnel order is metric → SEL (best-effort) → steer ifcause is not None→ reject.run.py: each of the 9 verdicts matches the PR's table. Spec gate and hook deny →POLICY. Headless and low-fidelity →SURFACE_POLICY. The three approver "no" answers and the two limit bails →None.
- The orphan reject in
_steer_host_denyusesevent.request_id, and the funnel rejectsrequest_id. Every caller passesevent.request_idfor both, so the orphan answers the same request. test/test_eval_subagent_deny_notice.py: 47 passed.test_subagent_coverage.py+test_hang_resilience_metrics.py: 275 passed.- Mutations. Each one was restored from a backup, and
git status --shortwas clean afterwards:- Funnel steer removed → 7 fail, including
test_funnel_policy_cause_steers_the_reason_before_rejectingandTestSubagentDeniesThroughOneSteeringFunnel::test_the_funnel_audits_then_steers_then_rejects. - Funnel SEL
try/exceptneutered so the raise propagates →test_funnel_failing_audit_still_steers_and_rejectsfails. - Runner unsafe-tool steer removed → 5 fail (
test_unsafe_tool_steers_the_surface_cause_before_rejecting, …). - Runner gate steer removed → 4 fail (
test_permission_gate_refusal_steers_the_gates_reason_before_rejecting,test_eval_gate_reason_is_redacted_before_it_reaches_the_model, …). - Judge steer removed → 4 fail (
test_judge_refusal_steers_the_surface_cause_before_rejecting, …). - Headless site, and separately the low-fidelity site, set to
cause=None→test_every_host_deny_names_a_deny_cause_and_a_reasonfails in each case. - Factory rejection given
cause=DENY_CAUSE_POLICY→test_user_rejections_and_bails_are_not_steeredfails.
- Funnel steer removed → 7 fail, including
- Owning spec:
docs/system-specs/common/injected-messages.mdgained the per-surface paragraph, in the same place as the earlier slices. - CI at the head SHA: 82 success, 12 skipped, 0 failure. Auto-merge is off. No human review yet (REVIEW_REQUIRED).
- Tests were not run locally beyond the three targeted files above. The full suite was left to CI.
Bot claims disposition
- GPT 6.1 / Opus 5.5 line review: no findings. Consistent with what I read.
- Design Review (PASS): says the change the description doesn't mention is that
_reject_and_lognow swallows SEL write failures. Real, and it does not block. The swallow is on a path whose decision is already a deny, so audit loss cannot weaken enforcement. With the audit now first, the swallow is also what stops a raising SEL from leaving the permission request unanswered and hanging the turn. It is logged atexceptionlevel. The only defect is that the PR body doesn't declare it (see follow-ups). - First Principles (CONCERNS), items 4/5 (reordering is "symmetry only"; the swallow is undeclared): partly real, and it does not block. The bot is right about one narrow point: a cancellation during the steer happens before anything is acted on, so the steer itself doesn't need audit-first. Even so, the reorder is defensible beyond symmetry, because
reject_toolis also an await on the same pipe. Under the old order, a cancellation after the reject's write left the decision acted on and unaudited. It also matches every other merged deny funnel (task_executor._reject_and_log, chat runner). Item 5 follows necessarily from item 4: without the swallow, an SEL raise would skip the reject. Both fail closed, so neither is a logic defect. The PR body should declare the behaviour change.
Follow-ups (non-blocking)
- PR body / squash message: state that a subagent SEL audit-write failure is now logged and swallowed rather than raised. Name the trade-off: the deny is still sent, the audit row is lost but logged, and the request is no longer stranded. This keeps the undeclared change out of main's permanent history.
- PR body says "46 tests"; the file has 47 at this head.
- Backwards-compatibility bullet "Only their ORDER relative to the wire call moves" is no longer complete, because the SEL raise semantics changed too (see 1).
Problem / Motivation
Goal: On the eval harness and the subagent surface, a tool call the HOST refuses must reach the model as a host-deny notice, never as "user denied".
A rejected
session/request_permissionreaches the model as kiro-cli's fixed "User denied tool execution" -- the ACP permission response carries onlyoutcome/optionId, so the host has no channel to say more. The dashboard chat runner (#5332/#5401),llm_helpers(#14141), the messaging surfaces (#15083) and the task runner (#15372, pending) steer the real reason into the running turn BEFORE the reject. Three surfaces still did not:eval/runner.py-- 3reject_tool(sites (permission gate refusal, the harness's own filesystem path check, a tool it does not know to be read-only): audit, then a bare reject.eval/judge.py-- 1 site (the judge refuses every tool): bare reject.subagent.py-- the_reject_and_logfunnel called from 9 sites insubagent_manager/run.py: reject FIRST, then the metric and the SEL row, never a notice.Measured on
origin/main@aec81410e:grep -c "_steer_host_deny\|steer_refusal_notice"is 0 in all four files;grep -c "reject_tool("is 3 / 1 / 1 / 0 (run.py denies only through the funnel).Why it matters
An eval scenario or a subagent that hits a policy block reads "the user denied this", apologises for a cancellation nobody made, and abandons or routes around a call that was never judged -- the model's recovery is aimed at the wrong actor. On the unattended subagent surface there is no person to notice the misattribution and correct it, so the whole fan-out inherits it.
Not a goal
dashboard/handlers/optimizer.py,dashboard/handlers/taskrunner.py,apps/builtins/auto_improvement/spine/agent_runner.py) -- later slices.Falsefor both, and the cause of thatFalseis not plumbed here. Both staycause=None(kiro-cli's wording is the truth for the "no"; the timeout arm is a separate change on the callback contract).DENY_CAUSE_POLICY/DENY_CAUSE_SURFACE_POLICYwording fromdeny_notice.py.turn_limit,child_escalation_limit): the run ends at that reject, so there is no continuing turn for a notice to correct.What changed (motivation → approach → change)
Motivation: the sites above hand the model kiro-cli's generic string with nothing to correct it.
Approach: the one established by the merged slices -- call
llm_helpers._steer_host_deny(provider, event, reason, cause=...)immediately BEFORE each host-denyreject_tool, after the SEL row. While the permission request is unanswered the turn is provably in flight, which is what gets the notice queued rather than dropped. Every site gets a per-site verdict; nothing is spot-fixed.Change, per site:
eval/runner.py(inline, audit → steer → reject)refusal_forreturned a reason) --DENY_CAUSE_POLICY, with the gate's own reason (redacted by the helper).DENY_CAUSE_POLICY; the sensitive-path reason uses the fixed "sensitive credential path" wording sodeny_guidancelands it in the same secret-file class as the hook gate's refusal.DENY_CAUSE_SURFACE_POLICY; the reason says the harness runs tools read-only. No remediation, because nothing about the call was judged.eval/judge.py(inline)DENY_CAUSE_SURFACE_POLICY("the eval judge runs no tools"). Steered inside the existingif event.request_id:guard: with no id nothing is answered on the wire, so there is no in-flight refusal to correct.subagent.py--SubagentManager._reject_and_loggains a REQUIRED keyword-onlycause: str | Noneandreason: str = "". Order becomes metric + SEL row → steer (whencause is not None) →reject_tool, matching every other deny surface (the row was written AFTER the wire call before; a backend that stops reading stdin could cancel the coroutine with the decision acted on and unaudited). Two reason constants live here becauserun.py's_implfunctions resolve globals throughbind_component_globals.subagent_manager/run.py-- the nine call sites, each with its verdict:spec_hook)POLICY, reason = the gate's block textTOOL_DENY(hook_deny)POLICY, reason =tool_result.reasonno_policy_deny_default)SURFACE_POLICY,_HEADLESS_DENY_REASON(names every tier that could still authorize:parent_policy=auto,hooks.auto_approve_tools, the read-only classification)child_origin_no_command_context)SURFACE_POLICY,_LOW_FIDELITY_DENY_REASONchild_interactive_rejected)Nonefactory_rejected)NoneNoneturn_limitbailNonechild_escalation_limitbailNonedocs/system-specs/common/injected-messages.md-- a paragraph recording the three surfaces and their per-site verdicts next to the messaging one.Backwards compatibility
SubagentManager._reject_and_logis a private static method; its newcause=keyword is required, so an in-tree caller that forgets it fails at call time rather than inheriting a wrong verdict. The two existing test call sites (test_subagent_coverage.py,test_hang_resilience_metrics.py) are updated withcause=None.supports_refusal_steeris unchanged: the helper probes the capability and returns without sending; the reject always runs. A steer failure is swallowed at debug; the reject always runs.outcome,error,metadata,request_id); the child-denial metric keeps its closed-enumreason. Only their ORDER relative to the wire call moves (now before it).Tests
test/test_eval_subagent_deny_notice.py(new, 46 tests) -- red onorigin/main@aec81410e(33 failed / 13 passed; the 13 are the no-steer and cause-None paths that were already bare), green on this head. Two halves:EvalRunner._run_turn,LLMJudge.judge_turnandSubagentManager._reject_and_logwith a provider double that records steer/approve/reject ORDER: gate refusal, sensitive path, unreadable path, unsafe tool, judge refusal, judge request without an id (neither steers nor rejects), policy / headless / low-fidelity funnel causes,cause=Nonesends nothing, no-steer backend only rejects, a raising steer still rejects, the SEL row (and the child metric) land before the steer, agent-authored text is redacted before it reaches the model.reject_tool(in the two eval modules and every_reject_and_log(call inrun.py, give each a per-site verdict, and fail when a host deny is not steered, a user rejection or a bail IS steered, a steer is not undercause=DENY_CAUSE_*, the SEL row comes after the steer,run.pygrows a wire reject of its own, or the funnel'scausegains a default.Mutation-verified on a scratch checkout: removing the runner's unsafe-tool steer -> 5 fail; changing the headless site to
cause=None-> 1 fail; reject-before-steer in the funnel -> 6 fail; reject-before-steer in the judge -> 4 fail.Sibling files run locally:
test_subagent_coverage.py,test_hang_resilience_metrics.py,test_eval_harness.py-- 342 passed. Static gates (black via the baseline script, isort, flake8, mypy) clean on every changed file. Full suites run in CI.Manual verification
Not applicable: no user-visible surface changes. The notice text itself is the existing
deny_notice.build_refusal_steer_noticeoutput, already verified on the chat runner,llm_helpersand the messaging surfaces.Screenshots / video
Not applicable (backend-only; no UI change).
Related Issues
Fixes the eval/subagent slice of #5403. Follows #14141 (slice 1), #15083 (slice 2) and #15372 (slice 3, task runner). Related: #5332, #5401 (the chat runner's notice).
Pattern harvest
Rule candidate: review-prompt -- a
reject_tool(on any surface must be immediately preceded by_steer_host_deny/steer_refusal_noticeunder an explicit host cause, or by an explicit not-a-host-deny verdict (cause=None/ a comment naming the user rejection or the teardown); a reject funnel must take its cause as a REQUIRED keyword so a new call site cannot inherit a wrong verdict by default.Not generalizable: the
bind_component_globalsindirection that forces the reason constants intosubagent.pyrather than next to their call sites inrun.pyis specific to the subagent manager's component split.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement