Skip to content

fix(eval-subagent): steer the in-band host-deny notice before every reject - #15463

Closed
CrysisDeu wants to merge 1 commit into
mainfrom
fix/deny-notice-eval-subagent-5403
Closed

CrysisDeu wants to merge 1 commit into
mainfrom
fix/deny-notice-eval-subagent-5403

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On the eval harness and the subagent surface, a tool call the HOST refuses must reach the model as a host-deny notice, never as "user denied".

A rejected session/request_permission reaches the model as kiro-cli's fixed "User denied tool execution" -- the ACP permission response carries only outcome/optionId, so the host has no channel to say more. The dashboard chat runner (#5332/#5401), llm_helpers (#14141), the messaging surfaces (#15083) and the task runner (#15372, pending) steer the real reason into the running turn BEFORE the reject. Three surfaces still did not:

  • eval/runner.py -- 3 reject_tool( sites (permission gate refusal, the harness's own filesystem path check, a tool it does not know to be read-only): audit, then a bare reject.
  • eval/judge.py -- 1 site (the judge refuses every tool): bare reject.
  • subagent.py -- the _reject_and_log funnel called from 9 sites in subagent_manager/run.py: reject FIRST, then the metric and the SEL row, never a notice.

Measured on origin/main @ aec81410e: grep -c "_steer_host_deny\|steer_refusal_notice" is 0 in all four files; grep -c "reject_tool(" is 3 / 1 / 1 / 0 (run.py denies only through the funnel).

Why it matters

An eval scenario or a subagent that hits a policy block reads "the user denied this", apologises for a cancellation nobody made, and abandons or routes around a call that was never judged -- the model's recovery is aimed at the wrong actor. On the unattended subagent surface there is no person to notice the misattribution and correct it, so the whole fan-out inherits it.

Not a goal

  • The remaining surfaces of In-band deny notice: host denies on surfaces other than the chat runner still say "user denied" #5403 (dashboard/handlers/optimizer.py, dashboard/handlers/taskrunner.py, apps/builtins/auto_improvement/spine/agent_runner.py) -- later slices.
  • Distinguishing an approver's timeout from an approver's "no" on the subagent surface: the factory / gateway callbacks return a bare False for both, and the cause of that False is not plumbed here. Both stay cause=None (kiro-cli's wording is the truth for the "no"; the timeout arm is a separate change on the callback contract).
  • A new notice text or a new cause: every site uses the existing DENY_CAUSE_POLICY / DENY_CAUSE_SURFACE_POLICY wording from deny_notice.py.
  • Steering on a run bail (turn_limit, child_escalation_limit): the run ends at that reject, so there is no continuing turn for a notice to correct.

What changed (motivation → approach → change)

Motivation: the sites above hand the model kiro-cli's generic string with nothing to correct it.

Approach: the one established by the merged slices -- call llm_helpers._steer_host_deny(provider, event, reason, cause=...) immediately BEFORE each host-deny reject_tool, after the SEL row. While the permission request is unanswered the turn is provably in flight, which is what gets the notice queued rather than dropped. Every site gets a per-site verdict; nothing is spot-fixed.

Change, per site:

eval/runner.py (inline, audit → steer → reject)

  • permission gate refusal (refusal_for returned a reason) -- DENY_CAUSE_POLICY, with the gate's own reason (redacted by the helper).
  • filesystem tool on a sensitive path / no readable path -- DENY_CAUSE_POLICY; the sensitive-path reason uses the fixed "sensitive credential path" wording so deny_guidance lands it in the same secret-file class as the hook gate's refusal.
  • tool not on the read-only allowlist -- DENY_CAUSE_SURFACE_POLICY; the reason says the harness runs tools read-only. No remediation, because nothing about the call was judged.

eval/judge.py (inline)

  • every tool call -- DENY_CAUSE_SURFACE_POLICY ("the eval judge runs no tools"). Steered inside the existing if event.request_id: guard: with no id nothing is answered on the wire, so there is no in-flight refusal to correct.

subagent.py -- SubagentManager._reject_and_log gains a REQUIRED keyword-only cause: str | None and reason: str = "". Order becomes metric + SEL row → steer (when cause is not None) → reject_tool, matching every other deny surface (the row was written AFTER the wire call before; a backend that stops reading stdin could cancel the coroutine with the decision acted on and unaudited). Two reason constants live here because run.py's _impl functions resolve globals through bind_component_globals.

subagent_manager/run.py -- the nine call sites, each with its verdict:

site verdict cause
spec PreToolUse gate blocked (spec_hook) host POLICY, reason = the gate's block text
stored hooks TOOL_DENY (hook_deny) host POLICY, reason = tool_result.reason
headless deny-by-default (no_policy_deny_default) host SURFACE_POLICY, _HEADLESS_DENY_REASON (names every tier that could still authorize: parent_policy=auto, hooks.auto_approve_tools, the read-only classification)
low-fidelity child request, no approver (child_origin_no_command_context) host SURFACE_POLICY, _LOW_FIDELITY_DENY_REASON
child approver said no (child_interactive_rejected) user None
per-subagent factory said no (factory_rejected) user None
gateway fallback said no user None
turn_limit bail teardown None
child_escalation_limit bail teardown None

docs/system-specs/common/injected-messages.md -- a paragraph recording the three surfaces and their per-site verdicts next to the messaging one.

Backwards compatibility

  • SubagentManager._reject_and_log is a private static method; its new cause= keyword is required, so an in-tree caller that forgets it fails at call time rather than inheriting a wrong verdict. The two existing test call sites (test_subagent_coverage.py, test_hang_resilience_metrics.py) are updated with cause=None.
  • Behaviour on a backend without supports_refusal_steer is unchanged: the helper probes the capability and returns without sending; the reject always runs. A steer failure is swallowed at debug; the reject always runs.
  • SEL rows keep their exact shape (outcome, error, metadata, request_id); the child-denial metric keeps its closed-enum reason. Only their ORDER relative to the wire call moves (now before it).
  • No config, wire format or public API change.

Tests

test/test_eval_subagent_deny_notice.py (new, 46 tests) -- red on origin/main @ aec81410e (33 failed / 13 passed; the 13 are the no-steer and cause-None paths that were already bare), green on this head. Two halves:

  • Behavioural, one per deny reason, driving the real EvalRunner._run_turn, LLMJudge.judge_turn and SubagentManager._reject_and_log with a provider double that records steer/approve/reject ORDER: gate refusal, sensitive path, unreadable path, unsafe tool, judge refusal, judge request without an id (neither steers nor rejects), policy / headless / low-fidelity funnel causes, cause=None sends nothing, no-steer backend only rejects, a raising steer still rejects, the SEL row (and the child metric) land before the steer, agent-authored text is redacted before it reaches the model.
  • Source-level guards that enumerate every reject_tool( in the two eval modules and every _reject_and_log( call in run.py, give each a per-site verdict, and fail when a host deny is not steered, a user rejection or a bail IS steered, a steer is not under cause=DENY_CAUSE_*, the SEL row comes after the steer, run.py grows a wire reject of its own, or the funnel's cause gains a default.

Mutation-verified on a scratch checkout: removing the runner's unsafe-tool steer -> 5 fail; changing the headless site to cause=None -> 1 fail; reject-before-steer in the funnel -> 6 fail; reject-before-steer in the judge -> 4 fail.

Sibling files run locally: test_subagent_coverage.py, test_hang_resilience_metrics.py, test_eval_harness.py -- 342 passed. Static gates (black via the baseline script, isort, flake8, mypy) clean on every changed file. Full suites run in CI.

Manual verification

Not applicable: no user-visible surface changes. The notice text itself is the existing deny_notice.build_refusal_steer_notice output, already verified on the chat runner, llm_helpers and the messaging surfaces.

Screenshots / video

Not applicable (backend-only; no UI change).

Related Issues

Fixes the eval/subagent slice of #5403. Follows #14141 (slice 1), #15083 (slice 2) and #15372 (slice 3, task runner). Related: #5332, #5401 (the chat runner's notice).

Pattern harvest

Rule candidate: review-prompt -- a reject_tool( on any surface must be immediately preceded by _steer_host_deny / steer_refusal_notice under an explicit host cause, or by an explicit not-a-host-deny verdict (cause=None / a comment naming the user rejection or the teardown); a reject funnel must take its cause as a REQUIRED keyword so a new call site cannot inherit a wrong verdict by default.
Not generalizable: the bind_component_globals indirection that forces the reason constants into subagent.py rather than next to their call sites in run.py is specific to the subagent manager's component split.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@CrysisDeu
CrysisDeu requested a review from a team as a code owner September 30, 2026 15:54
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 84c70cd659b176cba254319793fc33f257fb793a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the funnel's audit-before-wire reorder: a raising SEL write at an unwrapped call site now leaves the permission request unanswered.

Suggestions

[DESIGN-REVIEWED] 84c70cd

Description read

[DESCRIPTION-READ] 11db1ea596edcedb8c3d0211fc710609c0a18254b73ff73decefbfeb8b56ae27

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 84c70cd659b176cba254319793fc33f257fb793a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 84c70cd

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 84c70cd659b176cba254319793fc33f257fb793a: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 84c70cd659b176cba254319793fc33f257fb793a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Bare host-deny rejects still exist outside the declared later slices — e.g. the webhook handler at dashboard/handlers/hooks.py:1396 — confirm those are tracked.

What this change ships

Inventory (10 items) — 10 justified

Intent: a tool call the host refuses on the eval and subagent surfaces must reach the model as a host-deny notice, never kiro-cli's "user denied" — a FIX (red-on-base test shipped; mechanism already documented in injected-messages.md for the sibling surfaces).

  1. Eval gate refusals now tell the model the host refused, with the gate's reason — justified
  2. Eval sensitive-path / unreadable-path refusals carry the policy notice keyed to the secret-file remediation class — justified
  3. Eval's not-read-only refusal says the harness runs tools read-only — justified
  4. Eval judge refusals say the judge runs no tools — justified
  5. Subagent host denies (spec gate, hook deny, headless default, low-fidelity child) steer the real reason — justified
  6. Subagent user rejections and limit bails stay bare, by explicit cause=None — justified
  7. The funnel's cause= is required, so a new caller must write a verdict (9 counted callers in run.py) — justified
  8. SEL row and child metric now land before the wire reject — justified (documented audit-first invariant, llm_helpers.py:2814)
  9. Spec paragraph recording the surfaces and verdicts — justified (same-commit doc rule)
  10. Source-level guards making any future deny site in these modules declare a per-site verdict — justified

[FIRST-PRINCIPLES-REVIEWED] 84c70cd

Description read

[DESCRIPTION-READ] 1e03f8f8f6a6099f9d95e579e478627d3488d2481e84427ef36355f2a3a93e50

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@CrysisDeu CrysisDeu changed the title fix(eval,subagent): steer the in-band host-deny notice before every reject fix(eval-subagent): steer the in-band host-deny notice before every reject Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 84c70cd659b176cba254319793fc33f257fb793a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 84c70cd

Verdict parsed from the review's SHA-scoped output markers for commit 84c70cd659b176cba254319793fc33f257fb793a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 84c70cd659b176cba254319793fc33f257fb793a: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 30, 2026
bolichen97
bolichen97 previously approved these changes Oct 1, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: PR Readiness passed on this head, no open change requests.

@bolichen97
bolichen97 enabled auto-merge (squash) October 1, 2026 05:59
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 1, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-eval-subagent-5403 branch from a1bfd72 to 55842ba Compare October 1, 2026 07:37
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision labels Oct 1, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-eval-subagent-5403 branch 2 times, most recently from cb59a6a to 10ca950 Compare October 1, 2026 11:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
…eject

Outside the chat runner a rejected permission reaches the model as kiro-cli's
fixed "User denied tool execution", so on the eval harness and the subagent
surface a host verdict read as a person cancelling. Every host deny on
eval/runner.py, eval/judge.py and the subagent funnel now steers the shared
deny notice (llm_helpers._steer_host_deny) while the permission request is
still unanswered, then rejects; the subagent funnel takes a REQUIRED cause=
so each of its nine call sites in subagent_manager/run.py carries a per-site
verdict (policy / surface_policy / None for approver rejections and limit
bails). Audit rows stay first at every site.

Fixes the eval/subagent slice of #5403.
@CrysisDeu
CrysisDeu force-pushed the fix/deny-notice-eval-subagent-5403 branch from 10ca950 to 84c70cd Compare October 1, 2026 12:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Verify the funnel's audit-before-wire reorder: a raising SEL write at an unwrapped call site now leaves the permission request unanswered — verified on this head, accepted as designed; no code change.

SubagentManager._reject_and_log (subagent.py:3739) writes the SEL row first, then steers, then calls reject_tool, and does not wrap the write. That is the documented audit-first invariant every merged deny surface already holds (llm_helpers.py:2814, the Slack/channel/TurnDriver sites from #14141 and #15083, the handlers funnel in #15505): a reject the ledger cannot record is not sent, because an unaudited answer on the wire is the failure mode the ledger exists to rule out. The funnel's docstring names this order and why. An SEL write that raises here raises exactly where it raised before this PR -- the reorder moved the steer behind the audit, not the audit behind anything -- so no call site that previously answered the wire now leaves it unanswered.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • The new test's docstring cites test_taskrunner_deny_notice.py, which does not exist in this tree — accepted; deferred to the merge order, no push on this head.

Measured: test/test_eval_subagent_deny_notice.py:12 names test_taskrunner_deny_notice.py, which ships in the sibling slice PR #15372 (green, awaiting merge) and is not on main yet. The reference is a doc pointer only -- nothing imports or depends on that file -- so it cannot fail a test or a gate. The slices land in order (#15372 before this PR), at which point the citation is accurate; if the order flips, the one-word retarget to test_messaging_deny_notice.py (already on main) rides the next push that this branch needs for some other reason. It does not justify spending a CI round on this all-green head.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Bare host-deny rejects still exist outside the declared later slices — e.g. the webhook handler at dashboard/handlers/hooks.py:1396 — confirm those are tracked — confirmed tracked; out of this slice's scope, no code change.

This PR is slice 4 of issue #5403 (eval/* and subagent*), which enumerated its sites; hooks.py was never in the issue's list. The final slice, PR #15505, records the remaining un-enumerated reject_tool( sites by name and count under its "Not a goal" and "Related Issues" sections -- dashboard/handlers/hooks.py (2, one an audit_write_failed arm), cli_chat.py (7), acp/client.py (6), acp/session_handle.py (5), acp/session_provider.py (1), session_background.py (1), messaging/auto_title.py (1), review_pool.py (1) -- as follow-up material, and issue #8817 ("Audit reject_tool answer sites outside chat_runner.py for the same generic-attribution class") holds the audit. Adding hooks.py here would widen a slice whose test enumerates exactly the two eval modules and run.py.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @CrysisDeu for this fix! Your branch conflicted with main after readiness passed, so I rebased your commit onto current main and resolved the conflicts in a replacement PR: #16357 (your commit is preserved as author, so a rebase-and-merge lands it under your name, with a Co-authored-by trailer for the small audit-resilience follow-up a review bot requested). All checks are green there. Closing this one in favor of #16357.

@iamwhatever iamwhatever closed this Oct 3, 2026
auto-merge was automatically disabled October 3, 2026 05:08

Pull request was closed

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 3, 2026
@bolichen97
bolichen97 deleted the fix/deny-notice-eval-subagent-5403 branch November 6, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants