Repository navigation
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the funnel's audit-before-wire reorder: a raising SEL write at an unwrapped call site now leaves the permission request unanswered. Suggestions
[DESIGN-REVIEWED] 84c70cd Description read[DESCRIPTION-READ] 11db1ea596edcedb8c3d0211fc710609c0a18254b73ff73decefbfeb8b56ae27 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Bare host-deny rejects still exist outside the declared later slices — e.g. the webhook handler at What this change shipsInventory (10 items) — 10 justifiedIntent: a tool call the host refuses on the eval and subagent surfaces must reach the model as a host-deny notice, never kiro-cli's "user denied" — a FIX (red-on-base test shipped; mechanism already documented in injected-messages.md for the sibling surfaces).
[FIRST-PRINCIPLES-REVIEWED] 84c70cd Description read[DESCRIPTION-READ] 1e03f8f8f6a6099f9d95e579e478627d3488d2481e84427ef36355f2a3a93e50 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Approved: PR Readiness passed on this head, no open change requests.
a1bfd72 to
55842ba
Compare
cb59a6a to
10ca950
Compare
…eject Outside the chat runner a rejected permission reaches the model as kiro-cli's fixed "User denied tool execution", so on the eval harness and the subagent surface a host verdict read as a person cancelling. Every host deny on eval/runner.py, eval/judge.py and the subagent funnel now steers the shared deny notice (llm_helpers._steer_host_deny) while the permission request is still unanswered, then rejects; the subagent funnel takes a REQUIRED cause= so each of its nine call sites in subagent_manager/run.py carries a per-site verdict (policy / surface_policy / None for approver rejections and limit bails). Audit rows stay first at every site. Fixes the eval/subagent slice of #5403.
10ca950 to
84c70cd
Compare
|
|
|
|
Thank you @CrysisDeu for this fix! Your branch conflicted with main after readiness passed, so I rebased your commit onto current |
Pull request was closed
Problem / Motivation
Goal: On the eval harness and the subagent surface, a tool call the HOST refuses must reach the model as a host-deny notice, never as "user denied".
A rejected
session/request_permissionreaches the model as kiro-cli's fixed "User denied tool execution" -- the ACP permission response carries onlyoutcome/optionId, so the host has no channel to say more. The dashboard chat runner (#5332/#5401),llm_helpers(#14141), the messaging surfaces (#15083) and the task runner (#15372, pending) steer the real reason into the running turn BEFORE the reject. Three surfaces still did not:eval/runner.py-- 3reject_tool(sites (permission gate refusal, the harness's own filesystem path check, a tool it does not know to be read-only): audit, then a bare reject.eval/judge.py-- 1 site (the judge refuses every tool): bare reject.subagent.py-- the_reject_and_logfunnel called from 9 sites insubagent_manager/run.py: reject FIRST, then the metric and the SEL row, never a notice.Measured on
origin/main@aec81410e:grep -c "_steer_host_deny\|steer_refusal_notice"is 0 in all four files;grep -c "reject_tool("is 3 / 1 / 1 / 0 (run.py denies only through the funnel).Why it matters
An eval scenario or a subagent that hits a policy block reads "the user denied this", apologises for a cancellation nobody made, and abandons or routes around a call that was never judged -- the model's recovery is aimed at the wrong actor. On the unattended subagent surface there is no person to notice the misattribution and correct it, so the whole fan-out inherits it.
Not a goal
dashboard/handlers/optimizer.py,dashboard/handlers/taskrunner.py,apps/builtins/auto_improvement/spine/agent_runner.py) -- later slices.Falsefor both, and the cause of thatFalseis not plumbed here. Both staycause=None(kiro-cli's wording is the truth for the "no"; the timeout arm is a separate change on the callback contract).DENY_CAUSE_POLICY/DENY_CAUSE_SURFACE_POLICYwording fromdeny_notice.py.turn_limit,child_escalation_limit): the run ends at that reject, so there is no continuing turn for a notice to correct.What changed (motivation → approach → change)
Motivation: the sites above hand the model kiro-cli's generic string with nothing to correct it.
Approach: the one established by the merged slices -- call
llm_helpers._steer_host_deny(provider, event, reason, cause=...)immediately BEFORE each host-denyreject_tool, after the SEL row. While the permission request is unanswered the turn is provably in flight, which is what gets the notice queued rather than dropped. Every site gets a per-site verdict; nothing is spot-fixed.Change, per site:
eval/runner.py(inline, audit → steer → reject)refusal_forreturned a reason) --DENY_CAUSE_POLICY, with the gate's own reason (redacted by the helper).DENY_CAUSE_POLICY; the sensitive-path reason uses the fixed "sensitive credential path" wording sodeny_guidancelands it in the same secret-file class as the hook gate's refusal.DENY_CAUSE_SURFACE_POLICY; the reason says the harness runs tools read-only. No remediation, because nothing about the call was judged.eval/judge.py(inline)DENY_CAUSE_SURFACE_POLICY("the eval judge runs no tools"). Steered inside the existingif event.request_id:guard: with no id nothing is answered on the wire, so there is no in-flight refusal to correct.subagent.py--SubagentManager._reject_and_loggains a REQUIRED keyword-onlycause: str | Noneandreason: str = "". Order becomes metric + SEL row → steer (whencause is not None) →reject_tool, matching every other deny surface (the row was written AFTER the wire call before; a backend that stops reading stdin could cancel the coroutine with the decision acted on and unaudited). Two reason constants live here becauserun.py's_implfunctions resolve globals throughbind_component_globals.subagent_manager/run.py-- the nine call sites, each with its verdict:spec_hook)POLICY, reason = the gate's block textTOOL_DENY(hook_deny)POLICY, reason =tool_result.reasonno_policy_deny_default)SURFACE_POLICY,_HEADLESS_DENY_REASON(names every tier that could still authorize:parent_policy=auto,hooks.auto_approve_tools, the read-only classification)child_origin_no_command_context)SURFACE_POLICY,_LOW_FIDELITY_DENY_REASONchild_interactive_rejected)Nonefactory_rejected)NoneNoneturn_limitbailNonechild_escalation_limitbailNonedocs/system-specs/common/injected-messages.md-- a paragraph recording the three surfaces and their per-site verdicts next to the messaging one.Backwards compatibility
SubagentManager._reject_and_logis a private static method; its newcause=keyword is required, so an in-tree caller that forgets it fails at call time rather than inheriting a wrong verdict. The two existing test call sites (test_subagent_coverage.py,test_hang_resilience_metrics.py) are updated withcause=None.supports_refusal_steeris unchanged: the helper probes the capability and returns without sending; the reject always runs. A steer failure is swallowed at debug; the reject always runs.outcome,error,metadata,request_id); the child-denial metric keeps its closed-enumreason. Only their ORDER relative to the wire call moves (now before it).Tests
test/test_eval_subagent_deny_notice.py(new, 46 tests) -- red onorigin/main@aec81410e(33 failed / 13 passed; the 13 are the no-steer and cause-None paths that were already bare), green on this head. Two halves:EvalRunner._run_turn,LLMJudge.judge_turnandSubagentManager._reject_and_logwith a provider double that records steer/approve/reject ORDER: gate refusal, sensitive path, unreadable path, unsafe tool, judge refusal, judge request without an id (neither steers nor rejects), policy / headless / low-fidelity funnel causes,cause=Nonesends nothing, no-steer backend only rejects, a raising steer still rejects, the SEL row (and the child metric) land before the steer, agent-authored text is redacted before it reaches the model.reject_tool(in the two eval modules and every_reject_and_log(call inrun.py, give each a per-site verdict, and fail when a host deny is not steered, a user rejection or a bail IS steered, a steer is not undercause=DENY_CAUSE_*, the SEL row comes after the steer,run.pygrows a wire reject of its own, or the funnel'scausegains a default.Mutation-verified on a scratch checkout: removing the runner's unsafe-tool steer -> 5 fail; changing the headless site to
cause=None-> 1 fail; reject-before-steer in the funnel -> 6 fail; reject-before-steer in the judge -> 4 fail.Sibling files run locally:
test_subagent_coverage.py,test_hang_resilience_metrics.py,test_eval_harness.py-- 342 passed. Static gates (black via the baseline script, isort, flake8, mypy) clean on every changed file. Full suites run in CI.Manual verification
Not applicable: no user-visible surface changes. The notice text itself is the existing
deny_notice.build_refusal_steer_noticeoutput, already verified on the chat runner,llm_helpersand the messaging surfaces.Screenshots / video
Not applicable (backend-only; no UI change).
Related Issues
Fixes the eval/subagent slice of #5403. Follows #14141 (slice 1), #15083 (slice 2) and #15372 (slice 3, task runner). Related: #5332, #5401 (the chat runner's notice).
Pattern harvest
Rule candidate: review-prompt -- a
reject_tool(on any surface must be immediately preceded by_steer_host_deny/steer_refusal_noticeunder an explicit host cause, or by an explicit not-a-host-deny verdict (cause=None/ a comment naming the user rejection or the teardown); a reject funnel must take its cause as a REQUIRED keyword so a new call site cannot inherit a wrong verdict by default.Not generalizable: the
bind_component_globalsindirection that forces the reason constants intosubagent.pyrather than next to their call sites inrun.pyis specific to the subagent manager's component split.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement