Repository navigation
refactor(hooks): run the tool gate from a tier table - #17192
Conversation
Sensitive Change ReviewSensitive files changed:
|
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, confirm [DESIGN-REVIEWED] 695a2e9 Description read[DESCRIPTION-READ] ed9cea2af4956e19bb5c7f7fb9f2ccf0bc5e7d7444b3ff3ea620a7eddf3a1893 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The load-time soundness check is a second copy of the Not justified as shipped
What this change shipsInventory (9 items) — 7 justifiedIntent: make the tool gate's tier order one table that the gate, the denial differential and the tests all read, with no decision changed. This is a refactor, not a fix or an addition.
[FIRST-PRINCIPLES-REVIEWED] 695a2e9 Description read[DESCRIPTION-READ] 3d674f60e0ed98f5f818eb6ee2a6205eb2aaed840b4efd608f5c6063b1884ca8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Security Scope Review -- 🟡 CONCERNSWhich legitimate operations does Verdict basis: the reviewer's own judgement; no confirmed regression. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: CONCERNS TIGHTENING: The tool gate's composition is moved, not made stricter.
The three ADJUDICATED: 18 candidates, 0 confirmed newly-refused PLATFORM COVERAGE:
The gate code in this diff makes no new path, separator, interpreter or environment assumption. The only new path literal is UNADJUDICATED:
[SCOPE-REVIEWED] 695a2e9 |
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
Security Scope Review (CONCERNS, advisory): the inputs it would have submitted, run at the gate level. Each was judged through
The open risk does not hold as a change. The broader evidence is in the PR body: a 60,000-scenario BASE/HEAD differential that matches byte for byte through |
|
Design Review (PASS): the differential was run against this exact head. At
The merge ref adds one main commit, The harness is not committed. It is a campaign evidence script that imports the product from either tree, and it is not a repository test. The suite carries the interface tests and mutation proofs instead. I can post the harness in a comment if a reviewer wants to rerun it. |
|
For the Sensitive Change Review approval. The author's evidence for head |
|
Reviewed for merge triage, not approving from my side: ~2.8k added / ~1.2k removed lines moving the hook tool gate to a tier table, and the Sensitive Change Review check is red on this head. The tool gate is a security boundary that needs a security owner's sign-off. Auto-merge is already on, so a single approval merges it; please make sure a maintainer reviews it directly. CI is green. |
|
Heads-up: this branch went |
The tool gate's decision order lived only as the statement order of a 785-line HookManager.on_tool_call, guarded by tests that read its source and by two hand-copied tier tables. It is now data: hooks.GATE_TIERS lists the gate's rows (every deny tier, then the policy deny, then the grants, then the read-only classifier), HookManager.judge(call, ...) walks it, and each row's body lives in the composed owner hook_runtime/gate_tiers.py. ToolCall, in hook_runtime/tool_identity.py, is the gate's typed input; hook_gate_kwargs is its keyword form and on_tool_call a shim over judge with the same signature. Verdicts carry the deciding row in ToolHookResult.tier. scripts/deny_diff.py drops its literal table and reads the gate's SHELL_DENY_TIERS with ast.literal_eval from the tree the harness runs from; verify_fix.py keeps its copy, pinned equal. The source-reading gate pins become behavioural tests in test/test_gate_tiers.py and their own files. Every decision is unchanged: a recorded 60,000-scenario BASE/HEAD differential is byte-identical. Part of #16536
383ea46 to
695a2e9
Compare
|
Reviewed again at the new head, still not approving (crew-pr-reviewer). It rewrites the tool approval gate (+2886/-1203), and |
|
self-added: yes
|
|
self-added: yes
|
|
Security Scope Review on Each call went through
On measured-set provenance: the differential measures
The rows themselves are pinned by the order golden. So a narrowing must also delete a gate check, and that shows in the diff a reviewer reads. The broader evidence is in the PR body. Three 60,000-scenario BASE/head differentials match byte for byte through |
|
For the Sensitive Change Review approval of the CURRENT head |
iamwhatever
left a comment
There was a problem hiding this comment.
Sensitive change review
- What rule changed: none. The tool gate's decision order moved from the body of HookManager.on_tool_call into the GATE_TIERS table that HookManager.judge walks; on_tool_call is a shim with the same signature. This head is rebased onto main 9bf943b, and the denied-command handling main added for a shell tool reported under a non-execute kind (#16920) is carried into the table unchanged: the tool's own command/cmd argument meets the same shell checks and deny catalog, in the same order, never the path check.
- Before -> after (what was allowed/blocked, now): unchanged against main 9bf943b. Three 60,000-scenario base/head differentials (verdict, reason, flags, counter emits, log lines, governance arguments, context read count; the third gives most calls a command/cmd argument) are byte-identical through on_tool_call and judge. An independent review probe of 258,720 main-vs-head inputs matched on every field but the new tier stamp, with 0 security-event-log differences over 64,680 inputs.
- Worst case if wrong (breaks commands / leaks secret / locks user out): a misordered table would let a grant re-admit a call a deny tier refused. Pinned by test/test_gate_tiers.py (order golden, soundness check with self-tests, every deny under every grant, one scenario per reachable branch) and by the fail-closed soundness check at import.
- Evidence checked (which sandbox result proves it): Denial Differential green on macos-latest, ubuntu-latest and windows-latest at 695a2e9; Security Scope adjudication 0 newly refused on all three, and its unrun gate-level inputs judged identical at base and head (comment 6029728710).
- No regression:
- Tools still work (which tools/commands were run, result): E2E (stub ACP backend), E2E Gateway Boot (ubuntu, windows), Integration (in-process gateway): green at 695a2e9.
- Backward compatible (old config / old data / old callers still work, how checked): on_tool_call keeps its signature and verdicts and every caller is untouched; hook_gate_kwargs returns the same dict; deny_diff's CLI, exit codes and report are unchanged.
- Existing tests (which suites passed on this head): Backend Tests 3.12 x8, Windows x8, macOS x4, IPv6, namespace sandbox, crew container, Coverage Gate, CodeQL, SAST: green at 695a2e9; main's own test_hooks_raw_command_deny.py passes unmodified.
- How to undo: revert the single commit.
Problem / Motivation
Goal: Make the tool gate's decision order one table that every reader — the gate itself, the denial differential and the tests — takes from the same place, while every tool-call decision stays exactly what it is today.
HookManager.on_tool_callwas a 785-line method. Its tier order is a security invariant: every deny tier must run before every grant, or a grant re-admits what a deny blocked. That order lived only as the sequence of statements in the method. It was kept true by ten tests that read the method's source text, plus two hand-copied tier tables inscripts/deny_diff.pyand the security-conductor'sverify_fix.py. Adding a check meant editing the method, both copies and the source-reading pins.Why it matters
Not a goal
on_tool_callwith the same keywords. Moving them tojudgeis later work (W2-10 / PR-2b).tool_permission.pyengine, desk or settle work. Those belong to W2-10 and W2-12.What changed (motivation → approach → change)
The gate is now data plus one loop.
HookManager.judge(call, *, session_key, agent, app, classifier_only)walkshooks.GATE_TIERSand returns the first verdict a row hands back, stamped in the newToolHookResult.tier.on_tool_callis now a thin keyword shim over it with the same signature.ToolCallhook_runtime/tool_identity.pyfrom_eventis the one event-field extraction;hook_gate_kwargs(event) == ToolCall.from_event(event).kwargs()GATE_TIERS,GateTier,GateRule,SHELL_DENY_TIERShooks.pytargetsholds three per-target rules run target-major; with no recovered command and no MCP server, the tool's owncommand/cmdargument meets its two shell rules.SHELL_DENY_TIERSis the table's shell projection, written as a literalGateFactshook_runtime/gate_tiers.py(joins the existing compose)GateFactsderives each value on first use, in today's order, from one platform-context snapshot; the effective deny set is resolved twice from that snapshot, as at BASEgate_tier_problemsTwo boundaries were kept on purpose, and security review should check them:
deny_diffreads the table from the harness tree. It parseshooks.pywithast.literal_evalfrom the tree the script itself runs from. It never reads a tree it classifies, never imports the product, and reads lazily insidedifferential/classify, never at import. In the scope lanes the harness is the base or default-branch checkout, so the change under judgement cannot shrink the composite it is measured by. That is exactly where the deleted_TIERSliteral lived.verify_fix.pykeeps its literal (Master ruling). It runs as a synced skill file with no product tree beside it. Its only tree is the fixer's worktree, so reading the table there would let the fix being judged choose its own gate. A behavioural three-way equality replaces its source-regex pin.Docs follow the code in the same commit: the two owning specs (
memory-skills-hooks.md, the "Hook runtime owners" section and a new tier table; the hook-gate rows ofsecurity.md), plus statements the move made false, edited under the campaign Master's entry-scoped grants:docs/request-for-change/rfc-session-tag-change-event.md,rfc-crew-agent-sdk-boundary.mdhooks.py:<line>cites become symbols; their orphaned.github/docs-lint-baseline.txtrows are pruned (shrink only)docs/ci/denial-differential.md, golden-paths headerhooks.SHELL_DENY_TIERS(three checks).github/workflows/denial-differential.ymlgovernance.md,hook_runtime/__init__.pydocstringDeletion test.
gate_tiers.pyand the eleven tier bodies fall back into one method. The order returns to prose, guarded by source pins.GATE_TIERSand the order again lives in no single place.deny_diffandverify_fixwould need hand mirrors again.ToolCalland the gate's input becomes 15 loose keywords again, re-threaded at each of the 16 callers.Backwards compatibility
Compatible:
on_tool_callkeeps its exact signature and verdicts, and every caller, patch and fake still targets it.hook_gate_kwargsreturns the same dict.deny_diff's CLI, exit codes and report are unchanged.ToolHookResultgains a defaulted field that takes no part in equality.Reader: test/test_deny_diff.py:test_the_measured_checks_are_the_checks_the_tool_gate_applies -- entry: release -- read the deleted
deny_diff._TIERS; now readsload_shell_deny_tiersReader: test/test_security_conductor_verify_fix.py:test_the_tiers_are_the_gate_s_and_the_differential_s -- entry: release -- parsed
_TIERSout ofdeny_diff.py; now the three-way equalityTests
Evidence first:
uncounted_gate, and ill-typed inputs: aNonetitle or kind, anintresolved agent, app or command, a non-mappingraw_params) were run at BASE (main, this PR's parent) and at HEAD. A second set of 60,000 also gives most calls acommand/cmdargument (shell text, bare paths, oversize, non-string values). The full verdict records match byte for byte throughon_tool_calland throughjudge. Each verdict records action, reason, flags, counter emits, log lines, the exact_governance_denialarguments and how many times the gate read the platform context. The same harness detects every tier mutation (below).local-gate --dry-runselects 1,443 related backend files, more than this shared host runs at once, so CI owns that set. Locally, the owned, adjacent and gate-oracle tests (47 files) pass: 6,009 passed, 3 skipped. The new and rewritten tests passed 20 times serially and 10 times under-n 4, and the touched files pass alone, in file order and reversed. mypy, flake8, black, isort, docs-lint and semgrep are clean. The frontend is untouched.New interface suite,
test/test_gate_tiers.py:shell_ruleis the check its row really consults (make exactly that check refuse);classifier_onlydropping both grants, and refusing a host read-only built-in under any kind outside the read-only allow-list;command/cmdargument meets the shell rules and the deny catalog, never the path rule;.tierprovenance.test_gate_tiersgolden + soundness + deny × grant matrixon_tool_calltest_hooks: signature parityToolCallcompleteness throughfrom_event,kwargs()and the shim, pluson_tool_call/judgekeyword paritytest_hooks: AST over_READ_ONLY_TOOL_KINDSread/fetchauto-approve;other, mutators and arbitrary kinds prompt; the value pin staystest_name_grant:getsourcejudgereaches the same verdict for every scenario, and no module keeps a stub after the undotest_gate_crash_fail_closed:getsourcehalfjudgemetrics/test_business_counters: AST over returnsAPPROVAL_DECISIONSemit per judgement, for every reachable verdict of every row, a crash and an unsound tabletest_deny_diff:read_text+ regexhooks.SHELL_DENY_TIERS== projection; error modes; harness alone; one table for both sidestest_security_conductor_verify_fix: regex overdeny_diff.pyverify_fix.TIERS == hooks.SHELL_DENY_TIERS ==thedeny_diffreaderThe composition contract also gains the new owner, the two facade classes, and a globals sweep widened to class members.
test_hooks.py's caller splat scan stays (it guards callers, not the gate). It now also holds a typedjudge(...)site to the same rules: the call is built in place byToolCall.from_event, never wrapped or bound first, and overrides onlyresolved_agentplus the site's pinned overrides. The settle ladder's ownGateport (policy.gate.judge(ask)intool_permission.py) is not a hook-gate consultation; the scan skips it by that exact receiver, and pins that the site still exists.Mutation proofs: each mutation reds named tests, and the unmutated run (851 tests) is green.
ToolHookResult(...)in a tierToolCallfieldSHELL_DENY_TIERSverify_fix.TIERSdeny_diffreadercommand/cmdargument no longer judgedManual verification
N/A — unit coverage and the recorded BASE/HEAD differential are sufficient: the change is in-process with no UI or external service. The 3-OS Denial Differential and the e2e smoke run on this exact SHA in CI.
Related Issues
Part of #16536
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)