Skip to content

refactor(hooks): run the tool gate from a tier table - #17192

Merged
bolichen97 merged 1 commit into
mainfrom
refactor/w2-22-gate-tier-table
Oct 7, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
refactor/w2-22-gate-tier-table

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the tool gate's decision order one table that every reader — the gate itself, the denial differential and the tests — takes from the same place, while every tool-call decision stays exactly what it is today.

HookManager.on_tool_call was a 785-line method. Its tier order is a security invariant: every deny tier must run before every grant, or a grant re-admits what a deny blocked. That order lived only as the sequence of statements in the method. It was kept true by ten tests that read the method's source text, plus two hand-copied tier tables in scripts/deny_diff.py and the security-conductor's verify_fix.py. Adding a check meant editing the method, both copies and the source-reading pins.

Why it matters

  • The gate is the one place every surface consults before a tool runs, and its order is what reviewers must check on every security change. As prose inside one method, it could only be guarded by tests that pinned spelling, not behaviour.
  • The denial differential claims "no golden path is newly refused at the tool gate". That claim held only while a hand-copied list matched the method.
  • Those source pins froze the method in place, so wave-2 work on the three approval surfaces (W2-10, W2-12) could not move it.

Not a goal

  • No decision changes. The tier order and every decision are preserved byte for byte; two returns the table order makes unreachable are kept and named in the code.
  • No caller migration. The 16 production callers keep calling on_tool_call with the same keywords. Moving them to judge is later work (W2-10 / PR-2b).
  • No tool_permission.py engine, desk or settle work. Those belong to W2-10 and W2-12.
  • No new seal, mask or rule. No CI logic, workflow step or AUTOSDE change; the one workflow edit is comment-only, and the docs-lint baseline shrink is listed below. The only new refusal is the unsound-table refusal, a fail-closed guard the shipped table never reaches.

What changed (motivation → approach → change)

The gate is now data plus one loop. HookManager.judge(call, *, session_key, agent, app, classifier_only) walks hooks.GATE_TIERS and returns the first verdict a row hands back, stamped in the new ToolHookResult.tier. on_tool_call is now a thin keyword shim over it with the same signature.

Piece Where What it is
ToolCall hook_runtime/tool_identity.py The gate's typed input. from_event is the one event-field extraction; hook_gate_kwargs(event) == ToolCall.from_event(event).kwargs()
GATE_TIERS, GateTier, GateRule, SHELL_DENY_TIERS hooks.py 11 rows in order: deny tiers, then the policy deny, then the grant tiers, then the read-only classifier. targets holds three per-target rules run target-major; with no recovered command and no MCP server, the tool's own command/cmd argument meets its two shell rules. SHELL_DENY_TIERS is the table's shell projection, written as a literal
tier bodies, GateFacts hook_runtime/gate_tiers.py (joins the existing compose) One function per row. GateFacts derives each value on first use, in today's order, from one platform-context snapshot; the effective deny set is resolved twice from that snapshot, as at BASE
table soundness gate_tier_problems Runs when the module loads. An unsound table refuses every call (fail closed) instead of raising at import, which would fail every test at collection

Two boundaries were kept on purpose, and security review should check them:

  • deny_diff reads the table from the harness tree. It parses hooks.py with ast.literal_eval from the tree the script itself runs from. It never reads a tree it classifies, never imports the product, and reads lazily inside differential/classify, never at import. In the scope lanes the harness is the base or default-branch checkout, so the change under judgement cannot shrink the composite it is measured by. That is exactly where the deleted _TIERS literal lived.
  • verify_fix.py keeps its literal (Master ruling). It runs as a synced skill file with no product tree beside it. Its only tree is the fixer's worktree, so reading the table there would let the fix being judged choose its own gate. A behavioural three-way equality replaces its source-regex pin.

Docs follow the code in the same commit: the two owning specs (memory-skills-hooks.md, the "Hook runtime owners" section and a new tier table; the hook-gate rows of security.md), plus statements the move made false, edited under the campaign Master's entry-scoped grants:

File Edit
docs/request-for-change/rfc-session-tag-change-event.md, rfc-crew-agent-sdk-boundary.md hooks.py:<line> cites become symbols; their orphaned .github/docs-lint-baseline.txt rows are pruned (shrink only)
docs/ci/denial-differential.md, golden-paths header the measured list is read from hooks.SHELL_DENY_TIERS (three checks)
.github/workflows/denial-differential.yml comment-only edit to a workflow file, granted by the campaign Master; no key, step or logic changes
governance.md, hook_runtime/__init__.py docstring the Plane A threading and the new owner definitions

Deletion test.

  • Delete gate_tiers.py and the eleven tier bodies fall back into one method. The order returns to prose, guarded by source pins.
  • Delete GATE_TIERS and the order again lives in no single place. deny_diff and verify_fix would need hand mirrors again.
  • Delete ToolCall and the gate's input becomes 15 loose keywords again, re-threaded at each of the 16 callers.

Backwards compatibility

Compatible:

  • on_tool_call keeps its exact signature and verdicts, and every caller, patch and fake still targets it.
  • hook_gate_kwargs returns the same dict.
  • deny_diff's CLI, exit codes and report are unchanged.
  • ToolHookResult gains a defaulted field that takes no part in equality.

Reader: test/test_deny_diff.py:test_the_measured_checks_are_the_checks_the_tool_gate_applies -- entry: release -- read the deleted deny_diff._TIERS; now reads load_shell_deny_tiers
Reader: test/test_security_conductor_verify_fix.py:test_the_tiers_are_the_gate_s_and_the_differential_s -- entry: release -- parsed _TIERS out of deny_diff.py; now the three-way equality

Tests

Evidence first:

  • Recorded differential. 60,000 seeded scenarios (titles, commands, kinds, params incl. an over-cap path list, diff blocks, MCP identities, configs, governance, crashes, uncounted_gate, and ill-typed inputs: a None title or kind, an int resolved agent, app or command, a non-mapping raw_params) were run at BASE (main, this PR's parent) and at HEAD. A second set of 60,000 also gives most calls a command/cmd argument (shell text, bare paths, oversize, non-string values). The full verdict records match byte for byte through on_tool_call and through judge. Each verdict records action, reason, flags, counter emits, log lines, the exact _governance_denial arguments and how many times the gate read the platform context. The same harness detects every tier mutation (below).
  • Local run. local-gate --dry-run selects 1,443 related backend files, more than this shared host runs at once, so CI owns that set. Locally, the owned, adjacent and gate-oracle tests (47 files) pass: 6,009 passed, 3 skipped. The new and rewritten tests passed 20 times serially and 10 times under -n 4, and the touched files pass alone, in file order and reversed. mypy, flake8, black, isort, docs-lint and semgrep are clean. The frontend is untouched.

New interface suite, test/test_gate_tiers.py:

  • the row-and-kind golden, and that every per-target rule but the path rule names its shell check;
  • each shell_rule is the check its row really consults (make exactly that check refuse);
  • the soundness check, with a self-test for every way it can fail;
  • an unsound table refusing every call;
  • rows running on the facade's globals (the compose trap);
  • one scenario per tier, plus one per other return a row body can reach in table order (the operator grants' verified-identity arm and its title-only fallthrough included), and a check that every row and rule has one;
  • a malformed resolved agent is refused as a gate crash in either mode, as at BASE;
  • every deny under each grant;
  • classifier_only dropping both grants, and refusing a host read-only built-in under any kind outside the read-only allow-list;
  • exactly one platform-context read per judgement, taken after the title is normalized, and none for a call the first tier refuses;
  • a non-execute shell tool's command/cmd argument meets the shell rules and the deny catalog, never the path rule;
  • target-major order, rule order and the sandboxed-shell exemption;
  • .tier provenance.
Deleted pin (source text / getsource / AST) Replacement at the interface
composition contract: decision-order markers test_gate_tiers golden + soundness + deny × grant matrix
composition contract: factory set in on_tool_call per-tier counter test + closed kind vocabulary
test_hooks: signature parity ToolCall completeness through from_event, kwargs() and the shim, plus on_tool_call / judge keyword parity
test_hooks: AST over _READ_ONLY_TOOL_KINDS only read/fetch auto-approve; other, mutators and arbitrary kinds prompt; the value pin stays
test_name_grant: getsource every name resolver raises; judge reaches the same verdict for every scenario, and no module keeps a stub after the undo
test_gate_crash_fail_closed: getsource half deleted; the signature half is kept and extended to judge
metrics/test_business_counters: AST over returns exactly one APPROVAL_DECISIONS emit per judgement, for every reachable verdict of every row, a crash and an unsound table
test_deny_diff: read_text + regex the reader returns hooks.SHELL_DENY_TIERS == projection; error modes; harness alone; one table for both sides
test_security_conductor_verify_fix: regex over deny_diff.py verify_fix.TIERS == hooks.SHELL_DENY_TIERS == the deny_diff reader

The composition contract also gains the new owner, the two facade classes, and a globals sweep widened to class members. test_hooks.py's caller splat scan stays (it guards callers, not the gate). It now also holds a typed judge(...) site to the same rules: the call is built in place by ToolCall.from_event, never wrapped or bound first, and overrides only resolved_agent plus the site's pinned overrides. The settle ladder's own Gate port (policy.gate.judge(ask) in tool_permission.py) is not a hook-gate consultation; the scan skips it by that exact receiver, and pins that the site still exists.

Mutation proofs: each mutation reds named tests, and the unmutated run (851 tests) is green.

Mutation Tests that go red
swap a grant above a deny tier 11
drop a tier 10
direct ToolHookResult(...) in a tier the per-branch counter
double count 14
swap per-target rules 4
tier-major instead of target-major 14
drop the shell exemption 4
resolve names in the gate 2
widen the kind allow-list 7
replace the allow-list with a denylist 49
drop a ToolCall field 2
a row built before compose 101
drift SHELL_DENY_TIERS 4
drift verify_fix.TIERS 3
drift the deny_diff reader 6
skip the grant rows before the owner app resolves 2
add a row with no scenario 3
verified-identity grant built without the counted factory 2
verified-identity grant resolving a program name 1
unattended kind check turned into a denylist 4
deny-rules or governance re-reading the context 1 each
the context read before the first tier refuses 1
the snapshot taken before the title is normalized 1
a command/cmd argument no longer judged 11
the path rule judging that argument 1
the deny catalog no longer seeing it 4
an MCP-served call's argument read as shell text 3

Manual verification

N/A — unit coverage and the recorded BASE/HEAD differential are sufficient: the change is in-process with no UI or external service. The 3-OS Denial Differential and the e2e smoke run on this exact SHA in CI.

Related Issues

Part of #16536

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 6, 2026 01:11
@bolichen97
bolichen97 requested a review from iamwhatever October 6, 2026 01:11
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/builtin_skills/security-conductor/golden-paths.json

  • src/kiro_crew/hook_runtime/__init__.py

  • src/kiro_crew/hook_runtime/gate_tiers.py

  • src/kiro_crew/hook_runtime/tool_identity.py

  • src/kiro_crew/hooks.py

  • @iamwhatever approved 695a2e9 with a complete reasoning template.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 695a2e961ebb8d6b6c421dcbc86328505afafca2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, confirm test/test_gate_tiers.py's row-and-kind golden replaces the deleted test_the_gate_decision_order_is_unchanged, so the tier order stays pinned as a deliberate decision.

[DESIGN-REVIEWED] 695a2e9

Description read

[DESCRIPTION-READ] ed9cea2af4956e19bb5c7f7fb9f2ccf0bc5e7d7444b3ff3ea620a7eddf3a1893

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 695a2e961ebb8d6b6c421dcbc86328505afafca2 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The load-time soundness check is a second copy of the test_the_table_is_the_gate_order golden. The shipped table never trips it, so dropping it would change nothing anyone notices.

Not justified as shipped

  • Item 5 — duplicate of test/test_gate_tiers.py::test_the_table_is_the_gate_order: that golden test pins all 11 (name, kind) pairs exactly. GATE_TIERS is a module literal, so any reorder that gate_tier_problems could catch fails that test first. The PR's own words: "a fail-closed guard the shipped table never reaches."
    Subtraction: delete _GATE_TABLE_PROBLEMS, its logger.error, GATE_TABLE_UNSOUND_REASON and the if _GATE_TABLE_PROBLEMS: refusal in judge. Keep gate_tier_problems only if a test still calls it.
    Clears when: the guard is removed, or the PR names a way GATE_TIERS can change without failing the golden test.
  • Item 4 — zero consumers: grepping .tier in src/ finds only the writes in judge and _tier_targets. All 20+ reads are in tests. scripts/deny_diff.py reports its own tier and does not read this field.
    Subtraction: delete ToolHookResult.tier. Tests assert on reason, or on which tier body was called.
    Clears when: a production reader (log, SEL, counter or differential) consumes the field, or the field is deleted.

What this change ships

Inventory (9 items) — 7 justified

Intent: make the tool gate's tier order one table that the gate, the denial differential and the tests all read, with no decision changed. This is a refactor, not a fix or an addition.

  1. The gate's decision order now lives in one table, GATE_TIERS, which a loop walks — justified
  2. New HookManager.judge(ToolCall); on_tool_call becomes a shim over it — justified
  3. The gate takes a typed ToolCall input; hook_gate_kwargs now builds its dict from it — justified
  4. Each verdict is stamped with the tier that decided it (ToolHookResult.tier) — zero consumers (only tests read it)
  5. A table that fails the load-time soundness check refuses every call — duplicate of the golden test
  6. deny_diff reads a SHELL_DENY_TIERS literal through ast instead of keeping its own copy — justified
  7. verify_fix keeps its literal; its source-regex pin becomes a three-way equality test — justified
  8. Tests that read the method's source text are replaced with behavioural tests — justified
  9. RFC line cites become symbol names, the docs-lint baseline shrinks, one workflow comment changes — justified

[FIRST-PRINCIPLES-REVIEWED] 695a2e9

Description read

[DESCRIPTION-READ] 3d674f60e0ed98f5f818eb6ee2a6205eb2aaed840b4efd608f5c6063b1884ca8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 695a2e961ebb8d6b6c421dcbc86328505afafca2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 695a2e9

Verdict parsed from the review's SHA-scoped output markers for commit 695a2e961ebb8d6b6c421dcbc86328505afafca2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 695a2e961ebb8d6b6c421dcbc86328505afafca2: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 695a2e961ebb8d6b6c421dcbc86328505afafca2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 695a2e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 695a2e961ebb8d6b6c421dcbc86328505afafca2: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does 695a2e961ebb8d6b6c421dcbc86328505afafca2 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 15 classified, 0 newly refused, 15 still allowed.
  • ubuntu-latest (platform posix): 15 classified, 0 newly refused, 15 still allowed.
  • windows-latest (platform windows): 6 classified, 0 newly refused, 6 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: The tool gate's composition is moved, not made stricter. HookManager.on_tool_call becomes a walk over the ordered table hooks.GATE_TIERS (judge), with the tier bodies in the new hook_runtime/gate_tiers.py. The one refusal that did not exist before is the unsound-table guard, which refuses every call when the table fails its own consistency check: "if _GATE_TABLE_PROBLEMS: return ToolHookResult.deny(GATE_TABLE_UNSOUND_REASON)" (hooks.py, judge). The threat it names is a deny tier reordered below a grant, letting the grant re-admit what the deny blocked: "a deny tier that moved below a grant would let the grant re-admit something the deny blocked" (gate_tier_problems). With the shipped table that list is empty. I compared the tier bodies line by line against the base on_tool_call and found the same checks in the same order:

  • target-major path, then sensitive-bash, then exfil;
  • the command/cmd argument meets the shell rules only;
  • one context snapshot, taken after the unverifiable-shell deny;
  • the same deny targets, MCP auto-deny, search target, governance arguments, grants and classifier.

The three security.* checks are untouched.

ADJUDICATED: 18 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE:

  • posix: test, lint, differential and read-only search commands in the touched area. One is a compound cd … && grep shell, the shape the exempt_command note names. One is a python3 -c that parses SHELL_DENY_TIERS as text, the way load_shell_deny_tiers does.
  • windows: a .venv\Scripts\python.exe pytest run, a PowerShell Select-String, and a cmd /c "cd … && findstr …" compound with backslash paths.
  • any: scoped git diff/git log/flake8 over the gate files.

The gate code in this diff makes no new path, separator, interpreter or environment assumption. The only new path literal is Path("src") / "kiro_crew" / "hooks.py" in scripts/deny_diff.py, a harness-side read that the classifier never sees. So no assumed platform is left without a candidate.

UNADJUDICATED:

  • Guard: the gate composition itself (HookManager.judge / GATE_TIERS, GateFacts, ToolCall.from_event, and the unsound-table refusal).
    • Call path: every surface goes through on_tool_call(tool_name, **hook_gate_kwargs(event)) → judge(ToolCall(...)) → each GATE_TIERS row in turn.

    • The classifier calls is_sensitive_bash_command, audit_bash_exfiltration and is_denied directly. The new docs/ci/denial-differential.md text says a change made inside the gate itself "runs this gate and comes back empty". So the 0 above says nothing about whether the move kept behaviour; it only shows the rule functions are unchanged.

    • Inputs I would have submitted, none expressible as a shell row:

      • an execute frame with is_shell=True and both command and mcp_server set (does the path rule still run on that command?);
      • a DeepSeek bash-as-other call carrying raw_params={"cmd": "git status"} and no recovered command;
      • an edit-kind call with raw_params={} against a workspace file;
      • a builtin app's own-server MCP call where app="" and resolved_agent is set;
      • a kind="read" file read of config.json;
      • a classifier_only side-chat read of a host-known built-in with identity_trusted=True.

      Reading the code, each looks unchanged, but nothing ran them.

  • Guard: the unsound-table refusal (_GATE_TABLE_PROBLEMS, computed once at import). It reaches every surface but is inert while the shipped table is sound. Nothing can measure it short of a mis-built table, and the right check for that is test/test_gate_tiers.py, not a corpus row.
  • Measured-set provenance (scripts/deny_diff.py). The measured tier set is now read from the harness tree's SHELL_DENY_TIERS instead of a literal in the script. A future change could narrow what the differential measures by editing that literal in hooks.py, and only the test_gate_tiers.py pin stands in the way. That reduces what this lane can see, so it is a coverage gap rather than a refusal, and it is noted here for the reviewer who decides.

[SCOPE-REVIEWED] 695a2e9

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the import-time table soundness check, failing closed at judge

  • span=e82f0be65150 Item 4: the unsound-table refusal duplicates the soundness test

rebutted, kept: the accepted design for this seam (joint DIT §4.1, gate invariants) makes the monotone tier-kind order an invariant "checked at import", so the shipped module carries the check, not only the CI suite.
Raising at import is the failure AGENTS.md records (#11556: an import-time invariant made the import raise and failed every backend test at collection), so the import-time result is kept and judge fails closed on it instead.
Cost: one truthiness check per call. The shipped table never reaches it, as the Not-a-goal section states; the CI test pins the shipped table, this guard covers the installed module.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: ToolHookResult.tier, the deciding row's name

  • span=f415afcc3599 Item 5: ToolHookResult.tier has no production reader

rebutted, kept: it is the interface's observable for "the first row that decides wins". Several rows share one reason string (the sensitive-path rule and param-paths both return sensitive_path_refusal's text; param-paths and write-protected share the truncation text), so reason and action cannot tell which row decided.
The interface tests that replace the deleted source pins (one scenario per row, the deny x grant matrix, target-major and rule order) assert on it; without it those pins have no behavioural replacement.
The accepted design lists it as part of the gate's interface, and its first production reader is planned for the next change on this seam (W2-10 / PR-2b, where a gate verdict becomes a settled refusal). It takes no part in equality, so no existing reader changes.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: ToolCall.from_event field overrides

  • span=b4290589e43c Item 6: the ToolCall.from_event overrides have no production caller

rebutted, kept: the overrides are the accepted interface (from_event(event, **overrides), joint DIT §4.1). They carry the per-site fields an event cannot: resolved_agent at the dashboard, and the three overrides two existing sites already pass to hook_gate_kwargs (agent_runner and kas_wire: kind, command, is_shell).
Those sites move to judge in the next change on this seam (W2-10 / PR-2b); removing the parameter now only makes that change re-add it.
This PR already governs it: the judge-site scan in test_hooks.py holds every judge(...) site to resolved_agent plus that site's pinned overrides, and from_event refuses an unknown field with the TypeError hook_gate_kwargs raises.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Security Scope Review (CONCERNS, advisory): the inputs it would have submitted, run at the gate level.

Each was judged through on_tool_call at BASE 231561d2d (this PR's parent) and at head 383ea4697, isolated. The two result sets are byte-identical:

Input BASE Head
sandboxed shell, command="cd /x && grep foo bar" allow (not path-refused) same
MCP-served shell-kind call (mcp_server set), ls -la /workplace auto-approve, read-only same
edit kind, raw_params=None, no diff_path allow (falls through, no empty-target deny) same
kind="read" read of ~/.kirocrew/config.json auto-approve, read-only same
classifier_only, trusted host read-only built-in (fs_read) auto-approve, read-only same
a stray keyword (bogus=1) deny, gate crash (TypeError) same

The open risk does not hold as a change. on_tool_call was already wrapped by _fail_closed_on_gate_crash at BASE (hooks.py L867 at 231561d2d), so a stray keyword was already a fail-closed deny there. Nothing converts a TypeError that did not convert before.

The broader evidence is in the PR body: a 60,000-scenario BASE/HEAD differential that matches byte for byte through on_tool_call and judge, and the 3-OS Denial Differential on this head.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Design Review (PASS): the differential was run against this exact head.

At 383ea4697 (and its parent 231561d2d, a git archive of main), the recorded 60,000-scenario harness produced:

  • v5, no context counting: BASE, head on_tool_call and head judge all hash c20ef4028a432150…;
  • v6, with the gate's own current_context() read count per call: all three hash 68d3534b616234a7….

The merge ref adds one main commit, 782dedda3: a coverage-baseline row, which touches no gate file.

The harness is not committed. It is a campaign evidence script that imports the product from either tree, and it is not a repository test. The suite carries the interface tests and mutation proofs instead. I can post the harness in a comment if a reviewer wants to rerun it.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

For the Sensitive Change Review approval. The author's evidence for head 383ea4697, laid out in the gate's template. The approving reviewer verifies it and pastes their own copy into the approval; this comment does not satisfy the gate.

## Sensitive change review
- What rule changed: none. The tool gate's decision order moved from the body of HookManager.on_tool_call into the GATE_TIERS table that HookManager.judge walks; each tier body is the base branch it replaces, and on_tool_call is a shim with the same signature.
- Before -> after (what was allowed/blocked, now): unchanged. A 60,000-scenario base/head differential (verdict, reason, flags, counter emits, log lines, governance arguments, platform-context read count) is byte-identical through on_tool_call and judge.
- Worst case if wrong (breaks commands / leaks secret / locks user out): a misordered table would let a grant re-admit a call a deny tier refused. Pinned by test/test_gate_tiers.py (order golden, soundness check with self-tests, every deny under every grant) and by the fail-closed soundness check at import.
- Evidence checked (which sandbox result proves it): Denial Differential green on macos-latest, ubuntu-latest and windows-latest; Security Scope adjudication 0 newly refused on all three.
- No regression:
  - Tools still work (which tools/commands were run, result): E2E (stub ACP backend), E2E Gateway Boot (ubuntu, windows), Integration (in-process gateway), Pod Boot Canary (Windows): all green.
  - Backward compatible (old config / old data / old callers still work, how checked): on_tool_call keeps its signature and verdicts and all 16 callers are untouched; hook_gate_kwargs returns the same dict; deny_diff's CLI, exit codes and report are unchanged.
  - Existing tests (which suites passed on this head): Backend Tests 3.12 x8, Windows x8, macOS x4, IPv6 (ubuntu, windows), namespace sandbox, crew container, Coverage Gate: all green on 383ea4697.
- How to undo: revert the single commit.

@bolichen97
bolichen97 enabled auto-merge (squash) October 6, 2026 04:42
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed for merge triage, not approving from my side: ~2.8k added / ~1.2k removed lines moving the hook tool gate to a tier table, and the Sensitive Change Review check is red on this head. The tool gate is a security boundary that needs a security owner's sign-off. Auto-merge is already on, so a single approval merges it; please make sure a maintainer reviews it directly. CI is green.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Heads-up: this branch went CONFLICTING against main after the 16:5xZ batch landed (#17266, #17370, #17434), so the armed auto-merge cannot fire. A rebase onto main 9bf943b0f with conflict resolution + a full gate re-run is in flight; I'll push a lease-pinned update and report the new head here. No change to auto-merge.

The tool gate's decision order lived only as the statement order of a
785-line HookManager.on_tool_call, guarded by tests that read its source
and by two hand-copied tier tables. It is now data: hooks.GATE_TIERS
lists the gate's rows (every deny tier, then the policy deny, then the
grants, then the read-only classifier), HookManager.judge(call, ...)
walks it, and each row's body lives in the composed owner
hook_runtime/gate_tiers.py. ToolCall, in hook_runtime/tool_identity.py,
is the gate's typed input; hook_gate_kwargs is its keyword form and
on_tool_call a shim over judge with the same signature. Verdicts carry
the deciding row in ToolHookResult.tier.

scripts/deny_diff.py drops its literal table and reads the gate's
SHELL_DENY_TIERS with ast.literal_eval from the tree the harness runs
from; verify_fix.py keeps its copy, pinned equal. The source-reading
gate pins become behavioural tests in test/test_gate_tiers.py and their
own files. Every decision is unchanged: a recorded 60,000-scenario
BASE/HEAD differential is byte-identical.

Part of #16536
@bolichen97
bolichen97 force-pushed the refactor/w2-22-gate-tier-table branch from 383ea46 to 695a2e9 Compare October 6, 2026 19:38
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 6, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed again at the new head, still not approving (crew-pr-reviewer). It rewrites the tool approval gate (+2886/-1203), and Sensitive Change Review is red. A security owner must approve it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the import-time table soundness check, failing closed at judge

  • span=9875fe47dd1c Item 5: the load-time soundness check duplicates the order golden

rebutted, kept (same finding as on 383ea46, ruled keep): the accepted design for this seam (joint DIT §4.1, gate invariants) makes the monotone tier-kind order an invariant checked at import, so the installed module carries it, not only the test suite.
Raising at import is the failure AGENTS.md records (#11556: an import-time invariant failed every backend test at collection), so the result is kept and judge fails closed on it instead.
Cost: one truthiness check per call. No path changes the literal table without a code edit, which the golden pins; the guard covers the shipped module, as the Not-a-goal section states.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: ToolHookResult.tier, the deciding row's name

  • span=39f62af79826 Item 4: ToolHookResult.tier has no production reader

rebutted, kept (same finding as on 383ea46, ruled keep): it is the interface's observable for "the first row that decides wins". Several rows share one reason string (the sensitive-path rule and param-paths both return sensitive_path_refusal's text; param-paths and write-protected share the truncation text), so reason and action cannot tell which row decided.
The interface tests that replace the deleted source pins (one scenario per row, every deny under every grant, target-major and rule order, the command/cmd argument rows) assert on it; without it those pins have no behavioural replacement.
The accepted design lists it in the gate's interface, and its first production reader is planned for the next change on this seam (W2-10 / PR-2b). It takes no part in equality.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 7, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Security Scope Review on 695a2e961 (CONCERNS, advisory): its unrun gate-level inputs, run.

Each call went through on_tool_call at BASE 9bf943b0f (this PR's parent, which carries #16920) and at head 695a2e961, isolated. The two result sets are byte-identical (action, reason, read-only, identity grant):

Input BASE Head
execute frame, is_shell=True, with command and mcp_server both set (ls -la /workplace) auto-approve, read-only (path rule runs, nothing refused) same
bash streamed as other, raw_params={"cmd": "git status"}, no recovered command allow (not refused) same
edit kind with raw_params={} deny, "file edit names no target path to verify" same
builtin app's own-server call with app="" and resolved_agent set auto-approve, identity grant same
kind="read" read of ~/.kirocrew/config.json auto-approve, read-only same
classifier_only, trusted host read-only built-in (fs_read) auto-approve, read-only same

On measured-set provenance: the differential measures hooks.SHELL_DENY_TIERS, the gate's own shell projection, read from the harness tree. Narrowing that literal fails two pins:

  • test_gate_tiers.py: the literal must equal the projection of GATE_TIERS, and every per-target rule but the path rule must name its check;
  • test_security_conductor_verify_fix.py: a three-way equality with verify_fix.TIERS.

The rows themselves are pinned by the order golden. So a narrowing must also delete a gate check, and that shows in the diff a reviewer reads.

The broader evidence is in the PR body. Three 60,000-scenario BASE/head differentials match byte for byte through on_tool_call and judge. The third gives most calls a command/cmd argument, which exercises the #16920 path.

@bolichen97

Copy link
Copy Markdown
Contributor Author

For the Sensitive Change Review approval of the CURRENT head 695a2e961. This supersedes the evidence in the earlier comment, which named 383ea4697. The approving reviewer verifies it and pastes their own copy into the approval; this comment does not satisfy the gate.

## Sensitive change review
- What rule changed: none. The tool gate's decision order moved from the body of HookManager.on_tool_call into the GATE_TIERS table that HookManager.judge walks; on_tool_call is a shim with the same signature. This head is rebased onto main 9bf943b0f, and the denied-command handling main added for a shell tool reported under a non-execute kind (#16920) is carried into the table unchanged: the tool's own command/cmd argument meets the same shell checks and deny catalog, in the same order, never the path check.
- Before -> after (what was allowed/blocked, now): unchanged against main 9bf943b0f. Three 60,000-scenario base/head differentials (verdict, reason, flags, counter emits, log lines, governance arguments, context read count; the third gives most calls a command/cmd argument) are byte-identical through on_tool_call and judge. An independent review probe of 258,720 main-vs-head inputs matched on every field but the new tier stamp, with 0 security-event-log differences over 64,680 inputs.
- Worst case if wrong (breaks commands / leaks secret / locks user out): a misordered table would let a grant re-admit a call a deny tier refused. Pinned by test/test_gate_tiers.py (order golden, soundness check with self-tests, every deny under every grant, one scenario per reachable branch) and by the fail-closed soundness check at import.
- Evidence checked (which sandbox result proves it): Denial Differential green on macos-latest, ubuntu-latest and windows-latest at 695a2e961; Security Scope adjudication 0 newly refused on all three, and its unrun gate-level inputs judged identical at base and head (comment 6029728710).
- No regression:
  - Tools still work (which tools/commands were run, result): E2E (stub ACP backend), E2E Gateway Boot (ubuntu, windows), Integration (in-process gateway): green at 695a2e961.
  - Backward compatible (old config / old data / old callers still work, how checked): on_tool_call keeps its signature and verdicts and every caller is untouched; hook_gate_kwargs returns the same dict; deny_diff's CLI, exit codes and report are unchanged.
  - Existing tests (which suites passed on this head): Backend Tests 3.12 x8, Windows x8, macOS x4, IPv6, namespace sandbox, crew container, Coverage Gate, CodeQL, SAST: green at 695a2e961; main's own test_hooks_raw_command_deny.py passes unmodified.
- How to undo: revert the single commit.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sensitive change review

  • What rule changed: none. The tool gate's decision order moved from the body of HookManager.on_tool_call into the GATE_TIERS table that HookManager.judge walks; on_tool_call is a shim with the same signature. This head is rebased onto main 9bf943b, and the denied-command handling main added for a shell tool reported under a non-execute kind (#16920) is carried into the table unchanged: the tool's own command/cmd argument meets the same shell checks and deny catalog, in the same order, never the path check.
  • Before -> after (what was allowed/blocked, now): unchanged against main 9bf943b. Three 60,000-scenario base/head differentials (verdict, reason, flags, counter emits, log lines, governance arguments, context read count; the third gives most calls a command/cmd argument) are byte-identical through on_tool_call and judge. An independent review probe of 258,720 main-vs-head inputs matched on every field but the new tier stamp, with 0 security-event-log differences over 64,680 inputs.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): a misordered table would let a grant re-admit a call a deny tier refused. Pinned by test/test_gate_tiers.py (order golden, soundness check with self-tests, every deny under every grant, one scenario per reachable branch) and by the fail-closed soundness check at import.
  • Evidence checked (which sandbox result proves it): Denial Differential green on macos-latest, ubuntu-latest and windows-latest at 695a2e9; Security Scope adjudication 0 newly refused on all three, and its unrun gate-level inputs judged identical at base and head (comment 6029728710).
  • No regression:
    • Tools still work (which tools/commands were run, result): E2E (stub ACP backend), E2E Gateway Boot (ubuntu, windows), Integration (in-process gateway): green at 695a2e9.
    • Backward compatible (old config / old data / old callers still work, how checked): on_tool_call keeps its signature and verdicts and every caller is untouched; hook_gate_kwargs returns the same dict; deny_diff's CLI, exit codes and report are unchanged.
    • Existing tests (which suites passed on this head): Backend Tests 3.12 x8, Windows x8, macOS x4, IPv6, namespace sandbox, crew container, Coverage Gate, CodeQL, SAST: green at 695a2e9; main's own test_hooks_raw_command_deny.py passes unmodified.
  • How to undo: revert the single commit.

@bolichen97
bolichen97 merged commit f6046c2 into main Oct 7, 2026
109 of 110 checks passed
@bolichen97
bolichen97 deleted the refactor/w2-22-gate-tier-table branch October 7, 2026 07:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants