Skip to content

test(hooks): pin existing deny precedence for command arguments - #17647

Merged
bolichen97 merged 1 commit into
mainfrom
refactor/w2-22b-gate-tier-pins
Oct 7, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
refactor/w2-22b-gate-tier-pins

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Pin existing deny precedence for a shell tool's command argument at the tool gate's interface, so a reorder of the checks reds a test instead of passing unnoticed.

When no shell command was recovered (a harness that streams its shell tool under a kind other than execute), the tool gate judges the tool's own command/cmd argument with the shell checks and the deny-rule catalog. Three parts of that behaviour had no test:

  • the deny under an operator grant;
  • which check decides when the title and the argument both refuse;
  • that a recovered command, when there is one, is judged in place of the argument.

A reorder of any of them leaves the suite green.

Why it matters

  • The order of the gate's checks is a security invariant, and these three cases are where a refactor of the tier table could shift it silently.
  • They were found by an independent review of refactor(hooks): run the tool gate from a tier table #17192, which ported this behaviour into the tier table unchanged. Main has no test for them either.

Not a goal

  • No production code change. The gate's decisions are unchanged; this only pins them.
  • No change to which calls have their argument read, or to the shell checks or the deny catalog.
  • No new scenario coverage beyond these three cases.

What changed (motivation → approach → change)

Everything is in test/test_gate_tiers.py and runs at HookManager.judge:

Test Pins
test_every_deny_beats_every_grant[operator-*-argument] the three command-argument denies (bash scan, exfiltration shapes, deny catalog) still refuse under an operator grant that would approve everything, from the same tier
test_when_the_title_and_the_command_argument_both_refuse_the_title_decides the title is judged before the argument by every check, so a call refused on both carries the title's reason and tier, in the targets row and in the deny catalog
test_a_recovered_command_is_judged_in_place_of_the_command_argument with a recovered command, the argument changes nothing; the same argument on a call with no recovered command is refused

The three argument rows move into a named _ARGUMENT_SCENARIOS tuple that BRANCH_SCENARIOS still ends with, in the same order, so every existing scenario id is unchanged. The app-own-server half of the grant matrix does not apply to them: that grant re-addresses the call to the app's MCP server, and a call that names its server has no argument scan by design (test_hooks_raw_command_deny.py pins that).

Backwards compatibility

Compatible: test-only; no production code, config or public name changes.

Removes nothing: the three argument scenarios move into a named tuple and stay in BRANCH_SCENARIOS, so every test that reads the scenario tables keeps them.

Tests

  • Mutation proofs. Each mutation was applied to hook_runtime/gate_tiers.py and restored afterwards. Each one passes main's version of the test file and reds the new tests:

    Mutation Main's tests New tests that go red
    judge the argument before the title 619 passed the two targets-row precedence cases
    read the argument even when a command was recovered 619 passed the three recovered-command cases
    let a title-matching operator grant skip the argument scan 619 passed operator-sensitive-bash-argument, operator-exfil-argument
    ask the deny catalog about the argument before the title 619 passed the catalog precedence case
  • Determinism. The touched file and its importers (test_gate_tiers, metrics/test_business_counters, test_name_grant, test_hooks_raw_command_deny, test_hooks) pass with 628 tests in each of these runs: five times serially, once in reversed file order, and twice under -n 4. test_gate_tiers.py also passes alone. test/test_flake_pattern_ratchet.py passes.

  • Adjacent suites. The composition contract, crash fail-closed, deny_diff, verify_fix, tool_permission and name_grant surface suites pass (405 tests).

  • Static checks. black, flake8, isort, the brand and comment-history checks and semgrep are clean.

Manual verification

N/A — test-only change; the mutation proofs above show each new test can fail.

Related Issues

Part of #16536

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The tool gate reads a shell tool's own command/cmd argument when no
shell command was recovered (#16920). Three parts of that behaviour had
no test, so a reorder could change which check refuses a call, or
whether the argument is read at all, with the suite still green.

Pin them at the HookManager.judge interface in test_gate_tiers.py:

- the command-argument denies join the deny-under-grant matrix under
  the operator grant (the app-own-server grant names a server, which
  turns the argument scan off by design, so it cannot apply)
- when the title and the argument both refuse, the title's reason and
  tier decide, in the targets row and in the deny-rule catalog
- with a recovered command, the argument changes nothing

Each new test reds under the mutation it guards against and passes on
the code as it is; no production code changes.

Part of #16536
@bolichen97
bolichen97 requested a review from a team as a code owner October 7, 2026 08:15
@bolichen97
bolichen97 requested a review from iamwhatever October 7, 2026 08:15
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 989e2565bc7313d2d271f19380606cfcabe19796 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, confirm a maintainer wants this behaviour pinned: when a command is recovered, a rm -rf / argument changes nothing. A later fix that also scans the argument would have to delete this test.

[DESIGN-REVIEWED] 989e256

Description read

[DESCRIPTION-READ] 02f383835799d3cbe86e6748ec6835e702b39530dd984f81b8cccdd9c627837b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 989e2565bc7313d2d271f19380606cfcabe19796 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 989e256

Verdict parsed from the review's SHA-scoped output markers for commit 989e2565bc7313d2d271f19380606cfcabe19796.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 989e2565bc7313d2d271f19380606cfcabe19796: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 989e2565bc7313d2d271f19380606cfcabe19796 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 989e256

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 989e2565bc7313d2d271f19380606cfcabe19796: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Design Review (PASS) note: should the recovered-command case be pinned?

Yes, deliberately, as characterization. security.md documents the scope #16920 gave the argument scan: it applies "when no shell command was recovered". test_a_recovered_command_is_judged_in_place_of_the_command_argument pins that boundary at the interface, so main's behaviour is unchanged and only now observable.

The pin is not a ruling that the boundary is final. A later change that widens the scan to calls with a recovered command would be a decision about the gate's scope. That change should edit this test in the same diff, together with the security.md sentence, so the widening is visible to its reviewers instead of happening silently. Preventing a silent change is the purpose of the pin.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 7, 2026 09:53

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving (crew-pr-reviewer). Test-only. It pins today's deny precedence for a shell command argument at the gate's interface, so reordering the checks turns a test red. No production change. CI is green and the base is main.

@bolichen97
bolichen97 merged commit 256bddc into main Oct 7, 2026
101 checks passed
@bolichen97
bolichen97 deleted the refactor/w2-22b-gate-tier-pins branch October 7, 2026 16:10
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants