Repository navigation
test(hooks): pin existing deny precedence for command arguments - #17647
Conversation
The tool gate reads a shell tool's own command/cmd argument when no shell command was recovered (#16920). Three parts of that behaviour had no test, so a reorder could change which check refuses a call, or whether the argument is read at all, with the suite still green. Pin them at the HookManager.judge interface in test_gate_tiers.py: - the command-argument denies join the deny-under-grant matrix under the operator grant (the app-own-server grant names a server, which turns the argument scan off by design, so it cannot apply) - when the title and the argument both refuse, the title's reason and tier decide, in the targets row and in the deny-rule catalog - with a recovered command, the argument changes nothing Each new test reds under the mutation it guards against and passes on the code as it is; no production code changes. Part of #16536
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, confirm a maintainer wants this behaviour pinned: when a command is recovered, a [DESIGN-REVIEWED] 989e256 Description read[DESCRIPTION-READ] 02f383835799d3cbe86e6748ec6835e702b39530dd984f81b8cccdd9c627837b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Design Review (PASS) note: should the recovered-command case be pinned? Yes, deliberately, as characterization. The pin is not a ruling that the boundary is final. A later change that widens the scan to calls with a recovered command would be a decision about the gate's scope. That change should edit this test in the same diff, together with the |
iamwhatever
left a comment
There was a problem hiding this comment.
Approving (crew-pr-reviewer). Test-only. It pins today's deny precedence for a shell command argument at the gate's interface, so reordering the checks turns a test red. No production change. CI is green and the base is main.
Problem / Motivation
Goal: Pin existing deny precedence for a shell tool's command argument at the tool gate's interface, so a reorder of the checks reds a test instead of passing unnoticed.
When no shell command was recovered (a harness that streams its shell tool under a kind other than
execute), the tool gate judges the tool's owncommand/cmdargument with the shell checks and the deny-rule catalog. Three parts of that behaviour had no test:A reorder of any of them leaves the suite green.
Why it matters
Not a goal
What changed (motivation → approach → change)
Everything is in
test/test_gate_tiers.pyand runs atHookManager.judge:test_every_deny_beats_every_grant[operator-*-argument]test_when_the_title_and_the_command_argument_both_refuse_the_title_decidestest_a_recovered_command_is_judged_in_place_of_the_command_argumentThe three argument rows move into a named
_ARGUMENT_SCENARIOStuple thatBRANCH_SCENARIOSstill ends with, in the same order, so every existing scenario id is unchanged. The app-own-server half of the grant matrix does not apply to them: that grant re-addresses the call to the app's MCP server, and a call that names its server has no argument scan by design (test_hooks_raw_command_deny.pypins that).Backwards compatibility
Compatible: test-only; no production code, config or public name changes.
Removes nothing: the three argument scenarios move into a named tuple and stay in
BRANCH_SCENARIOS, so every test that reads the scenario tables keeps them.Tests
Mutation proofs. Each mutation was applied to
hook_runtime/gate_tiers.pyand restored afterwards. Each one passes main's version of the test file and reds the new tests:operator-sensitive-bash-argument,operator-exfil-argumentDeterminism. The touched file and its importers (
test_gate_tiers,metrics/test_business_counters,test_name_grant,test_hooks_raw_command_deny,test_hooks) pass with 628 tests in each of these runs: five times serially, once in reversed file order, and twice under-n 4.test_gate_tiers.pyalso passes alone.test/test_flake_pattern_ratchet.pypasses.Adjacent suites. The composition contract, crash fail-closed, deny_diff, verify_fix, tool_permission and name_grant surface suites pass (405 tests).
Static checks. black, flake8, isort, the brand and comment-history checks and semgrep are clean.
Manual verification
N/A — test-only change; the mutation proofs above show each new test can fail.
Related Issues
Part of #16536
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)