Repository navigation
docs(testing): a refusal test names the rule only at that rule's own bound - #38383
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change is a docs section and a review-rule item aimed at a real red- [DESIGN-REVIEWED] 77ae4a3 Description read[DESCRIPTION-READ] 381f0d42195a0c9dc7d7a4edaf245e4bf2e1a7273afa00d93fa2eabcbf72c668 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal: Refusal tests stay green when another change adds an earlier check that also refuses the same input, so two green PRs do not merge into a red
mainthis way.On 2026-11-16 #37820 added a test that sent a 128-level
$( )nest and asserted the ssh floor's rule refused it. #37875 added a 32-level substitution cap to the gate. Each was green on its base; merged, the cap refused the nest first, the rule-name assertion went red on every PR's backend shard, and #37964 fixed the test. The command was refused the whole time; only the name was wrong.Why it matters
Security code stacks checks (gate caps, floors, rules), and new caps keep landing. Each refusal test that names a rule for an input another check could also refuse is a red
mainwaiting for the next cap.Not a goal
What changed (motivation → approach → change)
testing-conventions.mdgets a section "Refusal tests". Assert the refusal alone (is_denied(x) is not None, no exception) when the point is "refused". Name the rule only at that rule's own bound, with the input asserted inside every other limit. Probe both sides of the bound. The wrong/right example is the fix(security): bound how deep the ssh floor nests its readings #37820/test: check the ssh floor's nesting bound below the gate's depth cap #37964 case.AUTOSDE.yamltests-are-deterministicgets item 16 for the same shape. It goes into an existing rule because a new rule id must also be listed incodex-review.yml's fence table (test_ai_review_workflows_gpt.py), and this PR changes no CI config.Backwards compatibility
Compatible: documentation and a review-rule item only.
Removes nothing: the diff only adds a spec section and a rule item.
Tests
mainas a scratch test: 2 passed (the 128-level nest is refused, the floor refuses at_MAX_SSH_READING_DEPTH= 32 and allows 31, and 32 <=_MAX_SUBSTITUTION_DEPTH).test_ai_review_workflows_gpt.py/_opus.pyandtest_prepare_pr_local_review.py(254 passed).docs-lint.sh: passed.Proof: not provable -- docs or config only
Manual verification
N/A — documentation only; the example was executed as described above.
Related Issues
Part of #38379
Checklist