Skip to content

docs(testing): a refusal test names the rule only at that rule's own bound - #38383

Merged
bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/reject-assert-convention
Nov 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/reject-assert-convention

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Refusal tests stay green when another change adds an earlier check that also refuses the same input, so two green PRs do not merge into a red main this way.

On 2026-11-16 #37820 added a test that sent a 128-level $( ) nest and asserted the ssh floor's rule refused it. #37875 added a 32-level substitution cap to the gate. Each was green on its base; merged, the cap refused the nest first, the rule-name assertion went red on every PR's backend shard, and #37964 fixed the test. The command was refused the whole time; only the name was wrong.

Why it matters

Security code stacks checks (gate caps, floors, rules), and new caps keep landing. Each refusal test that names a rule for an input another check could also refuse is a red main waiting for the next cap.

Not a goal

What changed (motivation → approach → change)

  • Cause: a test whose point is "this input is refused" asserted which check refused it, and an input far past the named rule's bound was also past another check's bound.
  • Change: testing-conventions.md gets a section "Refusal tests". Assert the refusal alone (is_denied(x) is not None, no exception) when the point is "refused". Name the rule only at that rule's own bound, with the input asserted inside every other limit. Probe both sides of the bound. The wrong/right example is the fix(security): bound how deep the ssh floor nests its readings #37820/test: check the ssh floor's nesting bound below the gate's depth cap #37964 case.
  • Review rule: AUTOSDE.yaml tests-are-deterministic gets item 16 for the same shape. It goes into an existing rule because a new rule id must also be listed in codex-review.yml's fence table (test_ai_review_workflows_gpt.py), and this PR changes no CI config.

Backwards compatibility

Compatible: documentation and a review-rule item only.

Removes nothing: the diff only adds a spec section and a rule item.

Tests

  • The "right" example was run on main as a scratch test: 2 passed (the 128-level nest is refused, the floor refuses at _MAX_SSH_READING_DEPTH = 32 and allows 31, and 32 <= _MAX_SUBSTITUTION_DEPTH).
  • Related backend set (34 files): passed, including test_ai_review_workflows_gpt.py / _opus.py and test_prepare_pr_local_review.py (254 passed). docs-lint.sh: passed.

Proof: not provable -- docs or config only

Manual verification

N/A — documentation only; the example was executed as described above.

Related Issues

Part of #38379

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is a docs section and a review-rule item aimed at a real red-main case.

[DESIGN-REVIEWED] 77ae4a3

Description read

[DESCRIPTION-READ] 381f0d42195a0c9dc7d7a4edaf245e4bf2e1a7273afa00d93fa2eabcbf72c668

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 77ae4a3

Verdict parsed from the review's SHA-scoped output markers for commit 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 77ae4a3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 77ae4a33a7d7ab2dbe8ca1fc563165fc614931f6 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 marked this pull request as ready for review November 17, 2026 10:33
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 17, 2026 10:33
@bolichen97
bolichen97 enabled auto-merge (squash) November 17, 2026 10:34
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@bolichen97
bolichen97 merged commit f220cae into main Nov 17, 2026
100 checks passed
@bolichen97
bolichen97 deleted the crew/LinuxFlackyGuard/reject-assert-convention branch November 17, 2026 11:33
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant