Skip to content

test: check security outcomes in weak security tests, n-z - #37308

Merged
iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/weak-sec-n-z
Nov 14, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/weak-sec-n-z

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Give nine weak tests in security-area test files (basename n-z) a check on the security outcome itself (what was granted, settled, kept or left on disk), so a bug in the guarded code makes them fail.

Each of these tests ended after the call with nothing checked, or only a mock's record. The list comes from goal_scan.py weak on 66dc266de3, filtered to security-area files whose name starts n-z (77 tests; the list is kept with the cleanup tools). This PR fixes the nine that have a real outcome to read; every existing check stays as it was.

Why it matters

A security test that only proves the code ran passes when the guard is broken. Each added line checks the state the guard leaves behind, and a planted bug in the source now fails the test while main's copy still passes (table below).

Not a goal

  • No source change: only test files change.
  • No security check is loosened, and no security test is deleted or renamed.
  • The other 68 listed tests stay as main has them: most already check through a helper that raises (_refused, _kept, _steered_then_rejected, finished), or their only outcome is a call on a mocked client.

What changed (motivation → approach → change)

Two commits: the second holds only the chmod_safe check, kept apart so it rebases cleanly over #35491, which edits the same file. Every row is a stored-state or return-value check; there are no log-line or wire checks.

Test New check on the security outcome
test_scoped_trust_reaches_children.py::test_projection_never_expires_the_grant a grant with seconds left still projects as Trust
test_steer_supersedes_approval.py::test_settled_permission_pushes_the_board_once both open cards are rejected when the board is pushed
test_sweep_stale_permissions.py::test_emits_sel_audit_event the audited card is really marked resolved: stale
test_safety_override.py::test_activate_fires_callback the grant the callback announces is live afterwards
test_snapshot_redaction.py::test_a_generated_column_with_nothing_sensitive_is_not_refused nothing redacted or dropped, and the harmless row is kept
test_review_mode.py::test_clean_draft_posts_no_redaction_notice the approve claims the draft, so it cannot post twice
test_shared_template_unowned_spec.py (2 tests) a spec that passes the ceiling check is left byte-for-byte unchanged
test_platform_compat_owner_only.py::test_chmod_safe_noop_on_missing_is_safe chmod_safe does not create a missing path

goal_scan.py weak on the same base (66dc266de3): 906 before, 897 after (9 fixed).

Backwards compatibility

Compatible: test-only change; no code, behaviour or contract moves.

Removes nothing: every existing check stays.

Tests

Proof: not provable -- test-only change; the planted-bug runs below show each new check fails on a broken source while main's copy of the test passes.

Planted bug (source restored after each run) This branch main's test
live_trust_scope hides a grant with under 10 s left fails passes
the steer settlement pushes the board but leaves the cards open fails passes
the stale sweep audits a card without writing resolved back fails passes
the override callback fires, then the grant is turned off fails passes
egress redaction deletes the rows of a harmless table fails passes
review approve reads the draft instead of claiming it fails passes
the ceiling check rewrites a spec it passes fails (2) passes
chmod_safe creates a missing file before chmod fails passes

Determinism: the eight changed files pass three runs in a row with -n 2. test/test_flake_pattern_ratchet.py and test/test_wait_discipline_ratchet.py pass. flake8 and isort are clean; the black, comment-history and brand gates pass.

Manual verification

N/A — test-only change.

Related Issues

N/A — part of the weak-test cleanup.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 14, 2026 19:01
@iamwhatever
iamwhatever requested a review from buluoray November 14, 2026 19:01
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

One undeclared rider: the chmod_safe check in a file the description says this PR leaves alone.

Not checked: whether the patch's many non-test hunks are main drift; they are not in the stated PR.

Notes

  • Item 9 — undeclared: the description says "test_platform_compat_owner_only.py is in another author's open PR and is left alone", but HEAD 36e998e5 adds assert not os.path.exists(missing) there. If both PRs land, they conflict on the same test. Subtraction: remove the test_chmod_safe_noop_on_missing_is_safe hunk and leave it to the other PR.

What this change ships

Inventory (9 items) — 8 justified

Intent: make eight weak security tests fail when the guard they cover is broken (a FIX to test strength).

  1. A trust grant with seconds left must still show as Trust — justified
  2. A steer-settled push must leave both approval cards rejected — justified
  3. A stale-permission sweep must really mark the card resolved: stale — justified
  4. The safety-override callback must leave a live grant — justified
  5. Egress redaction of a harmless generated-column row must keep the row and redact nothing — justified
  6. Approving a review draft claims it, so it cannot post twice — justified
  7. The ceiling check leaves a spec that passes it byte-identical — justified
  8. The same unchanged-file check for empty grant lists — justified
  9. chmod_safe on a missing path must not create the file — undeclared, contradicts "left alone"

[FIRST-PRINCIPLES-REVIEWED] 36e998e

Description read

[DESCRIPTION-READ] 11423ca5c5b9b475372e8c6d35d0b73026b0b003243c1ea0a198aeb68380dee6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. Only test files under test/ change, so this PR adds or changes nothing a user sees.

[UX-REVIEWED] 36e998e

Description read

[DESCRIPTION-READ] 7f1a12514bb5651931db1ee98802b2ce0de3c04ec9019b92563cd3ac5217328f

That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the PR only adds assertions to existing tests, and no source or contract changes.

Notes

  • The description says it leaves test_platform_compat_owner_only.py alone ("is in another author's open PR"), but the diff changes it (commit 36e998e5) and the tables don't list that change. Either drop the hunk to avoid clashing with that other PR, or add it to the description.

[DESIGN-REVIEWED] 36e998e

Description read

[DESCRIPTION-READ] 755056cc9d0f2dbf8bc020bae28b0c7690c64c8b4879086ea89978db9880ae52

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 36e998e

Verdict parsed from the review's SHA-scoped output markers for commit 36e998e5ffccf5148c1d78f60f55cb18ac5361a0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 36e998e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 36e998e5ffccf5148c1d78f60f55cb18ac5361a0 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/weak-sec-n-z branch from fd33a64 to 3202842 Compare November 14, 2026 19:34
@iamwhatever
iamwhatever enabled auto-merge (squash) November 14, 2026 19:35
@KiroCrewCaptain

KiroCrewCaptain commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This PR changes test files only: it adds checks on security outcomes to nine weak tests, and every existing check stays. Each new check matches the source. The steer settlement sets both futures to "rejected" before the board push. The stale sweep writes exactly {request_id, resolved: stale}. Review approve removes the draft from the _review_drafts dict that the test imports. activate() leaves is_active() true, the same pattern as the existing check at line 114. live_trust_scope returns the scope when scope_remaining_secs > 0, and the sibling test with the same setup passes the policy gates. redact_bundle_for_egress neither drops memory.db nor counts any replacement for harmless rows. require_shared_template_within_ceiling is documented as reads-only. chmod_safe only calls os.chmod. No source, contract or limit changes, and no trust boundary moves.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 36e998e5ff · 58 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@iamwhatever
iamwhatever disabled auto-merge November 14, 2026 20:37
@iamwhatever
iamwhatever enabled auto-merge (squash) November 14, 2026 20:37
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (8 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; adds outcome assertions to existing tests under test/ and changes no source file.

@iamwhatever
iamwhatever merged commit 6847c91 into main Nov 14, 2026
105 of 106 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/weak-sec-n-z branch November 14, 2026 22:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants