Repository navigation
test: check security outcomes in weak security tests, n-z - #37308
Conversation
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One undeclared rider: the Not checked: whether the patch's many non-test hunks are main drift; they are not in the stated PR. Notes
What this change shipsInventory (9 items) — 8 justifiedIntent: make eight weak security tests fail when the guard they cover is broken (a FIX to test strength).
[FIRST-PRINCIPLES-REVIEWED] 36e998e Description read[DESCRIPTION-READ] 11423ca5c5b9b475372e8c6d35d0b73026b0b003243c1ea0a198aeb68380dee6 That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. Only test files under [UX-REVIEWED] 36e998e Description read[DESCRIPTION-READ] 7f1a12514bb5651931db1ee98802b2ce0de3c04ec9019b92563cd3ac5217328f That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the PR only adds assertions to existing tests, and no source or contract changes. Notes
[DESIGN-REVIEWED] 36e998e Description read[DESCRIPTION-READ] 755056cc9d0f2dbf8bc020bae28b0c7690c64c8b4879086ea89978db9880ae52 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
fd33a64 to
3202842
Compare
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This PR changes test files only: it adds checks on security outcomes to nine weak tests, and every existing check stays. Each new check matches the source. The steer settlement sets both futures to "rejected" before the board push. The stale sweep writes exactly {request_id, resolved: stale}. Review approve removes the draft from the _review_drafts dict that the test imports. activate() leaves is_active() true, the same pattern as the existing check at line 114. live_trust_scope returns the scope when scope_remaining_secs > 0, and the sibling test with the same setup passes the policy gates. redact_bundle_for_egress neither drops memory.db nor counts any replacement for harmless rows. require_shared_template_within_ceiling is documented as reads-only. chmod_safe only calls os.chmod. No source, contract or limit changes, and no trust boundary moves. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (8 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; adds outcome assertions to existing tests under test/ and changes no source file.
Problem / Motivation
Goal: Give nine weak tests in security-area test files (basename n-z) a check on the security outcome itself (what was granted, settled, kept or left on disk), so a bug in the guarded code makes them fail.
Each of these tests ended after the call with nothing checked, or only a mock's record. The list comes from
goal_scan.py weakon66dc266de3, filtered to security-area files whose name starts n-z (77 tests; the list is kept with the cleanup tools). This PR fixes the nine that have a real outcome to read; every existing check stays as it was.Why it matters
A security test that only proves the code ran passes when the guard is broken. Each added line checks the state the guard leaves behind, and a planted bug in the source now fails the test while main's copy still passes (table below).
Not a goal
_refused,_kept,_steered_then_rejected,finished), or their only outcome is a call on a mocked client.What changed (motivation → approach → change)
Two commits: the second holds only the
chmod_safecheck, kept apart so it rebases cleanly over #35491, which edits the same file. Every row is a stored-state or return-value check; there are no log-line or wire checks.test_scoped_trust_reaches_children.py::test_projection_never_expires_the_granttest_steer_supersedes_approval.py::test_settled_permission_pushes_the_board_oncetest_sweep_stale_permissions.py::test_emits_sel_audit_eventresolved: staletest_safety_override.py::test_activate_fires_callbacktest_snapshot_redaction.py::test_a_generated_column_with_nothing_sensitive_is_not_refusedtest_review_mode.py::test_clean_draft_posts_no_redaction_noticetest_shared_template_unowned_spec.py(2 tests)test_platform_compat_owner_only.py::test_chmod_safe_noop_on_missing_is_safechmod_safedoes not create a missing pathgoal_scan.py weakon the same base (66dc266de3): 906 before, 897 after (9 fixed).Backwards compatibility
Compatible: test-only change; no code, behaviour or contract moves.
Removes nothing: every existing check stays.
Tests
Proof: not provable -- test-only change; the planted-bug runs below show each new check fails on a broken source while main's copy of the test passes.
live_trust_scopehides a grant with under 10 s leftresolvedbackchmod_safecreates a missing file before chmodDeterminism: the eight changed files pass three runs in a row with
-n 2.test/test_flake_pattern_ratchet.pyandtest/test_wait_discipline_ratchet.pypass. flake8 and isort are clean; the black, comment-history and brand gates pass.Manual verification
N/A — test-only change.
Related Issues
N/A — part of the weak-test cleanup.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.