Repository navigation
test: refuse new tests asserting an AWS key ID literal is removed - #38375
Conversation
The redactor keeps AWS key IDs, so a test that plants one and asserts it is gone tracks detector policy, not the code under test (#37421). A ratchet now AST-scans test/ for such removal assertions; the 70 existing sites in 43 files are a baseline that only shrinks.
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This is a test-only ratchet. It turns an existing written rule into a gate whose baseline only shrinks, takes nothing away, and is the right size for the problem. Notes
[DESIGN-REVIEWED] f81089a Description read[DESCRIPTION-READ] 3f551ff79330b87aaf1c2b000f29ee3c6384a5fab2f025b4b04bcf3648b03178 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
899c48e to
9e84071
Compare
Review found spellings the scan skipped (a bytes literal, self.KEY) and names it resolved too widely (a local rebinding, another class's attribute). Names now resolve the way Python resolves them: function scopes shadow, class bodies are their own scope, and self.KEY reads only the enclosing class. Baseline: 81 sites in 46 files.
9e84071 to
f81089a
Compare
Problem / Motivation
Goal: A new test that asserts an AWS key ID literal is removed from a value fails CI, so the testing-conventions rule from #37692 is enforced instead of only written.
#36640 planted the AWS documentation example key and asserted it was removed, while #29423 had made the redactor keep AWS key IDs. Both PRs were green on their own base and together turned
mainred on every platform (#37421, fixed by #37423). #37692 wrote the rule down indocs/system-specs/common/testing-conventions.md, but nothing stops the next such test.Why it matters
The same collision can red
mainfor every PR again, on every platform, whenever a new removal test plants anAKIA...literal and detector policy moves. A written rule alone did not stop it.Not a goal
synthetic_credentials. They are recorded in a baseline that only shrinks.What changed (motivation → approach → change)
test/test_redaction_literal_ratchet.py(283 lines). It AST-scanstest/**/*.pyfor<literal> not in <value>andassertNotIn(<literal>, ...)where the literal (a str or bytes literal, f-string, or+of literals, or a name the same function, module or class body binds to one, read bare or asself.KEY) holdsAKIA[0-9A-Z]{16}. Asserting the key is KEPT (in) is still allowed._BASELINErecords the existing sites as exact per-file counts: 46 files, 81 sites (the "70" under Not a goal was the count before review added the bytes andself.KEYspellings). A new site fails with a message that points at the testing-conventions rule and thesynthetic_credentialshelpers. A removed site must lower its count, so the baseline only shrinks.testing-conventions.md: the rule now names the ratchet.test/test_flake_pattern_ratchet.pyis at 1,932 lines, close to the 2,000-line cap.Existing sites that pass by accident
The structured redactor keeps every AWS key ID (
redact_credentials("key AKIA...")returns the input unchanged). I swapped the literal for a plain sentinel string in every baselined file and re-ran them: 18 tests failed, so those do depend on some detector removing the key. Of the tests that still passed, these three assert nothing real:test/test_voice_reply.py::test_credential_rejoined_by_comment_strip_is_redacted:strip_markdowndoes not remove the inline<!-- keep-visible -->tag, so the key halves are never rejoined. The output is the input, and the assertion holds trivially.test/test_webex_dispatch_cards.py::test_a_queue_receipt_scans_the_text_it_quotes: the receipt sent is⏳ Folded into the reply in progress., which quotes no text at all.test/test_imessage_renderer.py::test_credential_reassembled_through_link_grammar_is_caught: delivery text isAKIA (https://x.example)IOSFODNN7EXAMPLE. The link flattening never rejoins the key, so nothing is being caught.They are left in the baseline. A follow-up should rebuild each with a
synthetic_credentialsvalue and a split that the code really rejoins.Backwards compatibility
Compatible: test-only gate plus one doc sentence; no runtime code changes.
Removes nothing: adds a test and a doc sentence only.
Tests
test_the_detector_sees_every_removal_spelling/test_the_detector_allows_a_kept_key_and_other_values: the detector's catch and allow sets.test_no_new_test_asserts_an_aws_key_id_literal_is_removed: wholetest/tree against the baseline.test_the_baseline_has_no_empty_rows.Proof: not provable -- test-only change
Ran locally (Windows): the new module and
test/test_flake_pattern_ratchet.py(75 passed); black, isort, flake8,check_file_line_cap.py,check_comment_history.py,check_brand_name.py,scripts/docs-lint.shall clean.Manual verification
N/A -- unit coverage sufficient: the gate is itself a test, and its detector is proven on fixtures.
Related Issues
Refs #37421, #37423, #37692, #36640, #29423
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.