Repository navigation
docs: require synthetic credentials in tests that assert redaction - #37692
Conversation
A test that plants the AWS documentation example key and asserts it is removed depends on detector policy, not on the code under test: the structured redactor keeps AWS key IDs visible as identifiers. #36640 did this after #29423 and turned main red on every platform until #37423 (issue #37421). Record the rule in testing-conventions and point to it from AGENTS.md.
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This is a docs-only rule. The helpers it names ( Notes
[DESIGN-REVIEWED] f67adc7 Description read[DESCRIPTION-READ] 6df57f0ffaab9a4b6cb224fc6e499e31f5016007b92b0e13fd29185806ef9974 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal: Stop redaction tests from planting credential literals the redactor deliberately keeps, so two independently green PRs cannot turn
mainred on every platform again.#36640 added a test that planted the AWS documentation example key
AKIAIOSFODNN7EXAMPLEand asserted it was removed. #29423 had already made the structured redactor keep AWS key IDs (the documentation example key included) visible as identifiers. Each PR was green on its own base; together they mademainfail on Linux and Windows until #37423 switched the tests tokiro_crew.testing.synthetic_credentials.github_classic()(issue #37421). About 27 PR runs went red in that window. Nothing in the test conventions told a test author which credential to plant.Why it matters
A redaction test that uses a literal the redactor keeps tests detector policy, not the code under test. The next policy change or the next new test reds
mainfor every open PR, across all platforms, with a failure that looks like a product regression.Not a goal
AKIA…literals intest/. Many assert an identifier survives, and the ones asserting removal currently pass through paths that do remove them.AKIA…"not in" assertions. About 50 such assertions exist today, andtest/test_flake_pattern_ratchet.py(the natural home) is a flake-class ratchet already close to the 2,000-line cap.What changed (motivation → approach → change)
Symptom:
mainred on all platforms after #36640. Root cause: test data contradicting a documented redactor decision (the module docstring ofsrc/kiro_crew/testing/synthetic_credentials.pyalready says the redactor keeps AWS key IDs). Change:docs/system-specs/common/testing-conventions.md## Rules: a test asserting a credential is REMOVED builds it with akiro_crew.testing.synthetic_credentialshelper, never anAKIA…literal; anAKIA…literal belongs only in a test asserting an identifier survives. Cites feat(redaction): scrub every persisted conversation row #36640, feat(security): turn on the structured credential detectors, with owner engine and prompt-and-logs switches #29423, test: plant a secret the redactor removes in the row-scrub tests #37423, Eight row-scrub tests are red on main: they plant an AWS key ID the redactor keeps #37421.AGENTS.mdtesting section: one short paragraph pointing to that rule.Backwards compatibility
Compatible: docs only; no code, test or config changes.
Removes nothing: the diff adds a rule and a pointer and deletes no line.
Tests
Proof: not provable -- docs or config only
Manual verification
python scripts/docs_lint.pypasses ("All documentation checks passed").check_brand_name.pyandcheck_file_line_cap.pypass againstorigin/main.Related Issues
Refs #37421, #37423, #36640, #29423
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.