Skip to content

docs: require synthetic credentials in tests that assert redaction - #37692

Merged
bolichen97 merged 1 commit into
mainfrom
crew/mate/test-convention-synthetic-credentials
Nov 15, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/mate/test-convention-synthetic-credentials

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop redaction tests from planting credential literals the redactor deliberately keeps, so two independently green PRs cannot turn main red on every platform again.

#36640 added a test that planted the AWS documentation example key AKIAIOSFODNN7EXAMPLE and asserted it was removed. #29423 had already made the structured redactor keep AWS key IDs (the documentation example key included) visible as identifiers. Each PR was green on its own base; together they made main fail on Linux and Windows until #37423 switched the tests to kiro_crew.testing.synthetic_credentials.github_classic() (issue #37421). About 27 PR runs went red in that window. Nothing in the test conventions told a test author which credential to plant.

Why it matters

A redaction test that uses a literal the redactor keeps tests detector policy, not the code under test. The next policy change or the next new test reds main for every open PR, across all platforms, with a failure that looks like a product regression.

Not a goal

  • Rewriting the existing AKIA… literals in test/. Many assert an identifier survives, and the ones asserting removal currently pass through paths that do remove them.
  • A lint or ratchet that refuses new AKIA… "not in" assertions. About 50 such assertions exist today, and test/test_flake_pattern_ratchet.py (the natural home) is a flake-class ratchet already close to the 2,000-line cap.
  • Changing redactor behaviour.

What changed (motivation → approach → change)

Symptom: main red on all platforms after #36640. Root cause: test data contradicting a documented redactor decision (the module docstring of src/kiro_crew/testing/synthetic_credentials.py already says the redactor keeps AWS key IDs). Change:

Backwards compatibility

Compatible: docs only; no code, test or config changes.

Removes nothing: the diff adds a rule and a pointer and deletes no line.

Tests

Proof: not provable -- docs or config only

Manual verification

python scripts/docs_lint.py passes ("All documentation checks passed"). check_brand_name.py and check_file_line_cap.py pass against origin/main.

Related Issues

Refs #37421, #37423, #36640, #29423

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

A test that plants the AWS documentation example key and asserts it is
removed depends on detector policy, not on the code under test: the
structured redactor keeps AWS key IDs visible as identifiers. #36640 did
this after #29423 and turned main red on every platform until #37423
(issue #37421). Record the rule in testing-conventions and point to it
from AGENTS.md.
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 20:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f67adc7076b491a7f27a9109b3f60176340f1585 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f67adc7076b491a7f27a9109b3f60176340f1585 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f67adc7076b491a7f27a9109b3f60176340f1585 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This is a docs-only rule. The helpers it names (github_classic, aws_secret) exist, and it removes nothing.

Notes

  • The AGENTS.md paragraph adds a rule to the router that also lives in testing-conventions. The router is meant to hold only rules that cause damage before a pointer is read. This one qualifies, because a single wrong literal turns main red on every platform.

[DESIGN-REVIEWED] f67adc7

Description read

[DESCRIPTION-READ] 6df57f0ffaab9a4b6cb224fc6e499e31f5016007b92b0e13fd29185806ef9974

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f67adc7076b491a7f27a9109b3f60176340f1585 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f67adc7

Verdict parsed from the review's SHA-scoped output markers for commit f67adc7076b491a7f27a9109b3f60176340f1585.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f67adc7076b491a7f27a9109b3f60176340f1585 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f67adc7076b491a7f27a9109b3f60176340f1585 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f67adc7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f67adc7076b491a7f27a9109b3f60176340f1585 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97
bolichen97 merged commit f833e0a into main Nov 15, 2026
96 checks passed
@bolichen97
bolichen97 deleted the crew/mate/test-convention-synthetic-credentials branch November 15, 2026 21:38
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant