Skip to content

test: use synthetic credentials in the remaining test fixtures - #36127

Closed
mbriones98 wants to merge 2 commits into
crew/crew-tasks/redaction-fixturesfrom
crew/crew-tasks/redaction-fixtures-b
Closed

mbriones98 wants to merge 2 commits into
crew/crew-tasks/redaction-fixturesfrom
crew/crew-tasks/redaction-fixtures-b

Conversation

@mbriones98

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the rest of the redaction tests that need a credential use a synthetic one of a real shape, so they keep testing redaction once the structured detectors keep AWS key IDs visible.

Many tests use the AWS documentation example key or another AWS access key ID as "the credential the redactor must remove". #29423 turns the structured detectors on, and they treat an AWS access key ID and the documentation example values as identifiers that stay visible. Every test that relies on them being removed would then pass or fail for the wrong reason. #31863 swaps the first half of those fixtures and adds the shared helper; this PR swaps the second half.

Why it matters

In one PR, the swaps changed 416 test files that CI's New-test stability job repeats three times. On Windows that took longer than the job's 60-minute limit, so the job could never pass. Split in two, each PR repeats about 210 files.

Not a goal

What changed (motivation → approach → change)

  • 234 test files under test/, from test/test_mcp_*.py through test/test_z*.py: fixtures that used the AWS documentation example key, or an AWS key ID, as the value to redact now use a synthetic credential from kiro_crew.testing.synthetic_credentials (added in test: use synthetic credentials in redaction test fixtures #31863), or a split literal of the same shape.
  • Where an assertion named a piece of the old key, it names the same piece of the new credential. Where the old key was glued to padding, the padding now ends at a blank, because a GitHub token is matched at a word boundary; the cut still falls inside the credential.
  • test/test_messaging_auto_title.py, test/test_workflows_service.py and test/test_queue_discard_sender_notice.py name the test variables that hold synthetic values neutrally, so CodeQL does not read them as clear-text secrets. Values and assertions are unchanged.
  • test/test_outbound_files.py and test/test_search_chat_history.py keep the tests main added to them since; only the fixture values change.

Backwards compatibility

Compatible: only test files change; no production code, config or data changes.

Removes nothing: no test is deleted and no assertion is removed.

Tests

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: test fixtures only.

Screenshots / video

N/A — no UI change.

Related Issues

Part of #22552

Stack: #31863 (shared helper and first half, base main) → this PR (second half, base #31863's branch) → #29423 (turns the switch on).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Written by the Crew Tasks Bot on behalf of mbriones98.

@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners November 12, 2026 21:53
@mbriones98
mbriones98 requested review from iamwhatever and removed request for a team November 12, 2026 21:53
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 65367eec71d87043b2c8394c4db8f51738357142 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 65367eec71d87043b2c8394c4db8f51738357142 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 65367eec71d87043b2c8394c4db8f51738357142 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This PR only changes test fixtures. It removes no test or assertion and touches no production code, so nothing blocks.

Notes

  • Some hunks do more than swap a fixture value, and "What changed" does not mention them. test_redaction_structured_identifiers.py moves the temp_path boundaries ("Wrote {v}/tmpXy12/report.html"). test_stream_redactor_aws_keyed.py adds a _structured_detectors_off fixture and a new assert body not in out. Add one line to the description for each.

[DESIGN-REVIEWED] 65367ee

Description read

[DESCRIPTION-READ] c738533d351f613e3dbfd21fa018989374f27da6590d13bad06e09bd8e85b744

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 65367eec71d87043b2c8394c4db8f51738357142 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 65367ee

Verdict parsed from the review's SHA-scoped output markers for commit 65367eec71d87043b2c8394c4db8f51738357142.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 65367eec71d87043b2c8394c4db8f51738357142 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ff2fd5ac95104fe05ed4e3db23515c2010f32326 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ff2fd5a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ff2fd5ac95104fe05ed4e3db23515c2010f32326 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 12, 2026
Matt Briones added 2 commits November 12, 2026 22:57
Tests that need a credential the redactor removes used the AWS
documentation example key or an AWS access key ID. The structured
detectors keep both visible as identifiers, so once they are on those
tests would pass or fail for the wrong reason.

This first half adds kiro_crew.testing.synthetic_credentials, which
builds synthetic credentials of real shapes at run time, and moves the
shared fixtures (test/conftest.py, test/redaction_structured_cases.py,
the fake computer-use helper) and the built-in app tests onto it, along
with test files from test_a* through test_mate*. The rest of test/
follows in a second change. Each swapped test passes with the detectors
off, as on the base, and with them on. Production code is untouched.

Where an assertion named a piece of the old key (a prefix such as
"AKIAIO", a body fragment split by markup) it now names the same piece
of the new credential. Where the old key was glued to padding, the
padding now ends at a blank, because a GitHub token is matched at a
word boundary; the cut still falls inside the credential. A test whose
premise is pass 1's own reading (a credential that eats a JSON
delimiter) checks that premise with the detectors off.

The black baseline drops one test file that is now formatted.
The second half of the fixture swap. Tests that need a credential the
redactor removes used the AWS documentation example key or an AWS
access key ID. The structured detectors keep both visible as
identifiers, so once they are on those tests would pass or fail for the
wrong reason.

These fixtures, test files from test_mcp* through test_z*, now use
synthetic credentials of real shapes from
kiro_crew.testing.synthetic_credentials, which the first half adds, or
split literals of the same shape. Each swapped test passes with the
detectors off, as on the base, and with them on. Production code is
untouched.

Where an assertion named a piece of the old key it now names the same
piece of the new credential. Where the old key was glued to padding,
the padding now ends at a blank, because a GitHub token is matched at a
word boundary; the cut still falls inside the credential.
@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 0edd187 to 3a68d2c Compare November 12, 2026 22:58
@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures-b branch from ff2fd5a to 65367ee Compare November 12, 2026 22:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 12, 2026
@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 3a68d2c to 9fee228 Compare November 13, 2026 00:11
@mbriones98

Copy link
Copy Markdown
Contributor Author

Folded back into #31863, which again carries the full fixture swap as one commit (9fee228); the split is reverted and the branch is kept.

Written by the Crew Tasks Bot on behalf of mbriones98.

@mbriones98 mbriones98 closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant