Skip to content

test: use synthetic credentials in redaction test fixtures - #31863

Merged
bolichen97 merged 1 commit into
mainfrom
crew/crew-tasks/redaction-fixtures
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/crew-tasks/redaction-fixtures

Conversation

@mbriones98

@mbriones98 mbriones98 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: redaction tests that need a credential use a synthetic one of a real shape, so they keep testing redaction once the structured detectors keep AWS key IDs visible.

Many tests use the AWS documentation example key (AKIAIOSFODNN7EXAMPLE and its secret-key twin) or another AWS access key ID as "the credential the redactor must remove". #29369 (merged) adds the structured detectors behind a switch that is off on main; #29423 turns them on, and the detectors treat an AWS access key ID and the documentation example values as identifiers that stay visible. Every test that relies on them being removed would then pass or fail for the wrong reason.

Why it matters

Carried in #29423, these swaps take the combined diff past Design Review's 1,000,000-byte limit (it was 1,350,486 bytes folded), and they make the behaviour change hard to read. Landing them here first leaves #29423 with only the change in behaviour and the tests that describe it.

Not a goal

What changed (motivation → approach → change)

  • New src/kiro_crew/testing/synthetic_credentials.py builds synthetic credentials of real shapes at run time (an AWS-shaped 40-character secret value, GitHub classic tokens with a valid checksum, a Slack bot token), so no whole credential is written on one source line.
  • 480 files: test fixtures that used the AWS documentation example key, or an AWS key ID, as the value to redact now use a synthetic credential from that module, or a split literal of the same shape. This includes test/conftest.py's shared credential-straddle table, src/kiro_crew/testing/fake_computer_use.py's redaction fixture, and test files under src/kiro_crew/apps/builtins/*/tests/.
  • Rebased onto current main. feat(security): add structured credential detectors behind an internal switch #29369's own commit, which this branch used to carry, is dropped: its squash is on main. The swaps now also cover tests main added since this branch was cut (39 more files).
  • Where an assertion named a piece of the old key (a prefix such as AKIAIO, a body fragment split by markup), it names the same piece of the new credential. Where the old key was glued to padding, the padding now ends at a blank, because a GitHub token is matched at a word boundary; the cut still falls inside the credential. One test whose premise is pass 1's own reading (a credential that eats a JSON delimiter) checks that premise with the detectors off.
  • .github/black-baseline.txt drops test/test_session_start_timeout_diagnostics.py, which is now black-clean.
  • test/test_cron_preview_cmd.py and test/test_diagnostics.py write each AWS secret label and its synthetic value as separate literals, so the internal content scan does not read a labelled secret on one line.
  • CodeQL's alerts on this PR (clear-text logging and storage in messaging/auto_title.py, workflows/library.py and dashboard/chat_title.py) traced back to test variables holding synthetic values in test/test_messaging_auto_title.py, test/test_workflows_service.py and test/test_queue_discard_sender_notice.py. Those variables are renamed to neutral names; the values and assertions are unchanged.
  • Tests main added later get the same swap: test/test_channel_cards_render.py (the shared KEY, its split and markup pieces, and a short Slack token in the over-cap test, which needs a credential shorter than the placeholder), test/test_acp_mimocode_config.py (the logged server name) and the second credential test in test/test_ui_prefs.py.

Backwards compatibility

Compatible: only tests, the test-only kiro_crew.testing helpers and the black baseline change; no production code, config or data changes.

Removes nothing: no test is deleted and no assertion is removed; one baseline row goes because its file is now formatted. Three hunks of the earlier version edited tests main has since moved or deleted; they were dropped, and the moved copies get the same swap.

Tests

  • On main (619d46aaab, detectors off): every test file this PR changes plus every redaction, credential, secret, security and liveness test file, 642 files: 38,269 passed, 74 skipped, 11 failed. Two of the 11 were a name this PR had renamed in test_telegram_parity.py; fixed, that file passes (104 passed). The other 9 fail the same way on untouched main (no boto3, an essential-source path and the design-tweak tests on the test host, test_security_conductor_verify_fix). This change adds no failure.
  • With feat(security): turn on the structured credential detectors, with owner engine and prompt-and-logs switches #29423 on top (detectors on), 802 files: 49,294 passed, 39 failed, all of which fail the same way on untouched main.
  • flake8, isort, the black gate, mypy and scripts/docs-lint.sh pass.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: test fixtures only.

Screenshots / video

N/A — no UI change.

Related Issues

Part of #22552

Stack: this PR (fixtures, base main) → #29423 (turns the switch on, base this branch). A split into this PR and #36127 was tried and folded back; #36127 is closed.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Written by the Crew Tasks Bot on behalf of mbriones98.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f1b78d415f3458854a7db31d90451c544d1ca9b7 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f1b78d415f3458854a7db31d90451c544d1ca9b7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Only test fixtures change. No production code changes and nothing is taken away, so there are no design-level concerns.

[DESIGN-REVIEWED] f1b78d4

Description read

[DESCRIPTION-READ] 4e883a50524cf60ea2608e0123e2bbf3470af2ffbe5ce04e81855581f290b027

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of f1b78d415f3458854a7db31d90451c544d1ca9b7 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The change only touches test fixtures, and its premise comes from the linked #29369/#29423 detector switch. One note: the shared constant is spelled out again inline across the diff.

Notes

  • Item 2 — duplicate of src/kiro_crew/testing/synthetic_credentials.py:SAMPLE_AWS_SECRET: grepping added lines for Qh4Tm2Vx9KpRc finds 54. 9 of them exactly re-spell "Qh4Tm2Vx9KpRc/Z8Nw3Hd/" + "Jf1Lb5Gs0Ye9Ua2WkE", including fake_computer_use.py:FAKE_CREDENTIAL_FIXTURE in the same package. Subtraction: replace those 9 exact copies with an import of SAMPLE_AWS_SECRET.

What this change ships

Inventory (5 items) — 4 justified

Intent: keep the redaction tests checking real removal once the structured detectors leave AWS key IDs visible. This is preparation for a FIX (test-only).

  1. A new test helper builds synthetic credentials of real shapes at run time (GitHub checksum tokens, AWS secret, Slack token) — justified
  2. About 480 test files swap the AWS documentation example key or key IDs for synthetic credentials — duplicate of SAMPLE_AWS_SECRET: the literal is re-spelled inline 9 times
  3. The redaction fixture in fake_computer_use.py now holds a synthetic secret instead of the documentation key — justified
  4. Test variables in three files are renamed to neutral names to clear CodeQL alerts — justified
  5. The black baseline drops one file that is now black-clean — justified

[FIRST-PRINCIPLES-REVIEWED] f1b78d4

Description read

[DESCRIPTION-READ] ce463cddf690d474ef4492fcf8de378451b09be8eb3591653dad5a0a555987fa

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f1b78d415f3458854a7db31d90451c544d1ca9b7 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f1b78d4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f1b78d415f3458854a7db31d90451c544d1ca9b7 false-positive: <reason, 20 to 500 characters>

@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 39fc9c2 to 48446d8 Compare November 6, 2026 23:58
@mbriones98

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 48446d8
  • What rule changed: No redaction rule changes for any user. Every change is behind the credential rule table's internal switch (redaction_rules.compat.use_rule_table()), which is off and has no config key, CLI flag or API route. With the switch on, the table's rows and detectors, StreamRedactor's hold logic, the display caches and recorded-frame key handling are corrected so the table matches the fixed-pattern pass where it should (see the PR description's table).
  • Before -> after (what was allowed/blocked, now): Switch off: unchanged for every surface; display-cache keys are byte-identical to the base, and recorded frames take the base path. Switch on: ordinary text the table wrongly redacted (agent names, max_token, secret:// references, permission values, <hidden ...> placeholders, a bare session outside a cookie) is kept; values it wrongly kept (a link token after a dot, a URL password holding ', an AWS field split across lines, a token on the line after a bare Bearer, a password equal to a scheme word) are redacted; streamed output equals the batch result for every chunking of the covered shapes.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Nothing reachable while the switch is off. With it on, a stream/batch mismatch would show a value the batch pass hides, and a slow reader would stall the gateway loop; both are covered by tests below.
  • Evidence checked (which sandbox result proves it): Three new test files turn the switch on: 104 of their cases fail on the base and pass here. The stream file compares stream and batch output across fixed chunk sizes and every two-chunk cut, and counts newline-index work to show name/value reading is linear (8,000 unquoted pairs: 0.21 s, from 51.75 s before the last fix). Local GPT 6.1 and Opus 5.5 reviews of this head report no blocking findings after the repairs.
  • No regression:
    • Tools still work (which tools/commands were run, result): No tool or command handling changed. Recorded frames keep the base output with the switch off (88 key and ancestor combinations compared against the base).
    • Backward compatible (old config / old data / old callers still work, how checked): No config key, schema or stored data added. With the switch off the engine mark is empty, so stored display verdicts and cache keys are unchanged.
    • Existing tests (which suites passed on this head): Switch off, 979 passed across the redaction, stream-parity, compat, display-battery, display, chat-utils, frame-record, security-liveness and new suites; switch on, 898 passed. black, isort, flake8, mypy, the file line cap and Semgrep 1.78.0 with CI's packs pass.
  • How to undo: Revert this single commit. No config, schema or data migration is involved.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f1b78d415f3458854a7db31d90451c544d1ca9b7 — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • The module docstring at src/kiro_crew/testing/fake_computer_use.py:25 still calls the credential fixture "the well-known public documentation example key", but line 96 now holds a synthetic value.

[OPUS-REVIEWED] f1b78d4

Verdict parsed from the review's SHA-scoped output markers for commit f1b78d415f3458854a7db31d90451c544d1ca9b7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f1b78d415f3458854a7db31d90451c544d1ca9b7 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 9, 2026
@mbriones98
mbriones98 marked this pull request as ready for review November 9, 2026 18:20
@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners November 9, 2026 18:20
@mbriones98
mbriones98 requested review from tim-finnigan and removed request for a team November 9, 2026 18:20
Base automatically changed from mdbrio/redaction-cutover-v2 to main November 11, 2026 19:09
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

  • No sensitive path changed; nothing to check.

@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 48446d8 to 13c77f0 Compare November 12, 2026 02:05
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 12, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 4876 lines, more than the 400 it may approve on its own.

The PR only changes tests. It swaps AWS docs-example and AWS key-ID fixtures for synthetic credentials of real shapes (GitHub classic tokens with valid checksums, 40-character AWS secrets), built at run time by a new kiro_crew.testing helper. The helper functions were moved from redaction_structured_cases.py without changes and re-imported there, so existing imports keep working. The new values match main's existing redaction patterns (gh*_ pattern, the labelled AWS-secret pattern, and the pass that catches random-looking 40-character strings). The assertions I checked are not weakened. The detectors-off fixtures do nothing on main and restore the global switch. No production code changes. Approve.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit f1b78d415f · 4876 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 13c77f0 to 26765f8 Compare November 12, 2026 03:16
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
mbriones98 pushed a commit that referenced this pull request Nov 12, 2026
The credential plan now reads the structured detectors first by default.
The owner can choose main's regex passes instead (the engine selector
below), and can turn credential redaction off for the agent's prompt
and the logs (the output-seams switch below).

Plan:
- Secret hits are tagged; Identifier values (AWS key IDs, documentation
  example keys, git object names, bitmap image data-URI payloads,
  commit permalink SHAs, request ids, the macOS per-user temp
  directory) are written as a private-use placeholder while main's
  passes read the text, then restored. An SVG or other non-bitmap
  data-URI payload is read by main's passes.
- ARNs, public OAuth parameter values and placeholders are reported
  but not masked, so main's passes still judge them.
- Pass 1 skips the branches the detectors cover (private keys, JWTs,
  the AWS secret and session-token labels), the bare AWS key ID and a
  labelled AccessKeyId whose value is protected. A JWT match is
  dropped only when its first segment decodes to no JSON object; a
  signed token the detectors refuse (no alg, as in a Flask session
  cookie, alg none, glued on its left) is still removed by pass 1. A
  labelled AWS secret or session-token match is dropped only when its
  value is the detectors' tag or a placeholder, so a value they never
  read, or refuse for its shape (an S3-compatible secret that is not
  40 characters, a short session token), is still removed. Every other
  branch, including the URL userinfo branch, and passes 2-4 run
  unchanged.
- An Identifier mask is dropped before the passes read the text when
  the base64 run under it decodes to a credential, when it overlaps a
  pass-1 credential match pass 1 still reports (a hex Bearer token, a
  64-character hex value after an AWS secret label), or when it is a
  bitmap payload whose bytes do not open with that format's fixed
  header (a PNG IHDR, a JPEG marker, a GIF version, WEBP, a BMP header
  size) and that reads as a bare secret, however long it is padded or
  whatever file signature is typed in front of it.
- The rest of an Authorization: Bearer value a detector hit starts (an
  SD-JWT's disclosures) is claimed with the hit, as main's single match
  claims the whole value.
- A token= value a detector hit touches is claimed before the passes
  read the text, its uncovered parts redacted with the token
  parameter pass's own tag, warning and record.
- A JWT or JWE may follow a sentence's dot; the decoded header check
  decides.
- Pass-1 warnings give the length on the original text.

Stream:
- StreamRedactor's detector cut moves back to a word start and out of
  any pass-1 match, and the fixed tail hold ends at the last blank.
  Past the holdback cap the floor reads the detectors' own cut, so a
  long unbroken run is released and the buffer stays bounded.
- release_at_boundary() commits what only the detectors' tail hold
  kept back; the dashboard calls it at a tool call or prompt cut, and
  Slack at a tool card, a wait and the turn's end. The released text
  is redacted like any committed window.
- Dropped AWS runs are kept in the look-behind; flush() no longer
  clears the discard counter.

Engine selector:
- compat.use_rule_table() reads the owner's engine selector: "engine"
  in the credential-redaction keystone, structured (default) or legacy
  (main's regex passes alone). A missing field, any other value or an
  unreadable record selects structured. It is read once per process;
  the owner-gated PUT /api/security/credential-redaction with
  {"engine": ...} audits the change, writes it without moving the
  file-view switch, and republishes it to the gateway at once.
- Settings -> Security -> Redaction shows it as its own "Redaction
  engine" section, apart from the file-view switch.

Owner switches:
- The engine selector (above) and a second owner switch,
  seams_enabled (default on), share the credential-redaction keystone
  with the file-view switch; each writer keeps the others' fields, and
  each change has its own time. seams_enabled turns off the credential
  pass at the output seams listed in redaction_seams.py, and only
  there: the conversation replayed into the agent's prompt, and the
  shape passes over log lines. The exfiltration-URL pass, the literal
  vault-secret pass over log lines, every request-blocking gate and
  every refusal gate stay on. Chat, transcripts, the live chat stream
  and channel egress are not seams. A record without the field reads
  it on, so an earlier file-view OFF keeps its meaning.
- Settings -> Security -> Redaction shows three toggles under three
  headings: the file-view switch, the prompt-and-logs switch, and the
  redaction engine.

Display:
- display_battery.RULE_SOURCES lists every redaction_structured file
  and pass_compose.py, so a detector change gives stored verdicts a
  new version. The process caches and stored verdicts carry the
  engine's mark. A recorded frame keeps a field-name key under an
  authorization header legible, decided by a plain name check.

Tests: the fixture swaps to synthetic credentials are in the base
branch (#31863). Here, tests whose subject is a value the design keeps
visible expect it visible. Tests that pin main's pass-1 internals run
with the switch off and say why. Three Slack golden digests follow the
stream's tail hold; the final text is unchanged. An unsigned image
data-URI payload and a link after an AWS secret label are judged as
with the switch off, and their tests expect that again. Files whose
expectations are the structured engine's output pin it with an autouse
fixture, so the suite also passes with the legacy engine selected. The
security package's line budget is raised by 197, re-measured.

## Pattern harvest

Not generalizable: the fixture swaps follow from Identifier values
being kept visible by design.
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 0edd187 to 3a68d2c Compare November 12, 2026 22:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@mbriones98
mbriones98 force-pushed the crew/crew-tasks/redaction-fixtures branch from 3a68d2c to 9fee228 Compare November 13, 2026 00:11
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 13, 2026
@mbriones98

mbriones98 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/conftest.py since this head went green (0feaa8b -> 2151db2); CI re-running. The new base carries #36198's 90-minute Windows stability limit. The patch is unchanged (range-diff differs only in context lines); lint and 727 related tests pass locally. New head a220ead.

Written by the Crew Tasks Bot on behalf of mbriones98.

@mbriones98

mbriones98 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_mcp_session_report.py since this head went green (2151db2 -> 23dd50a); CI re-running. The new base carries #36457, which clears main's mypy error in src/kiro_crew/cli_commands_memory.py that failed Backend Lint on a220ead. The patch is unchanged (range-diff differs only in context lines); flake8, black, mypy and the related tests pass locally. New head fc00f1b.

Written by the Crew Tasks Bot on behalf of mbriones98.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_apps_routes_non_object_body.py since this head went green (23dd50a -> 6157e13); CI re-running. The new base carries #36504, which clears main's docs-lint break. The same commit swaps the AWS key ID for synthetic secrets in tests main added since: test/test_channel_cards_render.py, test/test_acp_mimocode_config.py and test/test_ui_prefs.py. GPT 6.1 and Opus 5.5 local reviews of 33a4138 both passed. New head 33a4138.

Written by the Crew Tasks Bot on behalf of mbriones98.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_warm_task_brief.py, test/test_warm_task_brief_facts.py, test/test_warm_task_brief_shape.py and test/test_warm_task_brief_unverified.py since this head went green (6157e13 -> 0e5b203); CI re-running. #33805 deleted the last three files and rewrote the first, so their fixture swaps were dropped and the one remaining swap was re-applied. No other file's patch changed. GPT 6.1 and Opus 5.5 local reviews of f153269 both passed. New head f153269. This re-sync follows mdbrio's re-sync budget reset, relayed by the conductor.

Written by the Crew Tasks Bot on behalf of mbriones98.

Tests that need a credential the redactor removes used the AWS
documentation example key or an AWS access key ID. The structured
detectors keep both visible as identifiers, so once they are on those
tests would pass or fail for the wrong reason.

The fixtures now use synthetic credentials of real shapes, built at run
time by the new kiro_crew.testing.synthetic_credentials, or split
literals of the same shape. Each swapped test passes with the detectors
off, as on the base, and with them on. Production code is untouched.

Where an assertion named a piece of the old key (a prefix such as
"AKIAIO", a body fragment split by markup) it now names the same piece
of the new credential. Where the old key was glued to padding, the
padding now ends at a blank, because a GitHub token is matched at a
word boundary; the cut still falls inside the credential. A test whose
premise is pass 1's own reading (a credential that eats a JSON
delimiter) checks that premise with the detectors off.

The black baseline drops one test file that is now formatted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants