Repository navigation
test: use synthetic credentials in redaction test fixtures - #31863
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Only test fixtures change. No production code changes and nothing is taken away, so there are no design-level concerns. [DESIGN-REVIEWED] f1b78d4 Description read[DESCRIPTION-READ] 4e883a50524cf60ea2608e0123e2bbf3470af2ffbe5ce04e81855581f290b027 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The change only touches test fixtures, and its premise comes from the linked #29369/#29423 detector switch. One note: the shared constant is spelled out again inline across the diff. Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: keep the redaction tests checking real removal once the structured detectors leave AWS key IDs visible. This is preparation for a FIX (test-only).
[FIRST-PRINCIPLES-REVIEWED] f1b78d4 Description read[DESCRIPTION-READ] ce463cddf690d474ef4492fcf8de378451b09be8eb3591653dad5a0a555987fa That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
39fc9c2 to
48446d8
Compare
Sensitive change review
|
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] f1b78d4 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Sensitive Change Review
|
48446d8 to
13c77f0
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 4876 lines, more than the 400 it may approve on its own. The PR only changes tests. It swaps AWS docs-example and AWS key-ID fixtures for synthetic credentials of real shapes (GitHub classic tokens with valid checksums, 40-character AWS secrets), built at run time by a new kiro_crew.testing helper. The helper functions were moved from redaction_structured_cases.py without changes and re-imported there, so existing imports keep working. The new values match main's existing redaction patterns (gh*_ pattern, the labelled AWS-secret pattern, and the pass that catches random-looking 40-character strings). The assertions I checked are not weakened. The detectors-off fixtures do nothing on main and restore the global switch. No production code changes. Approve. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
13c77f0 to
26765f8
Compare
The credential plan now reads the structured detectors first by default.
The owner can choose main's regex passes instead (the engine selector
below), and can turn credential redaction off for the agent's prompt
and the logs (the output-seams switch below).
Plan:
- Secret hits are tagged; Identifier values (AWS key IDs, documentation
example keys, git object names, bitmap image data-URI payloads,
commit permalink SHAs, request ids, the macOS per-user temp
directory) are written as a private-use placeholder while main's
passes read the text, then restored. An SVG or other non-bitmap
data-URI payload is read by main's passes.
- ARNs, public OAuth parameter values and placeholders are reported
but not masked, so main's passes still judge them.
- Pass 1 skips the branches the detectors cover (private keys, JWTs,
the AWS secret and session-token labels), the bare AWS key ID and a
labelled AccessKeyId whose value is protected. A JWT match is
dropped only when its first segment decodes to no JSON object; a
signed token the detectors refuse (no alg, as in a Flask session
cookie, alg none, glued on its left) is still removed by pass 1. A
labelled AWS secret or session-token match is dropped only when its
value is the detectors' tag or a placeholder, so a value they never
read, or refuse for its shape (an S3-compatible secret that is not
40 characters, a short session token), is still removed. Every other
branch, including the URL userinfo branch, and passes 2-4 run
unchanged.
- An Identifier mask is dropped before the passes read the text when
the base64 run under it decodes to a credential, when it overlaps a
pass-1 credential match pass 1 still reports (a hex Bearer token, a
64-character hex value after an AWS secret label), or when it is a
bitmap payload whose bytes do not open with that format's fixed
header (a PNG IHDR, a JPEG marker, a GIF version, WEBP, a BMP header
size) and that reads as a bare secret, however long it is padded or
whatever file signature is typed in front of it.
- The rest of an Authorization: Bearer value a detector hit starts (an
SD-JWT's disclosures) is claimed with the hit, as main's single match
claims the whole value.
- A token= value a detector hit touches is claimed before the passes
read the text, its uncovered parts redacted with the token
parameter pass's own tag, warning and record.
- A JWT or JWE may follow a sentence's dot; the decoded header check
decides.
- Pass-1 warnings give the length on the original text.
Stream:
- StreamRedactor's detector cut moves back to a word start and out of
any pass-1 match, and the fixed tail hold ends at the last blank.
Past the holdback cap the floor reads the detectors' own cut, so a
long unbroken run is released and the buffer stays bounded.
- release_at_boundary() commits what only the detectors' tail hold
kept back; the dashboard calls it at a tool call or prompt cut, and
Slack at a tool card, a wait and the turn's end. The released text
is redacted like any committed window.
- Dropped AWS runs are kept in the look-behind; flush() no longer
clears the discard counter.
Engine selector:
- compat.use_rule_table() reads the owner's engine selector: "engine"
in the credential-redaction keystone, structured (default) or legacy
(main's regex passes alone). A missing field, any other value or an
unreadable record selects structured. It is read once per process;
the owner-gated PUT /api/security/credential-redaction with
{"engine": ...} audits the change, writes it without moving the
file-view switch, and republishes it to the gateway at once.
- Settings -> Security -> Redaction shows it as its own "Redaction
engine" section, apart from the file-view switch.
Owner switches:
- The engine selector (above) and a second owner switch,
seams_enabled (default on), share the credential-redaction keystone
with the file-view switch; each writer keeps the others' fields, and
each change has its own time. seams_enabled turns off the credential
pass at the output seams listed in redaction_seams.py, and only
there: the conversation replayed into the agent's prompt, and the
shape passes over log lines. The exfiltration-URL pass, the literal
vault-secret pass over log lines, every request-blocking gate and
every refusal gate stay on. Chat, transcripts, the live chat stream
and channel egress are not seams. A record without the field reads
it on, so an earlier file-view OFF keeps its meaning.
- Settings -> Security -> Redaction shows three toggles under three
headings: the file-view switch, the prompt-and-logs switch, and the
redaction engine.
Display:
- display_battery.RULE_SOURCES lists every redaction_structured file
and pass_compose.py, so a detector change gives stored verdicts a
new version. The process caches and stored verdicts carry the
engine's mark. A recorded frame keeps a field-name key under an
authorization header legible, decided by a plain name check.
Tests: the fixture swaps to synthetic credentials are in the base
branch (#31863). Here, tests whose subject is a value the design keeps
visible expect it visible. Tests that pin main's pass-1 internals run
with the switch off and say why. Three Slack golden digests follow the
stream's tail hold; the final text is unchanged. An unsigned image
data-URI payload and a link after an AWS secret label are judged as
with the switch off, and their tests expect that again. Files whose
expectations are the structured engine's output pin it with an autouse
fixture, so the suite also passes with the legacy engine selected. The
security package's line budget is raised by 197, re-measured.
## Pattern harvest
Not generalizable: the fixture swaps follow from Identifier values
being kept visible by design.
0edd187 to
3a68d2c
Compare
3a68d2c to
9fee228
Compare
|
Rebased: main moved in test/conftest.py since this head went green (0feaa8b -> 2151db2); CI re-running. The new base carries #36198's 90-minute Windows stability limit. The patch is unchanged (range-diff differs only in context lines); lint and 727 related tests pass locally. New head a220ead. Written by the Crew Tasks Bot on behalf of mbriones98. |
|
Rebased: main moved in test/test_mcp_session_report.py since this head went green (2151db2 -> 23dd50a); CI re-running. The new base carries #36457, which clears main's mypy error in src/kiro_crew/cli_commands_memory.py that failed Backend Lint on a220ead. The patch is unchanged (range-diff differs only in context lines); flake8, black, mypy and the related tests pass locally. New head fc00f1b. Written by the Crew Tasks Bot on behalf of mbriones98. |
|
Rebased: main moved in test/test_apps_routes_non_object_body.py since this head went green (23dd50a -> 6157e13); CI re-running. The new base carries #36504, which clears main's docs-lint break. The same commit swaps the AWS key ID for synthetic secrets in tests main added since: test/test_channel_cards_render.py, test/test_acp_mimocode_config.py and test/test_ui_prefs.py. GPT 6.1 and Opus 5.5 local reviews of 33a4138 both passed. New head 33a4138. Written by the Crew Tasks Bot on behalf of mbriones98. |
|
Rebased: main moved in test/test_warm_task_brief.py, test/test_warm_task_brief_facts.py, test/test_warm_task_brief_shape.py and test/test_warm_task_brief_unverified.py since this head went green (6157e13 -> 0e5b203); CI re-running. #33805 deleted the last three files and rewrote the first, so their fixture swaps were dropped and the one remaining swap was re-applied. No other file's patch changed. GPT 6.1 and Opus 5.5 local reviews of f153269 both passed. New head f153269. This re-sync follows mdbrio's re-sync budget reset, relayed by the conductor. Written by the Crew Tasks Bot on behalf of mbriones98. |
Tests that need a credential the redactor removes used the AWS documentation example key or an AWS access key ID. The structured detectors keep both visible as identifiers, so once they are on those tests would pass or fail for the wrong reason. The fixtures now use synthetic credentials of real shapes, built at run time by the new kiro_crew.testing.synthetic_credentials, or split literals of the same shape. Each swapped test passes with the detectors off, as on the base, and with them on. Production code is untouched. Where an assertion named a piece of the old key (a prefix such as "AKIAIO", a body fragment split by markup) it now names the same piece of the new credential. Where the old key was glued to padding, the padding now ends at a blank, because a GitHub token is matched at a word boundary; the cut still falls inside the credential. A test whose premise is pass 1's own reading (a credential that eats a JSON delimiter) checks that premise with the detectors off. The black baseline drops one test file that is now formatted.
Problem / Motivation
Goal: redaction tests that need a credential use a synthetic one of a real shape, so they keep testing redaction once the structured detectors keep AWS key IDs visible.
Many tests use the AWS documentation example key (
AKIAIOSFODNN7EXAMPLEand its secret-key twin) or another AWS access key ID as "the credential the redactor must remove". #29369 (merged) adds the structured detectors behind a switch that is off on main; #29423 turns them on, and the detectors treat an AWS access key ID and the documentation example values as identifiers that stay visible. Every test that relies on them being removed would then pass or fail for the wrong reason.Why it matters
Carried in #29423, these swaps take the combined diff past Design Review's 1,000,000-byte limit (it was 1,350,486 bytes folded), and they make the behaviour change hard to read. Landing them here first leaves #29423 with only the change in behaviour and the tests that describe it.
Not a goal
What changed (motivation → approach → change)
src/kiro_crew/testing/synthetic_credentials.pybuilds synthetic credentials of real shapes at run time (an AWS-shaped 40-character secret value, GitHub classic tokens with a valid checksum, a Slack bot token), so no whole credential is written on one source line.test/conftest.py's shared credential-straddle table,src/kiro_crew/testing/fake_computer_use.py's redaction fixture, and test files undersrc/kiro_crew/apps/builtins/*/tests/.main. feat(security): add structured credential detectors behind an internal switch #29369's own commit, which this branch used to carry, is dropped: its squash is on main. The swaps now also cover tests main added since this branch was cut (39 more files).AKIAIO, a body fragment split by markup), it names the same piece of the new credential. Where the old key was glued to padding, the padding now ends at a blank, because a GitHub token is matched at a word boundary; the cut still falls inside the credential. One test whose premise is pass 1's own reading (a credential that eats a JSON delimiter) checks that premise with the detectors off..github/black-baseline.txtdropstest/test_session_start_timeout_diagnostics.py, which is now black-clean.test/test_cron_preview_cmd.pyandtest/test_diagnostics.pywrite each AWS secret label and its synthetic value as separate literals, so the internal content scan does not read a labelled secret on one line.messaging/auto_title.py,workflows/library.pyanddashboard/chat_title.py) traced back to test variables holding synthetic values intest/test_messaging_auto_title.py,test/test_workflows_service.pyandtest/test_queue_discard_sender_notice.py. Those variables are renamed to neutral names; the values and assertions are unchanged.test/test_channel_cards_render.py(the sharedKEY, its split and markup pieces, and a short Slack token in the over-cap test, which needs a credential shorter than the placeholder),test/test_acp_mimocode_config.py(the logged server name) and the second credential test intest/test_ui_prefs.py.Backwards compatibility
Compatible: only tests, the test-only
kiro_crew.testinghelpers and the black baseline change; no production code, config or data changes.Removes nothing: no test is deleted and no assertion is removed; one baseline row goes because its file is now formatted. Three hunks of the earlier version edited tests main has since moved or deleted; they were dropped, and the moved copies get the same swap.
Tests
main(619d46aaab, detectors off): every test file this PR changes plus every redaction, credential, secret, security and liveness test file, 642 files: 38,269 passed, 74 skipped, 11 failed. Two of the 11 were a name this PR had renamed intest_telegram_parity.py; fixed, that file passes (104 passed). The other 9 fail the same way on untouchedmain(no boto3, an essential-source path and the design-tweak tests on the test host,test_security_conductor_verify_fix). This change adds no failure.main.scripts/docs-lint.shpass.Proof: not provable -- test-only change
Manual verification
N/A — unit coverage sufficient: test fixtures only.
Screenshots / video
N/A — no UI change.
Related Issues
Part of #22552
Stack: this PR (fixtures, base
main) → #29423 (turns the switch on, base this branch). A split into this PR and #36127 was tried and folded back; #36127 is closed.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.
Written by the Crew Tasks Bot on behalf of mbriones98.