Skip to content

fix(security): classify redaction by what a value can do (awaiting design approval) - #22887

Closed
bolichen97 wants to merge 1 commit into
mainfrom
feat/redaction-tiers-22552-combined
Closed

bolichen97 wants to merge 1 commit into
mainfrom
feat/redaction-tiers-22552-combined

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Do not merge: waiting for design approval on #22552.

Problem / Motivation

Goal: Redact every secret and never an identifier, by deciding once, per rule, what a matched value can do.

Today each redaction rule decides sensitivity on its own. AWS access key IDs are removed from every log row and CloudTrail record, so an investigator loses the value they join on. A JSON field loses its label along with its value and stops parsing. A secret behind an upper-case or camel-case label, or in an escaped form, is missed. A guess by entropy for unlabelled values removes digests, request IDs and inline images, and needs a new exemption for every false positive.

Why it matters

People investigating incidents cannot read the logs and transcripts Kiro Crew writes, because identifiers are replaced. Secrets in common spellings still reach the screen, the log and chat channels. Every new false positive costs another one-off exemption, and those exemptions have been a source of leaks themselves.

Not a goal

  • Personal data (PII) redaction.
  • Rewriting rows already stored in transcripts; they stay as they are.
  • Validating keys against a vendor (no outbound calls).
  • Redacting the model's own turn (the backend hands raw tool output to the model).
  • A per-user "show identifiers" switch.

What changed (motivation → approach → change)

The design is in #22552 (tier model, section 3; detection, section 4); the tier decision is recorded in docs/decisions/2026-10-20-redaction-classifies-by-what-a-value-can-do.md.

  • Tiers. Every rule has a tier: Secret, Bearer link or Identifier. AWS key IDs are Identifier and stay visible everywhere; a signed link loses its signature and session token, not its key ID.
  • Labels. A labelled value keeps its label and loses only the value, so {"SecretAccessKey": "[REDACTED: credential]"} still parses. Labels match in any case and spelling, after =, :, =>, an XML tag or a sibling name/value pair; generic secret fields and Authorization Basic/Bearer are covered. The log filter now applies the same Secret tier.
  • Key pairs. A 40-character secret-shaped value next to an AWS key ID is a secret.
  • Normalisation. Rules also run over a percent-decoded, JSON-unescaped view with control and invisible characters removed; spans map back to the original text. Base64 decoding covers URL-safe and wrapped runs.
  • No entropy guess. The entropy pass is removed, with every exemption that existed only for it.
  • Streaming. The stream redactor holds a value while its label, its key ID or an escape sequence may still be arriving.
  • Vendor catalog. scripts/gen_secret_catalog.py generates src/kiro_crew/data/secret_catalog.generated.json from the Betterleaks rule set (MIT, licence kept in the file) at build time. Rules that need a filter program, a file path, or are not linear-time in Python's re are skipped; a finding whose value is an identifier shape is dropped. No new runtime dependency.
  • Publication gates. The deploy scan, the design preview's file check and get_credential_patterns() read the same records and labels; a deploy finding names its rule and length, never a character of the value.

Where. Shared spellings live in src/kiro_crew/credential_patterns.py and feed src/kiro_crew/log_redaction.py and src/kiro_crew/deploy/scan.py. The key-ID copies leave src/kiro_crew/sel.py, src/kiro_crew/metrics/schema.py, src/kiro_crew/decisions/gate.py, src/kiro_crew/security_posture.py, src/kiro_crew/platform/, the built-in apps under src/kiro_crew/apps/, src/kiro_crew/computer_use/, src/kiro_crew/testing/fake_computer_use.py, and the frontend website/src/utils/sanitize.ts, website/src/utils/errorReport.ts and website/src/apps/mochi/src/shared/utils.ts (signed links keyed on the signature in website/src/utils/credentialSignature.test.ts; fixtures in website/src/test/). setup.cfg packages the catalog; .github/ drops the removed module from a path list.

This carries the work of #22757, #22736 and #22737 in one commit.

Backwards compatibility

Breaking: AWS access key IDs are no longer redacted or blocked, an unlabelled value with no prefix, label or key ID beside it is no longer redacted, and the redacted text now keeps a value's label. Writer sweep on origin/main: every in-repo caller of the removed helpers is updated in this diff (listed below); stored transcripts are unchanged, and credential records naming the old rule id still resolve.

Reader: src/kiro_crew/security/exfil.py:_oauth_url_payload_diagnostic call site -- chat -- scans with _contains_fixed_credential only; test_security.py
Reader: src/kiro_crew/security/exfil_oauth.py:_safe_oauth_parameter_name -- chat -- same; test_security.py
Reader: src/kiro_crew/security/display_safety.py:sanitize -- chat -- same; test_display_redaction_source_guard.py
Reader: src/kiro_crew/dashboard/chat_utils_display.py:_display_scope_changes_battery -- crew page -- reads only the owner-view scope; test_display_time_redaction.py
Reader: src/kiro_crew/dashboard/handlers/member_dashboard.py:_template_text_safe -- crew page -- script and style text get the same pass; test_member_dashboard_routes.py
Reader: src/kiro_crew/dashboard/file_api/project_dirs.py:_redact_project_path -- crew page -- canonical redactor; test_project_git.py
Reader: src/kiro_crew/dashboard/display_verdicts.py:_rules_fingerprint -- crew page -- new rule files are in the digest, so stored verdicts are recomputed; test_display_time_redaction.py
Reader: src/kiro_crew/security/credential_sources.py:_AWS_TEMPLATE_KEYS -- chat -- keeps the old rule id for existing records; test_credential_sources.py
Reader: src/kiro_crew/security/stream_aws_anchor.py:names_a_key -- chat -- reads the shared label table; test_stream_redactor_aws_keyed.py

Tests

  • test/test_redaction_context_rules.py: everyday inputs (CloudTrail row, assume-role output, env dump, ~/.aws/credentials, escaped JSON, percent-encoded values, CSV pair, SigV4 header, presigned S3 and CloudFront links, base64 INI, Basic header) lose every secret and keep every identifier; label shapes beyond key=value; fields that name no secret stay; stream splits mid-label, mid-URL, after each separator, across invisible characters and beside a key ID.
  • test/test_secret_catalog.py: one test per ported vendor rule with a sample generated at test time from the rule itself, the Go-to-Python translation cases, the identifier veto, and a zero-noise check over the repository's own docs.
  • test/test_redaction_tiers.py, test/test_log_redaction.py, test/test_redaction_key_id_identifier.py: tiers for the new rules, the log Secret tier, and key IDs kept.
  • Tests that pinned the entropy pass or its exemptions are removed; tests that used a bare value as "some credential" now use a labelled or prefixed fake.

Manual verification

N/A — unit coverage sufficient: the change is text-in, text-out, and every surface is covered by the tests above. An independent read of the diff looked for secrets that survive; the shapes it found are fixed and pinned in test_redaction_context_rules.py.

Screenshots / video

Why no screenshot: the frontend change only swaps which strings the shared sanitizer replaces (key IDs stay, signing values go), so no panel, layout or component renders differently.

Related Issues

Closes #22552

Pattern harvest

Rule candidate: review-prompt
Pattern: a redaction rule that decides sensitivity from a value's shape instead of reading its tier from the shared table

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/log_redaction.py

  • src/kiro_crew/security/__init__.py

  • src/kiro_crew/security/_exports.py

  • src/kiro_crew/security/credential_sources.py

  • src/kiro_crew/security/denied_rules_catalog.py

  • src/kiro_crew/security/display_safety.py

  • src/kiro_crew/security/exfil.py

  • src/kiro_crew/security/exfil_oauth.py

  • src/kiro_crew/security/exfil_url_rules.py

  • src/kiro_crew/security/pass_compose.py

  • src/kiro_crew/security/redaction.py

  • src/kiro_crew/security/redaction_decode.py

  • src/kiro_crew/security/redaction_entropy.py

  • src/kiro_crew/security/redaction_labels.py

  • src/kiro_crew/security/redaction_normalise.py

  • src/kiro_crew/security/redaction_patterns.py

  • src/kiro_crew/security/redaction_switch.py

  • src/kiro_crew/security/secret_catalog.py

  • src/kiro_crew/security/stream_aws_anchor.py

  • src/kiro_crew/security/text_redaction.py

  • src/kiro_crew/security_posture_controls.py

  • src/kiro_crew/security_posture_sinks_delivery.py

  • No reasoning template for 56d1b35 yet.

How to clear this check:

  1. The PR author posts the template below as a PR comment, with
    every field filled (N/A needs a reason). Keep the
    Head commit line: it must name the current head SHA.
  2. The check turns green within a minute, with no push.
  3. A new push resets this check: post it again for the new head.

Only the author's comment counts this way. A reviewer can instead
paste the filled template into an "Approve" review on this head.
Merging still needs a reviewer's approval.

## Sensitive change review
- Head commit: 56d1b3521cf2723d2ca70c4410e2dccaa407ed99
- What rule changed:
- Before -> after (what was allowed/blocked, now):
- Worst case if wrong (breaks commands / leaks secret / locks user out):
- Evidence checked (which sandbox result proves it):
- No regression:
  - Tools still work (which tools/commands were run, result):
  - Backward compatible (old config / old data / old callers still work, how checked):
  - Existing tests (which suites passed on this head):
- How to undo:

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 21, 2026 20:05
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of 56d1b3521cf2723d2ca70c4410e2dccaa407ed99 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

The description says the entropy guess "is removed, with every exemption that existed only for it". The diff does the opposite in the aws_control bundle scan: it widens the entropy detector to every mode and adds new false-positive exemptions to it.

What this change ships

Inventory (10 items) — 8 justified

Intent: redact every secret and never an identifier, by tier. This is a FIX, under the maintainer decision docs/decisions/2026-10-20-redaction-classifies-by-what-a-value-can-do.md.

  1. AWS key IDs stay visible in chat, logs, SEL, metrics and the frontend — justified
  2. Redacted fields keep their label, so JSON still parses — justified
  3. Labels match in any case, separator, XML tag or name/value pair — justified
  4. An unlabelled 40-character value next to a key ID is redacted — justified
  5. Percent-encoded and JSON-escaped secrets are caught — justified
  6. Shared redactor's entropy pass and code_text scope removed — justified
  7. Vendor secret catalog (Betterleaks) ships, generated at build time — justified
  8. Deploy findings show rule and length, never characters — justified
  9. Tool output over 64 KB is redacted off the event loop — undeclared
  10. Bundle scan's entropy detector now runs in canonical mode too, with digest, inline-media and URL-segment exemptions — symptom-level

Blockers

Entropy guess kept and widened in the bundle scan — symptom-level
Trigger: the description says "The entropy pass is removed, with every exemption that existed only for it". In aws_control/.../pipeline/scan.py the diff moves leaks.extend(_scan_bare_secret_runs(text, origin)) out of the standalone-only branch. It also adds _DIGEST_PREFIX_RE, _DIGEST_PADDING and _is_inline_media as exemptions to the _BARE_SECRET_ENTROPY_MIN = 4.3 check.
Outcome: one surface keeps its own entropy guess, against the decision's "No surface keeps its own copy". Each new false positive there needs another exemption, which is the harm the Goal names.
Subtraction: put _scan_bare_secret_runs back in the standalone-only branch, and delete the three exemptions added to it.
Clears when: the diff no longer runs _scan_bare_secret_runs in canonical mode and no longer adds digest or media exemptions to it.

[FIRST-PRINCIPLES-REVIEWED] 56d1b35

Description read

[DESCRIPTION-READ] 8dc2246e52ce11038832e29a3e7ecbe36611298e46a4af3c3e0f7994885fbdd6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — 🔴 changes requested (blocking)

Reviewed 41bc201bebaea15c4881fb4daf1d61b970e7e3f7 — this comment is updated in place on each push.

Blocks: the label pass now treats JS identifiers and plain prose after a secret word as credentials. Deploy scan turns these into credential findings that can't be overridden, and the member dashboard rewrites page <script> text into broken JS.

BLOCKING — src/kiro_crew/deploy/scan.py:226
_, _, records = redact_credentials_with_records(text) → findings.append(Finding(... severity="info" if placeholder else "credential"))
Trigger: a site file with login({username: user, password: pass}), const body = {email, password: pwd};, secret: s.secret || '' or <p>Your password: must be at least 8 characters long.</p> goes through _scan_tree → scan_content → a secret_label record → Finding(severity="credential") → handlers_sites.py:454 is_credential_finding. The same label pass runs in member_dashboard._template_text_safe, which now calls map_page_text(text, redact_via_context, redact_via_context) on script text too.
Outcome: on head, each of these inputs produces a hard finding, and the deploy returns 409 "Credential/security findings cannot be overridden". On base 915f63f the same inputs give no finding. On the member dashboard, redact("secret: s.secret || ''") → secret: [REDACTED: credential] and password=document.getElementById("pw").value → password=[REDACTED: credential], so the page's JS silently becomes a syntax error. Base left both unchanged.
Anchor: defect
Fix: in the label classifier, count a value as a secret only when it looks like a literal (quoted, or a single token that is not an identifier, member access, call, || expression or multi-word prose). Failing that, make a secret_label record an overridable info finding in scan_content, and keep a code-text mode for script/style text.

[BLOCK-MERGE] 41bc201
[OPUS-REVIEWED] 41bc201

Verdict parsed from the review's SHA-scoped output markers for commit 41bc201bebaea15c4881fb4daf1d61b970e7e3f7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 41bc201bebaea15c4881fb4daf1d61b970e7e3f7 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 56d1b3521cf2723d2ca70c4410e2dccaa407ed99. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- scripts/gen_secret_catalog.py:715 -- Airtable PATs escape live redaction before vendor context arrives

if m is None or not any(c in "_-0123456789" for c in m.group(1)):
Trigger: A model response streams an Airtable PAT followed by “is the Airtable personal access token.” in small deltas. This filter discards the intrinsic pat prefix, leaving the rule gated on airtable; _on_text_chunk → StreamRedactor.feed releases the token before that keyword arrives. Reproduced with chunk sizes 1 and 7.
Outcome: The complete credential reaches dashboard WebSocket/SSE clients, although batch redaction removes it from the assembled message.
Anchor: residual/security
Fix: Retain alphabetic intrinsic prefixes for self-identifying token formats such as Airtable PATs, preserve full-format validation, and regenerate the catalog.
[BLOCK-MERGE] 56d1b35
[GPT-REVIEWED] 56d1b35

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 (fenced): I'm upholding it. Nothing has to line up unusually for this to happen. The Airtable PAT rule at src/kiro_crew/data/secret_catalog.generated.json:170-174 is selected only by the keyword airtable. scripts/gen_secret_catalog.py:715 drops the letters-only pat prefix, so a PAT that arrives before the word "Airtable" (as in " is the Airtable personal access token") has no keyword to select its rule. On main, nothing detected Airtable tokens at all, so this is a gap in new coverage rather than a regression. That softens the history but isn't one of the extreme conditions a FLAG needs, and the leaked credential is still unbounded harm.

[ADJUDICATION] 56d1b3521cf2723d2ca70c4410e2dccaa407ed99 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 56d1b3521cf2723d2ca70c4410e2dccaa407ed99

[ADJUDICATION-FENCED] 56d1b3521cf2723d2ca70c4410e2dccaa407ed99 fenced=1 flagged=0
UPHOLD-FENCED F1 scripts/gen_secret_catalog.py:715 -- Writing a token before naming its vendor is ordinary prose, and the catalog rule (secret_catalog.generated.json:170-174) can only be selected by "airtable" because the alphabetic "pat" prefix is dropped, so a streamed PAT can reach clients unredacted in normal use.
[GPT-ADJUDICATED-FENCED] 56d1b3521cf2723d2ca70c4410e2dccaa407ed99

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 56d1b3521cf2723d2ca70c4410e2dccaa407ed99 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🔴 could not complete (fails closed)

No scope verdict for aadfe699790147c27d10184cf5e28828b21b9864 (the candidate stage failed). This lane fails CLOSED: an unmeasured tightening must not read as "nothing newly refused". See the job logs, then re-run -- or record /ai-review override scope aadfe699790147c27d10184cf5e28828b21b9864: <reason> if a human has judged the scope by hand.

@bolichen97
bolichen97 disabled auto-merge October 21, 2026 20:48
@bolichen97
bolichen97 force-pushed the feat/redaction-tiers-22552-combined branch from aadfe69 to ca60662 Compare October 21, 2026 21:25
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Streaming releases a secret before its following key ID arrives span=8c954628938f

fixed in ca60662: the stream holds a secret-shaped 40-character run, and any key ID, until the pairing reach after it has arrived (redaction_labels.stream_hold_start); pinned by test_a_label_or_key_id_shape_is_held_with_its_value_in_the_stream.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Explicitly labelled passwords escape redaction when shaped like identifiers span=2b43374f05cb

fixed in ca60662: behind a secret label only a value that can only name something (key ID, account ID, ARN) is kept; a UUID or hex value is redacted. Pinned by test_a_hex_or_uuid_value_behind_a_secret_label_is_a_secret.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Two-line base64 wrapping leaves recoverable secret bytes span=871d244cd480

fixed in ca60662: the wrapped-run pattern accepts one full line plus a last line, so a two-line run is claimed whole. Pinned by test_base64_wrapped_over_two_lines_is_redacted_whole.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Publication warnings expose plaintext password characters span=52346a28f400

fixed in ca60662: a record-based deploy finding is described by rule and length only, no character of the value. Pinned by test_a_deploy_finding_carries_no_character_of_a_labelled_value.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Per-match sorting stalls the gateway on credential-dense output span=50b2215ffa8d

fixed in ca60662: _claim no longer re-sorts per span; each span is checked against the already-sorted added and taken lists by bisect. 64,000 labelled rows redact in about 1.4 s, linear.

@bolichen97

bolichen97 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author
  • design_tweak preview serves labelled AWS secrets and Bearer headers span=712f553b6342

fixed in ca60662: preview_files.contains_credential also refuses any AWS-label or Authorization span from label_spans; test_a_labelled_aws_secret_or_bearer_header_is_refused pins both shapes.

@bolichen97

bolichen97 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Stream anchors miss the no-separator label form and a secret before its key ID span=8d4983ffcc76

fixed in ca60662: the stream holds from a label word, a no-separator AWS label, a name/value pair or a secret-shaped run until its reach arrives; pinned at chunk sizes 1-64 by test_a_label_or_key_id_shape_is_held_with_its_value_in_the_stream.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • product-shape change without accepted RFC span=bfc88d73796a

needs-a-decision: the tier model (key IDs visible, no entropy guess) is the design under review in #22552, and this PR is held for that design approval. Only a maintainer can record it, by an RFC on main or /ai-review override first-principles <head>; the agent does not post this override.
Note on item 9 (catalog duplicates hand-written prefixes): kept on purpose. The hand-written alternation is the pre-filtered fast path that the stream holdback, the log floor and get_credential_patterns() read; catalog rules are keyword- and entropy-gated, regenerated per upstream release and may be skipped, so they add coverage and do not replace that floor.

@bolichen97
bolichen97 force-pushed the feat/redaction-tiers-22552-combined branch from ca60662 to 6a1aab9 Compare October 21, 2026 22:39
@bolichen97

bolichen97 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Records repeat a label the text now keeps span=be5fce1c4c0f

fixed in 6a1aab9: a record carries no label now that the label stays in the cleaned text, so a card or a copy shows the field name once (test_redaction_credential_records).

@bolichen97

bolichen97 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Type annotations and attribute reads behind a label are redacted span=61e11dd40c95

fixed in 6a1aab9: an unquoted value that is a call, a subscript or type parameter, a union, a dotted, constant, variable or type name is code, not a value; the AWS label branch exempts calls, subscripts and dotted reads (test_source_code_that_names_or_reads_a_secret_is_kept).

@bolichen97

bolichen97 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 11fbc2d. Fixed on 9adca1b.

F1 (credential_values.py:123), blank lines in quoted YAML: fixed. _continued_quote_line treats blank lines inside a quoted scalar as part of it: they fold to a line break. The next line that has text decides whether the value goes on. A buffer that ends inside blank lines keeps the value open, so the stream holds it. Test: test_a_blank_line_inside_a_quoted_yaml_value_does_not_end_it.

F2 (credential_values.py:160), XML values over lines: fixed.

  • xml_value_end reads an element's content to its closing </ across line breaks, and also when the content starts on the next line.
  • In the stream, open_xml_value and the xml-value drop span lines too.
  • A closing tag held by the stream (</password>) is held together with the element it closes, so the content is never committed apart from its tag.
  • Test: test_an_xml_value_over_lines_is_a_literal (batch, stream 1/7/64, log, deploy hard finding).

F3 (credential_patterns.py:236), triple-quoted literals: fixed. The separator reads """/''' as one opener (TRIPLE_OR_QUOTE). value_extent reads a triple-quoted value over lines to its closing triple quote, also from the stream trigger's single-quote view. Test: test_a_triple_quoted_password_is_read_to_its_triple_quote (single-line and multi-line, the line after kept).

F4 (credential_values.py:568), encoded config lines: fixed.

  • A normalised view is now read with its first line on the source line its window starts on (reading_view). A view line is therefore a config line exactly where the source line is one, or where a decoded %0A starts one.
  • Inside a view, a \ does not continue a line, since it may be a decoded escape, and an & ends the value, since it joins query parameters.
  • A label mid-line (?q=password%3Dnew_password) is still code.
  • The prefilter oracle in test_credential_prefilter.py now uses the same reader and joins touching spans, as the real view pass does.
  • Tests: test_an_encoded_config_line_is_still_a_config_literal and test_an_encoded_label_mid_line_is_still_code.

Other lanes on 11fbc2d:

  • CodeQL passed.
  • First Principles is the product-shape BLOCK that the owner's override covers on the final head.
  • Opus failed on infra on both attempts: "Discovery produced no [OPUS-DISCOVERY] marker". It reruns on this head.

Verification: the backend lint gates pass. The local battery passed 27,431, and the earlier CI-failing modules passed 626.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 9adca1b. Fixed on 2550983.

F1 (credential_patterns.py:138), next-line values containing : or =: fixed.

  • VALUE_SPACE stops at a line only when that line is a real YAML mapping key (key: then a blank or the line end) or an unindented dotenv NAME= line.
  • A : or = inside a scalar (AbCd1234:efGh5678, AbCd=1234) is the value.
  • secret: then secretName: my-tls is still a mapping.
  • Test: test_a_next_line_value_with_a_colon_or_equals_is_still_the_value (batch, stream, log, deploy hard finding).

F2 (credential_values.py:657), folded plain YAML values: fixed.

  • On a YAML config line, folded_plain_end reads the value on through the lines YAML folds into it. Each such line is indented deeper than the key's line and is not a key, a list item or a comment. Blank lines between them fold too.
  • Dotenv lines do not fold.
  • The stream's open_config_value holds a YAML value from its key while only its continuation or blank lines follow.
  • Tests: test_a_folded_plain_yaml_value_is_read_through_its_lines (sizes 1/7/64, siblings kept) and test_a_list_item_or_a_dotenv_line_does_not_fold.

F3 (scan.py:297), standalone media bypass: fixed. In standalone mode, a base64 run that decodes to binary is no longer skipped. Its printable runs (a PNG tEXt chunk, EXIF, XMP) are read by the same _HARD_PATTERNS, so a labelled secret in an inline image's metadata is found. Test: test_a_secret_in_an_inline_image_text_chunk_is_found_without_the_redactor.

Other lanes on 9adca1b:

  • First Principles is the product-shape BLOCK that the owner override covers on the final head.
  • Opus failed again before any review: "Discovery produced no [OPUS-DISCOVERY] marker". This is its fifth infra failure in a row.

Verification: the backend lint gates pass. The local battery passed 27,442, the earlier CI-failing modules plus the prefilter oracle passed 1,035, and the line budget was re-measured at 34,771.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 2550983. Fixed on a7238fa, which is also rebased onto fresh main: #22887 had gone CONFLICTING.

F1 (text_redaction.py:225), long values past the stream's hold: fixed. When a value runs past the hold, the stream drops it through the value's own close:

  • quoted-dq and quoted-sq go on across a line break when the next line is indented or blank, as a YAML quoted scalar folds.
  • A triple-quoted value drops as quoted-triple-dq / quoted-triple-sq through its closing triple quote, over lines.
  • A CDATA value drops as xml-cdata through its ]]>, past any < it holds. The drop is over once the ]]> is taken.
  • A line break, a quote or two, or a ] or two at the buffer end stays in the drop until the next chunk shows which it is.
  • Batch: value_extent past QUOTED_READ_MAX still reads an indented continuation line. The log filter uses the same reader.
  • Test: test_a_long_value_past_the_hold_drops_through_its_own_close (a 5,100-character value in YAML double and single quotes, a triple quote, and CDATA; chunk sizes 1, 64 and 1024; batch and log; the text after the value kept).

Rebase:

  • One conflict: main raised the security package line budget by 1, to 32,527, for member-records. Both notes are kept, and the budget is re-measured on the rebased tree at 34,803 (+2,276 from 32,527).
  • 2550983 was also pushed with the budget at a placeholder (33,000). This head corrects it.

Lanes on 2550983:

  • First Principles is the product-shape BLOCK that the owner override covers on the final head.
  • Opus failed again before reviewing (no [OPUS-DISCOVERY] marker, on both attempts).

Verification: the backend lint gates pass. The local battery passed 27,480. The earlier CI-failing modules, the prefilter oracle and the two files main touched passed 1,226.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on a7238fa. Fixed on 5d8d178.

F1 (credential_values.py:756), quoted config values behind weak labels: fixed.

  • A quoted value on a config line (DB_PASS="AbCdEfG", db_pass: 'AbCdEfG') is a literal, as its unquoted form is.
  • quoted_config_value uses the same line, key and separator test as config_value_end, now shared as _config_line_start.
  • Batch (label_spans) and the log floor (_on_config_line) both read it.
  • The weak-label length and word tests do not apply to such a value.
  • Off a config line, a quoted value keeps the weak-label tests: {"pwd": "deny"} and x db_pass="AbCdEfG".
  • Tests: test_a_quoted_config_value_behind_a_weak_label_is_a_literal (batch, stream, log, deploy hard finding) and test_a_quoted_word_off_a_config_line_keeps_the_weak_label_tests.
  • One existing case moved: in test_code_that_names_a_secret_is_left_alone, lane_pass: '\u7eff\u706f' alone on a line is now a config literal. It now sits in code, const lanes = {lane_pass: '…'}, where the weak-label test still keeps it.

Lanes on a7238fa:

  • First Principles is the product-shape BLOCK that the owner override covers on the final head.
  • Opus failed again before reviewing, with no [OPUS-DISCOVERY] marker.

Verification: the backend lint gates pass. The local battery passed 27,487; its one failure was the case moved above, which now passes. The earlier CI-failing modules and the prefilter oracle passed 1,035. The line budget was re-measured at 34,807.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 5d8d178. Fixed on 903e7a2.

F1 (credential_values.py:786), short quoted passwords behind weak labels: fixed. A quoted value behind a weak label (db_pass, pwd) now keeps only the word test. The length test is gone. So {"db_pass": "Ab1$cD"} and {"pwd": "x9!Q"} are redacted in batch, stream and log, while {"pwd": "deny"}, {"pwd": "on"} and a word of a script with no letter case (\u7eff\u706f) are kept.

Tests: test_a_short_quoted_password_behind_a_weak_label_is_redacted and test_a_quoted_word_behind_a_weak_label_is_kept. In last round's test_a_quoted_word_off_a_config_line_keeps_the_weak_label_tests, the off-line case is now a word (x db_pass="deny"); a mixed-case non-word there is now a secret.

Lanes on 5d8d178:

  • First Principles: the product-shape BLOCK, which the owner override covers on the final head.
  • Opus: failed again before reviewing (no [OPUS-DISCOVERY] marker).

Verification: the backend lint gates pass. The local battery passed 27,495. The earlier CI-failing modules and the prefilter oracle passed 1,035.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 903e7a2. Fixed on c51a175.

F1 (redaction_labels.py:1034, credential_values.py:704), long folded YAML values: fixed.

  • No line caps remain. folded_plain_end reads a plain YAML value through every line folded into it, up to its indentation boundary. The quoted-scalar reader past QUOTED_READ_MAX does the same.
  • open_config_value looks back over every continuation line in the held buffer for the value's key.
  • Past the hold, a YAML config value drops as block-N (N = its key's indent): the rest of its line, then every line indented deeper or blank, the same as a block scalar's body. It is never released line by line.
  • Test: test_a_long_folded_yaml_value_keeps_its_label_to_its_end (70 lines of about 100 characters each, chunk sizes 7, 64 and 1024, batch and log; the next key is kept).

F2 (credential_values.py:357), vendor Identifier tier behind a generic label: fixed.

  • names_vendor_identifier returns whether a value is, whole, the value of an Identifier-tier catalog rule whose keyword is in its context (twilio-api-key: SK<32 hex> beside twilio).
  • label_spans keeps such a value behind a generic label, so deploy gives no finding.
  • The same string behind password:, with no vendor keyword, is still a secret.
  • Tests: test_a_vendor_identifier_keeps_its_tier_behind_a_generic_label and test_the_same_string_behind_a_password_label_is_still_a_secret.

Lanes on 903e7a2:

  • First Principles is the product-shape BLOCK that the owner override covers on the final head.
  • Opus again failed before reviewing (no [OPUS-DISCOVERY] marker).

Verification: the backend lint gates pass. The local battery passed 27,501 and the earlier CI-failing modules passed 1,035. The line budget was re-measured at 34,847.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on c51a175. Fixed on dea0017.

F1 (credential_patterns.py:152), backtick-quoted values: fixed.

  • The separator takes a backtick as an opening quote (TRIPLE_OR_QUOTE), and the value grammar opens after it.
  • value_extent reads a template literal over lines to its unescaped closing backtick.
  • The batch reader, the log floor's quote test and the stream's label trigger all accept it.
  • Test: test_a_backtick_password_is_read_to_its_closing_backtick (single-line and multi-line with an escaped backtick, sizes 1 and 7, batch and log).

F2 (redaction_labels.py:121), name/value fields in any order: fixed.

  • _object_name_value_spans reads every quoted value field of the flat object that holds a secret-naming name field, in any field order and with any fields between them (CodeBuild's sorted name, type, value; or value before name). Each one gets the name's rule, so the AWS name keeps its blocking tier and deploy gives a hard finding.
  • The stream holds a value field while its object is still open and within reach, so a name that follows it is read with it.
  • A value beside an ordinary name (STAGE) is kept.
  • Tests: test_a_name_and_value_pair_in_any_order_is_one_secret (both orders, sorted JSON, sizes 1/7/64, deploy) and test_a_value_beside_an_ordinary_name_is_kept.

Lanes on c51a175:

  • First Principles is the product-shape BLOCK, which the owner override covers on the final head.
  • Opus again failed before reviewing, with no [OPUS-DISCOVERY] marker.

Verification: backend lint gates pass. Local battery 27,512 passed; earlier CI-failing modules 1,035 passed. Line budget re-measured at 34,897.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on dea0017. Fixed on 1274e13.

F1 (text_redaction.py:500), a value streamed before its secret name: fixed. The stream's name hold now starts at a value field earlier in the same object ({"value":"...", "name":"DB_PASSWORD"}). The whole pair reaches one window once the object closes, and the stream no longer cuts between them. Test: test_a_value_before_its_secret_name_never_streams (opaque values with blanks, punctuation and braces; chunk sizes 1, 3, 7 and 64).

F2 (redaction_labels.py:149), braces inside quoted values: fixed. _object_bounds and the stream's _same_object find an object's edges with strings skipped, so a { or } inside a quoted value is no boundary. Each object keeps its own name. Tests: the ab{cd}ef case above, and test_each_object_keeps_its_own_name.

F3 (redaction_labels.py:146), upper-case fields: fixed. The pre-check is now case-insensitive ("value" in text.lower()), the same as the matchers. Test: test_upper_case_sibling_fields_are_one_secret, which includes a deploy hard finding.

Lanes on dea0017: First Principles is the product-shape BLOCK that the owner override covers on the final head.

Verification: the backend lint gates pass. The local battery passed 27,534, and the earlier CI-failing modules passed 1,035. The line budget was re-measured at 34,965.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for CI, CodeQL and GPT 6.1 on 1274e13. Fixed on 1e2b431.

CI Backend Tests (3.12, 3) and (Windows, 3): fixed. test/test_dashboard_elicitation.py::test_display_redaction_touches_labels_and_taints_values used AKIAIOSFODNN7EXAMPLE1234 as its "secret". A key ID alone is now Identifier tier and stays visible, so the test now uses a Secret-tier token (glpat-…). It passes on main only because main still redacts key IDs. It recurred because my local battery did not include that file. The battery now also runs every test module that mentions AKIA or a redactor entry point: 294 modules, 20,932 passed.

CodeQL (1 new high): fixed. test/test_markdown_link_parentheses.py:90 re.match(r"\[\^(?:\\.|[^\]])*\]", …) could backtrack exponentially on \\ runs, because [^\]] also matched the backslash that \\. consumes. The class now excludes the backslash: (?:\\.|[^\]\\])*. The line predates this PR, but CodeQL counts it as new because this PR edits the file (its samples moved from AKIA… to glpat-…).

GPT F1 (redaction_labels.py:622), sibling fields far apart in the stream: fixed.

  • open_object_field holds a secret-naming name field, or a quoted value field, of an object still open at the buffer end, as far as the batch rule's object reach. The hold is like a quoted value's.
  • object_pair_extents keeps a pair already whole in the buffer from being cut between its fields, by the natural cut or by the 512 floor.
  • Test: test_sibling_fields_far_apart_never_stream_apart (an 800-character field between them, both orders, chunk sizes 7, 64 and 1024).

GPT F2 (text_redaction.py:1230), long backtick values: fixed. A backtick value past the hold now drops as quoted-bt: over lines, through its unescaped closing backtick. Test: test_a_long_backtick_value_drops_through_its_closing_backtick.

Verification:

  • The backend lint gates pass.
  • The local battery passed 27,543. The 294 redaction-touching modules passed 20,932.
  • The line budget was re-measured at 35,021.

@bolichen97

bolichen97 commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 1e2b431. Fixed on a5358d7.

F1 (credential_values.py:235), YAML list-item values swallowing sibling fields: fixed.

  • key_indent gives the column a key sits at with sequence indicators counted. In - password: |, the key is at column 2, so its body is the lines indented deeper than 2, and username: alice ends it.
  • A key's own opening quote ( "password": |) counts as part of the key. Text before an earlier quote ({"c": "- password: |) is the string the YAML sits in, so it is not counted.
  • Every reader uses it: block scalars, escaped block scalars, folded plain values, quoted continuations, the stream's open_block_scalar and block_scalar_header, and the config drop past the hold.
  • The batch and stream readers also read the part of the line the stream already emitted (LINE_PREFIX), so a - committed earlier still counts.
  • Test: test_a_list_item_value_ends_at_its_sibling_field (block, folded, quoted, escaped and quoted-key forms; sizes 1/7/64; batch, stream and log; the sibling field is kept).

Verification:

  • The backend lint gates pass.
  • The battery now includes every module that mentions AKIA or a redactor. It passed 42,008.
  • The line budget was re-measured at 35,024.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT on a5358d7, fixed on f4b2c33:

  • F1 (credential_values.py, folded plain YAML indent read from a later quoted field): fixed. folded_plain_end now takes the config key's own start and measures the indent from the line start to that key, so a later quoted field on the line cannot set it. Red/green: test_a_later_quoted_field_does_not_set_a_folded_value_indent.
  • F2 (credential_patterns.py, a triple-quoted value whose quote is followed by a newline was not read): fixed. LABEL_VALUE_OR_QUOTED accepts an empty match right after """ or ''' at end of line, so the value body on the next lines is consumed through its close. Red/green: test_a_triple_quoted_value_opening_on_the_next_line_is_read.

Local gate (black, isort, flake8, mypy, repo checks, comment-history, semgrep) and the 294-module redaction battery (42,017 passed) are green.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for Opus 5.5 on f4b2c33, fixed on 6448fce:

  • BLOCKING 1 (redaction_labels.py, _object_bounds rescanned the object reach once per name match): fixed at the root. _named_objects finds every secret-naming name field's flat object in one string-aware pass over the text (braces, quotes, escapes), then clamps to _OBJECT_REACH as before. Both callers (_object_name_value_spans, object_pair_extents) use it. 1,000 entries (31 KB): 1.8 s → 0.06 s; 8,000 (248 KB): 17 s → 0.55 s. Test: test_many_secret_named_objects_are_read_in_linear_time, plus test_each_object_keeps_its_own_fields (nested object, brace inside a string, value-before-name, sibling object stays).
  • BLOCKING 2 (deploy/scan.py, <your-api-key> / ******** hard-blocked): fixed as suggested. is_documented_placeholder also takes a value that fully matches REFERENCE_VALUE, so it is the overridable credential-placeholder finding. A <![CDATA[...]]> value is excluded, since it holds the value itself. Deploy severity for real values is unchanged.
  • The prose trigger (Set api_key: <your-api-key> in your config.) had a second cause: the XML > separator read api-key> in as an element holding in. A > followed by a blank on the same line now separates only when the element closes on that line (<password> v </password>). <password>v</password> and multi-line elements are unchanged.

Red on f4b2c33, green now: test/test_redaction_reference_and_object_scale.py. Local gate and the redaction battery (42,029 passed) are green.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 and Opus 5.5 on 6448fce, on fde20b7 (rebased onto main 2e0b123: one test conflict in test_cron_script.py, where the notify test now uses a Slack bot token, since a key ID alone is Identifier tier).

GPT F1 (redaction_normalise.py:50, a password with %20 plus %2C/%3B loses its tail): fixed. The blank-and-delimiter decode view keeps encoded blanks, quotes, backslashes, commas and semicolons together, and still decodes &, so the value is read whole and ends at the next parameter. Red/green: test_an_encoded_password_with_a_blank_and_a_delimiter_is_redacted_whole (batch, stream, deploy).

GPT F2 (sibling readers skip the Twilio SK veto): fixed with the shared predicate. names_vendor_identifier takes the sibling name as vendor context, and both sibling readers (_NAME_VALUE_RE, _object_name_value_spans) apply it, in either field order. Red/green: test_a_twilio_key_sid_in_a_sibling_value_keeps_its_identifier_tier (batch, stream, log, deploy), plus test_a_sibling_secret_beside_a_twilio_name_is_still_redacted.

Opus B3 (pass/pwd/token false positives, URL @ and #): fixed.

  • On a key: value line, a weak label (_pass, pwd, _token, and now refresh token, first pass) keeps the weak word, length and path tests and reads one token. A NAME=value dotenv line stays a literal (DB_PASS=hunter is still redacted), and a non-word value behind a weak colon label still goes (first_pass: Zq9vT2mXw8).
  • A labelled value ends at the host's @ when its label is a URL's user name, and at # in a query.
  • Red/green: test_a_status_or_path_behind_a_weak_label_stays, test_a_secret_behind_the_same_labels_is_still_redacted, test_a_url_password_ends_at_the_host_and_a_query_value_at_the_fragment.
  • The log filter reads no encoded views and masks URL user info its own way, as on main. Those two tests cover batch and stream.

Opus B1 (AKIA/ASIA key IDs visible): rebutted. #22552 section 3 / Q1: a cloud key ID is Identifier tier. It names a key, is sent in cleartext in every SigV4 request and logged by CloudTrail, and cannot authenticate without the secret access key, so it is never redacted. The hard deploy verdict follows the same tier. Owner-approved design; see #22552 comment 6301939162 and decision record #22577. No code change.

Opus B2 (a bare unlabelled 40-character secret is not caught after pass 3 was removed): rebutted. #22552 section 4: the entropy guess for a bare secret is removed, not narrowed. Case-insensitive labels and key-ID pairing are the control: an unlabelled 40-character value next to a key ID is a Secret. A secret pasted alone, with no label and no key ID, is the accepted residual. Owner-approved design; see #22552 comment 6301939162 and decision record #22577. No code change.

Local gate and the 294-module redaction battery (42,073 passed) are green.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on fde20b7, fixed on 6342533 (rebased onto main 4c1536c: main moved the warm-brief shape tests to test_warm_task_brief_shape.py, so its three key-ID samples are now a GitLab token there).

GPT F1 (redaction_labels.py:1248, an ANSI-coloured YAML block label streams its body): fixed, with two causes.

  • open_block_scalar now also reads the normalised views (terminal codes and escapes dropped) and maps the label back to the buffer, as stream_hold_start and open_quoted_start do.
  • A glued [_.-]pass in GENERIC_SECRET_LABEL matched first inside db_password, so a label scan found _pass with no separator, skipped password, and held nothing. [_.-]pass now gives way to password, passwd and passphrase. db_pass: is still a label.
  • Red/green: test_a_block_scalar_behind_a_coloured_label_streams_redacted (\x1b[33mpassword\x1b[0m, \x1b[1;33mdb_password\x1b[0m, pass\u0077ord; chunks of 1, 7 and 64; batch too) and test_a_glued_pass_gives_way_to_the_whole_label_word.

The Opus 5.5 run on fde20b7 failed in its discovery step (outcome=skipped) and posted no review, so this is infra, with nothing to dispose of. Local gate and the redaction battery (42,264 passed) are green.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition for GPT 6.1 on 6342533, fixed on 203eb13:

GPT F1 (credential_values.py:843, password: input.value, in a JS object hard-blocks a deploy): fixed at the root. A key: value line inside an open { (in_open_brace: braces in single-line strings skipped, read back 4,096 characters, cost per brace) is an object property in code, not a YAML line. config_value_end no longer reads it as a config literal, so its unquoted value goes through the shared code-expression tests, as the same property on one line already did ({password: input.value}).

Literals stay redacted:

  • a quoted or plain literal in the same object (password: 'Zq9…', password: Zq9…)
  • YAML under balanced Helm {{ }} lines
  • a NAME=value assignment inside a shell function's braces, since only the colon form is affected

Red/green: test_an_object_property_expression_in_code_is_no_config_literal (4 cases, batch and deploy), test_a_literal_beside_or_outside_an_object_is_still_redacted, test_many_config_lines_after_braces_stay_fast. Local gate and the redaction battery (42,274 passed) are green.

Comment thread src/kiro_crew/credential_values.py Fixed
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Padded sibling fields bypass credential redaction span=3c8f5b1d0a9e

fixed in 3465071. Root cause: the name/value object-field patterns capped the blanks around : at eight, while the shared label grammar reads any run. All three field patterns now use the shared separator. The stream also sent "name" before its : arrived, so a hold now covers a name/value key whose separator is still arriving. Red/green tests in test/test_redaction_reference_and_object_scale.py (padded name/value, both orders, batch and stream at chunk sizes 1/7/64).

Also on this head: the Fast Gate flake-ratchet red came from a wall-clock bound in that same test file; the bound is removed and only the correctness assertions remain.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Escaped sibling-field objects release passwords before their names arrive span=8c954628938f

fixed in ce6a792. Root cause: the stream's value-field hold read only quotes as written, so {\"value\": \"... was never held, and the pair extents read only the raw buffer. The hold pattern accepts JSON-escaped quotes (and an escape still arriving), and both object holds also run on each decoded view, mapped back to the buffer (_object_holds). Test: test_an_escaped_value_first_object_holds_the_value_for_its_name (3 field orders, 1 and 2 escape levels, chunks 1/2/7).

  • Closing an escaped YAML block releases its password suffix span=8c954628938f

fixed in ce6a792. Root cause: the natural cut was repaired against labelled spans only when a label sat near the cut. When the buffer holds escaped lines, the cut is now repaired against the batch and decoded-view label spans whatever the label's distance. Test: test_an_escaped_yaml_block_keeps_its_whole_value_when_it_closes (; , &, chunks 1/7).

  • A client_secret UUID that follows a client_id label is printed in the clear span=c3f638376b60

fixed in ce6a792. names_vendor_identifier vetoes only when the rule's keyword is in the label that holds the value and that label names no secret (secret, pass, pwd, token, private, credential). Tests: test_an_identifier_label_earlier_on_the_line_never_vouches_for_a_secret, test_an_azure_client_id_behind_its_own_label_keeps_its_tier; the Twilio SID tests still pass.

  • The deploy scan hard-blocks ordinary code and docs that contain sig= span=09cedbf06b4e

fixed in ce6a792. A signing value that is an interpolation or reference (${sig}, <signature>), a documented placeholder (YOUR_SAS_SIGNATURE) or a redaction tag is an overridable credential-placeholder finding (a tag is skipped), on both the fixed-pattern and the request_signature record paths. A real signing value stays a hard finding. Tests: test_a_signing_parameter_written_without_a_value_is_no_hard_finding, test_a_real_signing_value_is_still_a_hard_finding.



def _holding_label(text: str, start: int) -> str:
match = _HOLDING_LABEL_RE.search(text[max(0, start - 96) : start])
@bolichen97

Copy link
Copy Markdown
Contributor Author

Both findings fixed on ab259497e4df121f9260ab57292eb1469bff0767.

  • Normalization windows omit distant sibling credential fields span=50b2215ffa8d

Fixed. redaction_labels.VIEW_PAD = max(QUOTED_HOLD_REACH, _OBJECT_REACH) is the one pad for every decoded-view window, shared by redaction._view_spans and the blocking scans (redaction_decode.view_texts), so a sibling value anywhere within the object rule's reach is in the same window as its escaped name. Test: test_a_distant_sibling_value_is_read_in_the_decoded_view (gaps 800/2000/3500; redaction and deploy scan), red on the judged head, green now. No measured slowdown (escaped JSON, 1 MB percent text, ANSI and object corpora).

  • ANSI-colored sibling fields release secrets onto the live stream span=8c954628938f

Fixed. The \\/%/& shortcut in text_redaction._object_holds is removed, so the object holds always read the normalised views (ANSI escapes and invisible characters dropped). Test: test_an_ansi_coloured_value_key_holds_its_sibling_secret_in_the_stream (chunks 1/7/64), red on the judged head, green now; the views are memoised, so the stream cost is unchanged.

@bolichen97

Copy link
Copy Markdown
Contributor Author

All three findings fixed on 435921e94a634d6a97b4d153ada0afbde35a461d, through the shared predicate, so batch, stream, log filter and deploy scan agree.

  • Explicit Authorization headers expose short lowercase credentials span=0aec0a4a35a8

Fixed. After an explicit Authorization: header and scheme, AUTHORIZATION_VALUE is any non-empty b64token run; the short single-case word exemption is gone. A reference ($TOKEN, <token>) falls outside the character set and stays. Test: test_a_short_header_word_a_compound_pass_and_a_dotted_ini_key_are_redacted (Bearer abcdef, Basic ABCDEFG); test_a_short_authorization_credential_is_redacted now expects token/TOKEN redacted.

  • Compound password fields expose lowercase passwords span=860810596181

Fixed. is_weak_label(..., quoted=True) treats pass after an account or store word (db_pass, admin.pass, mysql_root_pass) as a strong label, read with the text before the matched label tail. Status names (lane_pass: 'green', first_pass: completed) keep the weak tests. Same test (db_pass, admin.pass cases).

  • Dotted INI password keys lose literal-value protection span=860810596181

Fixed. _config_key_start reads . as part of the key, as _INI_KEY_SEP_RE does, so db.password = ... in an INI section is a config line and its value a literal. Same test ([database] case). Code x self.password = new_password stays visible (test_the_word_reference_and_code_exemptions_still_hold).

Red 5 on the judged head, green 9 now. Full battery 42,085 passed (one unrelated push-guard timeout).

@bolichen97

Copy link
Copy Markdown
Contributor Author

Both findings fixed on 82e030d6bc5bc91d663c6046a9e350593db3319d. No override, including for the flagged edge case.

  • Value-based lookup suppresses a real credential finding span=52346a28f400

Fixed at the root. CredentialMatch now carries start, the value's source offset from the redaction plan, and deploy/scan.scan_content places and deduplicates each record by that offset; the text.find(rec.value, ...) lookup is gone, so an earlier placeholder copy of the same word can no longer absorb a real password. Tests: test_a_password_naming_an_earlier_placeholder_word_is_a_hard_deploy_finding, test_every_record_names_where_its_value_starts.

  • Unquoted YAML compound-password fields expose literal passwords span=2b43374f05cb

Fixed. An account or store compound password label (db_pass, admin_pass, mysql_root_pass; credential_values._PASS_TAIL_RE) is strong in every form, quoted or not. Both readers (redaction_labels label spans and the weak_colon config check, and the log filter) pass the full label with the text before the matched tail. Status labels keep the weak tests. Tests: test_an_unquoted_compound_password_is_redacted (batch, log, stream 1/7/64, deploy), test_a_status_pass_label_and_a_working_directory_stay_visible (first_pass: completed, lane_pass: green, pwd: /usr/local).

Red 5 on the judged head, green 8 now. Full battery 42,093 passed (one unrelated push-guard timeout).

@bolichen97

Copy link
Copy Markdown
Contributor Author

Disposition for GPT 6.1 on 82e030d6bc and Opus 5.5 on 435921e94a (Opus on 82e030d6bc failed at discovery, no review). Fixed in 41bc201bebaea15c4881fb4daf1d61b970e7e3f7.

Finding Status Fix
GPT F1: non-leading logfmt password read as code fixed _config_line_start accepts a key after logfmt fields (_LOGFMT_PREFIX_RE, = form only), so the value is a config literal and ends at the next field. f(a=1, password=new_password) stays code.
GPT F2: https:// exempted as a store reference fixed _STORE_REFERENCE_RE is a closed list of secret-store schemes (vault://, op://, ssm://, ...); a web URL behind a secret label is redacted.
GPT F3: narrowed log Bearer matcher fixed Main's rule restored (any 8+ token68 token), minus a closed list of prose words (Bearer authentication failed).
GPT F4: absolute wall-clock assert fixed Removed large_s < 3.0; only the relative comparison stays.
Opus 1: name/value fields rescanned per name (quadratic) fixed _object_pairs pairs each value field once with its nearest containing name on each side. 8 KB stream 0.77 s, 33 KB 2.98 s (linear). Test counts judgements: at most 2 per value.
Opus 2: escaped nested JSON token in the stream already fixed Stream and batch both give \"token\": \"[REDACTED: credential]\" on this head.
Opus 3: AKIA key IDs visible rebutted Owner-approved design, #22552 section 3 (cloud key IDs are Identifier tier); see #22552 comment 6301939162 and decision record #22577.

Tests: test/test_redaction_logfmt_store_and_bearer.py (13 cases; 6 fail on 82e030d6bc, all pass now). Battery: 42,107 passed. Local lint gates, mypy, semgrep and the flake ratchet pass.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Disposition for GPT 6.1 on 41bc201beb. Fixed in 89eac543c2badbd4af3ff670ff6ede12f176cc89.

Finding Status Fix
F1: tool_output_fingerprints placed the section by find(m.value) fixed It uses the recorded offset m.start (search only when no offset is recorded). Test: test_a_value_repeated_in_an_earlier_section_takes_its_own_section (fails on 41bc201beb, passes now).

Opus 5.5 on 41bc201beb (plain words and code after a secret word in JS and prose) is with the maintainer for a decision; it challenges the recorded fail-closed label design and the q2 deploy-severity ruling.

Battery: 42,107 passed; one push-guard timeout outside this PR (passes alone).

@bolichen97

Copy link
Copy Markdown
Contributor Author

Disposition for GPT 6.1 on 89eac543c2. Both findings fixed in 9a5c48f0f697e0886001f7c6f35aa616874f665d, no override.

Finding Status Fix
F1: an escaped \" before the value read as closing a caption literal fixed closes_label_literal compares the escaping of both quotes in raw text (an escaped value quote inside a plain string opens a value). In a decoded view, where escapes are gone, a quote closes a caption only when code punctuation alone follows it (getpass("Password:")).
F2: global (?i:false) dropped vendor keys that contain FALSE fixed The generator anchors upstream's placeholder-word filter to whole values (^(?i:true|false|null)$); catalog regenerated from betterleaks @ 81aff7a638.

Tests in test/test_redaction_logfmt_store_and_bearer.py: 3 cases fail on 89eac543c2, all 19 pass now; captions serialised as JSON stay. Battery: 42,114 passed.

Every redaction rule now carries a tier (Secret, Bearer link, Identifier),
decided once by what a value can do, not guessed from its shape.

- AWS access key IDs (AKIA/ASIA) are Identifier tier and stay visible on
  every surface; a signed link loses its signature and session token, not
  its key ID.
- Labelled values keep their label and lose only the value
  (`api_key = [REDACTED: credential]`), so structured output still parses.
  Labels match case-insensitively with the usual word separators, cover
  generic secret fields and Authorization Bearer/Basic, and are shared with
  the log filter, which now redacts the Secret tier too.
- A 40-character secret-shaped value next to an AWS key ID is the secret
  half of the pair; the key ID stays.
- Every rule also runs over a normalised view (percent-decoded,
  JSON-unescaped, control and invisible characters dropped), with spans
  mapped back to the original text. Base64 decoding covers URL-safe and
  wrapped runs.
- The entropy guess for unlabelled values is removed, with every exemption
  that existed only for it (document and diagram links, commit permalinks,
  macOS temp-dir ids, the code-text scope).
- The stream redactor holds a value with any secret label while it may
  still be arriving.
- A vendor token catalog is generated at build time from the Betterleaks
  rule set (MIT) by scripts/gen_secret_catalog.py into
  src/kiro_crew/data/secret_catalog.generated.json; a finding whose value
  is an identifier shape is dropped. No new runtime dependency.

Closes #22552
@bolichen97

Copy link
Copy Markdown
Contributor Author

Disposition for GPT on 9a5c48f, Opus on 41bc201 and CodeQL on 9a5c48f. Fixed in 56d1b35.

GPT F1 (fixed): a percent-encoded signed link (next=...%26sig%3D...) kept its signature. The normalised-view scan in redaction._view_spans now also reads signing and session-token parameters (_VIEW_SIGNING_PARAM_RE, _view_signing_spans) and maps them back to the source. The parameter after the signature stays. Tests: test_a_percent_encoded_sas_signature_is_redacted (3 shapes, batch and stream at chunk sizes 1, 7 and 64), test_the_parameter_after_an_encoded_signature_stays; 4 fail on 9a5c48f. The log filter already redacted these.

CodeQL (fixed): two polynomial-regex alerts.

  • credential_values.py:784: the quoted-run regex used to skip quotes while counting braces is replaced by _braces_outside_quotes, a linear scan with the same result (20,000 random inputs agree with the old regex). Test: test_brace_counting_skips_quotes_in_linear_time.
  • secret_catalog.py:271: _HOLDING_LABEL_RE now starts only at a word boundary, so it is not retried inside a run of -.

Opus, per the maintainer ruling:

  • password=document.getElementById("pw").value; inside an open {: fixed narrowly. It is code only when all hold: inside an open JS { block, the line ends in ;, and the value is a member chain or call (_js_statement_in_block). The stream now carries the emitted text's braces (BRACE_PREFIX), so it agrees with batch at every chunk size. Tests: test_a_js_member_chain_statement_in_a_block_is_kept kept; test_a_shell_function_or_dotenv_value_stays_redacted hides f() { PASSWORD=hunter2; } and PASSWORD=x.y.
  • login({username: user, password: pass}) and <p>Your password: must ...</p>: rebutted. Behind a strong label a plain word is a literal and is redacted at any length (F1). Owner-approved design; see Redaction: classify by what a value can do (Secret / Bearer link / Identifier) #22552 comment 6301939162 and decision record docs(security): record the redaction tiers and bring the docs in line with the code #22577. No code change.
  • Deploy severity: not lowered. Recorded as an owner decision item; the agent posts no override.

Checks: battery 42,123 passed; lint, mypy, semgrep and the flake ratchet pass.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Closed by the owner (bolichen97): cancelled for now. The branch feat/redaction-tiers-22552-combined is kept; the work can be reopened from it later. Tracking stays on #22552.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blocked Waiting on another issue or external system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redaction: classify by what a value can do (Secret / Bearer link / Identifier)

3 participants