Skip to content

fix(security): scan preview and deploy content for credentials in linear time - #21388

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/21006-pem-scan-callers
Oct 20, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/21006-pem-scan-callers

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Checking a previewed file or a deploy snapshot for credentials takes time that grows in proportion to its size, even when the text repeats a PEM header many times.

The preview check (contains_credential) and the deploy scan (scan_content) ran the raw pattern from get_credential_patterns() over file content. Its PEM branch ends in a lazy full-block tail that reads to the end of the text when no END marker follows, so a text that repeats -----BEGIN PRIVATE KEY----- paid one full read per header. Through the preview check, N = 2000 / 4000 / 8000 headers took 0.64 / 2.50 / 10.04 s.

Fixes #21006

Why it matters

A file in a previewed project, or in a folder someone deploys, can stall the process with input that holds no valid key. The regex holds the GIL while it runs.

Not a goal

What changed (motivation -> approach -> change)

The linear scanner from #20790 (security/pem_scan.py, reached through redaction._credential_matches) already finds the same matches as the raw pattern without letting a header that no END marker follows reach the full-block tail. Both callers now iterate that scanner instead of the raw pattern. contains_credential keeps its PEM_MARKER_RE / PEM_BODY_RE check on each hit, and scan_content keeps its findings cap and its own header pattern.

One difference: the shared scanner also drops a JWT-shaped hit whose first segment is not a JSON object, the same filter redaction applies. A real JWT is still found.

Backwards compatibility

Compatible: the same PEM blocks are refused and the same quoted markers are allowed; the only text that stops matching is a JWT-shaped string whose first segment is not JSON.

Removes nothing: no function, key or path is removed or renamed; get_credential_patterns() is unchanged and still exported.

Tests

test/test_pem_scan_callers_linear.py counts how many times the full-block pattern runs, so it does not depend on the clock. It checks that the preview never runs a repeated header through the full-block tail, that the deploy scan takes its matches from the shared scanner once per scan, that a real key block is refused by both callers (also after many headers), that the preview allows a quoted marker with no key body, that the deploy scan still flags a bare marker, and that other credentials are still found. A 3 MiB case guards against a hang. On main, 7 of these fail; on this branch all pass.

Also run and passing: test_design_tweak_backend.py, the deploy scan test files, test_redaction_pem_scan_linear.py, test_denied_root_parts_shared.py. flake8, isort and mypy are clean on the changed files.

Manual verification

N/A - unit coverage sufficient. Measured the preview check and the deploy scan on -----BEGIN PRIVATE KEY-----\n repeated 2000 / 4000 / 8000 times: preview 0.64 / 2.50 / 10.04 s before, 0.00 / 0.01 / 0.02 s after.

Screenshots / video

N/A - no user-visible UI change.

Related Issues

Fixes #21006

Follow-ups for the remaining callers of the raw pattern (deferred, out of this PR's scope):

Pattern harvest

Rule candidate: review-prompt
Pattern: a regex whose tail is a lazy scan to a closing marker, run once per opening marker, is quadratic; other callers of get_credential_patterns() (the artifact-deploy skill scripts) still carry it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The fix patches two callers rather than the root cause: get_credential_patterns() still hands out the quadratic regex, and both callers now import the private _credential_matches.

Watch

  • Symptom-level fix with private coupling. The root cause is the public accessor's regex. The PR leaves it alone ("No change to get_credential_patterns() itself") and instead points preview_files.py and deploy/scan.py at the private from kiro_crew.security.redaction import _credential_matches. Three consequences:

    • Other callers of get_credential_patterns() that scan file content still run the quadratic pattern on repeated PEM headers, so the same stall remains reachable there. ops_mission_control/backend/ledger_sync.py:641-647 is one, and the PR's pattern harvest does not mention it.
    • The private import goes against the stated purpose of get_credential_patterns: its docstring says it exists so callers avoid "coupling to the private ... name."
    • The next caller has no public linear path to use, so it will copy the quadratic pattern again.

    Clears when: a public linear match iterator (e.g. iter_credential_matches) is exported from kiro_crew.securityand both callers use it, or the PR listsledger_sync.py as a known remaining quadratic caller with a follow-up issue.

Suggestions

  • After this change, design_tweak/backend/server.py still imports and re-exports get_credential_patterns through _RUNTIME_COMPAT_IMPORTS, but nothing reads it anymore; remove it so the runtime seam doesn't advertise a hook that does nothing.

[DESIGN-REVIEWED] 6cdb2d0

Description read

[DESCRIPTION-READ] 45f6d0c0a4e2e559f38eeda3d1dcbcf9dd630a0e16b65ffd1fbed1ec9daeb9be

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One caller the description doesn't list: ledger_sync.py:650 still runs get_credential_patterns() through p.search(line), one line at a time. Confirm it is deliberately deferred.

What this change ships

Inventory (4 items) — 3 justified

Intent: let someone preview a project file or deploy a folder without a crafted text of repeated PEM headers stalling the process. This is a FIX (#21006; the new tests fail on base).

  1. Preview credential check now finishes in time linear in the file's size, even on repeated PEM headers. — justified
  2. Deploy-snapshot credential scan now finishes in time linear in the content's size, even on repeated PEM headers. — justified
  3. Preview and deploy no longer flag a JWT-shaped string whose first segment is not JSON. — rides along (declared; it comes from reusing the existing _credential_matches filter, no new surface)
  4. A new test file counts how often the full-block pattern runs and pins that real keys are still refused. — justified

Notes from the grep for get_credential_patterns\(\)|_credential_matches\b under src/:

  • The PR reuses the existing redaction._credential_matches (redaction.py:419) and adds no second spelling of it.
  • Three callers still use the raw pattern: deploy.sh:121, _common.sh:124 and ledger_sync.py:650. The description defers the first two by name. ledger_sync.py is missing from its "Pattern harvest" line. Because it scans line by line, the quadratic cost there is bounded by the length of one line.

[FIRST-PRINCIPLES-REVIEWED] 6cdb2d0

Description read

[DESCRIPTION-READ] eef9510138b93c9ba6b9c5875cdd752a6866be272de1793428d3bb9f50c7fc19

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6cdb2d0

Verdict parsed from the review's SHA-scoped output markers for commit 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6cdb2d0

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6cdb2d06f74ce92b0e1363bbc0c1dfb8ba5a4e0f: <one-sentence reason>

…ear time

The design_tweak preview check and the deploy scan ran the raw credential
pattern over file content. Its PEM branch ends in a lazy full-block tail that
reads to the end of the text when no END marker follows, so a text repeating a
PEM header paid one such read per header (0.69 / 2.67 / 11.04 s at 2000 / 4000 /
8000 headers).

Both callers now take their matches from the linear scanner that redaction
already uses (pem_scan via redaction._credential_matches). Real key blocks are
still refused and a quoted marker with no key body is still let through by the
preview.

Fixes #21006
@chenmingwei23
chenmingwei23 force-pushed the fix/21006-pem-scan-callers branch from 28e84b5 to 6cdb2d0 Compare October 16, 2026 21:44
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 16, 2026
@chenmingwei23
chenmingwei23 enabled auto-merge (squash) October 16, 2026 22:44
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 6cdb2d0.

The change itself reads as correct. Both scanners take their matches from the shared linear scanner, the preview keeps its PEM-body check, the deploy scan keeps its cap, and the new tests count full-pattern scans instead of timing them. Three things block approval:

  • It changes the credential scans that gate Design Tweak previews and deploys, so it needs a maintainer's review (policy hold).
  • No disposition exists for the Design CONCERNS on this head: Watch item (both callers import the private _credential_matches, and other callers still use the quadratic pattern) and its suggestion to remove the unused get_credential_patterns re-export. The First Principles PASS check on ledger_sync.py is also unanswered.
  • Not a goal defers moving artifact-deploy/scripts/deploy.sh and _common.sh as separate work, and ledger_sync.py still uses the raw pattern. Neither has a linked issue. Please open one issue per remaining item and link it in the description, or fix it here.

— crew-pr-reviewer (r108)

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code change looks correct to me. Approval waits on one open item.

Head reviewed: 6cdb2d0.

What I checked:

  • contains_credential and scan_content now iterate redaction._credential_matches, which is the same scanner the redaction path already uses. At a PEM header with no END marker after it, that scanner matches without the full-block tail, so the per-header read to the end of the text is gone.
  • The preview keeps its PEM_MARKER_RE / PEM_BODY_RE body check on each hit. The deploy scan keeps its findings cap. I compared the new loop with the old one, and the cap stops at the same point.
  • The only detection change is the declared one: a JWT-shaped hit whose first segment is not a JSON object is dropped, with a retry of the other branches at that start. The deploy scan still has its own header pattern, so a bare marker is still a finding there.
  • The tests count runs of the full pattern instead of timing them. On main, the preview test fails because get_credential_patterns() returns the patched pattern at call time. The deploy test fails because scan has no _credential_matches attribute to patch. A real block is still refused after many headers.
  • main does not contain this fix, and the touched lines did not move on main. No other open PR touches these files.
  • Readiness is passed and all required checks are green on this head.

Open item before approval:

  • The Design Review CONCERNS for this head has no reply. Please post a disposition for it: either the follow-up issue that covers a public linear accessor and the remaining raw-pattern callers, or a short rebuttal. The same reply can confirm that the line-by-line scan in ledger_sync.py is deliberately out of scope, which the First Principles note asks about.

When that reply is on this head, I will approve.

@chenmingwei23

chenmingwei23 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Re: review 2026-10-17

Dispositions for the open items on head 6cdb2d06f7. No code change: the fix is unchanged and all required checks are green on this head.

Design Review — Watch (private import + remaining raw-pattern callers). Deferred, with linked issues, keeping this PR's scope to the two callers that could stall on a previewed or deployed file:

Design Review — Suggestion (remove the unused get_credential_patterns re-export in design_tweak/backend/server.py). Deferred to #21696 rather than done here: that re-export is advisory cleanup, it is outside this bug's goal, and #21696 is where the accessor seam changes, so the removal belongs with it.

First Principles — ledger_sync.py line-by-line use. Deliberately out of scope, confirmed and tracked in #21696 (comment: #21388 (comment)). Correcting my earlier note: str.splitlines() bounds the cost by the length of the longest line, not by the whole file — but it does not make the per-line scan safe. The PEM header pattern needs no newline, so one physical line can repeat -----BEGIN ... PRIVATE KEY----- back to back many times, and p.search(line) over that line is still quadratic in that line's own length. Reach on main: each ledger line is one entry written as json.dumps(entry.to_dict()) + "\n", so a credential-shaped string field lands on a single physical line (a real newline is escaped as \n); the POST /ledger route already redacts entry content on the write path, and this per-line scan is the pre-push catch for an entry that reached the file by some other path (a legacy build, or a non-route write). It still carries the raw pattern, so it is listed in #21696 for the shared move to the public linear iterator.

@iamwhatever (comment #21388 (comment)):

@bolichen97 (review #21388 (review)): the disposition you asked for is here — follow-up issues (#21696, #21697) cover the public linear accessor and the remaining raw-pattern callers, and the First Principles note on ledger_sync.py is confirmed deliberately out of scope above. This reply is on head 6cdb2d06f7.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 6cdb2d0, for the policy reason only.

Since r108 the review items are settled. The author's dispositions answer the Design Watch item, the Design Suggestion and the First Principles note on ledger_sync.py. #21696 covers the public accessor, ledger_sync.py and the re-export removal. #21697 covers the deploy skill scripts. Both are labelled deferred-finding, assigned and dated. The code reads as correct: both callers iterate the shared linear scanner (preview_files.py L451, scan.py L120). The PEM body check and the findings cap are kept, and checks are green.

The remaining hold: this changes the credential scans that gate previews and deploys, including the declared change that a non-JSON JWT-shaped string is no longer flagged. Changes to what counts as a credential are reserved for a maintainer. @bolichen97's review says they will approve once the disposition is posted, and it now is. Nothing more is needed from the author.

— crew-pr-reviewer (r110)

@chenmingwei23
chenmingwei23 merged commit 534823e into main Oct 20, 2026
125 of 135 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/21006-pem-scan-callers branch October 20, 2026 00:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Credential scan in the design_tweak preview and the deploy scan takes quadratic time on text with many repeated PEM BEGIN headers

3 participants