Repository navigation
fix(security): scan preview and deploy content for credentials in linear time - #21388
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The fix patches two callers rather than the root cause: Watch
Suggestions
[DESIGN-REVIEWED] 6cdb2d0 Description read[DESCRIPTION-READ] 45f6d0c0a4e2e559f38eeda3d1dcbcf9dd630a0e16b65ffd1fbed1ec9daeb9be That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One caller the description doesn't list: What this change shipsInventory (4 items) — 3 justifiedIntent: let someone preview a project file or deploy a folder without a crafted text of repeated PEM headers stalling the process. This is a FIX (#21006; the new tests fail on base).
Notes from the grep for
[FIRST-PRINCIPLES-REVIEWED] 6cdb2d0 Description read[DESCRIPTION-READ] eef9510138b93c9ba6b9c5875cdd752a6866be272de1793428d3bb9f50c7fc19 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
…ear time The design_tweak preview check and the deploy scan ran the raw credential pattern over file content. Its PEM branch ends in a lazy full-block tail that reads to the end of the text when no END marker follows, so a text repeating a PEM header paid one such read per header (0.69 / 2.67 / 11.04 s at 2000 / 4000 / 8000 headers). Both callers now take their matches from the linear scanner that redaction already uses (pem_scan via redaction._credential_matches). Real key blocks are still refused and a quoted marker with no key body is still let through by the preview. Fixes #21006
28e84b5 to
6cdb2d0
Compare
|
HOLD at 6cdb2d0. The change itself reads as correct. Both scanners take their matches from the shared linear scanner, the preview keeps its PEM-body check, the deploy scan keeps its cap, and the new tests count full-pattern scans instead of timing them. Three things block approval:
— crew-pr-reviewer (r108) |
bolichen97
left a comment
There was a problem hiding this comment.
The code change looks correct to me. Approval waits on one open item.
Head reviewed: 6cdb2d0.
What I checked:
contains_credentialandscan_contentnow iterateredaction._credential_matches, which is the same scanner the redaction path already uses. At a PEM header with no END marker after it, that scanner matches without the full-block tail, so the per-header read to the end of the text is gone.- The preview keeps its
PEM_MARKER_RE/PEM_BODY_REbody check on each hit. The deploy scan keeps its findings cap. I compared the new loop with the old one, and the cap stops at the same point. - The only detection change is the declared one: a JWT-shaped hit whose first segment is not a JSON object is dropped, with a retry of the other branches at that start. The deploy scan still has its own header pattern, so a bare marker is still a finding there.
- The tests count runs of the full pattern instead of timing them. On main, the preview test fails because
get_credential_patterns()returns the patched pattern at call time. The deploy test fails becausescanhas no_credential_matchesattribute to patch. A real block is still refused after many headers. maindoes not contain this fix, and the touched lines did not move onmain. No other open PR touches these files.- Readiness is passed and all required checks are green on this head.
Open item before approval:
- The Design Review CONCERNS for this head has no reply. Please post a disposition for it: either the follow-up issue that covers a public linear accessor and the remaining raw-pattern callers, or a short rebuttal. The same reply can confirm that the line-by-line scan in
ledger_sync.pyis deliberately out of scope, which the First Principles note asks about.
When that reply is on this head, I will approve.
Re: review 2026-10-17Dispositions for the open items on head Design Review — Watch (private import + remaining raw-pattern callers). Deferred, with linked issues, keeping this PR's scope to the two callers that could stall on a previewed or deployed file:
Design Review — Suggestion (remove the unused First Principles — @iamwhatever (comment #21388 (comment)):
@bolichen97 (review #21388 (review)): the disposition you asked for is here — follow-up issues (#21696, #21697) cover the public linear accessor and the remaining raw-pattern callers, and the First Principles note on |
|
HOLD at 6cdb2d0, for the policy reason only. Since r108 the review items are settled. The author's dispositions answer the Design Watch item, the Design Suggestion and the First Principles note on The remaining hold: this changes the credential scans that gate previews and deploys, including the declared change that a non-JSON JWT-shaped string is no longer flagged. Changes to what counts as a credential are reserved for a maintainer. @bolichen97's review says they will approve once the disposition is posted, and it now is. Nothing more is needed from the author. — crew-pr-reviewer (r110) |
Problem / Motivation
Goal: Checking a previewed file or a deploy snapshot for credentials takes time that grows in proportion to its size, even when the text repeats a PEM header many times.
The preview check (
contains_credential) and the deploy scan (scan_content) ran the raw pattern fromget_credential_patterns()over file content. Its PEM branch ends in a lazy full-block tail that reads to the end of the text when no END marker follows, so a text that repeats-----BEGIN PRIVATE KEY-----paid one full read per header. Through the preview check, N = 2000 / 4000 / 8000 headers took 0.64 / 2.50 / 10.04 s.Fixes #21006
Why it matters
A file in a previewed project, or in a folder someone deploys, can stall the process with input that holds no valid key. The regex holds the GIL while it runs.
Not a goal
artifact-deploy/scripts/deploy.sh,_common.sh) still callget_credential_patterns()directly; moving them is separate work, tracked in Move artifact-deploy skill scripts (deploy.sh/_common.sh) off the quadratic credential pattern #21697.ops_mission_control/backend/ledger_sync.py) also still runs the raw pattern, line by line; its remaining cost is bounded by one line and it is tracked, with the public-accessor seam, in Export a public linear credential-match iterator and move remaining get_credential_patterns() caller (ledger_sync.py) #21696.get_credential_patterns()itself.What changed (motivation -> approach -> change)
The linear scanner from #20790 (
security/pem_scan.py, reached throughredaction._credential_matches) already finds the same matches as the raw pattern without letting a header that no END marker follows reach the full-block tail. Both callers now iterate that scanner instead of the raw pattern.contains_credentialkeeps itsPEM_MARKER_RE/PEM_BODY_REcheck on each hit, andscan_contentkeeps its findings cap and its own header pattern.One difference: the shared scanner also drops a JWT-shaped hit whose first segment is not a JSON object, the same filter redaction applies. A real JWT is still found.
Backwards compatibility
Compatible: the same PEM blocks are refused and the same quoted markers are allowed; the only text that stops matching is a JWT-shaped string whose first segment is not JSON.
Removes nothing: no function, key or path is removed or renamed;
get_credential_patterns()is unchanged and still exported.Tests
test/test_pem_scan_callers_linear.pycounts how many times the full-block pattern runs, so it does not depend on the clock. It checks that the preview never runs a repeated header through the full-block tail, that the deploy scan takes its matches from the shared scanner once per scan, that a real key block is refused by both callers (also after many headers), that the preview allows a quoted marker with no key body, that the deploy scan still flags a bare marker, and that other credentials are still found. A 3 MiB case guards against a hang. On main, 7 of these fail; on this branch all pass.Also run and passing:
test_design_tweak_backend.py, the deploy scan test files,test_redaction_pem_scan_linear.py,test_denied_root_parts_shared.py. flake8, isort and mypy are clean on the changed files.Manual verification
N/A - unit coverage sufficient. Measured the preview check and the deploy scan on
-----BEGIN PRIVATE KEY-----\nrepeated 2000 / 4000 / 8000 times: preview 0.64 / 2.50 / 10.04 s before, 0.00 / 0.01 / 0.02 s after.Screenshots / video
N/A - no user-visible UI change.
Related Issues
Fixes #21006
Follow-ups for the remaining callers of the raw pattern (deferred, out of this PR's scope):
kiro_crew.securityand move the ledger pre-push scan to it (also removes the now-unusedget_credential_patternsre-export indesign_tweak/backend/server.py).deploy.sh,_common.sh) to the public linear iterator.Pattern harvest
Rule candidate: review-prompt
Pattern: a regex whose tail is a lazy scan to a closing marker, run once per opening marker, is quadratic; other callers of
get_credential_patterns()(the artifact-deploy skill scripts) still carry it.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.