Skip to content

fix: run the deploy-skill and ledger credential scans in linear time - #31445

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/21697-linear-credential-iterator
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/21697-linear-credential-iterator

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The credential scans that run before an artifact deploy and before a team-ledger push finish in time linear in the file size, and still refuse the credentials they refuse today.

The artifact-deploy skill's pre-upload scans (deploy.sh, _common.sh) and the ops ledger's pre-push scan (ledger_sync._credential_bearing_lines) hand each whole file or ledger line to the raw get_credential_patterns() regex with search. That search re-walks a long eyJ run or digit run from every start, so its cost grows with the square of the run. Measured on main: 60 KB of eyJ takes 22 s, and 80 KB of eyJa takes 46 s. Two other callers reach the shared linear scanner only through the private name redaction._credential_matches.

Why it matters

A minified bundle or data file holding such a run stalls deploy.sh before upload, and one long ledger line stalls a team-ledger push, both with the GIL held. With no public linear scan, the next caller copies the raw pattern again.

Not a goal

  • Changing which shapes count as a credential: every caller gets the matches redact_credentials already acts on.
  • The deploy scripts' grep fallback for a Python without kiro_crew.
  • The ops app's own provider-token detector, redact_tokens.
  • Removing get_credential_patterns().

What changed (motivation → approach → change)

The slow scans are a raw search over a whole file or line. The security package already has a scanner that finds the same credentials in linear time, the one redact_credentials runs, but it was private.

kiro_crew.security now exports it as iter_credential_matches, and every caller outside the package uses it:

Caller Before Now
artifact-deploy deploy.sh, _common.sh (embedded Python) raw search per file iter_credential_matches
ops ledger_sync._credential_bearing_lines raw search per line iter_credential_matches
deploy/scan.py, design tweak preview_files.py private _credential_matches iter_credential_matches

The design tweak server drops its unused get_credential_patterns re-export. A scanner error still aborts the deploy, and a file with a real key is still refused. The specs and the security-posture text in security_posture_sinks_delivery.py that named the old accessor name the new one. The security package stays inside its existing line budget.

These three gates now skip the shape-only lookalikes the redactor already rejects, such as honeyJar.example.com (a JWT shape with no JSON header) or a CDK ARN resource name, so those no longer block a deploy or a push. The issues describe the cost as one read per repeated PEM header. For these stop-at-first-match scans that shape costs one extra read per file; the quadratic shapes are the eyJ and digit runs, which the tests use.

Backwards compatibility

Compatible: a credential the old scans refused is still refused; only the JWT/Telegram lookalikes redact_credentials already lets through stop blocking, and the new name is additive.

Reader: src/kiro_crew/apps/builtins/design_tweak/backend/server.py:get_credential_patterns -- app -- no reader: no runtime.get_credential_patterns or server.get_credential_patterns on origin/main; test_design_tweak_backend.py passes
Reader: src/kiro_crew/deploy/scan.py:_credential_matches -- app -- its only reader was the spy in test_pem_scan_callers_linear.py, now aimed at scan.iter_credential_matches
Reader: src/kiro_crew/apps/builtins/design_tweak/backend/preview_files.py:_credential_matches -- app -- no reader outside the module on origin/main

Tests

  • test_pem_scan_callers_linear.py::test_deploy_skill_scan_never_runs_the_full_pattern_over_a_file[deploy.sh|_common.sh] runs each script's embedded scan in-process: a file of 2,000 eyJ never reaches the raw alternation, and a file with a real key still exits 2 with that file named.
  • test_ledger_sync_coverage.py::test_credential_scan_never_runs_the_raw_search_over_a_line: a ledger line of 2,000 eyJ followed by an AWS key id is flagged with no raw search.
  • test_ledger_sync_git.py::test_the_scan_stays_in_step_with_the_write_path now asserts the push scan calls iter_credential_matches, and the deploy-scan spy targets the public name.

Proof: PROVEN src/kiro_crew/apps/builtins/ops_mission_control/tests/test_ledger_sync_coverage.py::test_credential_scan_never_runs_the_raw_search_over_a_line, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_ledger_sync_git.py::TestPushRefusesCredentialMaterial::test_the_scan_stays_in_step_with_the_write_path, test/test_pem_scan_callers_linear.py::test_deploy_skill_scan_never_runs_the_full_pattern_over_a_file[deploy.sh], test/test_pem_scan_callers_linear.py::test_deploy_skill_scan_never_runs_the_full_pattern_over_a_file[_common.sh] (head 50adb2d)

Determinism: the two new tests passed 20 of 20 runs at -n0 and 10 of 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside the rest of both files, and a shuffled run of both files (seed 21697, 124 passed). They count calls on a patched pattern object, with no clock or ordering dependence.

Manual verification

On main, get_credential_patterns()[0].search over 20,000 eyJ (60 KB) took 22.5 s and over 20,000 eyJa (80 KB) 45.8 s; iter_credential_matches returned on the same texts in 0.011 s and 0.015 s. Over 32,000 repeated PEM headers the raw search took 0.028 s, which is why the tests use the eyJ run.

Screenshots / video

N/A — no frontend change.

Related Issues

Closes #21697
Closes #21696

Sibling sweep

Swept: git grep -nE "get_credential_patterns()|_credential_matches(" -- src
Sibling: src/kiro_crew/deploy/skills/artifact-deploy/scripts/deploy.sh -- changed here
Sibling: src/kiro_crew/deploy/skills/artifact-deploy/scripts/_common.sh -- changed here
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/ledger_sync.py -- changed here
Sibling: src/kiro_crew/deploy/scan.py -- changed here
Sibling: src/kiro_crew/apps/builtins/design_tweak/backend/preview_files.py -- changed here
Sibling: src/kiro_crew/security/text_redaction.py -- not affected -- inside the security package, already runs the linear scanner
Sibling: src/kiro_crew/apps/builtins/aws_control/crew/packaging/tests/test_review_findings_security.py -- not affected -- a test that reads the pattern to probe token lengths, never scans a file

Pattern harvest

Rule candidate: semgrep
Pattern: a get_credential_patterns() pattern used with .search/.finditer outside kiro_crew/security/, where iter_credential_matches is the linear scan

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 13:58
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 13:58
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/__init__.py

  • src/kiro_crew/security/_exports.py

  • src/kiro_crew/security/redaction.py

  • src/kiro_crew/security/redaction_patterns.py

  • src/kiro_crew/security_posture_sinks_delivery.py

  • No reasoning template for 20e68f7 yet.

How to clear this check:

  1. The PR author posts the template below as a PR comment, with
    every field filled (N/A needs a reason). Keep the
    Head commit line: it must name the current head SHA.
  2. The check turns green within a minute, with no push.
  3. A new push resets this check: post it again for the new head.

Only the author's comment counts this way. A reviewer can instead
paste the filled template into an "Approve" review on this head.
Merging still needs a reviewer's approval.

## Sensitive change review
- Head commit: 20e68f7baef20e962f8904433902dd95d5f3228f
- What rule changed:
- Before -> after (what was allowed/blocked, now):
- Worst case if wrong (breaks commands / leaks secret / locks user out):
- Evidence checked (which sandbox result proves it):
- No regression:
  - Tools still work (which tools/commands were run, result):
  - Backward compatible (old config / old data / old callers still work, how checked):
  - Existing tests (which suites passed on this head):
- How to undo:

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 20e68f7baef20e962f8904433902dd95d5f3228f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 20e68f7baef20e962f8904433902dd95d5f3228f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change reuses the linear scanner redact_credentials already runs, adds one public name, and the description admits the one behaviour change (the shape-only lookalikes it lets through).

Notes

  • The "Pattern harvest" semgrep rule against get_credential_patterns() + .search/.finditer outside security/ is what stops a later caller from copying the raw pattern again. Without it, the "Why it matters" risk is only covered by the posture text.

[DESIGN-REVIEWED] 20e68f7

Description read

[DESCRIPTION-READ] 431112fbed3d197b995dda420e684db62efb1dacb4232c9afad9f6212a7bb71a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 20e68f7baef20e962f8904433902dd95d5f3228f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: each caller moves onto the linear scanner redact_credentials already runs, and the lookalikes that stop blocking are declared.

Notes

  • Item 3 — duplicate of src/kiro_crew/security/redaction_patterns.py:_credential_matches: the new iter_credential_matches just returns _credential_matches(text), so one function now has two names. Grepping _credential_matches\( found 4 callers inside the package that still use the private name (text_redaction.py:214, text_redaction.py:540, redaction.py:936, _contains_credential_pattern). Subtraction: rename _credential_matches to iter_credential_matches, point those 4 callers at the new name, and delete the wrapper.

What this change ships

Inventory (7 items) — 6 justified

Intent: stop the pre-deploy and pre-push credential scans stalling on long eyJ or digit runs, while still refusing the same keys. FIX.

  1. The artifact-deploy deploy.sh and _common.sh scans finish in linear time on large bundles — justified
  2. The ops ledger pre-push scan finishes in linear time on one long line — justified
  3. New public security.iter_credential_matches, with 5 callers outside the package — duplicate of redaction_patterns._credential_matches (thin alias)
  4. The JWT-shaped and Telegram-shaped lookalikes the redactor already lets through no longer block a deploy or a push — justified
  5. deploy/scan.py and design_tweak's preview_files.py switch from the private name to the public one — justified
  6. The design_tweak server drops its unused get_credential_patterns re-export — justified
  7. The specs and the security-posture text name the new scan — justified

[FIRST-PRINCIPLES-REVIEWED] 20e68f7

Description read

[DESCRIPTION-READ] 8fb1e28f09e10e0d3552fce821776b5aee2ec5a3a9187686ed9afe97629476c0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 20e68f7baef20e962f8904433902dd95d5f3228f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 20e68f7

Verdict parsed from the review's SHA-scoped output markers for commit 20e68f7baef20e962f8904433902dd95d5f3228f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 20e68f7baef20e962f8904433902dd95d5f3228f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 20e68f7baef20e962f8904433902dd95d5f3228f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 20e68f7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 20e68f7baef20e962f8904433902dd95d5f3228f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does 20e68f7baef20e962f8904433902dd95d5f3228f newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 9 classified, 0 newly refused, 8 still allowed.
  • ubuntu-latest (platform posix): 9 classified, 0 newly refused, 8 still allowed.
  • windows-latest (platform windows): 4 classified, 0 newly refused, 4 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

Reason: This diff doesn't make any shell-deny tier stricter. It adds iter_credential_matches, a public name for the existing linear _credential_matches scan, and moves five callers onto it. Four of them already used that scan (preview_files, deploy/scan.py) or the same raw pattern (ledger_sync, deploy.sh, _common.sh). For the last three, the move removes JWT and Telegram shape-only false hits, so their set of matches can only shrink. The ledger pre-push scan and the deploy-skill file scans are not shell-command guards, so the classifier can't give a verdict on them. That gap is named below.

TIGHTENING: No shell-tier mechanism got stricter. The behavioural change is in three scan sites, for example - for pat in patterns:\n- if pat.search(data): → + if any(iter_credential_matches(data)): in deploy.sh/_common.sh, and - if any(p.search(line) for p in patterns): → + if any(iter_credential_matches(line)): in ledger_sync._credential_bearing_lines. The threat it addresses is quadratic time (denial of service on the scan) from a long eyJ run or repeated unterminated PEM headers. It does not refuse any new credential shape.

ADJUDICATED: 13 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE: I wrote posix candidates for the artifact-deploy skill's operator commands (deploy.sh, deploy-app.sh, deploy-backend.sh), deploy.sh's python3 -c "import kiro_crew.security" probe, a manual scan with the new name, the touched test files, a caller search, and the ledger git push. I wrote windows candidates for the same manual scan (python, a backslash path), the test runs (.venv\Scripts\python.exe, backslash separators) and a findstr caller search. The diff adds no path, separator, interpreter or environment-variable assumption of its own. The skill heredocs were already posix-only (python3 - inside .sh), so no Windows spelling of those scripts exists to probe.

UNADJUDICATED: The changed guards are content scanners, not shell-command guards, so no row can reach them:

  • ledger_sync._credential_bearing_lines is reached from ledger sync before git push of the shared ledger. It reads ledger.ledger_path() line by line and checks each line with iter_credential_matches plus redact_tokens.
  • scan_source_dir in _common.sh and the inline scan in deploy.sh are reached from deploy*.sh <app_dir>. They walk the directory and test each file with iter_credential_matches.

The inputs I would have submitted, described in words: a ledger line quoting a minified bundle with a long eyJ-prefixed base64 run that is not a JWT; a ledger line holding an ARN-style account-id segment next to a colon and a long digit run that reads like a Telegram token; and a built bundle.js containing JWT-lookalike strings. With the new scan, each of these should be flagged less often than before, not more. That reading comes from the code of _credential_matches, which drops shape-only JWT and Telegram hits after _CREDENTIAL_PATTERNS matches. No script confirmed it.

[SCOPE-REVIEWED] 20e68f7

@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 14:13
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/21697-linear-credential-iterator branch from 2649c4a to 50adb2d Compare November 6, 2026 16:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Waiting on #31625, which fixes the current main red (#31648). Once it merges this PR rebases onto it and CI re-runs.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/21697-linear-credential-iterator branch from 50adb2d to d43a234 Compare November 6, 2026 23:46
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/security.md, src/kiro_crew/security_posture_sinks_delivery.py and test/test_security_gate_liveness.py since this head went green (38a8ef5 -> 4d9cb58); CI re-running. The only conflict was the package line budget, which main raised to 34,753; this PR now adds its 10 on top (34,763). Targeted tests pass on the new head.
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 6, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97
bolichen97 force-pushed the fix/21697-linear-credential-iterator branch from d43a234 to 45ed54a Compare November 7, 2026 04:04
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased: main moved in test/test_security_gate_liveness.py since this head (d43a234 -> 76ec06a); CI re-running. The only conflict was the security package line budget, which main raised to 34,777; this PR adds its 10 on top (34,787). The patch is otherwise byte-identical (same patch-id). The previous head's red AI and scope lanes were rate-limit failures with no verdict, not findings. Targeted tests, isort, flake8, mypy, semgrep and local GPT and Opus reviews pass on 45ed54a.

AI-assisted, posted on behalf of bolichen97

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 7, 2026
The artifact-deploy skill's pre-upload scans (deploy.sh, _common.sh)
and the ops ledger's pre-push scan ran get_credential_patterns()'s raw
search over whole files or lines. That search re-walks a long eyJ or
digit run from every start, so one such file or line stalls the scan
for seconds to minutes.

Export the shared linear scanner as kiro_crew.security's public
iter_credential_matches and move every caller outside the package to
it, including the two that imported the private _credential_matches.
Drop the unused get_credential_patterns re-export from the design
tweak server. Findings stay those redact_credentials sees, and a
scanner error still aborts the deploy.
@bolichen97
bolichen97 force-pushed the fix/21697-linear-credential-iterator branch from 45ed54a to 20e68f7 Compare November 7, 2026 05:46
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased: main moved in src/kiro_crew/security/redaction.py and src/kiro_crew/security/redaction_patterns.py since this head went green (76ec06a -> 480d162); CI re-running. Main's change there shrank the security package, so the package now fits main's existing line budget and this PR no longer raises it (the budget hunk is dropped). The rest of the patch is unchanged. On the previous head, every red lane failed on the shared token's API rate limit; the scope lane's rerun reached PASS but was held by the per-head floor that the rate-limited first run set. Targeted tests, isort, flake8, mypy, semgrep and local GPT and Opus reviews pass on 20e68f7.

AI-assisted, posted on behalf of bolichen97

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 20e68f7, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit 7aaf1b1 into main Nov 7, 2026
101 of 102 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/21697-linear-credential-iterator branch November 7, 2026 06:38
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
bolichen97 pushed a commit that referenced this pull request Nov 7, 2026
#31445 merged first and used the headroom #30976 was counted against, so main landed 10 lines over the budget (34,855 vs 34,845). Test-only; nothing is relaxed.
buluoray pushed a commit that referenced this pull request Nov 8, 2026
…31445)

The artifact-deploy skill's pre-upload scans (deploy.sh, _common.sh)
and the ops ledger's pre-push scan ran get_credential_patterns()'s raw
search over whole files or lines. That search re-walks a long eyJ or
digit run from every start, so one such file or line stalls the scan
for seconds to minutes.

Export the shared linear scanner as kiro_crew.security's public
iter_credential_matches and move every caller outside the package to
it, including the two that imported the private _credential_matches.
Drop the unused get_credential_patterns re-export from the design
tweak server. Findings stay those redact_credentials sees, and a
scanner error still aborts the deploy.

(cherry picked from commit 7aaf1b1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants