Skip to content

perf(redaction): native single-pass credential gate, Python fallback - #19628

Merged
buluoray merged 2 commits into
mainfrom
perf/native-redaction-gate
Oct 26, 2026
Merged

buluoray merged 2 commits into
mainfrom
perf/native-redaction-gate

Conversation

@buluoray

@buluoray buluoray commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the credential pre-filter that every redaction call runs about 5x cheaper in CPU, with identical answers, while installs without the native module keep working exactly as today.

Profiling a gateway on a copy of real data, redact_credentials was the largest pure-Python CPU path under load (22% of non-idle samples, mostly slot-detail rendering). Its pre-filter _might_contain_credential alone was 8%.

Why it matters

Redaction runs on every transcript string the dashboard renders and the gateway stores, so its CPU cost is paid on every session view and grows with session length.

Not a goal

  • Building or shipping kc_native in CI, the wheel or the installers. Until a later change does that, every install keeps the Python gate.
  • Moving the redaction patterns or replacements themselves to Rust. Only the "could this contain a credential?" pre-filter moves.
  • The Rust MCP stub default from the same campaign (separate PR).

What changed (motivation → approach → change)

  • Approach. The pre-filter is 21 substring searches plus 6 anchored regexes. Doing them in one regex-crate scan, with the GIL released for large strings, removes most of its cost without touching what redaction replaces.
  • native/kc_native/ (new). A PyO3 module with one function, might_contain_credential, and a GATE_FINGERPRINT constant. Python's \s and (?i) folding are spelled out by hand. The module reads each str from its own storage (PyUnicode_KIND / PyUnicode_DATA) instead of asking for UTF-8, because CPython caches that UTF-8 copy inside every non-ASCII str until it dies.
  • src/kiro_crew/security/redaction_patterns.py (the pre-filter's owner since the redaction split; redaction serves the new names through its facade). The existing pre-filter is renamed _might_contain_credential_py, body unchanged. _might_contain_credential uses the native gate by default when it is usable, and the Python one otherwise:
    • kc_native missing, or built for another CPython version: the import raises ImportError → Python.
    • GATE_FINGERPRINT does not match a hash of the Python gate's literals, every anchor pattern (both literal-led Authorization halves included) and Unicode version (the Python gate was edited without a rebuild) → Python.
    • KIROCREW_NATIVE_REDACTION=0 → Python.
    • A string with a lone surrogate: the native call raises UnicodeEncodeError → Python for that call.
  • .github/workflows/ci.yml. A new job, Native Redaction Gate (kc_native), builds the module (cargo test --release --locked, maturin build --release --locked), installs it, confirms it imports, and runs test/test_native_redaction_gate.py unskipped together with the fallback suite and test/test_credential_prefilter.py. This builds the module for testing only; the wheel and installers still do not ship it (added after the Design Review asked that CI hold the equivalence claim).
  • .github/dependency-review-config.yml. Two durable per-package license exceptions for build-time-only crates in the new Cargo.lock (target-lexicon: Apache-2.0 WITH LLVM-exception; unicode-ident: (MIT OR Apache-2.0) AND Unicode-3.0), with the evidence links the file asks for.

Product shape: unchanged. Redaction gives the same answer for every string on both paths, and no install gets the native path until it builds the module.

Measured

On current main, whose Python gate already leads each anchor with a literal: pre-filter CPU over 611,572 local transcript strings, best of 3, 2.05 s Python vs 0.42 s native. The table below was measured before that rewrite landed.

Python gate Native gate
Pre-filter CPU, 438,602 real transcript strings (3 runs) 6.57 s 0.93 s
redact_credentials CPU, same corpus 20.8 s 14.3 s
Fixed 60-cycle load run, gateway CPU 504.9 s 476.3 s
Fixed 60-cycle load run, wall 312.5 s 292.2 s
RSS growth, 200 live "中" * 100_000 strings scanned
Native build reading UTF-8 (PyUnicode_AsUTF8AndSize) 57.3 MB
This PR (reads str storage) 0.1 MB

Equivalence with the Python gate after the rebase: gate answers and full redact_credentials + redact_exfiltration_urls output hash identically over 611,572 local transcript strings for origin/main, this branch without the module, and this branch with it (626 strings flagged by all three). Before the rebase: 0 differences over 52,864 constructed inputs and a 708,249-string corpus of real strings. Full redact_credentials / redact_exfiltration_urls output: 0 differences over 989,663 real strings.

Risks / costs

  • Not abi3. Reading str storage needs the full C API, so the module must be built separately for each CPython minor version. A mismatched build fails to import and the Python path runs (test_missing_module_keeps_python); redaction does not break.
  • CPython 3.12 only, as shipped. The fingerprint covers unicodedata.unidata_version because the Rust side hand-encodes Python's \s set and i case folding, and the committed GATE_FINGERPRINT is the 3.12 (Unicode 15.0) value. On 3.13+ a build imports but its fingerprint does not match, so the Python gate runs and _load_native_gate logs one debug line saying so. Extending it to later minors means carrying their fingerprints and testing them, which belongs with the change that ships per-minor builds.
  • Security-relevant path. The Python gate is unchanged and stays the reference; the native gate is used only while its fingerprint matches, and KIROCREW_NATIVE_REDACTION=0 turns it off.
  • The profiling scripts that produced the measurements are not included (they run against copies of live data and are not product code).

Backwards compatibility

Compatible: with the module absent (every install today) the code path is the existing Python pre-filter; the new env var defaults to on and only matters when the module is built.

Removes nothing: _might_contain_credential keeps its name, signature and answers; the old body is kept verbatim as _might_contain_credential_py.

Tests

  • test/test_native_redaction_fallback.py (runs without the module): default on with a matching module, missing module, stale fingerprint, KIROCREW_NATIVE_REDACTION=0, and a gate that raises UnicodeEncodeError all answer like the Python gate.
  • test/test_native_redaction_gate.py (skipped when the module is not built): every code point at each character-class position, 200k random strings, lone-surrogate fallback, the no-growth property for non-ASCII strings, and redact_credentials output, native vs Python.
  • test/test_security_gate_liveness.py: _PACKAGE_LINE_BUDGET raised from 32,530 to 32,601 for the 71 lines the loader, fingerprint, mismatch log line, dispatcher and facade type-checking names add, with the reason recorded beside it as the file asks.
  • test/test_credential_prefilter.py::test_differential_test_detects_a_too_narrow_prefilter: the control disarms the Python gate by rebinding its anchors, which the native gate does not read, so it now also sets _NATIVE_GATE to None.
  • test/test_prepare_pr_profiles.py: cargo joins the floor's exempt_tools, with its reason: the new job's cargo test covers only the optional crate and a contributor host need not have a Rust toolchain.

Manual verification

After rebasing onto current main (CPython 3.12.13, Linux; macOS and Windows not run):

  • cargo test --release --locked 2 passed; cargo fmt --check and cargo clippy --release --locked clean; maturin build --release --locked with 1.15.0.
  • With the module live: test_native_redaction_gate.py, test_native_redaction_fallback.py, test_credential_prefilter.py, test_credential_prefilter_anchors.py: 542 passed, 1 skipped (corpus); the corpus case run separately over 611,572 strings: passed.
  • Without the module: the 28 test files that reference the redaction owners, display_verdicts or _NATIVE_GATE: 2,093 passed, 5 skipped, 1 failed (test_auto_improvement_members.py::test_assignments_build_real_context_from_only_their_private_store, which rejects this host's pytest temp dir under the crew scratch tree; it touches no redaction code).
  • Mutations that each turn a named test red: dropping the UnicodeEncodeError fallback (test_gate_that_cannot_encode_falls_back), the fingerprint check (test_stale_module_keeps_python), the env switch (test_env_zero_turns_it_off); in Rust, dropping the eyJ literal (test_random_differential) and the dotted/dotless i fold (test_every_codepoint_at_each_class_position[Author{c}zation]).
  • mypy src/kiro_crew/security/redaction_patterns.py: clean.

Before the rebase:

  • Built with maturin develop --release --locked into the repo venv (CPython 3.12); cargo fmt --check and cargo clippy --release --locked clean.
  • The two new files: 32 passed, 1 skipped.
  • Every test file that references redaction (130 files), native on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed. The same 22 tests fail in both runs and also fail on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
  • mypy src/kiro_crew/security/redaction.py: clean.

Related Issues

Closes #24137

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 13, 2026 00:09
@buluoray
buluoray requested a review from mbriones98 October 13, 2026 00:09
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/redaction.py

  • src/kiro_crew/security/redaction_patterns.py

  • @buluoray posted a complete reasoning template for b1bb2d4.

@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

The following issues were found:
  • ❌ 2 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ✅ 0 package(s) with unknown licenses.
See the Details below.

License Issues

native/kc_native/Cargo.lock

PackageVersionLicenseIssue Type
target-lexicon0.13.5Apache-2.0 WITH LLVM-exceptionIncompatible License
unicode-ident1.0.26(MIT OR Apache-2.0) AND Unicode-3.0Incompatible License
Allowed Licenses: 0BSD, CC0-1.0, Unlicense, WTFPL, Apache-2.0, BSD-2-Clause, BSD-3-Clause, BSD-3-Clause-Clear, BlueOak-1.0.0, ISC, MIT, MIT-0, PostgreSQL, Python-2.0, Zlib, MPL-2.0, Artistic-2.0
Excluded from license check: pkg:pypi/torch, pkg:pypi/regex, pkg:pypi/typing-extensions, pkg:npm/%40pierre/diffs, pkg:npm/%40pierre/theme, pkg:npm/%40pierre/theming, pkg:npm/%40pierre/trees, pkg:npm/%40anthropic-ai/claude-code, pkg:npm/%40anthropic-ai/claude-code-darwin-arm64, pkg:npm/%40anthropic-ai/claude-code-darwin-x64, pkg:npm/%40anthropic-ai/claude-code-linux-arm64, pkg:npm/%40anthropic-ai/claude-code-linux-arm64-musl, pkg:npm/%40anthropic-ai/claude-code-linux-x64, pkg:npm/%40anthropic-ai/claude-code-linux-x64-musl, pkg:npm/%40anthropic-ai/claude-code-win32-arm64, pkg:npm/%40anthropic-ai/claude-code-win32-x64, pkg:npm/opencode-darwin-arm64, pkg:npm/opencode-darwin-x64, pkg:npm/opencode-darwin-x64-baseline, pkg:npm/opencode-linux-arm64, pkg:npm/opencode-linux-arm64-musl, pkg:npm/opencode-linux-x64, pkg:npm/opencode-linux-x64-baseline, pkg:npm/opencode-linux-x64-baseline-musl, pkg:npm/opencode-linux-x64-musl, pkg:npm/opencode-windows-arm64, pkg:npm/opencode-windows-x64, pkg:npm/opencode-windows-x64-baseline

OpenSSF Scorecard

Scorecard details
PackageVersionScoreDetails
cargo/aho-corasick 1.1.5 🟢 5.7
Details
CheckScoreReason
Code-Review⚠️ 1Found 5/29 approved changesets -- score normalized to 1
Packaging⚠️ -1packaging workflow not detected
Binary-Artifacts🟢 10no binaries found in the repo
Maintained🟢 44 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 4
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Pinned-Dependencies🟢 10all dependencies are pinned
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing🟢 10project is fuzzed
Security-Policy⚠️ 0security policy file not detected
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/heck 0.5.0 🟢 3
Details
CheckScoreReason
Binary-Artifacts🟢 10no binaries found in the repo
Maintained⚠️ 00 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0
Code-Review⚠️ 2Found 7/29 approved changesets -- score normalized to 2
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Security-Policy⚠️ 0security policy file not detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/libc 0.2.190 🟢 6
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 21 issue activity found in the last 90 days -- score normalized to 10
Code-Review🟢 4Found 11/25 approved changesets -- score normalized to 4
Packaging⚠️ -1packaging workflow not detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Binary-Artifacts🟢 10no binaries found in the repo
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection🟢 3branch protection is not maximal on development and all release branches
Security-Policy🟢 10security policy file detected
SAST🟢 10SAST tool is run on all commits
Pinned-Dependencies⚠️ 2dependency not pinned by hash detected -- score normalized to 2
cargo/memchr 2.8.3 UnknownUnknown
cargo/once_cell 1.21.4 🟢 4
Details
CheckScoreReason
Packaging⚠️ -1packaging workflow not detected
Code-Review🟢 7Found 11/15 approved changesets -- score normalized to 7
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Maintained⚠️ 00 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0
Pinned-Dependencies🟢 8dependency not pinned by hash detected -- score normalized to 8
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Security-Policy⚠️ 0security policy file not detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/portable-atomic 1.15.0 🟢 7.3
Details
CheckScoreReason
Code-Review⚠️ 0Found 0/19 approved changesets -- score normalized to 0
Maintained🟢 1030 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Packaging⚠️ -1packaging workflow not detected
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Pinned-Dependencies🟢 10all dependencies are pinned
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 10security policy file detected
Branch-Protection🟢 4branch protection is not maximal on development and all release branches
SAST🟢 10SAST tool is run on all commits
cargo/proc-macro2 1.0.107 🟢 5.1
Details
CheckScoreReason
Binary-Artifacts🟢 10no binaries found in the repo
Maintained🟢 44 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 4
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Code-Review⚠️ 0Found 0/27 approved changesets -- score normalized to 0
Packaging⚠️ -1packaging workflow not detected
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing🟢 10project is fuzzed
License🟢 10license file detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 3security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/pyo3 0.29.3 🟢 6.7
Details
CheckScoreReason
Maintained🟢 1030 commit(s) out of 30 and 10 issue activity out of 30 found in the last 90 days -- score normalized to 10
Code-Review🟢 8GitHub code reviews found for 25 commits out of the last 30 -- score normalized to 8
CII-Best-Practices⚠️ 0no badge detected
Vulnerabilities🟢 10no vulnerabilities detected
Signed-Releases⚠️ -1no releases found
License🟢 10license file detected
Packaging⚠️ -1no published package detected
Token-Permissions⚠️ 0non read-only tokens detected in GitHub workflows
Dependency-Update-Tool🟢 10update tool detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies🟢 7dependency not pinned by hash detected -- score normalized to 7
Binary-Artifacts🟢 10no binaries found in the repo
Security-Policy⚠️ 0security policy file not detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Fuzzing⚠️ -1internal error: internal error: Client.Search.Code: Search.Code: GET https://api.github.com/search/code?q=github.com+PyO3+pyo3+repo%3Agoogle%2Foss-fuzz+in%3Afile+filename%3Aproject.yaml: 400 []
cargo/pyo3-build-config 0.29.3 🟢 6.7
Details
CheckScoreReason
Maintained🟢 1030 commit(s) out of 30 and 10 issue activity out of 30 found in the last 90 days -- score normalized to 10
Code-Review🟢 8GitHub code reviews found for 25 commits out of the last 30 -- score normalized to 8
CII-Best-Practices⚠️ 0no badge detected
Vulnerabilities🟢 10no vulnerabilities detected
Signed-Releases⚠️ -1no releases found
License🟢 10license file detected
Packaging⚠️ -1no published package detected
Token-Permissions⚠️ 0non read-only tokens detected in GitHub workflows
Dependency-Update-Tool🟢 10update tool detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies🟢 7dependency not pinned by hash detected -- score normalized to 7
Binary-Artifacts🟢 10no binaries found in the repo
Security-Policy⚠️ 0security policy file not detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Fuzzing⚠️ -1internal error: internal error: Client.Search.Code: Search.Code: GET https://api.github.com/search/code?q=github.com+PyO3+pyo3+repo%3Agoogle%2Foss-fuzz+in%3Afile+filename%3Aproject.yaml: 400 []
cargo/pyo3-ffi 0.29.3 🟢 6.7
Details
CheckScoreReason
Maintained🟢 1030 commit(s) out of 30 and 10 issue activity out of 30 found in the last 90 days -- score normalized to 10
Code-Review🟢 8GitHub code reviews found for 25 commits out of the last 30 -- score normalized to 8
CII-Best-Practices⚠️ 0no badge detected
Vulnerabilities🟢 10no vulnerabilities detected
Signed-Releases⚠️ -1no releases found
License🟢 10license file detected
Packaging⚠️ -1no published package detected
Token-Permissions⚠️ 0non read-only tokens detected in GitHub workflows
Dependency-Update-Tool🟢 10update tool detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies🟢 7dependency not pinned by hash detected -- score normalized to 7
Binary-Artifacts🟢 10no binaries found in the repo
Security-Policy⚠️ 0security policy file not detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Fuzzing⚠️ -1internal error: internal error: Client.Search.Code: Search.Code: GET https://api.github.com/search/code?q=github.com+PyO3+pyo3+repo%3Agoogle%2Foss-fuzz+in%3Afile+filename%3Aproject.yaml: 400 []
cargo/pyo3-macros 0.29.3 🟢 6.7
Details
CheckScoreReason
Maintained🟢 1030 commit(s) out of 30 and 10 issue activity out of 30 found in the last 90 days -- score normalized to 10
Code-Review🟢 8GitHub code reviews found for 25 commits out of the last 30 -- score normalized to 8
CII-Best-Practices⚠️ 0no badge detected
Vulnerabilities🟢 10no vulnerabilities detected
Signed-Releases⚠️ -1no releases found
License🟢 10license file detected
Packaging⚠️ -1no published package detected
Token-Permissions⚠️ 0non read-only tokens detected in GitHub workflows
Dependency-Update-Tool🟢 10update tool detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies🟢 7dependency not pinned by hash detected -- score normalized to 7
Binary-Artifacts🟢 10no binaries found in the repo
Security-Policy⚠️ 0security policy file not detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Fuzzing⚠️ -1internal error: internal error: Client.Search.Code: Search.Code: GET https://api.github.com/search/code?q=github.com+PyO3+pyo3+repo%3Agoogle%2Foss-fuzz+in%3Afile+filename%3Aproject.yaml: 400 []
cargo/pyo3-macros-backend 0.29.3 🟢 6.7
Details
CheckScoreReason
Maintained🟢 1030 commit(s) out of 30 and 10 issue activity out of 30 found in the last 90 days -- score normalized to 10
Code-Review🟢 8GitHub code reviews found for 25 commits out of the last 30 -- score normalized to 8
CII-Best-Practices⚠️ 0no badge detected
Vulnerabilities🟢 10no vulnerabilities detected
Signed-Releases⚠️ -1no releases found
License🟢 10license file detected
Packaging⚠️ -1no published package detected
Token-Permissions⚠️ 0non read-only tokens detected in GitHub workflows
Dependency-Update-Tool🟢 10update tool detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies🟢 7dependency not pinned by hash detected -- score normalized to 7
Binary-Artifacts🟢 10no binaries found in the repo
Security-Policy⚠️ 0security policy file not detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Fuzzing⚠️ -1internal error: internal error: Client.Search.Code: Search.Code: GET https://api.github.com/search/code?q=github.com+PyO3+pyo3+repo%3Agoogle%2Foss-fuzz+in%3Afile+filename%3Aproject.yaml: 400 []
cargo/quote 1.0.47 🟢 4.3
Details
CheckScoreReason
Code-Review⚠️ 0Found 1/28 approved changesets -- score normalized to 0
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Packaging⚠️ -1packaging workflow not detected
Maintained🟢 34 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 3
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy🟢 3security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/regex 1.13.1 🟢 6.3
Details
CheckScoreReason
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Maintained🟢 1021 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 1Found 3/19 approved changesets -- score normalized to 1
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Binary-Artifacts🟢 10no binaries found in the repo
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Fuzzing🟢 10project is fuzzed
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy🟢 10security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/regex-automata 0.4.18 🟢 6.3
Details
CheckScoreReason
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Maintained🟢 1021 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 1Found 3/19 approved changesets -- score normalized to 1
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Binary-Artifacts🟢 10no binaries found in the repo
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Fuzzing🟢 10project is fuzzed
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy🟢 10security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/regex-syntax 0.8.11 🟢 6.3
Details
CheckScoreReason
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Maintained🟢 1021 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 1Found 3/19 approved changesets -- score normalized to 1
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Binary-Artifacts🟢 10no binaries found in the repo
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Fuzzing🟢 10project is fuzzed
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy🟢 10security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/syn 2.0.119 🟢 5.7
Details
CheckScoreReason
Code-Review⚠️ 0Found 2/23 approved changesets -- score normalized to 0
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Maintained🟢 1030 commit(s) and 15 issue activity found in the last 90 days -- score normalized to 10
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Binary-Artifacts🟢 10no binaries found in the repo
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Fuzzing🟢 10project is fuzzed
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy🟢 3security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/target-lexicon 0.13.5 🟢 3.3
Details
CheckScoreReason
Code-Review🟢 5Found 14/25 approved changesets -- score normalized to 5
Maintained⚠️ 01 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0
Packaging⚠️ -1packaging workflow not detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Security-Policy⚠️ 0security policy file not detected
Fuzzing⚠️ 0project is not fuzzed
Signed-Releases⚠️ -1no releases found
License🟢 10license file detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
cargo/unicode-ident 1.0.26 🟢 5
Details
CheckScoreReason
Maintained🟢 1018 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Packaging⚠️ -1packaging workflow not detected
Code-Review⚠️ 0Found 1/23 approved changesets -- score normalized to 0
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Security-Policy🟢 3security policy file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0

Scanned Files

  • native/kc_native/Cargo.lock

@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from 922481e to f819e13 Compare October 13, 2026 00:26
@buluoray

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: f819e13
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns) may now be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches the Python gate's definition, KIROCREW_NATIVE_REDACTION is not 0, and no gate anchor was rebound; otherwise the Python pre-filter decides. The set of strings that get redacted is the same: 0 differences vs Python over 52,864 constructed inputs and 708,249 real strings, and 0 differences in full redact_credentials / redact_exfiltration_urls output over 989,663 real strings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: if the native gate answered False for a string the Python gate answers True for, that string would skip redaction. Bounded by the fingerprint check (a Python-side gate edit disables the native path until rebuilt) and by the differential tests. With the module not built, which is every install today because nothing in CI or packaging builds it, the code path is the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): test/test_native_redaction_gate.py with the module built (every code point at each character-class position, 200k random strings, lone-surrogate fallback, redact_credentials byte-identical); test/test_native_redaction_fallback.py without it (missing / stale / disabled / encode-failing module all answer like Python). 32 passed, 1 skipped locally.
  • No regression:
    • Tools still work (which tools/commands were run, result): All 130 test files that reference redaction run with the native gate on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed; the same 22 failures occur on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; no config or stored data changes. Checked by the fallback suite and the existing redaction suites passing on both paths.
    • Existing tests (which suites passed on this head): the redaction-referencing suites above, both new suites, mypy src/kiro_crew/security/redaction.py, cargo clippy --release --locked.
  • How to undo: set KIROCREW_NATIVE_REDACTION=0 (or uninstall kc_native) for the Python gate at runtime; revert this PR to remove the code.

@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from f819e13 to 4a85352 Compare October 13, 2026 00:53
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 4a85352
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns) may now be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches the Python gate's definition, KIROCREW_NATIVE_REDACTION is not 0, and no gate anchor was rebound; otherwise the Python pre-filter decides. The set of strings that get redacted is the same: 0 differences vs Python over 52,864 constructed inputs and 708,249 real strings, and 0 differences in full redact_credentials / redact_exfiltration_urls output over 989,663 real strings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: if the native gate answered False for a string the Python gate answers True for, that string would skip redaction. Bounded by the fingerprint check (a Python-side gate edit disables the native path until rebuilt) and by the differential tests. With the module not built, which is every install today because nothing in CI or packaging builds it, the code path is the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): test/test_native_redaction_gate.py with the module built (every code point at each character-class position, 200k random strings, lone-surrogate fallback, redact_credentials byte-identical); test/test_native_redaction_fallback.py without it (missing / stale / disabled / encode-failing module all answer like Python). 32 passed, 1 skipped locally.
  • No regression:
    • Tools still work (which tools/commands were run, result): All 130 test files that reference redaction run with the native gate on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed; the same 22 failures occur on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; no config or stored data changes. Checked by the fallback suite and the existing redaction suites passing on both paths.
    • Existing tests (which suites passed on this head): the redaction-referencing suites above, both new suites, mypy src/kiro_crew/security/redaction.py, cargo clippy --release --locked.
  • How to undo: set KIROCREW_NATIVE_REDACTION=0 (or uninstall kc_native) for the Python gate at runtime; revert this PR to remove the code.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b1bb2d48f0b6bdd9709af879df107bdccfed3c7a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b1bb2d48f0b6bdd9709af879df107bdccfed3c7a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the native gate is opt-in by build, falls back to the unchanged Python reference, and CI checks that the two agree.

Notes

  • GATE_FINGERPRINT only proves the Python gate has not changed since the build. It cannot catch a Rust edit that breaks equivalence. Only the new CI job's unskipped test_native_redaction_gate.py catches that, so that job needs to stay required. A false negative there means redact_credentials skips a real credential.
  • _NATIVE_GATE is bound once at import, so KIROCREW_NATIVE_REDACTION=0 only takes effect after a restart. That is fine for a kill switch, but the docs should say so.

[DESIGN-REVIEWED] b1bb2d4

Description read

[DESCRIPTION-READ] cf0cbe668d7030d08cde4a662c45f6036b307219c598c7fb7a006ecd1cd1eecc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b1bb2d48f0b6bdd9709af879df107bdccfed3c7a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The native gate sits next to the Python gate, the Python gate stays the default everywhere, and the speedup is measured. It does ship a Rust toolchain that no install receives yet.

Notes

  • Item 1 — oversized: "Building or shipping kc_native in CI, the wheel or the installers" is out of scope, so "every install keeps the Python gate." The crate, Cargo.lock, two license exceptions and a CI job are a permanent cost, and today they speed up nobody who hasn't built the module by hand. Subtraction: defer native/kc_native/, the CI job and the dependency-review exceptions to the PR that ships the wheel. Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 3 — inherited: KIROCREW_NATIVE_REDACTION has 1 consumer (_load_native_gate). Its only precedent is the sibling KIROCREW_NATIVE_SKILL_PROJECTION (symmetry). The fingerprint already handles drift, and pip uninstall kc_native already turns the gate off. Subtraction: delete the env-var branch.
  • Item 5 — undeclared rider: native/kc_native/README.md is a new markdown file, and AGENTS.md says "MUST NOT create additional markdown files unless explicitly instructed". Subtraction: delete it, since the description already carries its build steps.

What this change ships

Inventory (6 items) — 3 justified

Intent: make the credential pre-filter about 5x cheaper in CPU with identical answers. ADDITION (perf).

  1. An optional Rust module, kc_native, offers a single-pass credential pre-filter — oversized
  2. Redaction uses the native gate when it is importable and its fingerprint matches; otherwise it falls back to Python — justified
  3. New env var KIROCREW_NATIVE_REDACTION=0 turns the native gate off — inherited
  4. New CI job builds the module and runs the equivalence suite — justified
  5. New native/kc_native/README.md — undeclared
  6. Two license exceptions in dependency-review for build-time crates — justified

[FIRST-PRINCIPLES-REVIEWED] b1bb2d4

Description read

[DESCRIPTION-READ] e3e032779b26b366c6dec957b00ef411c9f194952927bb6e062d681befac17d8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b1bb2d48f0b6bdd9709af879df107bdccfed3c7a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b1bb2d4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b1bb2d48f0b6bdd9709af879df107bdccfed3c7a false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b1bb2d48f0b6bdd9709af879df107bdccfed3c7a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b1bb2d4

Verdict parsed from the review's SHA-scoped output markers for commit b1bb2d48f0b6bdd9709af879df107bdccfed3c7a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b1bb2d48f0b6bdd9709af879df107bdccfed3c7a false-positive: <reason, 20 to 500 characters>

@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from 4a85352 to 86373eb Compare October 13, 2026 01:26
@buluoray

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 86373eb
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns) may now be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches the Python gate's definition, KIROCREW_NATIVE_REDACTION is not 0, and no gate anchor was rebound; otherwise the Python pre-filter decides. The set of strings that get redacted is the same: 0 differences vs Python over 52,864 constructed inputs and 708,249 real strings, and 0 differences in full redact_credentials / redact_exfiltration_urls output over 989,663 real strings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: if the native gate answered False for a string the Python gate answers True for, that string would skip redaction. Bounded by the fingerprint check (a Python-side gate edit disables the native path until rebuilt) and by the differential tests. With the module not built, which is every install today because nothing in CI or packaging builds it, the code path is the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): test/test_native_redaction_gate.py with the module built (every code point at each character-class position, 200k random strings, lone-surrogate fallback, redact_credentials byte-identical); test/test_native_redaction_fallback.py without it (missing / stale / disabled / encode-failing module all answer like Python). 32 passed, 1 skipped locally.
  • No regression:
    • Tools still work (which tools/commands were run, result): All 130 test files that reference redaction run with the native gate on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed; the same 22 failures occur on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; no config or stored data changes. Checked by the fallback suite and the existing redaction suites passing on both paths.
    • Existing tests (which suites passed on this head): the redaction-referencing suites above, both new suites, mypy src/kiro_crew/security/redaction.py, cargo clippy --release --locked.
  • How to undo: set KIROCREW_NATIVE_REDACTION=0 (or uninstall kc_native) for the Python gate at runtime; revert this PR to remove the code.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does b1bb2d48f0b6bdd9709af879df107bdccfed3c7a newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 13 classified, 0 newly refused, 13 still allowed.
  • ubuntu-latest (platform posix): 13 classified, 0 newly refused, 13 still allowed.
  • windows-latest (platform windows): 7 classified, 0 newly refused, 7 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

Reason: This change tightens nothing. It adds an optional native (kc_native) version of the credential pre-filter, which is meant to give the same answer for every string, and redaction falls back to the Python gate whenever the module is missing, stale, disabled or fails. None of the shell deny tiers (sensitive-bash, exfil, deny-rules) changed. The exfil tier calls redact_credentials, so it reaches this pre-filter indirectly, and I wrote boundary candidates for that path. The redaction equivalence itself is a gate outside the classifier, named under UNADJUDICATED.

TIGHTENING: No rule got stricter. The pre-filter dispatch now goes through an optional native gate -- + if _NATIVE_GATE is not None: / + try: / + return _NATIVE_GATE(text) / + except UnicodeEncodeError: / + pass / + return _might_contain_credential_py(text). The threat it addresses is speed and memory (no cached UTF-8 copy kept inside each str), not a new refusal. The only security property at stake is that a wrong False would let a credential skip redaction, and that is the opposite direction from this lane.

ADJUDICATED: 18 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE: The diff assumes executable names (cargo, maturin, pip, python), path separators (native/kc_native, dist/kc_native-*.whl) and an env-var switch (KIROCREW_NATIVE_REDACTION).

  • posix: rows cover cargo test, maturin build and maturin develop (the README's VIRTUAL_ENV/PATH form), wheel install, the pytest suites, KIROCREW_NATIVE_REDACTION=0 and KC_REDACTION_CORPUS= prefixes, cargo tree, and a grep of the pre-filter anchors.
  • windows: rows cover cargo.exe/maturin.exe with \ paths, .venv\Scripts\pip.exe and python.exe, and the cmd.exe set VAR=0 && spelling.
  • any: the python -c "import kc_native" fingerprint check and a git diff of the two gate files.
  • Not probed: I wrote no PowerShell $env:KIROCREW_NATIVE_REDACTION="0" row and no UNC-path row; the diff does not rely on either.

UNADJUDICATED: The native redaction pre-filter, kc_native.might_contain_credential, sits behind _load_native_gate / _might_contain_credential. Its call path is: redact_credentials → _might_contain_credential → _NATIVE_GATE(text). It is reached from the exfil tier (exfil.py:513 _credential_clean, exfil.py:629 _blocked_link_safe_domain), transcript and history scrubbing, and messaging renderers. The classifier only settles shell command lines. It cannot show that the native and Python gates agree, and it most likely runs without kc_native built, so it will exercise the Python fallback. I would have submitted these text inputs:

  • latin-1 text containing U+0085 or U+00A0 next to ://user:pw@host
  • Authorİzation / authorızation
  • \x1c-separated URIs
  • CJK and emoji transcript rows longer than 4096 code points
  • a str holding a lone surrogate

Two places are the most fragile, though I have not confirmed any problem in either:

  • Unicode-version drift: the hand-listed PY_WS_WIDE / map_wide whitespace set is pinned to Unicode 15.0, while the fingerprint covers unicodedata.unidata_version.
  • The ≥4096-unit path: that path releases the GIL while reading the str's storage.

[SCOPE-REVIEWED] b1bb2d4

@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from 86373eb to a1dcced Compare October 13, 2026 02:01
@buluoray

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: a1dcced
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns) may now be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches the Python gate's definition, KIROCREW_NATIVE_REDACTION is not 0; otherwise the Python pre-filter decides. The set of strings that get redacted is the same: 0 differences vs Python over 52,864 constructed inputs and 708,249 real strings, and 0 differences in full redact_credentials / redact_exfiltration_urls output over 989,663 real strings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: if the native gate answered False for a string the Python gate answers True for, that string would skip redaction. Bounded by the fingerprint check (a Python-side gate edit disables the native path until rebuilt) and by the differential tests, which the new Native Redaction Gate (kc_native) CI job runs against a freshly built module on every PR. With the module not installed, which is every install today because the wheel and installers do not ship it, the code path is the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): test/test_native_redaction_gate.py with the module built (every code point at each character-class position, 200k random strings, lone-surrogate fallback, redact_credentials byte-identical); test/test_native_redaction_fallback.py without it (missing / stale / disabled / encode-failing module all answer like Python). Locally on this head: those two plus test/test_credential_prefilter.py, 536 passed, 1 skipped; cargo test --release --locked 2 passed.
  • No regression:
    • Tools still work (which tools/commands were run, result): All 130 test files that reference redaction run with the native gate on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed; the same 22 failures occur on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; no config or stored data changes. Checked by the fallback suite and the existing redaction suites passing on both paths.
    • Existing tests (which suites passed on this head): the redaction-referencing suites above, both new suites, mypy src/kiro_crew/security/redaction.py, cargo clippy --release --locked.
  • How to undo: set KIROCREW_NATIVE_REDACTION=0 (or uninstall kc_native) for the Python gate at runtime; revert this PR to remove the code.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from a1dcced to b8269ea Compare October 13, 2026 02:38
@buluoray

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: b8269ea
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns) may now be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches the Python gate's definition, KIROCREW_NATIVE_REDACTION is not 0; otherwise the Python pre-filter decides. The set of strings that get redacted is the same: 0 differences vs Python over 52,864 constructed inputs and 708,249 real strings, and 0 differences in full redact_credentials / redact_exfiltration_urls output over 989,663 real strings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: if the native gate answered False for a string the Python gate answers True for, that string would skip redaction. Bounded by the fingerprint check (a Python-side gate edit disables the native path until rebuilt) and by the differential tests, which the new Native Redaction Gate (kc_native) CI job runs against a freshly built module on every PR. With the module not installed, which is every install today because the wheel and installers do not ship it, the code path is the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): test/test_native_redaction_gate.py with the module built (every code point at each character-class position, 200k random strings, lone-surrogate fallback, redact_credentials byte-identical); test/test_native_redaction_fallback.py without it (missing / stale / disabled / encode-failing module all answer like Python). Locally on this head: those two plus test/test_credential_prefilter.py, 536 passed, 1 skipped; cargo test --release --locked 2 passed.
  • No regression:
    • Tools still work (which tools/commands were run, result): All 130 test files that reference redaction run with the native gate on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed; the same 22 failures occur on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; no config or stored data changes. Checked by the fallback suite and the existing redaction suites passing on both paths.
    • Existing tests (which suites passed on this head): the redaction-referencing suites above, both new suites, mypy src/kiro_crew/security/redaction.py, cargo clippy --release --locked.
  • How to undo: set KIROCREW_NATIVE_REDACTION=0 (or uninstall kc_native) for the Python gate at runtime; revert this PR to remove the code.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 13, 2026
@buluoray
buluoray enabled auto-merge (squash) October 13, 2026 03:07
@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from b8269ea to 1a816b9 Compare October 13, 2026 03:11
@buluoray

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 1a816b9
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns) may now be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches the Python gate's definition, KIROCREW_NATIVE_REDACTION is not 0; otherwise the Python pre-filter decides. The set of strings that get redacted is the same: 0 differences vs Python over 52,864 constructed inputs and 708,249 real strings, and 0 differences in full redact_credentials / redact_exfiltration_urls output over 989,663 real strings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: if the native gate answered False for a string the Python gate answers True for, that string would skip redaction. Bounded by the fingerprint check (a Python-side gate edit disables the native path until rebuilt) and by the differential tests, which the new Native Redaction Gate (kc_native) CI job runs against a freshly built module on every PR. With the module not installed, which is every install today because the wheel and installers do not ship it, the code path is the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): test/test_native_redaction_gate.py with the module built (every code point at each character-class position, 200k random strings, lone-surrogate fallback, redact_credentials byte-identical); test/test_native_redaction_fallback.py without it (missing / stale / disabled / encode-failing module all answer like Python). Locally on this head: those two plus test/test_credential_prefilter.py, 536 passed, 1 skipped; cargo test --release --locked 2 passed.
  • No regression:
    • Tools still work (which tools/commands were run, result): All 130 test files that reference redaction run with the native gate on and with KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed; the same 22 failures occur on unmodified origin/main in this environment (test/test_design_tweak_backend.py, one in test/test_auto_improvement_members.py).
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; no config or stored data changes. Checked by the fallback suite and the existing redaction suites passing on both paths.
    • Existing tests (which suites passed on this head): the redaction-referencing suites above, both new suites, mypy src/kiro_crew/security/redaction.py, cargo clippy --release --locked.
  • How to undo: set KIROCREW_NATIVE_REDACTION=0 (or uninstall kc_native) for the Python gate at runtime; revert this PR to remove the code.

@buluoray
buluoray marked this pull request as draft October 14, 2026 18:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 14, 2026
@Zedmor

Zedmor commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

@buluoray, data for this one from the measurements now in the hybrid RFC's §2.4 (#22065). I haven't touched this branch.

My suggestion is to park this until a shared native wheel pipeline exists (the RFC's Phase 3 kirocrew-native, which iamwhatever asked to settle first), and to revisit it then with a refreshed fingerprint.

kc_native.might_contain_credential does the 21 literal and 5 regex checks
of redaction's credential pre-filter in one regex-crate scan and drops the
GIL on large strings. Python's \s and (?i) folding are spelled out by hand,
and the module exports GATE_FINGERPRINT so the caller can refuse a build
that no longer matches the Python definition.

It reads each str from its own storage (PyUnicode_KIND / PyUnicode_DATA)
instead of PyUnicode_AsUTF8AndSize, which would keep a UTF-8 copy inside
every non-ASCII str for the rest of its life. Lone surrogates still raise
UnicodeEncodeError so the caller falls back. Reaching str storage needs the
full API, so the module is not abi3 and is built per CPython minor.
@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from 7162f58 to eec9817 Compare October 25, 2026 22:09
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 25, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]:

Sensitive change review

  • Head commit: eec9817
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns, now owned by redaction_patterns.py) may be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py. redaction.py only lists the four new names in its type-checking block; its facade serves them.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches a hash of the Python gate's literals, all six anchor patterns (both literal-led Authorization halves included) and the Unicode version, and KIROCREW_NATIVE_REDACTION is not 0; otherwise, and for any string the module cannot read (a lone surrogate), the Python pre-filter decides. The set of strings that get redacted is the same: on this head, gate answers and full redact_credentials + redact_exfiltration_urls output hash identically over 611,572 local transcript strings for origin/main, this head without the module, and this head with it.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: a string the native gate answers False for while the Python gate answers True would skip redaction. Bounded by the fingerprint (any edit to the Python gate turns the native path off until the module is rebuilt with the new value) and by the differential tests, which the Native Redaction Gate (kc_native) CI job runs against a freshly built module. No install ships the module today, so every install runs the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): With the module built from this head: test/test_native_redaction_gate.py, test/test_native_redaction_fallback.py, test/test_credential_prefilter.py, test/test_credential_prefilter_anchors.py: 542 passed, 1 skipped (corpus case, run separately over 611,572 strings: passed). Mutations that turn named tests red: dropping the UnicodeEncodeError fallback, the fingerprint check or the env switch in Python; dropping the eyJ literal or the dotted/dotless i fold in Rust.
  • No regression:
    • Tools still work (which tools/commands were run, result): cargo test --release --locked 2 passed; cargo fmt --check and cargo clippy --release --locked clean; mypy src/kiro_crew/security/redaction_patterns.py clean.
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; display_verdicts still finds redaction._NATIVE_GATE and redaction._native_gate_fingerprint through the facade. Without the module the corpus output hash equals origin/main's.
    • Existing tests (which suites passed on this head): Without the module, the 28 test files that reference the redaction owners, display_verdicts or _NATIVE_GATE: 2,093 passed, 5 skipped, 1 failed (test_auto_improvement_members.py::test_assignments_build_real_context_from_only_their_private_store, which rejects this host's pytest temp dir; it touches no redaction code). CI on this head is still running.
  • How to undo: Set KIROCREW_NATIVE_REDACTION=0, or uninstall kc_native; reverting this PR restores the previous code exactly.

… to Python

_might_contain_credential (redaction_patterns) uses kc_native when it
imports, its GATE_FINGERPRINT matches the Python pre-filter (literals,
anchors, Unicode version), and KIROCREW_NATIVE_REDACTION is not 0.
Otherwise, or when the module raises UnicodeEncodeError, the unchanged
Python pre-filter (_might_contain_credential_py) runs.

The fingerprint now covers the literal-led Telegram, Discord and both
Authorization anchors main spells today; the Rust gate keeps their plain
spellings, which accept the same strings.

test_native_redaction_gate diffs the native and Python gates code point by
code point, on random strings, and on redact_credentials output (skipped
when the module is not built). test_native_redaction_fallback covers the
missing, stale, disabled and failing-module paths without it. A new CI job,
Native Redaction Gate, builds the module and runs both suites plus the
Python pre-filter suite with the native gate live.
@buluoray
buluoray force-pushed the perf/native-redaction-gate branch from eec9817 to b1bb2d4 Compare October 25, 2026 22:59
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]:

Sensitive change review

  • Head commit: b1bb2d4
  • What rule changed: No redaction rule changed. _might_contain_credential (the pre-filter that decides whether a string needs the full credential patterns, now owned by redaction_patterns.py) may be answered by the optional native kc_native.might_contain_credential instead of the Python implementation, which is kept verbatim as _might_contain_credential_py. redaction.py only lists the four new names in its type-checking block; its facade serves them.
  • Before -> after (what was allowed/blocked, now): Before: the Python pre-filter decided. After: the native pre-filter decides when the module imports, its GATE_FINGERPRINT matches a hash of the Python gate's literals, all six anchor patterns (both literal-led Authorization halves included) and the Unicode version, and KIROCREW_NATIVE_REDACTION is not 0; otherwise, and for any string the module cannot read (a lone surrogate), the Python pre-filter decides. The set of strings that get redacted is the same: on this head, gate answers and full redact_credentials + redact_exfiltration_urls output hash identically over 611,572 local transcript strings for origin/main, this head without the module, and this head with it.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): Leaks a secret: a string the native gate answers False for while the Python gate answers True would skip redaction. Bounded by the fingerprint (any edit to the Python gate turns the native path off until the module is rebuilt with the new value) and by the differential tests, which the Native Redaction Gate (kc_native) CI job runs against a freshly built module. No install ships the module today, so every install runs the unchanged Python gate.
  • Evidence checked (which sandbox result proves it): With the module built from this head: test/test_native_redaction_gate.py, test/test_native_redaction_fallback.py, test/test_credential_prefilter.py, test/test_credential_prefilter_anchors.py: 542 passed, 1 skipped (corpus case, run separately over 611,572 strings: passed). Mutations that turn named tests red: dropping the UnicodeEncodeError fallback, the fingerprint check or the env switch in Python; dropping the eyJ literal or the dotted/dotless i fold in Rust.
  • No regression:
    • Tools still work (which tools/commands were run, result): cargo test --release --locked 2 passed; cargo fmt --check and cargo clippy --release --locked clean; mypy src/kiro_crew/security/redaction_patterns.py clean.
    • Backward compatible (old config / old data / old callers still work, how checked): _might_contain_credential keeps its name, signature and answers; display_verdicts still finds redaction._NATIVE_GATE and redaction._native_gate_fingerprint through the facade. Without the module the corpus output hash equals origin/main's.
    • Existing tests (which suites passed on this head): Without the module, the 28 test files that reference the redaction owners, display_verdicts or _NATIVE_GATE: 2,093 passed, 5 skipped, 1 failed (test_auto_improvement_members.py::test_assignments_build_real_context_from_only_their_private_store, which rejects this host's pytest temp dir; it touches no redaction code). The only change since eec9817 is .github/workflows/ci.yml: the native job now waits for the lint gate, as test/test_fast_gate_barrier.py requires (71 workflow test files: 3,980 passed, 13 skipped locally). No redaction file changed; on eec9817 every other CI check that had finished passed.
  • How to undo: Set KIROCREW_NATIVE_REDACTION=0, or uninstall kc_native; reverting this PR restores the previous code exactly.

@buluoray
buluoray marked this pull request as ready for review October 26, 2026 00:05
@buluoray
buluoray enabled auto-merge (squash) October 26, 2026 00:05
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@buluoray
buluoray merged commit 3168ce8 into main Oct 26, 2026
122 checks passed
@buluoray
buluoray deleted the perf/native-redaction-gate branch October 26, 2026 01:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(redaction): optional native single-pass credential pre-filter

4 participants