Repository navigation
perf(redaction): native single-pass credential gate, Python fallback - #19628
Conversation
Dependency ReviewThe following issues were found:
License Issuesnative/kc_native/Cargo.lock
OpenSSF ScorecardScorecard details
Scanned Files
|
922481e to
f819e13
Compare
Sensitive change review
|
f819e13 to
4a85352
Compare
Sensitive change review
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the native gate is opt-in by build, falls back to the unchanged Python reference, and CI checks that the two agree. Notes
[DESIGN-REVIEWED] b1bb2d4 Description read[DESCRIPTION-READ] cf0cbe668d7030d08cde4a662c45f6036b307219c598c7fb7a006ecd1cd1eecc That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The native gate sits next to the Python gate, the Python gate stays the default everywhere, and the speedup is measured. It does ship a Rust toolchain that no install receives yet. Notes
What this change shipsInventory (6 items) — 3 justifiedIntent: make the credential pre-filter about 5x cheaper in CPU with identical answers. ADDITION (perf).
[FIRST-PRINCIPLES-REVIEWED] b1bb2d4 Description read[DESCRIPTION-READ] e3e032779b26b366c6dec957b00ef411c9f194952927bb6e062d681befac17d8 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
4a85352 to
86373eb
Compare
Sensitive change review
|
Security Scope Review -- ✅ PASSWhich legitimate operations does Verdict basis: adjudicated, zero confirmed regressions. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: PASS Reason: This change tightens nothing. It adds an optional native ( TIGHTENING: No rule got stricter. The pre-filter dispatch now goes through an optional native gate -- ADJUDICATED: 18 candidates, 0 confirmed newly-refused PLATFORM COVERAGE: The diff assumes executable names (
UNADJUDICATED: The native redaction pre-filter,
Two places are the most fragile, though I have not confirmed any problem in either:
[SCOPE-REVIEWED] b1bb2d4 |
86373eb to
a1dcced
Compare
Sensitive change review
|
a1dcced to
b8269ea
Compare
Sensitive change review
|
b8269ea to
1a816b9
Compare
Sensitive change review
|
|
@buluoray, data for this one from the measurements now in the hybrid RFC's §2.4 (#22065). I haven't touched this branch.
My suggestion is to park this until a shared native wheel pipeline exists (the RFC's Phase 3 |
kc_native.might_contain_credential does the 21 literal and 5 regex checks of redaction's credential pre-filter in one regex-crate scan and drops the GIL on large strings. Python's \s and (?i) folding are spelled out by hand, and the module exports GATE_FINGERPRINT so the caller can refuse a build that no longer matches the Python definition. It reads each str from its own storage (PyUnicode_KIND / PyUnicode_DATA) instead of PyUnicode_AsUTF8AndSize, which would keep a UTF-8 copy inside every non-ASCII str for the rest of its life. Lone surrogates still raise UnicodeEncodeError so the caller falls back. Reaching str storage needs the full API, so the module is not abi3 and is built per CPython minor.
7162f58 to
eec9817
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Sensitive change review
|
… to Python _might_contain_credential (redaction_patterns) uses kc_native when it imports, its GATE_FINGERPRINT matches the Python pre-filter (literals, anchors, Unicode version), and KIROCREW_NATIVE_REDACTION is not 0. Otherwise, or when the module raises UnicodeEncodeError, the unchanged Python pre-filter (_might_contain_credential_py) runs. The fingerprint now covers the literal-led Telegram, Discord and both Authorization anchors main spells today; the Rust gate keeps their plain spellings, which accept the same strings. test_native_redaction_gate diffs the native and Python gates code point by code point, on random strings, and on redact_credentials output (skipped when the module is not built). test_native_redaction_fallback covers the missing, stale, disabled and failing-module paths without it. A new CI job, Native Redaction Gate, builds the module and runs both suites plus the Python pre-filter suite with the native gate live.
eec9817 to
b1bb2d4
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Sensitive change review
|
Problem / Motivation
Goal: Make the credential pre-filter that every redaction call runs about 5x cheaper in CPU, with identical answers, while installs without the native module keep working exactly as today.
Profiling a gateway on a copy of real data,
redact_credentialswas the largest pure-Python CPU path under load (22% of non-idle samples, mostly slot-detail rendering). Its pre-filter_might_contain_credentialalone was 8%.Why it matters
Redaction runs on every transcript string the dashboard renders and the gateway stores, so its CPU cost is paid on every session view and grows with session length.
Not a goal
kc_nativein CI, the wheel or the installers. Until a later change does that, every install keeps the Python gate.What changed (motivation → approach → change)
regex-crate scan, with the GIL released for large strings, removes most of its cost without touching what redaction replaces.native/kc_native/(new). A PyO3 module with one function,might_contain_credential, and aGATE_FINGERPRINTconstant. Python's\sand(?i)folding are spelled out by hand. The module reads eachstrfrom its own storage (PyUnicode_KIND/PyUnicode_DATA) instead of asking for UTF-8, because CPython caches that UTF-8 copy inside every non-ASCIIstruntil it dies.src/kiro_crew/security/redaction_patterns.py(the pre-filter's owner since the redaction split;redactionserves the new names through its facade). The existing pre-filter is renamed_might_contain_credential_py, body unchanged._might_contain_credentialuses the native gate by default when it is usable, and the Python one otherwise:kc_nativemissing, or built for another CPython version: the import raisesImportError→ Python.GATE_FINGERPRINTdoes not match a hash of the Python gate's literals, every anchor pattern (both literal-ledAuthorizationhalves included) and Unicode version (the Python gate was edited without a rebuild) → Python.KIROCREW_NATIVE_REDACTION=0→ Python.UnicodeEncodeError→ Python for that call..github/workflows/ci.yml. A new job,Native Redaction Gate (kc_native), builds the module (cargo test --release --locked,maturin build --release --locked), installs it, confirms it imports, and runstest/test_native_redaction_gate.pyunskipped together with the fallback suite andtest/test_credential_prefilter.py. This builds the module for testing only; the wheel and installers still do not ship it (added after the Design Review asked that CI hold the equivalence claim)..github/dependency-review-config.yml. Two durable per-package license exceptions for build-time-only crates in the newCargo.lock(target-lexicon: Apache-2.0 WITH LLVM-exception;unicode-ident: (MIT OR Apache-2.0) AND Unicode-3.0), with the evidence links the file asks for.Product shape: unchanged. Redaction gives the same answer for every string on both paths, and no install gets the native path until it builds the module.
Measured
On current
main, whose Python gate already leads each anchor with a literal: pre-filter CPU over 611,572 local transcript strings, best of 3, 2.05 s Python vs 0.42 s native. The table below was measured before that rewrite landed.redact_credentialsCPU, same corpus"中" * 100_000strings scannedPyUnicode_AsUTF8AndSize)strstorage)Equivalence with the Python gate after the rebase: gate answers and full
redact_credentials+redact_exfiltration_urlsoutput hash identically over 611,572 local transcript strings fororigin/main, this branch without the module, and this branch with it (626 strings flagged by all three). Before the rebase: 0 differences over 52,864 constructed inputs and a 708,249-string corpus of real strings. Fullredact_credentials/redact_exfiltration_urlsoutput: 0 differences over 989,663 real strings.Risks / costs
strstorage needs the full C API, so the module must be built separately for each CPython minor version. A mismatched build fails to import and the Python path runs (test_missing_module_keeps_python); redaction does not break.unicodedata.unidata_versionbecause the Rust side hand-encodes Python's\sset andicase folding, and the committedGATE_FINGERPRINTis the 3.12 (Unicode 15.0) value. On 3.13+ a build imports but its fingerprint does not match, so the Python gate runs and_load_native_gatelogs one debug line saying so. Extending it to later minors means carrying their fingerprints and testing them, which belongs with the change that ships per-minor builds.KIROCREW_NATIVE_REDACTION=0turns it off.Backwards compatibility
Compatible: with the module absent (every install today) the code path is the existing Python pre-filter; the new env var defaults to on and only matters when the module is built.
Removes nothing:
_might_contain_credentialkeeps its name, signature and answers; the old body is kept verbatim as_might_contain_credential_py.Tests
test/test_native_redaction_fallback.py(runs without the module): default on with a matching module, missing module, stale fingerprint,KIROCREW_NATIVE_REDACTION=0, and a gate that raisesUnicodeEncodeErrorall answer like the Python gate.test/test_native_redaction_gate.py(skipped when the module is not built): every code point at each character-class position, 200k random strings, lone-surrogate fallback, the no-growth property for non-ASCII strings, andredact_credentialsoutput, native vs Python.test/test_security_gate_liveness.py:_PACKAGE_LINE_BUDGETraised from 32,530 to 32,601 for the 71 lines the loader, fingerprint, mismatch log line, dispatcher and facade type-checking names add, with the reason recorded beside it as the file asks.test/test_credential_prefilter.py::test_differential_test_detects_a_too_narrow_prefilter: the control disarms the Python gate by rebinding its anchors, which the native gate does not read, so it now also sets_NATIVE_GATEtoNone.test/test_prepare_pr_profiles.py:cargojoins the floor'sexempt_tools, with its reason: the new job'scargo testcovers only the optional crate and a contributor host need not have a Rust toolchain.Manual verification
After rebasing onto current
main(CPython 3.12.13, Linux; macOS and Windows not run):cargo test --release --locked2 passed;cargo fmt --checkandcargo clippy --release --lockedclean;maturin build --release --lockedwith 1.15.0.test_native_redaction_gate.py,test_native_redaction_fallback.py,test_credential_prefilter.py,test_credential_prefilter_anchors.py: 542 passed, 1 skipped (corpus); the corpus case run separately over 611,572 strings: passed.display_verdictsor_NATIVE_GATE: 2,093 passed, 5 skipped, 1 failed (test_auto_improvement_members.py::test_assignments_build_real_context_from_only_their_private_store, which rejects this host's pytest temp dir under the crew scratch tree; it touches no redaction code).UnicodeEncodeErrorfallback (test_gate_that_cannot_encode_falls_back), the fingerprint check (test_stale_module_keeps_python), the env switch (test_env_zero_turns_it_off); in Rust, dropping theeyJliteral (test_random_differential) and the dotted/dotless i fold (test_every_codepoint_at_each_class_position[Author{c}zation]).mypy src/kiro_crew/security/redaction_patterns.py: clean.Before the rebase:
maturin develop --release --lockedinto the repo venv (CPython 3.12);cargo fmt --checkandcargo clippy --release --lockedclean.KIROCREW_NATIVE_REDACTION=0: 17,178 / 17,177 passed. The same 22 tests fail in both runs and also fail on unmodifiedorigin/mainin this environment (test/test_design_tweak_backend.py, one intest/test_auto_improvement_members.py).mypy src/kiro_crew/security/redaction.py: clean.Related Issues
Closes #24137
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.