Repository navigation
perf(security): cut the per-run decode and scan work in redaction passes 2 and 3 - #7067
Conversation
31a51b6 to
fdc8284
Compare
fdc8284 to
9fa64de
Compare
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of All callers verified: Design-Verdict: PASS Verdict-neutral gates on an already-pinned superset invariant, cross-version differential tests, and rejected alternatives each pinned by a test — sound and proportionate. [DESIGN-REVIEWED] 9f61023 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All claims verified against the base tree: First-Principles-Verdict: PASS Every item removes a measured cost on the redaction hot path or pins a named bypass risk, with byte-identical output enforced by an independent oracle. What this change shipsIntent: make credential redaction affordable on hash-heavy and blob-heavy content without moving any verdict — a FIX (measured perf defect, 68–78% of function time in two loop bodies).
WatchItem 6 hardens the base commit's pass-3 split rather than anything this diff changes — fine here, but it belongs with #7056 if that PR is still amendable. [FIRST-PRINCIPLES-REVIEWED] 9f61023 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
9fa64de to
8453625
Compare
…ses 2 and 3 Counts printable bytes with bytes.translate instead of a Python per-byte loop, and gates the decoded-blob scan behind the existing pre-filter. Byte-identical.
8453625 to
9f61023
Compare
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
Problem / Motivation
redact_credentialsruns three passes. #7056 cut most of the scanning cost bygating pass 1 behind a cheap superset pre-filter and feeding passes 2 and 3 from
one shared base64 scan. What it did not touch is the per-run work inside the
pass 2 and pass 3 loop bodies, which is now the largest remaining term on
content carrying many base64-alphabet runs.
Measured on this branch's base (#7056's head,
0235df78), pass 2 + pass 3 loopbodies with the shared scan excluded:
redact_credentialsTwo costs drive that:
_decodes_to_printable_textcounted printable bytes in a Python loop. Itruns once per base64 run and again per 40-char window as gate 7 of
_looks_like_secret_key, and its cost scaled with the decoded byte countrather than with the 40-char window. On a 2 KB encoded blob it was 86.3 us of
the 88.2 us
_contains_bare_secretspent on that run — 98%.decoded blobs are almost never credentials: 0 of 18 in the session corpus and
1 of 849 in the hash-heavy corpus.
Why it matters
This is the redaction boundary; a regression writes live credentials into
persisted chat history. So the constraint is byte-identical
(text, warnings)output, and the value has to be stated honestly:
because 981 KB of real text contains only 116 qualifying runs. The remaining
cost there is pass 1 and the shared scan, which perf(security): gate the credential scan behind a cheap superset pre-filter #7056 owns.
commit lists, encoded attachments — where the loop bodies were 68–78% of the
function.
A digest-keyed cache for
redact_credentialswas in flight concurrently(branch
perf/redact-credentials-cache), so these figures are miss-path cost.That is deliberate: a cache cannot help a miss, and this change targets exactly
the term a cache leaves behind. I checked for a cache at the function entry first
and found none —
redact_credentialscarries no memo, and the only sha256 cachenearby is
_entry_cacheinchat_persistence.py, keyed on the whole messagedict in front of the entry builder, which misses on every new message. Do not
read the numbers above as a whole-workload improvement a landed cache would absorb.
Stacking: #7056 is still open, so this branches off its head rather than
main, and the diff againstmainincludes its commit. I expect #7056 to mergefirst, then the cache PR and this one in either order — the cache lands at the
function entry and this lands in the pass 2/3 loop bodies.
What changed in code
Two changes in
src/kiro_crew/security.py, three executable lines.1.
_decodes_to_printable_textcounts printable bytes withbytes.translate.translate(None, _PRINTABLE_BYTES)returns exactly the bytes not in the set, solen(raw) - len(...)is the member count. An integer identity, so the ratio andthe comparison are bit-identical to the previous per-byte sum. Measured 14.6x at
48 bytes rising to 69x at 1500.
2. Pass 2's decoded-blob scan is gated behind the pre-filter, length-gated.
_might_contain_credentialis a proven superset of_CREDENTIAL_PATTERNS, so theverdict cannot move. It is not unconditionally cheaper, though: its ~590 ns
floor is fixed while the alternation scales with length, so the crossover was
measured rather than assumed — 8 chars 684 vs 494 ns (alternation wins), 12 chars
746 vs 898 (pre-filter wins), 3.4x by 256. Decoded blobs (measured 12–31 chars)
genuinely straddle that line.
_PREFILTER_MIN_LENis held at 16, above the measured crossover of 12, onpurpose. The gate is verdict-neutral — the pre-filter is a proven superset, so
either route reaches the same answer — which makes a conservative threshold cost at
most one alternation scan on a 12–15 character blob, and makes it robust to the
crossover drifting as the pre-filter's own cost changes. It has already drifted
once: this PR was rebased onto a revision of #7056 that replaced the
.lower()substring check with a case-insensitive
Authorizationanchor, moving the crossoverfrom 16 to 12. The figures above are re-measured against that implementation.
A third optimisation was written, then removed as unsound — read this before re-adding it
An earlier revision short-circuited pass 2's decode when
len(chunk) % 4wasnon-zero, reasoning that
validate=Truerejects such a length. That isinterpreter dependent and is a redaction bypass on the versions CI builds.
binascii.a2b_base64's padding leniency changed withstrict_mode:=(len 41, %4 = 1)==(len 42, %4 = 2)==(len 45, %4 = 1)So on 3.10 the guard skipped a chunk that does decode, leaving a base64-encoded
credential unredacted. No version-invariant form of the predicate exists either:
43 data chars + "=="decodes on 3.10 while failing both a total-length and astripped-length test. The guard is gone, the reasoning is recorded at the call
site, and
test_a_decode_length_precondition_would_be_version_dependentplustest_a_length_short_circuit_would_leak_an_encoded_credentialpin it.Removing it cost essentially nothing measurable — it was the least valuable of the
three changes, and the benchmark below is after its removal.
Also deliberately not done
order of
warnings, which occurrence eachstr.replace(..., 1)lands on, andwhether pass 3's
run not in resultguard sees pass 2's edits. Computation isshared; control flow is not.
matched with
validate=True; pass 3 reaches_decodes_to_printable_text, whichstrips
=, re-pads and decodes withvalidate=False. Those inputs coincide onlywhen the chunk already carries minimal padding — measured 867 same, 249
differing — so they are not one value. Pinned by
test_chunk_and_repadded_run_are_not_interchangeable_decode_inputs._decode_b64_safeis left unoptimised on purpose. It is the independentoracle for
_decode_b64_chunk; applying the same gates would make both sides ofthat comparison share the change and it would stop detecting anything.
_shannon_entropy(4.8 us, the costliest single window gate) and_vowel_ratioare untouched. Entropy is worth revisiting, but aprecomputed-term rewrite must be proven bit-identical or it can flip a verdict at
the
>= 4.3boundary, and that belongs in its own change.Tests
test/test_credential_prefilter.py, the differential test #7056 added, extendedfrom 157 to 213 passing. It pins the pre-optimisation implementation as a
reference oracle and asserts byte-identical
(text, warnings).Verified on Python 3.10 AND 3.12 — 213 passed on each. The removed guard is
exactly why: it passes on 3.12 and fails on 3.10, so a single-version run cannot
see it.
Corpus extended toward this change's blast radius: 40 hex digests in one string
(both cases), runs at every length residue 40–44, a long non-decodable run beside a
decodable one, padding variants per residue, blobs decoding to printable text / to
high-entropy garbage / to the empty string once invalid UTF-8 is dropped, encoded
credentials on both sides of the length gate, and an encoded credential in the
misaligned padding shapes whose decodability differs by interpreter.
New independent oracles.
_decodes_to_printable_textis reached through_contains_bare_secret, which the reference oracle also calls — so thebyte-identity assertion shares that helper and structurally cannot detect a change
inside it. Two tests close that gap against a verbatim copy of the pre-rewrite
body: all 256 single-byte values exhaustively (a complete proof for the membership
question, since
translatedecides each byte independently) plus 4000 fuzzedmulti-byte inputs, and 3000 fuzzed tokens across every residue and padding form.
Negative controls. Each sabotage was applied to the real source, the guarding
test observed, and the file restored and verified by sha256:
ghp_)len(chunk) % 4guard put backThe third row is the point: the same defect is caught on 3.10 and invisible on
3.12. The blob-gate sabotage fails
test_output_is_byte_identical_to_reference,i.e. an encoded credential slips through and the differential test notices.
Suite results.
test_credential_prefilter.py213 passed on 3.10 and on 3.12;with
test_security.py, 1208 passed / 1 skipped / 0 failed on 3.12. All 21redaction-related test files ran clean against a baseline worktree at the same
commit, the delta being the new tests.
scripts/check_black_formatting.pypasses,and
isort --check-only src/kiro_crew test conftest.py xdist_budget.pyis clean —note that the isort failure was pre-existing at #7056's head
(
_REDACTED_CREDENTIAL_TAGmisplaced in the import block), so #7056 will hit thesame gate on its own; this PR fixes it in passing.
Benchmark, baseline and optimised interleaved back to back in two passes so
neither is a stale capture, same session files and RNG seed on both sides: