Repository navigation
perf(security): gate the credential scan behind a cheap superset pre-filter - #7056
Conversation
0235df7 to
550c6e9
Compare
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Both hunks and the surrounding base code are verified: the pre-filter anchors are genuine necessary conditions of the branches at Design-Verdict: PASS A profiled hot-path fixed at the shared primitive, byte-identical by differential oracle, with the pre-filter's silent-bypass hazard guarded from every direction. Suggestions
[DESIGN-REVIEWED] eb347a3 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of I've verified everything I need from the base tree. The claims in the diff check out against the base: the three-pass structure exists as described, First-Principles-Verdict: PASS A measured 60%-of-event-loop cost removed by pure control flow, byte-identity pinned by a differential oracle — every item traces to a profile line or a demonstrated bypass. What this change shipsIntent: make persisted-chat redaction stop costing event-loop latency on credential-free text — a FIX.
Watch
[FIRST-PRINCIPLES-REVIEWED] eb347a3 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
550c6e9 to
0517c9a
Compare
0517c9a to
5cbc819
Compare
5cbc819 to
4bf73bc
Compare
4bf73bc to
6e3eb77
Compare
…filter Skips a 23-branch alternation scan that cannot match, shares one base64 run scan between passes 2 and 3, and drops a redundant chunk re-scan. 2.6-2.9x.
6e3eb77 to
eb347a3
Compare
Problem / Motivation
redact_credentialsis the single most expensive thing the gateway's event loopdoes. A
py-spycapture of a live gateway (2,945 samples,--rate 10 --nonblocking, window containing a 168-second API request) attributes:redact_credentials_flush_dirty_slots→flush_slot_now_save_slot_to_history_build_message_entryIt is reached per message, per flush, via
_flush_dirty_slots→_save_slot_to_history→_build_message_entry→_redact_meta_for_role→_redact_meta→_redact_value. On a host injecting 240K-char contexts thatrescans megabytes on every dirty-slot flush.
The function ran three separate full-text regex scans over every string, and
the first of them was the hottest line in the whole profile:
_CREDENTIAL_PATTERNS.finditer(result)_B64_CHUNK_RE.finditer(text)_BARE_SECRET_RUN_RE.finditer(text)_CREDENTIAL_PATTERNSis a 23-branch alternation, soreretries every branchat essentially every position — measured 117 ns/char. That cost is paid in
full even though real text almost never contains a credential: across 1,804 live
session-history messages (1.47 MB) the pattern matched zero times.
Why it matters
This is event-loop time on the request path, so it is latency users feel, not
background cost. It also scales the wrong way — with history length times flush
frequency, both of which grow during a session.
#7031 added a 3s memo to
/api/skillsso that one endpoint stops paying thiscost. That removes a caller; it does not make the redaction cheaper. This does.
What changed
Three changes, all pure control flow. Output is byte-identical — this is the
redaction boundary, so a regression here writes live credentials into persisted
chat history.
1. Gate pass 1 on a cheap superset pre-filter.
_might_contain_credentialanswers "could any branch match at all?" and pass 1 is skipped when the answer is
no. When it returns False,
finditerwould have yielded nothing and the loop bodywould not have run, so skipping cannot change the output.
It is built from each branch's necessary condition, never a restatement of the
branch — 21 literal anchors, plus five leading-shape regexes for the branches with
no usable literal (GitHub PAT, Telegram, Discord, URI userinfo, and the
case-insensitive
Authorization: Bearer). The superset direction is the securityproperty: a false positive costs a scan we would have run anyway, a false negative
would leak.
Two anchors were tightened after measuring them against real content, because the
obvious form was far too broad:
sk-matched "task-" → replaced withsk-proj-/sk-ant-[0-9]{6,}:matched an epoch timestamp followed by a colon (29 of 614 realmessages) → tightened to
[0-9]{6,}:[A-Za-z0-9_-]{30}The pre-filter fires on 5 of 1,804 real messages (0.3%) and costs 32
ns/char against pass 1's 117.
2. Share one base64-run scan between passes 2 and 3. They scanned the same
text twice for the same spans.
[A-Za-z0-9+/]{40,}is greedy and leftmost so ityields exactly the maximal runs of length ≥ 40, which is also precisely what
_BARE_SECRET_RUN_RE's(?<![A-Za-z0-9+/])/(?![A-Za-z0-9+/])boundariesselect. The only difference is the trailing
={0,2}that_B64_CHUNK_REadditionally consumes, and
=is not in the run's character class, sorstrip("=")recovers the bare run exactly.The two loops stay separate and in their original order. Fusing them into one
per-run loop would interleave the passes, which changes the order of
warningsand — because each pass mutates
resultviastr.replace(…, 1)— whichoccurrence each replacement lands on, and whether pass 3's
run not in resultguard sees pass 2's edits. Sharing the scan while keeping the loops ordered is
what makes this byte-identical.
3. Stop re-scanning a chunk that is already a chunk. Pass 2 called
_decode_b64_safe(chunk), which re-ran the chunk regex over a string that isitself one
_B64_CHUNK_REmatch. New_decode_b64_chunkskips that. The generichelper is unchanged for its other callers.
Both security properties called out in the existing comments are preserved and
now asserted: warnings carry length only, never a slice of the matched secret,
and passes 2 and 3 still scan the original
textrather than the mutatedresultso match offsets stay stable.The
Authorizationanchor must use the regex engine, notstr.lower()An earlier revision of this PR anchored the case-insensitive branch with
"authorization" in text.lower(). That was a redaction bypass, and it isfixed here.
str.lower()andre.IGNORECASEare two different case-foldingimplementations, so they disagree.
refolds viasre_compile._equivalences,which treats U+0131 (LATIN SMALL LETTER DOTLESS I) and U+0130 (LATIN CAPITAL
LETTER I WITH DOT ABOVE) as equivalent to
i/I;str.lower()leaves U+0131unchanged and expands U+0130 to two code points. So:
was matched by the
(?i:Authorization)branch but missed by the.lower()anchor — the gate returned False, pass 1 was skipped, and the bearertoken was persisted verbatim with no warning. The same disagreement exists for
U+017F/
sand U+212A/k, so this is a class of defect rather than onehomoglyph.
The anchor is now
_CREDENTIAL_PREFILTER_AUTHORIZATION_RE = re.compile(r"(?i:Authorization)")— the branch's own sub-pattern, so the supersetproperty holds by construction: same engine, same folding rules. The rule this
establishes is that a case-insensitive branch may only be anchored by the same
regex engine, never by a hand-rolled fold.
A widened branch is now caught, not just an added one
The branch-count assertion catches an added branch. It structurally cannot
catch a widened one, because widening does not change the count — and the
bypass above is exactly a branch accepting more than its anchor.
test_widening_a_branch_cannot_outgrow_its_anchorcloses that: it compiles eachbranch standalone, mutates that branch's registered sample (case flips and the
Unicode homoglyphs that break hand-rolled folds), and for every mutation the
branch still accepts, asserts the gate accepts it too. It fails on the pre-fix
implementation, naming branch 22.
_CREDENTIAL_PATTERNSis also now marked at its definition site as having adependent pre-filter, spelling out that adding, widening, or case-relaxing a
branch is a two-site change and which test catches each case. That missing signal
is what let the drift ship.
Deliberately not changed
result = result.replace(matched, tag, 1)inside pass 1 stays as-is. Rebuildingfrom collected spans would not be byte-identical: branches carrying a left
look-behind (Discord, the 2-segment link token) can match text that also occurs
earlier in the string at a position the look-behind rejects, and replace-first
then lands on that earlier occurrence — span-replacement would not. The corpus
pins that case (
"x" + token + " " + token). At 40 matches in a 240K string thestr.replaceloop costs 1.4 ms, so the risk buys almost nothing._B64_CHUNK_REitself is also untouched — adding a capture group would havechanged
findallsemantics for the external consumer atdashboard/handlers/artifacts.py:477.Benchmarks
Best of 5, Python 3.12.14, measured on two worktrees at the same base commit
(baseline has no pre-filter at all):
The last row is the pre-filter's worst case — an anchor is present, so the
pre-filter fires and the full scan runs anyway — and it is still faster, because
the shared base64 scan more than covers the pre-filter's cost. No input shape
regresses.
Switching the
Authorizationanchor fromstr.lower()to the(?i:…)regexcosts about 1.4 ms on a 240K input (14.55 → 15.95 ms), taking the clean-text
speedup from 2.93× to 2.55×. That is the correct trade: the
.lower()form wasfaster and wrong.
Tests
test/test_credential_prefilter.py(new, 181 cases) pins the originalthree-pass implementation as a reference oracle and asserts the live function is
byte-identical to it on both the returned text and the warnings list. The
oracle reuses the module's own compiled patterns and gate helpers, so what it
isolates is exactly the control-flow change.
Corpus covers: every one of the 23 branches (alone and embedded in prose), the
Unicode case-folding shapes, base64-encoded credentials, bare 40-char AWS secret
keys, the glued-secret sliding-window cases the existing comment calls out
(
X+key, key+A,SECRET=+key+ABC, key+X+key), adjacent and repeatedmatches, the look-behind-shadowed occurrence described above,
=paddingvariants, hex digests and long base64 blobs, PEM complete/truncated/inline-in-prose,
empty and whitespace-only strings, and 108K-char inputs.
test_corpus_actually_exercises_every_passasserts all three warning kindsappear, so the differential corpus cannot be vacuously all-clean.
test_warnings_never_carry_secret_materialasserts every warning matchesRedacted … (N chars)and contains no secret substring.test_prefilter_still_skips_credential_free_textpins the negative direction, soa "fix" that made the gate fire on everything — correct but worthless — fails.
Supporting equivalences are asserted rather than assumed:
test_b64_chunk_spans_match_bare_secret_run_spansproves the shared scan islegitimate, and
test_decode_b64_chunk_matches_generic_helper_on_chunksprovesthe new decode helper agrees with the one it replaces.
Negative controls — a test that cannot fail proves nothing on a redaction
boundary. Run against the pre-fix tree, the new tests produce 22 failures,
including the differential oracle test itself:
test_unicode_case_folding_cannot_bypass_the_prefilter— all four bypassshapes fail pre-fix, each asserting the branch matches (positive control, so
the test cannot go vacuous) and then that the gate does not skip it.
test_prefilter_is_a_superset_under_every_re_ignorecase_equivalence—generalises past the two reported homoglyphs by enumerating every code
point in U+0100–U+24FF that
re.IGNORECASEfolds into a letter of"Authorization" but
str.lower()does not, and asserting the property over allof them. Independently rediscovers U+0130.
test_widening_a_branch_cannot_outgrow_its_anchor— fails pre-fix namingbranch 22.
test_output_is_byte_identical_to_reference— the bypass shapes are now in thecorpus, so the buggy gate makes the live function diverge from the oracle.
Their absence from the corpus is what let the bypass ship.
test_differential_test_detects_a_too_narrow_prefilterdisarms every anchor viamonkeypatchand asserts the comparison differs, proving the byte-identityassertion can fail rather than passing because both sides share a defect.
test_each_sample_is_specific_to_its_own_branchremoves branch i and assertssample i stops matching, so no sample is vacuously covered by another branch.
test_every_pattern_branch_has_a_prefilter_anchorasserts branch count ==registered sample count, so a 24th branch fails loudly at the point of the edit.
test_case_insensitive_branches_must_be_anchored_by_the_same_enginepins whichbranches are
(?i:…), so a new one cannot quietly be gated by a literal.Also verified by sabotaging the source (dropping the
eyJliteral and disarmingthe GitHub-PAT shape): the differential test fails on exactly the
credential-bearing cases, then reverted byte-identically.
test_prefilter_soundness_under_fuzzruns 4,000 seeded random strings built fromthe real branch samples plus mutations and asserts both differential equality and
the superset property, with a coverage floor (
checked >= 500) so a fuzz run thatproduced almost no real matches fails instead of silently asserting nothing — that
floor already caught a first draft which produced only 5.
Independently of pytest, a cross-tree differential ran a baseline worktree
(no pre-filter at all) and this tree over 9,804 inputs (real session content
plus seeded fuzz), 5,031 of which produce warnings, and both produced an
identical aggregate output digest:
Gate results: every step of
Backend Lint & Type Checkrun locally — black(baselined), subprocess-encoding gate, lockdown-before-publish, isort, flake8,
mypy — all pass. Security and redaction suites (
test_credential_prefilter,test_security,test_log_redaction,test_redaction_mirror_parity,test_snapshot_redaction,test_streaming_meta_redaction,test_redact_meta_snapshot,test_handlers_artifacts_coverage) → 1,378 passed,1 skipped.
The full suite shows 96 failures; all 96 reproduce identically on an unmodified
baseline worktree at the same commit — same 31 files, same per-file counts — and
none is a security or redaction test. They are environmental (uid/ownership,
xdist host budget, install-path checks).
security.pyfails rawblack, identically on baseline (448-line diff bothsides) and
blackwants nothing in the added lines; the repo's own gate passes itvia the known-unformatted baseline. The new test file is
black-clean.