Skip to content

perf(search): native counting and snippet scans, Python fallback - #20106

Closed
buluoray wants to merge 2 commits into
mainfrom
perf/kc-search-native
Closed

buluoray wants to merge 2 commits into
mainfrom
perf/kc-search-native

Conversation

@buluoray

@buluoray buluoray commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Session search answers cold queries faster and stops starving other gateway threads while it scans.

A cold session search spends most of its time in two scans that hold the GIL: str.count per needle over every candidate session's folded text, and the snippet pass, which inflates and JSON-decodes a whole stored raw_texts blob per returned row. While a search runs, every other request in the gateway waits for the GIL; with 4 searches at once a thread that only sleeps 1 ms wakes up 3.6 ms late at p99 (measured below).

Why it matters

Search is typed into on every keystroke and served from the same process as chat, the session list and health checks, so its GIL time is latency every other endpoint pays.

Not a goal

  • No change to query parsing, scoring, ranking or the results. Python keeps all of it; the native code only counts and builds snippets, and must return what Python returns.
  • No new packaging: the code joins the existing optional kc_native crate and its wheel and CI job.
  • Not the warm-query memos (a separate pure-Python change); this covers the cold scans those memos do not.

What changed (motivation → approach → change)

Two scans hold the GIL → do exactly those scans in Rust with the GIL released, keep everything else in Python, fall back to Python whenever the native code is missing, not proven to match, or cannot vouch for its answer.

  • kc_native.count_needles(text, specs): count_needle over all of a query's needles in one call. Reads the str's own storage (latin-1 / UCS-2 / UCS-4), so it counts code points exactly as str.count does and never makes CPython keep a UTF-8 copy inside the str. Stops at the first required needle missing from both content and title.
  • kc_native.snippet_from_blob(blob, needles, lead, trail, cap): the snippet loop over the stored blob. It keeps the first text with a hit, but returns it only after the whole JSON string array has decoded and the whole zlib stream has inflated with a valid checksum. Texts after the first hit are validated without being folded or searched. A corrupt or truncated blob returns None, and the Python path re-reads the intact transcript, so a damaged index never yields a partially validated preview. Blobs whose inflated size fits a bounded prefix are parsed from a slice; larger ones replay that prefix and continue the same stream one string at a time.
  • Both release the GIL (py.detach) for large inputs.
  • str.isdigit / str.isspace / str.casefold come from tables generated from the interpreter (native/kc_native/gen_search_tables.py → src/search_tables.rs).
  • Python side: the scoring loop takes per-needle counts from _content_counts (native call or count_needle); _content_snippet tries the native snippet when the index row is fresh and the snippet memo misses; the snippet loop moved unchanged into snippet_from_texts, and blob decoding into history_index.decode_raw_texts / raw_blob.
  • docs/system-specs/modules/history.md describes the native path, its validation rule and its fallbacks.
  • Benchmarks used for the numbers below live next to the crate: bench_search.py (latency and 1 ms sleeper), bench_integrity.py (cold logical-cache miss vs immediate repeat), bench_memory.py (Linux RSS probe, one isolated child per arm; the child reads the library path as JSON on stdin), and examples/profile_decode.rs (decode-cost profile).

Fallback and fingerprint. The module is used only when kc_native.SEARCH_FINGERPRINT equals history_search._native_search_fingerprint(): a hash of the code tokens of count_needle, snippet_from_texts and decode_raw_texts (comments and docstrings excluded), the fold separator and unicodedata.unidata_version. A missing module, an import failure, a build without the search functions, a stale fingerprint, unavailable source, an exception from a native call, or KIROCREW_NATIVE_SEARCH=0 all run the Python code. The search opt-out is separate from KIROCREW_NATIVE_REDACTION, so an operator can isolate a search fault without turning off the shared credential gate. The constant is a Python module attribute, not a linker-visible static, so it cannot be stripped. CI asserts the module built from this tree is accepted by the loader, and test_rust_constant_matches_the_python_code (every job, no Rust needed) fails when the mirrored Python changes without the Rust constant.

One crate, one pipeline. native/kc_native/src/search.rs is a module of the existing crate. The existing CI job (renamed Native Extension (kc_native)) gains a step that checks the loader accepts the fresh build and runs the search parity and fallback suites.

Depends on #19628. The first commit here carries #19628's change as one commit (the crate, its CI job and the native credential gate); its files match #19628's current head 7162f58dd6 except test/test_security_gate_liveness.py, where this branch is rebased on newer main: its _PACKAGE_LINE_BUDGET is 29,246 (main's 29,181 plus #19628's 65 lines) where #19628 has 29,240 on its older base. The second commit is this change. Please merge #19628 first; a rebase then leaves only the second commit.

Measured (Python path vs native, same tree)

Both arms run the same source tree and the same freshly built library; only the search path differs. Synthetic indexed corpus from bench_search.py (300 sessions, 120 queries, seed 7 corpus / seed 11 queries); variants alternate round by round; median of 6 rounds. The search code measured is identical to this head (the last commit only changed bench_memory.py and its test).

Python native
Cold logical cache, search p50 (bench_integrity.py) 191.95 ms 148.44 ms
Immediate repeat, search p50 (bench_integrity.py) 144.70 ms 103.91 ms
4 threads, search p50 (bench_search.py) 461.05 ms 239.98 ms
1 ms sleeper lateness p99, 4 threads searching 3.59 ms 1.18 ms

"Cold" means a new ConversationLog per query (empty metadata, fold and snippet caches, new SQLite connection); OS page caches are never dropped, so it is not a cold disk read. Both cold and repeat queries include the full blob validation above. Not measured: end-to-end gateway latency under load, and memory.

Backwards compatibility

Compatible: without the module, or with a module that does not match, search runs the same Python code as before; with it, results are identical (parity suite). SessionSearchIndex.raw_texts keeps its signature and answers.

Removes nothing: the change adds an optional fast path and splits existing helpers without removing any caller-visible function, key or record.

Tests

  • test/test_native_search_parity.py (needs the module; runs unskipped in the Native Extension job): 400 queries x 42 folded texts through both counting paths, 400 queries x 31 stored blobs through both snippet paths, and search_sessions for 450 queries native vs Python, full rows including snippets: 0 differences. The corpus covers CJK, ß/İ/Σ/Dž/ffi, Deseret, Arabic, emoji, NUL, non-ASCII whitespace, full-width/superscript/Arabic-Indic digits touching forge references, a session large enough for the GIL-released branch, and sessions not yet indexed. A spy asserts both primitives actually served the run. test_a_fresh_build_is_accepted pins that the module built from this tree passes the loader.
  • Corruption regressions in the same file: a damaged blob never returns a partial snippet, every truncation is rejected, an unsupported tail defers to Python, a valid blob keeps the first hit, a damaged index makes search re-read the intact transcript, and the bounded-prefix decode boundary: 58 cases. The 34 written with the integrity fix (all but the 24 bounded-prefix boundary cases, added with the later decode speed-up) were run against the earlier head ccc31abeab, which returned a snippet before validating the rest of the blob: 24 of them failed there.
  • The parity suite catches real bugs: with the Rust digit table cut to ASCII, casefold cut to ASCII, or U+3000 dropped from whitespace, it fails.
  • test/test_native_search_fallback.py (no module needed): default-on, missing, stale, pre-search build, env off, import errors, native call errors keep Python results, Rust constant vs Python hash, committed tables vs generator, fingerprint ignores comments, a native snippet None falls back, native call sites rank like Python.
  • test/test_native_bench_memory.py: the parent keeps the library path out of the child's argv, the child loads the exact path it was given, and missing or conflicting modes fail before reading anything. 45 cases; 20 fail on the earlier head fc05407f02.
  • cargo test --release --locked (Native Extension job, this head); cargo clippy --release --locked --all-targets -- -D warnings clean.
  • On this head in CI: Native Extension (kc_native), all Backend Tests shards (Linux, Windows, macOS), Backend Lint & Type Check and SAST (Semgrep) pass.
  • Frontend Tests (2) fails on 5 ChatEmbed tests. This PR changes no frontend file, and the same 5 tests fail the same way on unmodified main e63d68d671 (run serially 3 times: 7 passed, 5 failed each time).

Manual verification

N/A — no UI or external service; behaviour is covered by the parity and corruption suites and the benchmarks above.

Related Issues

Depends on #19628.

Dependency licenses

New Rust crates: flate2 (miniz_oxide backend), serde, serde_json, plus their build-time dependencies in native/kc_native/Cargo.lock; all MIT and/or Apache-2.0. Two build-time crates report compound expressions whose every part is permissive and are listed with evidence in .github/dependency-review-config.yml: target-lexicon (Apache-2.0 WITH LLVM-exception) and unicode-ident ((MIT OR Apache-2.0) AND Unicode-3.0). No Python or npm dependency is added.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Risks

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 14, 2026 04:26
@buluoray
buluoray requested a review from mbriones98 October 14, 2026 04:26
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/redaction.py

  • No reasoning template for 5936963 yet.

How to clear this check:

  1. The PR author posts the template below as a PR comment, with
    every field filled (N/A needs a reason). Keep the
    Head commit line: it must name the current head SHA.
  2. The check turns green within a minute, with no push.
  3. A new push resets this check: post it again for the new head.

Only the author's comment counts this way. A reviewer can instead
paste the filled template into an "Approve" review on this head.
Merging still needs a reviewer's approval.

## Sensitive change review
- Head commit: 5936963bdd4eb8a6652f8e09378c956efd6c884c
- What rule changed:
- Before -> after (what was allowed/blocked, now):
- Worst case if wrong (breaks commands / leaks secret / locks user out):
- Evidence checked (which sandbox result proves it):
- No regression:
  - Tools still work (which tools/commands were run, result):
  - Backward compatible (old config / old data / old callers still work, how checked):
  - Existing tests (which suites passed on this head):
- How to undo:

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 14, 2026
@buluoray
buluoray force-pushed the perf/kc-search-native branch from 949c6da to 986a8b2 Compare October 14, 2026 04:31
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 5936963bdd4eb8a6652f8e09378c956efd6c884c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Any future edit to search semantics now needs a matching hand-written Rust change. Until that lands, the native path is silently off, and it only ever matches one CPython Unicode version.

Watch

  • Ongoing coupling cost. count_needle, snippet_from_texts and decode_raw_texts are written twice, and the two copies are tied together by a fingerprint you update by hand: pub const SEARCH_FINGERPRINT: &str = "66f0dcc621a31526";. Every later change to how search matches or builds snippets must also rewrite search.rs, regenerate the tables and update the constant. Otherwise test_rust_constant_matches_the_python_code goes red in every job, so a Python-only contributor can no longer change search on their own. That is a lasting tax for a 23–48% latency gain.
    Clears when: history.md (or the crate README) names who owns keeping the Rust side in line, and states that a Python-only search change may disable the native path instead of being blocked by the pin.
  • Fast path works on one Python version. The fingerprint hashes unicodedata.unidata_version, and the only CI build uses python-version: "3.12". Any other supported interpreter that installs the module rejects it at the debug log level and falls back to Python. The measured gains then apply only on 3.12, and nothing reports that they are missing.
    Clears when: the description or spec states which interpreters the wheel targets, or the loader logs a fingerprint mismatch above debug level once per process.
  • Two changes in one PR. The native credential gate from perf(redaction): native single-pass credential gate, Python fallback #19628 rides along, and a wrong False from it lets a credential skip redaction. The dependency is disclosed, but a security-relevant change shouldn't merge as a side effect of a perf PR.
    Clears when: #19628 merges first and this PR is rebased down to the search commit alone.

[DESIGN-REVIEWED] 5936963

Description read

[DESCRIPTION-READ] 27723dfcbb635baeaca96982e12b4ad6d5f269df1fc566b9f2cd965c0e830040

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5936963bdd4eb8a6652f8e09378c956efd6c884c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5936963

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5936963bdd4eb8a6652f8e09378c956efd6c884c: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5936963bdd4eb8a6652f8e09378c956efd6c884c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 5936963bdd4eb8a6652f8e09378c956efd6c884c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The PR bundles #19628's native credential gate inside security/, which the accepted Rust RFC lists under "Do not touch". No install receives kc_native.

Not justified as shipped

  • Item 3 — rides along: the accepted RFC §4.5 says "Do not touch: src/kiro_crew/security/", yet redaction.py now hands _might_contain_credential to Rust. The description gives no reason, and the search Goal doesn't need it.
    Subtraction: drop the redaction.py, lib.rs gate and test_native_redaction_* hunks here; they land through perf(redaction): native single-pass credential gate, Python fallback #19628.
    Clears when: those hunks are removed from this PR, or a maintainer records an override for security/.
  • Item 2 — inherited: the RFC says later PyO3 modules ship "in the same kirocrew-native", wheels only. kc_native is publish = false, isn't in setup.cfg and has no release wheel, so item 1 reaches nobody.
    Clears when: a maintainer records that kc_native replaces the RFC's kirocrew-native.
  • Item 8 — oversized: the description says "Not measured: … memory", yet it ships bench_memory.py plus a 147-line test for it.
    Subtraction: delete bench_memory.py, test/test_native_bench_memory.py and examples/profile_decode.rs.
    Clears when: the deletion is made.

What this change ships

Inventory (10 items) — 7 justified

Intent: let session search answer cold queries faster without stalling other gateway requests. This is an ADDITION.

  1. Search counting and snippets run in Rust with the GIL released, when the module is installed — justified
  2. A new native package, kc_native, with its own CI job — inherited: a second package name beside the RFC's kirocrew-native
  3. The credential pre-filter runs in Rust when the module is installed — rides along: from perf(redaction): native single-pass credential gate, Python fallback #19628, inside RFC-forbidden security/
  4. A KIROCREW_NATIVE_SEARCH=0 opt-out — justified
  5. The native path turns off whenever the mirrored Python code changes (source fingerprint) — justified
  6. Unicode tables generated from the interpreter — justified
  7. The snippet loop and blob decode split into helpers — justified
  8. Benchmark scripts, a decode profiler and a bench test committed to the repo — oversized
  9. The CI job builds the module and runs the parity suites unskipped — justified
  10. License allowances for two build-time crates — justified

[FIRST-PRINCIPLES-REVIEWED] 5936963

Description read

[DESCRIPTION-READ] 937773d412d65562e3fc8a61f389ba209514d0fe85b0a26d118171e84705f1a8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5936963bdd4eb8a6652f8e09378c956efd6c884c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5936963

Verdict parsed from the review's SHA-scoped output markers for commit 5936963bdd4eb8a6652f8e09378c956efd6c884c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5936963bdd4eb8a6652f8e09378c956efd6c884c: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does 5936963bdd4eb8a6652f8e09378c956efd6c884c newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 10 classified, 0 newly refused, 10 still allowed.
  • ubuntu-latest (platform posix): 10 classified, 0 newly refused, 10 still allowed.
  • windows-latest (platform windows): 7 classified, 0 newly refused, 7 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

TIGHTENING: No shell deny tier gets stricter. The patch leaves src/kiro_crew/hooks.py, SHELL_DENY_TIERS, the sensitive-bash tier, the exfil shapes and the deny-rules catalog alone. Its one security-adjacent change is a faster route through the credential pre-filter in src/kiro_crew/security/redaction.py, with no change to what the filter catches. Redaction pre-filter -- _might_contain_credential now goes through an optional native scan, used only when its fingerprint matches, and falls back to _might_contain_credential_py otherwise: "if getattr(kc_native, "GATE_FINGERPRINT", None) != _native_gate_fingerprint(): ... return None" and "if _NATIVE_GATE is not None: try: return _NATIVE_GATE(text) except UnicodeEncodeError: pass; return _might_contain_credential_py(text)". This filter only decides whether the full _CREDENTIAL_PATTERNS pass runs. It never refuses an operation. A native answer that matched more than the Python one would cost a regex pass, not a tool call. The patch names no new threat to refuse ("No pattern or pass changes", from the _PACKAGE_LINE_BUDGET comment in test/test_security_gate_liveness.py).

ADJUDICATED: 15 candidates, 0 confirmed newly-refused. The harness returned no confirmed rows. The candidates test the operations this patch adds against the unchanged deny composite: cargo test --release --locked, maturin develop / maturin build plus installing the wheel, pip install maturin==1.15.0, printing the search fingerprint, regenerating gen_search_tables.py, running with KIROCREW_NATIVE_REDACTION=0 / KIROCREW_NATIVE_SEARCH=0, the bench script, and a grep for the Authorization anchor.

PLATFORM COVERAGE: The patch assumes three platform spellings: the native/kc_native path, the cargo / maturin / python executable names, and the KIROCREW_NATIVE_* environment switches.

  • posix: forward-slash paths, .venv/bin/python, and the VAR=0 cmd environment prefix, for the cargo, maturin, table-generation, env-switch and bench rows.
  • windows: backslash paths, cargo.exe / maturin.exe / .venv\Scripts\python.exe, cmd set VAR=0 &&, and PowerShell $env:VAR='0';, for the cargo, maturin develop, table-generation and both env-switch rows.
  • any: the pinned pip install maturin==1.15.0 and the python -c fingerprint print. Neither string carries a separator or an executable suffix.

Two Windows spellings have no row, because the patch never spells them: the maturin build + wheel-install pipeline in its CI form, and the bench script, which is posix-only by design (/proc). bench_memory is described as Linux-only, and nothing on Windows reaches it.

[SCOPE-REVIEWED] 5936963

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@buluoray
buluoray force-pushed the perf/kc-search-native branch from 986a8b2 to ccc31ab Compare October 14, 2026 07:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 14, 2026
@buluoray
buluoray force-pushed the perf/kc-search-native branch 2 times, most recently from fc05407 to e29086c Compare October 14, 2026 09:52
… to Python

_might_contain_credential uses kc_native when it imports, its
GATE_FINGERPRINT matches the Python pre-filter (literals, anchors, Unicode
version), and KIROCREW_NATIVE_REDACTION is not 0. Otherwise, or when the
module raises UnicodeEncodeError, the unchanged Python pre-filter
(_might_contain_credential_py) runs.

test_native_redaction_gate diffs the native and Python gates code point by
code point, on random strings, and on redact_credentials output (skipped
when the module is not built). test_native_redaction_fallback covers the
missing, stale, disabled and failing-module paths without it. A new CI job,
Native Redaction Gate, builds the module and runs both suites plus the
Python pre-filter suite with the native gate live.
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 14, 2026
@buluoray
buluoray force-pushed the perf/kc-search-native branch from e29086c to 1841dd9 Compare October 14, 2026 16:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 14, 2026
Move cold needle counting and snippet scans into the optional kc_native
crate. Python keeps query parsing, scoring and the fallback implementation.
Read the string's own storage without a retained UTF-8 copy; release the
GIL on large inputs.

Retain the first snippet while decoding the entire JSON string array and
validating the zlib stream, including its checksum. Corrupt or truncated
blobs return None so the Python path rereads the intact transcript.
Later strings are validated without folding or searching them.

Accept only builds whose search source fingerprint and Unicode version
match the Python definitions. Module absence, import failures, scan errors
and explicit opt-out use Python. The separate search opt-out lets operators
isolate a search fault without disabling the shared credential gate.

Use the existing crate, wheel and CI job. Assert fresh-build acceptance,
compare 450 end-to-end search queries and both primitives against Python,
and pin corruption rejection with regressions that fail before the fix.
@buluoray
buluoray force-pushed the perf/kc-search-native branch from 1841dd9 to 5936963 Compare October 14, 2026 18:08
@buluoray
buluoray marked this pull request as draft October 14, 2026 18:24
@buluoray

Copy link
Copy Markdown
Contributor Author

Closing in favour of the pure-Python search fix in #20127, which measured faster under load and needs no native build. Branch kept; can be reopened if native wheels ship.

@buluoray buluoray closed this Oct 14, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 14, 2026
@bolichen97
bolichen97 deleted the perf/kc-search-native branch November 6, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant