Repository navigation
Conversation
Sensitive Change ReviewSensitive files changed:
How to clear this check:
Only the author's comment counts this way. A reviewer can instead |
949c6da to
986a8b2
Compare
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Any future edit to search semantics now needs a matching hand-written Rust change. Until that lands, the native path is silently off, and it only ever matches one CPython Unicode version. Watch
[DESIGN-REVIEWED] 5936963 Description read[DESCRIPTION-READ] 27723dfcbb635baeaca96982e12b4ad6d5f269df1fc566b9f2cd965c0e830040 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The PR bundles #19628's native credential gate inside Not justified as shipped
What this change shipsInventory (10 items) — 7 justifiedIntent: let session search answer cold queries faster without stalling other gateway requests. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 5936963 Description read[DESCRIPTION-READ] 937773d412d65562e3fc8a61f389ba209514d0fe85b0a26d118171e84705f1a8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Security Scope Review -- ✅ PASSWhich legitimate operations does Verdict basis: adjudicated, zero confirmed regressions. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: PASS TIGHTENING: No shell deny tier gets stricter. The patch leaves ADJUDICATED: 15 candidates, 0 confirmed newly-refused. The harness returned no confirmed rows. The candidates test the operations this patch adds against the unchanged deny composite: PLATFORM COVERAGE: The patch assumes three platform spellings: the
Two Windows spellings have no row, because the patch never spells them: the maturin build + wheel-install pipeline in its CI form, and the bench script, which is posix-only by design ( [SCOPE-REVIEWED] 5936963 |
986a8b2 to
ccc31ab
Compare
fc05407 to
e29086c
Compare
… to Python _might_contain_credential uses kc_native when it imports, its GATE_FINGERPRINT matches the Python pre-filter (literals, anchors, Unicode version), and KIROCREW_NATIVE_REDACTION is not 0. Otherwise, or when the module raises UnicodeEncodeError, the unchanged Python pre-filter (_might_contain_credential_py) runs. test_native_redaction_gate diffs the native and Python gates code point by code point, on random strings, and on redact_credentials output (skipped when the module is not built). test_native_redaction_fallback covers the missing, stale, disabled and failing-module paths without it. A new CI job, Native Redaction Gate, builds the module and runs both suites plus the Python pre-filter suite with the native gate live.
e29086c to
1841dd9
Compare
Move cold needle counting and snippet scans into the optional kc_native crate. Python keeps query parsing, scoring and the fallback implementation. Read the string's own storage without a retained UTF-8 copy; release the GIL on large inputs. Retain the first snippet while decoding the entire JSON string array and validating the zlib stream, including its checksum. Corrupt or truncated blobs return None so the Python path rereads the intact transcript. Later strings are validated without folding or searching them. Accept only builds whose search source fingerprint and Unicode version match the Python definitions. Module absence, import failures, scan errors and explicit opt-out use Python. The separate search opt-out lets operators isolate a search fault without disabling the shared credential gate. Use the existing crate, wheel and CI job. Assert fresh-build acceptance, compare 450 end-to-end search queries and both primitives against Python, and pin corruption rejection with regressions that fail before the fix.
1841dd9 to
5936963
Compare
|
Closing in favour of the pure-Python search fix in #20127, which measured faster under load and needs no native build. Branch kept; can be reopened if native wheels ship. |
Problem / Motivation
Goal: Session search answers cold queries faster and stops starving other gateway threads while it scans.
A cold session search spends most of its time in two scans that hold the GIL:
str.countper needle over every candidate session's folded text, and the snippet pass, which inflates and JSON-decodes a whole storedraw_textsblob per returned row. While a search runs, every other request in the gateway waits for the GIL; with 4 searches at once a thread that only sleeps 1 ms wakes up 3.6 ms late at p99 (measured below).Why it matters
Search is typed into on every keystroke and served from the same process as chat, the session list and health checks, so its GIL time is latency every other endpoint pays.
Not a goal
kc_nativecrate and its wheel and CI job.What changed (motivation → approach → change)
Two scans hold the GIL → do exactly those scans in Rust with the GIL released, keep everything else in Python, fall back to Python whenever the native code is missing, not proven to match, or cannot vouch for its answer.
kc_native.count_needles(text, specs):count_needleover all of a query's needles in one call. Reads the str's own storage (latin-1 / UCS-2 / UCS-4), so it counts code points exactly asstr.countdoes and never makes CPython keep a UTF-8 copy inside the str. Stops at the first required needle missing from both content and title.kc_native.snippet_from_blob(blob, needles, lead, trail, cap): the snippet loop over the stored blob. It keeps the first text with a hit, but returns it only after the whole JSON string array has decoded and the whole zlib stream has inflated with a valid checksum. Texts after the first hit are validated without being folded or searched. A corrupt or truncated blob returnsNone, and the Python path re-reads the intact transcript, so a damaged index never yields a partially validated preview. Blobs whose inflated size fits a bounded prefix are parsed from a slice; larger ones replay that prefix and continue the same stream one string at a time.py.detach) for large inputs.str.isdigit/str.isspace/str.casefoldcome from tables generated from the interpreter (native/kc_native/gen_search_tables.py→src/search_tables.rs)._content_counts(native call orcount_needle);_content_snippettries the native snippet when the index row is fresh and the snippet memo misses; the snippet loop moved unchanged intosnippet_from_texts, and blob decoding intohistory_index.decode_raw_texts/raw_blob.docs/system-specs/modules/history.mddescribes the native path, its validation rule and its fallbacks.bench_search.py(latency and 1 ms sleeper),bench_integrity.py(cold logical-cache miss vs immediate repeat),bench_memory.py(Linux RSS probe, one isolated child per arm; the child reads the library path as JSON on stdin), andexamples/profile_decode.rs(decode-cost profile).Fallback and fingerprint. The module is used only when
kc_native.SEARCH_FINGERPRINTequalshistory_search._native_search_fingerprint(): a hash of the code tokens ofcount_needle,snippet_from_textsanddecode_raw_texts(comments and docstrings excluded), the fold separator andunicodedata.unidata_version. A missing module, an import failure, a build without the search functions, a stale fingerprint, unavailable source, an exception from a native call, orKIROCREW_NATIVE_SEARCH=0all run the Python code. The search opt-out is separate fromKIROCREW_NATIVE_REDACTION, so an operator can isolate a search fault without turning off the shared credential gate. The constant is a Python module attribute, not a linker-visible static, so it cannot be stripped. CI asserts the module built from this tree is accepted by the loader, andtest_rust_constant_matches_the_python_code(every job, no Rust needed) fails when the mirrored Python changes without the Rust constant.One crate, one pipeline.
native/kc_native/src/search.rsis a module of the existing crate. The existing CI job (renamed Native Extension (kc_native)) gains a step that checks the loader accepts the fresh build and runs the search parity and fallback suites.Depends on #19628. The first commit here carries #19628's change as one commit (the crate, its CI job and the native credential gate); its files match #19628's current head
7162f58dd6excepttest/test_security_gate_liveness.py, where this branch is rebased on newer main: its_PACKAGE_LINE_BUDGETis 29,246 (main's 29,181 plus #19628's 65 lines) where #19628 has 29,240 on its older base. The second commit is this change. Please merge #19628 first; a rebase then leaves only the second commit.Measured (Python path vs native, same tree)
Both arms run the same source tree and the same freshly built library; only the search path differs. Synthetic indexed corpus from
bench_search.py(300 sessions, 120 queries, seed 7 corpus / seed 11 queries); variants alternate round by round; median of 6 rounds. The search code measured is identical to this head (the last commit only changedbench_memory.pyand its test).bench_integrity.py)bench_integrity.py)bench_search.py)"Cold" means a new
ConversationLogper query (empty metadata, fold and snippet caches, new SQLite connection); OS page caches are never dropped, so it is not a cold disk read. Both cold and repeat queries include the full blob validation above. Not measured: end-to-end gateway latency under load, and memory.Backwards compatibility
Compatible: without the module, or with a module that does not match, search runs the same Python code as before; with it, results are identical (parity suite).
SessionSearchIndex.raw_textskeeps its signature and answers.Removes nothing: the change adds an optional fast path and splits existing helpers without removing any caller-visible function, key or record.
Tests
test/test_native_search_parity.py(needs the module; runs unskipped in the Native Extension job): 400 queries x 42 folded texts through both counting paths, 400 queries x 31 stored blobs through both snippet paths, andsearch_sessionsfor 450 queries native vs Python, full rows including snippets: 0 differences. The corpus covers CJK, ß/İ/Σ/Dž/ffi, Deseret, Arabic, emoji, NUL, non-ASCII whitespace, full-width/superscript/Arabic-Indic digits touching forge references, a session large enough for the GIL-released branch, and sessions not yet indexed. A spy asserts both primitives actually served the run.test_a_fresh_build_is_acceptedpins that the module built from this tree passes the loader.ccc31abeab, which returned a snippet before validating the rest of the blob: 24 of them failed there.test/test_native_search_fallback.py(no module needed): default-on, missing, stale, pre-search build, env off, import errors, native call errors keep Python results, Rust constant vs Python hash, committed tables vs generator, fingerprint ignores comments, a native snippetNonefalls back, native call sites rank like Python.test/test_native_bench_memory.py: the parent keeps the library path out of the child's argv, the child loads the exact path it was given, and missing or conflicting modes fail before reading anything. 45 cases; 20 fail on the earlier headfc05407f02.cargo test --release --locked(Native Extension job, this head);cargo clippy --release --locked --all-targets -- -D warningsclean.e63d68d671(run serially 3 times: 7 passed, 5 failed each time).Manual verification
N/A — no UI or external service; behaviour is covered by the parity and corruption suites and the benchmarks above.
Related Issues
Depends on #19628.
Dependency licenses
New Rust crates:
flate2(miniz_oxide backend),serde,serde_json, plus their build-time dependencies innative/kc_native/Cargo.lock; all MIT and/or Apache-2.0. Two build-time crates report compound expressions whose every part is permissive and are listed with evidence in.github/dependency-review-config.yml:target-lexicon(Apache-2.0 WITH LLVM-exception) andunicode-ident((MIT OR Apache-2.0) AND Unicode-3.0). No Python or npm dependency is added.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Risks
Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.