Repository navigation
Conversation
Phase 3 of the session search RFC, backend half. While the flag is on (default
off), a session search embeds the query with the in-process model under a new
query role, scores the chunk vectors of the keyword search's own window, and
routes the two rankings: a query of at least 12 characters that occurs whole in
the keyword top session keeps it first over both legs fused; any other query
gets the vector rows first, except that keyword rows lead while fewer than 90%
of visible sessions are embedded. Each row carries match ("text" or "meaning"),
and a meaning row's snippet is its chunk, re-read under the transcript lock
through the derivation guard. Incognito and temporary sessions are never
candidates, and every row passes the keyword rows' gates. The dashboard search,
search_chat_history and the channel picker carry the match kind and a coverage
note; message search keeps the keyword ranking. With the flag off, no store or
a non-local embedder, rows are unchanged.
Part of kirodotdev#24476
kirocrew bench session-recall --meaning chunks and embeds the recall set with this host's in-process model and reports, beside the keyword ranking, R@1 per query class for the vector leg and the routed rows, both orders of the routed rows with the newest 50, 75 and 90% of sessions embedded, and the leg's cost: chunk count, matrix bytes, the matrix load's peak allocation, fill CPU per chunk and median query time. A host without the model is refused rather than scored as keyword search. These are the phase 3 exit measurements of the session search RFC. Part of kirodotdev#24476
Opus 5.5 Review (fork) —
|
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Everything ships behind What this change shipsInventory (10 items) — 10 justifiedIntent: let a person find a past conversation by describing it, not only by its exact words. This is an ADDITION, behind a flag that stays off.
[FIRST-PRINCIPLES-REVIEWED] 3258ba7 Description read[DESCRIPTION-READ] ce061c249fb2308911e966232b666e8cb5f9ff1bf33b10e9a9c76d808ccbfe6f That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the flag is off by default and every effect can be undone. The coverage denominator below needs fixing before phase 4. Notes
[DESIGN-REVIEWED] 3258ba7 Description read[DESCRIPTION-READ] 76c233ebc41599c03812a743e7b1182a40ba991ecc5e84541db0897ece3c63a1 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — 🔴 changes requested (blocking)Reviewed Adjudication upheld 3 of 3 blocking finding(s); those must be resolved before merging. BLOCKING -- src/kiro_crew/history_vectors_search.py:161 -- Apply caller eligibility before the ten-hit cap
BLOCKING -- src/kiro_crew/history_vectors_search.py:397 -- Exclude unindexable sessions from routing coverage
BLOCKING -- src/kiro_crew/history_vectors_search.py:220 -- Release matrices owned by one-shot MCP searches
[BLOCK-MERGE] 3258ba7 Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1 ( F2 ( F3 ( |
|
Thanks for the clear write-up and the bench numbers. I read the leg, the routing, and the privacy path ( I checked the three GPT 6.1 blockers on
The flag is off by default, so none of this hits users today, but phase 4 turns it on, so these should land in this PR. Push a fix and I'll re-review the new head right away. The Opus lane said "review incomplete" (no verdict), which is a pipeline issue on our side, not something you need to fix. AI-assisted, sent by Bolin Review bot |
Problem / Motivation
Goal: Let session search find a conversation from a description of it, as well as from its words, behind
session.semantic_search(default off), with today's rows unchanged while the flag is off.Phase 3 of the session search RFC (accepted on #24476, record updated in #39386), backend half, in two commits. The first adds the query side: the query role on the embed call, the vector leg over the chunk store #36002 built, the routing rule decided on set D,
matchand chunk snippets on result rows, the coverage note, and the backend surfaces. The second extends the recall harness with the vector and routed rankers, partial coverage and the leg's cost, as the RFC's phase 3 exit criteria ask. The dashboard rows ("Related" label in the sidebar and the command palette) follow in a separate UI PR.Why it matters
On the shipped recall fixture, keyword search finds no paraphrase, Chinese paraphrase or identifier plus description query in its top 10 (R@10 0.000 for each). These are the queries people use when they remember what a conversation was about but not its words. With the leg on, all answerable queries reach R@1 0.880 against 0.340 today.
Not a goal
match.What changed (motivation → approach → change)
Commit 1 (
search sessions by meaning) holds the leg and every surface; commit 2 (score the meaning leg in the session recall bench) holds the harness.embeddings_llama.py,embeddings_settings.py,embeddings.pyembedandembed_batchtakerole, defaultEMBED_ROLE_DOCUMENT, which sends text as given (stored chunks are embedded this way).EMBED_ROLE_QUERYprefixesSESSION_QUERY_INSTRUCTION; an unknown role raises. OnlyLlamaCppEmbeddertakes it, since only it embeds session text (embeds_on_this_host)history_vectors.pyvector_rows(signature)reads every reference's vector in the live space;vectors_fingerprint(signature)moves when those rows would change, including a re-chunk that leaves every count the samehistory_vectors_search.py(new)VectorLeg: query embedded atPRIORITY_INTERACTIVEwith the query role, memoised by text and space (64 entries); one resident normalised float32 matrix scored by a dot product, loaded on the first meaning query, rebuilt when the fingerprint moves but at most every 60 s while it keeps moving, released with the memo afterembedding_idle_evict_secswithout a meaning query. Best chunk per session, ties newer first, at most 10.routeapplies RFC 7.3 and 8.3;meaning_snippetre-reads a hit's chunkhistory_search.py,history_search_memo.py,history.py_search_parsedasks the leg after the keyword ranking, except for atitle:query, a quoted phrase or an excluded term (the vector leg cannot honour either), or a caller passingby_meaning=False, which also joins the single-flight key.SearchRows.semanticcarries{indexed, eligible};semantic_coverageandsemantic_coverage_noteread itsessions_parts/history_reads.py,sessions_parts/message_search.py,handlers_instances.py,handlers/sessions.pyGET /api/sessions/searchaddssemanticwhile the index is incomplete, and itsexclude_incognitoarm keeps a meaning row's chunk snippet while re-judging the row as before. Message search passesby_meaning=False, since it lists messages that hold the query's own words. A federated peer row keepsmatchonly when it is"text"or"meaning". The facade manifest listssemantic_coveragemcp_tools/sessions.py,mcp_tools/catalog/sessions.py,messaging/session_resume.py,discord/session_resume.py,teams/cards.py,telegram/session_resume.pysearch_chat_historynames each hit's match kind, prints a meaning hit's chunk, adds the coverage note, and its description says a meaning snippet need not contain the query's words (RFC 8.2).SessionChoice.relatedmarks a meaning row: each channel's picker button showsRelated: <title>throughlabel, a resume notice quotes the bare title, and the coverage note joins the scan noteeval/bench/session_recall.py,cli_bench.pykirocrew bench session-recall --meaningchunks and embeds the set with this host's model, holds the flag on for the run, and adds per-class R@1 for the vector and routed rankers, both orders of the routed rows with the newest 50, 75 and 90% of sessions embedded, and the leg's cost. A host without the in-process model is refused rather than scored as keywordconfig/sections_session.py,config-baseline.json,configuration.md,history.mdsession.semantic_searchhelp text describes search by meaning; the spec gains the leg and the harnessRouting (RFC 7.3, decided on set D): a query of at least
SEMANTIC_LITERAL_MIN_CHARS(12) characters whose case-folded whole occurs in the keyword top session is literal, and gets both legs fused byHybridRetriever._rrf_fuse(k 60, vector weight 1.5) with that session first. Any other query gets the vector rows first and the keyword rows the vector leg did not return after them. While fewer thanSEMANTIC_MIN_COVERAGE(0.9) of the visible sessions are embedded, keyword rows lead and meaning rows follow, so a half-built index never displaces a keyword result (RFC 8.3). Both constants stay named; before phase 4, 12 is scored again against 16 and 20 on a fresh set, as the RFC records.Privacy and ownership: candidates come only from the keyword search's own window, with incognito and temporary sessions removed, so a deleted or private session never ranks whatever the matrix still holds. Every row then passes the same
_result_rowsgates as a keyword row,keysincluded, so an app caller's search never returns another app's session from the vector leg. A meaning row's snippet is read again under the transcript lock through_withhold_if_restricted, at the chunk's message and offsets; a session that turned private since its vector was written is dropped there. The query is embedded only by the in-process model, so withollama_httpno query text leaves the host and the rows stay keyword rows. Thesemanticcounts leave private sessions out and areNonefor a search restricted bykeys.Measurements (RFC 10, phase 3 exit criteria)
kirocrew bench session-recall --meaningon the shipped fixture (40 sessions, 56 queries) with the bundledqwen3-embedding-0.6b, R@1:Partial coverage, newest sessions embedded first, R@1 / R@10:
Below full coverage, keyword rows first scores higher at every share measured, which supports keeping them first while the index is built. At 0.9 the two orders differ by one query, so the fixture is too small to place the threshold more precisely than that.
Cost at 160 chunks of dimension 1024: the matrix is 640 KiB (4096 bytes per chunk) and its load allocated at most 1992 KiB at once. The fill took 1.54 s of process CPU per chunk, and one vector ranking, query embedding included, took 121 ms wall and 677 ms CPU (median; llama.cpp runs several threads). At the 51,000 chunks RFC 6.5 projects for a large store, 4096 bytes per chunk is about 209 MB, the figure the RFC gives.
Gaps a maintainer is asked to accept here (RFC 10):
#2817: 5 characters, underSEMANTIC_LITERAL_MIN_CHARS, so it takes the vector ranking by the rule decided on set D, and the vector leg puts the issue's session outside the top row. Exempting a query that is wholly a forge reference would change the decided rule without a fresh measurement, so this PR does not.Backwards compatibility
Compatible. With the flag off (the default), no store, numpy absent or a non-local embedder, search returns today's rows row for row and they carry no
match(test 1 in RFC 7.3).matchandsemanticare new fields a caller can ignore. A federated peer on an older build sends nomatch, and its rows read as text.embedandembed_batchkeep their default behaviour;roleis optional.search_sessionsgains an optionalby_meaning, andSessionChoicean optionalrelated.Removes nothing.
Tests
test/test_history_vectors_search.py(new) andtest/test_bench_session_recall.py, with a word-counting tokenizer and a stand-in embedder whose vectors put words of one topic on one axis (test/session_meaning_helpers.py), over real transcript files chunked and filled as in production:match; a description finds its session by meaning with its chunk as snippet; a literal query of at least 12 characters keeps the keyword top first; a short query that occurs in the keyword top takes the vector ranking; a partial index puts every keyword row first. An excluded term keeps the keyword rows.keys, with no counts; a deleted session never ranks; a session made incognito or temporary never ranks while its vectors remain and is never offered to the leg; the snippet read refuses a session made private; private sessions never move the counts.semanticonly while incomplete), message search keeping the keyword ranking and never embedding,search_chat_historynaming the match kind and the note and printing today's lines without the leg, the channel picker marking a meaning row, a peermatchkept only when it is one of the two kinds (test_instances_search_federation.py).Every guard was disabled in turn and at least one test went red (28 of 28).
test_history_bindings.pyrecords the newsearch_sessionssignature.The full backend suite on this host fails the same 51 tests as main, all host-specific (no user namespace for the design-tweak, terminal and app sandboxes, a real codex-acp probe); one more,
test_monitor_provider_cli_orphaned_child.py, failed once under load and passes alone on repeated runs.Gate proof: tree 0540c8e31ac5; ran 24; skipped none
Related Issues
Part of #24476
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)