Skip to content

feat(history): search sessions by meaning behind session.semantic_search - #39791

Open
dma9527 wants to merge 2 commits into
kirodotdev:mainfrom
dma9527:feat/session-semantic-search-leg
Open

dma9527 wants to merge 2 commits into
kirodotdev:mainfrom
dma9527:feat/session-semantic-search-leg

Conversation

@dma9527

@dma9527 dma9527 commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let session search find a conversation from a description of it, as well as from its words, behind session.semantic_search (default off), with today's rows unchanged while the flag is off.

Phase 3 of the session search RFC (accepted on #24476, record updated in #39386), backend half, in two commits. The first adds the query side: the query role on the embed call, the vector leg over the chunk store #36002 built, the routing rule decided on set D, match and chunk snippets on result rows, the coverage note, and the backend surfaces. The second extends the recall harness with the vector and routed rankers, partial coverage and the leg's cost, as the RFC's phase 3 exit criteria ask. The dashboard rows ("Related" label in the sidebar and the command palette) follow in a separate UI PR.

Why it matters

On the shipped recall fixture, keyword search finds no paraphrase, Chinese paraphrase or identifier plus description query in its top 10 (R@10 0.000 for each). These are the queries people use when they remember what a conversation was about but not its words. With the leg on, all answerable queries reach R@1 0.880 against 0.340 today.

Not a goal

  • No change with the flag off, no store, or a non-local embedder: rows, order and snippets are today's, and the rows carry no match.
  • No default change. Phase 4 turns the flag on, after the measurements the RFC lists.
  • No dashboard UI change; the sidebar and palette label is the next PR.

What changed (motivation → approach → change)

Commit 1 (search sessions by meaning) holds the leg and every surface; commit 2 (score the meaning leg in the session recall bench) holds the harness.

Part What it does
Embed role: embeddings_llama.py, embeddings_settings.py, embeddings.py embed and embed_batch take role, default EMBED_ROLE_DOCUMENT, which sends text as given (stored chunks are embedded this way). EMBED_ROLE_QUERY prefixes SESSION_QUERY_INSTRUCTION; an unknown role raises. Only LlamaCppEmbedder takes it, since only it embeds session text (embeds_on_this_host)
Store reads: history_vectors.py vector_rows(signature) reads every reference's vector in the live space; vectors_fingerprint(signature) moves when those rows would change, including a re-chunk that leaves every count the same
The leg: history_vectors_search.py (new) VectorLeg: query embedded at PRIORITY_INTERACTIVE with the query role, memoised by text and space (64 entries); one resident normalised float32 matrix scored by a dot product, loaded on the first meaning query, rebuilt when the fingerprint moves but at most every 60 s while it keeps moving, released with the memo after embedding_idle_evict_secs without a meaning query. Best chunk per session, ties newer first, at most 10. route applies RFC 7.3 and 8.3; meaning_snippet re-reads a hit's chunk
Search: history_search.py, history_search_memo.py, history.py _search_parsed asks the leg after the keyword ranking, except for a title: query, a quoted phrase or an excluded term (the vector leg cannot honour either), or a caller passing by_meaning=False, which also joins the single-flight key. SearchRows.semantic carries {indexed, eligible}; semantic_coverage and semantic_coverage_note read it
Dashboard: sessions_parts/history_reads.py, sessions_parts/message_search.py, handlers_instances.py, handlers/sessions.py GET /api/sessions/search adds semantic while the index is incomplete, and its exclude_incognito arm keeps a meaning row's chunk snippet while re-judging the row as before. Message search passes by_meaning=False, since it lists messages that hold the query's own words. A federated peer row keeps match only when it is "text" or "meaning". The facade manifest lists semantic_coverage
Agent and channels: mcp_tools/sessions.py, mcp_tools/catalog/sessions.py, messaging/session_resume.py, discord/session_resume.py, teams/cards.py, telegram/session_resume.py search_chat_history names each hit's match kind, prints a meaning hit's chunk, adds the coverage note, and its description says a meaning snippet need not contain the query's words (RFC 8.2). SessionChoice.related marks a meaning row: each channel's picker button shows Related: <title> through label, a resume notice quotes the bare title, and the coverage note joins the scan note
Harness (commit 2): eval/bench/session_recall.py, cli_bench.py kirocrew bench session-recall --meaning chunks and embeds the set with this host's model, holds the flag on for the run, and adds per-class R@1 for the vector and routed rankers, both orders of the routed rows with the newest 50, 75 and 90% of sessions embedded, and the leg's cost. A host without the in-process model is refused rather than scored as keyword
Config and docs: config/sections_session.py, config-baseline.json, configuration.md, history.md The session.semantic_search help text describes search by meaning; the spec gains the leg and the harness

Routing (RFC 7.3, decided on set D): a query of at least SEMANTIC_LITERAL_MIN_CHARS (12) characters whose case-folded whole occurs in the keyword top session is literal, and gets both legs fused by HybridRetriever._rrf_fuse (k 60, vector weight 1.5) with that session first. Any other query gets the vector rows first and the keyword rows the vector leg did not return after them. While fewer than SEMANTIC_MIN_COVERAGE (0.9) of the visible sessions are embedded, keyword rows lead and meaning rows follow, so a half-built index never displaces a keyword result (RFC 8.3). Both constants stay named; before phase 4, 12 is scored again against 16 and 20 on a fresh set, as the RFC records.

Privacy and ownership: candidates come only from the keyword search's own window, with incognito and temporary sessions removed, so a deleted or private session never ranks whatever the matrix still holds. Every row then passes the same _result_rows gates as a keyword row, keys included, so an app caller's search never returns another app's session from the vector leg. A meaning row's snippet is read again under the transcript lock through _withhold_if_restricted, at the chunk's message and offsets; a session that turned private since its vector was written is dropped there. The query is embedded only by the in-process model, so with ollama_http no query text leaves the host and the rows stay keyword rows. The semantic counts leave private sessions out and are None for a search restricted by keys.

Measurements (RFC 10, phase 3 exit criteria)

kirocrew bench session-recall --meaning on the shipped fixture (40 sessions, 56 queries) with the bundled qwen3-embedding-0.6b, R@1:

Class n keyword vector routed
literal 8 1.000 0.750 0.875
paraphrase 16 0.000 0.750 0.750
paraphrase_cjk 8 0.000 0.875 0.875
short 12 0.750 1.000 1.000
identifier_plus_description 6 0.000 1.000 1.000
all answerable 50 0.340 0.860 0.880

Partial coverage, newest sessions embedded first, R@1 / R@10:

Coverage keyword rows first meaning rows first
0.50 0.700 / 0.720 0.600 / 0.620
0.75 0.800 / 0.860 0.720 / 0.800
0.90 0.860 / 0.920 0.840 / 0.900
1.00 (routed) 0.880 / 0.980

Below full coverage, keyword rows first scores higher at every share measured, which supports keeping them first while the index is built. At 0.9 the two orders differ by one query, so the fixture is too small to place the threshold more precisely than that.

Cost at 160 chunks of dimension 1024: the matrix is 640 KiB (4096 bytes per chunk) and its load allocated at most 1992 KiB at once. The fill took 1.54 s of process CPU per chunk, and one vector ranking, query embedding included, took 121 ms wall and 677 ms CPU (median; llama.cpp runs several threads). At the 51,000 chunks RFC 6.5 projects for a large store, 4096 bytes per chunk is about 209 MB, the figure the RFC gives.

Gaps a maintainer is asked to accept here (RFC 10):

  1. Fixture, literal R@1: routed 0.875 against keyword 1.000. The one miss is #2817: 5 characters, under SEMANTIC_LITERAL_MIN_CHARS, so it takes the vector ranking by the rule decided on set D, and the vector leg puts the issue's session outside the top row. Exempting a query that is wholly a forge reference would change the decided rule without a fresh measurement, so this PR does not.
  2. Set C, all queries: routed R@1 0.275 against vector R@1 0.312 (RFC 8.5), unchanged by this PR because the rule is the one RFC 7.3 records. On the fixture, routed R@1 is at or above vector R@1 (0.880 against 0.860).

Backwards compatibility

Compatible. With the flag off (the default), no store, numpy absent or a non-local embedder, search returns today's rows row for row and they carry no match (test 1 in RFC 7.3). match and semantic are new fields a caller can ignore. A federated peer on an older build sends no match, and its rows read as text. embed and embed_batch keep their default behaviour; role is optional. search_sessions gains an optional by_meaning, and SessionChoice an optional related.

Removes nothing.

Tests

test/test_history_vectors_search.py (new) and test/test_bench_session_recall.py, with a word-counting tokenizer and a stand-in embedder whose vectors put words of one topic on one axis (test/session_meaning_helpers.py), over real transcript files chunked and filled as in production:

  • Query role: a call without it sends the same text as main; the query role prefixes the instruction; an unknown role is refused.
  • RFC 7.3 tests 1 to 5: flag off, no store and a remote embedder each return today's rows with no match; a description finds its session by meaning with its chunk as snippet; a literal query of at least 12 characters keeps the keyword top first; a short query that occurs in the keyword top takes the vector ranking; a partial index puts every keyword row first. An excluded term keeps the keyword rows.
  • RFC 9 and 12: a foreign session from the vector leg is dropped under keys, with no counts; a deleted session never ranks; a session made incognito or temporary never ranks while its vectors remain and is never offered to the leg; the snippet read refuses a session made private; private sessions never move the counts.
  • Units: the literal test, the chunk snippet cut, the fingerprint moving on a re-chunk that keeps every count, the reload throttle, the idle release.
  • Surfaces: the dashboard search on both arms (match, chunk snippet, semantic only while incomplete), message search keeping the keyword ranking and never embedding, search_chat_history naming the match kind and the note and printing today's lines without the leg, the channel picker marking a meaning row, a peer match kept only when it is one of the two kinds (test_instances_search_federation.py).
  • Harness: each ranker scoring the same queries with the flag off in config, both orders at each share, the cost at the chunk count, a host without the model refused with the flag put back, the report and the CLI flag.

Every guard was disabled in turn and at least one test went red (28 of 28). test_history_bindings.py records the new search_sessions signature.

The full backend suite on this host fails the same 51 tests as main, all host-specific (no user namespace for the design-tweak, terminal and app sandboxes, a real codex-acp probe); one more, test_monitor_provider_cli_orphaned_child.py, failed once under load and passes alone on repeated runs.

Gate proof: tree 0540c8e31ac5; ran 24; skipped none

Related Issues

Part of #24476

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Phase 3 of the session search RFC, backend half. While the flag is on (default
off), a session search embeds the query with the in-process model under a new
query role, scores the chunk vectors of the keyword search's own window, and
routes the two rankings: a query of at least 12 characters that occurs whole in
the keyword top session keeps it first over both legs fused; any other query
gets the vector rows first, except that keyword rows lead while fewer than 90%
of visible sessions are embedded. Each row carries match ("text" or "meaning"),
and a meaning row's snippet is its chunk, re-read under the transcript lock
through the derivation guard. Incognito and temporary sessions are never
candidates, and every row passes the keyword rows' gates. The dashboard search,
search_chat_history and the channel picker carry the match kind and a coverage
note; message search keeps the keyword ranking. With the flag off, no store or
a non-local embedder, rows are unchanged.

Part of kirodotdev#24476
kirocrew bench session-recall --meaning chunks and embeds the recall set with
this host's in-process model and reports, beside the keyword ranking, R@1 per
query class for the vector leg and the routed rows, both orders of the routed
rows with the newest 50, 75 and 90% of sessions embedded, and the leg's cost:
chunk count, matrix bytes, the matrix load's peak allocation, fill CPU per
chunk and median query time. A host without the model is refused rather than
scored as keyword search. These are the phase 3 exit measurements of the
session search RFC.

Part of kirodotdev#24476
@dma9527
dma9527 requested review from a team and KiroCrewCaptain as code owners November 20, 2026 05:23
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ⚠️ review incomplete

Reviewed 3258ba753ce99dad93eba8baa1126735b851be2f via the fork AI-review pipeline; updated in place on each push.

No completed Opus verdict for this commit; see the Fork Opus 5.5 Review job logs.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 3258ba753ce99dad93eba8baa1126735b851be2f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 3258ba753ce99dad93eba8baa1126735b851be2f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Everything ships behind session.semantic_search, which stays off by default, and RFC rfc-session-semantic-search (status partial on base) covers the phase 3 shape.

What this change ships

Inventory (10 items) — 10 justified

Intent: let a person find a past conversation by describing it, not only by its exact words. This is an ADDITION, behind a flag that stays off.

  1. With the flag on, session search also ranks sessions by meaning, using the existing chunk store — justified
  2. Query embedding gets a query role that adds an instruction prefix; every other caller still embeds bare text — justified
  3. Rule that decides between both legs: a query of 12+ characters found in the top text match keeps that session first; otherwise meaning rows lead — justified
  4. While under 90% of sessions are embedded, text matches come first — justified
  5. Rows carry match (text or meaning), and a meaning row's snippet is its chunk, re-read under the privacy guard — justified
  6. Coverage note ("Meaning search covers N of M") in the API reply, search_chat_history and the channel pickers — justified
  7. Discord, Telegram and Teams picker buttons show Related: <title> for meaning rows — justified
  8. Message search and the federated peer match field stay keyword-only or are passed through checked — justified
  9. kirocrew bench session-recall --meaning: vector, routed, partial-coverage and cost figures (RFC 10 exit criteria) — justified
  10. Updated help text for the flag in config, docs and spec — justified

[FIRST-PRINCIPLES-REVIEWED] 3258ba7

Description read

[DESCRIPTION-READ] ce061c249fb2308911e966232b666e8cb5f9ff1bf33b10e9a9c76d808ccbfe6f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 3258ba753ce99dad93eba8baa1126735b851be2f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the flag is off by default and every effect can be undone. The coverage denominator below needs fixing before phase 4.

Notes

  • meaning_route computes visible from sessions, which is the whole searchable listing (history_search.py:892). The chunk store only covers sessions[:scan_window]. With more than about 2,222 visible sessions (default window 2000), coverage can never reach SEMANTIC_MIN_COVERAGE, so keyword rows always come first. Meaning rows then fall off routed[:limit] whenever the keyword hits fill the limit. The "Meaning search covers N of M sessions so far" note also stays up permanently. Fix: count over window, as VectorFill.coverage does.

[DESIGN-REVIEWED] 3258ba7

Description read

[DESCRIPTION-READ] 76c233ebc41599c03812a743e7b1182a40ba991ecc5e84541db0897ece3c63a1

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 3258ba753ce99dad93eba8baa1126735b851be2f via the fork AI-review pipeline; updated in place on each push.

Adjudication upheld 3 of 3 blocking finding(s); those must be resolved before merging.

BLOCKING -- src/kiro_crew/history_vectors_search.py:161 -- Apply caller eligibility before the ten-hit cap

for key, (score, position) in ranked[:SEMANTIC_MAX_ROWS]:
Trigger: A workspace-scoped search_chat_history query has no keyword matches, ten higher-scoring semantic hits outside its workspace, and an eligible hit ranked eleventh; VectorLeg.rank caps the hits before the MCP handler applies workspace/date filters.
Outcome: The tool reports no matching conversations despite an embedded, eligible match inside the search window. Its full-window request cannot recover the discarded hit.
Anchor: defect
Fix: Apply workspace/date eligibility before selecting the ten semantic hits, or defer that cap until those filters run.

BLOCKING -- src/kiro_crew/history_vectors_search.py:397 -- Exclude unindexable sessions from routing coverage

meta["key"] for meta in sessions if not is_incognito_transcript(meta.get("memory_mode"))
Trigger: With 3,000 visible sessions and the default 2,000-session window fully embedded, meaning_route counts all 3,000, while backfill_chunks indexes only the window and removes departed sessions’ vectors.
Outcome: Routing coverage remains 2,000/3,000 after the build finishes, permanently keeping non-literal searches keyword-first despite a fully embedded search window.
Anchor: defect
Fix: Calculate routing coverage against the visible, indexable window rather than the complete session listing; keep overall scan coverage separate.

BLOCKING -- src/kiro_crew/history_vectors_search.py:220 -- Release matrices owned by one-shot MCP searches

self._idle_timer = _start_idle_timer(idle_secs, self.release)
Trigger: Repeated semantic search_chat_history calls in the persistent MCP server each construct a fresh ConversationLog and VectorLeg, load a complete matrix, and return without releasing it; the timer retains the bound release callback.
Outcome: Every re-search retains another complete corpus matrix and daemon timer until its independent idle deadline—an hour with the normal automatic setting—so retained memory grows with the number of calls instead of reusing one matrix.
Anchor: defect
Fix: Release the one-shot MCP leg in a finally block after constructing the response, or reuse a directory-scoped leg with an explicit lifecycle.

[BLOCK-MERGE] 3258ba7
[GPT-REVIEWED] 3258ba7

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 (history_vectors_search.py:161): MEDIUM. The search silently misses an eligible match on the default path. rank keeps only the top 10 hits across the whole window (patch line 1187). search_chat_history scopes to the caller's workspace by default (mcp_tools/sessions.py:267) and only filters workspace and date afterwards (:302-308). Nothing recovers a hit cut at the cap. The fix is moderate (filter the candidates first, or defer the cap), and the miss is invisible to the user, so blocking is proportionate.

F2 (history_vectors_search.py:397): MEDIUM, and it never corrects itself. Coverage is counted over the full listing (history_search.py:892), but the chunk pass only indexes the window (history_vectors_index.py:204). With more than about 2,222 sessions, coverage stays under 0.9 for good: searches stay keyword-first and keep showing "covers N of M so far". The fix is about one line (count over window).

F3 (history_vectors_search.py:220): HIGH. Each tool call builds a new ConversationLog (mcp_tools/sessions.py:264), and with it a new projection and leg (history.py:519). Each leg keeps its full float32 matrix alive through the timer, for an hour by default (embeddings_settings.py:360-363). Repeated searches stack up whole-corpus matrices until the process runs out of memory. The fix is cheap: release in a finally, or keep one leg per directory.

[ADJUDICATION] 3258ba753ce99dad93eba8baa1126735b851be2f total=3 uphold=3 downgrade=0
UPHOLD F1 src/kiro_crew/history_vectors_search.py:161 reason=harm-warrants-remedy
UPHOLD F2 src/kiro_crew/history_vectors_search.py:397 reason=harm-warrants-remedy
UPHOLD F3 src/kiro_crew/history_vectors_search.py:220 reason=harm-warrants-remedy
[GPT-ADJUDICATED] 3258ba753ce99dad93eba8baa1126735b851be2f

@bolichen97

Copy link
Copy Markdown
Contributor

Thanks for the clear write-up and the bench numbers. I read the leg, the routing, and the privacy path (meaning_snippet re-reading under the transcript lock through _withhold_if_restricted, and incognito keys dropped before ranking). That part looks right to me.

I checked the three GPT 6.1 blockers on 3258ba753c against the code, and all three hold:

  1. Cap before caller filters (history_vectors_search.py:161): rank keeps ranked[:SEMANTIC_MAX_ROWS] over the whole window. search_chat_history only drops other-workspace and out-of-date rows afterwards (mcp_tools/sessions.py, the _ws_bucket(...) != current_ws check). So a workspace-scoped search can lose an eligible meaning hit ranked 11th. Passing the eligible keys into rank, or capping after the filters, fixes it.
  2. Coverage counted over the full listing (history_vectors_search.py:397): visible comes from sessions, but the chunk pass indexes only the window. With more sessions than the window, coverage never reaches 0.9, so search stays keyword-first and keeps printing "covers N of M so far" for good. Counting against the visible window fixes it.
  3. One matrix per MCP call (history_vectors_search.py:220): each search_chat_history call builds a new ConversationLog(), so a new VectorLeg, and the idle timer keeps that leg's whole matrix alive for embedding_idle_evict_secs. Repeated calls stack matrices. Releasing the leg in a finally after the MCP response is built, or sharing one leg per history directory, fixes it.

The flag is off by default, so none of this hits users today, but phase 4 turns it on, so these should land in this PR. Push a fix and I'll re-review the new head right away. The Opus lane said "review incomplete" (no verdict), which is a pipeline issue on our side, not something you need to fix.

AI-assisted, sent by Bolin Review bot

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants