Repository navigation
feat(history): build and fill a session chunk store behind session.semantic_search - #36002
Conversation
e236673 to
3efc43e
Compare
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the store is behind a flag that is off by default, nothing reads it yet, and it deletes on the keyword index's fail-closed terms. Notes
[DESIGN-REVIEWED] 77d4261 Description read[DESCRIPTION-READ] 031a3cf9e1e8e05104e639aa0b1fddf7abbf4d418d1a3bf00066aef8dbc0eee3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the store builds only behind an off-by-default flag, phase 2 of a Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: let a later phase find a past session by describing it, by building chunk vectors in the background now. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 77d4261 Description read[DESCRIPTION-READ] 82fedd0fcf0a32e6a0706436ee661ce227a8db3bd81820151b320416abfa9a8e That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] 77d4261 |
GPT 6.1 Review (fork) — ✅ human override accepted@bolichen97 overrode this lane for [GPT-OVERRIDE] 77d4261 |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks @dma9527. This is a careful phase-2 change. The store, its delete contract and its tests read well. One privacy gap stops me approving. It is small to fix.
Verdict: request changes (one finding).
🔴 Private and departed rows are only dropped while a tokenizer is loaded
- Where:
src/kiro_crew/history_index_worker.py:288and:292(_ChunkPass.run). - Cause:
runreturns early whensession.semantic_searchis off, or when no tokenizer is available (no model file, or a failed load).backfill_chunksis the only place that drops a session turned incognito or temporary, a session that left the window, and an owed WAL truncate (store.settle(),history_vectors_index.py:149). That code never runs on those paths. - Trigger: turn the flag on and let the store fill, so
pending_textholds the text. Turn the flag off, or remove the model. Then make a session incognito. - Consequence: the session's text stays in
session_vectors.dbwith no end date. RFC section 12 says this store "gets the FTS index's deletion and privacy contract". The FTS worker keeps dropping private sessions with no flag, so after the toggle the two contracts diverge. The person who made the session private has no sign that a copy is left. - Repro, run on this head with the test file's own helpers: flag on, one
_ChunkPass.run, then flag off, rewrite the session withmode="incognito", then run twice.pending_textstill holds('secret words here',). - Fix: the drop half needs no tokenizer. Split
backfill_chunkssosettle(), the private drop and the departed drop run on every pass wherestore.store_exists()is true, whatever the flag or model state. Only the chunking loop needs the flag and a tokenizer. With no store file, keep the current behavior: a flag-off host never creates one. Add one test per early return: flag off, and no model, each after a filled store.
Checked and fine
- Off by default:
session.semantic_searchdefaults tofalseinsections_session.py,loader_builders.py,config-baseline.jsonandconfiguration.md. With the flag off on a fresh host, no file is created (store_existsgatesdrop, and the worker returns before opening). - What is stored matches the updated RFC 7.1 table: hashes, token counts, offsets, per-session file identity, and the chunk text in
pending_textonly until a vector exists. No other text and no titles. - Bounds: work is limited to the keyword index's
_searchablewindow andsearch_scan_window, sessions that leave it are dropped, and each pass stops atpass_budget_secs. The worker loads the tokenizer withvocab_only=True, so no weights. The per-pass config read follows the same patternconfigured_search_scan_windowalready uses. - Incognito and temporary on the flag-on path: dropped from the listing before the quiet window, every read goes through
_withhold_if_restrictedunder_locked(key), andTranscriptBusydrops the rows. - Deletion:
delete_sessioncallsvector_store.drop(identities)under the same lock and refuses the delete when a store exists but cannot be erased.secure_delete=ONplus a checkedwal_checkpoint(TRUNCATE), and orphan chunks go while shared ones stay. The sync runs under the session lock, so a delete cannot race a half-written sync.
Local run on 3efc43e3a8: test/test_history_chunks.py and test/test_history_vectors_index.py gave 38 passed. test/test_history_index_worker.py and test/test_config_baseline.py gave 52 passed. flake8 and mypy are clean on the changed source.
After the fix, ping here and I'll re-review the new head.
AI-assisted, sent by Bolin Review bot
3efc43e to
41255ef
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Re-review at 41255efaba (de850e8 + 41255ef).
Earlier blocker: fixed. backfill_chunks now prunes private and departed sessions whenever a store file exists, and runs chunking only with a tokenizer (history_vectors_index.py:168-187). _ChunkPass.run calls it every round whatever the flag (history_index_worker.py:288-300). test_a_filled_store_still_drops_private_and_departed_sessions[flag-off|no-model] fills the store, turns the switch off (or removes the model), makes a session incognito and asserts pending_text is empty. That is the case I asked for. Thanks.
The vector-fill commit is in scope. RFC section 7.1 puts the fill in the gateway at PRIORITY_BULK, and phase 2's exit criterion ("the fixture indexes to coverage 1.0") needs vectors. It stays behind session.semantic_search (off), never creates the store, paces 8 texts per step at bulk priority, and the coverage counts leave private sessions out. Keeping it in this PR is fine.
Two things in the new commit need a change before merge:
1. A reused chunk_id gets the vector of a dropped chunk (blocker)
src/kiro_crew/history_vectors.py:40:chunk_id INTEGER PRIMARY KEYwithoutAUTOINCREMENT, so SQLite hands a dropped top id to the next insert.src/kiro_crew/history_vectors.py:194-213:write_vectorsmatches only onchunk_id.- Trigger: the fill reads a batch, then blocks in
embed_batch(seconds per row at bulk priority). Meanwhile the worker process drops a session (made incognito, or deleted) whose chunk holds the top id, and in the same pass chunks another session. The new chunk gets the same id. The fill then writes the dropped text's vector onto it and deletes the new chunk's pending text. - Consequence: a vector derived from an incognito or deleted session stays in the store after the drop returned, which breaks the section 12 deletion contract (a vector can leak its text). The other session keeps a wrong vector and never gets its own until a model change.
- Reproduced at this head, using the helpers in
test/test_history_vectors_fill.py:
def test_turned_private_mid_embed(tmp_path, space):
_write(tmp_path, "secret", "secret words")
_chunk(tmp_path)
store = chunking.vector_store_for(tmp_path)
def during():
_write(tmp_path, "secret", "secret words", mode="incognito")
_write(tmp_path, "other", "other words")
_chunk(tmp_path)
space["embedder"].during = during
fill.fill_batch(store)
# chunk 1 now: chunk_ref ('other', 1), vector == embedding of "secret words",
# pending_text empty- Fix (either one): declare
chunk_id INTEGER PRIMARY KEY AUTOINCREMENT(the store is new in this PR, so no migration), or makewrite_vectorsclaim the pending row by content:DELETE FROM pending_text WHERE chunk_id=? AND text=?and updatechunkonly when that deleted a row. Please add the test above with asserts that no row holds the secret's vector and thatotheris still pending.
2. The fill can send transcript text to a remote Ollama server
src/kiro_crew/history_vectors_fill.py:65-67embeds throughget_shared_embedder(). Withmemory.embedding_provider = "ollama_http"andmemory.ollama_allow_remote, that backend posts the text to a non-loopback host.ollama_allow_remote's help only discloses that "memory and knowledge text" leaves the machine. RFC section 12 says "Embedding runs on the local model, so no transcript text leaves the host", and the newsession.semantic_searchhelp (sections_session.py:258-260,configuration.md) says "the local embedding model". With both switches on, whole transcripts go off-host, which no setting tells the operator.- It also mismatches the chunker: chunks are cut with the local GGUF tokenizer, and vectors would come from a different model.
- Fix: make
VectorFill.step(andfill_batch) run only when the active provider is the in-process one (positive check,embedding_provider == "llama_cpp"), with a test that anollama_httpprovider fills nothing. Disclosing it in the help text instead is also acceptable if you prefer, but then the RFC's section 12 line needs to change in this PR too.
Checks run here
pytestontest_history_chunks.py,test_history_vectors_index.py,test_history_vectors_fill.py,test_dashboard_server_composition_contract.py,test_history_index_worker*.py: 133 passed.flake8on the changed src and tests: clean.mypyon the 10 changed src files: no issues.check_file_line_cap.py: 0 failures.Internal Content Scanis red with "NO VERDICT, the scan did not run to completion". That is the runner, not this diff; it will need a rerun.
Note, not blocking: when backfill_index raises, run_index_worker skips chunks.run for that round (history_index_worker.py:236-243), so the privacy prune waits for the keyword pass to recover. Running the prune before the keyword pass, or in its except branch too, would remove that dependency.
Once 1 and 2 are fixed this is ready from my side.
AI-assisted, sent by Bolin Review bot
|
@bolichen97 thanks for the repro. Fixed in de850e8: with no tokenizer (flag off or no model) |
41255ef to
69fe866
Compare
|
@bolichen97 both fixed in 69fe866:
From the bot reviews: a model change now keeps released sessions in the prune inventory, and both stores use one WAL check, On the non-blocking note: when Could you approve the held CI and re-review? |
…arch Phase 2a of the session search RFC. While the flag is on (default off), the session index worker cuts quiet sessions into chunks of at most 300 tokens of the embedding model's tokenizer, loaded vocabulary-only, and stores them in sessions/.index/session_vectors.db. Sessions are read through the derivation guard, so incognito and temporary sessions are never chunked. delete_session drops the store under the keyword index's contract and refuses the delete when it cannot. Search does not read the store. Part of kirodotdev#24476
Phase 2b of the session search RFC. While session.semantic_search is on, the gateway embeds the chunk store's pending text with the shared embedding model at bulk priority, writing each vector and deleting its text in one transaction. A batch is discarded when the model is not ready or its vector space changed; vectors from an old space are cleared and their sessions re-chunked. Coverage counts the visible search window only, so private sessions never move it. Search does not read the store. Part of kirodotdev#24476
Head branch was pushed to by a user without write access
8e950f3 to
77d4261
Compare
|
@bolichen97 rebased on bc8c588 in 77d4261.
Before pushing I ran the full backend suite the way CI does: 51 local failures, the same 51 test ids as on clean main (this host has no user namespaces for the sandbox). Pre-push gates, base gates and the 2a commit on its own are green. Could you approve the held CI and re-approve? |
|
Maintainer decision on the two fenced GPT 6.1 findings on 77d4261: risk accepted (Bolin Chen, @bolichen97).
The feature ships behind AI-assisted, sent by Bolin Review bot on Bolin's instruction |
|
/ai-review override gpt 77d4261 owner-decision: Bolin Chen (maintainer) accepted both fenced risks after review: narrow windows, data stays on host, next prune or truncate clears it; feature is off by default. Decision: #36002 (comment) |
Human judgment recorded@bolichen97 recorded a owner-decision release (class
This decision applies only to this commit. A new push requires a new judgment. |
Phase 2 of the session search RFC, the chunk store and its gateway vector fill behind session.semantic_search, merged as kirodotdev#36002. List it in implementation-prs, name it in the status line and the index row, and move the audit stamps to the commit the record was checked against. Record the answers to open questions 2 and 5, which phase 3 needs first, from a fresh blind query set scored once against pre-registered rules: the literal route stays on the whole query, and fires only for a query of at least 12 characters. Part of kirodotdev#24476
Phase 2 of the session search RFC, the chunk store and its gateway vector fill behind session.semantic_search, merged as kirodotdev#36002. List it in implementation-prs, name it in the status line and the index row, and move the audit stamps to the commit the record was checked against. Record the answers to open questions 2 and 5, which phase 3 needs first, from a fresh blind query set scored once against pre-registered rules: the literal route stays on the whole query, and fires only for a query of at least 12 characters. Part of kirodotdev#24476
…#39386) Phase 2 of the session search RFC, the chunk store and its gateway vector fill behind session.semantic_search, merged as #36002. List it in implementation-prs, name it in the status line and the index row, and move the audit stamps to the commit the record was checked against. Record the answers to open questions 2 and 5, which phase 3 needs first, from a fresh blind query set scored once against pre-registered rules: the literal route stays on the whole query, and fires only for a query of at least 12 characters. Part of #24476 Co-authored-by: dma9527 <221155142+dma9527@users.noreply.github.com>
Problem / Motivation
Goal: Build a store of session text chunks and their vectors in the background, so a later phase can search sessions by meaning, while search itself stays exactly as it is.
Phase 2 of the session search RFC (accepted on #24476), in two commits. The first builds the chunk store, the indexer that fills it, and its deletion path. The second fills each chunk's vector in the gateway and consumes the chunk text the first commit queues. Phase 3 adds the reader; nothing reads this store yet.
Why it matters
The vector leg needs every visible session cut into model-sized chunks and embedded before it can answer anything, and the first build takes days on a large store. Building it behind an off-by-default flag lets the store, its privacy rule and its deletion contract ship and be reviewed before any ranking changes.
Not a goal
What changed (motivation → approach → change)
Commit 1 (
build a session chunk store) holds the chunker, the store, the chunk pass and its deletion path; commit 2 (fill session chunk vectors in the gateway) holds the fill.history_chunks.pyhistory_vectors.pySessionVectorStoreatsessions/.index/session_vectors.db: one row per distinct chunk, references per session, a chunk's text kept inpending_textuntil its vector is written;chunk_idisAUTOINCREMENT, so a dropped chunk's id never goes to a new chunk. Its WAL check callswal_truncatedinsqlite_wal.py, whichhistory_index.pynow uses too, so the erasure check has one copy. Commit 2 addspending_batch,write_vectors,embedded_keys, andrelease_stale, which clears vectors from another space and re-queues their sessions while keeping them in the prune's inventoryhistory_vectors_index.pyhistory_index_worker.pysession.semantic_searchon and a tokenizer, loaded vocabulary-only so no model weights enter the worker, from a path that passes the embedder's gates (a rejected custom path or a protected one is never opened, a second name on the bundled weights refuses the load, and the file's stamp is re-checked once the constructor returns)history_projection.py,history_search.pydelete_sessiondrops the session from this store right after the keyword index, under the same fail-closed contract; the projection holds the store lazily besidesearch_indexhistory_vectors_fill.py(commit 2)fill_batchembeds up to 8 pending texts with the shared model atPRIORITY_BULK, only when that model is the in-processLlamaCppEmbedder(embeds_on_this_host). Just before embedding it re-reads, under each session's lock, the privacy of every session holding a batch chunk (session_embeddable): a holder that is gone, private or unreadable loses its rows, and a chunk is embedded only while some holder is visible. It then writes each vector with the space signature of the embedder it ran on and deletes its text in one transaction; a batch is discarded when the model returns nothing (still loading) or the space changed while it embedded.coveragecounts the visible search window only.VectorFill.steppaces on what the step did, retries an owed WAL truncate before any early return, and logs coverage every 10 minutesdashboard/server_runtime/maintenance.py(commit 2)_session_vector_fillstarts beside the index worker's supervisor, runs each step on a thread, and is cancelled when that supervision endssession.semantic_search(defaultfalse) inconfig/sections_session.pyandconfig/loader_builders.py,config-baseline.json,configuration.md,history.md; RFC 7.1 and 12 updated forpending_textand the counts ruleChunking runs in the index worker because transcript parsing is what that process exists to keep off the gateway's GIL. The worker cannot embed: the model is a process-wide singleton the gateway owns. So the worker stores each new chunk's text in
pending_text, and the gateway fill in the second commit writes the vector and deletes the text in one transaction. That text is a subset of what the keyword index already holds under the same deletion contract.Privacy: a session the listing marks incognito or temporary is dropped before the quiet window, because making a session private rewrites its file and deferring it would keep its rows. Every other session is read through
_withhold_if_restricted, and one whose privacy cannot be read loses its rows until it can. A session whose first line is damaged (not UTF-8 or not JSON) is withheld rather than retried, since no read fixes it and a retry would hold the worker at its busy pace for as long as it runs. Turning the flag off, or losing the model, stops chunking and embedding but not this prune: a session made private, leaving the window, or whose first line can no longer be read afterwards still loses its rows. A chunk dropped while its batch embedded matches no row, because its id is never reused, so its vector is discarded. A model change re-queues sessions without removing them from the prune. Withmemory.embedding_provider = "ollama_http"the fill embeds nothing, so no transcript text leaves the host. Coverage counts only sessions that are neither incognito nor temporary, so neither number reveals how many private sessions exist.Deletion:
dropsucceeds when no store file exists, so a host with the flag off never gets one, and neither the prune nor the fill creates one. Otherwise it removes the session's references and every chunk no other session references, truncates the WAL and reads the checkpoint's verdict. A store that exists but cannot be opened or erased refuses the delete. A truncate that fails after the rows are gone is retried at the start of every pass until it succeeds.Backwards compatibility
Compatible: one new config key, off by default. With it off and no store file, no file is created, the worker does what it did, and the fill sleeps.
Removes nothing: the diff adds modules, a config key, a step in
delete_session, a pass in the worker and a gateway task.Tests
test/test_history_chunks.py,test/test_history_vectors_index.py,test/test_history_vectors_fill.pyandtest/test_sqlite_wal.py, with a word-counting tokenizer and a fake embedder so no model is needed:wal_truncated, intest_sqlite_wal.py).pending_textempty, and a damaged first line pruned with no tokenizer.Every guard was disabled in turn and at least one of these tests went red. The four files pass on repeated runs and three seeded shuffles of test order.
Manual verification
On a temporary store with the bundled
qwen3-embedding-0.6b.gguf: three sessions chunked (largest chunk 300 tokens, a shared message stored once), the fill embedded all six chunks (pending_textempty, 4096-byte vectors) and coverage read 3 of 3. Then, with no tokenizer, one session made incognito and one removed: the prune-only pass dropped both, the private session's unique text was in neither the database nor its WAL, and the shared chunk stayed with the remaining session. Earlier,delete_sessionon the same kind of store left the deleted session's unique text in neither file.Related Issues
Part of #24476
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)