Skip to content

feat(history): build and fill a session chunk store behind session.semantic_search - #36002

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
dma9527:feat/session-chunk-store
Nov 19, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
dma9527:feat/session-chunk-store

Conversation

@dma9527

@dma9527 dma9527 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Build a store of session text chunks and their vectors in the background, so a later phase can search sessions by meaning, while search itself stays exactly as it is.

Phase 2 of the session search RFC (accepted on #24476), in two commits. The first builds the chunk store, the indexer that fills it, and its deletion path. The second fills each chunk's vector in the gateway and consumes the chunk text the first commit queues. Phase 3 adds the reader; nothing reads this store yet.

Why it matters

The vector leg needs every visible session cut into model-sized chunks and embedded before it can answer anything, and the first build takes days on a large store. Building it behind an off-by-default flag lets the store, its privacy rule and its deletion contract ship and be reviewed before any ranking changes.

Not a goal

  • No change to search results, ranking or snippets, with the flag on or off.
  • No query-side embedding and no fusion: that is phase 3.
  • No change to the keyword index or its worker cadence when the flag is off.

What changed (motivation → approach → change)

Commit 1 (build a session chunk store) holds the chunker, the store, the chunk pass and its deletion path; commit 2 (fill session chunk vectors in the gateway) holds the fill.

Part What it does
history_chunks.py Cuts each message into chunks of at most 300 tokens of the embedding model's tokenizer; a chunk never spans two messages; lone surrogates become U+FFFD
history_vectors.py SessionVectorStore at sessions/.index/session_vectors.db: one row per distinct chunk, references per session, a chunk's text kept in pending_text until its vector is written; chunk_id is AUTOINCREMENT, so a dropped chunk's id never goes to a new chunk. Its WAL check calls wal_truncated in sqlite_wal.py, which history_index.py now uses too, so the erasure check has one copy. Commit 2 adds pending_batch, write_vectors, embedded_keys, and release_stale, which clears vectors from another space and re-queues their sessions while keeping them in the prune's inventory
history_vectors_index.py The chunk pass: the keyword index's search window and quiet period, freshness by file identity, every read under the session lock and the derivation guard. The prune half (owed WAL truncate, private, departed and unreadable-header sessions) runs on every pass while a store file exists, whatever the flag or model state
history_index_worker.py Runs the pass after each keyword pass; chunking needs session.semantic_search on and a tokenizer, loaded vocabulary-only so no model weights enter the worker, from a path that passes the embedder's gates (a rejected custom path or a protected one is never opened, a second name on the bundled weights refuses the load, and the file's stamp is re-checked once the constructor returns)
history_projection.py, history_search.py delete_session drops the session from this store right after the keyword index, under the same fail-closed contract; the projection holds the store lazily beside search_index
history_vectors_fill.py (commit 2) fill_batch embeds up to 8 pending texts with the shared model at PRIORITY_BULK, only when that model is the in-process LlamaCppEmbedder (embeds_on_this_host). Just before embedding it re-reads, under each session's lock, the privacy of every session holding a batch chunk (session_embeddable): a holder that is gone, private or unreadable loses its rows, and a chunk is embedded only while some holder is visible. It then writes each vector with the space signature of the embedder it ran on and deletes its text in one transaction; a batch is discarded when the model returns nothing (still loading) or the space changed while it embedded. coverage counts the visible search window only. VectorFill.step paces on what the step did, retries an owed WAL truncate before any early return, and logs coverage every 10 minutes
dashboard/server_runtime/maintenance.py (commit 2) _session_vector_fill starts beside the index worker's supervisor, runs each step on a thread, and is cancelled when that supervision ends
Config and docs session.semantic_search (default false) in config/sections_session.py and config/loader_builders.py, config-baseline.json, configuration.md, history.md; RFC 7.1 and 12 updated for pending_text and the counts rule

Chunking runs in the index worker because transcript parsing is what that process exists to keep off the gateway's GIL. The worker cannot embed: the model is a process-wide singleton the gateway owns. So the worker stores each new chunk's text in pending_text, and the gateway fill in the second commit writes the vector and deletes the text in one transaction. That text is a subset of what the keyword index already holds under the same deletion contract.

Privacy: a session the listing marks incognito or temporary is dropped before the quiet window, because making a session private rewrites its file and deferring it would keep its rows. Every other session is read through _withhold_if_restricted, and one whose privacy cannot be read loses its rows until it can. A session whose first line is damaged (not UTF-8 or not JSON) is withheld rather than retried, since no read fixes it and a retry would hold the worker at its busy pace for as long as it runs. Turning the flag off, or losing the model, stops chunking and embedding but not this prune: a session made private, leaving the window, or whose first line can no longer be read afterwards still loses its rows. A chunk dropped while its batch embedded matches no row, because its id is never reused, so its vector is discarded. A model change re-queues sessions without removing them from the prune. With memory.embedding_provider = "ollama_http" the fill embeds nothing, so no transcript text leaves the host. Coverage counts only sessions that are neither incognito nor temporary, so neither number reveals how many private sessions exist.

Deletion: drop succeeds when no store file exists, so a host with the flag off never gets one, and neither the prune nor the fill creates one. Otherwise it removes the session's references and every chunk no other session references, truncates the WAL and reads the checkpoint's verdict. A store that exists but cannot be opened or erased refuses the delete. A truncate that fails after the rows are gone is retried at the start of every pass until it succeeds.

Backwards compatibility

Compatible: one new config key, off by default. With it off and no store file, no file is created, the worker does what it did, and the fill sleeps.

Removes nothing: the diff adds modules, a config key, a step in delete_session, a pass in the worker and a gateway task.

Tests

test/test_history_chunks.py, test/test_history_vectors_index.py, test/test_history_vectors_fill.py and test/test_sqlite_wal.py, with a word-counting tokenizer and a fake embedder so no model is needed:

  • Chunker: the token bound, no chunk across messages, offsets that address the message, CJK cut between characters, long words split, whitespace skipped, lone surrogates.
  • Store: identical text stored once, a drop keeping a chunk another session holds, no dropped bytes left in the file or the WAL, a resync, freshness by file identity and tokenizer signature, a version change.
  • Deletion: no store file on a flag-off host, a delete removing the session's chunks, a delete refused when the store cannot be opened, a busy or raising WAL truncate reported as failure and retried (wal_truncated, in test_sqlite_wal.py).
  • Pass: incognito and temporary sessions never chunked, a session made private losing its rows even while recent, unreadable privacy, the quiet window, the search window, a restart keeping what was built, a damaged first line dropping its rows without holding the pace, a tokenizer change re-chunking, one failing session not stopping the rest.
  • Model gate: the bundled model gives a tokenizer, a rejected custom path and a protected path are never opened, an aliased model file is never opened, a file replaced during the load gives no tokenizer, a failing load gives no tokenizer, no model file gives no signature, and the flag reads off when the config cannot be read.
  • Worker: the flag off writes nothing, no model means no pass, a failed tokenizer load not retried for the same model, and, after a flag-on pass filled the store, the flag off and no model each still dropping a session made incognito and a session that left the window, with pending_text empty, and a damaged first line pruned with no tokenizer.
  • Fill: a batch writing vectors and deleting their text, an unavailable model writing nothing, a model change mid-batch discarding it, a chunk dropped while it embedded skipped, vectors from another space released once and refilled, a session released by a model change still pruned when it turns private, a session turned private while its batch embedded leaving no vector and the other session's text still pending, a remote Ollama backend filling nothing, a session made private or gone or unreadable before its batch not embedded, a shared chunk embedded while one holder is visible, a held lock skipping a chunk without dropping it, a model swapped after the batch captured its embedder not labelling old vectors as new, an owed erasure retried with the flag off or on a remote backend and not retried when nothing is owed, only the in-process model counting as on this host, private sessions never moving the counts, vectors from an old space not counted as embedded, the recall fixture reaching full coverage, no store meaning no file, the flag off filling nothing, the pace following what the step did.
  • Composition: the gateway imports the fill lazily, inside the supervisor.

Every guard was disabled in turn and at least one of these tests went red. The four files pass on repeated runs and three seeded shuffles of test order.

Manual verification

On a temporary store with the bundled qwen3-embedding-0.6b.gguf: three sessions chunked (largest chunk 300 tokens, a shared message stored once), the fill embedded all six chunks (pending_text empty, 4096-byte vectors) and coverage read 3 of 3. Then, with no tokenizer, one session made incognito and one removed: the prune-only pass dropped both, the private session's unique text was in neither the database nor its WAL, and the shared chunk stayed with the remaining session. Earlier, delete_session on the same kind of store left the deleted session's unique text in neither file.

Related Issues

Part of #24476

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@dma9527
dma9527 requested review from a team and KiroCrewCaptain as code owners November 12, 2026 18:23
@dma9527
dma9527 requested a review from mbriones98 November 12, 2026 18:23
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 12, 2026
@dma9527
dma9527 force-pushed the feat/session-chunk-store branch from e236673 to 3efc43e Compare November 13, 2026 19:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 77d42610c2762fbb8cae55eef33ab5b1f2ff7fca touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 77d42610c2762fbb8cae55eef33ab5b1f2ff7fca via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the store is behind a flag that is off by default, nothing reads it yet, and it deletes on the keyword index's fail-closed terms.

Notes

  • When session.semantic_search is on and session_vectors.db cannot be opened, delete_session now refuses the delete. Unlike the search-index branch beside it, the warning doesn't name the file to remove to unblock it, so the user has no remedy in hand. Naming that path in the log fixes this.
  • Removed readers I looked for: history_index._WAL_TRUNCATE_* has no reader outside the patch.

[DESIGN-REVIEWED] 77d4261

Description read

[DESCRIPTION-READ] 031a3cf9e1e8e05104e639aa0b1fddf7abbf4d418d1a3bf00066aef8dbc0eee3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 77d42610c2762fbb8cae55eef33ab5b1f2ff7fca via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the store builds only behind an off-by-default flag, phase 2 of a partial RFC that covers this shape, and search output does not change.

Notes

  • Item 6 — oversized: def _truncate_wal(self, conn) -> bool: return wal_truncated(conn) is now a one-line pass-through with 2 callers (history_index.py:658, :901). Subtraction: delete SessionSearchIndex._truncate_wal and call wal_truncated(conn) at both sites.
  • Item 6 — zero consumers: the sleep= parameter on wal_truncated has 0 production callers (3 call sites in the patch pass none; only test_sqlite_wal.py sets it). Subtraction: drop the parameter and patch sqlite_wal.time.sleep in the test.

What this change ships

Inventory (8 items) — 6 justified

Intent: let a later phase find a past session by describing it, by building chunk vectors in the background now. This is an ADDITION.

  1. New config key session.semantic_search, off by default, in settings and docs — justified
  2. With the flag on, the index worker cuts quiet, visible sessions into chunks of up to 300 model tokens and stores them in sessions/.index/session_vectors.db — justified
  3. With the flag on, the gateway embeds pending chunks with the local llama.cpp model at bulk priority, and with ollama_http embeds nothing — justified
  4. Deleting a session also erases its chunk rows, and the delete is refused when the store cannot be erased — justified
  5. Sessions made private, unreadable or out of the window lose their rows whatever the flag says — justified
  6. WAL-truncate check moved into shared sqlite_wal.wal_truncated and used by both stores — oversized: a leftover wrapper and a test-only parameter
  7. Changing the model clears old vectors and re-queues those sessions — justified
  8. Gateway log line every 10 minutes with embedded and pending counts while the build runs — undeclared: the 10-minute log is not in the Goal or the docs

[FIRST-PRINCIPLES-REVIEWED] 77d4261

Description read

[DESCRIPTION-READ] 82fedd0fcf0a32e6a0706436ee661ce227a8db3bd81820151b320416abfa9a8e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 77d42610c2762fbb8cae55eef33ab5b1f2ff7fca via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

Notes:

  • The chunk pass tokenizes inside _locked(key) (history_vectors_index.py:1280). For a very large transcript that lock could be held much longer than the "short critical section" history_locks.py assumes, but nobody has measured tokenizer throughput, so this is not shown to reach the 10 s writer timeout.
  • VectorFill.close() runs on the event-loop thread, so the connections that steps opened on executor threads stay open until the process exits. There are only a few of them, so this is cleanup, not a defect.

[OPUS-REVIEWED] 77d4261

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ human override accepted

@bolichen97 overrode this lane for 77d42610c2762fbb8cae55eef33ab5b1f2ff7fca (the recorded decision, this commit only); the model review was skipped.

[GPT-OVERRIDE] 77d4261

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dma9527. This is a careful phase-2 change. The store, its delete contract and its tests read well. One privacy gap stops me approving. It is small to fix.

Verdict: request changes (one finding).

🔴 Private and departed rows are only dropped while a tokenizer is loaded

  • Where: src/kiro_crew/history_index_worker.py:288 and :292 (_ChunkPass.run).
  • Cause: run returns early when session.semantic_search is off, or when no tokenizer is available (no model file, or a failed load). backfill_chunks is the only place that drops a session turned incognito or temporary, a session that left the window, and an owed WAL truncate (store.settle(), history_vectors_index.py:149). That code never runs on those paths.
  • Trigger: turn the flag on and let the store fill, so pending_text holds the text. Turn the flag off, or remove the model. Then make a session incognito.
  • Consequence: the session's text stays in session_vectors.db with no end date. RFC section 12 says this store "gets the FTS index's deletion and privacy contract". The FTS worker keeps dropping private sessions with no flag, so after the toggle the two contracts diverge. The person who made the session private has no sign that a copy is left.
  • Repro, run on this head with the test file's own helpers: flag on, one _ChunkPass.run, then flag off, rewrite the session with mode="incognito", then run twice. pending_text still holds ('secret words here',).
  • Fix: the drop half needs no tokenizer. Split backfill_chunks so settle(), the private drop and the departed drop run on every pass where store.store_exists() is true, whatever the flag or model state. Only the chunking loop needs the flag and a tokenizer. With no store file, keep the current behavior: a flag-off host never creates one. Add one test per early return: flag off, and no model, each after a filled store.

Checked and fine

  • Off by default: session.semantic_search defaults to false in sections_session.py, loader_builders.py, config-baseline.json and configuration.md. With the flag off on a fresh host, no file is created (store_exists gates drop, and the worker returns before opening).
  • What is stored matches the updated RFC 7.1 table: hashes, token counts, offsets, per-session file identity, and the chunk text in pending_text only until a vector exists. No other text and no titles.
  • Bounds: work is limited to the keyword index's _searchable window and search_scan_window, sessions that leave it are dropped, and each pass stops at pass_budget_secs. The worker loads the tokenizer with vocab_only=True, so no weights. The per-pass config read follows the same pattern configured_search_scan_window already uses.
  • Incognito and temporary on the flag-on path: dropped from the listing before the quiet window, every read goes through _withhold_if_restricted under _locked(key), and TranscriptBusy drops the rows.
  • Deletion: delete_session calls vector_store.drop(identities) under the same lock and refuses the delete when a store exists but cannot be erased. secure_delete=ON plus a checked wal_checkpoint(TRUNCATE), and orphan chunks go while shared ones stay. The sync runs under the session lock, so a delete cannot race a half-written sync.

Local run on 3efc43e3a8: test/test_history_chunks.py and test/test_history_vectors_index.py gave 38 passed. test/test_history_index_worker.py and test/test_config_baseline.py gave 52 passed. flake8 and mypy are clean on the changed source.

After the fix, ping here and I'll re-review the new head.

AI-assisted, sent by Bolin Review bot

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@dma9527
dma9527 force-pushed the feat/session-chunk-store branch from 3efc43e to 41255ef Compare November 16, 2026 21:57
@dma9527 dma9527 changed the title feat(history): build a session chunk store behind session.semantic_search feat(history): build and fill a session chunk store behind session.semantic_search Nov 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 16, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 41255efaba (de850e8 + 41255ef).

Earlier blocker: fixed. backfill_chunks now prunes private and departed sessions whenever a store file exists, and runs chunking only with a tokenizer (history_vectors_index.py:168-187). _ChunkPass.run calls it every round whatever the flag (history_index_worker.py:288-300). test_a_filled_store_still_drops_private_and_departed_sessions[flag-off|no-model] fills the store, turns the switch off (or removes the model), makes a session incognito and asserts pending_text is empty. That is the case I asked for. Thanks.

The vector-fill commit is in scope. RFC section 7.1 puts the fill in the gateway at PRIORITY_BULK, and phase 2's exit criterion ("the fixture indexes to coverage 1.0") needs vectors. It stays behind session.semantic_search (off), never creates the store, paces 8 texts per step at bulk priority, and the coverage counts leave private sessions out. Keeping it in this PR is fine.

Two things in the new commit need a change before merge:

1. A reused chunk_id gets the vector of a dropped chunk (blocker)

  • src/kiro_crew/history_vectors.py:40: chunk_id INTEGER PRIMARY KEY without AUTOINCREMENT, so SQLite hands a dropped top id to the next insert.
  • src/kiro_crew/history_vectors.py:194-213: write_vectors matches only on chunk_id.
  • Trigger: the fill reads a batch, then blocks in embed_batch (seconds per row at bulk priority). Meanwhile the worker process drops a session (made incognito, or deleted) whose chunk holds the top id, and in the same pass chunks another session. The new chunk gets the same id. The fill then writes the dropped text's vector onto it and deletes the new chunk's pending text.
  • Consequence: a vector derived from an incognito or deleted session stays in the store after the drop returned, which breaks the section 12 deletion contract (a vector can leak its text). The other session keeps a wrong vector and never gets its own until a model change.
  • Reproduced at this head, using the helpers in test/test_history_vectors_fill.py:
def test_turned_private_mid_embed(tmp_path, space):
    _write(tmp_path, "secret", "secret words")
    _chunk(tmp_path)
    store = chunking.vector_store_for(tmp_path)

    def during():
        _write(tmp_path, "secret", "secret words", mode="incognito")
        _write(tmp_path, "other", "other words")
        _chunk(tmp_path)

    space["embedder"].during = during
    fill.fill_batch(store)
    # chunk 1 now: chunk_ref ('other', 1), vector == embedding of "secret words",
    # pending_text empty
  • Fix (either one): declare chunk_id INTEGER PRIMARY KEY AUTOINCREMENT (the store is new in this PR, so no migration), or make write_vectors claim the pending row by content: DELETE FROM pending_text WHERE chunk_id=? AND text=? and update chunk only when that deleted a row. Please add the test above with asserts that no row holds the secret's vector and that other is still pending.

2. The fill can send transcript text to a remote Ollama server

  • src/kiro_crew/history_vectors_fill.py:65-67 embeds through get_shared_embedder(). With memory.embedding_provider = "ollama_http" and memory.ollama_allow_remote, that backend posts the text to a non-loopback host.
  • ollama_allow_remote's help only discloses that "memory and knowledge text" leaves the machine. RFC section 12 says "Embedding runs on the local model, so no transcript text leaves the host", and the new session.semantic_search help (sections_session.py:258-260, configuration.md) says "the local embedding model". With both switches on, whole transcripts go off-host, which no setting tells the operator.
  • It also mismatches the chunker: chunks are cut with the local GGUF tokenizer, and vectors would come from a different model.
  • Fix: make VectorFill.step (and fill_batch) run only when the active provider is the in-process one (positive check, embedding_provider == "llama_cpp"), with a test that an ollama_http provider fills nothing. Disclosing it in the help text instead is also acceptable if you prefer, but then the RFC's section 12 line needs to change in this PR too.

Checks run here

  • pytest on test_history_chunks.py, test_history_vectors_index.py, test_history_vectors_fill.py, test_dashboard_server_composition_contract.py, test_history_index_worker*.py: 133 passed.
  • flake8 on the changed src and tests: clean. mypy on the 10 changed src files: no issues. check_file_line_cap.py: 0 failures.
  • Internal Content Scan is red with "NO VERDICT, the scan did not run to completion". That is the runner, not this diff; it will need a rerun.

Note, not blocking: when backfill_index raises, run_index_worker skips chunks.run for that round (history_index_worker.py:236-243), so the privacy prune waits for the keyword pass to recover. Running the prune before the keyword pass, or in its except branch too, would remove that dependency.

Once 1 and 2 are fixed this is ready from my side.

AI-assisted, sent by Bolin Review bot

@dma9527

dma9527 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 thanks for the repro. Fixed in de850e8: with no tokenizer (flag off or no model) backfill_chunks now runs only the prune half, settle() plus the private and departed drops, whenever a store file exists; with no store file nothing runs, so a flag-off host still never gets one. test_a_filled_store_still_drops_private_and_departed_sessions[flag-off|no-model] covers both early returns, and your repro now prints []. The PR also carries 41255ef, the gateway vector fill, so pending_text has its consumer. The RFC draft finding clears once #35435 merges; I will rebase then. Could you approve the held workflows and re-review the new head?

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@dma9527
dma9527 force-pushed the feat/session-chunk-store branch from 41255ef to 69fe866 Compare November 17, 2026 00:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@dma9527

dma9527 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 both fixed in 69fe866:

  1. chunk_id is AUTOINCREMENT. Your test is in as test_a_session_turned_private_mid_embed_leaves_no_vector: no row holds a vector and other is still pending.
  2. The fill runs only when the shared embedder is the in-process LlamaCppEmbedder (embeds_on_this_host). test_a_remote_backend_fills_nothing covers ollama_http with ollama_allow_remote.

From the bot reviews: a model change now keeps released sessions in the prune inventory, and both stores use one WAL check, sqlite_wal.wal_truncated. Rebased on 5ff3fed, which fixes the two red Backend Tests shards (they fail on the old base).

On the non-blocking note: when backfill_index raises, the keyword index's own private drop stalls too, so the two stores stay in step. I left it as is.

Could you approve the held CI and re-review?

@bolichen97
bolichen97 enabled auto-merge (squash) November 17, 2026 22:37
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

@dma9527 this now conflicts with main, so auto merge can't land it. Could you rebase onto current main? My approval is on 8e950f3; I'll check the delta and re-approve on the new head.

AI-assisted, sent by Bolin Review bot

…arch

Phase 2a of the session search RFC. While the flag is on (default off), the
session index worker cuts quiet sessions into chunks of at most 300 tokens of
the embedding model's tokenizer, loaded vocabulary-only, and stores them in
sessions/.index/session_vectors.db. Sessions are read through the derivation
guard, so incognito and temporary sessions are never chunked. delete_session
drops the store under the keyword index's contract and refuses the delete when
it cannot. Search does not read the store.

Part of kirodotdev#24476
Phase 2b of the session search RFC. While session.semantic_search is on, the
gateway embeds the chunk store's pending text with the shared embedding model
at bulk priority, writing each vector and deleting its text in one transaction.
A batch is discarded when the model is not ready or its vector space changed;
vectors from an old space are cleared and their sessions re-chunked. Coverage
counts the visible search window only, so private sessions never move it.
Search does not read the store.

Part of kirodotdev#24476
auto-merge was automatically disabled November 18, 2026 19:16

Head branch was pushed to by a user without write access

@dma9527
dma9527 force-pushed the feat/session-chunk-store branch from 8e950f3 to 77d4261 Compare November 18, 2026 19:16
@dma9527

dma9527 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 rebased on bc8c588 in 77d4261.

  • Conflict: test/test_dashboard_server_composition_contract.py. Main replaced the _LAZY_IMPORTS table with a scan, so the line this PR added is gone and the file is main's. The scan passes on our lazy import in maintenance.py.
  • GPT's fenced finding (unreadable header with the flag off): fixed in the same push so it does not cost another round. backfill_chunks now also drops held window sessions whose first line cannot be read (get_metadata_status(key)[1] is False), before the no-tokenizer return. The listing reads those as persistent, so the private set never held them. A readable line comes from the metadata cache, so the cost is one stat per held session. test_a_damaged_first_line_is_pruned_with_no_tokenizer covers a non-JSON and a non-UTF-8 line, and goes red without the fix.

Before pushing I ran the full backend suite the way CI does: 51 local failures, the same 51 test ids as on clean main (this host has no user namespaces for the sandbox). Pre-push gates, base gates and the 2a commit on its own are green. Could you approve the held CI and re-approve?

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving 77d4261. The delta from 8e950f3 tightens privacy: the chunk pass also drops the rows of a session whose first line cannot be read (the listing reads those as persistent), with the spec updated to match.

AI-assisted, sent by Bolin Review bot

@bolichen97
bolichen97 enabled auto-merge (squash) November 18, 2026 19:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Maintainer decision on the two fenced GPT 6.1 findings on 77d4261: risk accepted (Bolin Chen, @bolichen97).

  • F1 (history_vectors_fill.py:118, a vector written for a session that turned incognito during one embed batch): accepted. It needs a privacy flip inside a single batch's few-second window. The vector stays on this host, the same session's pending_text already sits under the same per-pass prune rule, and that prune (history_vectors_index.py:246) removes the vector on the next worker pass. So it adds no exposure beyond what the pending text already has, and it lives no longer.
  • F2 (history_vectors.py:88, erasure debt lost across a worker restart): accepted. It needs a reader to outlast three TRUNCATE tries at the 10 s busy timeout on autocommit-only connections, then a worker restart before the next settle(), and no later drop. The leftover bytes stay in the owner-only local -wal until the next truncate or until the gateway's last connection closes.

The feature ships behind session.semantic_search, off by default. If either path shows up in practice, a follow-up should persist the erasure debt (F2) and revalidate holders before write_vectors (F1).

AI-assisted, sent by Bolin Review bot on Bolin's instruction

@bolichen97

Copy link
Copy Markdown
Contributor

/ai-review override gpt 77d4261 owner-decision: Bolin Chen (maintainer) accepted both fenced risks after review: narrow windows, data stays on host, next prune or truncate clears it; feature is off by default. Decision: #36002 (comment)

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 recorded a owner-decision release (class owner-decision) for the gpt AI finding on 77d42610c2762fbb8cae55eef33ab5b1f2ff7fca.

Bolin Chen (maintainer) accepted both fenced risks after review: narrow windows, data stays on host, next prune or truncate clears it; feature is off by default. Decision: #36002 (comment)

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026
@bolichen97
bolichen97 merged commit a91af96 into kirodotdev:main Nov 19, 2026
109 of 112 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
dma9527 added a commit to dma9527/KiroCrew that referenced this pull request Nov 19, 2026
Phase 2 of the session search RFC, the chunk store and its gateway vector
fill behind session.semantic_search, merged as kirodotdev#36002. List it in
implementation-prs, name it in the status line and the index row, and move
the audit stamps to the commit the record was checked against.

Record the answers to open questions 2 and 5, which phase 3 needs first,
from a fresh blind query set scored once against pre-registered rules: the
literal route stays on the whole query, and fires only for a query of at
least 12 characters.

Part of kirodotdev#24476
dma9527 added a commit to dma9527/KiroCrew that referenced this pull request Nov 19, 2026
Phase 2 of the session search RFC, the chunk store and its gateway vector
fill behind session.semantic_search, merged as kirodotdev#36002. List it in
implementation-prs, name it in the status line and the index row, and move
the audit stamps to the commit the record was checked against.

Record the answers to open questions 2 and 5, which phase 3 needs first,
from a fresh blind query set scored once against pre-registered rules: the
literal route stays on the whole query, and fires only for a query of at
least 12 characters.

Part of kirodotdev#24476
bolichen97 pushed a commit that referenced this pull request Nov 19, 2026
…#39386)

Phase 2 of the session search RFC, the chunk store and its gateway vector
fill behind session.semantic_search, merged as #36002. List it in
implementation-prs, name it in the status line and the index row, and move
the audit stamps to the commit the record was checked against.

Record the answers to open questions 2 and 5, which phase 3 needs first,
from a fresh blind query set scored once against pre-registered rules: the
literal route stays on the whole query, and fires only for a query of at
least 12 characters.

Part of #24476

Co-authored-by: dma9527 <221155142+dma9527@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants