Repository navigation
fix(knowledge): sweep interrupted-ingest residue in the agent aggregate source - #18063
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS A same-content re-add that runs before the sweep still leaves a permanent duplicate and a marker that never clears, and the "self-healing" claim does not hold. Watch
[DESIGN-REVIEWED] 0f74f26 Description read[DESCRIPTION-READ] 67bc6651b51493d40de4282e34135c15bf72621bb40ea3ad29795d7678baf388 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (9 items) — 6 justifiedIntent: stop a hard kill during an agent-document add from leaving a duplicate that can never be removed. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 0f74f26 Description read[DESCRIPTION-READ] 268a2046e1187f98ff3bc6f938b660f45841f0f65e1edc3a07ec4e50d79cb6c7 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
c1adb48 to
83d1b6f
Compare
83d1b6f to
24dc98d
Compare
24dc98d to
fc78567
Compare
fc78567 to
9eec01a
Compare
9eec01a to
7f7272d
Compare
7f7272d to
815664a
Compare
|
Both findings on Root cause. The sweep's shield excluded a Fix. The exclusion is removed. The shield is now unconditional: if any healthy or F1 ( Opus ( Tests pin both: a sole copy at the same slug as its marker survives, the empty-group |
|
This fixes a re-added agent document being stored twice after a hard kill lands between the item commit and the agent_item_state ownership commit, by writing a pre-commit intent marker and adding a maintenance-window sweep (reclaim_agent_source_residue) that deletes unowned agent:// items only when a stale marker names their hash. I read the diff and the lane comments on head 815664a. GPT and Opus have no findings, and the author's replies close the earlier F1-F3 shield and timestamp findings in code. HOLD for three reasons. First, it adds a new path that deletes user data (~650 prod lines in store.py and agent_source.py). Second, the description describes a different design from the one that ships: it says the marker is an |
…attempt The maintenance sweep reaps an unowned agent item only on positive intent evidence: a stale agent_ingest_intent marker naming its content hash AND the item created at or after that marker's started_at. Tying the proof to the attempt in time means a pre-existing or bundle-imported item that merely shares a hash (created before any crash) is never residue and never deleted. - store.py: _ingesting_hashes returns hash -> earliest started_at; the candidate read, the under-lock recheck and marker retirement all require created_at >= started_at, so a pre-existing copy neither gets swept nor pins a spent marker open. - store.py: the under-lock recheck filters status = 'active', so an item a concurrent writer flips out of active in the read-to-lock gap is not swept. - agent_source.py: the budget-refusal, exception and non-completed cleanup paths clear the marker with preserve_if_orphans, so a same-content retry refused after an earlier crash no longer erases that crash's recovery evidence; the preserve check is attempt-scoped too. Tests: a pre-existing/bundle item sharing a hash survives; a spent marker is retired despite a pre-existing same-hash item; a non-active marker-named item is left alone; the same-slug retry fixture models the real marker-before-items crash order.
815664a to
0f74f26
Compare
|
Thanks — all three are addressed on head 1. Description and Backwards-compatibility rewritten to match the shipped code. The description now says the marker is a new permanent 2. Design and First Principles answered on this head (separate replies on each lane), including the root-cause question — why the items and the ownership row can't share one 3. Duplicated owned-id derivation removed. The owned-id query/parse/fail-safe now lives in one place, Gate green (flake8 / isort / mypy / baselined-black / comment-history); 61 touched tests pass. Rebased onto current |
|
On the root-cause question — why not commit the items and the ownership row in one The item commits and the ownership write are not adjacent — a per-chunk embedding step sits between them. The two writes are on different surfaces. The chunk So the fix reaps on positive evidence instead (a pre-commit intent marker + a maintenance-window sweep), which keeps the writer lock short and the pipeline generic. The description's "Why not one transaction" section now records this. On the "stays duplicated forever by design" watch item: the same-content retry case is bounded — the marker keeps the earliest |
|
Both items addressed on head Item 2 — undeclared Item 6 — duplicate owned-id derivation. Subtracted as you proposed: deleted the copy in Gate green; 61 touched tests pass. Rebased onto current |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Deep review passed: fix (5 files). Adds an agent_ingest_intent marker table and a maintenance-window sweep that deletes only crash residue with positive evidence (marker hash plus created_at after start, live-owned rows shielded, unreadable groups fail closed). The author answered every lane concern on this head; the one known residual (a same-content re-add before the sweep) is declared, not a regression. Approved at a maintainer's direction.
|
Reviewed after merge. The sweep is idempotent, stays inside the agent source, and spares completed adds. One window remains where it can delete the only remaining copy of a document: when a replace crashes between the old-group delete and the new ownership-row write, which are two separate commits. Tracked with acceptance criteria in #18423. This was traced from source; I have not reproduced it with a probe. |
Problem / Motivation
Goal: stop a re-added agent document from being silently stored twice after a hard kill interrupts its ingest.
An agent-document add commits its item chunks, the previous group's deletion, and its
agent_item_stateownership row as separate autocommit transactions. A plain cancellation is already handled — the ownership row is written from inside the ingest's uncancellable finalize hop. But a hard kill between the first item commit and the ownership-row commit leaves committed items that no state row names. The replacement path keys off that row, so the next add of the same document stores a second copy instead of replacing the first, and nothing on the agent path reaps that residue — so the duplicate persists across every later add with no way to clear it.Why it matters
Anyone whose gateway is hard-killed (OOM, power loss,
kill -9) mid-ingest of an agent document ends up with a permanently duplicated document in their knowledge library: every search over it returns both copies, and re-adding never repairs it. The content is silently wrong with no user-visible way to fix it.Not a goal
Why not one transaction (the obvious smaller fix)
The root cause is a non-atomic write: the items and the ownership row commit separately. The obvious subtraction is to commit them in one
BEGIN IMMEDIATE, which would remove the sweep entirely. I investigated this and it is not feasible for this path, for two concrete reasons:The item commits and the ownership write are not adjacent — an embedding step sits between them, per chunk.
IngestionPipeline._ingest_file_bodycommits each chunk'sitemsrow in its ownadd_itemtransaction inside a loop, and between chunks calls_embed_item, which is anawaiton an executor-run embedder behind an async rate limiter — a network/CPU step that straddles the event loop. The ownership row is written only in the final_finalizehop, after the whole loop and afterdelete_items_batch(which rebuilds the entity graph). A singleBEGIN IMMEDIATEspanning items + ownership would have to hold SQLite's writer lock across every per-chunk embedding call and the graph rebuild — seconds of held lock that block every other knowledge writer (and trip the loop-stall watchdog the pipeline is built to avoid). The chunked-hop design exists precisely so the writer lock is never held across an embed.The ownership state lives on a different write surface than the chunk items. The chunk
itemsare written by the sharedIngestionPipeline(so every source gets the same chunker/extractor/embedder), while the agent aggregate's ownership (agent_item_state) is an agent-source concept the pipeline does not know about; the pipeline only calls back throughon_committed. Folding the ownership write into the pipeline's own transaction would push an agent-source-specific table into the generic ingest path.So the fix reaps on positive evidence of an interrupted ingest instead: a pre-commit intent marker plus a maintenance-window sweep. This keeps the writer lock short and leaves the pipeline generic.
What changed (motivation → approach → change)
Symptom: a re-add after a hard kill stores a second copy. Root cause: the ownership row that the replacement path keys off is never written, so the committed items are orphaned and the next add cannot find them to replace.
Approach considered and rejected: reap by absence — delete any
activeagent://item that noagent_item_staterow names. This is unsafe:_import_bundle_stateleaves a bundle's agent items unowned on several legitimate paths (a row skipped asownership_row_key_held_locally, an empty or over-claiming group, a bundle with no state tables), and a single-itemexport_item→import_bundleround trip writes no state row either. Absence is therefore not proof of crash residue, and reaping by absence would permanently delete imported content.Approach taken — reap only on positive evidence of an interrupted ingest:
agent_ingest_intent(CREATE TABLE IF NOT EXISTS, primary key(source_id, slug, content_hash), carryingstarted_at), is written byadd_agent_documentbefore it commits any item, throughmark_ingesting. It is a separate table, not a status value onagent_item_state: the marker must coexist with a live ownership row, because a re-add that crashes while replacing a live group must leave a marker exactly as a first add does. The marker is written on every add that reaches the ingest (it does not depend on whether a live group already owns the slug); the finalize hop deletes it viaclear_ingesting, and every refusal/error exit clears it too, so only a hard kill leaves one behind. A same-content retry at the same slug shares the one marker row and keepsstarted_atat the earliest attempt (ON CONFLICT … DO UPDATE SET started_at = MIN(…)), so an earlier crash's evidence is not lost.add_agent_documentnow calls_normalize_newlines(text)(CRLF / bare CR → LF) before computingcontent_hashand writing the temp file, so the marker hash equals the hash the pipeline stamps on each chunk (the file reader does the same normalization). See the compatibility note below for the effect on documents stored before this change.KnowledgeStore.reclaim_agent_source_residue()deletes an unownedactiveagent://item only when a stale marker names itscontent_hashand the item was created at/after that marker'sstarted_at(items.created_at >= started_at) — positive, attempt-scoped proof an ingest for exactly that content started and never finished. An imported item has no such marker, and a pre-existing item created before the marker is excluded by the timestamp, so neither is ever a candidate. A hash claimed by any healthy ordedupedownership row is shielded, so a sole surviving copy is never deleted on bare hash evidence. An unreadableitem_idson any row stands the whole sweep down (a wrongly-deleted item is unrecoverable where a stale one is not). The sweep runs in chunkedBEGIN IMMEDIATEtransactions that re-derive ownership and the marker set under the writer lock, so an ingest that finalized in the gap protects its items.KnowledgeStore.agent_owned_item_ids; both the sweep andclear_ingestingcall it._kick_knowledge_orphan_reclaim, inside the same drainedmaintenance_windowasreclaim_orphans, so it only ever sees fully-settled ingests.Backwards compatibility
Compatible: adds one maintenance-window sweep, one public store method, one new table, and one pre-commit marker write; it rejects, renames and removes no existing input, key, or capability.
agent_ingest_intentis a new permanent table created withCREATE TABLE IF NOT EXISTSon store init; it is additive and touches no existing table. No migration of existing rows is needed._normalize_newlineschanges thecontent_hashof any already-stored document that contains a carriage return (\r\nor bare\r). On that document's next add, the "unchanged since last add" shortcut and thefind_document_by_hashdedup both compare the new (normalized) hash against the old (un-normalized) stored hash and miss. The effect is a one-time re-ingest (replace), not a duplicate:get_statereturns the previous group by slug (the slug is derived fromsource_uri, not the hash), so the add proceeds down the normal replace path andingest_file's finalize deletes the old group (delete_items_batch(old_item_ids)) before recording the new one. After that one re-ingest the stored hash is the normalized value and the shortcut hits again. No duplicate is produced, so no extra handling is required; a CR-only document simply re-chunks once on its next add. A document with no carriage returns is unaffected (its hash is unchanged).The diff has a take-away shape (the sweep deletes items), so the readers of a swept item:
src/kiro_crew/knowledge/store.py:reclaim_agent_source_residue— crew page / cron (maintenance window) — only deletes an unownedactiveagent://item whose hash a stale marker names and that was created at/after the marker; proven by the imported-unowned-survives, sole-copy-survives, and real-residue-removed tests.src/kiro_crew/knowledge/agent_source.py:get_state— chat (knowledge.agent_document.add) — readsitem_idsfromagent_item_state; the marker lives in a separate table and never changes whatget_statereturns, so the re-add path behaves as before.src/kiro_crew/knowledge/agent_source.py:find_document_by_hash— chat — filtersagent_item_state.status = 'active', so an intent marker (a different table) never falsely refuses a re-add.Tests
test/test_agent_source_residue_sweep.py(new): real crash residue (items + a stranded intent marker) is removed and a re-add then stores one copy; a bundle-imported unowned item survives; an owned item sharing a crashed hash is shielded while only the unowned orphan is swept; a sole copy at the same slug as its marker survives; a pre-existing item created before the marker survives; the earlieststarted_atis kept across a same-content retry; the add holds the ingestion gate before writing the intent;dedupedmarkers, unreadable-group fail-safe, and co-location detach covered; artifact and folder sources never touched.test/test_knowledge_reclaim_deferred.py: pins that the kicker runs the residue sweep in the same drained window asreclaim_orphans.Manual verification
N/A — unit coverage sufficient: the sweep and its fail-safe boundaries are exercised directly, including the imported-unowned-survives case that proves no legitimate import is deleted.
Related Issues
Fixes #2670
Supersedes #17504.
Pattern harvest
Rule candidate: review-prompt
Pattern: a reaper that infers "crash residue" from the ABSENCE of a tracking row — safe only if absence is unreachable by any legitimate path; prefer a positive pre-commit intent marker over reaping by absence whenever an import or restore path can also produce the absent state.