Repository navigation
fix(knowledge): sweep interrupted-ingest residue in the agent aggregate source - #17504
Closed
chenmingwei23 wants to merge 1 commit into
Closed
chenmingwei23 wants to merge 1 commit into
chenmingwei23 wants to merge 1 commit into
Conversation
…te source A hard kill between an agent-document ingest's item commit and its agent_item_state ownership-row commit leaves committed items no row names. Nothing on the agent path reaps by absence, so the next add of that document duplicates instead of replacing it (issue kirodotdev#2670, first half). Add KnowledgeStore.reclaim_agent_source_residue(), run in the same drained maintenance window as reclaim_orphans, scoped to the agent:// source alone: a bundle restores agent_item_state (so imported agent docs are owned) but not folder_file_state/artifact_item_state, so an unowned agent-source item is provable crash residue. Co-held items are detached, not destroyed; an unreadable group stands the sweep down. Fixes kirodotdev#2670.
chenmingwei23
force-pushed
the
fix/2670-agent-residue-sweep
branch
from
October 6, 2026 22:28
acb2c8b to
6af41d9
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem / Motivation
Goal: After a hard kill during an agent-document ingest, re-adding the same document replaces the old copy instead of silently duplicating it.
An agent-document add commits its item chunks, the previous group's deletion and its
agent_item_stateownership row as three separate autocommit transactions. A plain cancellation is already handled — the ownership row is written from inside the ingest's uncancellable finalize hop (_record_ownership). But a hard kill between the item commit and the row commit still leaves committed items that no state row names. The replacement path keys off that row, so the next add of the same document stores a second copy instead of replacing the first. Nothing on the agent path reaps by absence (there is no reconcile pass, unlike the folder watcher andreconcile_artifacts), so the residue is permanent.This is the remaining half of the originating report; the bundle round-trip half already shipped, and this PR adds the residue repair that half made safe.
Why it matters
A user who interrupts an ingest (process crash, OOM kill, machine power-off) and then re-adds the same document ends up with duplicate items in their knowledge base, with no way to clear them — the duplicate persists across every subsequent add. Left undone, the aggregate agent source accumulates unreclaimable residue for every interrupted ingest.
Not a goal
reconcile_artifactsreaping paths.What changed (motivation → approach → change)
Symptom: duplicate agent-source items survive a re-add. Root cause: committed items with no
agent_item_staterow after a hard kill, which nothing reaps. The change adds a sweep that removes exactly that residue, scoped so it can never delete a legitimately-unowned item.KnowledgeStore.reclaim_agent_source_residue(): deletesactiveitems in theagent://aggregate source that noagent_item_staterow names. Co-held items (a dedup co-location) are detached to the surviving holder, not destroyed. The sweep runs in chunkedBEGIN IMMEDIATEtransactions that re-derive ownership under the writer lock, so a row written between the candidate read and the delete still protects its items._kick_knowledge_orphan_reclaim, inside the same drainedmaintenance_windowasreclaim_orphans, so it only ever sees fully-settled ingests (never an item mid-ingest, committed but not yet named by its finalize hop).Scope rationale (
agent://alone): the sweep keys on "no state row names this item", a safe residue test only where every legitimate item is owned. A bundle import breaks that elsewhere —_BUNDLE_STATE_RESTORED_TABLESis{agent_item_state}, so a bundle restores an agent document's ownership row but deliberately carries nofolder_file_state/artifact_item_staterow (those two tables are reaped by absence by their owners, so importing one would be an order to delete the items the import just brought). Folder- and artifact-backed items therefore arrive unowned by design, and a sweep keyed on absence would delete every one a bundle ever imported. For the agent aggregate alone the ambiguity does not exist: a bundle brings the owning row with the items, so an unowned agent-source item is provably crash residue and never a legitimate import — the owner-provable subset the design-review constraint on this issue asked for.Fail-safe choices:
item_idson any row stands the whole sweep down for that source (returns 0): the owned set would be missing genuinely-owned items, and a wrongly-deleted item is unrecoverable where a stale one is not.dedupedmarker row owns an empty group; its items are named by the winner's row, so the winner is not mistaken for residue.Backwards compatibility
Removes nothing: the change adds a new maintenance-window sweep and a new public method; it rejects, renames and removes no existing input, key, or capability. The only deletions it performs are of provable crash residue that no code path references.
Tests
test/test_agent_source_residue_sweep.py(new): residue is removed and a re-add then stores one copy; a fully-owned store is untouched;dedupedmarkers, unreadable groups, and co-location detach are all covered; artifact and folder sources are never touched; an imported (owned) agent document survives.test/test_knowledge_reclaim_deferred.py: pins that the kicker runs the residue sweep in the same drained window asreclaim_orphans.Locally ran the files that directly test the changed code (residue sweep, reclaim/maintenance wiring, agent/bundle ownership, dedup/co-location, artifact reconcile, composition contract) with
-n 2: 499 passed; flake8, isort and mypy clean on the changed modules. CI runs the cross-surface set.Manual verification
N/A — unit coverage sufficient; the crash is simulated by deleting the state row directly (the issue's own reproducer) and the sweep's effect is asserted on the store.
Related Issues
Fixes #2670
Pattern harvest
Rule candidate: review-prompt
Pattern: a reconcile/reclaim sweep that deletes by "no state row names this item" is safe only where every legitimate item is owned; where a legitimate item can arrive unowned (e.g. bundle import), absence is not a residue signal and the sweep must scope to the owner-provable subset.