Skip to content

fix(knowledge): sweep interrupted-ingest residue in the agent aggregate source - #17504

Closed
chenmingwei23 wants to merge 1 commit into
kirodotdev:mainfrom
chenmingwei23:fix/2670-agent-residue-sweep
Closed

chenmingwei23 wants to merge 1 commit into
kirodotdev:mainfrom
chenmingwei23:fix/2670-agent-residue-sweep

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: After a hard kill during an agent-document ingest, re-adding the same document replaces the old copy instead of silently duplicating it.

An agent-document add commits its item chunks, the previous group's deletion and its agent_item_state ownership row as three separate autocommit transactions. A plain cancellation is already handled — the ownership row is written from inside the ingest's uncancellable finalize hop (_record_ownership). But a hard kill between the item commit and the row commit still leaves committed items that no state row names. The replacement path keys off that row, so the next add of the same document stores a second copy instead of replacing the first. Nothing on the agent path reaps by absence (there is no reconcile pass, unlike the folder watcher and reconcile_artifacts), so the residue is permanent.

This is the remaining half of the originating report; the bundle round-trip half already shipped, and this PR adds the residue repair that half made safe.

Why it matters

A user who interrupts an ingest (process crash, OOM kill, machine power-off) and then re-adds the same document ends up with duplicate items in their knowledge base, with no way to clear them — the duplicate persists across every subsequent add. Left undone, the aggregate agent source accumulates unreclaimable residue for every interrupted ingest.

Not a goal

  • The artifact-source crash window is deliberately not repaired here. Absence of a row in the artifact source is not decidable as residue (a bundle import legitimately arrives unowned there), so repairing it needs a different, store-decidable signal and is tracked separately.
  • No change to the folder watcher or reconcile_artifacts reaping paths.
  • No change to the bundle export/import ownership tables.

What changed (motivation → approach → change)

Symptom: duplicate agent-source items survive a re-add. Root cause: committed items with no agent_item_state row after a hard kill, which nothing reaps. The change adds a sweep that removes exactly that residue, scoped so it can never delete a legitimately-unowned item.

  • KnowledgeStore.reclaim_agent_source_residue(): deletes active items in the agent:// aggregate source that no agent_item_state row names. Co-held items (a dedup co-location) are detached to the surviving holder, not destroyed. The sweep runs in chunked BEGIN IMMEDIATE transactions that re-derive ownership under the writer lock, so a row written between the candidate read and the delete still protects its items.
  • Wired into _kick_knowledge_orphan_reclaim, inside the same drained maintenance_window as reclaim_orphans, so it only ever sees fully-settled ingests (never an item mid-ingest, committed but not yet named by its finalize hop).

Scope rationale (agent:// alone): the sweep keys on "no state row names this item", a safe residue test only where every legitimate item is owned. A bundle import breaks that elsewhere — _BUNDLE_STATE_RESTORED_TABLES is {agent_item_state}, so a bundle restores an agent document's ownership row but deliberately carries no folder_file_state / artifact_item_state row (those two tables are reaped by absence by their owners, so importing one would be an order to delete the items the import just brought). Folder- and artifact-backed items therefore arrive unowned by design, and a sweep keyed on absence would delete every one a bundle ever imported. For the agent aggregate alone the ambiguity does not exist: a bundle brings the owning row with the items, so an unowned agent-source item is provably crash residue and never a legitimate import — the owner-provable subset the design-review constraint on this issue asked for.

Fail-safe choices:

  • An unreadable item_ids on any row stands the whole sweep down for that source (returns 0): the owned set would be missing genuinely-owned items, and a wrongly-deleted item is unrecoverable where a stale one is not.
  • A deduped marker row owns an empty group; its items are named by the winner's row, so the winner is not mistaken for residue.

Backwards compatibility

Removes nothing: the change adds a new maintenance-window sweep and a new public method; it rejects, renames and removes no existing input, key, or capability. The only deletions it performs are of provable crash residue that no code path references.

Tests

  • test/test_agent_source_residue_sweep.py (new): residue is removed and a re-add then stores one copy; a fully-owned store is untouched; deduped markers, unreadable groups, and co-location detach are all covered; artifact and folder sources are never touched; an imported (owned) agent document survives.
  • test/test_knowledge_reclaim_deferred.py: pins that the kicker runs the residue sweep in the same drained window as reclaim_orphans.

Locally ran the files that directly test the changed code (residue sweep, reclaim/maintenance wiring, agent/bundle ownership, dedup/co-location, artifact reconcile, composition contract) with -n 2: 499 passed; flake8, isort and mypy clean on the changed modules. CI runs the cross-surface set.

Manual verification

N/A — unit coverage sufficient; the crash is simulated by deleting the state row directly (the issue's own reproducer) and the sweep's effect is asserted on the store.

Related Issues

Fixes #2670

Pattern harvest

Rule candidate: review-prompt
Pattern: a reconcile/reclaim sweep that deletes by "no state row names this item" is safe only where every legitimate item is owned; where a legitimate item can arrive unowned (e.g. bundle import), absence is not a residue signal and the sweep must scope to the owner-provable subset.

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 6, 2026 19:39
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 6, 2026
…te source

A hard kill between an agent-document ingest's item commit and its agent_item_state ownership-row commit leaves committed items no row names. Nothing on the agent path reaps by absence, so the next add of that document duplicates instead of replacing it (issue kirodotdev#2670, first half). Add KnowledgeStore.reclaim_agent_source_residue(), run in the same drained maintenance window as reclaim_orphans, scoped to the agent:// source alone: a bundle restores agent_item_state (so imported agent docs are owned) but not folder_file_state/artifact_item_state, so an unowned agent-source item is provable crash residue. Co-held items are detached, not destroyed; an unreadable group stands the sweep down. Fixes kirodotdev#2670.
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from acb2c8b to 6af41d9 Compare October 6, 2026 22:28
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Superseded by #18063, which carries the same fix as a same-repo draft rebased onto current main, plus the failing-on-main reproduction test. Closing this one in favour of #18063.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Knowledge: interrupted artifact ingest leaves duplicate items, and bundle import loses ownership metadata

1 participant