Skip to content

fix(knowledge): sweep interrupted-ingest residue in the agent aggregate source - #18063

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/2670-agent-residue-sweep
Oct 9, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/2670-agent-residue-sweep

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: stop a re-added agent document from being silently stored twice after a hard kill interrupts its ingest.

An agent-document add commits its item chunks, the previous group's deletion, and its agent_item_state ownership row as separate autocommit transactions. A plain cancellation is already handled — the ownership row is written from inside the ingest's uncancellable finalize hop. But a hard kill between the first item commit and the ownership-row commit leaves committed items that no state row names. The replacement path keys off that row, so the next add of the same document stores a second copy instead of replacing the first, and nothing on the agent path reaps that residue — so the duplicate persists across every later add with no way to clear it.

Why it matters

Anyone whose gateway is hard-killed (OOM, power loss, kill -9) mid-ingest of an agent document ends up with a permanently duplicated document in their knowledge library: every search over it returns both copies, and re-adding never repairs it. The content is silently wrong with no user-visible way to fix it.

Not a goal

  • Does not change how a successful ingest, a cancellation, or a bundle import behaves.
  • Does not add a reconcile pass for the folder or artifact sources — only the agent aggregate's crash residue is in scope.
  • Does not touch the duplicate-detection or dedup logic itself.

Why not one transaction (the obvious smaller fix)

The root cause is a non-atomic write: the items and the ownership row commit separately. The obvious subtraction is to commit them in one BEGIN IMMEDIATE, which would remove the sweep entirely. I investigated this and it is not feasible for this path, for two concrete reasons:

  1. The item commits and the ownership write are not adjacent — an embedding step sits between them, per chunk. IngestionPipeline._ingest_file_body commits each chunk's items row in its own add_item transaction inside a loop, and between chunks calls _embed_item, which is an await on an executor-run embedder behind an async rate limiter — a network/CPU step that straddles the event loop. The ownership row is written only in the final _finalize hop, after the whole loop and after delete_items_batch (which rebuilds the entity graph). A single BEGIN IMMEDIATE spanning items + ownership would have to hold SQLite's writer lock across every per-chunk embedding call and the graph rebuild — seconds of held lock that block every other knowledge writer (and trip the loop-stall watchdog the pipeline is built to avoid). The chunked-hop design exists precisely so the writer lock is never held across an embed.

  2. The ownership state lives on a different write surface than the chunk items. The chunk items are written by the shared IngestionPipeline (so every source gets the same chunker/extractor/embedder), while the agent aggregate's ownership (agent_item_state) is an agent-source concept the pipeline does not know about; the pipeline only calls back through on_committed. Folding the ownership write into the pipeline's own transaction would push an agent-source-specific table into the generic ingest path.

So the fix reaps on positive evidence of an interrupted ingest instead: a pre-commit intent marker plus a maintenance-window sweep. This keeps the writer lock short and leaves the pipeline generic.

What changed (motivation → approach → change)

Symptom: a re-add after a hard kill stores a second copy. Root cause: the ownership row that the replacement path keys off is never written, so the committed items are orphaned and the next add cannot find them to replace.

Approach considered and rejected: reap by absence — delete any active agent:// item that no agent_item_state row names. This is unsafe: _import_bundle_state leaves a bundle's agent items unowned on several legitimate paths (a row skipped as ownership_row_key_held_locally, an empty or over-claiming group, a bundle with no state tables), and a single-item export_item → import_bundle round trip writes no state row either. Absence is therefore not proof of crash residue, and reaping by absence would permanently delete imported content.

Approach taken — reap only on positive evidence of an interrupted ingest:

  • A new persisted table, agent_ingest_intent (CREATE TABLE IF NOT EXISTS, primary key (source_id, slug, content_hash), carrying started_at), is written by add_agent_document before it commits any item, through mark_ingesting. It is a separate table, not a status value on agent_item_state: the marker must coexist with a live ownership row, because a re-add that crashes while replacing a live group must leave a marker exactly as a first add does. The marker is written on every add that reaches the ingest (it does not depend on whether a live group already owns the slug); the finalize hop deletes it via clear_ingesting, and every refusal/error exit clears it too, so only a hard kill leaves one behind. A same-content retry at the same slug shares the one marker row and keeps started_at at the earliest attempt (ON CONFLICT … DO UPDATE SET started_at = MIN(…)), so an earlier crash's evidence is not lost.
  • Submitted agent text is newline-normalized before hashing and ingesting. add_agent_document now calls _normalize_newlines(text) (CRLF / bare CR → LF) before computing content_hash and writing the temp file, so the marker hash equals the hash the pipeline stamps on each chunk (the file reader does the same normalization). See the compatibility note below for the effect on documents stored before this change.
  • KnowledgeStore.reclaim_agent_source_residue() deletes an unowned active agent:// item only when a stale marker names its content_hash and the item was created at/after that marker's started_at (items.created_at >= started_at) — positive, attempt-scoped proof an ingest for exactly that content started and never finished. An imported item has no such marker, and a pre-existing item created before the marker is excluded by the timestamp, so neither is ever a candidate. A hash claimed by any healthy or deduped ownership row is shielded, so a sole surviving copy is never deleted on bare hash evidence. An unreadable item_ids on any row stands the whole sweep down (a wrongly-deleted item is unrecoverable where a stale one is not). The sweep runs in chunked BEGIN IMMEDIATE transactions that re-derive ownership and the marker set under the writer lock, so an ingest that finalized in the gap protects its items.
  • Each agent add now holds the ingestion gate across the whole span — the marker write, the ingest, and the marker cleanup — so a maintenance sweep cannot begin between the marker write and the ingest's own gate entry and retire a live add's fresh marker. The hold is re-entrant per task.
  • The owned-id derivation (which items a live state row names) lives in one place, KnowledgeStore.agent_owned_item_ids; both the sweep and clear_ingesting call it.
  • Wired into _kick_knowledge_orphan_reclaim, inside the same drained maintenance_window as reclaim_orphans, so it only ever sees fully-settled ingests.

Backwards compatibility

Compatible: adds one maintenance-window sweep, one public store method, one new table, and one pre-commit marker write; it rejects, renames and removes no existing input, key, or capability.

  • New schema. agent_ingest_intent is a new permanent table created with CREATE TABLE IF NOT EXISTS on store init; it is additive and touches no existing table. No migration of existing rows is needed.
  • Newline normalization of already-stored CR documents. _normalize_newlines changes the content_hash of any already-stored document that contains a carriage return (\r\n or bare \r). On that document's next add, the "unchanged since last add" shortcut and the find_document_by_hash dedup both compare the new (normalized) hash against the old (un-normalized) stored hash and miss. The effect is a one-time re-ingest (replace), not a duplicate: get_state returns the previous group by slug (the slug is derived from source_uri, not the hash), so the add proceeds down the normal replace path and ingest_file's finalize deletes the old group (delete_items_batch(old_item_ids)) before recording the new one. After that one re-ingest the stored hash is the normalized value and the shortcut hits again. No duplicate is produced, so no extra handling is required; a CR-only document simply re-chunks once on its next add. A document with no carriage returns is unaffected (its hash is unchanged).

The diff has a take-away shape (the sweep deletes items), so the readers of a swept item:

  • Reader: src/kiro_crew/knowledge/store.py:reclaim_agent_source_residue — crew page / cron (maintenance window) — only deletes an unowned active agent:// item whose hash a stale marker names and that was created at/after the marker; proven by the imported-unowned-survives, sole-copy-survives, and real-residue-removed tests.
  • Reader: src/kiro_crew/knowledge/agent_source.py:get_state — chat (knowledge.agent_document.add) — reads item_ids from agent_item_state; the marker lives in a separate table and never changes what get_state returns, so the re-add path behaves as before.
  • Reader: src/kiro_crew/knowledge/agent_source.py:find_document_by_hash — chat — filters agent_item_state.status = 'active', so an intent marker (a different table) never falsely refuses a re-add.

Tests

  • test/test_agent_source_residue_sweep.py (new): real crash residue (items + a stranded intent marker) is removed and a re-add then stores one copy; a bundle-imported unowned item survives; an owned item sharing a crashed hash is shielded while only the unowned orphan is swept; a sole copy at the same slug as its marker survives; a pre-existing item created before the marker survives; the earliest started_at is kept across a same-content retry; the add holds the ingestion gate before writing the intent; deduped markers, unreadable-group fail-safe, and co-location detach covered; artifact and folder sources never touched.
  • test/test_knowledge_reclaim_deferred.py: pins that the kicker runs the residue sweep in the same drained window as reclaim_orphans.

Manual verification

N/A — unit coverage sufficient: the sweep and its fail-safe boundaries are exercised directly, including the imported-unowned-survives case that proves no legitimate import is deleted.

Related Issues

Fixes #2670

Supersedes #17504.

Pattern harvest

Rule candidate: review-prompt
Pattern: a reaper that infers "crash residue" from the ABSENCE of a tracking row — safe only if absence is unreachable by any legitimate path; prefer a positive pre-commit intent marker over reaping by absence whenever an import or restore path can also produce the absent state.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A same-content re-add that runs before the sweep still leaves a permanent duplicate and a marker that never clears, and the "self-healing" claim does not hold.

Watch

  • Same-content retry is shielded forever. A same-slug retry writes a healthy row that owns hash H. _live_owned_hashes shields H "UNCONDITIONAL[ly]", so the orphan is never reaped. clear_ingesting(preserve_if_orphans=True) then keeps the marker for as long as that orphan exists, so every boot rescans it. The code calls this duplicate "self-healing: a later identical add dedups it", but _skip_as_duplicate passes exclude_source_id=source_id, so a re-add within agent:// never dedups against it. test_a_same_hash_retry_preserves_the_shielded_duplicate pins this as deliberate. The window is narrow (the re-add has to beat the boot sweep, or the sweep has to skip on a drain timeout), but this is exactly the harm the Goal names, and in this window it stays.
    Clears when: the "self-healing" wording is corrected and this residual is recorded as a known gap, or a test shows a later add actually removes the shielded copy.
  • The spec is out of date. This PR adds a permanent persisted table (agent_ingest_intent), a second post-bind sweep and a newline-normalization change to content_hash. docs/system-specs/modules/knowledge.md documents agent_item_state and reclaim_orphans in detail but is not updated.
    Clears when: knowledge.md describes the intent table, its lifecycle, the residue sweep's evidence rule and the CR-hash change in this commit.

[DESIGN-REVIEWED] 0f74f26

Description read

[DESCRIPTION-READ] 67bc6651b51493d40de4282e34135c15bf72621bb40ea3ad29795d7678baf388

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

preserve_if_orphans keeps a crash marker alive in the very case where the sweep's shield won't reap. So "crash, then same-content re-add" still ends with two copies.

Not justified as shipped

  • Item 5 — oversized: two of the four preserve_if_orphans=True sites can't lead to a reap while the row they just wrote exists. These are the finalize site in _record_ownership and the one in _finalize_deduped. That row's hash is shielded unconditionally, which the test admits: "the sweep leaves EVERY unowned copy of it alone".
    Subtraction: drop preserve_if_orphans at those two sites and clear unconditionally. Keep it only on the deferred/error exits, where no row is written.
    Clears when: the flag is removed from the finalize and deduped sites, or a test shows a reap that only that preservation makes possible.
  • Item 6 — undeclared, symptom-level: the Goal says "stop a re-added agent document from being silently stored twice", but test_a_same_hash_retry_preserves_the_shielded_duplicate asserts removed == 0 with the orphan surviving. A re-add that reaches the gate before the single boot sweep runs keeps the duplicate for good. _skip_as_duplicate skips same-source hits, so the "(self-healing)" claim is unsupported.
    Clears when: the description says this re-add-before-sweep case is left unfixed.
  • Item 9 — zero consumers: mark_ingesting reads 0 of its name/source_uri parameters, and it only wraps _write_ingest_intent, which has 1 caller. The _owned_ids closure is a second name for agent_owned_item_ids.
    Subtraction: merge the two functions into one mark_ingesting(store, source_id, slug, content_hash) and call self.agent_owned_item_ids(source_id) directly.
    Clears when: the unused parameters and both wrappers are deleted.

What this change ships

Inventory (9 items) — 6 justified

Intent: stop a hard kill during an agent-document add from leaving a duplicate that can never be removed. This is a FIX.

  1. The boot maintenance window deletes unowned agent items that a stale intent marker names — justified
  2. A new persisted table records an intent marker before every agent add — justified
  3. Submitted agent text has its CR/CRLF line endings normalized before hashing, so stored CR documents get a new hash — justified
  4. An agent add holds the ingestion gate across the marker write, the ingest and the cleanup — justified
  5. Finalize, dedup and refusal exits keep the marker while earlier orphans remain — oversized
  6. A crash followed by a same-content re-add before the sweep still leaves two copies — undeclared
  7. New public agent_owned_item_ids shared by the sweep and the cleanup — justified
  8. Deleting the agent source also deletes its intent rows — justified
  9. Public mark_ingesting with unused parameters, plus wrapper layers — zero consumers

[FIRST-PRINCIPLES-REVIEWED] 0f74f26

Description read

[DESCRIPTION-READ] 268a2046e1187f98ff3bc6f938b660f45841f0f65e1edc3a07ec4e50d79cb6c7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0f74f26

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0f74f26

Verdict parsed from the review's SHA-scoped output markers for commit 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0f74f26abd74f877fa7d126d12d6fecd8e8f0d72: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from c1adb48 to 83d1b6f Compare October 8, 2026 09:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from 83d1b6f to 24dc98d Compare October 8, 2026 11:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from 24dc98d to fc78567 Compare October 8, 2026 12:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from fc78567 to 9eec01a Compare October 8, 2026 13:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from 9eec01a to 7f7272d Compare October 8, 2026 15:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from 7f7272d to 815664a Compare October 8, 2026 17:11
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both findings on 7f7272dd0a are addressed in code on the new head 815664acb0. They share one root cause and one fix, so I describe the fix once and then each finding.

Root cause. The sweep's shield excluded a (slug, hash) pair from protection whenever that same pair also carried a crash marker. That exclusion existed so a genuine same-slug retry's leftover copy could be reaped as a duplicate. But it also stripped protection from the one case where the leftover copy is the slug's only surviving content — a reassignment that could not establish ownership leaves the row with no items and a single unowned copy, or a hard kill during a replace leaves the new copy unowned while the old items are already deleted. In both, a bare content-hash match from the marker then authorized deleting the last copy.

Fix. The exclusion is removed. The shield is now unconditional: if any healthy or deduped row claims a content hash — whether it still names live items or its group is empty — the sweep leaves every unowned copy of that hash alone. The only cost is that a genuine same-slug retry's duplicate is no longer reaped by this path; it is left in place and a later identical add deduplicates it. Between leaving a self-healing duplicate and deleting a slug's last copy, the shield takes the side that never deletes.

F1 (store.py, hash whose only copy is unowned). A deduped row whose (slug, hash) also had a marker added nothing to the protected set, so its sole surviving copy — created after the marker — passed the attempt-time check and was deleted. The row now shields its hash regardless of a co-located marker, so the sole copy survives.

Opus (store.py, replace-path). Re-adding a slug with edited content, then a hard kill before the ownership write, leaves the old row naming deleted items and the new copy unowned under a marker whose start precedes it. The same unconditional shield covers this: the slug's deduped/healthy row claims the content hash, so the new copy is not deleted and the document stays searchable.

Tests pin both: a sole copy at the same slug as its marker survives, the empty-group deduped shield holds, and the same-slug retry's duplicate is preserved (not reaped). The full agent-source / dedup / ownership / ingestion suite passes and the local gate is clean.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@chenmingwei23
chenmingwei23 marked this pull request as ready for review October 8, 2026 17:45
@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 8, 2026 17:45
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

This fixes a re-added agent document being stored twice after a hard kill lands between the item commit and the agent_item_state ownership commit, by writing a pre-commit intent marker and adding a maintenance-window sweep (reclaim_agent_source_residue) that deletes unowned agent:// items only when a stale marker names their hash. I read the diff and the lane comments on head 815664a. GPT and Opus have no findings, and the author's replies close the earlier F1-F3 shield and timestamp findings in code. HOLD for three reasons. First, it adds a new path that deletes user data (~650 prod lines in store.py and agent_source.py). Second, the description describes a different design from the one that ships: it says the marker is an ingesting status in agent_item_state, written only when no live group owns the slug, but the diff adds a new permanent agent_ingest_intent table that is written on every add. Third, the description never mentions _normalize_newlines, which changes content_hash for any CR-containing document already stored and so makes the unchanged/dedup shortcuts miss on them. Design and First Principles are both at CONCERNS on this head and have no disposition, including the root-cause question of why the items and the ownership row can't share one BEGIN IMMEDIATE transaction (that would remove the need for the sweep) and the duplicated owned-id derivation (agent_source.py clear_ingesting vs store.py _owned_ids). Before a maintainer approves, the author should rewrite the description and Backwards compatibility to match the shipped table and the newline normalization, and answer both whole-design lanes. — crew-pr-reviewer

…attempt

The maintenance sweep reaps an unowned agent item only on positive intent
evidence: a stale agent_ingest_intent marker naming its content hash AND the
item created at or after that marker's started_at. Tying the proof to the
attempt in time means a pre-existing or bundle-imported item that merely shares
a hash (created before any crash) is never residue and never deleted.

- store.py: _ingesting_hashes returns hash -> earliest started_at; the candidate
  read, the under-lock recheck and marker retirement all require
  created_at >= started_at, so a pre-existing copy neither gets swept nor pins a
  spent marker open.
- store.py: the under-lock recheck filters status = 'active', so an item a
  concurrent writer flips out of active in the read-to-lock gap is not swept.
- agent_source.py: the budget-refusal, exception and non-completed cleanup paths
  clear the marker with preserve_if_orphans, so a same-content retry refused
  after an earlier crash no longer erases that crash's recovery evidence; the
  preserve check is attempt-scoped too.

Tests: a pre-existing/bundle item sharing a hash survives; a spent marker is
retired despite a pre-existing same-hash item; a non-active marker-named item is
left alone; the same-slug retry fixture models the real marker-before-items
crash order.
@chenmingwei23
chenmingwei23 force-pushed the fix/2670-agent-residue-sweep branch from 815664a to 0f74f26 Compare October 9, 2026 00:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 9, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Thanks — all three are addressed on head 0f74f26abd.

1. Description and Backwards-compatibility rewritten to match the shipped code. The description now says the marker is a new permanent agent_ingest_intent table (primary key (source_id, slug, content_hash), carrying started_at), created with CREATE TABLE IF NOT EXISTS, written on every add — not an ingesting status on agent_item_state. The Backwards-compatibility section now has a dedicated note on _normalize_newlines: it changes content_hash for any already-stored document containing a CR, so on that document's next add the unchanged/dedup shortcuts miss. I traced the effect: it is a one-time re-ingest (replace), not a duplicate. get_state returns the previous group by slug (derived from source_uri, not the hash), so the miss falls through to the normal replace path and ingest_file's finalize deletes the old group (delete_items_batch(old_item_ids)) before writing the new one. After that one re-ingest the stored hash is the normalized value and the shortcut hits again — no duplicate, so no extra handling is needed. The widened ingestion-gate hold is also now described.

2. Design and First Principles answered on this head (separate replies on each lane), including the root-cause question — why the items and the ownership row can't share one BEGIN IMMEDIATE. Short version: they are not adjacent. The chunk items commit per-chunk inside a loop that calls the embedder (an await behind a rate limiter) between chunks, and the ownership row is written only in the final _finalize hop after the whole loop and the old-group delete. One transaction would hold SQLite's writer lock across every per-chunk embed and the graph rebuild, blocking every other knowledge writer and tripping the loop-stall watchdog the chunked-hop design exists to avoid. The reply gives the file/line detail.

3. Duplicated owned-id derivation removed. The owned-id query/parse/fail-safe now lives in one place, KnowledgeStore.agent_owned_item_ids; both the sweep (store.py _owned_ids) and agent_source.py clear_ingesting call it. grep "SELECT item_ids FROM agent_item_state WHERE source_id = ?" over src/ now finds one copy. Two tests pin it: the shared helper returns None on an unreadable group, and clear_ingesting keeps the marker (stands down) on that signal, matching the sweep.

Gate green (flake8 / isort / mypy / baselined-black / comment-history); 61 touched tests pass. Rebased onto current main and pushed once.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

On the root-cause question — why not commit the items and the ownership row in one BEGIN IMMEDIATE and drop the sweep. I investigated it; it is not feasible for this path, for two concrete reasons.

The item commits and the ownership write are not adjacent — a per-chunk embedding step sits between them. IngestionPipeline._ingest_file_body commits each chunk's items row in its own add_item transaction inside a loop, and between chunks calls _embed_item, which is await loop.run_in_executor(..., embed_call) behind an async rate limiter — a network/CPU step that straddles the event loop. The ownership row is written only in the final _finalize hop, after the whole loop and after delete_items_batch rebuilds the entity graph. A single BEGIN IMMEDIATE spanning items → ownership would have to hold SQLite's writer lock across every per-chunk embed and the graph rebuild — seconds of held lock blocking every other knowledge writer, and the exact loop-stall the chunked-hop design (and its watchdog comments) exist to avoid.

The two writes are on different surfaces. The chunk items are written by the shared IngestionPipeline so every source gets the same chunker/extractor/embedder; the agent aggregate's ownership (agent_item_state) is an agent-source concept the pipeline does not know about and only reaches through the on_committed callback. Folding ownership into the pipeline's transaction would push an agent-source-specific table into the generic ingest path.

So the fix reaps on positive evidence instead (a pre-commit intent marker + a maintenance-window sweep), which keeps the writer lock short and the pipeline generic. The description's "Why not one transaction" section now records this.

On the "stays duplicated forever by design" watch item: the same-content retry case is bounded — the marker keeps the earliest started_at across retries, and the shield leaves at most one self-healing duplicate rather than deleting a sole copy. The description's compatibility note states the one case (a CR document re-ingests once) explicitly as a known, repairing behaviour.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both items addressed on head 0f74f26abd.

Item 2 — undeclared agent_ingest_intent table. The description's What changed and Backwards compatibility sections now describe the actual shipped surface: a new permanent agent_ingest_intent table (CREATE TABLE IF NOT EXISTS, primary key (source_id, slug, content_hash), carrying started_at), written on every add through mark_ingesting, explicitly not a status value on agent_item_state. The reason it is a separate table is stated: the marker must coexist with a live ownership row so a crash while replacing a live group leaves a marker just as a first add does. The _normalize_newlines behaviour (item 3 in your inventory, undeclared) is now declared too, with its hash effect and the one-time re-ingest it causes for already-stored CR documents.

Item 6 — duplicate owned-id derivation. Subtracted as you proposed: deleted the copy in clear_ingesting and routed both callers through one store method, KnowledgeStore.agent_owned_item_ids. grep "SELECT item_ids FROM agent_item_state WHERE source_id = ?" over src/ now finds one copy (in store.py). The sweep's _owned_ids and clear_ingesting both call it, and both honour its None ("unreadable group → cannot tell") return by standing down — the sweep sweeps nothing, clear_ingesting keeps the marker. Two tests pin this.

Gate green; 61 touched tests pass. Rebased onto current main and pushed once.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 9, 2026 05:49

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review passed: fix (5 files). Adds an agent_ingest_intent marker table and a maintenance-window sweep that deletes only crash residue with positive evidence (marker hash plus created_at after start, live-owned rows shielded, unreadable groups fail closed). The author answered every lane concern on this head; the one known residual (a same-content re-add before the sweep) is declared, not a regression. Approved at a maintainer's direction.

@KiroCrewCaptain
KiroCrewCaptain merged commit 8f9333b into main Oct 9, 2026
106 of 114 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/2670-agent-residue-sweep branch October 9, 2026 05:49
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 9, 2026
@buluoray

buluoray commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Reviewed after merge. The sweep is idempotent, stays inside the agent source, and spares completed adds. One window remains where it can delete the only remaining copy of a document: when a replace crashes between the old-group delete and the new ownership-row write, which are two separate commits. Tracked with acceptance criteria in #18423. This was traced from source; I have not reproduced it with a probe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Knowledge: interrupted artifact ingest leaves duplicate items, and bundle import loses ownership metadata

4 participants