Repository navigation
fix(knowledge): carry per-document ownership through a bundle round-trip - #13448
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Follow-up sweep for #2670 must stay scoped to agent-owned residue: folder/artifact bundle imports still arrive unowned, so crash-vs-import indistinguishability persists for those tables. [DESIGN-REVIEWED] 785c72d |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify one premise: no bundle outside this repo hand-builds items naming an undeclared source — that previously-importing shape is now a typed 400 (both in-repo exporters ship Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: make importing a
[FIRST-PRINCIPLES-REVIEWED] 785c72d |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- test/test_knowledge_bundle_ownership.py:159 -- False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks. FINDING — src/kiro_crew/knowledge/store.py:3409 — the FINDING — src/kiro_crew/knowledge/store.py:722 — the header [OPUS-REVIEWED] 785c72d Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bef7363 to
219bfcf
Compare
cadb9d0 to
25f7f1e
Compare
Attribution for the non-green lanesNone of these come from this diff, which touches the knowledge store, its own test file and one spec file. Current head All four reds are jobs of one
These are inherited, not local and not flake. The CI run on A caution for anyone else checking this: reading Nor is it a shard artefact: Three independent base-side breakages, then, plus one dependent — not one shared cause. Also seen together on #13640, #13727, #13936 and #14095. Earlier heads of this PR had different reds, all since resolved: two upheld |
25f7f1e to
dc43317
Compare
dc43317 to
33d3ea9
Compare
33d3ea9 to
832bf79
Compare
832bf79 to
4d0faac
Compare
cb62692 to
f2c00fa
Compare
f2c00fa to
017b94d
Compare
UX Review (Fable 5) — ⏭️ skippedRevision |
export_all ships folder_file_state, artifact_item_state and agent_item_state, and import_bundle resolves every bundle source by uri before rewriting the rows that point at it.
…t do not own it
export_all ships folder_file_state, artifact_item_state and agent_item_state, and import_bundle resolves every bundle source by uri before rewriting the rows that point at it.
A restored ownership row owns the items THIS import inserted, and for a live local row on the same document key the arriving ids join its existing group. The blocking pass already promised this ("its absent ids arrive, to be merged into that row's group by the accepting pass") and the accepting pass never merged, so a newer pruned backup restored before an older complete one left the older bundle's chunk inserted and FTS-indexed with no row naming it: agent_item_state is the one table a bundle restores, nothing reaps it, and remove_document deletes only the ids its row names. The merge takes only ids this import wrote, exempts the row's own group from the claim check without exempting anyone else's, and UPDATEs item_ids alone so an older bundle cannot repoint the local row's hash and name at a revision this store does not hold.
The merge no longer trusts the blocking pass for its own precondition. That pass never blocks an id two bundle rows both claim -- the bundle does not agree who owns it -- so such an id reached the accepting pass with a live local row on its key whose group shares nothing with the bundle's: a different document by the pass's own rule. The merge extended that row's group anyway, and the local document's next delete took the imported chunk with it, with nothing that reaps the row or reports it. The accepting pass now tests the overlap itself and skips a live disjoint row, naming it in the account as ownership_row_key_held_locally.
The claimed-id rule's exemption is keyed on the bundle's own row at the same (table, source, key), which is what makes it this document meeting itself rather than a blanket amnesty -- so it covers every table the bundle states, not only the one whose rows get written. _delete_item_cascade never prunes a group, so a per-chunk delete on a folder- or artifact-backed document leaves the row naming that chunk while the file or slug is still here, and withholding the id loses it for good: FolderWatcher._do_scan step 4 reaps only a row whose path the walk MISSED, and step 5 skips a done row whose mtime and content hash did not change, so nothing re-ingests it. The opposite risk -- a row whose document is gone here and not yet reconciled away -- needs the owning subsystem's view of whether the document still exists, which is not reachable inside the import transaction, so it is named in the code and left to the reap it belongs to.
Every IN list on this path is chunked under _MAX_SQL_PARAMS, the ceiling vector_memory already keeps and for the same reason: sqlite binds 32,766 on the bundled build but only 999 on a host still on a pre-3.32 library. The list lengths are untrusted bundle input -- a bundle names as many sources as it likes, and one state row's group runs to _MAX_ITEM_GROUP_MEMBERS -- so both pass 999 on ordinary input, and the OperationalError lands mid-transaction: the whole import rolls back and the endpoint answers 500 where it should have succeeded. Four sites are covered, the two source-keyed reads and the two group-keyed probes; _items_exist already chunked and now shares the one constant.
A bundle source reference absent from source_id_map is refused as a typed 400 instead of being resolved against this store's own source rows. The declared path already refuses to file a bundle's content under a local row that merely shares an id -- it mints a fresh uuid4 when a bundle uri is absent but its id is held by a different uri -- while the undeclared path resolved the bare id, so the foreign key refused the harmless case (an id that exists nowhere) and admitted the harmful one (an id this store holds under a source the bundle never names).
The import response's withheld account carried bundle-authored text unredacted. Its fields are the ones the handler's inbound loop exempts on purpose, because the store matches them against a PRIMARY KEY and against items.id and needs them byte-exact, and a document key has no length bound. They are redacted after the store has used and stored the real values, so the stored key stays byte-exact.
Problem / Motivation
Goal: a
.knowledgebundle round-trips the ownership of its own content -- imported items arrive owned by the document that exported them, and a store that already holds a bundle's source imports it instead of rolling the whole import back.A
.knowledgebundle moves content without moving the ownership of that content, and on many stores it cannot be imported at all.export_allshipsitems,sources,entities,relations,source_locationsandmentions. It does not ship the three per-document state tables that say which document owns which items:folder_file_state,artifact_item_state,agent_item_state.Separately,
sources.uriis UNIQUE while each store mints its own source ids.import_bundleinserted each bundle source under the id the EXPORTING store had used, so when the receiving store already held that uri the unique index skipped the row and every item then referenced a source id this store did not have. Measured onmain:The whole import rolls back. That is reachable by re-importing a bundle into the store it came from, and by moving any bundle with an
artifact://oragent://aggregate between two machines.Why it matters
Two distinct user-visible failures.
The import simply fails. The user picks a bundle they exported, gets an error, and nothing tells them that the reason is a source they already have.
When the import does land, the knowledge it brings is permanently unmanaged. An item with no state row has no owner, so: nothing claims it for de-duplication, the Sources UI has no per-document group label for it, and a later ingest of the same document adds a second searchable copy instead of replacing the first. The user ends up with two answers to one question and no way to tell which is current.
That second failure is also what blocks the repair for interrupted ingest. Unowned items from a crash and unowned items from an import are indistinguishable, so a sweep that cleans up residue would delete legitimately imported knowledge. #2452 shipped exactly that sweep and had to be reverted for this reason.
Not a goal
Refs, notFixes.folder_file_stateorartifact_item_staterows. Both owners reap by ABSENCE, so restoring their rows is a deletion order for the items the import just brought:FolderWatcher._do_scanstep 4 drops every row its walk did not yield, andreconcile_artifactsremoves each provably-absent slug's group._known_kindsreads every row regardless ofstatus, so no status value hides an imported row from that pass. The export still ships all three tables; onlyagent_item_stateis restored.source_iddoes not map, the row is skipped and nothing is recorded inwithheld, while every neighbouring unowned-arrival path does record. Filed as knowledge: the bundle-state skip for an undeclared source records nothing in the withheld account, while every neighbouring unowned-arrival path does #14395. The shape is not reachable through any exporter in this store, so it is a disclosure gap rather than live data loss.export_allcannot produce that shape, and partially trusting a bundle that contradicts itself about who owns an id is a separate decision.What changed
Symptom: importing a bundle fails, or delivers knowledge nothing can manage.
Root cause: values local to the exporting side were treated as portable. A source's id belongs to the store that minted it,
mtimebelongs to the host the file sits on, and ownership was never in the bundle's vocabulary at all.A source's cross-store identity is its uri.
import_bundleresolves each bundle source by uri and records a map from the bundle's id to the local row's id, then rewritesitems.source_id,source_locations.source_idand the state rows through it. Every case has a verdict:KnowledgeBundleError, a 400. No export can produce it (sources.idis a PRIMARY KEY), and the later entry would overwrite the earlier one's place in the mapsources.idorsources.uriis not a non-empty stringKnowledgeBundleError. Both become dictionary keys during the rewrite, so a list or dict raises an unhashable-typeTypeErrorthat no handler arm catches -- a 500 in place of the typed 400The last row is deliberate, and it is where the declared and undeclared paths had disagreed. The declared path already refuses to file a bundle's content under a local row that merely shares an id: when a bundle source's uri is absent here but its id is held by a different uri, the uri arrives under a freshly minted
uuid4rather than the unrelated row, because reusing that id "would file this bundle's documents under an unrelated source". The undeclared path did not have that rule -- it resolved the bare id against this store'ssourcesand passed it to the insert -- so the foreign key refused exactly the harmless case (an id that exists nowhere) and admitted the harmful one (an id this store holds under a source the bundle never names)._source_fknow refuses any non-null reference absent fromsource_id_map, which completes the rule rather than adding a second one; a NULL reference is not a reference and imports unchanged. This also makes the plainINSERTin the sources loop defence in depth rather than a hope: were it to fail and leave a uri unmapped, its items are refused instead of misfiled -- the outcome that insert's own comment calls "the worst outcome available".The withheld account is redacted on the way OUT, because its fields are the ones the inbound loop cannot touch. A withheld entry names what did not arrive in the bundle's own words: the document key, the ids it claimed, and the source they resolved to. Those are exactly the values
_BUNDLE_STATE_PLAIN_FIELDSexempts from the handler's inbound_redact()loop, because the store matches them against a PRIMARY KEY and againstitems.idand so needs them byte-exact -- and a document key carries no length bound. That left the HTTP response as the one path on which bundle-authored text reached a reader unfiltered, sinceweb.json_responsereturns the account verbatim, while the same handler already routesnameandsource_urion those very rows through_redact()and the SEL copy is covered by_redact_and_clip._WITHHELD_REDACTED_FIELDSis applied after the store has used and stored the real values, so the stored key stays byte-exact and the response carries no bundle-authored credential. Pinned from both sides: the credential does not appear anywhere in the response, and the storedslugis byte-identical.The identity-column rule is enforced at the writer rather than in the endpoint's validator, matching the invariant this module already holds for
sources.propertiesandentities.aliases: the dashboard endpoint is the store's only production caller today, and enforcing at the writer makes any future one safe by construction.Ownership travels with the content.
export_allships the three state tables, keyed off the existing_DOC_STATE_TABLESdefinition so there is no second list to drift, andstore.BUNDLE_STATE_KEY_COLis the one name the importer and the dashboard's bundle validator both read. A row owns the items THIS import inserted for it, and those items must have landed under the mapped source; a row written fresh carries that table's own live status (donefor folder rows,activefor the aggregates, which are not interchangeable vocabularies).A collision is only a DIFFERENT document holding the key, and no withholding is silent. When the local row's own group overlaps the bundle's, this is that same document coming back. Deleting one chunk of a chunked document through the dashboard's single-item route leaves the group naming the deleted id while a surviving sibling keeps the row live -- so the absolute rule held the whole group back, the deleted chunk included, in the one case where the bundle is the only remaining copy. Its only exit was deleting the document's remaining chunks so the row owned nothing live, which asks the user to destroy content in order to recover content. Such a group is no longer withheld, and the ids that arrive JOIN the live row's group.
The arriving ids join the live row's group, which restores the design rather than changing it. The module already stated this behaviour twice and implemented it nowhere.
store.py:3264-3266, in the blocking pass, is explicit that a same-document collision lets the absent ids through "to be merged into that row's group by the accepting pass" — the accepting pass never merged. And_adopt_reassigned_itematstore.py:1556-1563calls the outcome of a row that owns an item it does not name "the strand this whole model exists to prevent", for exactly the reason that matters here: its own delete path consults onlyitem_ids.Without the merge the gap is reachable by the ordinary restore order. Restore the newest backup, then an older one to recover what the newest had already lost:
export_allpruned the newer group to the chunks it still carried, so the older bundle's row names ids the first restore already put down alongside the one it brings.INSERT OR IGNOREskips the present ids, so they never enterinserted_items; the whole row was then dropped while the new chunk was inserted and FTS-indexed.agent_item_stateis the one table a bundle restores and nothing reaps it, soremove_documenttakes the ids its row names and walks past that chunk — which stays searchable, owned by nothing, and removable by no path at all.Three properties make the merge safe, and each is pinned in both directions:
INSERT OR IGNORE, so the row in the store is local content; adopting it would hand a foreign document the authority to replace or delete it. Unowned residue from an interrupted ingest is the reachable case._claimed_item_idsincludes the base group, so testingarriving & claimed_idswould refuse the very merge it exists to allow, while testing nothing would put one chunk in two groups.INSERT OR REPLACEwould leave the row'scontent_hashandnamedescribing a revision this store no longer holds — a silent data regression inside a data-integrity fix. The merge is anUPDATE item_ids.One item in two groups is not a shape this schema has, which is why the claim check is sufficient rather than a heuristic: a de-duplication loser holds an empty group plus a
source_locationsclaim by hash, and_adopt_reassigned_item,detach_source_location_by_hashandclaimed_idseach refuse the overlap independently.Where the group already names the returning id — a chunk deleted locally and restored from a bundle of the same revision — nothing has to be added, because no delete path rewrites a group (
_delete_item_cascadeleaves it alone and_adopt_reassigned_itemonly appends) and the row becomes accurate again the moment the chunk returns. Withholding that DOES still happen is now named:withheldcarries an entry per held-back document or id with its reason, beside theitems_withheldcount.A document key this store already holds blocks the bundle's copy of that document entirely. A
(source, key)pair IS a document's identity within a source, so two documents cannot share one. A local row whose group is empty or names only deleted items is a marker rather than ownership -- a folder file that lost a de-duplication keeps adedupedrow with an empty group, and every pending or failed scan row carries one too -- so the imported row takes the key, and the marker's claim on another source's items is released with it throughdetach_source_location_by_hash. A local row that DOES own items keeps the key, and_bundle_blocked_itemsthen withholds that document's items from the import: letting them in would leave them permanently unowned, which is the defect this round-trip exists to remove rather than a lesser version of it. A local row's group counts as ownership only for the items filed under that row's OWN source._adopt_reassigned_itemleaves a stale claim by design when a content hash is ambiguous -- it refuses to cross-wire one item into two groups, so the un-adopted row keeps naming itemsdelete_source_cascademoved to the recipient -- and those items are live. Counting a stale claim as ownership hands the key to a row that owns nothing here and withholds the bundle's document, losing its content silently, which is the same rule the accepting pass applies to the group a bundle ships. And the collision is computed for EVERY state table, not only the one whose rows travel: withholding is not reserving items for a claim someone goes on to make, it is the answer to "this store already holds that document", which a folder path and an artifact slug each answer for themselves. Restricting it to the restored table left a local folder file or artifact slug importing a second copy of its own text, unowned and removed by nothing._bundle_state_restores()governs row restoration alone.That is the only refusal reason that blocks an item. Every other one rejects a ROW as an untrustworthy statement about items that are themselves legitimate -- an unparsable group, one whose ids another row already owns, one the bundle files under a different source -- and those items arrive unowned exactly as they do from a bundle carrying no state tables at all. An over-claiming group is not among them: the ids the import wrote are owned and the rest are only named in the account.
An exported ownership group names only items that export carries, and only ones filed under that row's own source. The accepting pass keeps only the ids the import wrote, so an id the export omits is named on every restore of that document in the account of what arrived owned by nothing, and the group reaches the other side one id short of what the row claims. That shape is routine rather than exotic: no delete path prunes a state row's group --
_delete_item_cascadeanddelete_items_batch_in_txnboth leave it, by design, because the group is accurate again once the chunk returns -- so a document that lost one chunk has a row naming a dead id, and shipping it verbatim reports content as unowned that nothing is actually missing.export_allprunes each group to the items it carries keyed to the source each is filed under, and drops a row that prunes to empty; this subsumes the namespace-only filter, since a row belonging wholly to another namespace prunes to nothing and that namespace'sfile_path, name and hashes stay in the machine. A bundle built elsewhere can still name an absent item; that id stays out of the group -- partial ownership would claim items another document holds -- and it is reported inwithheldasownership_row_names_absent_itemswith the document key and the ids. One id the bundle files under a DIFFERENT source is not a partial statement but an inconsistent one, and refuses the whole row: this store's own export cannot produce that shape.Parsing a group is linear, and both of its fields are bounded. The membership test used
entry not in groupagainst the growing list, so anitem_idsarray of N unique strings cost O(N^2) -- and this parser runs insideasyncio.to_threadafterBEGIN IMMEDIATE, with the import body read atmax_bytes=None, so one large array spins a worker thread while holding the knowledge write lock. Asetcarries the membership test, and the two retained fields are bounded by name:_MAX_ITEM_GROUP_MEMBERS(4096) and_MAX_ITEM_ID_CHARS(128). Both bounds REFUSE rather than trim, in_bundle_group_boundsbeside the existing identifier gate, because a trimmed group would under-claim in silence and hand the missing chunks to no one.The claim exemption names the IDS a bundle's own row claims, not the key. Two independent reviewers found the two directions a key-level exemption fails in. A full export ships empty-group MARKER rows verbatim, so a key carried by such a row would vouch for ids it never claimed and hand them to whatever unrelated local row holds a stale claim -- the next delete of that document then takes the imported item. And a bundle that ships NO state row for a table states no ownership through it at all (the single-item export endpoint, and every bundle written before this change), so judging its items against that table's local claims withholds exactly what a delete removed, with no exemption able to exist because nothing was shipped to carry one. So a table the bundle is silent about is skipped, and within a table only the ids the bundle's own row for that key names are exempt.
An id a local row already claims does not arrive, unless the claiming row is the document this bundle is restoring and holds nothing live. Both halves fail in opposite directions. Without the rule, a stale group belonging to an UNRELATED document captures the item: claims are read before the item loop, so the id does not exist yet, the insert creates it, the claim goes live over content it never owned, and the bundle's own row is skipped as claimed -- leaving the item held only by that row, so deleting that unrelated document destroys imported content. Without the exception the ordinary restore breaks:
_delete_item_cascaderemoves an item and leaves the doc-state group naming it, which is the documented normal shape, so exporting a document, deleting it and re-importing the export would meet the deleter's own stale row and withhold the item -- the one case where the file is gone and the bundle is the only copy. That row is the same (source, key) the bundle restores and owns nothing live, so the accepting pass repoints it; withholding there contradicts that pass instead of guarding it. An id that already exists here is outside both halves:INSERT OR IGNOREleaves the local row alone and the inserted-items rule already refuses to hand it to an imported claim.One item is never owned by two rows. A group whose ids are already claimed, by a local row of the same source or by an earlier row in the same bundle, is refused. Two rows naming one item is content-destroying rather than untidy: the next document-level delete hands
delete_items_batchan item whose only other holder is a state row it does not consult, finds nothing else holding it, and removes it, leaving the second row naming deleted content._adopt_reassigned_itemanddetach_source_location_by_hashalready refuse the same shape, so this closes a gap in an invariant the module holds elsewhere. The subset rule is not redundant with it: only the subset rule stops a bundle claiming pre-existing UNOWNED local content -- residue -- which it never shipped.mtimedoes not travel; the content hashes do. The folder scan skips hashing a live row whose recorded mtime is at or above the file's, so a carried mtime from the exporting host would stop the local file ever being hashed again, silently leaving the imported text indexed in place of the local content with no self-correcting path. The importer writes the most negative FINITE double as the floor -- zero would strand a file dated at the epoch and a pre-epoch file carries a negative mtime, so neither is low enough, and-infis not valid JSON in a bundle that gets exported again.content_hashandtext_hash, which DO travel, are what keep the re-hash cheap by skipping re-extraction on a match.Ownership covers only what this import inserted. An id the bundle ships that already exists here was skipped by
INSERT OR IGNORE, so the row in the store is local content -- residue from an interrupted ingest is exactly the reachable case -- and naming it would hand a foreign document the authority to replace or delete something this store owns. A row must additionally own only items filed under its OWN source, which is a separate case: both sources can be in the bundle and differ.A withheld document takes its dependent rows with it. Locations, mentions and
entity_relations.source_item_idall point at an item by foreign key, so a row naming a withheld item would refuse the whole import -- losing an entire valid bundle over one document this store already has its own copy of. And a colliding row blocks nothing when its group overlaps another bundle row's: the bundle does not agree who owns such an item, so treating one row's collision as a verdict would omit a sibling document's valid content.A folder row is never restored for a file this host does not have. The folder scan detects deletions by walking every state row and dropping the ones whose path the walk does not find, without consulting
status-- so restoring adonerow for an absent file is not a stale marker, it is an instruction to delete the items the import just brought. That made this change a data-loss regression in the ordinary "same folder watched on two machines" case, where the two hosts hold different file sets. Leaving the row out costs those items their ownership, which is the state they already had; restoring it costs them their existence. Any error answers "absent", because the failure mode of guessing present is deletion. The aggregate tables are unaffected: an artifact slug is not a path and no scanner reasons from its absence.A namespace-scoped export ships only the scoped documents' state rows. An out-of-namespace row always fails the import's group test anyway, so shipping it would carry another namespace's
file_path, document name and content hashes out of the machine for nothing. Filtered inexport_all's namespace arm, exactly assource_locationsalready is. An unscoped export still ships every row.Every bundle string that reaches a SQLite bind is UTF-8-validated at the writer. A lone surrogate is a
strSQLite cannot encode, and theUnicodeEncodeErrorsits outside every arm the import endpoint catches, so it surfaces as a 500 instead of the typed 400. The state key is validated in_bundle_blocked_items, which is the first bind it reaches.Only
agent_item_stateis restored on import, and the reaper test is why. A state row is a claim that a LOCAL subsystem owns this document, and importing one makes that claim on the subsystem's behalf without its knowledge -- so wherever that subsystem deletes a document because its state row names something the subsystem cannot see, the row is not a stale marker but an order to delete the items the import just brought. Two of the three owners reap exactly that way, and neither is a corner case:folder_file_state--FolderWatcher._do_scanstep 4 walks every state row and drops the ones whose path its walk did not yield, WITHOUT consultingstatus. What the walk yields depends on the MAPPED source's own live filters (extension allowlist,min_file_bytes,ignore_patterns,confine_to_root, and a root.kiroignorere-read every sweep), so a receiving store that filters the same folder more narrowly than the sender deletes the arriving documents. Checking eligibility at import does not close it: the walk's answer changes afterwards whenever the user edits.kiroignoreor narrows the source, so no check made at import stays true, and an exact check would be a second copy of the walk's per-file rules.artifact_item_state--reconcile_artifactsruns on everyArtifactKnowledgeSync.start, takesknown.keys() - live, and removes each provably absent slug's group. A bundle imported on a host whose artifact store does not hold those slugs (any second machine, any restore) loses them on the next start._known_kindsreads every row regardless ofstatus, so no status value hides the row from that pass.agent_item_state-- nothing reaps it.agent_source.remove_documentruns only when a caller explicitly deletes that document; there is no reconcile pass, no inventory comparison and no absence test anywhere on the agent path.The export still ships all three tables, because a bundle is a faithful record of the store and a future importer that can prove liveness could use them. Items from the two unrestored tables arrive UNOWNED, which is the state a bundle carrying no state tables already leaves them in, and each owning subsystem re-derives its row the next time it sees the document.
A repeated
items.idis refused.item_sourceis keyed on it, so a repeat overwrites the earlier row's mapped source -- and because the withheld set holds the ID rather than the row, a collision resolved against the surviving mapping skips BOTH rows while the import reports success, with only theitems_withheldcount hinting that anything went missing.items.idis a PRIMARY KEY, so only one of the two could ever land throughINSERT OR IGNOREin any case.sources.idis already refused when repeated under two uris, for this same overwrite.Every row identifier the import tests for membership is validated as text at the writer.
items.id,entities.id,relations.source_item_id,source_locations.item_idandmentions.item_idare each tested against aset[str]of withheld items, and set membership hashes the key unconditionally: a JSON array or object raises aTypeErrorthat sits outside every arm the endpoint catches, answering 500 where the typed 400 belongs. A number is worse than the crash -- it hashes cleanly and matches nothing, so a withheld item's dependent row is written as though the item had arrived, and the foreign key then fails.Nonestays legitimate, because a relation need not name a source item and an absent key reads the same way.Every source column a bundle supplies is UTF-8-validated at the writer,
created_atincluded. A lone surrogate passes a type-and-emptiness check -- it is a non-emptystr-- and is refused only by an encodability gate.sources.idandsources.uriwere checked for type and emptiness only, so a lone surrogate -- astrSQLite cannot encode -- reached the bind and raised aUnicodeEncodeError, which is outside every arm the import endpoint catches and so surfaced as a 500 where the typed 400 belongs. The transaction still rolled back atomically, so the residual was a wrong status code rather than corruption, but the check was already there and the gate is the same one every other bundle string takes.Every NOT NULL column a bundle supplies for a source is validated, and the source insert no longer suppresses its own failures.
INSERT OR IGNORE INTO sourcessilently swallowed a NOT NULL violation, so a bundle source carrying"name": nullnever landed -- and because the row never landed, the uri stayed absent, nothing mapped that bundle source id, and_source_fk's id fallback then resolved the UNRELATED local source whose id happened to match. The documents were filed under it and granted ownership over, with no error and no recovery path. That is precisely the id/uri collision the remap detects and routes around a few lines earlier, arriving through the fallback instead.name,source_typeandcreated_atare now validated besideidanduri(a MISSINGcreated_atstill falls back to the import clock; only an explicit null is refused), so no bundle-supplied null can reach the constraint and the answer is this module's typed rejection. The insert is a plainINSERTas defence in depth: this branch has already established that the uri is absent and the id is free, so nothing can fire for a legitimate row, and a suppressed failure here is the worst outcome available. Note for anyone extending this: the store's SQLite driver raises anIntegrityErrorclass the endpoint'sexceptarms do not name, so a constraint violation surfaces as a server error rather than the typed 400 -- which is why the validations, not the constraint, are what make the rejection typed.An entity stranded by a withheld item is dropped rather than left as a graph node no document supports. Mentions and relations both skip a withheld item, so an entity the bundle referenced only from those rows arrives reachable from nothing and inflates
entities_createdpast what is reachable. The prune is scoped three ways -- to entities this import inserted, to ones the bundle actually referenced, and to ones nothing in the store references now -- so a bundle carrying a standalone entity with no mentions keeps it (hand-built bundles do that deliberately), and a pre-existing local orphan is left alone.The new lists go through the same handler-side treatment as the existing ones: shape validation, and redaction of the display name and the agent document's locator.
file_pathis deliberately not redacted, because it is half of a PRIMARY KEY matched against a real path on disk and rewriting it would leave the row unable to find its own file.Three review findings on this branch's own heads, all fixed here rather than overridden. Each was reachable, and each is pinned by a test proven to redden when only that fix is reverted.
ownership_row_key_held_locally._BUNDLE_STATE_RESTORED_TABLESlooked safer and was not:_delete_item_cascadenever prunes a group, so a per-chunk delete on a folder- or artifact-backed document leaves the row naming that chunk while the file or slug is still here, and withholding the id loses it for good -- the scan reaps only a row whose path the walk MISSED and skips adonerow whose mtime and hash are unchanged, so nothing re-ingests it. The opposite risk, a row whose document is gone here and not yet reconciled away, needs the owning subsystem's view of whether the document still exists, which is not reachable inside the import transaction; it is left to the reap it belongs to and named in the code.INlist on this path is chunked under_MAX_SQL_PARAMS, the ceilingvector_memoryalready keeps and for the same reason: SQLite binds 32,766 on the bundled build but only 999 on a host still on a pre-3.32 library. The lengths are untrusted bundle input -- a bundle names as many sources as it likes, and one state row's group runs to_MAX_ITEM_GROUP_MEMBERS(4096) -- so both pass 999 on ordinary input and theOperationalErrorlands mid-transaction, rolling the whole import back and answering 500 where it should have succeeded. Four sites are covered, the two source-keyed reads and the two group-keyed probes;_items_existalready chunked and now shares the one constant.Tests
Two new files,
test/test_knowledge_bundle_ownership.pyandtest/test_knowledge_bundle_host_backed.py, 161 tests between them, all passing (pytest -n0), covering the round-trip for all three tables, the uri-collision case, re-import idempotence, every row of the case table above, the membership predicate, the owns-live-items collision rule from both sides plus the blocked-items consequence, the double-ownership refusal, the mtime floor, the released claim, the preserved 400 contract, and a legacy bundle with the keys absent. Two further pins live beside the handler they cover, intest/test_knowledge_handlers_coverage.py.Four existing tests in this PR asserted behaviour these fixes disprove, and all four were inverted deliberately. Each is named here with what it asserted and why that assertion was wrong, because an inverted assertion inside a large test file is indistinguishable, to a reader scanning a diff, from a fix legitimising its own defect:
test_group_is_dropped_when_one_item_lands_under_another_source->test_an_item_landing_under_another_source_does_not_strand_its_sibling. Assertedownership_rows_imported == 0when one id of the group was already present locally. Wrong because that is the strand itself: the sibling this import DID write then has no row naming it, and only the document's own delete could ever have removed it. It now asserts the row is restored owning the sibling alone, that the already-local id keeps its own source and is not named by the row, and that the id left out is reported inwithheld.test_a_bundle_naming_an_absent_item_is_refused_BY_NAME->test_an_absent_item_is_named_and_left_out_of_the_group. Asserted, in its name and its docstring, that the whole row is refused when it names one absent id. Wrong because refusing the row discards ownership of the ids the import DID write, over an id it never had. The absent id is now named inwithheldand left out of the group while the arrived ids are owned; the assertions on thewithheldaccount are unchanged and one was added on the resulting group.test_item_naming_an_unknown_source_is_refused-- name kept, assertedmatch="FOREIGN KEY". Wrong because it made the foreign key the whole contract, and the foreign key cannot see the case that loses data: an undeclared id this store DOES hold satisfies the constraint, so the content is filed under an unrelated local source. It now expects the typedKnowledgeBundleErrornamingitems.source_id, and a sibling pin covers the locally-held id the foreign key would have admitted.test_location_naming_an_unknown_source_is_refused-- the same inversion forsource_locations.source_id, for the same reason.test_a_folder_row_does_not_exempt_a_stale_local_claim->TestTheSameRowExemptionCoversEveryStatedTable::test_a_folder_rows_own_group_exempts_the_chunk_it_names. Asserted that afolder_file_staterow could not exempt a claim on an id it names, because no row of that table is restored. Wrong because the exemption is keyed on the bundle's own row at the same(table, source, key), so what it forgives is this document meeting itself -- and for a file this host still has, that row owns the chunk correctly while nothing re-derives an id withheld there. A sibling pin now covers the rule the exemption is carved out of: a claim at a DIFFERENT key is still contested.Three fixtures were repaired rather than inverted, listed for the same reason: their assertions are untouched and only the bundles they feed became well-formed.
test_agent_source_ownership_durability.py's_imported_itembecame_imported_bundle(two call sites). It handedimport_bundlean item referencing a local source id with nosourcesentry, as a shortcut for "an independent writer commits mid-ingest". It now declares the aggregate source, which every producer ships, and the item still lands under the same source -- which is what those two pins are actually about.test_bundle_violating_foreign_keys_is_400_not_500andtest_client_error_carries_no_raw_exception_textboth reached the foreign key through an UNDECLARED source, an arm the typed refusal now takes first. Their bundles declare the source and let the missing ITEM trip the foreign key, so both keep exercising thesqlite3.IntegrityError-> 400 arm they exist for; confirmed against the store directly.The reproduction was written and run before the fix existed. Pre-fix the control passed --
items_imported == 1, the older bundle's chunk does arrive -- while 5 assertions inTestASecondBundleOwnsTheChunksItBringsfailed: the chunk is owned by nothing, the row names one id,remove_documentleaves it alive, it still answerssearch_items_fts("chunk two"), and one item exists that no ownership row names.Mutation-verified in BOTH directions, one mutation at a time, restored from a pristine copy held outside the repository and sha256-verified after each round, with no verdict accepted from a run that did not print an explicit pytest count. Baseline 153 passed in the ownership file; 7/7 caught for the original rule, and a further 6/6 for the three findings above -- one substitution site apiece, each verdict carrying an explicit pytest count.
The full revert reddens 5 of the 6 reproduction pins rather than all 6. The sixth is the control and stays green by design: the chunk still arrives either way, which is what makes the other five diagnostic instead of merely red.
Earlier rounds, with their pins still passing on this head:
Red on base, measured with a script driving the real ingest path rather than the unit fixtures:
96f4a157a1IntegrityError: FOREIGN KEY constraint faileditems_imported: 1, ownership_rows_imported: 1Mutation-verified, one mutation at a time with an exact restore after each; all twenty-six reddened their own pin and none another's:
created_atis not encodability-checkeditems.idis acceptedentities.idis left out of the identifier gateThree pins were weak on an earlier pass and were rewritten rather than accepted, which is worth recording because each hid a different mistake. The local-presence pin's scenario tripped the subset rule instead, so it stayed green under that mutation; it now puts an already-present item id under an unrelated local source, the only way a bundle item legitimately fails to land under the mapped source. The collision pin covered only the case where the local row owns items, so the empty-row side is now pinned separately from both directions. And once the double-ownership check went in, the subset pin went green under its own mutation because the new check caught its cases too -- the case only the subset rule catches is a bundle claiming pre-existing unowned local content, which is now what that pin asserts.
Existing suites, narrow selections on the files that reference this diff:
test_knowledge.py test_knowledge_handlers_coverage.py test_knowledge_embedding_space.py test_agent_source_ownership_durability.py -k "import or export or bundle or TestImportedFolderSourcesWaitForConfirmation", 103 passed. No existing test file is modified.Gates: flake8, isort, mypy, black (the new file is black-clean; the ratchet's baseline covers only pre-existing files), comment-history with its selftest, sync-io-in-async, brand and docs-lint all clean.
One red in
Backend Tests (Windows)is not from this diff:test_crew_log_crew_emitter.py::test_a_worker_report_through_the_route_records_the_answerasserts{'from': 3} == {'from': 1}. That file was added by61c08b1f71and this branch does not touch it;src/kiro_crew/knowledge/store.pyhas zero references tocrew_log,CrewLogoremit, so nothing here can write the session-log watermark it reads. It passes in isolation and the whole file passes alone (33 passed) behind its own autouseKIROCREW_HOMEfixture, so the leak is cross-file: another file in the same xdist worker advances the watermark withoutemit.reset_caches(). Adding a test file changes whole-file shard composition, so this PR co-scheduled it rather than caused it.Pattern harvest
Rule candidate: review-prompt
Pattern: a record crossing a store or host boundary carries a column that describes the SENDING side, and the receiving side reads it as a statement about itself. Both blocking findings on the first head of this PR were instances:
mtimedescribes a file on the exporting host and gates hashing on the importing one, andsources.iddescribes the exporting store's own row and was used to file items in the receiving store. The check is per column, not per record: when a table becomes part of an export format, every column has to be classified as travelling (a statement about content, like a content hash) or sender-local (a statement about one machine's filesystem or one store's ids), and a sender-local column must be re-derived or floored on import, never carried.folder_file_stateis the worked example --content_hashandtext_hashtravel,mtimeandattemptsdo not.Second candidate, from the review round after that: an identity column that becomes a dictionary key needs a type gate at the writer, not only a column type.
sources.idandsources.uriare TEXT in SQLite and therefore trusted as strings, but a JSON bundle can carry a list there, and theTypeErrorsurfaces as a 500 rather than the endpoint's typed 400.Any other suggestions on the work
The decision to split, and the options rejected, are recorded on the issue. The short version: half 1's stated root cause is already closed on
main(ingest_artifactwrites the ownership row inside the pipeline's finalize hop viaon_committed), so only residue repair remains, and residue repair is unsafe until imported items arrive owned. Rejected: both halves in one PR, which re-creates the shape that was reverted and puts a deletion sweep beside an import fix; residue repair first, already reverted for this reason; and splitting the remap into its own PR ahead of this one, which cannot work because the state tables are keyed onsource_id.One finding for the follow-up PR, so it does not have to be re-derived: the remaining orphan window is narrower than #2670 describes but not closed. The item chunks, the old group's
delete_items_batch, and the state-row write are three separate commits on an autocommit connection, so a hard kill between them still leaves committed items with no row naming them. A cancellation cannot land there any more; a process kill can.One residual of the same class is deliberately left in place rather than widened into: the same-source check is still all-or-nothing, so a bundle that files one group id under a different source refuses the whole row and leaves the row's other arriving ids owned by nothing. This store's own
export_allcannot produce that shape -- it prunes each group to ids filed under the row's own source -- andtest_a_row_cannot_own_an_item_filed_under_a_different_sourceencodes the refusal with its own reasoning, so partially trusting a bundle that contradicts itself about who owns an id is a separate decision from the strand fixed here.The scoped-export disclosure that Design Review and First Principles both raised is fixed in this PR rather than deferred: shipping rows the importer is guaranteed to drop is not a conservative choice, it is a new per-document disclosure in an artifact whose purpose is to leave the machine.
Refs #2670