fix: Track exact source edits and hidden replacements

docs/resource-budgets.md

@@ -787,3 +787,210 @@ caller's later outer COMMIT. Other adapters with an owned publication guard must
 supply an explicit borrowed guard or report unavailable. This checkpoint does
 not route engines or ingest, change algorithms or establish a measured latency,
 memory or throughput improvement.
+
+## Standalone rebuild source streaming (issue #756, first checkpoint)
+
+`build_vectors()` and `build_separation_vectors()` read database-backed inputs
+in ascending-ID pages, using the existing native batch size. They capture a
+fixed upper ID, row count and A10 source revision, including its epoch. Each
+page read ends before inference. Publication checks the source revision,
+model generation, target manifest and database schema version under writer
+ownership. Writer acquisition precedes the nonblocking model fence; that
+fence remains held through the terminal COMMIT or SAVEPOINT RELEASE. A source correction, deletion, lower-ID insertion, epoch change or
+schema change rejects the unfinished generation. An unrelated schema change
+is conservatively invalidating. New high-ID source rows remain outside the
+captured range.
+
+A versioned metadata manifest contains model/dimension/target identity,
+revision counters, the last **consumed input ID**, consumed count and inserted
+vector count. It contains no source text. Skipped zero/nonfinite vectors still
+advance consumed input, so restart cannot infer progress from `MAX(rowid)`.
+A saved manifest without matching identity and revision is restarted once on
+the next explicit call; the builder does not run an automatic restart loop.
+Legacy boolean markers alone are insufficient resume evidence.
+
+The native batch size and `txn_batch` durability cadence remain configurable.
+Each group is encoded and serialized before its owned writer transaction.
+Serialized payload is bounded by
+`max(1, txn_batch) * batch_size * dimension * 4` bytes per resident index,
+plus Python objects, one input page, the current native output and allocator
+memory. Defaults at 256 dimensions give:
+
+- CPU: `100 * 100 * 256 * 4 = 10,240,000` bytes (`9.765625 MiB`).
+- MPS: `100 * 16 * 256 * 4 = 1,638,400` bytes (`1.5625 MiB`).
+- Custom 384-dimensional CPU example: `100 * 100 * 384 * 4 = 15,360,000`
+  bytes (`14.6484375 MiB`).
+
+Completion and separation standalone calls stage their groups separately.
+This bound is independent of corpus rows with fixed controls, but can increase
+staging for small corpora compared with immediate insertion. It is not a
+whole-process memory cap. A single record is not truncated and can itself be
+large. Native model residency and thermal behavior are not measured by the
+synthetic regression tests.
+
+One maintenance owner covers the complete operation. Same-thread nesting in
+an existing A10 maintenance worker reuses its owner. A caller's open SQLite
+transaction is preserved with savepoints; the rebuild cannot release locks
+that the caller already owns. Such callers also retain control of durability.
+Otherwise, no rebuild-owned read or writer transaction remains open during
+inference. Short page snapshots avoid pinning a job-long WAL reader; they do
+not promise globally bounded WAL when other readers exist.
+
+Explicit `messages` lists preserve caller order and text. A streamed digest
+binds restart to that exact list without making another corpus-sized copy;
+the consumed list position preserves out-of-order IDs. The caller's allocated
+list is outside the database paging bound. Concurrent mutation of a supplied
+list is not supported. Database-backed calls reuse the canonical source
+tracker only with the exact built-in `messages` definition and verified tracking.
+Older connections, including deprecated
+`engine.open()` and the minimal five-column separation schema, instead install
+a rebuild-only bridge when canonical tracking is unavailable. Completion also
+supports a legacy table containing only `id INTEGER PRIMARY KEY` and `content`.
+
+The bridge adds one namespaced state table and three source triggers. It does
+not add columns to `messages`, migrate other storage features, or mark the
+canonical maintenance tracker ready. Installation and repair own the writer
+before model loading or clearing vectors; caller transactions use savepoints.
+The token explicitly identifies its tracker, epoch, covered source fields,
+table schema and index definitions. Page/publication checks verify that source
+schema and the exact trigger definitions; missing or altered tracking starts
+a new epoch on the next call, invalidating untracked intervals. These checks
+read schema metadata and scalar counters, not corpus text. Read-only legacy
+connections and unsupported source schemas fail before model allocation.
+Bridge updates compare each covered value's SQLite storage type and bytes,
+so custom `NOCASE`/`RTRIM` collations cannot hide changed encoder input.
+Other table definitions retain the bridge even after a full storage migration
+reports canonical tracking ready. This conservative definition whitelist avoids
+inferring equality semantics from arbitrary SQL; compatible custom definitions
+remain usable with the additional bridge counters.
+
+Same-ID replacement invalidates a bridge generation even with recursive
+triggers disabled. Custom schemas with unique indexes, including indexed primary keys that
+are not rowid aliases, use a conservative bridge: every successful insert or update invalidates the
+captured prefix. This includes expression and partial unique indexes, where
+`OR REPLACE` can silently delete another row without a delete trigger. Such
+schemas remain supported, but concurrent high-ID appends cannot preserve a
+running generation. They retain this bridge even if the canonical tracker
+reports ready, because its append certificate does not cover those hidden
+replacement deletions. There is no repeated automatic restart loop.
+
+When the trusted built-in definition and canonical tracking become suitable,
+the next rebuild changes tracker
+identity and starts a fresh generation; it atomically retires only the bridge's
+owned triggers. Its inactive state table remains. The bridge never certifies
+the readiness of other maintenance layers. Normal canonical schemas need no
+bridge table, triggers or additional writer commit during setup.
+
+Foreground engine add/update and `embed_single` bind inference to the active
+model generation. Same-model appends above an in-progress database range are
+published normally and cannot advance its consumed cursor. Initial clear and
+source capture share one writer transaction, so a committed earlier append
+joins the captured range. A correction within the range publishes its current
+vector and invalidates the rebuild's source fence. A late model mismatch is
+reported categorically: add rolls back its uncommitted source row; update's
+existing storage helper has already committed its source, so its error says
+to retry vector publication. Explicit-input or unattested in-progress targets
+reject concurrent publication rather than silently reporting success.
+
+Foreground publication acquires the SQLite writer before its nonblocking
+model fence, then validates the target manifest and retains both owners
+through the owned commit or caller savepoint release. Inference finishes
+before either is acquired. Add source insertion shares that transaction;
+rejection rolls back only the new add, preserving unrelated pending caller
+writes. Successful add and `embed_single` calls inside an existing caller
+transaction also leave durability to that caller. Update retains its existing
+source-helper commit behavior; the subsequent vector publication is fenced
+separately and cannot claim the source update was rolled back.
+
+This checkpoint does **not** remove the tier-switch manager/worker's retained
+message lists or solve tier configuration publication across SQLite, config
+file and process state. Issue #756 remains open for that next checkpoint.
+No global embedder-compatibility authority, models, precision, dimensions,
+reranker selection, retrieval depth or deployment defaults change here.
+
+### Legacy rebuild value precision
+
+The rebuild bridge also compares values with explicit `BINARY` equality.
+SQLite's conversion of a REAL value to BLOB formats it as text and can lose
+precision, so the byte and storage-type checks alone cannot certify an
+unchanged source. Numeric comparison supplements both existing checks.
+
+SQL numeric equality cannot distinguish stored positive and negative REAL
+zero. A source UPDATE involving a covered REAL zero therefore invalidates
+the generation conservatively, including assigning the same zero again.
+For ordinary source columns, the trigger listens only to covered source
+fields and unshadowed `rowid` aliases. Unrelated derived-only updates do not
+invalidate solely because a source field contains zero. When a covered
+source field is generated, the trigger must listen to all updates because
+its dependencies can be other columns. Custom UNIQUE schemas retain their
+existing all-update invalidation for hidden replacement deletions.
+
+Changed trigger definitions fail the existing identity check. The next
+writer-owned repair starts a fresh epoch before model loading or clearing
+the target, so manifests captured under the earlier comparison cannot
+resume. Synthetic SQLite tests cover precise REAL metadata rendered into
+separation text, signed zero, collations, storage types, rowid aliases,
+generated fields, repair failures and caller rollback. These tests do not
+measure native model behavior or add a process memory budget.
+
+### Canonical source comparisons and FTS replacement cleanup
+
+Canonical maintenance tracking and FTS synchronization compare storage types
+and values with explicit `BINARY` collation. Declared `NOCASE` or `RTRIM`
+collations cannot hide source edits, and adjacent REAL values do not depend on
+SQLite's lossy REAL-to-text conversion. SQL equality cannot distinguish retained
+positive and negative REAL zero, so an explicit source UPDATE involving REAL
+zero invalidates conservatively, including assigning the same zero again.
+
+For ordinary columns, comparison triggers listen to their covered fields and
+unshadowed `rowid`, `_rowid_`, and `oid` aliases. Default derived-only updates
+therefore remain clean. A generated source field can depend on any column, so
+its comparison trigger listens to all updates; REAL zero in such a field can
+conservatively invalidate an otherwise-derived update. The rebuild bridge keeps
+its separate, existing schema contract and canonical-reuse restriction.
+
+Any UNIQUE index on messages selects conservative source tracking dynamically:
+every INSERT fences append-only reuse, and every UPDATE is a correction. This
+includes ordinary, expression, partial and composite UNIQUE indexes. It also
+includes writes that do not delete a conflicting row. Default-schema inserts
+remain append-only, and their derived-only updates remain clean. Hidden deletes
+from `INSERT OR REPLACE` or `UPDATE OR REPLACE` cannot escape this fence when
+`recursive_triggers` is disabled.
+
+Separate FTS cleanup triggers run only when messages has a UNIQUE index. They
+remove indexed row IDs absent from messages. Their trigger-level guard avoids a
+content scan on the default schema. A custom UNIQUE schema pays an O(N) orphan
+scan for each inserted or updated message, where N is its FTS row count. Source
+certification does not depend on FTS completeness or trigger execution order.
+
+These canonical guarantees require `messages.id` to be an ordinary INTEGER
+PRIMARY KEY rowid alias. An indexed `INTEGER PRIMARY KEY DESC` can retain NULL
+and REAL IDs; a WITHOUT ROWID INTEGER primary key can retain REAL IDs. FTS may
+allocate an unrelated rowid for NULL or reject a REAL ID. Unsupported IDs leave
+canonical source tracking unavailable, and FTS repair is skipped with a warning;
+the database remains readable. Missing source or indexed fields likewise retain
+the degraded legacy path. This checkpoint does not change the rebuild bridge's
+broader ID support or establish FTS synchronization for these unsupported schemas.
+
+Changed source-trigger definitions rotate the epoch and mark maintenance layers
+pending. Changed or missing owned FTS definitions are repaired and the index is
+refilled from current messages in one transaction. The refill repairs stale,
+missing and orphan indexed rows. It happens once per definition repair, not on
+ordinary repeated opens. A caller transaction retains its writes and ownership;
+rollback restores both trigger definitions and indexed contents. The runtime
+must provide FTS5, `table_xinfo`, and table-valued `pragma_index_list`.
+
+Synthetic SQLite 3.50.4 progress-handler measurements were identical at 100 and
+10,000 source rows: 289 virtual-machine steps for append, 232 for a derived-only
+update, and 15 for revision read. The row-count ratio is 10,000 / 100 = 100;
+each measured step ratio is 1. These counts cover the tested SQL paths, not
+native inference, overall database-open cost, memory use, or UNIQUE schemas.
+An authorizer-based regression also rejects any source/FTS content read during
+ordinary repeated tracking initialization and FTS migration.
+
+Interval trust remains a separate unresolved contract. A writer can drop
+tracking triggers, mutate messages, and recreate identical definitions without
+the current definition checks exposing that interval. Arbitrary direct FTS
+mutations can also defeat synchronization. This checkpoint adds neither a
+schema-cookie fence nor a readiness gate to FTS search, and does not certify
+either kind of external mutation.

tests/test-episode-index-writes-751.py

@@ -202,9 +202,11 @@ def test_existing_database_migration_preserves_data_and_is_idempotent(self) -> N
         opened = self.open_db()
         self.assertEqual(opened.execute("PRAGMA schema_version").fetchone()[0], version + 2)
         self.assertEqual(self.snapshot(), before)
-        self.assertEqual(self.audit(), {})
+        self.assertEqual(self.audit(), {"fts_delete": 1, "fts_insert": 1})
+        self.clear_audit()
         reopened = self.open_db()
         self.assertEqual(reopened.execute("PRAGMA schema_version").fetchone()[0], version + 2)
+        self.assertEqual(self.audit(), {})
         reopened.execute("UPDATE messages SET episode_id = 5")
         reopened.commit()
         self.assertEqual(self.audit(), {"message_update": 1})