fix: Stream rebuild source pages with publication fences

[buildingjoshbetter]

Scope

First checkpoint for #756, stacked on the source-revision/maintenance foundation #790. Covers standalone completion/separation builders and their foreground publication boundaries. Tier-switch manager/worker corpus lists and active-tier configuration publication remain separate work under #756 and #795. Includes the MCP dependency correction from #801 for current installation gates.

Reproduced problems and changes

• Replace database-source full-message materialization with ascending-primary-key pages. Capture source epoch/revision, upper ID and count; release each owned read snapshot before inference.
• Persist an input-consumed cursor independently of emitted vector rows, so zero/nonfinite skips cannot corrupt restart progress. Bind restart to model, dimension, target, schema, source identity and generation.
• Stage bounded groups, then publish vectors and progress atomically. Acquire the SQLite writer before the nonblocking model fence and retain both through terminal COMMIT/RELEASE.
• Preserve explicit-list order and caller transaction ownership. Concurrent high-ID appends stay outside captured input; edits, deletes, replacements and changed source/schema identity reject an unfinished generation.
• Fence foreground add/update and embed_single against late model or target changes. Add rollback preserves unrelated caller writes. Update retains its existing committed-source boundary and reports retry-required vector publication truthfully.
• Restore legacy/minimal-schema API compatibility with a rebuild-only source bridge. Exact trusted built-in schemas reuse canonical tracking. Custom schemas compare storage type and bytes, avoiding NOCASE/RTRIM blind spots; unique indexes use conservative invalidation for hidden REPLACE deletions. Changed/missing triggers get a fresh epoch, never an invented certificate over an untracked interval.

Memory bound

Serialized staged payload is bounded by max(1, txn_batch) * native_batch_size * dimension * 4 bytes per resident index. Existing defaults at 256 dimensions yield 100 * 16 * 256 * 4 = 1,638,400 bytes on the MPS path and 100 * 100 * 256 * 4 = 10,240,000 bytes on the CPU path. Python objects, one source page, model/native allocations and caller-provided lists remain additional. One large record is not truncated. This is not a whole-process memory cap or measured Mac saving.

Validation

• 78 focused synthetic tests passed. A separate actual sqlite-vec/legacy-MPS-regression run passed 85 tests with 34 subtests.
• Final exact-source installed-wheel GPUBox suite, with actual sqlite-vec cases enabled: 2,397 passed, 22 skipped, 3 deselected; 507 subtests passed.
• Full-suite log SHA256: 519fb87f7946c222da572bbfb004c2861337a9b7f3dc025f93e6ac83f25df8f8.
• Independent review and concrete controls caught transaction ownership, minimal-schema compatibility, indexed-primary-key replacement and collation defects. Those failures remain recorded; corrected controls pass. One earlier proposed formatter finding was withdrawn after checking the real formatter.
• Existing issue MPS GPU out-of-memory during embedding — silent data loss (missing separation vectors) #459 regression tests were not modified. Synthetic tests inject model boundaries; actual sqlite-vec verification does not establish native MPS behavior.
• Ruff, Python 3.10 syntax, changed-source hash verification, and new-diff/commit-identity scans passed. Test source loaders explicitly use UTF-8 after reproducing a Windows default-encoding failure.
• Combined branch 4fdcb50fa056f17eac3ce6c792a5e798b8be890d passed 2,597 tests and 1,169 subtests with actual sqlite-vec rebuild cases enabled, 22 skips and 3 deselections. Ruff passed.
• Hosted Python 3.10 exposed a fault-injection defect: its SQLite statement cache can reuse an earlier RELEASE without rerunning the conditional authorizer. The previous callback also queried its invoking connection, contrary to SQLite's authorizer contract. A test-only follow-up arms a pure real authorizer immediately before the target RELEASE. Cache sizes 0, 5 and 100 now each verify actual denial, rollback, preserved caller/target data and zero model work. Production is unchanged; 78 focused synthetic tests pass. The original cached fixture remains a failing negative control. Hosted verification is rerunning on the new head; it is not yet a pass.

Primary behavior references: CPython 3.10 statement cache, SQLite authorizer contract. The full native-suite count above belongs to the preceding source head; the new correction changes only the test fault injector.

Limits

A caller-owned transaction retains its own locks and final durability responsibility. Explicit input lists must not mutate concurrently. The bridge certifies only rebuild input; canonical maintenance and FTS behavior on custom collations are not fixed by this patch. Other derived-state invalidation and tier activation remain open.

No model, reranker, precision, dimension or retrieval-depth change. No personal memory database, transcript or credential in test inputs. No live Mac change, main merge, release or issue closure. Issue #756 remains open.