Skip to content

fix: Track exact source edits and hidden replacements - #807

Draft
buildingjoshbetter wants to merge 7 commits into
fix/maintenance-adapters-753from
fix/canonical-source-fts-publish-753
Draft

buildingjoshbetter wants to merge 7 commits into
fix/maintenance-adapters-753from
fix/canonical-source-fts-publish-753

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Scope

Source/FTS correctness follow-up for #753 and #751, stacked on #803. Includes #805 through a normal prerequisite merge. This fixes tracked comparisons and hidden replacement deletions; schema-interval trust remains a separate checkpoint.

Changes

  • Compare source storage types and values with explicit BINARY collation, so NOCASE/RTRIM declarations cannot hide edits and adjacent REAL values are distinguished.
  • Conservatively invalidate retained REAL-zero source assignments. Ordinary derived-only updates stay clean; generated source fields listen to all updates because their dependencies can be other columns. Unshadowed rowid aliases remain covered.
  • Detect UNIQUE indexes dynamically. Every INSERT/UPDATE on a custom UNIQUE schema fences append-only reuse, including writes without a conflict. This safely covers hidden deletions from INSERT/UPDATE OR REPLACE when recursive triggers are off.
  • Use separate trigger-level-guarded FTS orphan cleanup. Default schemas skip its scan body. Source certification does not depend on FTS completeness or trigger order.
  • Repair changed/missing owned FTS definitions and refill from current messages transactionally. Repeated opens do not repeat that refill. Changed source definitions rotate the epoch and mark layers pending.
  • Unsupported source IDs leave tracking unavailable and skip FTS repair with warnings, while existing databases remain readable. The rebuild bridge retains its separate schema contract.

Validation

  • 21 new synthetic SQLite tests, including 48 UNIQUE/index/recursive-trigger/order/operation combinations, plus multiple-victim and same-ID replacements.
  • Baseline counterfactuals produced 21 behavioral assertion failures and zero setup errors. Root independently reran the new suite.
  • Source revision, runner, publication, rebuild-streaming and 25 episode-index cases pass. Existing canonical assertions now require invalidation for newly covered edits; bridge, output, progress and rollback assertions remain intact.
  • The existing migration test now requires exactly one delete/refill, unchanged source/index contents, and zero repeated-open writes. Its prior zero-first-repair expectation conflicts with correcting potentially stale legacy contents.
  • At 100 versus 10,000 rows, measured SQLite VM steps are identical: append 289, derived update 232, source read 15. The row-count ratio is 10,000 / 100 = 100; each measured step ratio is 1. An authorizer test also rejects source/FTS content reads during ordinary repeated initialization.
  • Integration-based exact-source suite: 2,697 passed, 22 skipped, 3 deselected; 1,849 subtests passed. Log SHA256: d62840b57ea8b5b9d04778571618f76680112bd401cd11a40fcf8ffa50f72e6b.
  • Isolated publication-branch suite with actual sqlite-vec enabled: 2,581 passed, 22 skipped, 3 deselected; 1,235 subtests passed. Log SHA256: 46a0426562cdd71f4e8c248ecf4a13fefefcbe630797dee70e4a214887050fd3.
  • Configured Ruff, Python 3.10 syntax, whitespace, content privacy and commit-identity checks passed. Hosted checks are pending.

Limits

Custom UNIQUE schemas pay an O(N) FTS orphan scan on each insert/update and conservatively invalidate even nonconflicting writes. Default-schema append/derived-only behavior remains cheap. These measurements concern tested SQL paths, not whole-app memory or inference latency.

Canonical guarantees require an INTEGER PRIMARY KEY rowid alias. Indexed descending primary keys and WITHOUT ROWID keys can retain FTS-incompatible IDs; their synchronization and writes are not certified here. Missing indexed/source fields retain the degraded legacy path.

Identical trigger drop/mutate/recreate intervals and arbitrary direct FTS mutations remain outside these guarantees. No schema-cookie fence or search readiness gate is added. No model, reranker, precision, dimension, retrieval-limit or release change. All inputs are generated; no personal memories, transcripts or credentials are included. No issue closure or native MPS/thermal claim.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant