Skip to content

fix: Preserve raw sums for exact style profile updates - #809

Draft
buildingjoshbetter wants to merge 4 commits into
fix/canonical-source-fts-publish-753from
fix/l0-style-accumulator-publish-747
Draft

buildingjoshbetter wants to merge 4 commits into
fix/canonical-source-fts-publish-753from
fix/l0-style-accumulator-publish-747

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Scope

Raw-sum foundation for #747, stacked on #807 with the atomic L0 builders from #794 included by a normal prerequisite merge. This draft is preparatory: legacy-profile migration and maintenance routing are still required before release or issue closure.

Problem and correction

The incremental updater multiplied an already normalized profile by its count. Normalization discards the magnitude needed for an exact running sum. For vectors [1, 0], [0, 1], [0, 1], the old incremental result was approximately [0.505449, 0.862856]; batch arithmetic produces [0.447214, 0.894427].

  • Persist 256 little-endian float64 sum components (256 * 8 = 2,048 bytes), a format version and the existing count. Normalize only the exposed profile using the existing divide-then-normalize arithmetic.
  • Preserve the 256-dimensional character-style algorithm, existing profile JSON and getter behavior. No embedding model, reranker, inference precision or retrieval limit changes.
  • Serialize accumulator read-modify-write. Incremental success remains pending in the caller's transaction; failures roll back only owned work.
  • Publish batch profiles, raw sums and versions atomically under the existing source-validation fence. Empty generations retain the same full-replacement behavior.
  • Add columns transactionally without upgrading legacy rows. Legacy, invalid or unsupported accumulators retain their readable profiles and raise a categorical rebuild requirement on append. Raw sums are never reconstructed from normalized profiles.

Validation

  • 31 focused tests passed. The existing algorithm and source-validation functions were checked for unintended changes; one existing incremental test was tightened to compare the actual full-batch result.
  • Independent production review passed 92 in-memory assertions, including negative controls for the original arithmetic bug, non-atomic schema migration and partial batch publication.
  • Test portability probes passed 93/93 cases across SQLite statement-cache sizes 0, 5 and 100 with simulated Python 3.10 API and CP1252 restrictions. Failure injectors were corrected after proving cached statements could bypass the original authorizer condition. Existing release gates were retained.
  • Integration-based candidate: 2,742 passed, 22 skipped, 3 deselected; 1,908 subtests passed, with actual sqlite-vec enabled. Log SHA256: b95fda3983a90643f7d145d58703f4f48957dead264cbada5629819f72150787.
  • Exact isolated publication branch: 2,631 passed, 22 skipped, 3 deselected; 1,323 subtests passed, with actual sqlite-vec enabled. Log SHA256: eeac59b7ad372053b78c61dbc16fed8f0a1c71add9d398660d537f8ce763fc35.
  • Configured Ruff, Python 3.10 syntax, whitespace, new-content privacy and commit-identity checks passed. Hosted checks are pending.

Release requirements and limits

The migration bridge must rebuild legacy sums from source, preserve the old generation until atomic publication, and avoid appending a triggering message twice when the rebuild snapshot already contains it. Normal startup must not synchronously scan the corpus. Dirty source coverage and failed rebuilds need explicit pending/degraded state. These integration steps are not implemented by this draft.

Arithmetic equivalence is established for the same ordered vectors. Different message accumulation orders are not certified as bit-identical. No native latency, process-memory, MPS or thermal improvement is claimed. Only generated synthetic test data was used; no personal memories, transcripts or credentials. No main merge, issue closure or release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant