Skip to content

fix: Prevent maintenance from rejecting foreground writes - #806

Draft
buildingjoshbetter wants to merge 1 commit into
fix/rebuild-value-exactness-756from
fix/foreground-model-admission-publish-753
Draft

buildingjoshbetter wants to merge 1 commit into
fix/rebuild-value-exactness-756from
fix/foreground-model-admission-publish-753

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Scope

Foreground admission prerequisite for #753, stacked on #805. Prevent background model-lock ownership from rejecting a valid foreground add after it has already acquired the SQLite writer. This PR does not activate the new scheduler or close #753.

Changes

  • Owned add and incremental vector publication acquire model identity admission before opening the SQLite write transaction and retain it through commit or rollback.
  • Add always acquires the engine write lock before model admission. Add/update also classify model capture under that engine lock, preventing another same-engine mutation from being mistaken for a borrowed caller transaction. Encoding runs after the capture lock releases.
  • Caller-owned transactions use nonblocking capture and accept only a protected cached model. Busy ownership or a missing model returns typed deferral without loading a model or silently accepting a source-only write.
  • Genuine model-generation changes still reject publication. Update retains its existing distinction between capture failure before source mutation and later vector failure after the source update committed.
  • Preserve nonblocking model fences for background writers and streamed rebuild publication.

Reproductions and review

An event-driven test executes the actual clustering snapshot and add paths: add waits without SQLite writer ownership, an independent connection can write, then source and both vectors commit after clustering releases the model lock. Other controls cover a background writer yielding model admission, same-engine add/update ordering, borrowed sentinels, stale generations and lock release after SQLite failures.

The first full suite exposed a second genuine race: connection-wide transaction state was inspected outside the engine lock. The correction serializes capture classification only. The original concurrent-add and concurrent-consolidate tests remain unchanged. Its new add/update negative control fails twice before correction and passes afterward.

Two existing publication tests now require the intentional new model-before-writer order while retaining commit, RELEASE, rollback and output assertions. The cached-model fixture now represents a loaded model. A source-position assertion for style computation was replaced by AST containment/order checks and a runtime test proving style computation occurs outside engine/model/SQLite ownership and passes the exact computed object into publication. Independent mutation controls reject style computation inside a lock and duplicate computation.

Validation

  • 14 new runtime tests passed. Independent review also passed the existing foreground and applicable streaming tests, plus lock-cleanup probes.
  • Exact-source GPUBox suite, actual sqlite-vec enabled: 2,425 passed, 22 skipped, 3 deselected; 1,072 subtests passed.
  • Full-suite log SHA256: 53d7e66cc34feb5f24963d8a25206e6c85d441930f5fb41debad65648d9f8883.
  • Configured Ruff, Python 3.10 syntax, whitespace, content privacy and commit-identity checks passed. Earlier failed-run evidence is retained. Hosted checks are pending.

Limits

Waiting before writer ownership is not a latency bound; cluster snapshots can still retain the model lock while scanning source data. Cached inference can still run inside a caller-owned transaction. The borrowed fence covers savepoint release, not a later caller-controlled outer commit. If SQLite rejects rollback itself, pending changes may remain; the connection owner must recover that transaction.

All test inputs are generated. No personal memories, transcripts, credentials or local identity are included. No native performance, memory-budget or MPS/thermal improvement is established. Models, rerankers, dtype, dimensions and retrieval depth are unchanged. No main merge, release or issue closure.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant