Skip to content

fix: Add bounded tier worker and verified background handoff - #824

Draft
buildingjoshbetter wants to merge 2 commits into
fix/tier-streaming-prereqs-publish-v1from
fix/tier-worker-job-publish-v1
Draft

buildingjoshbetter wants to merge 2 commits into
fix/tier-streaming-prereqs-publish-v1from
fix/tier-worker-job-publish-v1

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Adds an opt-in bounded tier rebuild worker and a verified background database handoff. The public manager still uses its existing full-list path; this prerequisite does not yet reduce production tier-switch memory or activate a new tier.

The worker consumes one certified source page plus bounded prefix buffers, prepares the frozen target outside SQLite transactions, and commits completion/separation vectors with the paired receipt. It preserves the existing text construction, serialization, model choices and OOM retry bound. Its result distinguishes committed progress, captured-range completion and current-source completion; activated remains false. Ambiguous publication failures propagate instead of returning an obsolete cursor.

The frozen job descriptor carries explicit target and database/source identity without retaining a corpus, connection or ownership token. A worker acquires its own maintenance owner, reopens with existing-only mode=rw, checks identity before yielding, and closes/releases on interruption. Stable pathname and known-current caller association remain required. Durable selected-job binding and certified activation are subsequent work.

Dependencies: #822 and #823. The base branch fix/tier-streaming-prereqs-publish-v1 combines those already-published runtime/source prerequisites. This PR adds only five files across two atomic commits; no unrelated private integration changes are included.

Validation:

  • Sequential isolated GPUBox installed-wheel gates with real sqlite-vec and the opt-in file-backed job test: combined 3,189 tests / 2,733 subtests; publication 3,057 tests / 2,088 subtests. Each had 17 explicit skips and 3 deselections, exit 0, no timeout or OOM kill.
  • Independent worker review: 45 probes; a 1,025-row source with page size 7 retained at most 14 source texts. This is a row-retention bound, not a byte bound or process-memory measurement.
  • Independent job review: 30 probes, plus 10 separate root probes. Prior worker methods and assertions remain unchanged; the job fixture's Windows path correction preserves all 104 assertions.
  • Ruff, Python 3.10 syntax, source/manifest/remote hashes and public diff/commit-identity privacy checks passed.

Publication full log SHA256: a497ad44348a8387fce21e8110812f97867af0900ddccd30801d2f34865763ca.
Combined full log SHA256: e8c2699392e5c85ca2514cb90a5e09c2f007ab71077931f1038d623acf7237d6.

Cancellation is cooperative at call boundaries; it cannot interrupt internal native or SQLite waits. Path observations cannot prove SQLite file-descriptor association, ABA replacement or same-epoch clones. The existing public route, selected-job lifecycle, final source fence, configuration/runtime reconciliation, inactive legacy reset and active-force staging still need work. Original Mac unified-memory improvement and native allocator reclamation remain unproven.

Refs #756, #795, #752, #297. No issue closure or release claim.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant