Skip to content

fix: Add durable maintenance adapters and transient deferral - #803

Draft
buildingjoshbetter wants to merge 3 commits into
fix/cluster-outer-publication-753from
fix/maintenance-adapters-753
Draft

buildingjoshbetter wants to merge 3 commits into
fix/cluster-outer-publication-753from
fix/maintenance-adapters-753

Conversation

@buildingjoshbetter

Copy link
Copy Markdown
Owner

Scope

Adapter checkpoint for #753, stacked on #800. Enroll all eight persisted maintenance layers with durable coverage and explicit borrowed-transaction support. Production engine scheduling is not routed here yet. Includes #801's MCP/Pydantic dependency correction for current installation gates.

Changes

  • Preserve the original six standard-library adapters and add connection-aware clustering plus Dunbar generation/ownership coverage. Keep execution outcome, source freshness and successful coverage separate.
  • Cluster dependency probes read package metadata and bounded schema/registry metadata without importing native packages, loading models, scanning vectors or waiting for the model lock. Ordinary vector progress timestamps/counts do not reset the 25-append threshold. Changed model/schema/generation, missing dependencies and incomplete or malformed descriptors remain distinct.
  • Treat temporary model contention as ephemeral deferral. Initial contention, including force, performs no SQL writes or fabricated attempt. Mid-attempt contention rolls back output and conditionally restores every prior checkpoint field or prior absence. Changed owner/diagnostic, rollback failure or restoration failure propagates instead of claiming successful deferral.
  • Use a typed transient cluster exception. Initial clustering acquisition is nonblocking inside a transaction, closing a lock-order window when the caller already holds SQLite's writer. Existing standalone capture can still wait before opening a transaction.
  • Normalize SQLite Row values before dependency hashing. Invalid metadata is still rejected.
  • Migrate successful coverage atomically, defaulting legacy rows to unverified while preserving existing source epochs/provenance. Dunbar counts only verified owned output and reports legacy unowned contacts separately. Cluster generation coverage remains explicitly unverified.
  • Require explicit opt-in for a caller-owned transaction. Use protected random per-layer savepoints, retain caller work, keep output/checkpoint/coverage together, and report pending caller commit. Final cluster guards retain model ownership through the layer RELEASE. The caller still owns the later outer COMMIT.

Reproduced review failures

Review caught model contention resetting the 25-append baseline after one append, Row-factory incompatibility, and an initial model-lock wait inside a caller transaction. Corrected controls cover initial/forced/mid-compute/final-publication contention, whole-row restoration, absent rows, ownership replacement, cancellation and rejected rollback/restoration. A plain RuntimeError containing busy-like text remains a real failure; exception text does not select deferral.

Validation

  • 47 focused adapter tests plus 166 existing synthetic regressions passed. Root independently reran all 47 adapter tests.
  • Five mutation controls each produce one assertion failure and zero errors. The initial-lock regression separately fails both owned and borrowed cases before correction and passes afterward.
  • Exact-source installed-wheel GPUBox suite at 5de4cdc72ae32099f981765abb5443366db89105: 2,442 passed, 22 skipped, 3 deselected; 573 subtests passed.
  • Full-suite log SHA256: 816a2a2faf7cc318a8605bd59a2be822789ebe8e31e53eaa123f4c16e1454c08.
  • The repository's ruff check truememory/ tests/ CI gate, Python 3.10 syntax, whitespace, staged-content and commit-identity scans passed. GPUBox source hashes matched the reviewed local source. Hosted matrix remains pending.
  • An additional whole-repository Ruff check found 253 existing findings in unchanged benchmarks/scripts, outside the configured CI lint scope. Those were not changed or represented as passing.

Remaining work and limits

Engine startup/foreground wake routing, durable shared scheduling, preference extraction and stale getter policy remain separate checkpoints. This patch does not activate automatic processing or close #753.

A caller retaining an existing writer can retain it during computation. Borrowed cluster coverage does not certify arbitrary later writes or the final caller commit. The owned guard's initial capture can still wait outside its database transaction. Dependency metadata checks are not certification of every legacy vector writer.

The separate custom-schema collation/replacement follow-up remains open; canonical maintenance/FTS correctness on those schemas is not certified here. Standalone rebuild bridging belongs to #802.

All tests use generated inputs and controlled dependency boundaries. No native model-performance, whole-process memory or MPS/thermal improvement is established by this suite. Models, rerankers, precision, dimensions and retrieval depth are unchanged. No personal memory database, transcript or credential is included. No main merge, release or issue closure.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant