Repository navigation
perf: stream rebuild inputs instead of retaining the full message corpus before batching #756
Description
A17: perf: stream rebuild inputs instead of retaining the full message corpus before batching
- Priority: P2
- Evidence: code-confirmed
- Audited source:
063e5b8844af735a52fde886217a5d26a0f13064 - Audit date: 2026-10-08
- Disposition: new
Problem
Rebuild functions encode in batches but first fetch every message and build an additional list of dictionaries. Tier rebuild retains that list throughout the job. Clustering similarly materializes blobs, per-row arrays, stacked and normalized matrices. Encoding batch size therefore does not bound host-side corpus materialization.
Advertised or expected contract
- https://arxiv.org/html/2605.04897v1#S5
- Long-horizon memory and bounded local resource use; no claim that these allocations explain26.24GB.
Source evidence
- Source: Full completion rebuild input fetched before batching.
- Source: Full separation rebuild repeats materialization.
- Source: Tier full/delta queries fetch all pending message text.
- Source: Full input list retained while slicing batches.
- Source: Embeddings represented as blobs, per-row arrays, and stacked array.
- Source: Normalization adds another matrix before HDBSCAN.
Reproduction
- Executed source-only allocation-lifetime audit.
- Prescribed, not executed: tracemalloc/OS memory measurement with synthetic length distribution matching mean233.67,max4140 and9942 rows, then2x/10x; compare streaming reference output.
Observed result
Memory grows with full pending corpus independently of inference batch size. Current data text alone is roughly9942*233.67=2323147.14 characters, so this is not evidence for a26GB cause.
Expected result
Message-text rebuild staging should be bounded by a fixed page/batch and preserve stable resume checkpoints. Clustering algorithm workspace needs separate explicit budgeting.
Proposed implementation
- Page input by stable primary-key generation/high-watermark; preserve source snapshots and resume state.
- Avoid redundant completion/separation text materialization.
- Do not replace HDBSCAN or other algorithms solely on source inspection; first isolate its actual workspace cost.
Acceptance tests
- Identical vector IDs/content and interruption/resume outcome to the existing reference.
- Peak host staging memory remains bounded as corpus grows.
- Concurrent add/update behavior across snapshot boundaries is specified and tested.
Dependencies and sequencing
- A09
Regression risks
- Paging a live changing source without a stable version boundary can skip or mix evidence.
- Do not claim a clustering matrix can be streamed without considering HDBSCAN requirements.
Scope and affected paths
Rebuild staging memory; preserve fixed model and algorithm quality.
Related history
No matching issue identified in the reviewed history.
Uncertainty and measurement limits
No live workload or model inference was run; source and synthetic evidence do not establish this as the cause of the screenshot memory footprint.
Release constraints
Preserve current embedding models, rerankers, internal retrieval depth, and durable source data. A synthetic control-flow reproduction establishes the stated defect only; hardware savings and retrieval quality require the separate release validation gates. Do not close this issue merely because a unit test passes.
Audit package
Full plan and catalogue | Portable reproductions and receipts
Activity
buildingjoshbetter commentedon Oct 8, 2026
Follow-up: legacy rebuild tokens can miss numeric input changes
Independent synthetic SQLite review found a remaining publication blocker in the standalone rebuild bridge. The existing storage-type plus
CAST AS BLOBcomparison does not distinguish every value consumed by separation embedding.Reproduction
content TEXT,sender REAL, and text recipient/timestamp columns.1.0000000000000002, capture the rebuild source and start encoding its separation text.1.0000000000000004before publication.A separate control preserves signed zero in a column without affinity:
0.0and-0.0can produce different separation strings while comparing numerically equal. A BINARY comparison alone is therefore insufficient. Genuine TEXT affinity converts the adjacent precision values to the same stored string; that control does not represent a changed encoder input.Correction being prepared
Retain the existing type/byte checks and add the explicit BINARY value comparison. Conservatively invalidate covered-field UPDATEs involving stored REAL zero, because SQLite numeric equality cannot distinguish its sign. Restrict the nonconservative update trigger to covered source columns so unrelated derived-field writes do not cause false invalidation. Preserve the existing conservative custom-UNIQUE behavior.
Regression gates include actual separation publication races, both signed-zero directions, unchanged nonzero REAL and TEXT controls, NOCASE/RTRIM, null/type/blob values, old-trigger epoch repair, rollback and old-target preservation. Same-value REAL-zero source updates can conservatively restart a rebuild; that limit must be documented.
The prior 2,397-test suite and all 34 hosted checks for draft #802 remain valid evidence for their tested cases. They did not cover this defect. A separate correction is being prepared; no native model, personal database or transcript was used in the investigation. Issue #756 remains open.