fix(knowledge): store compact vector search candidates

[icecrasher321]

Summary

• Store binary search candidates in a narrow, transactionally maintained projection so ANN traversal does not reload and quantize full vectors. Keep full-precision reranking and the shared permission predicates.
• Bound the visibility probe with document lookups, retain per-stage diagnostics, and preserve the separate Search and Assistant budgets.
• Add an additive, replay-safe migration and batched backfill. Keep the existing expression indexes available during rollout and mark their later removal.

Type of Change

• Bug fix

Testing

• Real PostgreSQL application integration: 16 passed, including both search surfaces, varied-query recall, authorization, cancellation, and query-plan assertions that candidate retrieval never touches the full embedding table.
• Retrieval unit tests: 60 passed. Database unit tests: 142 passed; migration integration: 9 passed, including every vector width, concurrent writes, rollback, deletion, and replay.
• App and database type checks, lint, all repository audits, API boundary validation, docs manifest, migration safety, and generated-schema parity passed.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 14, 2026 7:27pm UTC

[greptile-apps]

Confidence Score: 5/5

This PR appears safe to merge.

What we checked:

• Projection drifts during backfill: This risk is blocked. The migration installs the trigger first, locks each source row during copying, lets trigger writes win conflicts, and cascades deletes.
• Candidate search bypasses access: This risk is blocked. Candidate lookup applies document access inside the lateral query. Reranking and hydration apply the full visibility checks again before content is returned.

Summary

• Searches use width-specific stored bit columns and HNSW indexes.
• Visibility checks still use the shared document permissions.
• A replay-safe migration creates, synchronizes, and backfills the projection.
• Integration tests cover query recall and plans that avoid the full embedding table.

Diagram

sequenceDiagram
    participant Deploy as Migration job
    participant Emb as embedding
    participant Proj as embedding_search
    participant Search as Search request
    participant Doc as document
    participant Auth as Access provider

    Deploy->>Proj: Create table and HNSW indexes
    Deploy->>Emb: Install sync trigger
    loop Missing rows in batches of 500
        Deploy->>Emb: Read rows with key-share locks
        Deploy->>Proj: Insert stored bit vectors
    end
    Emb->>Proj: Trigger inserts or updates later writes
    Emb-->>Proj: Delete cascades by id

    Search->>Proj: Probe visible candidate IDs
    Proj->>Doc: Check access and filters
    alt Small or filtered scope
        Search->>Emb: Rank full vectors exactly
    else Broad scope
        Search->>Proj: Rank stored bit vectors with HNSW
        Proj->>Doc: Check access and filters
        Search->>Emb: Rerank candidate IDs by full vector
    end
    Search->>Auth: Check live connector access
    Search->>Emb: Hydrate authorized IDs
    Search-->>Search: Sort and return top results

Loading

_{Reviews (1) · Last reviewed commit: "fix(knowledge): store compact vector sea..."}