Skip to content

fix(search): retire legacy embeddings through scoped backfill - #8430

Merged
icecrasher321 merged 4 commits into
stagingfrom
codex/retire-sim-search-embeddings
Sep 29, 2026
Merged

icecrasher321 merged 4 commits into
stagingfrom
codex/retire-sim-search-embeddings

Conversation

@icecrasher321

@icecrasher321 icecrasher321 commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add a resumable script migration that discovers and retires the sole legacy Search KB without cleanup flags, preserving other KBs and live source configuration. Persist the selected target and defer ambiguous databases.
  • Save each bounded page's cursor in the same transaction as its changes; defer unfinished work and recheck for late documents or chunks before recording completion.
  • Reject dormant Search uploads before creation and stop dispatch and queued processing before billing or embedding writes. Document deployment, continuation, and storage maintenance.

Type of Change

  • Bug fix and maintenance

Testing

  • The full DB PostgreSQL suite (110 checks) and ten app integration checks passed, covering cleanup scope, cascades, bounded resumption, rollback, late documents, upload admission, queued processing, and indexed-mode transaction integrity.
  • Upload admission, late-document retirement, and both queued-processing regressions were reproduced before their fixes.
  • The DB unit suite and all 43 affected document-processing unit tests passed. Billing and quota fixtures now account for the KB eligibility read. Existing transaction fixtures now explicitly enable indexed Search; removed a duplicate registry test and scoped the older migration-journal assertion to the upgrade it proves.
  • App and DB type-checks, lint, all 52 audits (including API validation), docs-manifest and block-registry checks passed. Migration safety check reports no new schema migrations.
  • Broader local document suite: 319/320 passed; the unchanged SDK cancellation test failed locally on Node 23.11 and passed in CI.
  • No production cleanup or storage maintenance was executed.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 29, 2026 9:58pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 8 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Critical risk] Adds database migration to delete search embeddings and retire legacy search index.

The PR appears safe to merge based on the changes reviewed.

Summary

This PR retires embeddings from a selected legacy Search knowledge base through a resumable, bounded migration while preserving other knowledge bases and live source configuration. It also blocks dormant Search uploads and processing, and documents the cleanup and recovery lifecycle.

Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Discover sole legacy Search KB] --> B[Persist selected target]
  B --> C[Retire documents in bounded pages]
  C --> D[Delete target embeddings in bounded pages]
  D --> E{Unretired documents or chunks remain?}
  E -- Yes --> C
  E -- No --> F[Record completion]
Loading

Reviews (4) · Last reviewed commit: "chore(tests): align document dispatch bi..."

Comment thread apps/sim/lib/knowledge/documents/service.ts
Comment thread packages/db/script-migrations/0027_retire_search_embeddings.ts
Comment thread packages/db/script-migrations/0027_retire_search_embeddings.ts
@icecrasher321
icecrasher321 force-pushed the codex/retire-sim-search-embeddings branch from 3f1ab4a to ff7268b Compare September 29, 2026 21:24
@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 11 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

Comment thread packages/db/script-migrations/0027_retire_search_embeddings.ts
@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 12 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@icecrasher321
icecrasher321 force-pushed the codex/retire-sim-search-embeddings branch from 93543a3 to 474204e Compare September 29, 2026 21:56
@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 14 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@icecrasher321
icecrasher321 merged commit 16c7e1a into staging Sep 29, 2026
32 of 33 checks passed
@waleedlatif1
waleedlatif1 deleted the codex/retire-sim-search-embeddings branch September 29, 2026 22:32

This branch was successfully deployed

1 active deployment
Preview — 474204eb Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant