Implement new embed vector and similar search

[hungpham10]

Summary by CodeRabbit

• New Features

  • Added semantic and hybrid symbol search modes.
  • Added optional embedding generation, caching, persistence, and vector-based retrieval.
  • Added an embed command to pre-download embedding models.
  • Added support for embedding storage across configured database backends.
  • Added Apple acceleration options for supported macOS builds.

• Documentation

  • Expanded setup, configuration, installation, search, embedding, storage, and development guidance.

• Chores

  • Improved CI coverage with database and Redis integration services, coverage reporting, and manual workflow triggering.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a58c55d7-ec68-4451-a093-3cb79b9b6edd

📥 Commits

Reviewing files that changed from the base of the PR and between bbe1274 and 45bdc64.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (27)

• .github/workflows/ci.yml
• .github/workflows/integration.yml
• README.md
• crates/codegraph-api/Cargo.toml
• crates/codegraph-api/src/lib.rs
• crates/codegraph-api/tests/api.rs
• crates/codegraph-bench/src/lib.rs
• crates/codegraph-context/src/lib.rs
• crates/codegraph-core/src/semgraph.rs
• crates/codegraph-extract/src/config.rs
• crates/codegraph-extract/src/walker.rs
• crates/codegraph-extract/tests/extract.rs
• crates/codegraph-graph/Cargo.toml
• crates/codegraph-graph/src/embeddings.rs
• crates/codegraph-graph/src/lib.rs
• crates/codegraph-graph/src/storage.rs
• crates/codegraph-graph/src/storage/lmdb.rs
• crates/codegraph-graph/src/storage/mysql.rs
• crates/codegraph-graph/src/storage/postgres.rs
• crates/codegraph-graph/src/storage/redis.rs
• crates/codegraph-graph/src/storage/sqlite.rs
• crates/codegraph-graph/src/vector_index.rs
• crates/codegraph-graph/tests/lmdb.rs
• crates/codegraph-graph/tests/sqlite.rs
• crates/codegraph-mcp/src/tools.rs
• crates/codegraph/Cargo.toml
• crates/codegraph/src/main.rs

---

Walkthrough

This change adds opt-in embedding backends, persistent vector storage, semantic and hybrid symbol search, and resumable search API migration. It also updates MCP and CLI integrations, expands CI coverage, removes the integration workflow, and documents the new configuration and commands.

Changes

Semantic Search Platform

Layer / File(s)	Summary
Embedding configuration and backends crates/codegraph-graph/..., crates/codegraph-extract/..., crates/codegraph/src/main.rs, crates/codegraph-core/src/semgraph.rs	Adds hashing and fastembed backends, embedding configuration, model caching, Apple acceleration, semantic match modes, and the embed command.
Vector storage and indexing crates/codegraph-graph/src/storage*, crates/codegraph-graph/src/vector_index.rs	Adds embedding persistence across storage backends and an in-memory vector index with KNN and deterministic k-means operations.
Graph indexing and semantic search crates/codegraph-graph/src/lib.rs, crates/codegraph-graph/Cargo.toml	Builds and persists embeddings during ingest, supports storage-native or in-memory KNN, and adds resumable semantic and hybrid search.
Search API migration and integrations crates/codegraph-api/..., crates/codegraph-context/..., crates/codegraph-bench/..., crates/codegraph-mcp/..., crates/codegraph-graph/tests/..., crates/codegraph-extract/tests/...	Removes legacy search methods and updates callers, tests, and MCP entry resolution to use paged resumable search.
CI and repository documentation .github/workflows/*, README.md	Adds manual CI runs, database services, integration tests, coverage upload, and documentation for storage and semantic search configuration. The separate integration workflow is removed.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• hungpham10/codegraph-rs#1: Introduces the earlier graph symbol-search API replaced by this change.
• hungpham10/codegraph-rs#3: Adds storage work extended here with embedding persistence.
• hungpham10/codegraph-rs#5: Provides the resumable paged search APIs used by the updated callers.

Suggested reviewers: hungpham7-tiki, cleboost

Poem

I am a rabbit with vectors to sow,
Through paged search paths, the answers now flow.
Redis and SQL keep embeddings in line,
Fast models wake when the cache bells chime.
CI checks each burrow with care,
Semantic hops fill the air.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/setup-new-vector-embed-to-help-similarity-search

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing feature/setup-new-vector-embed-to-help-similarity-search (45bdc64) with main (bbe1274)}

[codecov-commenter]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️