[WIP] Add an optional PostgreSQL backend for pgvector and pg_cuvs

[ysys143]

Add a cuvs-bench PostgreSQL backend for pgvector HNSW/IVFFlat, pg_cuvs CAGRA, and CAGRA-to-pgvector HNSW export. It uses the current plugin/config-loader API, standard dataset YAML and Cartesian parameter grids, with one result per search setting and corpus-offset IDs for the existing recall calculation.

Each build owns a fresh schema, loads its corpus, builds one index and reuses it across that build cell's search sweep. Cleanup drops only that schema. Custom YAML overrides matching algorithm groups; explicit algo_groups augments the selected groups and unmatched requests fail before execution.

Dependencies are optional. Driver installation errors give the postgres-extra guidance; connection failures return a diagnostic that excludes raw conninfo errors. CPU tests exclude GPU cases even when a GPU DSN is configured.

CAGRA sessions disable CPU fallback and check the actual daemon mode and query-count delta outside timed queries. Plan checks before and after each search point reject sequential scans. HNSW export searches on CPU. Build timing excludes COPY/ANALYZE and includes CAGRA construction plus conversion for export. CPU disk bytes and CAGRA daemon VRAM bytes are labeled separately in CSV metadata.

Reproduce

fern/pages/cuvs_bench/postgres.md covers optional client installation, a pinned public pg_cuvs server bootstrap/build, standard wiki_all_1M data, CLI runs, bounded parameter sweeps, role privileges and measurement boundaries.

python -m venv .venv
. .venv/bin/activate
python -m pip install -e './python/cuvs_bench[postgres]' -Crapidsai.disable-cuda=true
python -m pip install pytest
python -m pytest python/cuvs_bench/cuvs_bench/tests/test_postgres.py -m 'not postgres'

Live CPU tests use CUVS_BENCH_POSTGRES_DSN; GPU cases separately require CUVS_BENCH_POSTGRES_GPU_DSN and a real CUDA/cuVS server. CAGRA roles need SELECT on pg_stat_gpu_search in addition to database CREATE. The optional index_dir override requires permission to set the server's PGC_SUSET parameter.

Validation

Current client source, macOS arm64/Python 3.12.14/PostgreSQL 17.11/pgvector 0.8.6:

• Available suite: 196 passed, 28 skipped, 4 deselected. The excluded cases require C++ benchmark outputs or a /data directory unavailable in this environment.
• A newly built wheel in an isolated environment passed 44 backend tests against the real CPU database; 2 real-GPU cases were skipped. Installed backend/test bytes and entry-point discovery were checked. A separate core-only installation without psycopg/pgvector preserved discovery and returned the extra installation guidance at the build entrypoint.
• Standard CLI with groups=base, an additional pgvector_hnsw.test group and --search-threads 1 produced 2 build rows and 3 validated search rows on small synthetic data.
• Full changed-file pre-commit passed.

GPU validation on an earlier client revision used fresh Ubuntu 22.04.5, A100-SXM4-80GB/driver 580.126.09, PostgreSQL 16.15, pgvector 0.8.7, pg_cuvs 0.8.0 at public commit 8b73d65fb8a3d6f66cdfe69eec4969c1f2fc753c, CUDA 12.4/cuVS 26.04 and Python 3.12.15:

• Public pinned bootstrap and native CUDA build/install completed. That revision passed 30 server tests, including 2 real-GPU cases, plus a four-algorithm 4096-by-32 synthetic smoke. GPU mode/counters and CPU HNSW serving were observed.
• Full wiki_all_1M, valid supplied ground truth, k=10 and 10,000 queries per point: four algorithms each with one build/two search settings, followed by CAGRA/export graph_degree=[32,64] build sweeps. All 8 build rows and 16 search points (160,000 timed queries) were retained and validated. CAGRA rows recorded gpu_cagra and 10,000 GPU dispatches each; export CPU serving did not increment its source CAGRA GPU counter.
• An ordinary effective-role CAGRA smoke passed after granting the required view SELECT. Authentication began with a host peer session and SET ROLE; direct nonadmin password authentication was not tested.

These runs validate execution and reproduction. No speedup is claimed, and this PR does not change GPU kernels. GPU execution of the current client revision has not been repeated.

Limits

Euclidean vectors and one serial SQL client only. The SQL boundary includes client adaptation, planning/execution and result transfer. Separate-process search-only/persistent index reuse, concurrent/throughput clients, other GPUs and upstream PR CI are unverified. Abrupt termination can leave an isolated schema.

Always provide valid ground truth: the shared result interface otherwise leaves recall at zero rather than unmeasured. Python API dry-run results must not be passed to the generic CSV exporter as measurements because skipped search rows retain zero values; the CLI skips dry-run export. These existing interface limitations remain documented.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.