Skip to content

Add Azure Cosmos DB for NoSQL vector benchmark engine - #407

Open
fcenedes wants to merge 10 commits into
redis:masterfrom
fcenedes:codex/cosmosdb-nosql
Open

fcenedes wants to merge 10 commits into
redis:masterfrom
fcenedes:codex/cosmosdb-nosql

Conversation

@fcenedes

@fcenedes fcenedes commented Nov 18, 2026 •

Copy link
Copy Markdown

Closes #406.

Adds Azure Cosmos DB for NoSQL as a dense vector benchmark engine using Microsoft's azure_data_cosmos 1.0 SDK. The shipped Flat and DiskANN configs run global top-K queries with the same recall, latency, and throughput reporting as the other engines.

The runner creates a database if needed and uses a dedicated container per engine config and dataset. Manual throughput is configurable per run or per config and checked against Azure's realized offer. Upload and search parallelism are independently configurable. When the SDK exhausts its HTTP 429 retries on an upload write, the runner backs off and resends it, up to 60 times per row, and reports how many writes it resent. If the upload fails, the runner removes the container it created. A complete corpus is kept when a later step fails, so --skip-upload can reuse it. The runner checks the realized vector policy and exact corpus size, including when reusing a container.

DiskANN results record the configured index and Azure's realized options. Azure can fall back to a scan while the index builds or when a partition has too few vectors, so the runner probes every feed range before search, after upload and on reuse. It compares the request charge of an index-requested query with a forced scan and waits until the index-requested query is clearly cheaper on each range. It polls every 2 seconds at first and less often up to every 30, and prints the index and scan charge of each waiting range. The run fails if that does not happen within AZURE_COSMOS_INDEX_PROBE_SECONDS (default 1800; 0 skips the probe and records index use as unverified). Results record the per-range probe charges and the wait, and upload total_time covers the upload plus that wait.

Search entries can set partition_concurrency to query feed ranges in parallel and merge the global top K on the client. Before timing, the runner samples queries and runs the stock query twice and the fan-out once for each. The fan-out may miss no more IDs than a small multiple of what the stock query misses against itself, and both counts are recorded under partition_fanout.parity.

DiskANN search entries can set search_list_size_multiplier and quantized_vector_list_multiplier to compare query-time settings on the same index. The shipped cosmosdb-diskann preset keeps the service-default search points and adds search-list multipliers of 1, 5, 10, and 20 at parallelism 1 and 8. Cosmos search artifacts record SDK request charge, request attempts, and observed HTTP 429s. The README includes setup and DNS verification for a same-region Azure benchmark VM using a NoSQL private endpoint.

The separately named cosmosdb-diskann-build-* presets compare one index-build setting at a time on dedicated containers. The runner verifies explicit indexing_search_list_size and quantization_byte_size values against the realized policy before search, including on reuse. An opt-in DBpedia config under experiments/examples/ runs four index builds at 10,000 RU/s per container with top-100 searches.

Validation

  • make agent-check
  • Release mutation check. Removing the distance-policy comparison caused the matching unit test to fail.
  • make integration-test-cosmosdb against Azure. Flat achieved recall@1 of 1.0 at 400 RU/s and the second run confirmed container cleanup.
  • Live DiskANN smoke at a verified 1,000 RU/s. Uploaded 1,200 vectors, verified all rows on reuse, and scored recall@10 of 1.0 over 24 queries against a brute-force oracle. This smoke verifies the query path and configured policy. It predates the index-use probe, which a corpus this small does not pass.
  • DiskANN query mutation check. Changing the emitted searchListSizeMultiplier option caused the SQL serialization test to fail.
  • DiskANN index-build tests check both SDK wire properties and reject missing or mismatched explicit settings on readback. The wire and readback tests each failed under a targeted mutation.
  • Index-use probe, live at 7b5a9e7. On a multi-partition DiskANN container the probe passed on every feed range some minutes after the row count already matched, and passed immediately on reuse. On a corpus too small for Azure to use the index it failed at the deadline with equal scan and index charges.
  • Fan-out parity, live at 7b5a9e7. Flat and DiskANN fan-out matched the stock query on every sampled query. The new env-gated fan-out integration test (AZURE_COSMOS_TEST_FANOUT=1) passed.
  • Mutation checks for the probe ratio, the forced-scan flag, the probe skip, the parity verdict and the config validation each turned the matching unit test red.
  • Upload behaviour, live at 4316e91. An upload at the default throughput with parallel: 8 completed after resending throttled writes. A forced upload failure removed the container the run had created. A probe failure after a complete upload kept the corpus.
  • At 4316e91 the probe loop, the per-range row count, the fan-out merge and parity sampling are unit-tested in src/cosmosdb.rs. Nine targeted mutations of those guards and of the upload retry each turned a unit test red.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🤖 Automated first-pass review. A human maintainer still needs to review this before merge.

This is an outside contributor's new engine, so I reviewed it the way the KiviDB engine (#203) was reviewed. I checked: filter refusal (#219), per-config container naming (#151-4), configure/--skip-upload destructiveness, corpus-size and readiness checks, the timed worker path, the fan-out merge direction for each metric, secret redaction for the new AZURE_COSMOS_* prefix scan, whether the integration test owns what it touches, and config-key validation. I found no correctness, isolation or secrets defects in the production paths. Several things are done right and are worth saying so:

Worth fixing before merge

  1. No live run is reported on the head commit. Every live result in the PR body is "at 7b5a9e7". 4316e91 came after it and changes paths CI cannot reach:

    • the 60-per-row 429 resend loop;
    • deleting the container when an upload fails;
    • the new 2–30 s poll schedule;
    • moving the probe, row-count, merge and parity code behind closures.

    Without credentials, integration_cosmosdb prints SKIP and passes, the same as Vertex. A green CI run therefore says nothing about the Azure path. Please rerun make integration-test-cosmosdb on 4316e91, ideally with AZURE_COSMOS_TEST_FANOUT=1. Also run one DiskANN corpus with several partitions through the probe on head, and paste the numbers.

  2. total_time leaves out the row-count scan, but the DiskANN index keeps building during it. In upload_corpus the count runs after upload_time stops (engine/cosmosdb.rs:302). The index-wait clock in verify_index_use only starts after that (engine/cosmosdb.rs:336). The result is total_time = upload_time + index_wait_seconds (:446), which is shorter than the real time until the corpus is searchable, by however long the ID-projection count took.

    That matters most for the cosmosdb-diskann-build-* presets and the DBpedia sweep, because time-to-searchable is the column they exist to compare. If the build finishes during the count, every preset reports roughly upload_time plus one probe round, and the build-setting difference disappears. I can't size this from the diff; the 975K DBpedia run would show it.

    Two fixes would work. Start the index-wait clock where upload_time stops and record the count's duration separately. Or add the count's duration to total_time and say so in the README. As it stands, the README's "The row-count check is in neither" is accurate, but it describes the bias rather than avoiding it.

Optional / follow-up (not blockers)

  • The index probe is a proxy, like taxonomy item 3. A range counts as indexed when index_ru < 0.5 × scan_ru (src/cosmosdb.rs:481, :495). That reliably catches "still scanning". It doesn't prove "fully built": a partly built index could plausibly clear half the scan cost. Worth asking whether the service or SDK exposes an index-progress signal for vector indexes that could replace or back up the ratio. It's not a blocker: the PR is upfront that this is a heuristic, it fails loudly at the deadline, and it records vector_index_use_verified: false when the probe is skipped.
  • Fan-out queries also project the distance; stock queries don't. With fan-out on, the measured query adds VectorDistance(...) AS distance to the SELECT (engine/cosmosdb.rs:533 → query_text, src/cosmosdb.rs:463). So fan-out and stock rows differ in more than routing when you compare RU or latency. The README already says to compare them on throughput and recall. It may be worth naming the projection there too.
  • The DBpedia example has no shared center point. The list-size pair (50/200) uses q256 and the quantization pair (128/512) uses list100. Each pair changes one variable, but no row sits at list100/q256, so the two pairs can't be compared with each other. The shipped *-build-* presets do share list100/bytes32, so this only affects the example file.
  • --skip-upload reuse checks the row count, not which rows are there. That matches the rest of the harness (--skip-upload verifies corpus SIZE, not corpus IDENTITY (cardinality is not identity) #279), so it's not specific to this engine.
  • The fan-out integration test provisions 12,000 RU/s (tests/integration_cosmosdb.rs:123). If it fails after upload, the README says the container stays provisioned and billing. A drop guard in the test that deletes its own container would cap that cost.

Net: a careful new engine. It is unusually honest about where the Azure service can return a scan and present it as an index search, and its mutation-tested claims match the bar this repo sets for its own PRs. Both "before merge" items are small: one re-verification on head, and one timing boundary. Accepting a new third-party cloud engine is the maintainer's decision.

@fcenedes

fcenedes commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

Follow-up to the first-pass review is in b1b33a8.

  • The adapter now rejects a response that exceeds global TOP K across its pages. The Rust SDK 1.0.0 planner uses a global ordered merge and TOP stage for supported cross-partition plans. The new unit test covers the cardinality guard. A live multi-partition run is still outstanding.
  • Each measured search result now records SDK-reported RU charge, request attempts, HTTP 429s, and diagnostic gaps. The CLI warns when measured requests encountered throttling.
  • The README now explains how to handle a temporary container left by a failed run and how to interpret RU-capped latency. A regression test covers account-key redaction.

make agent-check passed. Temporarily removing the TOP K guard or the key-redaction marker caused the corresponding test to fail. The cloud integration target compiled, and its live case requires Azure credentials.

@fcenedes

fcenedes commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

The Cosmos adapter now has two query modes for the same global top-K workload.

  • Stock SDK mode uses the SDK's full-container query. Leave partition_concurrency unset to select it.
  • Parallel feed-range mode queries disjoint physical ranges concurrently and merges the scored candidates. Set partition_concurrency to "auto" to use the discovered physical range count, or use a positive integer to cap per-query parallelism.
[
  { "parallel": 1, "top": 100 },
  { "parallel": 1, "top": 100, "search_params": { "partition_concurrency": "auto" } }
]

The parallel mode caps concurrency per query and across all workers. It compares sampled top-K IDs against the stock SDK before timing. A mismatched sample can be retried up to three times because even stock DiskANN queries can vary at the cutoff. Every sample still has to match exactly before timing. The adapter checks topology stability and saves effective fan-out limits, preflight retries, RU charge, and HTTP attempts. The merge follows the service's score direction for cosine, dot product, and Euclidean distance.

make agent-check and the live Cosmos integration test passed. A separate five-partition live Flat test exercised both modes with matching sampled IDs and recall.

@fcenedes fcenedes left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review round 2, at de9bf3c. I read the whole adapter (src/cosmosdb.rs, engine/cosmosdb.rs), the harness call sites in experiment.rs and the shipped configs. I am not re-raising what the first pass covered and b1b33a8 answered (TOP K guard, RU/429 diagnostics, key redaction).

Checked and clean:

  • Namespacing (#151-4). One container per config and dataset, length-prefixed hash.
  • Non-destructive configure (#238). A 409 on create is a hard error, and delete() only removes a container this run created.
  • Reuse. check_index_layout runs on both the upload and --skip-upload paths (experiment.rs:1678), so the policy and realized RU/s are verified before every search.
  • Filters (#219). Both read_queries() sites refuse a declared filter.
  • Duration mode. --search-duration is refused for this engine at experiment.rs:474, so it cannot silently run as a single pass.
  • Merge direction. Cosine and dot product sort descending, L2 ascending, ties by ID, with a unit test per metric.

Not verified in this round: I could not run make agent-check on this head. cargo fmt --check passed, but the build stopped in hdf5-sys 0.8.1, which rejects the HDF5 2.2.0 on the review machine. That is a local toolchain problem, not this diff. Separately, the only check that has run on this PR is claude-review. ci.yml and Docker Build Validation have not run on any commit, so clippy, the release test suites, and the Linux and release-static builds with the new native_tls dependency have no CI result yet.

Worth fixing before merge

  1. DiskANN index use is still unverified in the tree.

    • First-pass item 1 got a README sentence, not code. upload() returns once the ID count matches (engine/cosmosdb.rs:366), check_index_layout prints "Index use is unverified", server_metadata hardcodes vector_index_use_verified: false, and search starts.
    • For the cosmosdb-diskann-build-* presets the risk tracks the swept variable. A larger indexing_search_list_size builds more slowly, so those configs are the most likely to be timed mid-build, against a scan.
    • Add a per-partition probe before timing, wait on it with a deadline, fail the run if it never passes, and set the flag from its result.
  2. The parity preflight is retry-until-green.

    • check_partition_parity (engine/cosmosdb.rs:644) passes if any one of four stock/fan-out pairs agrees exactly.
    • Its own comment says two stock DiskANN requests can differ at the cutoff. Exact equality is then stricter than the engine's own repeatability: it can abort a valid run, and a pass shows one agreement in up to four tries.
    • A noise-floor check is the better shape. For each sample, run stock twice and fan-out once, require the fan-out/stock overlap to be no worse than the stock/stock overlap, and record both numbers in partition_fanout. Keep the hard checks on unique IDs, finite scores and global top K.
    • Worth confirming alongside: that both modes score the same recall at one worker. The fan-out query projects VectorDistance ... AS distance and the stock one does not, and an overlap threshold would hide a real difference between the two routes.

Optional / follow-up (not blockers)

  • Fan-out latency at high parallel includes client-side queueing. shared_request_limit = parallel.max(per_query_concurrency) (engine/cosmosdb.rs:496). When parallel exceeds the range count, every worker's range requests share parallel permits, and the semaphore wait is inside the timed window. The limit is recorded in partition_fanout, but the row keeps its parallel label. One README sentence would stop a reader comparing its p99 with the stock row.
  • Upload total_time measures the count scan, not the index build. It includes the full ID-projection scan (engine/cosmosdb.rs:366-378) and no build wait. The build sweep therefore has no build-cost metric at all. The probe in item 1 would give it one.
  • Untested logic in src/bin/. check_partition_parity, the range fold in search_hits, and the knob validation in CosmosDbEngine::new have no unit tests. AGENTS.md asks for this logic in src/. The retry loop is the part item 2 is about.
  • No committed test exercises fan-out. integration_cosmosdb is Flat, stock mode, one feed range at 400 RU/s. The five-partition Flat run mentioned above would make a good second env-gated case.

Net: the adapter itself is careful and follows the repo's conventions. What is open is whether the DiskANN and fan-out rows measure what they are labelled as. Accepting a new cloud engine is the maintainers' call.

Probe every feed range before search: compare the RU charge of an
index-requested query with a forced scan and wait, up to
AZURE_COSMOS_INDEX_PROBE_SECONDS, until the index serves each range.
The run fails if it never does. Upload total_time now covers the upload
and the index wait, and server metadata records the probe result.

Replace the retry-until-green fan-out parity check with a noise-floor
check: each sample runs the stock query twice and the fan-out once, and
the fan-out may miss no more IDs than a small multiple of what stock
misses against itself. The counts are recorded under
partition_fanout.parity (was preflight_retries).

Also: validate config keys and values in src/cosmosdb.rs with unit
tests, build the query text once per search, count rows per feed range
concurrently, record AZURE_COSMOS_* settings the SDK reads, and add an
env-gated fan-out integration test.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@fcenedes

Copy link
Copy Markdown
Author

Follow-up to the round 2 review, at 7b5a9e7.

Before-merge items

  1. DiskANN index use. Addressed. The runner now probes every feed range before search, after upload and on reuse (verify_index_use in engine/cosmosdb.rs). It compares the request charge of an index-requested query with a forced scan, waits until the index serves each range, and fails the run at the deadline. vector_index_use_verified is set from the probe result, and the per-range charges and the wait are recorded.
  2. Parity preflight. Addressed. The retry loop is gone. Each sample runs the stock query twice and the fan-out once, and the fan-out may miss no more IDs than a small multiple of what stock misses against itself (ParityTally in src/cosmosdb.rs). The counts are recorded under partition_fanout.parity, which replaces preflight_retries.

Optional items

  • Fan-out queueing at high parallel. README now says the shared request limit is inside the timed window.
  • Upload total_time. Now the upload plus the index wait; the row-count scan is excluded.
  • Logic in src/bin/. Config validation, the parity rule and the probe criterion moved to src/cosmosdb.rs with unit tests. The concurrent request plumbing stays in the adapter and has no unit tests.
  • Fan-out test. Added flat_fanout_matches_the_stock_query_across_feed_ranges, gated on AZURE_COSMOS_TEST_FANOUT=1.

Verified

  • make agent-check passes locally.
  • Both integration tests pass against a live account.
  • Live DiskANN on a multi-partition container: the probe passed on every range, some minutes after the row count already matched, and immediately on reuse. On a corpus too small for Azure to use the index it failed at the deadline with equal scan and index charges.
  • Mutation checks on the probe ratio, forced-scan flag, probe skip, parity verdict and config validation each went red.

Not verified

  • The parity allowance above exact agreement. Stock DiskANN repeated exactly in the live run, so only the exact-agreement path was exercised.
  • ci.yml and Docker Build Validation still have not run on this PR.

The parity description in my earlier comment (retries, exact match per sample) is superseded by item 2.

When the SDK exhausts its HTTP 429 retries on an upload write, back off
and send the write again, up to 60 times per row, and report how many
writes were resent. The shipped presets at the default throughput no
longer fail on the first throttled row.

Remove the container this run created when the upload fails. A complete
corpus is kept if the index-use probe then fails, so --skip-upload can
reuse it.

Print the index and forced-scan charge of each waiting feed range on
every probe poll and in the deadline error. Poll every 2 seconds at
first and less often up to every 30, so the recorded index wait is
accurate to about a tenth of its length.

Move the probe loop, the per-range row count, the fan-out merge and
parity sampling into src/cosmosdb.rs behind closures, with unit tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@fcenedes

Copy link
Copy Markdown
Author

Follow-up at 4316e91, on top of the reply above.

  • Throttled uploads. When the SDK exhausts its HTTP 429 retries on a write, the runner backs off and resends it, up to 60 times per row, and reports how many writes it resent. The shipped presets at the default throughput no longer fail on the first throttled row. Upload time includes that backoff, and the README says so.
  • Failed uploads. The run removes the container it created. A complete corpus is kept when the index-use probe then fails, so --skip-upload can reuse it.
  • Probe output and polling. Each poll prints the index and forced-scan charge of every waiting feed range, as does the deadline error. Polling starts at 2 seconds and grows to 30, so the recorded index wait is accurate to about a tenth of its length. While an index builds, each poll costs about as much as a scan of the waiting ranges.
  • Unit tests. This replaces "the concurrent request plumbing has no unit tests" from the earlier reply. The probe loop, the per-range row count, the fan-out merge and parity sampling now sit in src/cosmosdb.rs behind closures, with tests.

Verified

  • make agent-check passes locally.
  • Nine targeted mutations (probe range check, poll interval, waiting-range list, retry budget, retry condition, split detection, shared permits, per-query limit, parity verdict) each turned a unit test red.
  • Live: a throttled upload completed, a forced upload failure removed its container, both integration tests passed, and a multi-partition DiskANN run passed the probe and fan-out parity.

Still not verified

  • The parity allowance above exact agreement.
  • ci.yml and Docker Build Validation have not run on this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CosmoDB comparison

1 participant