Repository navigation
Conversation
|
🤖 Automated first-pass review. A human maintainer still needs to review this before merge. This is an outside contributor's new engine, so I reviewed it the way the KiviDB engine (#203) was reviewed. I checked: filter refusal (#219), per-config container naming (#151-4), configure/
Worth fixing before merge
Optional / follow-up (not blockers)
Net: a careful new engine. It is unusually honest about where the Azure service can return a scan and present it as an index search, and its mutation-tested claims match the bar this repo sets for its own PRs. Both "before merge" items are small: one re-verification on head, and one timing boundary. Accepting a new third-party cloud engine is the maintainer's decision. |
|
Follow-up to the first-pass review is in
|
|
The Cosmos adapter now has two query modes for the same global top-K workload.
[
{ "parallel": 1, "top": 100 },
{ "parallel": 1, "top": 100, "search_params": { "partition_concurrency": "auto" } }
]The parallel mode caps concurrency per query and across all workers. It compares sampled top-K IDs against the stock SDK before timing. A mismatched sample can be retried up to three times because even stock DiskANN queries can vary at the cutoff. Every sample still has to match exactly before timing. The adapter checks topology stability and saves effective fan-out limits, preflight retries, RU charge, and HTTP attempts. The merge follows the service's score direction for cosine, dot product, and Euclidean distance.
|
There was a problem hiding this comment.
Review round 2, at de9bf3c. I read the whole adapter (src/cosmosdb.rs, engine/cosmosdb.rs), the harness call sites in experiment.rs and the shipped configs. I am not re-raising what the first pass covered and b1b33a8 answered (TOP K guard, RU/429 diagnostics, key redaction).
Checked and clean:
- Namespacing (#151-4). One container per config and dataset, length-prefixed hash.
- Non-destructive configure (#238). A 409 on create is a hard error, and
delete()only removes a container this run created. - Reuse.
check_index_layoutruns on both the upload and--skip-uploadpaths (experiment.rs:1678), so the policy and realized RU/s are verified before every search. - Filters (#219). Both
read_queries()sites refuse a declared filter. - Duration mode.
--search-durationis refused for this engine atexperiment.rs:474, so it cannot silently run as a single pass. - Merge direction. Cosine and dot product sort descending, L2 ascending, ties by ID, with a unit test per metric.
Not verified in this round: I could not run make agent-check on this head. cargo fmt --check passed, but the build stopped in hdf5-sys 0.8.1, which rejects the HDF5 2.2.0 on the review machine. That is a local toolchain problem, not this diff. Separately, the only check that has run on this PR is claude-review. ci.yml and Docker Build Validation have not run on any commit, so clippy, the release test suites, and the Linux and release-static builds with the new native_tls dependency have no CI result yet.
Worth fixing before merge
-
DiskANN index use is still unverified in the tree.
- First-pass item 1 got a README sentence, not code.
upload()returns once the ID count matches (engine/cosmosdb.rs:366),check_index_layoutprints "Index use is unverified",server_metadatahardcodesvector_index_use_verified: false, and search starts. - For the
cosmosdb-diskann-build-*presets the risk tracks the swept variable. A largerindexing_search_list_sizebuilds more slowly, so those configs are the most likely to be timed mid-build, against a scan. - Add a per-partition probe before timing, wait on it with a deadline, fail the run if it never passes, and set the flag from its result.
- First-pass item 1 got a README sentence, not code.
-
The parity preflight is retry-until-green.
check_partition_parity(engine/cosmosdb.rs:644) passes if any one of four stock/fan-out pairs agrees exactly.- Its own comment says two stock DiskANN requests can differ at the cutoff. Exact equality is then stricter than the engine's own repeatability: it can abort a valid run, and a pass shows one agreement in up to four tries.
- A noise-floor check is the better shape. For each sample, run stock twice and fan-out once, require the fan-out/stock overlap to be no worse than the stock/stock overlap, and record both numbers in
partition_fanout. Keep the hard checks on unique IDs, finite scores and global top K. - Worth confirming alongside: that both modes score the same recall at one worker. The fan-out query projects
VectorDistance ... AS distanceand the stock one does not, and an overlap threshold would hide a real difference between the two routes.
Optional / follow-up (not blockers)
- Fan-out latency at high
parallelincludes client-side queueing.shared_request_limit = parallel.max(per_query_concurrency)(engine/cosmosdb.rs:496). Whenparallelexceeds the range count, every worker's range requests shareparallelpermits, and the semaphore wait is inside the timed window. The limit is recorded inpartition_fanout, but the row keeps itsparallellabel. One README sentence would stop a reader comparing its p99 with the stock row. - Upload
total_timemeasures the count scan, not the index build. It includes the full ID-projection scan (engine/cosmosdb.rs:366-378) and no build wait. The build sweep therefore has no build-cost metric at all. The probe in item 1 would give it one. - Untested logic in
src/bin/.check_partition_parity, the range fold insearch_hits, and the knob validation inCosmosDbEngine::newhave no unit tests. AGENTS.md asks for this logic insrc/. The retry loop is the part item 2 is about. - No committed test exercises fan-out.
integration_cosmosdbis Flat, stock mode, one feed range at 400 RU/s. The five-partition Flat run mentioned above would make a good second env-gated case.
Net: the adapter itself is careful and follows the repo's conventions. What is open is whether the DiskANN and fan-out rows measure what they are labelled as. Accepting a new cloud engine is the maintainers' call.
Probe every feed range before search: compare the RU charge of an index-requested query with a forced scan and wait, up to AZURE_COSMOS_INDEX_PROBE_SECONDS, until the index serves each range. The run fails if it never does. Upload total_time now covers the upload and the index wait, and server metadata records the probe result. Replace the retry-until-green fan-out parity check with a noise-floor check: each sample runs the stock query twice and the fan-out once, and the fan-out may miss no more IDs than a small multiple of what stock misses against itself. The counts are recorded under partition_fanout.parity (was preflight_retries). Also: validate config keys and values in src/cosmosdb.rs with unit tests, build the query text once per search, count rows per feed range concurrently, record AZURE_COSMOS_* settings the SDK reads, and add an env-gated fan-out integration test. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Follow-up to the round 2 review, at Before-merge items
Optional items
Verified
Not verified
The parity description in my earlier comment (retries, exact match per sample) is superseded by item 2. |
When the SDK exhausts its HTTP 429 retries on an upload write, back off and send the write again, up to 60 times per row, and report how many writes were resent. The shipped presets at the default throughput no longer fail on the first throttled row. Remove the container this run created when the upload fails. A complete corpus is kept if the index-use probe then fails, so --skip-upload can reuse it. Print the index and forced-scan charge of each waiting feed range on every probe poll and in the deadline error. Poll every 2 seconds at first and less often up to every 30, so the recorded index wait is accurate to about a tenth of its length. Move the probe loop, the per-range row count, the fan-out merge and parity sampling into src/cosmosdb.rs behind closures, with unit tests. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Follow-up at
Verified
Still not verified
|
Closes #406.
Adds Azure Cosmos DB for NoSQL as a dense vector benchmark engine using Microsoft's
azure_data_cosmos1.0 SDK. The shipped Flat and DiskANN configs run global top-K queries with the same recall, latency, and throughput reporting as the other engines.The runner creates a database if needed and uses a dedicated container per engine config and dataset. Manual throughput is configurable per run or per config and checked against Azure's realized offer. Upload and search parallelism are independently configurable. When the SDK exhausts its HTTP 429 retries on an upload write, the runner backs off and resends it, up to 60 times per row, and reports how many writes it resent. If the upload fails, the runner removes the container it created. A complete corpus is kept when a later step fails, so
--skip-uploadcan reuse it. The runner checks the realized vector policy and exact corpus size, including when reusing a container.DiskANN results record the configured index and Azure's realized options. Azure can fall back to a scan while the index builds or when a partition has too few vectors, so the runner probes every feed range before search, after upload and on reuse. It compares the request charge of an index-requested query with a forced scan and waits until the index-requested query is clearly cheaper on each range. It polls every 2 seconds at first and less often up to every 30, and prints the index and scan charge of each waiting range. The run fails if that does not happen within
AZURE_COSMOS_INDEX_PROBE_SECONDS(default 1800;0skips the probe and records index use as unverified). Results record the per-range probe charges and the wait, and uploadtotal_timecovers the upload plus that wait.Search entries can set
partition_concurrencyto query feed ranges in parallel and merge the global top K on the client. Before timing, the runner samples queries and runs the stock query twice and the fan-out once for each. The fan-out may miss no more IDs than a small multiple of what the stock query misses against itself, and both counts are recorded underpartition_fanout.parity.DiskANN search entries can set
search_list_size_multiplierandquantized_vector_list_multiplierto compare query-time settings on the same index. The shippedcosmosdb-diskannpreset keeps the service-default search points and adds search-list multipliers of 1, 5, 10, and 20 at parallelism 1 and 8. Cosmos search artifacts record SDK request charge, request attempts, and observed HTTP 429s. The README includes setup and DNS verification for a same-region Azure benchmark VM using a NoSQL private endpoint.The separately named
cosmosdb-diskann-build-*presets compare one index-build setting at a time on dedicated containers. The runner verifies explicitindexing_search_list_sizeandquantization_byte_sizevalues against the realized policy before search, including on reuse. An opt-in DBpedia config underexperiments/examples/runs four index builds at 10,000 RU/s per container with top-100 searches.Validation
make agent-checkmake integration-test-cosmosdbagainst Azure. Flat achieved recall@1 of 1.0 at 400 RU/s and the second run confirmed container cleanup.searchListSizeMultiplieroption caused the SQL serialization test to fail.7b5a9e7. On a multi-partition DiskANN container the probe passed on every feed range some minutes after the row count already matched, and passed immediately on reuse. On a corpus too small for Azure to use the index it failed at the deadline with equal scan and index charges.7b5a9e7. Flat and DiskANN fan-out matched the stock query on every sampled query. The new env-gated fan-out integration test (AZURE_COSMOS_TEST_FANOUT=1) passed.4316e91. An upload at the default throughput withparallel: 8completed after resending throttled writes. A forced upload failure removed the container the run had created. A probe failure after a complete upload kept the corpus.4316e91the probe loop, the per-range row count, the fan-out merge and parity sampling are unit-tested insrc/cosmosdb.rs. Nine targeted mutations of those guards and of the upload retry each turned a unit test red.