Conversation
Add a producer that queries a running memd via POST /v1/search over every dataset query and emits a mem.recall-rankings.v1 file. This closes the gap where the recall benchmark had a measurement harness but no way to produce rankings from the search path that actually ships. - benchmarks/recall/live_producer.py: queries memd, maps API results back to dataset doc_ids by path (with snippet overlap for disambiguation), records client-side latency per query - benchmarks/recall/__main__.py: add `produce` subcommand with --memd-url, --token, --dimension, --mode, --provider, --model flags - benchmarks/recall/tests/test_live_producer.py: unit tests for path index, snippet matching, success and error paths - benchmarks/recall/README.md: document the live producer recipe and explain filter translation (path_prefix→scope, source_kind→type) The emitted configuration block is populated from CLI flags, not hand-written, so the artifact cannot be mistaken for the lexical reference. Fixes bytefolk#175
|
HOLD — the five producer unit tests pass locally on current head |
- Remove benchmark/recall files (belong to bytefolk#184, not this PR) - Rename migration 0024 → 0025 to resolve collision with bytefolk#183 - Fix verify script: use valid pgvector literals via array_fill, add index-scan assertion instead of just printing EXPLAIN output - Update VALIDATION_HNSW.md references to 0025
- Remove bytefolk#184 stacking (recall benchmark is a separate PR) - Rename migration 0025 → 0026 to avoid collision with bytefolk#180
|
Current-head CI approval readback (2026-09-10) Head:
Failures at this snapshot: |
|
Superseded by @PeterGuy326's #196 which covers the same changes on current main. Closing per the '本轮先不合并' directive. |
The producer code and tests are checked in but no end-to-end run against a running memd has been recorded. Make this explicit in the README and state that bytefolk#184's HOLD still requires a real production-class run; the fixture-level tests in this PR are not a substitute for live acceptance.
…open Fixture acceptance passes. Live acceptance (real memd run with ranking artifacts) is still required before #184 can be closed. Make this boundary unmissable at the top of the README.
) (#196) ## Linked draft successor — original #184 remains open Refs #175. Preserves @sun-970 / liyuanyang's authored chain through `89bb12bdf2ae5edbdfa17260953ec6352ab9239f` for #175. Do not close #184 before a replacement is independently verified and merged. This draft remains HOLD for live acceptance. Source-fork Git-data writes returned HTTP 404, with repository `push:false`; no ACL override was attempted. The canonical branch preserves original commit hashes and identities. Added blobs, trees, and commits were individually hash-verified; no force update was used. ## Bounded corrections - Fail closed on ambiguous cross-workspace mappings, unknown result paths, malformed responses, non-finite scores, and failed requests; retain an error artifact and exit 2. - Map the shipping folder `path` plus file `name`; do not silently discard unknown hits or infer tenant identity from snippets. - Vector mode sends `route=text`; lexical mode sends `route=lexical` with null provider/model/dimension metadata. Do not claim `auto` is lexical/vector hybrid. - Reject structured-memory queries this file-search endpoint cannot serve. Document the existing file-only `profile-text-v1` fixture as the bounded corpus. - Remove hostname collection and invented provider/index identity. Configuration labels remain explicitly operator-declared, not server-verified. - Carry #192 audit remediation as a separate `cherry-pick -x` of `11e02e21ef2c3dbd2dae26e4376872e54e78ecb5`; audit threshold unchanged. ## Validation at `651bec1679c50a9cd07cf77b27b5556c20e3273e` - PASS: Python 3.11.14, `python3.11 -m unittest discover -s benchmarks/recall/tests`: 39 tests, including the reproduced fail-closed regressions and a loopback HTTP fixture. - PASS: `python3.11 -m benchmarks.recall verify`: deterministic harness and intentional leakage failure gate. - PASS: Web audit with the explicit shared fix. - NOT VERIFIED: real memd retrieval, actual embedding-provider quality, real index selection, production latency, or full structured-memory corpus acceptance. The HTTP handler is a fixture, not memd; its timing is not live benchmark evidence. ## Exact remaining live prerequisites (no external provider authorized) 1. An isolated, authorized test deployment of real memd and its Worker, with PostgreSQL/pgvector and ingest dependencies configured, plus a token verified to belong to the test workspace. No existing user deployment or provider credentials have been used. 2. Ingest all five synthetic files from `benchmarks/recall/data/profile-text-v1/corpus.jsonl`, preserving their full paths and contents, into that workspace. The producer is not an ingestor. Confirm indexing completed and file/result identities match the fixture. 3. For the bounded fixed-text experiment, use the same locally available, explicitly selected 768-dimensional text embedding model for corpus and query. Verify the corpus/provider metadata and which active generation or fixed table the server actually uses. A label passed to the producer proves none of these facts. No paid provider, external endpoint, model download, or provider configuration was enabled by this correction. 4. Execute all four file queries through real `/v1/search` with `--mode vector`, retain sanitized rankings, then score with `run --rankings` and record the exact memd head, actual model/dimension/index, environment, and errors. An empty/error run does not satisfy live acceptance. 5. Model-free lexical is a separate optional real-server experiment requiring #183's server capability (draft successor #194); it cannot establish vector quality. Full v1 structured-memory acceptance remains unsupported by this producer and must not be reported as passed. Fresh exact-head CI and independent review/human approval remain separate required gates. No fake or paid live run is substituted for the missing evidence.
Summary
memdviaPOST /v1/searchover every dataset query and emits amem.recall-rankings.v1file, closing the gap where the recall benchmark had a measurement harness but no way to produce rankings from the search path that actually ships.producesubcommand onpython -m benchmarks.recallwith--memd-url,--token,--dimension,--mode,--provider,--modelflags.doc_ids by matching thepathfield, with snippet overlap for disambiguation when multiple documents share a path.Changes
benchmarks/recall/live_producer.py: producer module — queries memd, records client-side latency, maps results to dataset doc_idsbenchmarks/recall/__main__.py: addproducesubcommandbenchmarks/recall/tests/test_live_producer.py: unit tests for path index, snippet matching, success and error pathsbenchmarks/recall/README.md: document the live producer recipe and filter translationUsage
Test plan
Fixes #175