Skip to content

test(recall): add live memd producer for recall benchmark - #184

Closed
sun-970 wants to merge 1 commit into
bytefolk:mainfrom
sun-970:feat/175-recall-live-producer
Closed

sun-970 wants to merge 1 commit into
bytefolk:mainfrom
sun-970:feat/175-recall-live-producer

Conversation

@sun-970

@sun-970 sun-970 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add a producer that queries a running memd via POST /v1/search over every dataset query and emits a mem.recall-rankings.v1 file, closing the gap where the recall benchmark had a measurement harness but no way to produce rankings from the search path that actually ships.
  • New produce subcommand on python -m benchmarks.recall with --memd-url, --token, --dimension, --mode, --provider, --model flags.
  • Maps API results back to dataset doc_ids by matching the path field, with snippet overlap for disambiguation when multiple documents share a path.

Changes

  • benchmarks/recall/live_producer.py: producer module — queries memd, records client-side latency, maps results to dataset doc_ids
  • benchmarks/recall/__main__.py: add produce subcommand
  • benchmarks/recall/tests/test_live_producer.py: unit tests for path index, snippet matching, success and error paths
  • benchmarks/recall/README.md: document the live producer recipe and filter translation

Usage

# Produce rankings from a live memd
python3 -m benchmarks.recall produce \
  --memd-url http://localhost:8080 \
  --token "$MEM_TOKEN" \
  --output /tmp/live-rankings.json \
  --dimension 1536 \
  --mode hybrid

# Score against the lexical baseline
python3 -m benchmarks.recall run \
  --rankings /tmp/live-rankings.json \
  --output /tmp/live-artifact.json \
  --compare benchmarks/recall/baselines/lexical-reference.v1.json

Test plan

  • All 33 existing recall benchmark tests pass
  • 5 new unit tests for the producer (path index, snippet matching, success, error)
  • Maintainer verifies against a live memd instance with the benchmark corpus loaded

Fixes #175

Add a producer that queries a running memd via POST /v1/search over every
dataset query and emits a mem.recall-rankings.v1 file. This closes the gap
where the recall benchmark had a measurement harness but no way to produce
rankings from the search path that actually ships.

- benchmarks/recall/live_producer.py: queries memd, maps API results back to
  dataset doc_ids by path (with snippet overlap for disambiguation), records
  client-side latency per query
- benchmarks/recall/__main__.py: add `produce` subcommand with --memd-url,
  --token, --dimension, --mode, --provider, --model flags
- benchmarks/recall/tests/test_live_producer.py: unit tests for path index,
  snippet matching, success and error paths
- benchmarks/recall/README.md: document the live producer recipe and explain
  filter translation (path_prefix→scope, source_kind→type)

The emitted configuration block is populated from CLI flags, not hand-written,
so the artifact cannot be mistaken for the lexical reference.

Fixes bytefolk#175
@PeterGuy326

Copy link
Copy Markdown
Collaborator

HOLD — the five producer unit tests pass locally on current head 89bb12bdf2ae5edbdfa17260953ec6352ab9239f, but the PR explicitly leaves the live memd acceptance unchecked and no CI is reported. Please run the producer against a real memd with the benchmark corpus loaded, attach the generated ranking artifact/command output, and trigger exact-head CI.

sun-970 pushed a commit to sun-970/mem that referenced this pull request Sep 10, 2026
- Remove benchmark/recall files (belong to bytefolk#184, not this PR)
- Rename migration 0024 → 0025 to resolve collision with bytefolk#183
- Fix verify script: use valid pgvector literals via array_fill,
  add index-scan assertion instead of just printing EXPLAIN output
- Update VALIDATION_HNSW.md references to 0025
sun-970 pushed a commit to sun-970/mem that referenced this pull request Sep 10, 2026
- Remove bytefolk#184 stacking (recall benchmark is a separate PR)
- Rename migration 0025 → 0026 to avoid collision with bytefolk#180
@PeterGuy326

Copy link
Copy Markdown
Collaborator

Current-head CI approval readback (2026-09-10)

Head: 89bb12bdf2ae5edbdfa17260953ec6352ab9239f. The original pending-approval runs on this head have been approved and real check-runs now exist (17 at this snapshot). No old-head run was approved. Auto-merge remains disabled; this is not a code approval or merge-ready declaration.

Failures at this snapshot:

@sun-970

sun-970 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by @PeterGuy326's #196 which covers the same changes on current main. Closing per the '本轮先不合并' directive.

@sun-970 sun-970 closed this Sep 10, 2026
sun-970 pushed a commit to sun-970/mem that referenced this pull request Sep 17, 2026
The producer code and tests are checked in but no end-to-end run against
a running memd has been recorded. Make this explicit in the README and
state that bytefolk#184's HOLD still requires a real production-class run; the
fixture-level tests in this PR are not a substitute for live acceptance.
PeterGuy326 added a commit that referenced this pull request Sep 17, 2026
…open

Fixture acceptance passes. Live acceptance (real memd run with ranking
artifacts) is still required before #184 can be closed. Make this
boundary unmissable at the top of the README.
waterbro-8 pushed a commit that referenced this pull request Sep 17, 2026
) (#196)

## Linked draft successor — original #184 remains open

Refs #175.

Preserves @sun-970 / liyuanyang's authored chain through
`89bb12bdf2ae5edbdfa17260953ec6352ab9239f` for #175. Do not close #184
before a replacement is independently verified and merged. This draft
remains HOLD for live acceptance.

Source-fork Git-data writes returned HTTP 404, with repository
`push:false`; no ACL override was attempted. The canonical branch
preserves original commit hashes and identities. Added blobs, trees, and
commits were individually hash-verified; no force update was used.

## Bounded corrections

- Fail closed on ambiguous cross-workspace mappings, unknown result
paths, malformed responses, non-finite scores, and failed requests;
retain an error artifact and exit 2.
- Map the shipping folder `path` plus file `name`; do not silently
discard unknown hits or infer tenant identity from snippets.
- Vector mode sends `route=text`; lexical mode sends `route=lexical`
with null provider/model/dimension metadata. Do not claim `auto` is
lexical/vector hybrid.
- Reject structured-memory queries this file-search endpoint cannot
serve. Document the existing file-only `profile-text-v1` fixture as the
bounded corpus.
- Remove hostname collection and invented provider/index identity.
Configuration labels remain explicitly operator-declared, not
server-verified.
- Carry #192 audit remediation as a separate `cherry-pick -x` of
`11e02e21ef2c3dbd2dae26e4376872e54e78ecb5`; audit threshold unchanged.

## Validation at `651bec1679c50a9cd07cf77b27b5556c20e3273e`

- PASS: Python 3.11.14, `python3.11 -m unittest discover -s
benchmarks/recall/tests`: 39 tests, including the reproduced fail-closed
regressions and a loopback HTTP fixture.
- PASS: `python3.11 -m benchmarks.recall verify`: deterministic harness
and intentional leakage failure gate.
- PASS: Web audit with the explicit shared fix.
- NOT VERIFIED: real memd retrieval, actual embedding-provider quality,
real index selection, production latency, or full structured-memory
corpus acceptance. The HTTP handler is a fixture, not memd; its timing
is not live benchmark evidence.

## Exact remaining live prerequisites (no external provider authorized)

1. An isolated, authorized test deployment of real memd and its Worker,
with PostgreSQL/pgvector and ingest dependencies configured, plus a
token verified to belong to the test workspace. No existing user
deployment or provider credentials have been used.
2. Ingest all five synthetic files from
`benchmarks/recall/data/profile-text-v1/corpus.jsonl`, preserving their
full paths and contents, into that workspace. The producer is not an
ingestor. Confirm indexing completed and file/result identities match
the fixture.
3. For the bounded fixed-text experiment, use the same locally
available, explicitly selected 768-dimensional text embedding model for
corpus and query. Verify the corpus/provider metadata and which active
generation or fixed table the server actually uses. A label passed to
the producer proves none of these facts. No paid provider, external
endpoint, model download, or provider configuration was enabled by this
correction.
4. Execute all four file queries through real `/v1/search` with `--mode
vector`, retain sanitized rankings, then score with `run --rankings` and
record the exact memd head, actual model/dimension/index, environment,
and errors. An empty/error run does not satisfy live acceptance.
5. Model-free lexical is a separate optional real-server experiment
requiring #183's server capability (draft successor #194); it cannot
establish vector quality. Full v1 structured-memory acceptance remains
unsupported by this producer and must not be reported as passed.

Fresh exact-head CI and independent review/human approval remain
separate required gates. No fake or paid live run is substituted for the
missing evidence.
waterbro-8 pushed a commit that referenced this pull request Sep 18, 2026
- Remove #184 stacking (recall benchmark is a separate PR)
- Rename migration 0025 → 0026 to avoid collision with #180
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(recall): the recall benchmark has no producer for the search path that ships

3 participants