test(recall): the recall benchmark has no producer for the search path that ships

[waterbro-8]

Summary

benchmarks/recall/ is a well-built measuring half of a retrieval benchmark. Its
scoring, dataset, leakage detection and artifact contract are all real. What it
has no counterpart for is the producer: nothing in this repository can
generate a rankings file from the search path that actually ships, so every
quality claim about production retrieval — including the Chinese-language
condition GOAL.md §6 makes standing — is currently unmeasured rather than
measured and bad.

The gap is narrow and I want to be precise about it, because the first draft of
this issue overstated it: the harness is not missing a live mode, it is missing
the tool that drives a live system.

Source reproduction

On main @ 7a194f1eba4167d54bd46cf84cdbe86e00532319:

• benchmarks/recall/README.md:5-20 states plainly that the default run is
  "deliberately model-free and offline", reports engine=lexical-reference, is
  "not production recall", and that the lexical adapter's 0 ms is a sentinel
  that "must not be used as a performance result". adapters.py:108 produces
  that label; adapters.py:95 writes the 0.0 latency.
• The opt-in lane exists and is strict: --rankings
  (__main__.py:36-40) feeds load_external_rankings (adapters.py:189-311),
  which validates schema_version: mem.recall-rankings.v1, per-query
  latency_ms, status, error_code, citation length and result count.
  README.md:112-150 documents it.
• But the contract's own instruction is "Run the candidate system separately"
  (README.md:114) and no such runner exists here. The only producer in the
  repository is lexical_rankings. grep over benchmarks/**/*.py finds no
  HTTP client at all (urllib/requests/httpx are absent); the single
  subprocess import, runner.py:9, is used only to read git metadata for the
  artifact's commit field (runner.py:38-56). There is no fixture of real
  memd responses either.
• The measurement side is therefore wired to a source that can only ever be the
  sentinel: metrics.py:52 computes recall_at_1/5/10, :90-93 emits them with
  MRR and nDCG@10.
• One detail worth keeping: the documented example artifact configuration is
  "index": {"kind": "hnsw", "distance": "cosine"} (README.md:137). The
  harness is ready to record an ANN index that the server does not have — see
  perf(index): add an ANN index so vector search stops scanning every embedding #173, filed the same day.
• test(recall): establish a versioned multilingual retrieval benchmark #44 ("establish a versioned multilingual retrieval benchmark") is closed completed. Of its acceptance bullets, the ones that describe the harness
  itself are met; "Real embedding runs are opt-in and provider-agnostic" is met
  as a schema, and the run itself has no in-repo way to be produced. This
  issue is that residue, not a re-open of test(recall): establish a versioned multilingual retrieval benchmark #44.

What is missing

One script and one recipe: query a running memd (compose ships
pgvector/pgvector:pg16 by digest at docker-compose.test.yml:7, plus worker,
Postgres and MinIO) over the versioned dataset's queries, and emit
mem.recall-rankings.v1 — with real latency, per-query status, and the
corpus/index configuration that produced it. No change to metrics.py,
dataset.py, or the artifact contract is needed.

Scope boundary

In scope: the producer, the documented run recipe, and checking a result into
baselines/ so the next change has a delta to compare against.

Out of scope: changing ranking, adding a reranker, or chasing a score target.
Ranking changes without a measuring harness is the failure mode this issue exists
to prevent, and #71 (hybrid candidates, reranking, diversity, budgets) was closed
not_planned — reviving any part of it should happen after something can measure
it. Also out of scope: making this a CI gate; #44 already recorded that threshold
policy needs a separate review.

Acceptance

• A single documented command produces a mem.recall-rankings.v1 file from a
  live memd, and the emitted artifact cannot be mistaken for
  lexical-reference (its engine and configuration are populated from the
  running system, not hand-written).
• The artifact records engine, model/profile identity, corpus size, index
  configuration and host, so a later reader can tell what was measured — i.e.
  the fields _validate_configuration already demands are filled with truth.
• Recall@1/5/10 and nDCG@10 are reported for English and Chinese queries
  separately, from the same run, with the Chinese set including image
  description queries, so GOAL.md §6's Chinese condition becomes answerable.
• Latency is measured, not sentinel: latency_ms comes from the client
  timing each request, and the README's warning about 0 ms stays true only for
  the lexical lane.
• A first real-memd baseline is checked into benchmarks/recall/baselines/
  and README.md:5-20 stops describing the offline default as the only way to
  run.
• Whether the producer runs in CI or stays on-demand is decided and recorded
  here.

Evidence level

E2 — source-level. Every line above was read directly, and the negative claim
(no network use anywhere under benchmarks/) is a completed grep of that tree.
I did not run the benchmark, and I did not attempt a live memd retrieval; the
size of the quality gap is unknown precisely because nothing has measured it.

Proposed triage

Applied on filing, per docs/maintainers/triage.md and the precedent in #135:
type:maintenance, area:recall, evidence:e2-source, status:needs-triage.
Left to a maintainer: priority, and any change to the set below.

This is the dependency for judging an ANN index change (#173, filed the same
day) and any future ranking change; the audit found no existing issue covering
the producer.

Raised from the data/index audit on 2026-09-08.

[waterbro-8]

Still open. Draft candidate: #196 (codex/fix-pr-184, successor to #184). I am not opening a duplicate producer PR.

main still has no HTTP producer under benchmarks/recall/ (no urllib/httpx/requests client). Offline lexical-reference remains the default measuring half. Tracking #196.

[waterbro-8]

平台组分工补派（冯浩然 / waterbro-8）

• 跟进人：李圆阳（GitHub sun-970，仓库暂不能指派，以钉钉+本评论为准）
• 仓库：bytefolk/mem test(recall): the recall benchmark has no producer for the search path that ships #175
• 动作：现在提 PR
• 说明：recall benchmark 缺 search path producer，直接补测试提 PR
• 流程：fork 后提 PR，标题带 test(recall): the recall benchmark has no producer for the search path that ships #175，然后 @waterbro-8 审核。我不能合自己的 CODEOWNER PR。

[waterbro-8]

平台组群成员派活（冯浩然 / waterbro-8）

• 跟进人：施卓凡（钉钉平台组；本仓库可能无法 GitHub 指派，以钉钉+本评论为准）
• 仓库：bytefolk/mem test(recall): the recall benchmark has no producer for the search path that ships #175
• 动作：现在提 PR
• 说明：recall benchmark 缺 search producer，直接提 PR
• 流程：有 GitHub 权限就直接开 PR；没有就 fork 后提。标题带 test(recall): the recall benchmark has no producer for the search path that ships #175。提完 @waterbro-8 审核。
• 李彬彬这批不用做。

[sun-970]

Verified current main after #196. Met: documented produce command, client-timed latency_ms, EN+ZH scoring including Chinese image-description queries, engine/configuration discovered from running memd, CI vs on-demand recorded as on-demand.

Remaining gap (not invented): no real-memd baseline in benchmarks/recall/baselines/ — that needs a populated isolated memd + embedding-provider run.

PR for the remaining in-repo gaps: #224 (sun-970:fix/175-discover-live-config-20260920)

@waterbro-8 please review.