Skip to content

HUB-044: report source coverage at k beside recall - #14

Merged
funsaized merged 1 commit into
mainfrom
hub-044-coverage-at-k
Aug 13, 2026
Merged

HUB-044: report source coverage at k beside recall#14
funsaized merged 1 commit into
mainfrom
hub-044-coverage-at-k

Conversation

@funsaized

Copy link
Copy Markdown
Owner

Closes HUB-044.

Two corrections before the work

  • Breadth was not untracked. RetrievalDiagnostics has computed sources_available/sources_represented since HUB-024, app/synthesis.py writes both into job progress, and hub_report_retrieval_items carries both. The backlog entry was wrong and is corrected in this PR.
  • The motivating case no longer reproduces. "15 chunks from 7 of 22 sources" (job bc3e5297) re-measures today at 44 chunks from 15 of 22.

What was actually missing

The tracked number is a single post-cap total. It is taken after max_chunks_per_source has already forced diversity, so it cannot separate breadth the ranking found from breadth the cap manufactured; it says nothing about how breadth grows with k; and it never existed for the corpus-wide path HUB-043 opened.

Changes

  • tests/coverage_at_k.py — one shared definition. Unit is the document (chunking is a fixed 800/100 split, so chunk_index marks position, not structure). sources_reachable is required with no default, because the obvious default makes saturation 1.0 at every tail k by construction. scope_coverage_at_k is null for the unfiltered corpus rather than dividing by 679.
  • benchmark_report_retrieval.py, benchmark_retrieval_exact_terms.py — coverage reported beside recall. Each case retrieved twice: at the deployed cap and with it lifted.
  • benchmark_retrieval_coverage.py + fixture — live read-only baseline over both scopes (6 real jobs, 5 corpus-wide queries, one topic-filtered and one tag-filtered).
  • Nothing under app/ changes. This item adds a number; it does not change retrieval.

Coverage gates nothing

It is maximised by returning one chunk per source. Stated in the docstring, encoded in a test showing a broad-but-shallow ranking scoring 1.0, and enforced by a test asserting the report benchmark's gate set is still exactly citation_validity + critical_recall_at_k.

Baseline (2026-08-13, deployed limits: 120 candidates, cap 3, rrf_k 60)

Pooled saturation_at_k:

Scope k=4 k=8 k=16 k=40
Job, capped 0.750 0.604 0.583 0.795
Job, uncapped 0.750 0.583 0.524 0.692
Corpus, capped 0.700 0.675 0.712 0.701
Corpus, uncapped 0.650 0.575 0.562 0.535

Three findings:

  1. The candidate pool is the binding ceiling, not the cap or the ranking — five of six jobs reach fewer sources than they have embedded (p-hacking 8 of 19). This redirects HUB-045.
  2. The paper's failure mode reproduces exactly: on "how does reciprocal rank fusion combine rankings" the uncapped fused ranking takes its first eleven chunks from one document; coverage@8 is 1 source uncapped, 5 capped.
  3. Recall and coverage move independently: on the exact-term manifest, adding the lexical channel lifts hit@4 0.69231.0 while pooled breadth is identical dense and hybrid.

Verification

  • 387 tests + 567 subtests green in-container (throwaway image, tests/ and bin/ mounted, Redis DB 15).
  • Exact-term probe unchanged: dense 0.6923, hybrid 1.0, no fixture drift.
  • Job 48c9247e reproduces HUB-043's recorded 93 chunks / 39 sources exactly.
  • Read-only audit after the runs: Qdrant 68,072 points green, 679 docs / 758 observations / 71,125 lexical rows, attempt-11 report 068d60b2…, v4 seal 762e7a19…. No deployed container recreated.

🤖 Generated with Claude Code

Breadth was not untracked, as the backlog claimed: RetrievalDiagnostics has
reported sources_available/sources_represented since HUB-024, synthesis writes
both into job progress, and hub_report_retrieval_items carries both. What was
missing is that the tracked number is a single post-cap total — taken after
max_chunks_per_source has already forced diversity — so it cannot separate
breadth the ranking found from breadth the cap manufactured, says nothing
about how breadth grows with k, and never covered the corpus-wide path HUB-043
opened.

tests/coverage_at_k.py is one shared definition, reported beside recall in
both retrieval benchmarks, with tests/benchmark_retrieval_coverage.py as a
live read-only baseline over both scopes. Every case is retrieved twice, at
the deployed cap and with it lifted, because the gap between those curves is
the question the existing total cannot answer.

Coverage gates nothing anywhere; it is trivially maximised by returning one
chunk per source. That is stated in the module docstring, encoded in a test
that shows a broad-but-shallow ranking scoring 1.0, and enforced by a test
asserting the report benchmark's gate set is still exactly the two relevance
gates. The unit is the document, not the paper's section: chunking is a fixed
800/100 split, so chunk_index marks position rather than structure.
Denominators are named rather than assumed — sources_reachable is required and
has no default, and scope_coverage_at_k is null for the unfiltered corpus
instead of dividing by 679.

Nothing under app/ changes. Baseline over six real jobs and five corpus-wide
queries, and the corrected backlog entry, are in docs/CURRENT_STATE.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@funsaized
funsaized merged commit abec770 into main Aug 13, 2026
4 checks passed
@funsaized
funsaized deleted the hub-044-coverage-at-k branch August 14, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant