HUB-044: report source coverage at k beside recall - #14
Merged
Conversation
Breadth was not untracked, as the backlog claimed: RetrievalDiagnostics has reported sources_available/sources_represented since HUB-024, synthesis writes both into job progress, and hub_report_retrieval_items carries both. What was missing is that the tracked number is a single post-cap total — taken after max_chunks_per_source has already forced diversity — so it cannot separate breadth the ranking found from breadth the cap manufactured, says nothing about how breadth grows with k, and never covered the corpus-wide path HUB-043 opened. tests/coverage_at_k.py is one shared definition, reported beside recall in both retrieval benchmarks, with tests/benchmark_retrieval_coverage.py as a live read-only baseline over both scopes. Every case is retrieved twice, at the deployed cap and with it lifted, because the gap between those curves is the question the existing total cannot answer. Coverage gates nothing anywhere; it is trivially maximised by returning one chunk per source. That is stated in the module docstring, encoded in a test that shows a broad-but-shallow ranking scoring 1.0, and enforced by a test asserting the report benchmark's gate set is still exactly the two relevance gates. The unit is the document, not the paper's section: chunking is a fixed 800/100 split, so chunk_index marks position rather than structure. Denominators are named rather than assumed — sources_reachable is required and has no default, and scope_coverage_at_k is null for the unfiltered corpus instead of dividing by 679. Nothing under app/ changes. Baseline over six real jobs and five corpus-wide queries, and the corrected backlog entry, are in docs/CURRENT_STATE.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes HUB-044.
Two corrections before the work
RetrievalDiagnosticshas computedsources_available/sources_representedsince HUB-024,app/synthesis.pywrites both into job progress, andhub_report_retrieval_itemscarries both. The backlog entry was wrong and is corrected in this PR.bc3e5297) re-measures today at 44 chunks from 15 of 22.What was actually missing
The tracked number is a single post-cap total. It is taken after
max_chunks_per_sourcehas already forced diversity, so it cannot separate breadth the ranking found from breadth the cap manufactured; it says nothing about how breadth grows with k; and it never existed for the corpus-wide path HUB-043 opened.Changes
tests/coverage_at_k.py— one shared definition. Unit is the document (chunking is a fixed 800/100 split, sochunk_indexmarks position, not structure).sources_reachableis required with no default, because the obvious default makes saturation 1.0 at every tail k by construction.scope_coverage_at_kisnullfor the unfiltered corpus rather than dividing by 679.benchmark_report_retrieval.py,benchmark_retrieval_exact_terms.py— coverage reported beside recall. Each case retrieved twice: at the deployed cap and with it lifted.benchmark_retrieval_coverage.py+ fixture — live read-only baseline over both scopes (6 real jobs, 5 corpus-wide queries, one topic-filtered and one tag-filtered).app/changes. This item adds a number; it does not change retrieval.Coverage gates nothing
It is maximised by returning one chunk per source. Stated in the docstring, encoded in a test showing a broad-but-shallow ranking scoring 1.0, and enforced by a test asserting the report benchmark's gate set is still exactly
citation_validity+critical_recall_at_k.Baseline (2026-08-13, deployed limits: 120 candidates, cap 3, rrf_k 60)
Pooled
saturation_at_k:Three findings:
0.6923→1.0while pooled breadth is identical dense and hybrid.Verification
0.6923, hybrid1.0, no fixture drift.48c9247ereproduces HUB-043's recorded 93 chunks / 39 sources exactly.068d60b2…, v4 seal762e7a19…. No deployed container recreated.🤖 Generated with Claude Code