diff --git a/PRDs/hub-047-answer-completeness-scoring.md b/PRDs/hub-047-answer-completeness-scoring.md new file mode 100644 index 0000000..926d99f --- /dev/null +++ b/PRDs/hub-047-answer-completeness-scoring.md @@ -0,0 +1,236 @@ +# HUB-047 — Scoring design for end-to-end answer quality + +Status: DESIGN ONLY, 2026-08-13. No question set is built and no code is +written. This document settles *how a report is scored* first, because the +scoring scheme determines what the question set must contain, and building +the set first would force the scoring to fit whatever was collected. + +Prior art reviewed: 3 arXiv abstracts fetched and read 2026-08-13 for this +design, plus 16 read for the HUB-049 pass (citations at the end; every claim +below is attributed to a fetched abstract or marked as this design's own +choice). + +## Problem + +The system verifies every claim it publishes and measures nothing about what +it failed to say. + +The claim gate checks each drafted claim against its cited evidence, with +per-reference necessity, and rejects padding. `benchmark_report_retrieval.py` +checks citation validity. `benchmark_retrieval_exact_terms.py` checks that +needles are retrievable. HUB-044 measures retrieval breadth. Every one of +these is a **precision-side** instrument. + +Consequence: a report that states three true, trivially-supported facts and +omits everything that mattered passes every gate this system has, at +`citation_validity = 1.0`. Nothing notices. + +That gap now blocks work. HUB-049 is implemented and cannot be adopted; +HUB-045 and HUB-048 would hit the same wall. Two candidate proxies are ruled +out by the prior art rather than by opinion: retrieval recall stops predicting +accuracy once a context budget forces evidence to be discarded (2607.00725), +which is this system's measured condition, and source coverage does not stand +in either (2603.22633 measured large coverage gains worth roughly 0.01 F1). + +## The scoring scheme + +### Two axes, never merged + +Precision and completeness are scored separately and reported separately +(2603.09891 treats completeness as its own axis, measured independently of +whether the cited support checks out; 2509.26184 decomposes a cited report +along the same two axes). + +| Axis | Question | Instrument | Status | +|---|---|---|---| +| Precision | Is what the report said supported by what it cited? | claim gate, `citation_validity` | **exists** | +| Completeness | Did the report say what a good answer must say? | nugget recall (below) | **missing — this item** | + +A single blended score is explicitly rejected. The two axes trade against each +other — a report can buy completeness with unsupported claims — and blending +them hides exactly the trade a change needs to be judged on. This is the same +discipline HUB-044 already applies to coverage versus recall. + +### The completeness metric: nugget recall + +Nugget evaluation comes from the TREC QA track and was revived for RAG by the +AutoNuggetizer framework (2504.15068): a question is distilled into a short +list of **atomic facts a good answer ought to contain**, each labelled +**vital** (must be present) or **okay** (worth having, not essential); a +report is then scored by which nuggets it contains. + +Per question, per report: + +- Each nugget is assigned **supported / partial / absent** against the report + text (the three-way assignment of 2504.15068 — a binary label discards the + common case of a nugget stated incompletely). +- `vital_nugget_recall` — **the primary number.** Fraction of vital nuggets + assigned supported. +- `weighted_nugget_recall` — vital and okay pooled, vital counted double. + *(The 2:1 weight is this design's choice, not an attributed value; it is + reported alongside the unweighted vital figure so the weight can never be + the thing that moves a decision.)* +- `all_nugget_recall` — unweighted, for the record. +- Partial assignments count as 0 in every recall figure and are reported as a + separate `partial_rate`. Crediting half a fact would make a report that + gestures at everything beat one that establishes anything. + +Retrieval coverage@k (HUB-044) continues to be reported beside these and +continues to gate nothing. + +### A nugget is only credited when a *verified* claim carries it + +A nugget is assigned supported only if the report text carrying it is a claim +that passed the claim gate (2509.26184 scores completeness over the report's +*supported* claims specifically). Otherwise completeness could be raised by +emitting unverified assertions, which is the failure the gate exists to +prevent. + +## Construction: where nuggets come from + +This is where the design can quietly destroy its own validity, so it is +specified tightly. + +### Nuggets are mined from documents, not from retrieval + +**The trap:** nuggets mined from what retrieval surfaced can only ever reward +facts retrieval already surfaced (2504.15068's own stated limitation — whole +corpus blind spots stay hidden). The same self-reference already flattered a +published study in the neighbouring literature: SciRet's relevance labels were +generated by the very hybrid system being scored, mechanically producing +perfect recall for that configuration (2608.03860). + +**The rule:** candidate nuggets are extracted from the **retained documents +resolved by scope** — `documents_for_job` for a job-scoped question, +`documents_matching` for a filtered corpus question — never from +`ScopedRetrievalService` output. The pipeline under test must not choose the +yardstick it is measured against. This is testable and will be asserted: the +construction command must not import the retrieval service. + +### The metered judge is not spent on construction + +Drafting runs on the **local** model (`qwen3.5:9b`); the quality bar is human +verification, not the drafter. The sealed v4 protocol already in this repo +established that shape — draft with a model, verify blind by hand, seal the +result — and the drafter's cost is therefore irrelevant to the set's validity. + +**Metered judge calls required by HUB-047: zero.** The only metered calls in +the loop are the claim gate's, which a report generation spends anyway. This +is what makes the item affordable, and it should stay true: any future design +that needs the metered judge per evaluation run has regressed. + +### The expensive resource is operator time, not compute + +Approximately 15 questions × ~12 candidate nuggets ≈ **180 nuggets to verify +and label** vital/okay by hand. That is the real cost of this item and it is +the operator's own hours. It cannot be delegated to the drafter without +reintroducing the circularity the whole design exists to avoid. + +Mitigation for the residual blind spot: for a stated subset of questions the +verifier reads the scope's source documents directly and *adds* nuggets the +drafter missed. Deletion-only verification can prune a bad list but cannot +discover an omission, and omissions are the thing being measured. + +### Two sets, two different disciplines + +These are easy to conflate and the standing v4 obligation makes the confusion +costly: + +| Set | Purpose | Discipline | +|---|---|---| +| **Calibration set** | human labels validating the local assigner | blind, sealed, **consumed — never reused** | +| **Evaluation set** | questions + verified nuggets | **stable, versioned, re-run freely** | + +A regression benchmark that could only be run once would be useless; a +calibration set reused after consumption invalidates the agreement number. +Both statements are true simultaneously and the two artefacts must be stored +and named separately. + +### The local assigner must be calibrated before it is trusted + +Nugget assignment is done locally and unmetered, so its agreement with human +labels is the load-bearing assumption. Calibrate exactly as v4 did: blind +human assignment on the calibration set, then agreement against the local +assigner, recorded before any score is quoted. If agreement fails to clear a +bar stated in advance, the assigner is the finding and the metric does not +ship. Lexical-only matching is expected to be insufficient here — it silently +misses paraphrased-but-correct coverage (2603.09891) — which is precisely why +the assigner is a model and why it must be measured. + +## What the instrument is, and is not, allowed to be used for + +All three fetched abstracts converge on the same caveat: nugget-style scoring +correlates with human judgment **at the system level**, while per-topic and +per-report agreement is noticeably noisier (2504.15068, 2509.26184). + +Therefore: + +- **Allowed:** ranking configurations. Is `EVIDENCE_PACKING=marginal_gain` + better than `rank`? Does late chunking (HUB-045) help? Does a knowledge + graph earn its keep (HUB-048)? +- **Not allowed:** diagnosing a single report, or gating a single report's + publication. A per-report number from this instrument is noise. + +This also sizes the question set: enough questions that a system-level +comparison is meaningful, not enough to make hand-verification impossible. +**15 questions is the design's starting point**, split across the two scopes +HUB-043 created — job-topic questions exercising the report path, and +cross-job corpus questions exercising `/query` and `/rag` — because a metric +that only covers one scope would leave the other unjudgeable. + +## Acceptance criteria + +1. Two axes reported separately; no blended score anywhere. +2. Nuggets provably mined from document scope, not retrieval output — + asserted by test, not by convention. +3. Zero metered judge calls per evaluation run. +4. Local assigner agreement with blind human labels recorded before any + score is quoted, against a bar set in advance. +5. Calibration and evaluation sets stored separately with their different + reuse rules stated in the files themselves. +6. A baseline recorded for the current deployed configuration, so HUB-049's + flag becomes a decidable question. + +## Known limits, recorded rather than discovered later + +- **Recall is against a self-authored reference.** Omissions outside the + nugget lists are invisible (2509.26184, 2603.09891). Directly mitigated only + by the document-reading subset above, and never fully. +- **Per-report scores are noise.** See above; enforced by how the instrument + is allowed to be used. +- **Private corpus, no assessor pool.** TREC-style reference lists rest on + pooled human assessment over a shared public corpus; here there is one + annotator and no inter-assessor agreement to report (2603.09891). Single + annotator is a stated limitation of every number this produces. +- **The corpus grows.** A nugget verified against today's 679 documents may + become answerable in new ways later. The evaluation set is versioned so a + score is always quoted against a corpus state. + +## Citations + +Fetched and read for this design, 2026-08-13: + +- **arXiv:2504.15068** — *The Great Nugget Recall: Automating Fact Extraction + and RAG Evaluation with Large Language Models*. Primary mechanism: vital/okay + nuggets, three-way assignment, weighted recall; automation calibrated + against manual nugget creation on the TREC 2024 RAG Track. Also the source + of the pool-blind-spot limitation. + +- **arXiv:2509.26184** — *Auto-ARGUE: LLM-Based Report Generation Evaluation*. + Supporting evidence: decomposes a citation-backed report into sentence-level + claims and scores attribution and completeness as independent axes, with + completeness measured over supported claims. + +- **arXiv:2603.09891** — *Overview of the TREC 2025 Retrieval Augmented + Generation (RAG) Track*. Supporting evidence for separating completeness + from attribution; source of the private-corpus and lexical-matching limits. + + +Carried from the HUB-049 pass, 2026-08-13: + +- **arXiv:2607.00725** — recall stops predicting accuracy under a context + budget; the reason a retrieval-side proxy cannot stand in here. +- **arXiv:2603.22633** — large source-coverage gains worth roughly 0.01 F1; + the reason coverage cannot stand in either. +- **arXiv:2608.03860** — relevance labels generated by the system under test + mechanically flatter it; the failure mode the construction rule avoids. diff --git a/backlog.md b/backlog.md index 2504ba1..db858fc 100644 --- a/backlog.md +++ b/backlog.md @@ -1521,12 +1521,26 @@ scored on answer correctness and source coverage. HUB-044 is its first metric. This is what would let HUB-045, HUB-048 and HUB-049's flag be decided by measurement rather than argument. -**Open design question, unanswered by the prior art.** Every paper read -validates on short-answer QA with gold spans. This system's output is -long-form cited synthesis over a private, growing corpus, where "correct" is -not a string match. Settle the scoring design before building the set — the -existing sealed judge protocol (v4) is the closest thing already in the repo -and is the obvious starting point. +**Scoring design settled 2026-08-13** — `PRDs/hub-047-answer-completeness-scoring.md`, +grounded in three further fetched abstracts. Stage 1 of the item is done; no +question set is built yet. The design in one paragraph: score precision and +completeness as **two axes that are never blended**, keep the existing claim +gate as the precision axis, and add **nugget recall** as the completeness axis +— per question a short list of atomic facts a good answer must contain, +labelled vital or okay, assigned three-way against the report, and credited +only when carried by a claim that passed the gate. Nuggets are mined from +**document scope, never from retrieval output**, or the pipeline picks its own +yardstick. Drafting is local and human verification is the quality bar, so +**HUB-047 costs zero metered judge calls** — the expensive resource is roughly +180 nuggets of operator hand-verification. Calibration and evaluation sets are +kept separate because one is consumed on use and the other must be re-runnable. + +The instrument ranks **configurations**, not reports: nugget-style scoring +agrees with human judgment at the system level and is noisy per report +(arXiv:2504.15068, arXiv:2509.26184). It may never gate a single report. + +**Next stage:** build the 15-question set against that design — the operator +hand-verification pass is the gating cost and needs scheduling, not code. ### HUB-048 — Knowledge-graph go/no-go, decided by measurement