From 28a97322dbd9c82f3660e436f85072564163142b Mon Sep 17 00:00:00 2001 From: funsaized Date: Thu, 13 Aug 2026 21:37:25 -0400 Subject: [PATCH] docs(prd): settle the HUB-047 scoring design before building the set (stage 1) The system verifies every claim it publishes and measures nothing about what it failed to say: a report stating three true trivial facts passes every gate at citation_validity 1.0. That gap now blocks HUB-045, HUB-048 and HUB-049's flag, and the two available proxies are ruled out by prior art rather than opinion -- recall stops predicting accuracy under a context budget (arXiv:2607.00725) and coverage is worth ~0.01 F1 (arXiv:2603.22633). The design: precision and completeness as two axes that are never blended, the existing claim gate serving the first, and nugget recall serving the second -- atomic facts a good answer must contain, labelled vital or okay, assigned three-way, credited only when carried by a gate-passed claim (arXiv:2504.15068, arXiv:2509.26184, arXiv:2603.09891). Three decisions worth naming. Nuggets are mined from document scope and never from retrieval output, because a pipeline that picks its own yardstick scores itself -- the failure that mechanically flattered arXiv:2608.03860, and to be enforced by test rather than convention. Drafting is local with human blind verification as the quality bar, so the item costs zero metered judge calls and the real price is ~180 hand-verified nuggets of operator time. The calibration set and the evaluation set are separate artefacts because one is consumed on use and the other must be re-runnable. The instrument ranks configurations, not reports: all three fetched abstracts agree that nugget scoring tracks human judgment at system level and is noisy per report, so it may never gate a single report. Design only. No question set, no code. Co-Authored-By: Claude Opus 5 (1M context) --- PRDs/hub-047-answer-completeness-scoring.md | 236 ++++++++++++++++++++ backlog.md | 26 ++- 2 files changed, 256 insertions(+), 6 deletions(-) create mode 100644 PRDs/hub-047-answer-completeness-scoring.md diff --git a/PRDs/hub-047-answer-completeness-scoring.md b/PRDs/hub-047-answer-completeness-scoring.md new file mode 100644 index 0000000..926d99f --- /dev/null +++ b/PRDs/hub-047-answer-completeness-scoring.md @@ -0,0 +1,236 @@ +# HUB-047 — Scoring design for end-to-end answer quality + +Status: DESIGN ONLY, 2026-08-13. No question set is built and no code is +written. This document settles *how a report is scored* first, because the +scoring scheme determines what the question set must contain, and building +the set first would force the scoring to fit whatever was collected. + +Prior art reviewed: 3 arXiv abstracts fetched and read 2026-08-13 for this +design, plus 16 read for the HUB-049 pass (citations at the end; every claim +below is attributed to a fetched abstract or marked as this design's own +choice). + +## Problem + +The system verifies every claim it publishes and measures nothing about what +it failed to say. + +The claim gate checks each drafted claim against its cited evidence, with +per-reference necessity, and rejects padding. `benchmark_report_retrieval.py` +checks citation validity. `benchmark_retrieval_exact_terms.py` checks that +needles are retrievable. HUB-044 measures retrieval breadth. Every one of +these is a **precision-side** instrument. + +Consequence: a report that states three true, trivially-supported facts and +omits everything that mattered passes every gate this system has, at +`citation_validity = 1.0`. Nothing notices. + +That gap now blocks work. HUB-049 is implemented and cannot be adopted; +HUB-045 and HUB-048 would hit the same wall. Two candidate proxies are ruled +out by the prior art rather than by opinion: retrieval recall stops predicting +accuracy once a context budget forces evidence to be discarded (2607.00725), +which is this system's measured condition, and source coverage does not stand +in either (2603.22633 measured large coverage gains worth roughly 0.01 F1). + +## The scoring scheme + +### Two axes, never merged + +Precision and completeness are scored separately and reported separately +(2603.09891 treats completeness as its own axis, measured independently of +whether the cited support checks out; 2509.26184 decomposes a cited report +along the same two axes). + +| Axis | Question | Instrument | Status | +|---|---|---|---| +| Precision | Is what the report said supported by what it cited? | claim gate, `citation_validity` | **exists** | +| Completeness | Did the report say what a good answer must say? | nugget recall (below) | **missing — this item** | + +A single blended score is explicitly rejected. The two axes trade against each +other — a report can buy completeness with unsupported claims — and blending +them hides exactly the trade a change needs to be judged on. This is the same +discipline HUB-044 already applies to coverage versus recall. + +### The completeness metric: nugget recall + +Nugget evaluation comes from the TREC QA track and was revived for RAG by the +AutoNuggetizer framework (2504.15068): a question is distilled into a short +list of **atomic facts a good answer ought to contain**, each labelled +**vital** (must be present) or **okay** (worth having, not essential); a +report is then scored by which nuggets it contains. + +Per question, per report: + +- Each nugget is assigned **supported / partial / absent** against the report + text (the three-way assignment of 2504.15068 — a binary label discards the + common case of a nugget stated incompletely). +- `vital_nugget_recall` — **the primary number.** Fraction of vital nuggets + assigned supported. +- `weighted_nugget_recall` — vital and okay pooled, vital counted double. + *(The 2:1 weight is this design's choice, not an attributed value; it is + reported alongside the unweighted vital figure so the weight can never be + the thing that moves a decision.)* +- `all_nugget_recall` — unweighted, for the record. +- Partial assignments count as 0 in every recall figure and are reported as a + separate `partial_rate`. Crediting half a fact would make a report that + gestures at everything beat one that establishes anything. + +Retrieval coverage@k (HUB-044) continues to be reported beside these and +continues to gate nothing. + +### A nugget is only credited when a *verified* claim carries it + +A nugget is assigned supported only if the report text carrying it is a claim +that passed the claim gate (2509.26184 scores completeness over the report's +*supported* claims specifically). Otherwise completeness could be raised by +emitting unverified assertions, which is the failure the gate exists to +prevent. + +## Construction: where nuggets come from + +This is where the design can quietly destroy its own validity, so it is +specified tightly. + +### Nuggets are mined from documents, not from retrieval + +**The trap:** nuggets mined from what retrieval surfaced can only ever reward +facts retrieval already surfaced (2504.15068's own stated limitation — whole +corpus blind spots stay hidden). The same self-reference already flattered a +published study in the neighbouring literature: SciRet's relevance labels were +generated by the very hybrid system being scored, mechanically producing +perfect recall for that configuration (2608.03860). + +**The rule:** candidate nuggets are extracted from the **retained documents +resolved by scope** — `documents_for_job` for a job-scoped question, +`documents_matching` for a filtered corpus question — never from +`ScopedRetrievalService` output. The pipeline under test must not choose the +yardstick it is measured against. This is testable and will be asserted: the +construction command must not import the retrieval service. + +### The metered judge is not spent on construction + +Drafting runs on the **local** model (`qwen3.5:9b`); the quality bar is human +verification, not the drafter. The sealed v4 protocol already in this repo +established that shape — draft with a model, verify blind by hand, seal the +result — and the drafter's cost is therefore irrelevant to the set's validity. + +**Metered judge calls required by HUB-047: zero.** The only metered calls in +the loop are the claim gate's, which a report generation spends anyway. This +is what makes the item affordable, and it should stay true: any future design +that needs the metered judge per evaluation run has regressed. + +### The expensive resource is operator time, not compute + +Approximately 15 questions × ~12 candidate nuggets ≈ **180 nuggets to verify +and label** vital/okay by hand. That is the real cost of this item and it is +the operator's own hours. It cannot be delegated to the drafter without +reintroducing the circularity the whole design exists to avoid. + +Mitigation for the residual blind spot: for a stated subset of questions the +verifier reads the scope's source documents directly and *adds* nuggets the +drafter missed. Deletion-only verification can prune a bad list but cannot +discover an omission, and omissions are the thing being measured. + +### Two sets, two different disciplines + +These are easy to conflate and the standing v4 obligation makes the confusion +costly: + +| Set | Purpose | Discipline | +|---|---|---| +| **Calibration set** | human labels validating the local assigner | blind, sealed, **consumed — never reused** | +| **Evaluation set** | questions + verified nuggets | **stable, versioned, re-run freely** | + +A regression benchmark that could only be run once would be useless; a +calibration set reused after consumption invalidates the agreement number. +Both statements are true simultaneously and the two artefacts must be stored +and named separately. + +### The local assigner must be calibrated before it is trusted + +Nugget assignment is done locally and unmetered, so its agreement with human +labels is the load-bearing assumption. Calibrate exactly as v4 did: blind +human assignment on the calibration set, then agreement against the local +assigner, recorded before any score is quoted. If agreement fails to clear a +bar stated in advance, the assigner is the finding and the metric does not +ship. Lexical-only matching is expected to be insufficient here — it silently +misses paraphrased-but-correct coverage (2603.09891) — which is precisely why +the assigner is a model and why it must be measured. + +## What the instrument is, and is not, allowed to be used for + +All three fetched abstracts converge on the same caveat: nugget-style scoring +correlates with human judgment **at the system level**, while per-topic and +per-report agreement is noticeably noisier (2504.15068, 2509.26184). + +Therefore: + +- **Allowed:** ranking configurations. Is `EVIDENCE_PACKING=marginal_gain` + better than `rank`? Does late chunking (HUB-045) help? Does a knowledge + graph earn its keep (HUB-048)? +- **Not allowed:** diagnosing a single report, or gating a single report's + publication. A per-report number from this instrument is noise. + +This also sizes the question set: enough questions that a system-level +comparison is meaningful, not enough to make hand-verification impossible. +**15 questions is the design's starting point**, split across the two scopes +HUB-043 created — job-topic questions exercising the report path, and +cross-job corpus questions exercising `/query` and `/rag` — because a metric +that only covers one scope would leave the other unjudgeable. + +## Acceptance criteria + +1. Two axes reported separately; no blended score anywhere. +2. Nuggets provably mined from document scope, not retrieval output — + asserted by test, not by convention. +3. Zero metered judge calls per evaluation run. +4. Local assigner agreement with blind human labels recorded before any + score is quoted, against a bar set in advance. +5. Calibration and evaluation sets stored separately with their different + reuse rules stated in the files themselves. +6. A baseline recorded for the current deployed configuration, so HUB-049's + flag becomes a decidable question. + +## Known limits, recorded rather than discovered later + +- **Recall is against a self-authored reference.** Omissions outside the + nugget lists are invisible (2509.26184, 2603.09891). Directly mitigated only + by the document-reading subset above, and never fully. +- **Per-report scores are noise.** See above; enforced by how the instrument + is allowed to be used. +- **Private corpus, no assessor pool.** TREC-style reference lists rest on + pooled human assessment over a shared public corpus; here there is one + annotator and no inter-assessor agreement to report (2603.09891). Single + annotator is a stated limitation of every number this produces. +- **The corpus grows.** A nugget verified against today's 679 documents may + become answerable in new ways later. The evaluation set is versioned so a + score is always quoted against a corpus state. + +## Citations + +Fetched and read for this design, 2026-08-13: + +- **arXiv:2504.15068** — *The Great Nugget Recall: Automating Fact Extraction + and RAG Evaluation with Large Language Models*. Primary mechanism: vital/okay + nuggets, three-way assignment, weighted recall; automation calibrated + against manual nugget creation on the TREC 2024 RAG Track. Also the source + of the pool-blind-spot limitation. + +- **arXiv:2509.26184** — *Auto-ARGUE: LLM-Based Report Generation Evaluation*. + Supporting evidence: decomposes a citation-backed report into sentence-level + claims and scores attribution and completeness as independent axes, with + completeness measured over supported claims. + +- **arXiv:2603.09891** — *Overview of the TREC 2025 Retrieval Augmented + Generation (RAG) Track*. Supporting evidence for separating completeness + from attribution; source of the private-corpus and lexical-matching limits. + + +Carried from the HUB-049 pass, 2026-08-13: + +- **arXiv:2607.00725** — recall stops predicting accuracy under a context + budget; the reason a retrieval-side proxy cannot stand in here. +- **arXiv:2603.22633** — large source-coverage gains worth roughly 0.01 F1; + the reason coverage cannot stand in either. +- **arXiv:2608.03860** — relevance labels generated by the system under test + mechanically flatter it; the failure mode the construction rule avoids. diff --git a/backlog.md b/backlog.md index 2504ba1..db858fc 100644 --- a/backlog.md +++ b/backlog.md @@ -1521,12 +1521,26 @@ scored on answer correctness and source coverage. HUB-044 is its first metric. This is what would let HUB-045, HUB-048 and HUB-049's flag be decided by measurement rather than argument. -**Open design question, unanswered by the prior art.** Every paper read -validates on short-answer QA with gold spans. This system's output is -long-form cited synthesis over a private, growing corpus, where "correct" is -not a string match. Settle the scoring design before building the set — the -existing sealed judge protocol (v4) is the closest thing already in the repo -and is the obvious starting point. +**Scoring design settled 2026-08-13** — `PRDs/hub-047-answer-completeness-scoring.md`, +grounded in three further fetched abstracts. Stage 1 of the item is done; no +question set is built yet. The design in one paragraph: score precision and +completeness as **two axes that are never blended**, keep the existing claim +gate as the precision axis, and add **nugget recall** as the completeness axis +— per question a short list of atomic facts a good answer must contain, +labelled vital or okay, assigned three-way against the report, and credited +only when carried by a claim that passed the gate. Nuggets are mined from +**document scope, never from retrieval output**, or the pipeline picks its own +yardstick. Drafting is local and human verification is the quality bar, so +**HUB-047 costs zero metered judge calls** — the expensive resource is roughly +180 nuggets of operator hand-verification. Calibration and evaluation sets are +kept separate because one is consumed on use and the other must be re-runnable. + +The instrument ranks **configurations**, not reports: nugget-style scoring +agrees with human judgment at the system level and is noisy per report +(arXiv:2504.15068, arXiv:2509.26184). It may never gate a single report. + +**Next stage:** build the 15-question set against that design — the operator +hand-verification pass is the gating cost and needs scheduling, not code. ### HUB-048 — Knowledge-graph go/no-go, decided by measurement