HUB-047 stage 1: scoring design for end-to-end answer quality

[funsaized]

Design only — no question set, no code. Settling how a report is scored first, because the scheme determines what the set must contain.

The gap

The system verifies every claim it publishes and measures nothing about what it failed to say. A report stating three true trivial facts and omitting everything that mattered passes every existing gate at citation_validity = 1.0. Every instrument in the repo — claim gate, citation validity, exact-term recall, coverage@k — is precision-side.

The design

Two axes, never blended. The claim gate stays the precision axis. Completeness becomes nugget recall: per question a short list of atomic facts a good answer must contain, labelled vital/okay, assigned supported/partial/absent, and credited only when carried by a claim that passed the gate — otherwise completeness could be bought with unverified assertions.

Three decisions worth naming:

1. Nuggets are mined from document scope, never from retrieval output. A pipeline that picks its own yardstick scores itself — the exact self-reference that mechanically flattered arXiv:2608.03860. To be enforced by test: the construction command must not import the retrieval service.
2. Zero metered judge calls. Drafting is local; human blind verification is the quality bar, per the v4 protocol already in the repo. The real cost is ~180 hand-verified nuggets of operator time, and that cannot be delegated to the drafter without reintroducing the circularity.
3. Calibration set ≠ evaluation set. One is blind, sealed and consumed on use; the other must be stable and re-runnable. Conflating them either invalidates the agreement number or produces a benchmark that can only run once.

It ranks configurations, not reports. All three fetched abstracts agree nugget scoring tracks human judgment at system level and is noisy per topic. It may never gate a single report — only answer "is marginal_gain better than rank", "does late chunking help", "does a knowledge graph earn its keep".

Citations

Fetched and read for this design: arXiv:2504.15068 (vital/okay nuggets, three-way assignment, weighted recall) · arXiv:2509.26184 (attribution and completeness as independent axes over supported claims) · arXiv:2603.09891 (completeness as its own axis; private-corpus and lexical-matching limits). Carried from the HUB-049 pass: 2607.00725, 2603.22633, 2608.03860.

Known limits are recorded in the PRD rather than left to be discovered: recall is against a self-authored reference, single annotator with no inter-assessor agreement, and a growing corpus means scores are always quoted against a corpus state.

🤖 Generated with Claude Code