design(rag): measure hallucination_detection against the groundedness judgment prompts before swapping #1721

GroundednessRequirement decides whether each part of a response needs a citation, and whether its citations support it, by sending two hand-written prompts to the chat model. #1669 proposes replacing both with one call to the hallucination_detection adapter. That swap is on hold until a comparison on the same inputs shows it's an improvement. Nobody has measured both on the same data yet, and the adapters themselves are due to change.

What matters to users is that this requirement gates the response. A false "not grounded" sends a correct answer through repair. A false "grounded" lets an invented claim through. So the comparison measures both, for each candidate.

What to compare

Arms, on the same inputs and base model:

1. Current prompts. The necessity and support prompts as they ship, after bug(rag): GroundednessRequirement reports its own failures as "not grounded" #1718.

2. Trained adapter. find_citations is still called for the citation output, but the verdict comes from flag_hallucinated_content alone:

   • any unfaithful sentence fails the response;
   • a partial sentence fails it unless allow_partial_support is set;
   • NA sentences are ignored;
   • a sentence with no record, or a contract error from bug(adapters): hallucination_detection output isn't checked against its own contract #1719, gives an errored result, which counts as a hard failure.

   Run it with allow_partial_support both off and on.

3. Prompt fallback, if one exists by then. feat(adapters)!: prompt fallback for adapter functions without a trained adapter #1654 resolves aLoRA, then LoRA, then a prompt. Every Granite base model the catalogue lists has a trained LoRA for this adapter, and the public ibm-granite/granitelib-rag-r1.0 tree has no prompt/ or any/ directory for hallucination_detection. So this arm needs a published prompt io.yaml and a way to force the prompt rung, and it runs on LocalHF only. If it can run, report it as information; the decision below compares arms 1 and 2.

The ten e2e cases in test/stdlib/requirements/test_groundedness_requirement_e2e.py are too few to decide anything. Build a labelled set covering:

• fully grounded responses, including ones that combine facts from several documents without quoting any one of them;
• genuinely partially supported responses (no fixture has one today);
• responses with an invented claim;
• responses with non-factual sentences (greetings, connecting text), where the right answer is "needs no citation";
• long responses, to see how output length behaves against the adapter's 4096-token cap.

A public set such as RAGTruth (MIT licence) could supply some of these. It has no "needs no citation" labels, and its source passages have their own terms, so check those before committing anything as fixtures.

What to measure

Per arm:

• False fails on grounded responses. This is the number that decides the swap, because in a gate it's what causes repair.
• Misses on responses with an invented claim.
• Whether partial and NA are used where they apply.
• Hard failures: errored results, truncated output, out-of-range sentence indexes.
• Stability: both arms run at temperature 0.0 (mellea/stdlib/requirements/rag.py:340, :440, and the adapter's io.yaml). Repeat each item to confirm verdicts are stable, and compare verdicts across the two binding shapes for arm 2.
• Cost: latency and tokens per validation on both binding shapes. On the local-file path (LocalFileBinding, mellea/backends/adapters/_core.py:374) the LoRA is loaded and swapped; on Granite Switch (EmbeddedBinding, _core.py:856) it's already in the served model. The current prompts are capped at max_new_tokens=500 per call (rag.py:125), a cap that only takes effect on HF until bug(rag): GroundednessRequirement reports its own failures as "not grounded" #1718 fixes it; the adapter's io.yaml allows 4096.

Arm 2 on Granite Switch also gives GroundednessRequirement its first functional run on a non-HF backend.

Two design questions from #1669 are answered here too:

• Is find_citations still needed for the verdict? Arm 2 doesn't use it for the verdict, so if arm 2 wins, the answer is no, and the pipeline needs citations only for the citation output.
• What should the reason string report? Today spans are cut at citation boundaries (_extract_response_spans, rag.py:467), so a half-cited sentence becomes two spans. The adapter judges whole sentences.

Decision rule

Record these in this issue before running, so the result can't move the bar:

• the size of the labelled set, and how many items of each kind it has;
• the margin by which arm 2's false fails and misses may differ from arm 1's and still count as "no worse";
• the paired test on the items where the arms disagree (McNemar's test is the usual choice for paired pass/fail outcomes);
• the hard-failure rate arm 2 must stay under.

The swap in #1669 goes ahead only if arm 2 is no worse than arm 1 on false fails and misses, by that margin and test, and stays under the hard-failure threshold. Otherwise we keep the prompts, record the numbers here, and close #1669.

When to start

Once #1718 and #1719 have merged, and the catalogue pins the adapter revisions we expect to ship with. Running earlier measures adapters that are about to change. Needs a GPU.

Deliverable

A results comment here with the per-arm numbers and a go or no-go for #1669, plus the labelled set committed as fixtures so a later adapter bump can re-run it.

Separately, and without waiting for any of the above, a genuinely partially supported fixture would give test_groundedness_e2e_allow_partial_support (test_groundedness_requirement_e2e.py:132) a real assertion; today it checks only isinstance(result.as_bool(), bool).

Line references are against 46ad3b3a5.