feat(rag): replace groundedness judgment prompts with the hallucination_detection adapter #1669

Status (2026-10-08). The swap proposed below is on hold until a measured comparison shows it's an improvement. The work is split into sub-issues:

• bug(rag): GroundednessRequirement reports its own failures as "not grounded" #1718: GroundednessRequirement reports its own failures as errors, not verdicts. Ready.
• bug(adapters): hallucination_detection output isn't checked against its own contract #1719: hallucination_detection output is checked against its contract. Ready.
• design(sampling): a requirement that couldn't reach a verdict still triggers repair #1720: sampling stops repairing against verdicts that were never computed. The stop-or-raise decision below is settled there: re-validate once, then stop. Follows bug(rag): GroundednessRequirement reports its own failures as "not grounded" #1718.
• design(rag): measure hallucination_detection against the groundedness judgment prompts before swapping #1721: measure the adapter against the current prompts. That result decides whether the swap goes ahead. Blocked by bug(rag): GroundednessRequirement reports its own failures as "not grounded" #1718 and bug(adapters): hallucination_detection output isn't checked against its own contract #1719.

Checks 1 to 4 under "What to check while doing it" now live in #1721, check 5 in #1720 and check 6 in #1719. From "Also in scope", the docstring fix is in #1719, the backend question in #1718, and the partial-support test and binding-shape coverage in #1721. The coverage check and the swap itself stay here.

A RAG application answers from retrieved documents, and what matters to the people using it
is whether the answer actually came from those documents or from the model's own stock of
plausible-sounding facts. GroundednessRequirement is mellea's check for that. Attach it as
a requirement to a session call, or invoke validate() directly, and you get a pass or fail
with a reason; a failure feeds the repair loop, so the response is generated again.

The reasonable expectation is that the answer to "is this grounded" depends on the response
and the documents. For half of the pipeline it currently also depends on which chat model is
behind the backend, and on the wording of a prompt we ship. There is a published adapter that
answers both of those judgments directly, and we are not using it.

Current behaviour

GroundednessRequirement runs the four stages #775 specified:

1. Citation generation. Link each part of the response to the passage it came from. Asks
   a trained adapter (find_citations).
2. Citation necessity. Decide which parts need a citation at all, so that "I'm happy to
   help" is not treated as an uncited claim. Sends a hand-written prompt to the chat model
   (rag.py:336).
3. Citation support. For the parts that need one, judge whether the citations fully,
   partly or do not support them. Sends a second hand-written prompt (rag.py:436).
4. Verdict. Combine those judgments into the pass or fail and the reason. Plain code.

flowchart LR
    IN["response + documents"] --> S1
    S1["1. citation generation<br/>trained adapter"] --> S2
    S2["2. citation necessity<br/>hand-written prompt"] --> S3
    S3["3. citation support<br/>hand-written prompt"] --> S4
    S4["4. verdict<br/>plain code"] --> OUT["pass / fail + reason"]

    classDef trained fill:#cde7cd,stroke:#4a7a4a,color:#111
    classDef prompt fill:#f5cfcf,stroke:#a35050,color:#111
    classDef plain fill:#d3e2f5,stroke:#48688f,color:#111
    class S1 trained
    class S2,S3 prompt
    class S4 plain

Problem

Take a response where every claim is properly cited. The support stage shows the chat model
each span with its evidence and asks for a flat answer, one entry per span:

[{"span_id": 0, "support_level": "FULLY_SUPPORTED"}]

Until #1618 the prompt labelled each span's evidence with the key citations, and
granite-4.0-micro answered by copying that nested shape back instead, about 70% of the time
(#1316):

[{"span_id": 0, "citations": [{"support_level": "FULLY_SUPPORTED"}]}]

The parser looked for support_level at the top of each entry, found none, and fell back to
its default of NOT_SUPPORTED. A fully grounded response was reported as ungrounded, the
reason quoted the user's own sentences back as unsupported, and the repair loop rewrote an
answer that had been correct.

Two changes hold that off today: #1292 taught the parser the nested shape, and #1618 renamed
the prompt's key so it no longer resembles the shape being requested. Both are workarounds
fitted to one model's habit and one prompt's wording. A later prompt edit can reintroduce the
mirroring, a different chat model can mirror something else, and the user sees a confident
wrong verdict either way. Every groundedness defect so far has been in this layer: #1316,
#1614, and the parser bug #1292 fixed. Three defects across five references, since #1618 is the
fix for #1316 and #1625 the fix for #1614.

The adapter already answers both judgments

hallucination_detection takes the response and the documents and returns one record per
sentence. Its output contract, from io.yaml at the revision the catalogue pins
(_RAG_SHA = 2f0b2c79), is a four-value enum, identical for granite-4.0-micro, 4.1-3b,
4.1-8b and 4.1-30b:

So NA is the necessity judgment and the other three are the support judgment, from one
call, with trained weights, under a contract that raises AdapterSchemaMismatchError when
the shape is wrong. mellea already wraps it as flag_hallucinated_content
(mellea/stdlib/components/intrinsic/rag.py:235), returning response_begin,
response_end, response_text, faithfulness and explanation. No requirement calls it; the
callers today are the examples, the docs, and test/stdlib/components/intrinsic/.

Proposal

Replace both judgment prompts with one flag_hallucinated_content call.

flowchart LR
    IN["response + documents"] --> S1
    S1["citation generation<br/>trained adapter"] --> S2
    S2["necessity + support<br/>hallucination_detection"] --> S3
    S3["verdict<br/>plain code"] --> OUT["pass / fail + reason"]

    classDef trained fill:#cde7cd,stroke:#4a7a4a,color:#111
    classDef plain fill:#d3e2f5,stroke:#48688f,color:#111
    class S1,S2 trained
    class S3 plain

What that removes: the necessity prompt (rag.py:552), the support prompt (rag.py:588),
all three parsers (rag.py:653, :706, :818), the label and identifier normalisers
(rag.py:31-67), and _SUPPORT_LEVEL_RANK. The verdict logic in
_build_groundedness_result stays, reading adapter records instead of parsed judgments.

What changes for a user: allow_partial_support keeps its meaning, since partial is a real
adapter value rather than a word in a prompt. Verdicts become per sentence rather than per
citation-derived span, so the reason string names sentences instead of fragments. That is a
visible output change and needs a changelog note.

Every part of the response must get a verdict, and an incomplete answer is not a verdict.
This is the part the swap does not fix by itself. The contract validates the keys of each record
and nothing else: _ListContract.parse (mellea/backends/adapters/_core.py:213-268) checks that
the output is a JSON array of objects each carrying the required keys. It does not check that the
records account for the whole response, that faithfulness holds one of the four contract values,
or that the offsets are integers in range and non-overlapping.
With max_completion_tokens: 4096, and the io.yaml noting that the LoRA "can be quite verbose
in its explanations", a long response can come back with fewer records than it has sentences.
Filling the gaps with a default reproduces exactly the bug described above: default to
unsupported and a grounded response fails again, default to grounded and it is worse, a silent
pass.

So coverage has to be checked and reported rather than patched over. The records carry
response_begin and response_end, so the check is whether those ranges account for the
response text, which mellea can verify without reimplementing the adapter's sentence
splitting. An incompletely covered response fails closed with the exception on
result.error, which is the convention Requirement.validate already uses for unparsable
adapter output (mellea/core/requirement.py:433-453): reason stays None so a repair
strategy falls back to the requirement description rather than being handed a verdict nobody
computed.

An unrecognized faithfulness value fails closed the same way. The verdict maps four contract
values onto three support levels, so a fifth value has no defined meaning and must not be quietly
treated as any of them. The enum and offset checks belong in the contract rather than in
GroundednessRequirement, since every caller of flag_hallucinated_content wants the same
guarantee.

GroundednessRequirement sets error nowhere today. It builds ValidationResult ten times
without it, and the three handlers at rag.py:254, :272 and :293 hold an exception and
discard it, which is why an adapter or backend failure currently reads as "not grounded".
Those handlers get error=e as part of this change.

Reporting honestly is still not enough on its own, because the retry machinery cannot see the
difference. The sampling loop builds its retry list by filtering on as_bool()
(mellea/stdlib/sampling/base.py:725-727), and as_bool() is False whenever error is set
(requirement.py:110). Nothing under mellea/stdlib/sampling/ reads .error at all. So an
undetermined verdict counts as a failure, the response is regenerated up to the loop budget,
and the model is told "The following requirements failed before" followed by the requirement's
description (base.py:729-740).

Measured against the three harms in the Problem section: the coverage check and error fix the
invented reason and the wrong verdict, and leave the third untouched. A correct answer is still
rewritten because of a check that never ran.

So the fix has a sampling-side half. A requirement that did not reach a verdict must not drive
repair:

• If it is the only failure, the loop stops and returns the response already produced, with the
  error still on the result for the caller to inspect.
• If other requirements genuinely failed, repair proceeds on those alone, and the undetermined
  one contributes no instructions.

This half is not new with this change. ALoraRequirement already produces an errored result
when its adapter output cannot be parsed (requirement.py:110 and :447-453), so the loop
already repairs against undetermined verdicts in shipped code. It belongs here because this
change is what makes that state routine rather than rare.

Decision needed before implementation: quiet stop or raise. Stopping quietly matches the
existing convention, since ValidationResult fails closed and carries the exception rather than
propagating it, and it keeps a transient adapter problem from breaking a caller who would have
accepted an unvalidated response. Raising is harder to ignore and surfaces a broken deployment
at once. Either is defensible; the change should not start until it is settled.

What to check while doing it

The contract says the adapter can express all four values. Whether the model uses them well
on real responses is a separate question, and these are the checks the work carries rather
than blockers on starting it:

1. Does it emit partial and NA in practice? Run flag_hallucinated_content over the
   e2e fixtures in test/stdlib/requirements/test_groundedness_requirement_e2e.py, extended
   with a genuinely partially supported response, since the fixtures have none today. Compare
   the verdicts against the current prompts'.

2. Sentence boundaries. Today a span is cut at citation boundaries
   (_extract_response_spans, rag.py:467), so a half-cited sentence becomes two spans. The
   adapter splits by sentence. Decide what the reason string reports, and whether
   _extract_response_spans is still needed at all.

3. Is find_citations still required for the verdict? The adapter compares the response
   against the documents itself, so citations may only be needed for the citation output, not
   the judgment. If so the pipeline is two calls rather than three.

4. Cost. One call with max_completion_tokens: 4096 (the io.yaml notes the LoRA "can be
   quite verbose in its explanations") against today's two calls capped at 500. Measure on
   both binding shapes; a Granite Switch deployment has no weight-swap cost
   (EmbeddedBinding, _core.py:856), the local-file path does.

5. No repair on an undetermined verdict. With a forced coverage failure, assert that the
   sampling loop does not regenerate when groundedness is the only failing requirement, and
   that when another requirement does fail, the repair instructions name only that one.

6. Contract rejection is tested, not assumed. Unit tests for a record with an unlisted
   faithfulness value, a non-integer offset, an out-of-range offset, and two overlapping
   records, each asserting AdapterSchemaMismatchError, and that GroundednessRequirement
   reports it through result.error with reason still None. No GPU needed for these.

Checks 1 to 5 need a GPU, and granite-4.0-micro publishes hallucination_detection as LoRA only,
which is the model the fixtures already use.

Also in scope

Small things this work runs into anyway, recorded here rather than as separate issues:

• Fix the adapter docstring. flag_hallucinated_content gives the faithfulness values as
  "e.g. "faithful", "hallucinated"" (rag.py:249-251). The contract has no hallucinated,
  and partial and NA are missing entirely, which is why this adapter looked unusable for a
  three-level verdict.
• Settle which backends this requirement supports. The class docstring says a Hugging Face
  backend is required; the gate at rag.py:212 accepts any AdapterMixin, so OpenAIBackend
  and OllamaModelBackend pass too. The work needs an answer anyway, since it decides where
  hallucination_detection can run. Make the docstring and the gate agree.
• Give the partial path a real test. test_groundedness_e2e_allow_partial_support asserts
  only isinstance(result.as_bool(), bool), and no fixture produces a genuinely partially
  supported response, so nothing would notice if partial stopped working. The fixture this
  work adds for check 1 fixes both.
• Exercise both binding shapes for this requirement while measuring cost, rather than only
  the local-file path the fixtures use today. Making Granite Switch coverage general, beyond
  groundedness, is wider than this issue and may want its own once we see how this goes.

Related

• feat: intrinsic-backed Requirement subclass for Guardian safety validation in repair loops #1071 proposes an intrinsic-backed Requirement for Guardian safety validation. Same shape of
  change: a requirement doing its own thing beside a published adapter that already does it.
  Worth keeping the two consistent.
• feat(rag): constrain groundedness judgments with response schemas #1629 asked for response schemas on the two judgment prompts. Closed, and moot if this lands,
  since the prompts go away.