bug(rag): GroundednessRequirement reports its own failures as "not grounded" #1718

When GroundednessRequirement can't do its job, the caller is told the response isn't grounded. That covers a missing or broken citations adapter, a backend error during a judgment call, judgment output that can't be read, and setup problems like a context with no documents. None of those are a verdict on the response, but they all come back as one.

Two things then go wrong:

• The caller can't tell a failure from a verdict. result.as_bool() is False and result.error is None, so a broken deployment looks exactly like an ungrounded answer.
• The model gets the wrong repair instruction. RepairTemplateStrategy uses a truthy reason as prompt text (mellea/stdlib/sampling/base.py:631-646). After an adapter failure the model is asked to fix "Citation generation adapter function failed: ...". With no documents attached, it's told "No documents provided for grounding validation". When a judgment can't be read, the parser fills in a default, so the reason names the user's own sentences as unsupported.

Where

mellea/stdlib/requirements/rag.py:

• Setup checks (:179-216): five early returns (empty context, last item not a Message, last message not from the assistant, no documents, backend without adapter support) return a failure with only a reason.
• Stage handlers (:254, :272, :293): catch the exception and return ValidationResult(False, reason=f"... failed: {e!s}"). error is never set.
• Support parser defaults. Unreadable output marks every span NOT_SUPPORTED (:808-814); that except also catches anything raised inside the parser. A missing span (:799-806), an empty label (:788) and an unrecognised label (_norm, :742) also become NOT_SUPPORTED.
• Necessity parser defaults. Unreadable output (:871-877), a missing span (:862-869) and an unrecognised value (_normalise_needs_citation, :55-69) all become "needs a citation".

Requirement.validate already handles this case for unparsable adapter output (mellea/core/requirement.py:440-453): reason=None, error=exc. The comment there explains why: a truthy reason becomes repair text, and None sends repair to the requirement's description.

Proposed fix

1. Setup checks and stage handlers return ValidationResult(False, reason=None, error=...). For the setup checks, raise the existing message as the exception (for example a ValueError) so it isn't lost. For stages 2 and 3, the handler receives the ValueError("LLM judgment failed: ...") wrapper with the original on __context__; store what the handler caught.

2. Parsers mark what they couldn't read as unknown instead of filling in a default. That covers all the defaults listed above. Remove the support parser's own catch-all at :808, so a span it couldn't read isn't turned back into NOT_SUPPORTED.

3. One rule in _build_groundedness_result: an unknown part never overrides a determined verdict.

   • If any span determinately fails, return the ordinary failure, naming only those spans. A span determinately fails when its necessity was read as "needs a citation" and either it has no citation (decided without a model call, :408-410) or its support was read as failing.
   • Otherwise, if any span's outcome is unknown (its support couldn't be read, or it would fail only because its necessity couldn't be read), return an errored result.
   • Otherwise pass. A pass that relied on assuming "needs a citation" for an unread necessity is kept: that's the strictest assumption, so the response is grounded either way.

   Net effect: nothing that passes today fails, and no determined failure is lost. The only results that change are failures that become errored results, which are still False.

4. Keep the failure detail visible.

   • ModelFriendlyRepairStrategy's generic branch prints Details: {validation_result.reason} (mellea/stdlib/sampling/feedback.py:346-349), so the model would read "Details: None". Leave the details clause out when there's no reason.
   • The debug plugins only print a truthy reason (mellea/plugins/builtin_debug/validation.py:121-123, mellea/plugins/builtin_debug/sampling.py:108-110). Fall back to str(result.error).

   ALoraRequirement produces the same kind of result today, so it benefits too. Telemetry for errored results is design(sampling): a requirement that couldn't reach a verdict still triggers repair #1720.

5. Fix the docstrings.

   • The class docstring (rag.py:87-88) and validate docstring (:167-168) say a LocalHFBackend is required. The gate at :213 accepts any AdapterMixin, and OpenAI (Granite Switch) and Ollama (with a citations tag in adapter_models) both reach find_citations. Update the docstrings to match the gate. Keep the gate as it is: narrowing it would fail every validation for OpenAI and Ollama users.
   • ValidationResult's class docstring (mellea/core/requirement.py:43-47), its error property (:91-99) and as_bool (:101-106) describe error as "output could not be parsed". Widen them to cover a validator that couldn't run.

6. Make max_new_tokens work off HF. The judgment calls pass it as a plain "max_new_tokens" key (rag.py:341, :441). Only the HF backend reads that; OpenAI drops it and Ollama ignores it. Use ModelOption.MAX_NEW_TOKENS, which both map (mellea/backends/openai.py:201, mellea/backends/ollama.py:320).

Tests

No test drives validate() as far as the stage handlers: the mocked-backend tests in test/stdlib/requirements/test_groundedness_requirement.py stop earlier, and the parser tests call the parsers directly. Add unit tests with a mocked AdapterMixin backend for:

• each setup check and each stage raising: as_bool() is False, reason is None, error is set;
• each kind of unreadable parser input producing an unknown;
• the verdict rule: a determinate failure plus an unknown gives the ordinary failure naming only the determinate span; only unknowns give an errored result; an unread necessity with passing support gives a pass;
• ModelFriendlyRepairStrategy given an errored result: no "Details: None";
• a non-HF backend receiving the token cap.

These existing tests change on purpose. Update them rather than delete them:

• setup-check reason assertions at :73, :92 and :158: assert on str(result.error) instead;
• support parser tests pinning NOT_SUPPORTED defaults at :514, :524, :540, :848, :894, :939, :961 and :976: assert unknown instead;
• the necessity parser test at :563, which pins invalid labels to True.

One e2e effect: test_groundedness_e2e_ungrounded_claim asserts reason is not None (test/stdlib/requirements/test_groundedness_requirement_e2e.py:81). Today an adapter failure satisfies it without testing anything; after this change it fails instead, which is right. Assert result.error is None first so the failure message says why.

Out of scope

• The sampling loop still repairs after an errored result, and telemetry still counts it as an ordinary failure. Both are design(sampling): a requirement that couldn't reach a verdict still triggers repair #1720. Until then the validation tracing plugin, which skips results with no reason (mellea/telemetry/tracing_plugins.py:601-602), stops showing the exception text for these results.
• feat(rag): replace groundedness judgment prompts with the hallucination_detection adapter #1669's adapter swap would delete these parsers. They're worth fixing while it's on hold.

Line references are against 46ad3b3a5.