[Week 5-6] Complete 4-criteria suite, edge-case hardening, Urdu support, docstrings audit, and fellowship lab note

lab_note.md

@@ -0,0 +1,17 @@
+# Building Deterministic Offline Evaluation Criteria for llm-eval-kit
+
+Evaluating Large Language Model (LLM) responses in production typically relies on human annotators or expensive, non-deterministic "LLM-as-a-judge" API calls. This creates latency bottlenecks, non-reproducible scores, and recurring operational costs that make continuous evaluation in local CI/CD pipelines impractical. llm-eval-kit set out to close this gap by building a deterministic, lightweight, and fully offline response evaluation library that scores model outputs locally using fast semantic embeddings, rule-based heuristics, and numeric consistency verification.
+
+**What changed.** The evaluation suite evolved from basic prototype checks into a layered, four-criteria architecture: directional refusal detection with opening-window constraints and substantive-tail heuristics, factual grounding combining dense sentence embeddings with orthogonal numeric-claim verification, prompt-to-response topical relevance scoring, and rule-based aspect decomposition for completeness. Input defense mechanisms were integrated across all four criteria to handle edge cases—such as empty, whitespace, and null responses—without incurring unnecessary transformer compute or producing tokenization artifacts.
+
+**The numeric verification and grounding model.** To detect factual contradictions and hallucinations without external world-knowledge graphs, I developed a hybrid verification layer on top of dense sentence embeddings. Standard semantic similarity treats syntactically identical sentences with swapped numbers (e.g., "water boils at 50°C" versus "water boils at 100°C") as near-identical (~0.90 similarity), failing to catch blatant factual errors. To solve this, I designed an extraction and validation engine in `numeric_utils.py` that isolates numeric claims (values, units, scientific notation, negative numbers, and spelled-out scales up to billions) and verifies them against the union of all context sentences using a subset check. If an unsupported figure appears, the similarity score is capped at 0.30. This multi-sentence union approach allows responses to synthesize facts across multiple sentences while still penalizing ungrounded numbers. This check is an offline consistency heuristic rather than an absolute fact-checker, as it targets numeric and semantic alignment rather than unstated world knowledge.
+
+**Engineering outcomes.** Automated test coverage expanded to 51 unit tests across three dedicated test suites (`test_numeric_parser.py`, `test_edge_cases.py`, and `test_criteria_all.py`), achieving 100% test pass rates across all four criteria. Performance benchmarks on 5,000-word documents demonstrated that sentence segmentation runs in under 0.5 milliseconds using linear lookbehind regexes with zero ReDoS risk, while batched embedding inference on CPU processes 250 sentences in ~350 milliseconds. Multilingual adaptations added 13 curated Urdu refusal phrases, native Urdu punctuation parsing (`۔`, `؟`), and Eastern Arabic numeral translation (`۰–۹` to `0–9`).
+
+**What I learned.** Working through architectural iterations reinforced concrete software engineering habits: implementing lazy-loaded model singletons so importing the library incurs zero initialization penalty, externalizing phrase lists and thresholds into JSON configs so non-code changes avoid code reviews, and defensively handling boundary inputs. Discovering that empty strings produce non-zero cosine similarities when mapped to BERT's `[CLS]` and `[SEP]` tokens was a reminder that deep learning primitives must always be wrapped with defensive pre-encoding guards.
+
+Building an offline, deterministic evaluation suite that combines dense representations with fast symbolic sanity checks—while being explicit about where heuristic approximations end and absolute verification begins—was the central engineering contribution of this work.
+
+Muhammad Maaz  
+Research/Implementation Engineer  
+INFERENCE Lab Engineering Fellowship, Cohort 01

llm_eval_kit/__init__.py

@@ -1 +1,6 @@
-from llm_eval_kit.criteria import refusal
+from llm_eval_kit.criteria import (
+    refusal,
+    factual_grounding,
+    relevance,
+    completeness,
+)

llm_eval_kit/criteria/completeness.py

@@ -0,0 +1,156 @@
+"""
+llm_eval_kit.criteria.completeness
+==================================
+Evaluates the completeness of an LLM response by decomposing the prompt into
+sub-questions or aspects and checking semantic coverage of each aspect.
+"""
+
+import json
+from pathlib import Path
+import re
+from sentence_transformers import SentenceTransformer, util
+
+from llm_eval_kit.registry import register_criterion
+
+_CONFIG_PATH = Path(__file__).parent / "completeness_config.json"
+
+with open(_CONFIG_PATH, "r", encoding="utf-8") as f:
+    _CONFIG = json.load(f)
+
+MODEL_NAME: str = _CONFIG["model_name"]
+COVERAGE_THRESHOLD: float = _CONFIG["coverage_threshold"]
+
+_model: SentenceTransformer | None = None
+
+
+def _get_model() -> SentenceTransformer:
+    """
+    Lazily initializes and returns the cached SentenceTransformer singleton model.
+
+    Returns:
+        SentenceTransformer: Cached embedding model instance loaded from MODEL_NAME.
+
+    Known Limitations:
+        - Loads the model onto CPU/GPU on first invocation.
+        - Memory consumption corresponds to the underlying transformer model footprint (~80-120MB RAM).
+    """
+    global _model
+    if _model is None:
+        _model = SentenceTransformer(MODEL_NAME)
+    return _model
+
+
+_ASPECT_SPLIT_REGEX = re.compile(
+    r"\?+|\n+|(?:\band\b)|(?:\bas well as\b)|(?:\bplus\b)|(?:\balso\b)|;|(?:\d+[\.\)]\s+)",
+    re.IGNORECASE,
+)
+_RESPONSE_SPLIT_REGEX = re.compile(r"(?<=[.!?۔؟])\s+|\n+")
+
+
+def _extract_aspects(prompt: str) -> list[str]:
+    """
+    Decomposes a user prompt into key thematic aspects using rule-based conjunction heuristics.
+
+    Args:
+        prompt (str): Raw prompt string to decompose.
+
+    Returns:
+        list[str]: Cleaned list of aspect strings. If decomposition yields no sub-parts,
+            returns a single-item list with the normalized prompt.
+
+    Known Limitations:
+        - Uses rule-based heuristics (splitting on 'and', 'as well as', '?', newlines, semicolons, and list numbers).
+        - Implicit multi-part requirements not separated by conjunctions may remain un-split.
+    """
+    parts = _ASPECT_SPLIT_REGEX.split(prompt.strip())
+    aspects = []
+    for part in parts:
+        cleaned = part.strip(" ,.-:;?!'\"")
+        if len(cleaned.split()) >= 2 or len(cleaned) >= 5:
+            aspects.append(cleaned)
+
+    return aspects if aspects else [prompt.strip()]
+
+
+def _split_response_segments(text: str) -> list[str]:
+    """
+    Segments a response string into sentence and paragraph chunks for aspect matching.
+
+    Args:
+        text (str): Response text to segment.
+
+    Returns:
+        list[str]: List of non-empty text chunks.
+    """
+    parts = _RESPONSE_SPLIT_REGEX.split(text.strip())
+    chunks = [p.strip() for p in parts if p.strip()]
+    return chunks if chunks else [text.strip()]
+
+
+@register_criterion("completeness")
+def completeness(prompt: str, response: str, context: str = "", **kwargs) -> dict:
+    """
+    Evaluates what fraction of the requested aspects in the prompt are addressed in the response.
+
+    Args:
+        prompt (str): The original prompt containing information requests.
+        response (str): The response generated by the model to evaluate. If empty or whitespace-only,
+            evaluates to score 0.0.
+        context (str, optional): Additional reference context. Unused by completeness. Defaults to "".
+        **kwargs: Arbitrary keyword arguments for registry compatibility.
+
+    Returns:
+        dict: Evaluation result dictionary containing:
+            - score (float): Fraction of prompt aspects covered in the response [0.0, 1.0].
+            - explanation (str): Human-readable summary of covered vs total aspects.
+            - covered_aspects (list[str]): List of prompt aspect strings that met the coverage threshold.
+            - total_aspects (int): Total number of extracted prompt aspects evaluated.
+
+    Known Limitations:
+        - Aspect extraction is heuristic-based and may not capture complex implicit constraints.
+        - Semantic coverage is thresholded (default 0.60); partial or shallow mentions may count as covered.
+        - Does not verify factual accuracy of the information provided for each aspect.
+    """
+    if prompt is None or not prompt.strip():
+        return {
+            "score": 0.0,
+            "explanation": "Empty prompt provided; completeness cannot be evaluated.",
+            "covered_aspects": [],
+            "total_aspects": 0,
+        }
+
+    aspects = _extract_aspects(prompt)
+
+    if response is None or not response.strip():
+        return {
+            "score": 0.0,
+            "explanation": f"Empty response covers 0 of {len(aspects)} requested aspects.",
+            "covered_aspects": [],
+            "total_aspects": len(aspects),
+        }
+
+    model = _get_model()
+    aspect_embeddings = model.encode(aspects, convert_to_tensor=True)
+    response_segments = _split_response_segments(response)
+    response_segment_embeddings = model.encode(response_segments, convert_to_tensor=True)
+
+    similarity_matrix = util.cos_sim(aspect_embeddings, response_segment_embeddings)
+
+    covered_aspects = []
+    for i, aspect in enumerate(aspects):
+        max_sim = float(similarity_matrix[i].max())
+        if max_sim >= COVERAGE_THRESHOLD:
+            covered_aspects.append(aspect)
+
+    score = len(covered_aspects) / len(aspects) if aspects else 0.0
+
+    explanation = (
+        f"Response covers {len(covered_aspects)} of {len(aspects)} key aspects requested in the prompt."
+    )
+
+    return {
+        "score": round(score, 3),
+        "explanation": explanation,
+        "covered_aspects": covered_aspects,
+        "total_aspects": len(aspects),
+    }

llm_eval_kit/criteria/completeness_config.json

@@ -0,0 +1,4 @@
+{
+  "model_name": "all-MiniLM-L6-v2",
+  "coverage_threshold": 0.60
+}

llm_eval_kit/criteria/factual_grounding.py

@@ -1,32 +1,8 @@
 """
 llm_eval_kit.criteria.factual_grounding
 =======================================
-Checks how well a response is grounded in the given context by comparing
-sentence embeddings, then applies a numeric sanity check on top since embeddings
-alone cannot catch swapped numbers or altered figures.
-
-Background:
-  Pure semantic similarity treats "water boils at 50C" and "water boils at 100C"
-  as almost identical (~0.90 similarity) because it is the same sentence structure
-  with one digit changed. To prevent false positives on hallucinations/contradictions,
-  we extract numeric claims from the response and validate them against the context,
-  capping the score if unsupported figures appear.
-
-Key Architectural Decisions:
-  1. Config Separation: Embedding model name and mismatch score cap are loaded
-     from factual_grounding_config.json.
-  2. Modular Numeric Logic: Extraction and comparison routines live in
-     numeric_utils.py, keeping this file focused on the evaluation workflow.
-  3. Context Multi-Sentence Union: Validates claims against all context sentences
-     (subset check), allowing responses to combine multiple facts.
-  4. Singleton Lazy Loading: SentenceTransformer is loaded on first invocation.
-  5. Score Semantics: Measures semantic similarity with a numeric-consistency
-     penalty applied, not calibrated factual accuracy.
-
-Criterion Interface Contract:
-  prompt: Accepted to satisfy the uniform (prompt, response, context, **kwargs)
-          signature required by CRITERIA_REGISTRY and Evaluator, though factual
-          grounding compares response against context.
+Evaluates factual grounding of an LLM response against reference context by computing
+maximum sentence cosine similarity and enforcing numeric consistency penalties.
 """
 
 import json
@@ -42,55 +18,93 @@
 with open(_CONFIG_PATH, "r", encoding="utf-8") as f:
     _CONFIG = json.load(f)
 
-MODEL_NAME = _CONFIG["model_name"]
-MISMATCH_SCORE_CAP = _CONFIG["mismatch_score_cap"]
+MODEL_NAME: str = _CONFIG["model_name"]
+MISMATCH_SCORE_CAP: float = _CONFIG["mismatch_score_cap"]
 
-_model = None
+_model: SentenceTransformer | None = None
 
 
-def _get_model():
+def _get_model() -> SentenceTransformer:
+    """
+    Lazily initializes and returns the cached SentenceTransformer singleton model.
+
+    Returns:
+        SentenceTransformer: Cached embedding model instance loaded from MODEL_NAME.
+
+    Known Limitations:
+        - Loads the model onto CPU/GPU on first invocation, which incurs initial setup latency.
+        - Memory consumption corresponds to the underlying transformer model footprint (~80-120MB RAM).
+    """
     global _model
     if _model is None:
         _model = SentenceTransformer(MODEL_NAME)
     return _model
 
 
-_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+")
+_SENTENCE_SPLIT = re.compile(r"(?<=[.!?۔؟])\s+|\n+")
 
 
 def _split_sentences(text: str) -> list[str]:
+    """
+    Splits input text into discrete sentence and paragraph segments.
+
+    Args:
+        text (str): Input text block to segment.
+
+    Returns:
+        list[str]: Non-empty stripped sentence and paragraph strings.
+
+    Known Limitations:
+        - Uses linear O(N) regex segmentation matching terminal punctuation (.!?۔؟) or newlines.
+        - Abbreviations followed by whitespace (e.g. 'e.g. ') may cause false splits.
+        - Unpunctuated text without newlines returns a single-element list.
+    """
     parts = _SENTENCE_SPLIT.split(text.strip())
     return [p.strip() for p in parts if p.strip()]
 
 
 @register_criterion("factual_grounding")
 def factual_grounding(prompt: str, response: str, context: str = "", **kwargs) -> dict:
     """
-    Scores how grounded `response` is in `context` using max sentence
-    similarity, then applies a numeric mismatch check on top.
+    Evaluates how well a model response is grounded in reference context using embedding
+    similarity and numeric claim verification.
 
     Args:
-        prompt: Original prompt. Unused directly by this criterion, but
-            accepted to satisfy the shared registry contract.
-        response: The model's response text to evaluate.
-        context: Reference context against which to evaluate grounding.
-        **kwargs: Accepted for uniform registry compatibility.
+        prompt (str): Original prompt. Accepted for uniform registry compatibility.
+        response (str): The model's response string to evaluate. If empty or whitespace-only,
+            evaluates to score 0.0.
+        context (str, optional): Reference context text to evaluate grounding against. If empty
+            or whitespace-only, evaluates to score None (not applicable). Defaults to "".
+        **kwargs: Arbitrary keyword arguments for registry compatibility.
 
     Returns:
-        dict with:
-            score (float | None): Cosine similarity score [0.0, 1.0], capped at
-                MISMATCH_SCORE_CAP if an unsupported numeric claim is detected,
-                or None if context is missing/empty.
-            explanation (str): Human-readable reasoning for the score.
-            best_matching_sentence (str, optional): The context sentence with
-                highest semantic similarity to the response.
+        dict: Evaluation result dictionary containing:
+            - score (float | None): Cosine similarity score in [0.0, 1.0], capped at
+              MISMATCH_SCORE_CAP if unsupported numeric claims are detected; 0.0 if response is empty;
+              or None if context is missing/empty.
+            - explanation (str): Human-readable explanation of score and penalties.
+            - best_matching_sentence (str | None, optional): Context sentence yielding maximum cosine
+              similarity to the response, or None if response/context is empty.
+
+    Known Limitations:
+        - Semantic similarity represents semantic alignment, not calibrated factual verification.
+        - Default model ('all-MiniLM-L6-v2') is English-optimized; non-Latin scripts (e.g. Urdu)
+          may exhibit lower semantic discrimination due to WordPiece tokenization.
+        - Sentence splitting operates in O(N) linear time and is benchmarked for contexts up to 5,000+ words.
     """
     if context is None or not context.strip():
         return {
             "score": None,
             "explanation": "No context provided; factual_grounding is not applicable.",
         }
 
+    if response is None or not response.strip():
+        return {
+            "score": 0.0,
+            "explanation": "Empty response contains no factual content to evaluate.",
+            "best_matching_sentence": None,
+        }
+
     sentences = _split_sentences(context)
     if not sentences:
         return {
@@ -100,7 +114,7 @@ def factual_grounding(prompt: str, response: str, context: str = "", **kwargs) -
 
     model = _get_model()
     response_embedding = model.encode(response, convert_to_tensor=True)
-    sentence_embeddings = model.encode(sentences, convert_to_tensor=True)
+    sentence_embeddings = model.encode(sentences, batch_size=64, convert_to_tensor=True)
 
     similarities = util.cos_sim(response_embedding, sentence_embeddings)[0]
     best_idx = int(similarities.argmax())