Add integration test suite: fixtures, criteria tests, evaluator tests…

[varishasiddiqui]

Add integration test suite, fixtures, and performance benchmark

What does this PR do?

Adds the fixture dataset, pytest test suite, and performance benchmark for the refusal_check and factual_grounding criteria (Integration/Evaluation Engineer deliverables, DESIGN.md Section 5).

Which module(s) does it touch?

• tests/fixtures/criteria_fixtures.json — new. Hand-verified fixture cases (6 for refusal_check, 6 for factual_grounding) with known expected outcomes, per Section 5.6.
• tests/fixtures/verify_grounding.py — new. Standalone script to manually spot-check factual_grounding fixtures against the real embedding model, outside of pytest.
• tests/test_refusal_check.py — new. Parametrized pytest suite over the refusal_check fixtures, plus contract checks (score never None, required keys present).
• tests/test_factual_grounding.py — new. Parametrized pytest suite over the factual_grounding fixtures (score ranges + numeric-mismatch capping), plus a singleton-model regression test.
• tests/test_evaluator.py — new. Tests for Evaluator.evaluate() itself: default vs. requested criteria, fail-fast on unknown criterion names, overall_score averaging, and the score:None-excluded-not-zero rule (Section 3.3).
• tests/test_registry.py — new. Tests for the @register_criterion decorator: registration, duplicate-name rejection, available_criteria(), direct callability.
• tests/test_cli.py — new. Tests for cli.py using Typer's CliRunner: basic evaluation, --context, missing required --prompt, unknown-criterion error handling, default-to-all-criteria behavior.
• tests/test_performance.py — new. Benchmark per Section 5.4: full evaluate() call under the 2-second target (two independent timing methods), plus a singleton-not-reloaded regression test at the Evaluator level.

No existing source files (evaluator.py, registry.py, cli.py, criteria files) were modified — this PR is tests/fixtures only.

How did you test it?

Ran the full suite with coverage:

pytest tests/ -v --cov=llm_eval_kit --cov-report=term-missing

Result: 57 passed, 96% coverage on llm_eval_kit/ (target: 90%+ per Section 8.3).

Also ran the performance benchmark on its own:

pytest tests/test_performance.py -v

Result: 4 passed in 21.82s — full evaluate() call comfortably under the 2-second target, and the embedding-model singleton confirmed not to reload across repeated calls.

Coverage detail: how we got from 89% to 96%

The first full run (refusal_check + factual_grounding + evaluator + registry + numeric_parser tests, before test_cli.py existed) gave:

File	Coverage
evaluator.py	100%
registry.py	100%
factual_grounding.py	97%
numeric_utils.py	96%
refusal.py	94%
cli.py	0%
Overall	89%

cli.py was completely untested, which is what kept the overall number under the 90% target. Adding tests/test_cli.py (using Typer's CliRunner to exercise --prompt, --criteria, missing-argument handling, and error paths) brought cli.py from 0% to 95% and the overall project to 96%.

Edge cases handled:

• Empty response and whitespace-only response → treated as a refusal (refusal_check fixtures)
• False-positive refusal phrase mid-response ("I cannot stress enough...") followed by a substantive answer → correctly NOT flagged as a refusal
• Refusal phrase after a soft greeting lead-in → still detected within the opening window
• Missing/empty context passed to factual_grounding → returns score: None, excluded from overall_score rather than counted as 0
• Numeric value swapped in an otherwise near-identical sentence (e.g. "50°C" vs. context's "100°C") → score correctly capped despite high semantic similarity
• Same number, mismatched unit (e.g. "10 feet" vs. context's "10 meters") → correctly flagged as a mismatch
• Evaluator given an unknown criterion name → fails fast with ValueError before running any checker, even if other requested criteria are valid
• A dummy criterion returning score: None → correctly excluded from overall_score, verified not to silently count as 0
• CLI given an unknown criterion → clean exit code 1 with an error message, not an unhandled traceback
• CLI missing the required --prompt flag → fails cleanly before reaching Evaluator

Known blockers (not in scope for this PR)

• relevance and completeness criteria are not yet implemented by the Research Engineer — fixtures/tests for these will be added once available.
• No pyproject.toml/setup.py yet, so pip install . and the llm-eval console-script entry point can't be verified end-to-end (DESIGN.md Section 5.5). CLI was verified via python -m llm_eval_kit.cli instead.

Checklist:

• Tests added or updated
• Docstrings on all new public functions
• No unused imports
• README updated if public API changed — N/A, no public API was changed by this PR
• Lead Engineer has reviewed and approved before submitting to director

[varishasiddiqui]

@MahrukhBaig @Maazkorejo

[Khubaib8281]

Test coverage is good. lets get this merged