Skip to content

feat(eval): add Ragas-style context precision and recall metrics - #341

Merged
Kaap10 merged 1 commit into
codeforstartups:developmentfrom
Isha-Zaka:feat/issue-303-ragas-metrics
Oct 13, 2026
Merged

Kaap10 merged 1 commit into
codeforstartups:developmentfrom
Isha-Zaka:feat/issue-303-ragas-metrics

Conversation

@Isha-Zaka

Copy link
Copy Markdown
Collaborator

Description

Adds Ragas-style context precision and context recall metrics to the RAG evaluation framework, with model-layer judge support and batch evaluation for labeled examples.

Related issue

Fixes #303

Changes

  • Added Ragas-style context precision with rank-sensitive relevance scoring and context recall based on reference-answer claim coverage.
  • Added ChatModelJudge to support evaluation through Dynavec's existing ChatModel abstraction.
  • Extended evaluate_rag() and EvalRunner to support all four RAG evaluation metrics and batch evaluation with optional reference answers, with fake-judge test coverage.

Testing

Validation: 36 passed in tests/test_eval.py; full make run-ci passed; mypy passed on 81 source files; 1 snapshot passed; git diff --check passed.

  • Tests pass locally
  • Ruff checks pass
  • Documentation updated, if applicable

Checklist

  • My changes are focused and relevant to this pull request.
  • I have added or updated tests where appropriate.
  • I have reviewed my changes for unrelated modifications.
  • I have updated documentation where necessary.

@Kaap10 Kaap10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The context precision and recall metrics adhere to standard Ragas formulations, integrate cleanly with ChatModelJudge, and maintain backwards compatibility for existing batch evaluation workflows. Verified locally with all tests and linters passing.

@Kaap10
Kaap10 merged commit a57bbe0 into codeforstartups:development Oct 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ragas-style metrics: faithfulness, answer relevance, context precision/recall

2 participants