Repository navigation
Publish wider RAGAS results and preserve failed evaluation attempts - #7
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The wider training run now has a redacted result and a six-metric RAGAS summary in the README. The diagram includes the shipped feedback workflow. Raw records, questions, references and judge traces stay local or in the private LangSmith project.
The run processed 15 of 30 planned questions before a provider rate limit. It released 8 answers, withheld 2, and recorded 3 citation-format errors, 1 local quota-wait deadline and 1 provider HTTP429. The labeled article was selected for 14/15 questions. Semantic metrics have explicit sample sizes and reference-review flags.
The collector had treated a local request deadline as a batch-wide account limit. It now records that failure and continues. HTTP429 still stops collection, and the previous failed attempt is archived before retry. A regression test covers both cases.
Validation passed with 158 local Python tests and 3 subtests, plus 14 separate RAGAS-environment checks. The final 15-row experiment and review notes were published to LangSmith without new model calls. The remaining questions and three budget-exhausted recall checks are recorded as pending.