Repository navigation
Collect user feedback and export reviewed RAGAS regression cases - #6
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Users can now mark a completed response Helpful or Not helpful and describe what went wrong. The server accepts feedback only for runs in the current session, then saves the answer, sources, private diagnostics and pipeline versions before syncing a stable feedback record to LangSmith.
A review command requires a checked reference answer and review notes before exporting feedback cases to a separate RAGAS regression cohort. Votes remain available as user ratings. Feedback records persist after session end and cloud export failures preserve the local copy. Pending feedback is excluded from evaluation.
The evaluation runner also supports separate judge canaries, stops further HTTP calls after a quota reset exceeds the metric deadline, and paces shorter quota waits explicitly. The judge output limit is 2,048 tokens. Reports handle reviewed feedback without inventing a gold article label.
Validation passed with 157 local tests and 3 subtests, plus 13 separate RAGAS-environment checks. Browser checks used a temporary service without model calls and verified both ratings, comments, durable storage, session reset and responsive layouts at 1440, 768 and 390 pixels. Typecheck, lint and production build passed.
A new 30-question training run is in progress with a 90-attempt generation cap. GPT-OSS-120B generates answers. GPT-OSS-20B is the budgeted RAGAS judge, checked on a supported/unsupported faithfulness pair with scores 1.0/0.0. Fresh quality scores will be reported separately with their denominators.