Repository navigation
Evaluation: Improve summary identifiers #1174
Description
Is your feature request related to a problem?
The evaluation summary currently uses internal Langfuse identifiers, which reviewers cannot map back to specific questions without accessing Langfuse. Additionally, questions with duplication appear under multiple ids, leading to redundant entries in the summary.
Describe the solution you'd like
- Provide the summary model with
question_idinstead oftrace_idand update the prompt to reference questions as "Question N". - Upgrade
EVAL_JUDGE_MODELto a stronger model. - Modify the test in
app/tests/crud/evaluations/test_run_ai_summary.pyto accommodate the new payload key.
Original issue
Describe the current behavior
The evaluation AI summary (app/crud/evaluations/summary.py) keys every per-question judge trace on the Langfuse dataset item id (trace_id, formatted item_{row}_{dup}). The system prompt instructs the summary model to reference each question by that id, so the generated "Top 3 to check" list comes back as item_3_1, item_12_4, etc.
Two problems with that:
- Those ids are internal Langfuse identifiers and mean nothing to a reviewer reading the summary — you cannot map
item_12_4back to a question without opening Langfuse. - With a duplication factor > 1, the same question appears under several distinct ids (
item_3_1…item_3_5), so the model can list what is really one problematic question as several separate items and burn the three slots on duplicates.
Separately, EVAL_JUDGE_MODEL in app/core/config.py is pinned to gpt-5-mini, which is weaker than what the LLM-as-judge scoring warrants.
Describe the enhancement you'd like
- Hand the summary model
question_id(the 1-based dataset row number already assigned at Langfuse push time and merged inapp/crud/evaluations/merge.py) instead oftrace_id, and update the system prompt to reference questions as "Question N". - Bump
EVAL_JUDGE_MODELto a stronger judge model. - Update
app/tests/crud/evaluations/test_run_ai_summary.pyfor the new payload key.
Why is this enhancement needed?
Usability: the run summary is meant to be a go-verify pointer list for a human reviewer. "Question 3" is directly actionable against the golden dataset; item_3_1 is not. Collapsing duplicate runs of the same question onto one id also stops the "Top 3" from being padded with the same question repeated, so the three slots surface three genuinely distinct problems.
Additional context
question_id is already populated end to end (app/crud/evaluations/langfuse.py writes it into trace metadata, merge.py preserves and sorts on it), so this is a consumer-side change only — no new plumbing.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsClosed
Activity
[-]Reference evaluation summary findings by question number instead of Langfuse trace id[/-][+]Evaluation: Improve summary identifiers[/+]