Repository navigation
Evaluation: Streamline AI summary model #1130
Description
Is your feature request related to a problem?
Currently, the AI run-level summary for judge evaluations relies on OpenAI, while prompt improvements target Anthropic. This dual-provider approach complicates credential management and error handling.
Describe the solution you'd like
- Change the default
EVAL_SUMMARY_MODELfromgpt-5-minitoclaude-sonnet-4-6. - Modify
generate_run_ai_summaryto create an Anthropic client using theANTHROPIC_API_KEY, removingsession/openai_clientparameters; update the caller incrud/evaluations/fast.py. - Implement a single
messages.createcall with structured JSON output, superseding the current free-text Responses parsing and removing the mapper's DB lookup. - Maintain the existing contract: if no API key, provider error, bad JSON, or empty note occurs, set
ai_summarytoNonewhile still allowing the run to succeed. - Move the shared 6-branch Anthropic error handling into
log_anthropic_error(exc, *, fn_name, context)inservices/llm/providers/claude.pyand use it in bothsummary.pyandprompt_improvement.py. - Revise tests (
test_run_ai_summary.py,test_fast_judge.py) and updatedocs/wiki/modules/evaluations.md.
Comments: No changes to the API surface. Deployments using EVAL_SUMMARY_MODEL with an OpenAI model must transition to a Claude model, and ANTHROPIC_API_KEY needs to be set; otherwise, summaries will remain empty.
Original issue
Is your feature request related to a problem?
The best-effort run-level AI summary on judge evaluations was generated with OpenAI (gpt-5-mini, Responses API), while prompt improvement already ran on Anthropic. Two providers on the same feature area meant two credential paths and a duplicated Anthropic error-handling ladder.
Describe the solution you'd like
- Switch
EVAL_SUMMARY_MODELdefault fromgpt-5-minitoclaude-sonnet-4-6. generate_run_ai_summarybuilds its own Anthropic client from the platform-ownedANTHROPIC_API_KEY— drop thesession/openai_clientparams; update the caller incrud/evaluations/fast.py.- Use one
messages.createcall with structured JSON output (output_configjson_schema,{"summary": ""}) instead of free-text Responses parsing viamap_kaapi_to_openai_params+extract_response_text, removing the mapper's DB lookup from this path. - Keep the best-effort contract: no API key, provider error, bad JSON, or empty note →
ai_summaryisNoneand the run still succeeds. - Extract the shared 6-branch Anthropic error ladder into
log_anthropic_error(exc, *, fn_name, context)inservices/llm/providers/claude.py; use it from bothsummary.pyandprompt_improvement.py. - Update tests (
test_run_ai_summary.py,test_fast_judge.py) anddocs/wiki/modules/evaluations.md.
Comments: No API surface change. Deployments pinning EVAL_SUMMARY_MODEL to an OpenAI model must switch to a Claude model, and ANTHROPIC_API_KEY must be set or summaries silently stay empty.
Tracked and completed in #1128.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsIn Progress
Activity
[-]Evaluation: Move AI run summary to Anthropic[/-][+]Evaluation: Streamline AI summary model[/+]