Skip to content

Evaluation: Streamline AI summary model #1130

Closed

Description

@AkhileshNegi

Is your feature request related to a problem?
Currently, the AI run-level summary for judge evaluations relies on OpenAI, while prompt improvements target Anthropic. This dual-provider approach complicates credential management and error handling.

Describe the solution you'd like

  • Change the default EVAL_SUMMARY_MODEL from gpt-5-mini to claude-sonnet-4-6.
  • Modify generate_run_ai_summary to create an Anthropic client using the ANTHROPIC_API_KEY, removing session / openai_client parameters; update the caller in crud/evaluations/fast.py.
  • Implement a single messages.create call with structured JSON output, superseding the current free-text Responses parsing and removing the mapper's DB lookup.
  • Maintain the existing contract: if no API key, provider error, bad JSON, or empty note occurs, set ai_summary to None while still allowing the run to succeed.
  • Move the shared 6-branch Anthropic error handling into log_anthropic_error(exc, *, fn_name, context) in services/llm/providers/claude.py and use it in both summary.py and prompt_improvement.py.
  • Revise tests (test_run_ai_summary.py, test_fast_judge.py) and update docs/wiki/modules/evaluations.md.

Comments: No changes to the API surface. Deployments using EVAL_SUMMARY_MODEL with an OpenAI model must transition to a Claude model, and ANTHROPIC_API_KEY needs to be set; otherwise, summaries will remain empty.

Original issue

Is your feature request related to a problem?
The best-effort run-level AI summary on judge evaluations was generated with OpenAI (gpt-5-mini, Responses API), while prompt improvement already ran on Anthropic. Two providers on the same feature area meant two credential paths and a duplicated Anthropic error-handling ladder.

Describe the solution you'd like

  • Switch EVAL_SUMMARY_MODEL default from gpt-5-mini to claude-sonnet-4-6.
  • generate_run_ai_summary builds its own Anthropic client from the platform-owned ANTHROPIC_API_KEY — drop the session / openai_client params; update the caller in crud/evaluations/fast.py.
  • Use one messages.create call with structured JSON output (output_config json_schema, {"summary": ""}) instead of free-text Responses parsing via map_kaapi_to_openai_params + extract_response_text, removing the mapper's DB lookup from this path.
  • Keep the best-effort contract: no API key, provider error, bad JSON, or empty note → ai_summary is None and the run still succeeds.
  • Extract the shared 6-branch Anthropic error ladder into log_anthropic_error(exc, *, fn_name, context) in services/llm/providers/claude.py; use it from both summary.py and prompt_improvement.py.
  • Update tests (test_run_ai_summary.py, test_fast_judge.py) and docs/wiki/modules/evaluations.md.

Comments: No API surface change. Deployments pinning EVAL_SUMMARY_MODEL to an OpenAI model must switch to a Claude model, and ANTHROPIC_API_KEY must be set or summaries silently stay empty.

Tracked and completed in #1128.

Activity

  1. moved this from To Do to Closed in Kaapi-devon Aug 14, 2026
  2. changed the title [-]Evaluation: Move AI run summary to Anthropic[/-] [+]Evaluation: Streamline AI summary model[/+] on Aug 14, 2026
  3. self-assigned this
    on Aug 14, 2026
  4. moved this from Closed to In Progress in Kaapi-devon Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions