BatchEvaluationRunner does not expose the batch insights workflow.
Requested behavior
Support requesting batch insights through BatchEvaluationRunner, with job polling and access to the resulting insights. The original report identifies these StartBatchEvaluation insights:
Builtin.Insight.FailureAnalysisBuiltin.Insight.UserIntentBuiltin.Insight.ExecutionSummary
The workflow should enforce the API constraint that insights and evaluators cannot be requested in the same job.
Why this matters
The reporter uses insights to cluster failures and identify root causes that can be addressed through prompt changes. Without SDK support, the evaluation improvement workflow requires custom orchestration outside BatchEvaluationRunner.
Open design question
Confirm whether the runner's dataset-based workflow is compatible with insights and which inputs are supported. Use that finding to determine how to expose insights through the runner, or whether a separate insights runner is needed. This compatibility question remains unresolved.
Original report environment: bedrock-agentcore 1.23.0, Python 3.13, us-west-2.
BatchEvaluationRunnerdoes not expose the batch insights workflow.Requested behavior
Support requesting batch insights through
BatchEvaluationRunner, with job polling and access to the resulting insights. The original report identifies theseStartBatchEvaluationinsights:Builtin.Insight.FailureAnalysisBuiltin.Insight.UserIntentBuiltin.Insight.ExecutionSummaryThe workflow should enforce the API constraint that insights and evaluators cannot be requested in the same job.
Why this matters
The reporter uses insights to cluster failures and identify root causes that can be addressed through prompt changes. Without SDK support, the evaluation improvement workflow requires custom orchestration outside
BatchEvaluationRunner.Open design question
Confirm whether the runner's dataset-based workflow is compatible with insights and which inputs are supported. Use that finding to determine how to expose insights through the runner, or whether a separate insights runner is needed. This compatibility question remains unresolved.
Original report environment:
bedrock-agentcore1.23.0, Python 3.13,us-west-2.