Conversation
StartBatchEvaluation and CreateOnlineEvaluationConfig both take an outputConfig, and batch-evaluation/online-eval already expose it, but the insights commands built on the same APIs had no way to send one — so insight results always landed in the service-managed default log group, which blocks the per-tenant results routing the eval commands support. online-insight update forwards the flag without merging the stored value: GET echoes back the service-managed group under /aws/bedrock-agentcore/evaluations/, which the write APIs reject.
|
Claude Security Review: no high-confidence findings. (run) |
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Looks good
This is a small, well-scoped feature that wires the already-existing --output-config flag modules (BatchOutputConfig, OnlineEvalOutputConfigFlag) into the three insights commands, plus corresponding forwarding in core/eval.tsx. Behavior mirrors the sibling batch-evaluation / online-eval commands:
- Types (
StartBatchInsightsInput,CreateOnlineInsightInput,UpdateOnlineInsightInput) get an optionaloutputConfig, and the service commands are updated accordingly. SourceResolveris threaded through so the single-stdin guard still spans--filters,--data-source-config, and--output-config.- The intentional non-merge on
updateOnlineInsightis documented both in the code comment (src/core/eval.tsx:1164-1168) and the commit message, and matchesupdateOnlineEvaluationConfig. - Tests exercise the real handler +
TestCoreClient(no over-mocking) and cover both the happy path and malformed-JSON rejection for each command.
No telemetry is added, but these handlers don't have any today; this PR isn't the place to introduce it.
Nothing that needs to change before merge.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #2386 +/- ##
=========================================
Coverage 97.25% 97.25%
=========================================
Files 613 613
Lines 41028 41040 +12
=========================================
+ Hits 39902 39914 +12
Misses 1126 1126 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
What
eval batch-insights run,eval online-insight create, andeval online-insight updatenow take--output-config, the same flagbatch-evaluationandonline-evalalready expose.Why
Insights ride the same APIs as their evaluation siblings —
StartBatchEvaluationfor batch insights,CreateOnlineEvaluationConfig/UpdateOnlineEvaluationConfigfor online insights — and both modeloutputConfig. Without the flag, insight results always land in the service-managed default log group, so a customer can't route them to their own log group, log stream, or metrics namespace, and can't useresultDestination: SOURCE_LOG_GROUPat all.No new flag shapes:
BatchOutputConfigandOnlineEvalOutputConfigFlagare reused as-is, so help text, JSON/file:///stdin handling, and the shared-stdin guard behave exactly as on the existing commands.Note on
online-insight updateThe flag is forwarded without merging the stored value, unlike
rule/insights/clusteringConfigon the same command.GetOnlineEvaluationConfigechoes back the service-managed destination under/aws/bedrock-agentcore/evaluations/, and the write APIs reject a log group in that namespace — re-sending it would break every update that omits the flag. Verified live that omitting the flag leaves the stored destination untouched (see below). Matchesonline-eval update.Testing
bun test— 3563 pass, 0 failbun run typecheck,bun run lint:check, prettier — clean--output-configto the job; malformed JSON is rejected onbatch-insights run,online-insight create,online-insight updateLive verification (us-west-2)
online-insight create --output-config(SOURCE_LOG_GROUP + custom metrics namespace)getshowsresultDestination: SOURCE_LOG_GROUPandmetricsNamespacepersistedonline-insight update --sampling-ratewith no--output-configoutputConfigintact — no resetonline-insight update --output-config(new namespace)batch-insights run --output-config(custom log group + namespace, DEDICATED_LOG_GROUP)--output-configwith a log group under/aws/bedrock-agentcore/evaluations/--output-configwithmetricsNamespace: AWS/...Test resources created during verification were deleted.
Findings for the service team (not addressed here)
GetBatchEvaluationomitsoutputConfig.StartBatchEvaluationechoes the destination back, but the GET response returnsnull, so a job's result destination is not readable after start — reproduced against the raw API, so it is not a CLI-side drop.batch-insights get/batch-evaluation gettherefore cannot show where results went.CreateOnlineEvaluationConfigreturnsoutputConfig: {"cloudWatchConfig": {}}on create even whenresultDestinationandmetricsNamespacewere supplied; the values are present on a subsequent GET.Possible follow-up
online-insight createrequires--role-arn, so a custom destination fails with "The provided execution role does not have permissions to create log group: ..." unless the caller has already scoped that role.online-eval createavoids this by provisioning a role and widening it to theoutputConfigdestination when--role-arnis omitted. Extending the same auto-provisioning toonline-insight createwould remove the manual IAM step; deliberately out of scope here.