Skip to content

feat(eval): accept --output-config on insights commands - #2386

Draft
jariy17 wants to merge 1 commit into
refactorfrom
feat/insights-output-config
Draft

jariy17 wants to merge 1 commit into
refactorfrom
feat/insights-output-config

Conversation

@jariy17

@jariy17 jariy17 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

What

eval batch-insights run, eval online-insight create, and eval online-insight update now take --output-config, the same flag batch-evaluation and online-eval already expose.

Why

Insights ride the same APIs as their evaluation siblings — StartBatchEvaluation for batch insights, CreateOnlineEvaluationConfig/UpdateOnlineEvaluationConfig for online insights — and both model outputConfig. Without the flag, insight results always land in the service-managed default log group, so a customer can't route them to their own log group, log stream, or metrics namespace, and can't use resultDestination: SOURCE_LOG_GROUP at all.

No new flag shapes: BatchOutputConfig and OnlineEvalOutputConfigFlag are reused as-is, so help text, JSON/file:///stdin handling, and the shared-stdin guard behave exactly as on the existing commands.

Note on online-insight update

The flag is forwarded without merging the stored value, unlike rule/insights/clusteringConfig on the same command. GetOnlineEvaluationConfig echoes back the service-managed destination under /aws/bedrock-agentcore/evaluations/, and the write APIs reject a log group in that namespace — re-sending it would break every update that omits the flag. Verified live that omitting the flag leaves the stored destination untouched (see below). Matches online-eval update.

Testing

  • bun test — 3563 pass, 0 fail
  • bun run typecheck, bun run lint:check, prettier — clean
  • New unit tests: batch insights forwards a full --output-config to the job; malformed JSON is rejected on batch-insights run, online-insight create, online-insight update

Live verification (us-west-2)

Check Result
online-insight create --output-config (SOURCE_LOG_GROUP + custom metrics namespace) created; get shows resultDestination: SOURCE_LOG_GROUP and metricsNamespace persisted
online-insight update --sampling-rate with no --output-config sampling changed, stored outputConfig intact — no reset
online-insight update --output-config (new namespace) destination replaced, sampling preserved
batch-insights run --output-config (custom log group + namespace, DEDICATED_LOG_GROUP) accepted; service created the destination log group and assigned a log stream
--output-config with a log group under /aws/bedrock-agentcore/evaluations/ rejected with a clear message
--output-config with metricsNamespace: AWS/... rejected with a clear message

Test resources created during verification were deleted.

Findings for the service team (not addressed here)

  1. GetBatchEvaluation omits outputConfig. StartBatchEvaluation echoes the destination back, but the GET response returns null, so a job's result destination is not readable after start — reproduced against the raw API, so it is not a CLI-side drop. batch-insights get / batch-evaluation get therefore cannot show where results went.
  2. CreateOnlineEvaluationConfig returns outputConfig: {"cloudWatchConfig": {}} on create even when resultDestination and metricsNamespace were supplied; the values are present on a subsequent GET.

Possible follow-up

online-insight create requires --role-arn, so a custom destination fails with "The provided execution role does not have permissions to create log group: ..." unless the caller has already scoped that role. online-eval create avoids this by provisioning a role and widening it to the outputConfig destination when --role-arn is omitted. Extending the same auto-provisioning to online-insight create would remove the manual IAM step; deliberately out of scope here.

StartBatchEvaluation and CreateOnlineEvaluationConfig both take an
outputConfig, and batch-evaluation/online-eval already expose it, but the
insights commands built on the same APIs had no way to send one — so
insight results always landed in the service-managed default log group,
which blocks the per-tenant results routing the eval commands support.

online-insight update forwards the flag without merging the stored value:
GET echoes back the service-managed group under
/aws/bedrock-agentcore/evaluations/, which the write APIs reject.
@github-actions github-actions Bot added the size/s PR size: S label Sep 22, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 22, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 22, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Looks good

This is a small, well-scoped feature that wires the already-existing --output-config flag modules (BatchOutputConfig, OnlineEvalOutputConfigFlag) into the three insights commands, plus corresponding forwarding in core/eval.tsx. Behavior mirrors the sibling batch-evaluation / online-eval commands:

  • Types (StartBatchInsightsInput, CreateOnlineInsightInput, UpdateOnlineInsightInput) get an optional outputConfig, and the service commands are updated accordingly.
  • SourceResolver is threaded through so the single-stdin guard still spans --filters, --data-source-config, and --output-config.
  • The intentional non-merge on updateOnlineInsight is documented both in the code comment (src/core/eval.tsx:1164-1168) and the commit message, and matches updateOnlineEvaluationConfig.
  • Tests exercise the real handler + TestCoreClient (no over-mocking) and cover both the happy path and malformed-JSON rejection for each command.

No telemetry is added, but these handlers don't have any today; this PR isn't the place to introduce it.

Nothing that needs to change before merge.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 22, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.25%. Comparing base (824d5a8) to head (6e0e06b).
⚠️ Report is 3 commits behind head on refactor.

Additional details and impacted files
@@            Coverage Diff            @@
##           refactor    #2386   +/-   ##
=========================================
  Coverage     97.25%   97.25%           
=========================================
  Files           613      613           
  Lines         41028    41040   +12     
=========================================
+ Hits          39902    39914   +12     
  Misses         1126     1126           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Sep 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants