Skip to content

fix(evaluation): expose shared span helpers and evaluator level lookup - #674

Merged
jariy17 merged 1 commit into
mainfrom
fix/public-evaluation-span-helpers
Sep 23, 2026
Merged

jariy17 merged 1 commit into
mainfrom
fix/public-evaluation-span-helpers

Conversation

@jariy17

@jariy17 jariy17 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Consumers reading evaluation spans currently need private SDK helpers or copies of their implementation. This exposes is_tool_span, tool_span_ids, and trace_ids through bedrock_agentcore.evaluation.spans and the evaluation package, and routes both EvaluationClient and OnDemandEvaluationDatasetRunner through the shared implementation.

EvaluationClient.get_evaluator_level() also becomes public, reusing its existing control plane client and cache. Its existing SESSION fallback and caching behavior are preserved. The on-demand runner keeps its existing error propagation and locking behavior.

Related to #663 (span helpers and evaluator-level lookup). Includes usage examples and tests for all three span conventions, malformed attributes, missing IDs, ordering, duplicates, trace filtering, and evaluator caching.

Validation:

  • Full suite: 3,562 passed, 10 skipped, 4 xpassed; 91.09% overall coverage and 100% for the new span helpers.
  • All pre-commit checks and the Bandit security scan passed.
  • Built wheel verified in a clean Python 3.12 environment with base dependencies only; span imports and pure helper calls require no AWS credentials.
  • Bug bash: 1,000 generated span batches matched the original helpers across five trace filters, with input non-mutation checks.

@jariy17
jariy17 requested a review from a team September 22, 2026 20:16
@github-actions github-actions Bot added the size/m PR size: M label Sep 22, 2026
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@jariy17
jariy17 deployed to auto-approve September 22, 2026 20:16 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

✅ No Breaking Changes Detected

No public API breaking changes found in this PR.

@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 22, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 22, 2026
@jariy17 jariy17 changed the title fix(evaluation): expose public span helpers and evaluator level lookup fix(evaluation): expose shared span helpers and evaluator level lookup Sep 22, 2026
@github-actions github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 22, 2026
@jariy17

jariy17 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

CI note: both red checks here are pre-existing and unrelated to this change. I'll address them in a follow-up PR.

Test Python 3.11 / 3.12 / 3.13payments/integrations/langgraph/test_stage3.py::TestRetryDelay::test_zero_delay_skips_sleep fails with Expected 'sleep' to not have been called. Called 898 times. Calls: [call(1), call(1), ...].

Root cause is a cross-test interaction that exists on main:

  1. ragas constructs its AnalyticsBatcher at import time, which starts a daemon thread looping on time.sleep(1). It is not gated by RAGAS_DO_NOT_TRACK, and it outlives the test that imported ragas.
  2. Our tests patch sleep via a module's own time reference (e.g. patch("...middleware.time.sleep")). That attribute lives on the shared stdlib time module, so the patch is process-global.
  3. When that daemon thread wakes inside a patched window, the mocked sleep returns instantly, its loop spins, and the mock accumulates hundreds of phantom sleep(1) calls — breaking call-count assertions in whichever test holds the patch.

That makes it timing-dependent, which matches 3.10 passing while 3.11–3.13 failed, and the same test passing on rerun in #675. Neither the failing test nor the patch pattern is touched by this PR.

Compat (evaluation) — the compat job installs pytest plugins only, so v1.23.1's tests_integ/evaluation/test_third_party_adapters.py errors at setup with ModuleNotFoundError: No module named 'ragas' / 'deepeval' (16 collection errors). The same failure occurs on unrelated branches, and the job is continue-on-error: true, so the workflow run itself reports success.

@jariy17
jariy17 merged commit 5554fb8 into main Sep 23, 2026
38 of 43 checks passed

This branch was successfully deployed

1 active deployment
auto-approve 90892ae5 Deployed Sep 22, 2026 by jariy17 via Test (payments) #1592
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants