You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Consumers reading evaluation spans currently need private SDK helpers or copies of their implementation. This exposes is_tool_span, tool_span_ids, and trace_ids through bedrock_agentcore.evaluation.spans and the evaluation package, and routes both EvaluationClient and OnDemandEvaluationDatasetRunner through the shared implementation.
EvaluationClient.get_evaluator_level() also becomes public, reusing its existing control plane client and cache. Its existing SESSION fallback and caching behavior are preserved. The on-demand runner keeps its existing error propagation and locking behavior.
Related to #663 (span helpers and evaluator-level lookup). Includes usage examples and tests for all three span conventions, malformed attributes, missing IDs, ordering, duplicates, trace filtering, and evaluator caching.
Validation:
Full suite: 3,562 passed, 10 skipped, 4 xpassed; 91.09% overall coverage and 100% for the new span helpers.
All pre-commit checks and the Bandit security scan passed.
Built wheel verified in a clean Python 3.12 environment with base dependencies only; span imports and pure helper calls require no AWS credentials.
Bug bash: 1,000 generated span batches matched the original helpers across five trace filters, with input non-mutation checks.
jariy17
changed the title
fix(evaluation): expose public span helpers and evaluator level lookup
fix(evaluation): expose shared span helpers and evaluator level lookup
Sep 22, 2026
CI note: both red checks here are pre-existing and unrelated to this change. I'll address them in a follow-up PR.
Test Python 3.11 / 3.12 / 3.13 — payments/integrations/langgraph/test_stage3.py::TestRetryDelay::test_zero_delay_skips_sleep fails with Expected 'sleep' to not have been called. Called 898 times. Calls: [call(1), call(1), ...].
Root cause is a cross-test interaction that exists on main:
ragas constructs its AnalyticsBatcher at import time, which starts a daemon thread looping on time.sleep(1). It is not gated by RAGAS_DO_NOT_TRACK, and it outlives the test that imported ragas.
Our tests patch sleep via a module's own time reference (e.g. patch("...middleware.time.sleep")). That attribute lives on the shared stdlib time module, so the patch is process-global.
When that daemon thread wakes inside a patched window, the mocked sleep returns instantly, its loop spins, and the mock accumulates hundreds of phantom sleep(1) calls — breaking call-count assertions in whichever test holds the patch.
That makes it timing-dependent, which matches 3.10 passing while 3.11–3.13 failed, and the same test passing on rerun in #675. Neither the failing test nor the patch pattern is touched by this PR.
Compat (evaluation) — the compat job installs pytest plugins only, so v1.23.1's tests_integ/evaluation/test_third_party_adapters.py errors at setup with ModuleNotFoundError: No module named 'ragas' / 'deepeval' (16 collection errors). The same failure occurs on unrelated branches, and the job is continue-on-error: true, so the workflow run itself reports success.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consumers reading evaluation spans currently need private SDK helpers or copies of their implementation. This exposes
is_tool_span,tool_span_ids, andtrace_idsthroughbedrock_agentcore.evaluation.spansand the evaluation package, and routes bothEvaluationClientandOnDemandEvaluationDatasetRunnerthrough the shared implementation.EvaluationClient.get_evaluator_level()also becomes public, reusing its existing control plane client and cache. Its existingSESSIONfallback and caching behavior are preserved. The on-demand runner keeps its existing error propagation and locking behavior.Related to #663 (span helpers and evaluator-level lookup). Includes usage examples and tests for all three span conventions, malformed attributes, missing IDs, ordering, duplicates, trace filtering, and evaluator caching.
Validation: