feat: copy aws.agent.lineage baggage to GenAI inference spans and support key:scope baggage entries by syed-ahsan-ishtiaque · Pull Request #965 · aws-observability/aws-otel-python-instrumentation

syed-ahsan-ishtiaque · 2026-11-05T01:04:57Z

Why

The X-Ray and AgentCore backend will compute model cost from the token counts on GenAI inference spans and attribute it to agents. It needs the agent lineage on those spans. The lineage travels in the aws.agent.lineage baggage entry. Today the distro can only copy a baggage entry to every span (OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS). Copying lineage to every span would add an attribute that no other span needs. It would also put cost attribution on spans that carry no cost.

Companion JS PR: aws-observability/aws-otel-js-instrumentation#586. Both PRs implement the same behavior.

What changes for users

Key	Copied to	When
`session.id` (agent observability) and `key` entries in `OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS`	every span	at start (unchanged)
`aws.agent.lineage` (agent observability only)	GenAI inference spans	at end
`key:scope` entries in `OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS`	spans that match any of the key's scopes	at end

aws.inference is the only built-in scope. It matches a CLIENT span whose gen_ai.operation.name is chat, text_completion, generate_content or embeddings. When a framework span wraps the model SDK span with the same operation, GenAINestedClientSpanProcessor demotes the outer span to INTERNAL first, so only the innermost span matches.
OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS=session.id,my.key:aws.inference copies session.id to every span and my.key only to inference spans.
Scope names starting with aws. are reserved for built-in scopes, so a future built-in cannot change the meaning of an application's scope.
Entries are split on the first :. W3C baggage keys cannot contain : or ,, so existing values parse as before.
An unknown scope matches no span. A warning is logged once per scope name.
An attribute already set on the span is never overwritten.
If the span is at its attribute count limit, the key is skipped rather than evicting an attribute. The value is truncated to the attribute length limit.

Precedence

`OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS`	Agent observability off	Agent observability on
`k,k:a` (any order)	every span	every span
`k:a,k:b`	spans matching `a` or `b`	same
`aws.agent.lineage` not listed	not copied	`aws.inference` spans
`aws.agent.lineage`	every span	every span
`aws.agent.lineage:a`	spans matching `a`	spans matching `a`
`session.id:a`	spans matching `a`	every span, scope ignored with a warning

An entry without a scope means every span, so it wins over the same key with a scope.
Configuration wins over the aws.agent.lineage default.
session.id is reserved under agent observability, because AgentCore groups spans by session.
Python and JS behave the same in all of these cases.

GenAINestedClientSpanProcessor now runs with or without agent observability. Before, it ran only with agent observability. Without agent observability, the outer span of a nested same-operation inference call is now INTERNAL instead of CLIENT, so the call is not counted twice as a dependency in Application Signals. This also gives aws.inference the same result in both modes. It is listed in the CHANGELOG.

Backward compatibility: apart from the change above, nothing changes without agent observability and without key:scope entries, and the scoped processor is not installed. With agent observability, the only change is the new aws.agent.lineage attribute on inference spans. The session.id and aws.agent.lineage defaults apply only with agent observability.

Undocumented support hook: register_baggage_scope(name, predicate) in amazon.opentelemetry.distro.scoped_baggage_span_processor lets application code define more scopes without a release. Keys are still mapped to the scope in the env var. It is intentionally left out of the public docs.

# OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS=my.key:tools
from amazon.opentelemetry.distro.scoped_baggage_span_processor import register_baggage_scope

register_baggage_scope("tools", lambda span: span.attributes.get("gen_ai.operation.name") == "execute_tool")

It returns False for an empty name, a name with surrounding whitespace, : or ,, a name starting with aws., or a predicate that is not callable. A predicate that raises does not match, and the key's other scopes are still checked.

How it works

ScopedBaggageSpanProcessor (new file):

on_start captures the scoped baggage entries from the parent context. The context active when the span ends may not hold the same baggage (streaming, callbacks).
on_end checks the key's scopes against the ending span and writes the attribute if any matches. Scopes are checked only at end, because instrumentations set gen_ai.operation.name after start and processors can change span kind.
Scope names are looked up when each span ends. A scope registered after the distro starts still applies.

Points worth a close look:

Writing to an ended span. In SDK 1.45.1 (pinned) the span's BoundedAttributes is immutable by on_end, and also during _on_ending. _set_attribute_if_absent writes BoundedAttributes._dict under its lock, the same approach AttributeRedactingSpanProcessor already uses. It keeps the count and length limits.
Pending state for spans that never end. Each pending entry has a weakref.finalize on the span. A garbage-collection finalizer can run while this processor holds its lock, so the finalizer only appends the span key to a lock-free deque. Garbage collection can also end a span on the same thread (for example by closing an abandoned streaming generator), so the pending entry and key are built before the lock is taken and the lock is an RLock. on_start, on_end and force_flush drain the deque, so collected spans are removed even if later spans carry no scoped baggage. shutdown detaches the remaining finalizers.
Configuration. _parse_baggage_span_attribute_keys parses the env var once into key -> scopes, with an empty scope meaning every span. _get_baggage_span_attribute_keys applies the session.id and lineage rules and splits the result between BaggageSpanProcessor and ScopedBaggageSpanProcessor.
Processor order. GenAINestedClientSpanProcessor, then ScopedBaggageSpanProcessor, then AttributeRedactingSpanProcessor, then the exporter processors. The nested processor demotes the outer framework CLIENT span to INTERNAL first, so only the innermost CLIENT span gets lineage, and redaction rules apply to it.
Registry. Registered scopes live in a module-level dict replaced on write (copy-on-write), so span processing reads it without a lock.

Files

scoped_baggage_span_processor.py: new processor, built-in scope, registration hook.
aws_opentelemetry_configurator.py: installs GenAINestedClientSpanProcessor with or without agent observability. Parses key:scope entries, adds the lineage default and the session.id rule, installs the processor.
gen_ai_nested_client_span_processor.py: the inference operation list becomes a shared constant GEN_AI_INFERENCE_OPERATIONS. No behavior change.
CHANGELOG.md, tests.

Testing

Processor tests:
- matching and non-matching spans; operation name set after start; kind changed to CLIENT after start
- baggage captured at start when the span ends in another context
- no overwrite; count and length limits
- nested CLIENT spans with GenAINestedClientSpanProcessor; unscoped keys still on every span
- pending state released for spans collected without ending, including when later spans carry no lineage
- garbage collection while the processor lock is held does not deadlock, and neither does on_end re-entered on the same thread
Registration tests: a registered scope applies; registration after span start; replacement; any of several scopes matches; a predicate that raises does not block other scopes; invalid names, including the aws. prefix; an unknown scope warns once.
End to end through _init_tracing with agent observability off: tenant.id:aws.inference reaches only the inner span of a framework chat span wrapping an SDK chat span, and the outer span is INTERNAL.
Configurator tests: env parsing, the precedence table above, processor order, session.id:<scope> with agent observability on and off, processor installed for key:scope without agent observability.
Locally: the new tests, the configurator tests, and black, isort, flake8 and pylint pass. In my local venv the full run has 26 failures (mostly MCP instrumentation tests). They fail the same way without this change. The CrewAI, LangChain, LlamaIndex and OpenAI Agents tests do not load because those packages are not installed there. CI is the check for those.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

With AGENT_OBSERVABILITY_ENABLED, copy the aws.agent.lineage baggage entry only to GenAI inference spans: CLIENT spans whose gen_ai.operation.name is chat, text_completion, generate_content or embeddings. The backend uses it to attribute model cost to agents. ScopedBaggageSpanProcessor captures the baggage in on_start and applies it in on_end, when instrumentations have set gen_ai.operation.name. It runs after GenAINestedClientSpanProcessor, so only the innermost CLIENT span matches, and before the redacting and exporter processors. It never overwrites an existing attribute and never evicts one to make room. session.id and keys in OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS are still copied to every span at start. If a customer lists aws.agent.lineage there, it goes to every span and the scoped processor is not added.

syed-ahsan-ishtiaque

Found one pending-state lifetime issue; see the inline comment. The scoped baggage, nested CLIENT, redaction, and configurator suites passed (104 tests and 45 subtests). A separate garbage-collection reproduction confirmed that pending baggage survives after unended CLIENT spans are collected.

Pending baggage was keyed by span ID and removed only in on_end or shutdown, so CLIENT spans abandoned without end() kept their entry after they were garbage-collected. Tie each entry to the span's lifetime with weakref.finalize, and detach the finalizer when the span ends. Also stop filtering by span kind at start. Kind can change after start, so the scope is now checked only when the span ends.

syed-ahsan-ishtiaque

Verified that the original pending-baggage leak is fixed: the 10,000-span garbage-collection reproduction now leaves zero pending entries, and normal end/shutdown detach finalizers. The focused suites passed (106 tests and 45 subtests). The finalizer change introduces a separate shutdown deadlock, reproduced below.

Garbage collection can run the weakref finalizer at any allocation, including while the processor holds its non-reentrant lock, for example in shutdown(). The finalizer then blocked on the same lock and hung shutdown and process exit. The finalizer now only appends the span key to a thread-safe deque. The locked paths remove queued entries. Replace DictWithLock with a plain dict guarded by a processor lock, and add a regression test that collects an abandoned span while the lock is held.

syed-ahsan-ishtiaque

Verified the shutdown deadlock fix at 442b2e6: the original default-GC-threshold reproduction completes, collection while the processor lock is held completes, and an eight-thread check exported 4,000 spans with correct lineage and no deadlock. Broader unit suites passed: Python 2,181 tests plus 351 subtests (51 skipped), JS 1,737 tests; a real Vercel AI generateText reproduction also still passes. The deferred cleanup introduces a separate baggage retention regression described inline.

Queued removals from the span finalizer were only processed when a new span with scoped baggage started. If abandoned spans were collected and later traffic had no lineage, their pending baggage stayed in memory. Process the queue in on_start for every span, in on_end, and in force_flush. The queue is checked without the lock first, so spans with nothing to clean up stay lock-free.

Add an undocumented hook so a customer can scope their own baggage keys without a new release: register_scoped_baggage_key(key, scope) in Python and registerScopedBaggageKey(key, scope) in JS, exported from ./register. scope is a built-in scope name such as "inference" or a predicate that receives the ending span. Registration replaces the key map (copy-on-write), so span processing reads it without a lock. A predicate that raises skips the key for that span. The scoped processor is now installed whenever agent observability is enabled, even when the customer allowlist removes the lineage default, so registrations have a processor to attach to.

syed-ahsan-ishtiaque

Verified the entire change and the new registration hook at 53bf96b against the proposed handbook contract. The actual Python import and distro startup work with both the built-in inference scope and an ending-span predicate. Registration replacement, unknown-scope/disabled-mode warnings, predicate exceptions, existing-attribute preservation, and explicit baggage allowlist precedence behave as described. Registration also works when the default lineage key is allowlisted and the scoped processor starts with no default keys.

The main distro unit suite passes: 2,190 tests and 351 subtests, with 51 skips. I also reran the previous 10,000-span cleanup reproductions, the shutdown/GC deadlock reproduction, and the eight-thread 4,000-span GC stress check; all pass. No additional confirmed Python issues found in this revision.

If the distro configuration runs twice, the second tracer provider is not the global one, so its processor never sees spans. Claim the hook only if no processor holds it, and release it when the processor shuts down. This matches the JS fix for a second copy of /register.

OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS entries of the form key:scope copy the baggage key only to spans that match the scope. The built-in inference scope works without code. Applications can define more scopes with an undocumented register hook; scope names are resolved when each span ends, so the hook can run after the distro starts. This replaces the per-key registration hook.

syed-ahsan-ishtiaque

Reviewed the full PR and the revised scope-only registration API against the new handbook. The examples work with agent observability both on and off through actual distro startup. Built-in inference matching, custom tools, registration after span start, scope replacement, rejected built-in overrides, unknown-scope warnings, predicate exceptions, explicit unscoped precedence, and existing attributes all behave as expected.

The comment below records a reproduced reserved-key behavior for verification: an explicit session.id scope is overridden by the automatic all-span default with agent observability enabled. The author considers this likely intentional because the key is reserved; it is not being reported as a confirmed defect. No other confirmed issues found in this revision.

Local validation: 2,192 main distro tests and 351 subtests pass (51 skipped). The prior 10,000-span cleanup checks, default-GC-threshold shutdown reproduction, and eight-thread 4,000-span GC stress check still pass.

Agent observability copies session.id to every span, so a session.id:scope entry had no effect in that mode. Drop the scope explicitly and log a warning, and cover session.id:scope with agent observability on and off.

…ineage # Conflicts: # CHANGELOG.md

github-actions · 2026-11-06T00:56:11Z

Claude finished @syed-ahsan-ishtiaque's task in 3m 24s —— View job

Claude PR Review

Read prior reviews/comments to dedupe
Review the PR diff
Read key files (scoped processor, configurator, tests)
Post findings (or confirm no new issues)

No new substantive findings.

The author's own inline reviews have already covered the significant risk areas with reproductions and fixes: the GC retention of pending state (67ad406), the finalizer/shutdown lock deadlock (442b2e6), the collected-queue drain on every lifecycle call (7e98e26), and the reserved-key precedence for session.id:<scope> under agent observability (c05d235). I reviewed scoped_baggage_span_processor.py, the parser and installer changes in aws_opentelemetry_configurator.py, the move of GEN_AI_INFERENCE_OPERATIONS to a shared constant, and the new tests, and did not find additional substantive issues to raise.

A few spot checks that look fine:

Copy-on-write of _registered_scopes with lock-protected writes and lock-free dict reads is safe in CPython (atomic reference swap).
on_end order is GenAINestedClientSpanProcessor → ScopedBaggageSpanProcessor → AttributeRedactingSpanProcessor → batch, so the scope sees the demoted kind and redaction still runs on the written attribute.
_set_attribute_if_absent writes via BoundedAttributes._lock, honors max_value_len, and increments dropped without evicting an instrumentation-set attribute. Mirrors the existing AttributeRedactingSpanProcessor pattern.
_parse_baggage_span_attribute_keys handles trimming, empty entries, and lets unscoped entries override a scoped entry for the same key as documented.
finalizer.atexit = False + explicit shutdown() detachment avoids finalizer execution during interpreter teardown.
· branch feat/scoped-baggage-lineage

…es, reserved aws. prefix - The built-in inference scope now matches only the innermost of nested CLIENT inference spans with the same operation. The scoped processor tracks this itself, so key:aws.inference gives the same result with agent observability off, where GenAINestedClientSpanProcessor is not installed. - A key can list several scopes and is copied when any of them matches. An entry without a scope means every span. The env var is parsed once into key -> scopes, which also handles session.id and the lineage default for both processors. - Built-in scopes use the reserved aws. prefix (aws.inference). Names starting with aws. cannot be registered, so a future built-in cannot collide with an application scope.

github-actions · 2026-11-08T21:46:40Z

Claude finished @syed-ahsan-ishtiaque's task in 6m 35s —— View job

Claude PR Review

Read prior reviews/comments to dedupe
Review the latest PR diff (focus on commits since last review)
Read key files (scoped processor, configurator, tests)
Post findings (or confirm no new issues)

No new substantive findings.

The latest commit (f5451db) adds three things: a reserved aws. scope-name prefix, OR semantics across multiple scopes per key, and internal tracking of nested CLIENT inference spans so aws.inference works even without GenAINestedClientSpanProcessor. I traced through each change against the existing pending-state and finalizer lifecycle work in scoped_baggage_span_processor.py, the parser/installer changes in aws_opentelemetry_configurator.py:933-975, and the new test_scoped_baggage_span_processor.py and configurator tests, and did not find additional substantive issues.

Spot checks that look fine:

Nested-tracking in on_end (scoped_baggage_span_processor.py:164-178) records the child's inference operation on the parent entry under the processor lock, even when the child itself has no pending baggage (the parent_entry lookup is outside the if entry is None guard). This matches the GenAINestedClientSpanProcessor demotion rule (same-operation only) and keeps aws.inference correct whether or not that processor is installed.
_scope_matches uses _gen_ai_inference_operation(span) (CLIENT + inference op), so when GenAINestedClientSpanProcessor has already demoted the parent to INTERNAL, the built-in matcher independently rejects it; when it is absent, entry.inference_child_operations provides the equivalent rejection — both paths agree.
The reserved-prefix check in register_baggage_scope rejects aws.<anything> after trimming/:/, checks and warns, while leaving inference (unprefixed) as a legal custom scope name — the test test_unprefixed_inference_is_a_custom_scope_name pins this.
_get_baggage_span_attribute_keys() keeps the previously-verified session.id precedence under agent observability and now adds the aws.agent.lineage:{aws.inference} default via setdefault, so an explicit customer-configured scope (or unscoped listing) still wins.
Pending-state deque drain is still exercised on every lifecycle path (on_start for spans with no matching baggage via _remove_collected(), on_end under the lock, force_flush, and shutdown clearing), so the retention-regression fix from 7e98e26 is unaffected by this commit.
Multi-scope OR semantics (any(self._scope_matches(scope, span, entry) for scope in self._scoped_keys[key])) is covered by test_key_matching_any_of_its_scopes_is_copied and test_predicate_error_does_not_block_other_scopes.
· branch feat/scoped-baggage-lineage

syed-ahsan-ishtiaque

Reviewed the complete PR and the newly published f5451db revision. No new confirmed issues found in Python.

Verified the aws.inference scope with agent observability on and off through actual ADOT startup and exported spans: two and three nested same-operation spans copy the key only to the innermost span, different inference operations both match, and reconstructed parent contexts also work. With agent observability off, span kinds stay CLIENT. An inference child that carries no mapped baggage still excludes its matching parent.

The built-in and custom-scope examples, OR matching, throwing-predicate fallback, reserved aws. names, custom unprefixed inference, late registration and replacement, unknown-scope warnings, existing attributes, prototype names, unscoped precedence, reserved session IDs, and explicit scoped/unscoped lineage overrides all passed runtime checks with both flag settings.

Local validation: 2,199 core distro tests plus 361 subtests pass (51 skipped; optional framework suites excluded). The earlier pending-state and GC regression tests pass. The companion JS review records two reproduced issues in its new parent-link bookkeeping; the Python reconstructed-parent control passes.

syed-ahsan-ishtiaque

Re-reviewed the complete PR at f5451db against its description and the revised JS companion at 091a1b7. The Python head is unchanged since the previous review. All 20 Python runtime checks rerun through actual tracing startup pass: built-in/custom scope registration with agent observability on/off, OR and unscoped precedence, reserved names and session/lineage rules, and inference nesting using original or reconstructed parent contexts, including children without baggage. The results agree with JS. I found no new confirmed issues or regressions.

Install GenAINestedClientSpanProcessor whether or not agent observability is enabled. It demotes the outer span of nested CLIENT inference spans, so aws.inference matches only the innermost span in both modes. This removes the nesting tracking from ScopedBaggageSpanProcessor. The processor is generic again: aws.inference is a plain built-in predicate in the scope table, looked up by name like a registered scope. Defaults (session.id and aws.agent.lineage) still apply only with agent observability.

github-actions · 2026-11-09T19:27:53Z

Claude finished @syed-ahsan-ishtiaque's task in 5m 37s —— View job

Claude PR Review

Read prior reviews/comments to dedupe
Review the latest PR diff (focus on fe5f40e, new since the last review)
Read key files (scoped processor, configurator, nested processor, tests)
Post findings (or confirm no new issues)

No new substantive findings.

The one new commit since the previous review (fe5f40e) is a clean refactor:

GenAINestedClientSpanProcessor is now installed unconditionally in _init_tracing (aws_opentelemetry_configurator.py:352), and the duplicated nested-inference tracking in ScopedBaggageSpanProcessor (added in f5451db) is removed. _PendingSpan.inference_child_operations and the parent-entry bookkeeping in on_end are gone, and _scope_matches is a simple _get_scope(name) lookup. aws.inference is a plain built-in in _BUILTIN_SCOPES, resolved like a registered scope.
Processor order is unchanged and still correct: GenAINestedClientSpanProcessor → ScopedBaggageSpanProcessor → AttributeRedactingSpanProcessor → batch. The nested processor demotes the outer framework CLIENT span to INTERNAL before ScopedBaggageSpanProcessor.on_end runs, so is_gen_ai_inference_span naturally rejects the outer span.
The resulting user-visible change — framework spans wrapping an SDK inference call become INTERNAL even without AGENT_OBSERVABILITY_ENABLED — is called out in the new CHANGELOG entry at CHANGELOG.md:14-17 and in the updated test_inference_scope_stamps_only_innermost_span_without_agent_observability assertion (SpanKind.INTERNAL, test_aws_opentelementry_configurator.py:596). The author's intent to avoid double-counting the inference call as a dependency is stated in the commit message.
All earlier risk areas remain addressed by prior commits: pending-state GC retention (67ad406), finalizer/shutdown lock deadlock (442b2e6), collected-queue drain on every lifecycle call (7e98e26), reserved-key precedence for session.id:<scope> (c05d235), and the reserved aws. scope prefix / OR semantics (f5451db).

Spot checks that look fine:

is_gen_ai_inference_span at scoped_baggage_span_processor.py:29-37 is unchanged in behavior and now serves as both the module-public predicate and the registered aws.inference built-in.
The removed inference_child_operations tracking relied on "children end before parents"; the retained path through GenAINestedClientSpanProcessor uses the same assumption, so no new ordering risk is introduced.
_get_scope consults built-ins before registered scopes, so even if _clear_registered_scopes() is called in tests, the built-in resolution is unaffected.
The two deleted tests (test_only_innermost_client_span_gets_lineage_without_nested_processor, test_nested_inference_spans_with_different_operations_both_match) are no longer meaningful because the nested processor is always installed; the retained test_only_innermost_client_span_gets_lineage_with_nested_processor and the end-to-end test_inference_scope_stamps_only_innermost_span_without_agent_observability cover the two modes.

Minor nit (not blocking, not posting inline): the PR description still contains the older paragraph stating the processor "does not change span kind, so installing the scoped processor without agent observability does not affect Application Signals", which the new unconditional nested-processor install supersedes. The CHANGELOG entry is the authoritative description now.
· branch feat/scoped-baggage-lineage

syed-ahsan-ishtiaque

Reviewed fe5f40e against the updated description and commit message, including the companion JS change. Installing the nested processor with agent observability off and demoting the outer inference span is an intentional behavior change; the runtime checks confirm it. I found one P2 regression in the now-unconditional nested processor's parent-operation cache, described inline.

The core distro suite passes 2,197 tests and 361 subtests (51 skipped; optional framework suites excluded locally). All 20 Python functional runtime checks pass and agree with JS on built-in/custom scopes with agent observability on/off, OR and unscoped precedence, reserved names and defaults, reconstructed parents, three nesting levels, different operations, and baggage-free children. Additional lifetime comparisons against the previous heads confirm the new orphaned nesting-record growth. No other confirmed findings.

syed-ahsan-ishtiaque · 2026-11-09T20:40:43Z

    # processors to ensure exporters observe the updated kind.
-    if is_agent_observability_enabled():
-        trace_provider.add_span_processor(GenAINestedClientSpanProcessor())
+    trace_provider.add_span_processor(GenAINestedClientSpanProcessor())


[P2] Bound nested-parent state before enabling the processor globally

Installing this processor outside the agent-observability gate exposes its unbounded parent-operation cache to every ADOT application. GenAINestedClientSpanProcessor.on_end() records an entry for any inference child with a parent ID, and only the parent's local on_end() or shutdown removes it. A valid remote parent represented by NonRecordingSpan never receives that local callback, so otherwise completed model calls permanently grow the cache.

I reproduced this at fe5f40e through _init_tracing with AGENT_OBSERVABILITY_ENABLED=false and no baggage-key configuration. Two batches of 10,000 CLIENT/chat spans, each under a distinct remote parent context, leave 10,000 then 20,000 entries in _parent_span_id_and_operation_to_gen_ai_client_child, although every child ended. GC, force_flush(), and 1,000 ordinary completed requests leave all 20,000 entries; only shutdown clears them. The same startup/workload using f5451db retains zero. Collected local parents and local parents ending before their children show the same growth; normal child-before-parent completion leaves zero. The JS companion has the same regression.

Please keep records only for tracked local parents and release them with the parent's lifetime, including GC, without recreating records for already-ended parents. Add regressions for remote/non-recording parents, collected parents, and children ending after their parents before installing this processor unconditionally.

Garbage collection can run while the processor holds its lock and end a span on the same thread, for example by closing an abandoned streaming generator that holds a span open. Build the pending entry and span key before taking the lock, and use an RLock so a re-entrant on_end cannot deadlock. Same hardening as the GenAINestedClientSpanProcessor fix.

github-actions · 2026-11-09T23:04:51Z

Claude finished @syed-ahsan-ishtiaque's task in 5m 42s —— View job

Claude PR Review

Read prior reviews/comments to dedupe
Review the latest PR diff (focus on commits since last review)
Read key files (scoped processor, configurator, nested processor, tests)
Post findings (or confirm no new issues)

No new substantive findings.

The one new commit since the previous review (9593444) is a small hardening of ScopedBaggageSpanProcessor and only touches that file plus its tests:

self._lock becomes a threading.RLock.
The two allocations inside on_start and on_end critical sections (_PendingSpan(pending, finalizer) and _span_key(span)) are moved outside the with self._lock: block.
New regression test test_span_ended_while_lock_is_held_on_the_same_thread_does_not_deadlock holds the processor lock and calls span.end() on the same thread, verifying re-entry succeeds and the lineage attribute still lands on the exported span (test_scoped_baggage_span_processor.py:201-215).

Why this is correct: a tracked-object allocation under the old threading.Lock could trigger cyclic GC, and a finalizer run during that collection could end another span on the same thread, which re-enters on_end on this processor and would have deadlocked on the non-reentrant lock. The fix matches the same hardening pattern already in GenAINestedClientSpanProcessor (via DictWithLock), and keeps the deque-based cross-thread finalizer handling introduced in 442b2e6 intact. Thread-safety of concurrent writers is unaffected because RLock still serializes different threads.

Spot checks that look fine:

on_start still assigns self._pending_by_span[span_key] = entry under the lock and still drains _collected first, so pending-state and queued-removal invariants from 7e98e26 are preserved (scoped_baggage_span_processor.py:165-167).
on_end keeps the pop under the lock and only calls entry.finalizer.detach() and _set_attribute_if_absent outside — the detach is idempotent and the attribute write is already lock-protected via BoundedAttributes._lock (scoped_baggage_span_processor.py:174-182, 232-249).
The span passed to on_start/on_end is kept alive by the caller's frame, so the newly-created weakref.finalize(span, ...) cannot fire between its creation and the dict insertion. No orphaned-entry window.
shutdown is unchanged and still snapshots entries under the lock, then detaches finalizers outside it, so the pattern established in 442b2e6 continues to hold under RLock.
All earlier risk areas remain addressed by prior commits: pending-state GC retention (67ad406), finalizer/shutdown lock deadlock (442b2e6), collected-queue drain on every lifecycle call (7e98e26), reserved-key precedence for session.id:<scope> (c05d235), reserved aws. prefix / OR semantics (f5451db), and the unconditional nested-processor install (fe5f40e).

The author's own open P2 against aws_opentelemetry_configurator.py:352 (unbounded parent-operation cache in GenAINestedClientSpanProcessor for remote/non-recording parents) is not touched by this commit and the original inline comment is still on the line, so I am not re-posting it here.
· branch feat/scoped-baggage-lineage

syed-ahsan-ishtiaque added 2 commits November 4, 2026 17:04

chore: add PR number to CHANGELOG entry

01ccdf6

syed-ahsan-ishtiaque commented Nov 5, 2026

View reviewed changes

Comment thread aws-opentelemetry-distro/src/amazon/opentelemetry/distro/scoped_baggage_span_processor.py Resolved

syed-ahsan-ishtiaque commented Nov 5, 2026

View reviewed changes

Comment thread aws-opentelemetry-distro/src/amazon/opentelemetry/distro/scoped_baggage_span_processor.py Resolved

syed-ahsan-ishtiaque commented Nov 5, 2026

View reviewed changes

Comment thread aws-opentelemetry-distro/src/amazon/opentelemetry/distro/scoped_baggage_span_processor.py Resolved

syed-ahsan-ishtiaque added 2 commits November 5, 2026 12:43

syed-ahsan-ishtiaque commented Nov 5, 2026

View reviewed changes

syed-ahsan-ishtiaque added 2 commits November 5, 2026 16:04

syed-ahsan-ishtiaque commented Nov 6, 2026

View reviewed changes

Comment thread aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_configurator.py Resolved

syed-ahsan-ishtiaque added 2 commits November 5, 2026 16:46

fix: ignore a scope on session.id under agent observability

c05d235

Agent observability copies session.id to every span, so a session.id:scope entry had no effect in that mode. Drop the scope explicitly and log a warning, and cover session.id:scope with agent observability on and off.

Merge remote-tracking branch 'origin/main' into feat/scoped-baggage-l…

e5d1f98

…ineage # Conflicts: # CHANGELOG.md

syed-ahsan-ishtiaque marked this pull request as ready for review November 6, 2026 00:55

syed-ahsan-ishtiaque requested a review from a team as a code owner November 6, 2026 00:55

syed-ahsan-ishtiaque changed the title ~~feat: copy aws.agent.lineage baggage to GenAI inference spans~~ feat: copy aws.agent.lineage baggage to GenAI inference spans and support key:scope baggage entries Nov 6, 2026

syed-ahsan-ishtiaque mentioned this pull request Nov 6, 2026

feat: copy aws.agent.lineage baggage to GenAI inference spans and support key:scope baggage entries aws-observability/aws-otel-js-instrumentation#586

Closed

syed-ahsan-ishtiaque commented Nov 8, 2026

View reviewed changes

syed-ahsan-ishtiaque commented Nov 9, 2026

View reviewed changes

syed-ahsan-ishtiaque closed this Nov 10, 2026

Conversation

syed-ahsan-ishtiaque commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes for users

Precedence

How it works

Files

Testing

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude PR Review

Uh oh!

github-actions Bot commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude PR Review

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude PR Review

Uh oh!

syed-ahsan-ishtiaque left a comment

Choose a reason for hiding this comment

Uh oh!

syed-ahsan-ishtiaque Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude PR Review

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

syed-ahsan-ishtiaque commented Nov 5, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 8, 2026 •

edited

Loading

github-actions Bot commented Nov 9, 2026 •

edited

Loading

github-actions Bot commented Nov 9, 2026 •

edited

Loading