Skip to content

feat(tracing): session lifecycle spans: open, close, reset, compaction, switch, adopt (6, #1643) - #36830

Open
kyleseaman wants to merge 2 commits into
mainfrom
crew/devmate/otel-pr6
Open

kyleseaman wants to merge 2 commits into
mainfrom
crew/devmate/otel-pr6

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: With tracing on, each live session's lifecycle (opened, compacted between turns, switched, adopted, closed or reset) shows up as a short trace that sits beside its turn traces under the same session.id, and a worker, fork or linked session names its parent.

Today only model turns have spans. Nothing records when a session opened, when Crew compacted it between turns, when its agent, conductor or model was switched, or why it ended. A worker session's traces do not say which session created it.

Why it matters

The tracing RFC's phase 6 (§10) and Kyle's "track all the steps/sessions/turns" both need the session itself on the timeline. An operator debugging a slow or costly chat cannot tell from turn traces alone that the session was recycled after a compaction, reset by a model switch, or moved onto a new conductor, and cannot find a worker's parent session.

Not a goal

  • Manager-change spans for handoff, release and hand-back on close (phase 5's scope); this PR records the adopt tree edge only.
  • Compaction inside a turn (phase 4a's kirocrew.compaction step span).
  • A manual /compact slash command or the in-turn model fallback swap.
  • New config keys, export destinations or settings UI.

What changed (motivation → approach → change)

With tracing on, a trace backend now shows each session's opens, between-turn compactions, switches, adoptions and its close or reset as a short trace under the session's own session.id, beside its turn traces.

Design record: the native tracing RFC §5.8 (session lifecycle row) and §10 row 6.

Approach: hook the few funnels every surface already passes, so one small module covers them all. A generation is one stay of a session key in the session registry, from record_session_started to its end record; it gets one trace.

Span Comes from
kirocrew.session.open (root) metrics/sessions.py::record_session_started, every surface
kirocrew.session.close / .reset record_session_ended and record_sessions_ended, with the end reason
kirocrew.session.compaction session_parts/turn_context.py::_compact_session, the funnel of both automatic triggers, with the coordinator's answer
kirocrew.session.switch chat_api/slot_guards.py::_reset_slot_session_or_warn once a switch's reset lands, plus a live model switch in slot_model.py; conductor when the Crew Mode migration switches a root (agent_switch.py)
kirocrew.session.adopt session_tree.py and session_handoff.py once their tree edge lands
  • New tracing/sessions.py. Every span carries the same session.id and kirocrew.session_key_hash as the session's turns, so an ACP recycle keeps one session. Children parent to the open span by id, as run anchors do.
  • Ids: trace H(trace.session, session_key, generation), span H(session.<event>, session_key, generation, n). The generation is the registry start time in microseconds, so a session resumed after a restart (same ACP session id) gets a new trace instead of reusing span ids. Two conformance vectors are added.
  • Parents: trace_binding.py::note_slot_lineage notes the parent before either dashboard allocation can register a session, the first turn's claim (turn_admission.py) and the eager spawn's prefetch (eager_spawn.py): a creator only when this process minted the lineage, then forked_from, then the dashboard slot a channel session is linked to. Lineage stamps and the session tree name bare slot keys, so slot_session_key maps them to the registry key a session is traced under. The open span carries the hashed parent key and kirocrew.session.relation, and links to the parent's open span when it is live in this process.
  • A switch is recorded on the generation it reaches, read before the reset or the live set_model RPC, and under the key the registry folds onto (trace_binding.py::registry_key), so a successor registered mid-teardown keeps its own trace and an alias key still finds the session. A live switch records only once its caller's rebind check passes. The ACP session id is kept only within the shared bounded_session_id limit.
  • Allowlist: five closed-vocabulary keys under kirocrew.session.*. No titles, prompts or message text; session keys are hashed.
  • The metrics spec's Traces section documents the spans and ids.

Rejected: hooking the crew-log emitters (they only run with the crew log on, and are keyed by ACP session id), and keying ids on the ACP session id as the RFC table sketched (a resume after restart would reuse every span id).

Backwards compatibility

Compatible: tracing is off by default, and off every hook returns at its first check. The only behaviour change outside tracing is the log text of a failed teardown during a Crew Mode migration switch, which now says conductor switch instead of agent switch.

Removes nothing: the diff adds spans, five allowlisted attributes and two conformance vectors.

Tests

test/tracing/test_tracing_sessions.py (new, 25 tests):

  • A session's spans, lifecycle and turns, share one session.id across an ACP recycle, and each generation is its own trace with open, compaction, switch, close, then open, reset.
  • Every event hangs under its generation's open span.
  • The same inputs give the same ids, matching the formula.
  • Tracing off emits nothing and keeps no state.
  • A worker's open span names its creator hashed and links to the creator's open span; lineage precedence is creator, then fork, then linked; a session that is its own slot names no parent.
  • An adoption names the new parent and links to it.
  • Bare slot keys resolve to the registry keys sessions are traced under: an ordinary tab names no parent, a worker stamped with its creator's bare slot key matches that session's hash and links to it, and a channel-linked session names its dashboard twin.
  • The real eager prefetch (_spawn_admitted_prefetch) has noted a worker's creator when its allocation runs.
  • A switch whose reset lets a successor register mid-teardown lands in the torn-down generation's trace, with its ACP session id.
  • An ACP session id over MAX_ACP_SESSION_ID_LEN is not retained or put on a span.
  • A live model switch records its span only after the caller's rebind check, in the generation read before the RPC; a rebind mid-RPC records none.
  • A reset or live switch named by an alias key lands in the session registered under the folded key.
  • A failed or cancelled compaction is an error; an unknown outcome is other; an unknown switch kind is dropped.
  • No attribute except the local raw key carries a session key or text.
  • The real record_session_started / record_session_ended / record_sessions_ended drive open, close and reset.
  • The real compaction funnel records the coordinator's answer.
  • The real switch funnel records a switch only once its reset lands (a declined reset records nothing).

test_the_attribute_allowlist_is_pinned gains the five keys; the conformance fixture gains two vectors. The tests use no sleeps; the metrics-record test pins metrics/sessions.py's own clock binding to distinct readings.

Determinism: the file passed repeated runs, a run in reversed test order, and a run under -n 4, each 25/25. The prefetch, switch-generation, live-switch, folded-key and id-bound tests each fail on the production code without their fix.

Manual verification

N/A for manual steps: the tests drive the real metrics records, compaction funnel and switch funnel with an in-memory exporter; no UI changes.

Related Issues

Part of #1643 (phase 6).

Overlap with #37088

#37088 adds a kirocrew.session.parent_change span in the acting turn's trace. This PR's kirocrew.session.adopt span sits in the adopted session's own lifecycle trace. So one adoption gives one span in each trace, by design: the turn that made the change shows it, and so does the session it changed.

Merge order: #37088 first. This PR then rebases onto it once the re-sync reset comment lands.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners November 13, 2026 19:46
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ca28bedaf6eedc7c9279f1918450c66f8b9e9144 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ca28bedaf6eedc7c9279f1918450c66f8b9e9144 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. Every hook returns at its first line while tracing is off and swallows its own errors, nothing is taken away, and no user-visible surface changes, so no screenshot is needed.

Notes

  • slot_session_key documents a known gap in its docstring: a closed channel-linked parent falls back to its dashboard: key, so that worker's parent hash names a session that was never traced. Either resolve it from the session map or state the gap in the metrics spec's lifecycle section, so operators don't read the missing parent as a bug.

[DESIGN-REVIEWED] ca28bed

Description read

[DESCRIPTION-READ] 5e1e3c50223706baadf13f759ae6346f42736c6b909794acc794f55fe4a4bd77

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ca28bedaf6eedc7c9279f1918450c66f8b9e9144 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Phase 6 of the tracing RFC is accepted and listed as partial on base, and tracing stays off by default. Each item hooks a funnel that already exists, and nothing in the diff copies an existing mechanism.

What this change ships

Inventory (9 items) — 8 justified

Intent: let an operator see a session's opens, compactions, switches, adoptions and its close or reset in a trace backend, beside its turn traces. ADDITION.

  1. With tracing on, a kirocrew.session.open root trace appears per session generation. — justified
  2. Close and reset spans carry the end reason. — justified
  3. Between-turn automatic compaction shows up as a timed span with its outcome. — justified
  4. Agent, conductor, model, workspace and effort switches show up once they commit. — justified
  5. A session-tree adoption or handoff shows up as an adopt span linked to the new parent. — justified
  6. A worker, fork or linked session's open span names its hashed parent and links to it. — justified
  7. Five new allowlisted kirocrew.session.* attributes and two conformance vectors. — justified
  8. Switching tracing off and on again drops lifecycle state kept from before (provider.off_epoch, one consumer). — undeclared
  9. A failed teardown during a Crew Mode migration now logs "conductor switch" instead of "agent switch". — rides along

[FIRST-PRINCIPLES-REVIEWED] ca28bed

Description read

[DESCRIPTION-READ] 931c38ed6df18113670f46b8f742fecbc93b63ba8ce938de82af0ce8f6dc76db

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ca28bedaf6eedc7c9279f1918450c66f8b9e9144 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ca28bed

Verdict parsed from the review's SHA-scoped output markers for commit ca28bedaf6eedc7c9279f1918450c66f8b9e9144.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ca28bedaf6eedc7c9279f1918450c66f8b9e9144 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ca28bedaf6eedc7c9279f1918450c66f8b9e9144 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ca28bed

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ca28bedaf6eedc7c9279f1918450c66f8b9e9144 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own.

This PR adds session lifecycle spans (open, close, reset, compaction, switch, adopt) by hooking existing funnels. Tracing is off by default. Every hook returns early or swallows its own errors, and the always-evaluated arguments (_crew_log_lineage, slot_session_key) cannot raise. Switch spans are recorded only after each handler's last rollback or rebind check. The agent switch records at its own commit point. Runtime name resolution through the facade globals checks out. The allowlist stays closed: keys are hashed and the ACP session id is length-bounded. There are 25 targeted tests, and the metrics spec is updated in the same change. CI is green. No blocking defects found. The remaining gaps are minor observability omissions, plus a merge-conflict label and a rebase order the author already plans for.

Tier: 3, held for a person: merge conflict.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit ca28bedaf6 · 1329 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr6 branch from ad007d0 to 61aca79 Compare November 13, 2026 21:15
@kyleseaman

kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=0bbdc6f6edee Eager spawn writes a worker open span before its lineage is noted

fixed in 61aca79: trace_binding.note_slot_lineage runs before both dashboard allocations, the eager prefetch (before its get_or_create) and the first turn claim, so whichever registers the session first already has the creator.
Test test_the_eager_prefetch_notes_the_workers_parent_before_it_registers drives the real _spawn_admitted_prefetch; it fails on ad007d0 and passes on 61aca79.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 13, 2026
@kyleseaman

kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=4803882cd425 Eagerly created workers lose their parent attribution

fixed in 61aca79: the shared note_slot_lineage runs before the eager prefetch get_or_create as well as before the first turn claim, so the worker open span carries its creator hash, relation and link.
Test test_the_eager_prefetch_notes_the_workers_parent_before_it_registers fails on ad007d0 and passes on 61aca79.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@kyleseaman

kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=f91e082943bb A successor receives the previous generation switch

fixed in 61aca79: _reset_slot_session_or_warn reads trace_sessions.current(session_key) before the reset and both emit paths pass that generation to switched(), so a successor registered during the teardown keeps its own trace.
Test test_a_switch_names_the_session_its_reset_tore_down fails on ad007d0 and passes on 61aca79.

@kyleseaman

kyleseaman commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=31c7088fea7c Bound backend-authored IDs before retaining generations

fixed in 61aca79: opened() passes the ACP session id through validation.bounded_session_id before retaining it, so an id over MAX_ACP_SESSION_ID_LEN is kept as empty and never put on a span.
Test test_an_oversized_acp_session_id_is_not_retained fails on ad007d0 and passes on 61aca79.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/metrics.md (ba3a1ca -> ac4062d), carried in the same push as the review fixes at 61aca79 (clean merge-tree, unchanged patch-id); CI re-running.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/session_allocation_parts/get_or_create.py, src/kiro_crew/dashboard/handlers/sessions.py, src/kiro_crew/dashboard/routes/chat.py and docs/system-specs/modules/ since this head went green (ac4062d -> 46707f8); head 7734c16, unchanged patch-id; CI re-running.

@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr6 branch from 61aca79 to 7734c16 Compare November 13, 2026 22:12
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr6 branch from 7734c16 to 4f8b53b Compare November 14, 2026 00:34
@kyleseaman

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/tracing/correlation.py, src/kiro_crew/dashboard/trace_binding.py, test/tracing/test_tracing_ids_attrs.py and docs/system-specs/modules/metrics.md (#36833, OTel phase 5), which left this PR CONFLICTING (7734c16 on ac4062d -> 4f8b53b on c275f70). Conflicts resolved by keeping both sides; CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 14, 2026
…n, switch, adopt (6, #1643)

Review fixes on the rebase: session state kept from before tracing was last
switched off is dropped (provider.off_epoch), and a fork's parent key is
folded through registry_key before note_lineage.
@iamwhatever
iamwhatever force-pushed the crew/devmate/otel-pr6 branch from 4f8b53b to 5f062e0 Compare November 19, 2026 18:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Hi @kyleseaman, this is part of a cleanup of older open PRs. I rebased this on main and pushed 5f062e0 (one commit, your authorship kept). It folds in fixes for the two upheld findings on 4f8b53b:

  • Opus: provider.off_epoch() counts the times tracing resolves off. A generation or lineage note kept from before the last off is now dropped, not reused.
  • GPT F2: note_slot_lineage folds forked_from through registry_key.

Two new tests cover these. They fail on the old code, and test/tracing passes (395).

Opus now has no findings. GPT raised a new finding in the same place, so I am stopping here and leaving the next step to you. After a toggle, _start makes a fallback _Generation("") whose counter starts again at 1. Two off/on cycles then give two compactions the same (trace_id, span_id). A suggested fix: in _start, when there is no live generation, make the fallback's id unique, e.g. _Generation(f"e{provider.off_epoch()}") (or use the start time). Then take gen from that object, not from "". The fallback in switched and adopted goes through the same path.

Also, Backend Lint fails on black formatting: black --target-version py310 test/tracing/test_tracing_sessions.py.

@kyleseaman

kyleseaman commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @iamwhatever for the rebase, and for the off_epoch rework of the stale-generation and fork-alias fixes.

ca28bedaf6 is one commit on top of 5f062e01fb:

  • Lint: black-formats test/tracing/test_tracing_sessions.py (the Backend Lint & Type Check red; Coverage Gate failed closed behind it).
  • GPT F1, span-id collision after an off/on cycle (sessions.py:171): the repeatable events' counter now carries the generation's off epoch (<epoch>:<count>; epoch 0 ids are unchanged). That covers the "" fallback GPT named and a real generation rebuilt by a reset after the toggle. Each case has a test that fails on 5f062e01fb.
  • Opus, switch span on a rolled-back switch (slot_guards.py): a kirocrew.session.switch span is recorded only once the switch can no longer be rolled back.
    • Model, effort and workspace: _reset_slot_session_or_warn records only while the slot still resolves to the session it reset, the condition each handler's rebind guard checks next, so a session_rebound 409 leaves no span.
    • Agent and conductor: switch_slot_agent can still roll back after that (the metadata write's 503 history_unavailable, its second rebind 409, the owner and member-grant checks), so it passes record_switch=False and records the span itself past its last rollback point.
    • Tests: test_a_reset_switch_rebound_mid_reset_records_no_switch, test_an_agent_switch_refused_after_its_reset_records_no_switch, test_an_agent_switch_that_commits_records_one_switch.
  • Test isolation: provider.reset_for_testing() resets the off epoch, so test_the_same_inputs_give_the_same_ids no longer depends on which tests ran first.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr6 branch from cff02b2 to 76b19c3 Compare November 19, 2026 21:18
…tches

A generation recorded again after tracing is switched off and on, the
unknown fallback or a real one rebuilt by a reset, restarted its event
counters at 1 under the same trace, so two compactions, switches or
adopts could export the same trace and span id. The repeatable events'
counter now carries the generation's off epoch; epoch 0 is unchanged.
provider.reset_for_testing() also resets the off epoch, so the per-test
id assertions do not depend on which tests ran first.

_reset_slot_session_or_warn recorded kirocrew.session.switch as soon as
the reset landed, before its callers could still roll the switch back.
The model, effort and workspace handlers roll back only at their rebind
guard, the next statement, so the helper records only while the slot
still resolves to the session it reset. The agent switch can also roll
back after that (the metadata write's 503, the second rebind guard, the
owner checks), so it asks the helper not to record and records the span
itself once past its last rollback point.

Also black-formats test/tracing/test_tracing_sessions.py.
@kyleseaman
kyleseaman force-pushed the crew/devmate/otel-pr6 branch from 76b19c3 to ca28bed Compare November 19, 2026 22:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants