Repository navigation
feat(otel): session tree depth and child counters + root-count gauge (#21273) - #21308
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the session-tree projection is seeded before the first turns after boot. Until it is, depths are silently left out with no counter showing the gap. [DESIGN-REVIEWED] 9365558 Description read[DESCRIPTION-READ] a7c9ac6a3c5feb6f765744e3677e45c73160cfa9565590ec89aa12bb042d6f04 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The Goal asks for two measurements, but the PR ships three instruments: the per-ordinal Not justified as shipped
What this change shipsInventory (7 items) — 4 justifiedIntent: let maintainers see how deep delegated session trees go and how many branch, via OTel behind consent — ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 9365558 Description read[DESCRIPTION-READ] 89aef110c262259ebfe8813fdc59d795278504bbfb47855b6002ec30722c9dbd That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Nothing here blocks the merge; one advisory finding remains. FINDING — src/kiro_crew/messaging/dispatch_crew_log.py:235 — [OPUS-REVIEWED] 9365558 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/crew_log/otel_session_tree_fold.py:32 -- Summing adds at False positive or not applicable? A repository writer can comment: |
2de446e to
fef1f8a
Compare
fef1f8a to
0a02558
Compare
0a02558 to
f625224
Compare
|
Thanks for the first-principles pass. Addressed on head Item 6 (dirty-flag cache) -- accepted, removed. Item 7 (second gauge-registration path) -- accepted, removed. The roots gauge is now registered through one Item 4 (the Items 8 and 10 -- not this change. Those belong to the export-skeleton commits from #21272 (the slot-scope loop, the |
6af4019 to
aa8e99f
Compare
aa8e99f to
0ca9402
Compare
725ca61 to
d799b6f
Compare
9e2f2e6 to
e4f5f82
Compare
|
AI-review override not recorded: |
|
/ai-review override first-principles e4f5f82: The crew log is the declared source for session-tree telemetry (design #21271); emitting from the write path would add work to the append path and lose the restart baseline. The write-path change is one integer field known at session creation; all aggregation stays in the fold. |
Human judgment recorded@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
HOLD at e4f5f82. The branch conflicts with main and needs a rebase. It also stacks on #21342, which waits on the maintainer agreement on design and data scope that #21271 requires (RFC #21288 is still a draft). Before the next review, please also:
|
|
/ai-review override first-principles 4c1b678: The crew log is the declared source for session-tree telemetry (design #21271); emitting from the write path would add work to the append path and lose the restart baseline. The write-path change is one integer field known at session creation; all aggregation stays in the fold. |
Human judgment recorded@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override first-principles 0b4cbc2: The crew log is the declared source for session-tree telemetry (design #21271); emitting from the write path would add work to the append path and lose the restart baseline. The write-path change is one integer field known at session creation; all aggregation stays in the fold. |
Human judgment recorded@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override first-principles c834de5: The crew log is the declared source for session-tree telemetry (design #21271); emitting from the write path would add work to the append path and lose the restart baseline. The write-path change is one integer field known at session creation; all aggregation stays in the fold. |
Human judgment recorded@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override first-principles ac74b08: The crew log is the declared source for session-tree telemetry (design #21271); emitting from the write path would add work to the append path and lose the restart baseline. The write-path change is one integer field known at session creation; all aggregation stays in the fold. |
Human judgment recorded@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
/ai-review override first-principles ce06ecf: The crew log is the declared source for session-tree telemetry (design #21271); emitting from the write path would add work to the append path and lose the restart baseline. The write-path change is one integer field known at session creation; all aggregation stays in the fold. |
Human judgment recorded@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
buluoray
left a comment
There was a problem hiding this comment.
Approve the code at ce06ecf. Merge only after #21342 lands and the #22180 / #21271 design is agreed.
Both r108 items are fixed. A root depth now goes through the same readable-tree check as a child depth (_depth_readable_nodes and root_depth, session_tree_projection.py L2131 and L2175), and both writers use it (crew_log_attribution.py L282, dispatch_crew_log.py L235). The over-cap test is there. Labels are bounded: depth is 0, 1, 2 or 3+, nth is 1, 2, 3, 4-7 or 8+, and parent_kind goes through the closed agent_kind check. The roots gauge is recalculated from the folded tree on every collect, keeps no running state, and reports nothing when the tree is unseeded or incomplete. Mutating any of these bounds fails a named test.
Follow-ups, not blocking:
- The GPT point is right: the sum of adds at nth >= k counts creators only for the single-value buckets, so please fix that sentence in the fold docstring, the exporter and metrics.md.
- The subscriber is installed only in start_dashboard, so an API-only gateway emits the roots gauge but no depth, child or usage counters.
- The comment at crew_log_export.py L648 says at or after install, but the code uses strictly greater than.
- Please open and link the issue for the creator.depth + 1 follow-up that r109 asked for.
- The Design scope concern and the provider.py black reformat need a disposition.
CI is green, and the related tests pass locally (455).
|
Rebased onto the new #21342 head
|
|
New head: c97f4ad. Still stacked on the #21342 commit, unchanged. Since the last head: removed the unused session-tree row counter, the second SESSION_OPENED_TYPE constant (now imported from projection_otel_usage, and the facade is back to excluded=()) and the _read_for_tests helper. The description now says what the #21342 part of the diff contains, and declares the provider.py black reformat. The raising-subscriber test still skips the bus.subscriber_error counter, which depends on test order. |
buluoray
left a comment
There was a problem hiding this comment.
Approve the code at c97f4ad. Merge only after #21342 lands, this PR is rebased onto it, and the #21271 design is agreed.
The install helper is no longer duplicated: its one definition is server_runtime/maintenance.py L392, and this PR only rewords the comment beside the call in server.py. The tree depth and width code is unchanged in substance since my earlier approve; the related tests passed locally on 105416d (453), and this push only removes code.
First Principles items 8 to 10 are gone on this head: _otel_session_tree_rows and its count_rows entry, the second SESSION_OPENED_TYPE and its excluded entry, and the test-only _read_for_tests.
Before merge:
- This head is still stacked on 7ded32a, not on the current #21342 head 95c52b9. Those two differ only in test_a_raising_subscriber_does_not_silence_this_one, which this PR also edits in a different way, so the rebase will conflict there. Please rebase onto 95c52b9 and keep one fix for that test.
- The First Principles blocker (the diff against main carries all of #21342) clears once #21342 merges and this PR is rebased; until then it needs an override that names the stack.
Follow-ups still open from my earlier approve:
- The sentence that creators with at least k children equals the sum of adds at nth >= k is still wrong (otel_session_tree_fold.py L32-L33, crew_log_export.py L222, metrics.md L451, and the description). That sum counts children at ordinal k or higher. For k of 1, 2 and 3 the adds at nth equal to k give the creators with at least k children; the 4-7 and 8+ buckets cannot give that.
- The subscriber is installed only in start_dashboard, so an API-only gateway exports the roots gauge but no depth or child counters.
- Please open and link an issue for the creator.depth + 1 follow-up.
- The provider.py black reformat (and the black-baseline row it drops) still has no disposition in the description.
- A leftover fragment, ONE ROW PER FOLD -- a later story, still sits at crew_log_export.py L393.
…21273) Record session-tree depth on session/opened and export three OTel metrics from the crew log, all riding the crew-log export skeleton. session/opened gains a depth field: how many creators stand above a session in the session tree, resolved by the caller from the in-memory tree and passed in, never derived on the writer thread. depth is a declared JSON_INT; absent means not established, never zero; 0 is written only when the caller knows the session has no creator (a channel conversation). slot_depth walks the folded tree to resolve it, returning None on every unresolvable case (unseeded, wrong store, incomplete, missing node, cycle, unplaceable ancestor). kirocrew.crew.session.tree.depth (counter, attr depth in 0 / 1 / 2 / 3+): a SESSION-keyed counting fold (otel_session_tree) renders this session's own depth, and one add is recorded per session this process can prove is new. A counter per bucket label rather than a histogram, so the distribution survives backends that do not keep explicit histogram buckets. Entries written before this change carry no depth and contribute no add. kirocrew.crew.session.tree.child (counter, attrs parent_kind and nth): one add per child session a creator makes. parent_kind is the creator session's closed agent_kind; nth is the child's ordinal among its creator's children, in 1 / 2 / 3 / 4-7 / 8+. Creators with at least k children is the sum of the adds at nth >= k, so the width distribution is read without waiting for a creator to end. The ordinal comes from the creator's own log: session/spawned is written there once per child at creation time, so the same fold adds one per entry and keeps one more integer plus the creator's closed kind (folded in step with the classifier otel_usage uses). No child identity, raw agent name or slot is stored and no parent link is walked. A creator this process cannot prove is new is baselined on first sight; the count resumes from the savepoint, so its next child is still counted at its true ordinal. kirocrew.crew.session.tree.roots (observable GAUGE, no attribute): the count of depth-0 sessions with children is a level of the whole tree that rises and falls, so it is a gauge, not a counter. The O(n) count over the folded tree runs on the OTel SDK collect thread, bounded by TREE_UNIT_CAP. All are gated by telemetry.enabled (default false); no attribute carries a session id, name or path. Registered in the fold catalogue (INTERNAL_PROJECTION_NAMES, _FOLDS, the generated FOLDS.md/folds.json, a nullable depth leaf) and in the export's _EXPORTERS table and the provider's gauge registration. The entry-types golden table and render digest are regenerated for the new field. Part of #21270.
Problem / Motivation
Goal: Record how deep delegated session trees get, and how many trees actually branch, and export both to OpenTelemetry -- off the append path and behind the existing telemetry consent gate.
Maintainers cannot say how deep delegated session trees run or how many roots have children. The crew log records who created whom (
session/opened.parent), but nothing turns that into a shape a maintainer can read. This is the session-tree story on the crew-log OTel epic (#21270), built on the export skeleton from #21272.Builds on #21342 (the OTel export skeleton), which is now merged into
main. This branch is rebased ontomain, so the diff contains only the session-tree change below.Why it matters
Delegation depth and branching are the clearest signal of how much work is actually delegated in practice -- the question the whole crew-log-to-OTel effort exists to answer a module at a time. Without it, the maintainers tuning subagent concurrency, staleness windows and dispatch policy are guessing at the shape of what they tune. The metrics are distribution-only: a depth histogram and a root count across an install, never a per-person timeline.
Not a goal
depthfield whose value is already known whensession/openedis written.What changed (motivation -> approach -> change)
Goal: get the two measurements out without a join and without slowing the append path.
Approach: record depth at write time, and read branching from the folded tree at collect time.
session/openedgains adepthfield -- how many creators stand above a session -- resolved by the caller from the in-memory tree and passed in, never derived on the writer thread. A session-keyed counting fold (otel_session_tree) renders this session's own depth; the export skeleton's subscriber turns it into one counter add per session, labelled by depth bucket. The same fold counts thesession/spawnedentries on a creator's own log, which gives each child its ordinal among its creator's children without walking any parent link. The root count is a property of the whole tree, so it is an observable gauge read on the SDK's collect thread (register_session_tree_gaugeinprovider.py, the samemeter.create_observable_gaugepath the inventory and process gauges take), counting depth-0 sessions that have children off the folded tree.Change:
kirocrew.crew.session.tree.depth(counter, attrdepthin0/1/2/3+): one add per opened session carrying a depth. A counter per bucket label rather than a histogram, so the distribution survives backends that do not keep explicit histogram buckets. Entries written before this change carry no depth and contribute no add.kirocrew.crew.session.tree.child(counter, attrsparent_kindandnth): one add per child session a creator makes.parent_kindis the creator session's closedagent_kind;nthis the child's ordinal among its creator's children, in1/2/3/4-7/8+. Creators with at leastkchildren is the sum of the adds atnth >= k, so the width distribution is read without waiting for a creator to end. The fold keeps one more integer (the child count) and the creator's closed kind; no child identity, raw agent name or slot is stored. Same never-over-count rule: a creator this process cannot prove is new is baselined on first sight, and because the count resumes from the savepoint, its next child is still counted at its true ordinal.kirocrew.crew.session.tree.roots(observable gauge, no attribute): the current count of depth-0 sessions with children. A gauge, not a counter, because the quantity rises and falls (a forgotten child drops a root) and is read off the folded tree.session/opened.depth(declared JSON_INT): birth depth, absent when not establishable, never inferred.slot_depthwalks the folded tree to resolve a child's;root_depthanswers a root's.docs/system-specs/modules/metrics.mdgains a row per new instrument in itsInstrumented signalstable.metrics/provider.pyregisters the roots gauge. Touching it meant the file had to passblack, so the whole file is black-reformatted (formatting only, no behaviour change) and its row is dropped from.github/black-baseline.txt.Depth is censored on both sides, never just one
A depth is only written against a tree that can place sessions: seeded for the store being read, and a reading not reported
incomplete.incompleteholds when the projection is unseeded, when a unit's bytes faulted, or when the population passedTREE_UNIT_CAP.How common is that, honestly. An earlier draft of this section called the over-cap case the permanent state of a busy install, and that overstated it against the cap's own declaration:
session_tree.pysays 4096 admitted units is far above any real population because retention keeps the store far smaller, and that a store reaching it usually has retention disabled. So over-cap is the rare arm. The arms that are not rare are the other two -- an unseeded projection (every reader before the first seed completes) and a faulted unit -- and they produce exactly the same missing edges. The censoring is therefore a correctness guard for a reading that cannot place a child, not a resignation about busy installs, and it is worth having whichever arm fires.What the censoring deliberately does NOT do is report its own extent. A maintainer reading thin depth counts cannot currently tell "few sessions" from "the tree could not be read", and a censored-opens counter beside them is the obvious way to close that. It is a new instrument rather than a fix to this one, so it is noted for the epic rather than added here.
A child's depth already refused in that state, because it needs the walk. A root's did not -- zero needs no walk, so it was written unconditionally. That combination is the one way this field can bias its own distribution: on exactly the installs with the most delegation, every child would be withheld and every root still recorded, so the depth counter would report almost all sessions at depth 0 while the roots gauge reported nothing. The answer to "how deep does delegation go" would come back backwards, with no second metric showing the gap.
So both sides now go through one readable-tree check (
session_tree_projection._depth_readable_nodes, used byslot_depthand by the newroot_depth). A tree the fold cannot read produces no observation at all rather than a root-only distribution, which is the honest reading of "absent, never inferred": a session left out of a distribution, not an invented tree.Bucket labels
The depth labels (
0/1/2/3+) and thenthlabels (1/2/3/4-7/8+) are closed sets, so neither attribute's cardinality grows with the deepest or widest tree.0is its own label, so "no delegation" and "one level" never share a bucket.Backwards compatibility
This PR adds a new fold, a new field, and three metrics; reads only already-declared entry types and changes no existing record, validator or public function. An entry written before this change simply carries no
depthand contributes no depth add. The fold-catalogue anddashboard_typesadditions only add; they hide and remove nothing.Tests
depthfield: stored when valid, absent when unresolved, refused for a bool/negative, the entry still validates, an old (no-depth) entry replays with no observation.slot_depth,root_depthandcount_roots_with_children: driven over roots, chains, citations and cycles.root_depthanswers 0 on a readable tree andNonewhen the projection is unseeded or its seed faulted; an over-cap tree withholds a child's depth AND the root's, so it cannot produce a root-only distribution. The writer-side resolver (_crew_log_tree_depth) is driven over both.otheroutside the set), baselines a resumed creator and then counts its next child at its true ordinal, and a real crew log + eager worker + bus path drives threesession/spawnedentries tonth1, 2, 3; the roots read folds each collect, reportsNone(no observation) when unseeded/incomplete, a real 0 for a seeded empty tree, and never raises into collect; its bus-free callback is proven; the gauge registers throughprovider.py.dashboard_typesnullable-driver were regenerated for the new field and fold.Manual verification
N/A -- unit coverage exercises the real fold, the real tree projection, and the gauge callback; there is no UI or external service in this change.
Related Issues
Part of #21270.
Closes #21273.
Builds on #21342 (the OTel export skeleton, merged); the boot-time install helper (
_kick_otel_crew_log_subscriber) lives there. Not for merge until the design in #21271 is agreed and the data scope is confirmed.