Repository navigation
feat(tracing): worker creation, report-to-wake links, manager changes and patrol counts (5b, #1643) - #37088
feat(tracing): worker creation, report-to-wake links, manager changes and patrol counts (5b, #1643)#37088kyleseaman wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: all of it is additive tracing that stays off by default, each fact is hooked where it is already known, and nothing is taken away. Notes
[DESIGN-REVIEWED] 4643b77 Description read[DESCRIPTION-READ] 3506f2297ee5cc077c464e4046b8f7dc4340a15eca0508d74ebc19f510ec576e That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item maps to a §5.5 producer in the tracing RFC (base status Notes
What this change shipsInventory (9 items) — 8 justifiedIntent: let an operator see in a conductor's trace who started each worker, which reports each wake answered, which reports never arrived, who manages each session, and the board's health. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 4643b77 Description read[DESCRIPTION-READ] 4b27ce172e96b43841171003e523b1bc7b76aa090a8a250ad572157aba6ab270 That sha256 names the PR description this verdict read. Recompute it with |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 1272 lines, more than the 400 it may approve on its own. Additive, off-by-default tracing hooks for worker creation, report-to-patrol links, undelivered reports, manager and parent changes, and board-health counts. Checked against the code: pending report links are stored and looked up under the same board key. The patrol mark is set only for the work-ledger watch's turn, can be claimed once, carries into a new task, and is spent at the queue hand-off. Moving the work_report unlanded-refusal earlier keeps behaviour the same, because the mirror was already skipped when the record didn't land. The undelivered wake still fires in a finally if the trace is cancelled. Create/fork/handoff/adopt/release result keys match what the routes return, and the hand-back loop handles both shapes of the change map. Every hook catches its own errors. Allowlist and spec are updated, and the tests drive the real routes and binding. The CI reds are redaction tests that don't exist at this head, so they come from the newer main CI merged with, not from this PR. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
1ca0e7b to
5ba5c3d
Compare
|
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
5ba5c3d to
fbb101f
Compare
|
fbb101f to
9dec790
Compare
|
Rebased: main moved in #36481 (host memory pressure on the work ledger) and #37005 (OTel steps inside a turn), which conflicted with this PR. Head fbb101f -> 9dec790 on main 12a03ac.
No behaviour change in this PR's own code. Related tests (7,036), the pre-push gates and semgrep pass on the new head. |
9dec790 to
5e2c63a
Compare
|
Rebased: main moved in #37095 (finish the conductor mode rename), which conflicted with this PR in
No behaviour change in this PR's own code (range-diff checked by an independent verifier). Related tests (7,036), the pre-push gates and semgrep pass on the new head. |
5e2c63a to
c7d358b
Compare
|
c7d358b to
c801216
Compare
|
|
|
This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/). Review result on this head: nothing blocks. We read the whole diff. The work-report reorder changes no behaviour, because the mirror append was already gated on Optional:
|
|
This head conflicts with main (GitHub reports |
… and patrol counts (5b, #1643)
c801216 to
4643b77
Compare
|
Rebased: main moved in #37158 (queue wait and the links a send, a broadcast and a workflow launch leave), which conflicted with this PR. Head c801216 -> 4643b77 on main 5c3b2cd.
No behaviour change in this PR's own code; an independent verifier checked the range-diff and found it unchanged. Related tests (7,117, OTLP extra installed), the pre-push gates and semgrep pass on the new head. |
Problem / Motivation
Goal: With tracing on, a conductor mode or Bot conductor's trace shows who started each worker, which reports each wake answered, when a report never arrived, who manages each session, and the board's health on every patrol.
5a made a round one trace, but the edges between sessions are still missing. A worker's first turn does not point back at the conductor turn that created it. A wake turn says nothing about which reports it is answering, so a batch of three finished workers looks like an unexplained conductor turn. A report the harness never delivered leaves no trace at all. A handoff, adopt or release changes who reads a session, and the trace cannot show it. And the stale, orphaned and overdue counts the conductor acts on live only in its tool output.
Why it matters
These are the questions an operator asks of a conductor mode run when something goes wrong: why did this worker exist, why did the conductor wake, why did it not hear from item X, and who was supposed to be watching. The tracing RFC (§5.5, phase 5) lists each of them. Without them a round's waterfall shows the work but not the causes.
Not a goal
What changed (motivation → approach → change)
Design record: the native tracing RFC §5.5 and §10 row 5; this is 5b, the half split off from #36833.
Approach: each producer already has one place where the fact is known, so each gets one hook there, and the turn that answers a fact gets a link when it binds.
kirocrew.session.create/.forksession_create/session_forkroutes (work_dispatchcreates through the same route)nudge_dashboard_fire.pymarks the turn a work-ledger watch delivers (work.patrol_turn);dashboard/trace_binding.pyadds the links when the turn binds, andchat_turn/queue_cycle.pyspends an unclaimed mark at the turn's queue hand-offkirocrew.work.reportand undelivered event written on the board since the last patrol turn, at most 64kirocrew.work.report_undeliveredsrc/kiro_crew/undelivered_call_watch.py'srecord_undelivered_call, next to its wakekirocrew.session.manager_changesession_handoff.hand_back_on_closekirocrew.session.parent_changekirocrew.work.{open,stale,orphaned,pending_overdue,undelivered}_itemswork_ledger_readtracing/workers.py(creation, manager changes and tree-parent changes); the rest is intracing/work.py. Named apart from feat(tracing): session lifecycle spans: open, close, reset, compaction, switch, adopt (6, #1643) #36830'stracing/sessions.pyso the two do not collide.kirocrew.session.opand the five hashed session keys (target, manager, previous manager, parent, previous parent), plus five counts. Values are ids, a closed set of operations, and counts: never a title or any text.Rejected: linking every conductor turn to the pending reports. A person's own turn does not answer them, so only the watch's turns take them; a wake steered into a turn already running adds none, because a span's links are fixed at its start, and the reports wait for the next patrol turn.
Overlap with #36830
This PR's
kirocrew.session.parent_changespan for an adopt sits in the acting turn's trace and records who moved the session. #36830's adopt event sits in the session's own lifecycle trace and records what happened to the session. So one adopt gives one span in each trace. That is by design, not a duplicate. Merge order: this PR first, then #36830 rebases onto it.Backwards compatibility
Compatible: tracing is off by default, and off, each hook is one
peek()returningNone. No route, ledger or wake behaviour changes; the hooks run after the operation committed and never raise.Removes nothing: the diff adds spans, links and allowlisted attributes.
Tests
test/tracing/test_tracing_work.pygains twenty-two tests, driving the real ledger routes andtrace_binding.bind_turn:session_createregisters the creation link before its first message starts the worker, so the worker's first turn takes it.kirocrew.session.parent_changespan with the parent hashes and no manager keys.test_session_handoff.py: a close's hand-back records one manager change per session moved, naming the creator as the new manager when the move removes an override.test_tracing_ids_attrs.py: the pinned allowlist gains the eleven keys.Determinism: 20 runs at
-n0and 10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_tracing_runs.py,test_session_handoff.pyandtest_chat_runner_coverage_queue.py, all green; a shuffled order (seed 4219) green. No sleeps.Local: related backend tests, including the chat queue hand-off and workflow suites (12,406 passed; the 224 red ones, 221 in the
test_ai_review_workflows_*CI-script tests and 3 intest_design_tweak_backend_queue.py, fail the same on a cleanorigin/mainworktree on this host; an earlier wider run's 10 app-backend, app-scaffold, codex-ACP and client-binding reds did too), semgrep rules on the diff (0 findings), the flake-pattern ratchet and the pre-push gates pass. Twotest_tracing_otlp.pytests (test_the_exit_flush_sends_spans_still_queued,test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header) fail onorigin/mainitself, from phase 4a and #36721; a separate fix PR covers them.Manual verification
N/A — unit coverage drives the real routes, the real turn binding and the real dashboard fire; no UI or live collector is involved.
Follow-ups
ultra outcomemarker: ending it needs a ledger signal for a finished goal.crew_log/emit_surfaces._note_eager) can still start a wake that binds during the trace call's thread hop; that wake misses the link and the next patrol turn takes it. Closing it needs a post-append, pre-publication hook inemit_surfaces, which changes the shared crew-log write path, so it is left for its own change.AcpAuthRequiredwhile the provider initialises) has already taken its creation or report links, and releasing the unused root does not give them back; restoring them on release, or taking them whenTurnTracestarts, would close it.TurnTracerather than declared in its__init__; declaring it would make a later__slots__change fail loudly instead of through the span tests.work_ledger_read.py, the fork hook, and the key pairssession_control.pypasses for adopt, handoff and release are covered only through the helpers today.MAX_ANSWERED_LINKS) and at most 4096 boards (_MAX_BOARDS).work.in_patrol_turn(): it has no production caller, and the tests could read_PATROLdirectly.Related Issues
Part of #1643
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)