Skip to content

feat(tracing): kirocrew trace show prints one turn as a waterfall from the local span files - #37537

Merged
javenciu merged 1 commit into
mainfrom
feat/trace-show
Nov 15, 2026
Merged

javenciu merged 1 commit into
mainfrom
feat/trace-show

Conversation

@javenciu

@javenciu javenciu commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a developer can read one traced turn, its model calls, tool calls, sub-agents and the waits between them, without a collector.

With telemetry.traces_enabled on, every turn already writes spans to <data home>/traces/spans-YYYY-MM-DD-<pid>.jsonl (tracing/exporter.py). Nothing in Kiro Crew reads those files back. To see why one turn took 40 seconds, a developer has to set up an OTLP backend or read raw JSON lines, where parent links, times and ids are spread over hundreds of records and sorted by finish time.

Why it matters

The local files are the only trace output that needs no account, no network and no extra install. They are also the first thing someone checks when a turn feels slow or stuck. Without a reader they can only be inspected by hand.

Not a goal

  • A dashboard view of the same data (a later change; this one is the CLI half).
  • New spans, new attributes or any change to what the exporter writes. A hook span does not exist yet, so none is drawn; the view shows whatever spans the files hold.
  • A setting. The command only reads, so there is nothing to turn on.

What changed (motivation → approach → change)

Symptom → cause: spans are written and never read.

  • src/kiro_crew/tracing/render.py (new): reads the shards, picks the most recent trace holding the key, and prints it as an indented tree. Each row has the span's start offset, duration and a proportional bar. A gap row marks a stretch under a parent where no child ran, which is where "the turn felt slow" usually hides. A failed span shows its error.type. Imports no OpenTelemetry SDK and no regular expression.
  • src/kiro_crew/cli_parser_agent.py, src/kiro_crew/cli_help.py, src/kiro_crew/cli.py: the trace show <key> verb, its help entry and its dispatch. The key is a turn id, a session key or id, an ACP session id or a trace id.
  • docs/guides/telemetry-otlp-traces.md, docs/system-specs/modules/cli.md: the command, with what it prints.
  • test/tracing/test_trace_render.py (new), test/test_cli_split_dispatch.py: the tests below.

Reading is bounded and read-only:

  • Each shard goes through safe_read_file; a shard over 50 MB is skipped with a warning, a line over 256 KB, one nested past the JSON parser's limit, or one that is not a span (an id that is not hex counts) is skipped, and one warning per shard counts them. At most 100,000 spans are kept, newest shards first; at most 500 rows print, and a cut trace ends with the exact number of spans never shown (the trace's span count minus the span rows printed, so the subtrees under a cut-off span are counted).
  • An unreadable shard, or a traces path that cannot be listed, stops the run with Error: cannot read <file name>: <exception class>, so a lookup never answers "nothing matched" for data it could not read. The text of the exception, which quotes the resolved path, is not printed.
  • Only the span name, timing and error.type are printed, each through safe_terminal_line. Span attributes are allowlisted in tracing/correlation.py and never hold prompt or tool content; the command prints none of them anyway.
  • Nothing is written, and the tree walk is one pass with an explicit stack, so a deep or cyclic parent chain cannot recurse.
  • An unknown key prints one line (No trace matches <key>.) and exits 0, as memory search does for no hits.

Backwards compatibility

A new command; no existing command, setting, file or span changes. With tracing off, the directory is absent and the command prints the one-line no-match answer.

Tests

  • Round 2: the row-cap line counted only entries still queued, so a cut trace read as nearly complete; it now reports every span not printed.
  • Before: on main 4c530d0bd2, test_a_command_hands_its_namespace_to_its_handler[trace] fails (the parser has no trace command) and test/tracing/test_trace_render.py fails to import.
  • After: 21 new tests pass, plus the dispatch case. They cover the waterfall layout and gaps, error marks, key lookup by turn, session and trace id, the most-recent pick and the count of others, unknown, empty and oversized keys, a missing directory, malformed and oversized lines, an oversized shard, the span cap, an unreadable shard, hostile names and ids, a deeply nested line, an unlistable folder, orphan spans, the row cap, the exact count of spans a cut trace hides (a root with two subtrees of 600 spans), no writes, and a round trip through the real exporter. No test needs an installed backend CLI.
  • Related: the tracing tests, the CLI dispatch, help, facade and lazy-import tests and the import-budget test (659 passed); the flake and wait ratchets (122 passed).
  • flake8, isort, mypy, the black gate, the comment-history gate, the sync-I/O gate and the lane CI checks pass.
  • Reproduce: python -m pytest test/tracing/test_trace_render.py -n0
  • Proof: PROVEN test/tracing/test_trace_render.py::test_a_turn_prints_as_an_indented_waterfall_with_its_gaps test/test_cli_split_dispatch.py::test_a_command_hands_its_namespace_to_its_handler[trace] test/tracing/test_trace_render.py::test_a_line_nested_past_the_parser_limit_is_skipped_like_any_other_bad_line test/tracing/test_trace_render.py::test_the_cut_line_counts_every_span_never_shown_not_just_the_queued_ones (head ff77066)

Manual verification

A turn of 10 s with two overlapping tool calls (one failed), read back through the command:

trace aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa  5 spans  10.00s  2026-11-14 22:00:00 UTC
invoke_agent kirocrew                             <1ms  10.00s  ##############################
  · gap 500ms                                     <1ms  ..
  kirocrew.step                                  500ms   2.00s   #######
  · gap 500ms                                    2.50s         ..
  execute_tool shell                             3.00s   1.00s           ###
  execute_tool read                              3.50s   1.00s            ####   ! Timeout
  · gap 4.50s                                    4.50s               ..............
  kirocrew.step                                  9.00s   1.00s                             ###

Sibling sweep

Sibling: src/kiro_crew/tracing/exporter.py -- not affected -- the writer is unchanged; the reader uses its shard-name rule (shard_name_re("spans")) so a stray file in the folder is left out
Sibling: src/kiro_crew/tracing/provider.py -- not affected -- traces_dir() is the one place the folder is named, and the command uses it
Sibling: src/kiro_crew/cli_commands_memory.py -- not affected -- the no-match answer follows memory search

Pattern harvest

Rule candidate: review-prompt
Pattern: a local log written for one purpose has no reader in the product, so the only way to use it is to open the raw file.

Related Issues

Part of the tracing work tracked in #1643 (the local-sink reader; no open pull request or assigned issue covers a trace viewer).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 15, 2026 06:45
@javenciu
javenciu enabled auto-merge (squash) November 15, 2026 06:45
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ff770665d91ff476f214f00fa863261af0156b9a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ff770665d91ff476f214f00fa863261af0156b9a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: it adds a read-only CLI reader for span files that are already written, takes nothing away, and changes no UI.

Notes

  • The goal names "a developer" as the user, so the MCP-first rule in docs/architecture/mcp.md doesn't apply. If the agent should later diagnose its own slow turns, that would need an MCP twin in a follow-up PR, not in this one.

[DESIGN-REVIEWED] ff77066

Description read

[DESCRIPTION-READ] e202964155e3e7ce2d771bd5968105afc3d5df500d0977c3e440e7589b898769

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ff770665d91ff476f214f00fa863261af0156b9a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (8 items) — 8 justified

Intent: a developer can read one traced turn's timing locally, with no collector. This is an ADDITION.

  1. New kirocrew trace show <key> command prints a turn as a waterfall from the local span files. — justified
  2. gap rows mark stretches under a parent where no child ran. — justified
  3. A turn id, session key or id, ACP session id or trace id picks the most recent matching trace, and a line says how many others match. — justified
  4. A failed span shows its error.type. — justified
  5. Reads are bounded: shards over 50 MB, lines over 256 KB, at most 100,000 spans, at most 500 rows, and a count of spans not shown. — justified
  6. An unreadable shard or folder exits 1 and prints only the exception class. — justified
  7. A help entry under the agent command group. — justified
  8. A guide section and a cli.md spec row. — justified

[FIRST-PRINCIPLES-REVIEWED] ff77066

Description read

[DESCRIPTION-READ] 52f5c886478faee89e999c65cb52621e773c21bc03e432ee5c7fa30ba19a26e1

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ff770665d91ff476f214f00fa863261af0156b9a — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • render.py:336: if the row cap is hit just as the last span is drawn, and only an ancestor's trailing gap is still on the stack, the output ends with "... 0 more spans not shown". It's cosmetic and needs an exact row-count coincidence.

[OPUS-REVIEWED] ff77066

Verdict parsed from the review's SHA-scoped output markers for commit ff770665d91ff476f214f00fa863261af0156b9a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ff770665d91ff476f214f00fa863261af0156b9a false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ff770665d91ff476f214f00fa863261af0156b9a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ff77066

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ff770665d91ff476f214f00fa863261af0156b9a false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own.

Adds a read-only kirocrew trace show <key> command that prints one trace as a waterfall from the local span shards. Checked against the code: shards are picked with the exporter's own shard-name pattern and read through safe_read_file, whose 50 MB ceiling matches the PR's shard cap, and the data home is not on the sensitive-path list. The parser matches the SDK's to_json output (0x-prefixed hex ids, status_code ERROR, timestamps ending in Z, which fromisoformat reads on the required Python 3.12+). All five key attributes it matches on exist in correlation.py's allowlist. The tree walk uses an explicit stack bounded by the row cap, so it cannot loop forever. The earlier automated review's blocking finding (the cut line undercounted hidden spans) is fixed with len(spans) - drawn and a test using two 600-span subtrees. The command is registered, grouped in the help listing, dispatched lazily and covered by the dispatch test, and the docs are updated. It changes no existing state, so there is nothing to undo or migrate. Only non-blocking notes remain.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit ff770665d9 · 574 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto 2e56b3b as f6f51e3 with no change to this PR's code. The failures in backend shard 6 (Linux and Windows) were test/metrics/test_kiro_export_destination.py, which failed on main itself until #37545; the rebase picks that fix up. On this head test/tracing/test_trace_render.py, test/test_cli_split_dispatch.py and test/test_cli_help.py give 98 passed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Row cap under-reports what was cut (Opus, blocking, render.py:223). Fixed in ff77066. The line counted only the entries still on the stack, so the subtrees under queued spans were missing from the number. render now counts the span rows it prints and, at the cap, reports len(spans) - printed as ... N more spans not shown. Gap rows are not spans and are not counted. The wording changed from "rows" to "spans" because the number is spans. test_the_cut_line_counts_every_span_never_shown_not_just_the_queued_ones builds the reviewer's case (a root with two subtrees of 600 spans) and fails on the previous head with ... 104 more rows not shown where the exact figure is 703; it passes now. test_the_rows_are_bounded still pins the cap and its last line. The ops guide gained one sentence about the 500-row stop and the count.

Design lane (PASS, note). The handler trace_show sits beside the reader in tracing/render.py; the note suggests a cli_commands_* home if the later dashboard view reuses the reader. Not changed in this PR: the dashboard is a separate change, the handler is about a dozen lines, and read_spans, pick_trace and render already import nothing from the CLI, so the handler can move when a second consumer exists. Adding a cli_commands_* owner now would also put a name into the facade manifests without a reader beyond this one.

First-principles lane (PASS). Its remark is that nothing else reads the span files yet. That is the reason for the PR: kirocrew trace show is the first reader, and its reads reuse shard_name_re, safe_read_file and safe_terminal_line.

GPT lane. No findings.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/).

This PR is part of the tracing work in #1643, which waits on a maintainer decision. The new top-level kirocrew trace command and the planned dashboard view are not in the phase plan of the accepted tracing RFC, so a maintainer should confirm the command name and the follow-up.

Review result on ff77066: no blocking findings. The parser matches what JsonlSpanExporter writes. Shards are picked with the exporter's own shard_name_re and read through safe_read_file. Every printed field passes safe_terminal_line. The tree walk is bounded by the row cap. The earlier cut-count fix has a test that fails on the previous head. The tests use fixed timestamps and tmp_path, and they include a round trip through the real exporter, which is a good way to pin the reader to the writer.

Optional:

  • render.py:120: a live shard between 50 MB and the default 100 MB rotation is skipped, so trace show can miss the newest turns on a busy day. Consider streaming each shard line by line, or at least document the skip and treat EFBIG at read time like the size skip.
  • render.py:224: if stack: prints "... 0 more spans not shown" when only a gap entry is left, and spans in a parent cycle drop without a count. if drawn < len(spans): fixes both.
  • render.py:26: MAX_SHARD_BYTES repeats hooks.MAX_FILE_BYTES. Importing that constant next to safe_read_file keeps the two caps from drifting apart.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve (no auto; the author holds it).

kirocrew trace show <key> is a new, read-only CLI command that prints one trace from the local span files as a waterfall.

What I checked:

  • Read-only: it lists and reads traces/spans-*.jsonl through the existing safe_read_file and writes nothing (tested by comparing size and mtime before and after). No gateway, no SDK.
  • Output: only the span name, timing and error.type; every name goes through safe_terminal_line. A non-hex trace or span id is not a span, so it never reaches the terminal. Tests cover control sequences and an attribute that must stay out.
  • Bounds: shard size, line length, span count, rows and depth are all capped, and a line nested past the JSON parser's limit is skipped like any bad line. The tree walk uses an explicit stack, so a deep trace cannot recurse.
  • Errors: an unreadable shard or folder stops with the file name and the exception class only, never a full path; a missing folder reads as "nothing matched".
  • No new refusal: it is a new command, nothing on main changes. The shard filter reuses the existing shard_name_re from metrics/local_exporter.py; no new regex in product code.

I read the whole diff on ff77066, including the 286-line test file.

Not checked: I did not run it locally against a real span folder.

@javenciu
javenciu merged commit a880cd0 into main Nov 15, 2026
99 checks passed
@javenciu
javenciu deleted the feat/trace-show branch November 15, 2026 17:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants