Repository navigation
feat(dashboard): show what each session delivered beside its spend - #37407
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The change is additive and read-only, it goes through the existing derivation seam, it falls back to "no outcome" whenever a lock or read fails, and screenshots of this revision are attached. Notes
[DESIGN-REVIEWED] e89169b Description read[DESCRIPTION-READ] 25f22fc9b7156dbfed5374f83055f9672fcb5adec30ce6228eb4ccfa3b266711 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (8 items) — 8 justifiedIntent: let a person see what each session's spend bought (commits, lines, PRs, merges) next to its cost. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] e89169b Description read[DESCRIPTION-READ] 421500ca6a00752c3c9a51a8f5e5353e6982f2a7f006bbc053fb0f6ed07b63bb That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS No blockers. Every new column and both new tips appear in the screenshots in both themes, and the blind reader read the Commits and Merged tips correctly. Notes
[UX-REVIEWED] e89169b Description read[DESCRIPTION-READ] b14f195cd609385a0bd6aad92e897605e1c5e13ec4e17f96aea58ba6853d55df That sha256 names everything this verdict read: the PR description, plus the 10 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 2193 lines, more than the 400 it may approve on its own. An additive, read-only feature that shows commits, changed lines and PRs (from transcripts) beside each session's spend. I checked the APIs it calls (derive_messages_chained_full_with_keys, publication_hold with expected_keys, derivation_hold raising TranscriptBusy, reentrant RLock/flock file locks, read_commands never returning an empty list, _parse_row_ts). Privacy checks go through the existing guarded read and are repeated under the locks before the response is built. A refused session is dropped from both spend units, and a busy lock or a large set drops every outcome. The cached cost block is never written into, and a window with no delivery returns the same object as before. Merge state goes only to the owner and comes from the existing chip cache. Backend and frontend tests are extensive. I found nothing that blocks the merge. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
9f0c808 to
593da7d
Compare
|
All three GPT findings on 9f0c808 are fixed in 593da7d, which is also rebased onto af94fac.
On the previous code, 7 of the 8 new cases fail. The eighth pins git's UX punchline, the |
593da7d to
014f96c
Compare
|
Round 2 on 593da7d: the GPT finding, the two CI failures from this PR and the UX notes are fixed in 014f96c, rebased onto 2092903. GPT, Backend Tests (3.12, 3), Frontend Tests (1), The remaining red shards are main-side (main's own tests at |
014f96c to
92e5af9
Compare
|
First principles, span 13c83ecd4e97 (second spelling of the guarded transcript read). Fixed in 92e5af9. Opus, span 369e1a8835cc ( GPT, span 0052fa7534cd ( Maintainer rule, no new regular expression in product code. Fixed in 92e5af9. Screenshots. The eight PNGs are removed from the commit ( |
92e5af9 to
c3fc2ff
Compare
|
GPT, span 0052fa7534cd ( First principles PASS punchline: "the PR adds a fourth shell-command splitter to the codebase". Fixed in c3fc2ff. This PR's own scanner is deleted.
None of these returns which operator joins two commands. This PR needs that, because only Body. It now has an r4 line, the measured counts (118 / 155 / 2,475 passed) and the proof head c3fc2ff. It is 7,955 bytes and keeps all seven attachment URLs. |
c3fc2ff to
aad8df4
Compare
|
|
aad8df4 to
383cb97
Compare
|
CI Shard 6, |
383cb97 to
185bdaf
Compare
|
|
The Telemetry Spend table ranks sessions by credits but cannot say what a session produced for them, so a long run that merged three changes and one that only talked cost the same line. The gateway already holds the answer in each session's own transcript: the shell tool rows carry the command and its output. Read the 50 costliest sessions' transcripts through the derivation seam (memoised by file mtime, reread at most once a minute while a live file keeps changing, and never memoised past a 64-transcript chain) and attach an outcome to each row that delivered something: commits and changed lines from git's own summary line (a quiet commit prints none and is not counted), pull requests from the URL a create command printed unless it says the pull request already exists, and merged pull requests from the chip-status cache, never refreshed here. Commands are read by the shared shell reader the batch guard uses, so quoting is the shell's. Each row's command is read from the call's own arguments, since a long title can arrive cut short. The owner alone gets merge figures. Amended commits, commit previews (--dry-run and the formats that imply it) and commits whose output is mixed with other text are not counted, so every figure is a minimum; an incognito or temporary session, or one with such a transcript earlier in its tab, gives none. The table gains five columns once a row has an outcome, and a window with none returns the payload as it was.
185bdaf to
e89169b
Compare
|
|
|
This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/). Review result on this head: no blocking finding.
The privacy handling is careful and well tested from both sides, through the real route and a real second thread. Optional:
|
bolichen97
left a comment
There was a problem hiding this comment.
Approve. Read the whole diff on this head: usage_outcome.py, the telemetry.py hook, both test files, the frontend, and every locale.
- Additive read API:
/api/telemetry/startuponly gains an optionaloutcomeper row andoutcome_scanon the block, andwith_outcomesreturns the same object when nothing was delivered (pinned bytest_telemetry_outcome_unchanged.py, which passes on main too). The only frontend reader is the Spend table, and it shows the five columns only when some row has an outcome. Every failure path (no log, unreadable transcript, busy holds, over 256 transcripts) drops outcomes and answers as main does. Nothing is refused. - Privacy: rows come through
derive_messages_chained_full_with_keys, so incognito/temporary transcripts give nothing; the memo re-asks every chain member before a hit is served, and the response is built underpublication_hold, so a session tightened after the scan is dropped from both units. Merge figures come from the chip cache, owner only, never refreshed here. - Parsing reuses the shared shell reader (
read_commands,_shell_tokens) andparse_source_url. No regex in product code (theRegExpis in the vitest file). - The memoised cost block is copied, never written into.
Non-blocking:
- A live session's transcript is reread through the derivation seam at most once a minute while the Spend tab is open. For a chain of hundreds of MB, that is one worker thread for seconds every minute. Fine as bounded, but worth a size cap later if the System page shows it.
- The response now enters
derivation_holdover every contributing chain's lock stems while it builds the JSON. The build is short and busy locks drop the outcome rather than wait, so I see no write stall, but a comment saying the acquire is non-blocking would help the next reader.
Not checked: no local run; I relied on the 97 green checks.
Problem / Motivation
Goal: the Telemetry Spend table shows, beside what each session cost, what it delivered: commits, changed lines, pull requests, how many are merged, and the spend per merged pull request.
The table ranks sessions by credits only, so a run that merged three changes and one that only talked read alike. Each shell tool row in a transcript already stores the command and its output, the handler already maps sessions to transcripts, and the session chips already parse pull request URLs and cache merge state. This change reads those together, with no git command and no provider call.
Before:
After (
≥2: two of three pull requests confirmed merged, one not checked yet):The Commits and Merged tips:
Why it matters
Credits alone cannot say what the spend bought. On 40 large local transcripts, 8 held a delivery: 17 commits and 100 distinct pull requests. Quiet commits are not counted (an earlier rule took 137 more from
git commit -q && git logrows), so commits are a floor. Nothing to configure or store.Not a goal
What changed
src/kiro_crew/dashboard/handlers/usage_outcome.py(new).outcome_eventswalks a transcript's tool rows once. A row's command is the call's owncommandargument (viaparse_tool_args), else its title, since a long title can arrive cut short. Commands are read by the shared shell reader the batch guard uses (security.shell_commands.read_commands, words viashell_normalizer._shell_tokens): an operator inside quotes does not split, a redirection and its target are not words, and a here-document body is data. No new regular expression.git commitwhose output holds git's[branch sha]summary line, with lines from git's statistics line only (never the subject or a later command's output); agh pr create/glab mr createthat is the command's one pull request call, followed only by&&or a pipe, whose output has the change's URL on its own line (validated byparse_source_url).-q), even joined to a log: it prints no summary line, and nothing else says which commit that call made; amends (any abbreviation of--amend); commit previews,--dry-runand the formats git documents as implying it (--short,--porcelain,--long,-z/--null); output mixed with other text; a create whose output says the pull request already exists; a command with a quote left open or longer than 16,384 characters.ConversationLog.derive_messages_chained_full_with_keys. An incognito or temporary transcript, or one whose line cannot be read, raisesTranscriptWithheldand its session gets no outcome. A tightened line keeps its mtime, so the memo keeps every transcript the chained read covered and asks each one's line on every poll; a chain over 64 is not memoised.respond_with_session_outcomesbuilds the response off the loop while every contributing chain is held: their lock stems in one sorted acquire, then eachpublication_holdin sorted order; a refused outcome is dropped from both units. A busy lock, or over 256 transcripts, sends none.with_outcomesfolds events into the 50 costliest rows in the window, memoised by mtime, reread at most once a minute, bounded (128 transcripts, 2,048 events each, 16,384 in all), and copies rows rather than writing into the memoised cost block.telemetry.py:api_telemetry_startupattaches outcomes before the title step, in both units, and returns the response built under the holds.TelemetryPanelConversations.tsx,telemetryPanelModel.ts: five columns, shown only once a row has anoutcome. A session with none reads—, never 0. A≥before the merged count means unchecked pull requests may also be merged.metrics.md, the feature map anddashboard.mddescribe the rules.Backwards compatibility
Additive. A row gains an optional
outcomeand the cost blockoutcome_scan, only when a session delivered something; a window with none returns the same object as before, pinned bytest_telemetry_outcome_unchanged.py, which passes on main. Merge figures go only to the dashboard owner, as with the chips. No setting, since nothing existing changes.Tests
KeyError: 'outcome'; 7 of 8 frontend tests fail with main's two frontend files.test_usage_outcome.py(116 cases),test_telemetry_outcome_unchanged.py,test_transcript_derivation_seam.pyandtest_usage_facade.py: 159 passed.test/metrics/, seam, facade, telemetry handler and usage tests: 3,025 passed, 31 skipped. Ratchets 122 passed; frontend 8 passed; flake8, isort, mypy, black, comment-history, sync-io pass.TestPublicationRevalidation).python -m pytest test/metrics/test_usage_outcome.py test/metrics/test_telemetry_outcome_unchanged.py test/test_transcript_derivation_seam.py test/test_usage_facade.pyandcd website && npx vitest run src/test/TelemetryPanel.outcome.test.tsxManual verification
Rendered the real Spend tab on loopback in both themes from a fabricated payload (above). Ran the reader over 40 local transcripts. A scratch repository confirmed
git commit -qprints no summary line on success.Sibling sweep
Sibling:
src/kiro_crew/dashboard/state_parts/source_links.py-- not affected -- chips from message text; its parser and cache are reusedSibling:
src/kiro_crew/slack/code_channels/wrapup.py-- not affected -- it reads its own workspace recordPattern harvest
Rule candidate: none
Pattern: a count beside spend comes only from stored text, through the derivation seam, and an unproven event is left out.
Related Issues
None.
Checklist
Contribution License Agreement