Skip to content

feat(dashboard): show what each session delivered beside its spend - #37407

Merged
javenciu merged 1 commit into
mainfrom
feat/session-cost-per-outcome
Nov 15, 2026
Merged

javenciu merged 1 commit into
mainfrom
feat/session-cost-per-outcome

Conversation

@javenciu

@javenciu javenciu commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the Telemetry Spend table shows, beside what each session cost, what it delivered: commits, changed lines, pull requests, how many are merged, and the spend per merged pull request.

The table ranks sessions by credits only, so a run that merged three changes and one that only talked read alike. Each shell tool row in a transcript already stores the command and its output, the handler already maps sessions to transcripts, and the session chips already parse pull request URLs and cache merge state. This change reads those together, with no git command and no provider call.

Before:

Spend table before, light

After (≥2: two of three pull requests confirmed merged, one not checked yet):

Spend table after, light
Spend table after, dark

The Commits and Merged tips:

Commits tip, light
Commits tip, dark
Merged tip, light
Merged tip, dark

Why it matters

Credits alone cannot say what the spend bought. On 40 large local transcripts, 8 held a delivery: 17 commits and 100 distinct pull requests. Quiet commits are not counted (an earlier rule took 137 more from git commit -q && git log rows), so commits are a floor. Nothing to configure or store.

Not a goal

  • Exact counts. Every rule fails toward counting less, and the column tip says the figures are minimums.
  • Sessions beyond the 50 costliest in the window, and sub-agent rows.
  • A setting, a store or a metric counter. The change is read-only and additive.

What changed

  • src/kiro_crew/dashboard/handlers/usage_outcome.py (new). outcome_events walks a transcript's tool rows once. A row's command is the call's own command argument (via parse_tool_args), else its title, since a long title can arrive cut short. Commands are read by the shared shell reader the batch guard uses (security.shell_commands.read_commands, words via shell_normalizer._shell_tokens): an operator inside quotes does not split, a redirection and its target are not words, and a here-document body is data. No new regular expression.
    • Counted: a git commit whose output holds git's [branch sha] summary line, with lines from git's statistics line only (never the subject or a later command's output); a gh pr create / glab mr create that is the command's one pull request call, followed only by && or a pipe, whose output has the change's URL on its own line (validated by parse_source_url).
    • Not counted: a quiet commit (-q), even joined to a log: it prints no summary line, and nothing else says which commit that call made; amends (any abbreviation of --amend); commit previews, --dry-run and the formats git documents as implying it (--short, --porcelain, --long, -z/--null); output mixed with other text; a create whose output says the pull request already exists; a command with a quote left open or longer than 16,384 characters.
    • Privacy: rows are read through the derivation seam, ConversationLog.derive_messages_chained_full_with_keys. An incognito or temporary transcript, or one whose line cannot be read, raises TranscriptWithheld and its session gets no outcome. A tightened line keeps its mtime, so the memo keeps every transcript the chained read covered and asks each one's line on every poll; a chain over 64 is not memoised. respond_with_session_outcomes builds the response off the loop while every contributing chain is held: their lock stems in one sorted acquire, then each publication_hold in sorted order; a refused outcome is dropped from both units. A busy lock, or over 256 transcripts, sends none.
    • with_outcomes folds events into the 50 costliest rows in the window, memoised by mtime, reread at most once a minute, bounded (128 transcripts, 2,048 events each, 16,384 in all), and copies rows rather than writing into the memoised cost block.
  • telemetry.py: api_telemetry_startup attaches outcomes before the title step, in both units, and returns the response built under the holds.
  • TelemetryPanelConversations.tsx, telemetryPanelModel.ts: five columns, shown only once a row has an outcome. A session with none reads —, never 0. A ≥ before the merged count means unchecked pull requests may also be merged.
  • Seven i18n keys in every catalog; metrics.md, the feature map and dashboard.md describe the rules.

Backwards compatibility

Additive. A row gains an optional outcome and the cost block outcome_scan, only when a session delivered something; a window with none returns the same object as before, pinned by test_telemetry_outcome_unchanged.py, which passes on main. Merge figures go only to the dashboard owner, as with the chips. No setting, since nothing existing changes.

Tests

  • Before: on main the outcome module does not exist and the route test fails with KeyError: 'outcome'; 7 of 8 frontend tests fail with main's two frontend files.
  • After: test_usage_outcome.py (116 cases), test_telemetry_outcome_unchanged.py, test_transcript_derivation_seam.py and test_usage_facade.py: 159 passed. test/metrics/, seam, facade, telemetry handler and usage tests: 3,025 passed, 31 skipped. Ratchets 122 passed; frontend 8 passed; flake8, isort, mypy, black, comment-history, sync-io pass.
  • r1-r5: statistics-line figures, previews, the shell reader, the seam fence, withheld older transcripts, no quiet-commit inference, the chain cap.
  • r6-r8: a transcript tightened after the scan, or an older one of the tab, loses its outcome in both units; a checked chain stays held until the response is built; the hold bound (TestPublicationRevalidation).
  • Reproduce: python -m pytest test/metrics/test_usage_outcome.py test/metrics/test_telemetry_outcome_unchanged.py test/test_transcript_derivation_seam.py test/test_usage_facade.py and cd website && npx vitest run src/test/TelemetryPanel.outcome.test.tsx
  • Proof: PROVEN test/metrics/test_usage_outcome.py::TestPublicationRevalidation, test/test_usage_facade.py, test/metrics/test_usage_outcome.py::TestWithOutcomes, test/metrics/test_usage_outcome.py::TestAttachSessionOutcomes, test/test_transcript_derivation_seam.py::TestTheFence, website/src/test/TelemetryPanel.outcome.test.tsx (head e89169b)

Manual verification

Rendered the real Spend tab on loopback in both themes from a fabricated payload (above). Ran the reader over 40 local transcripts. A scratch repository confirmed git commit -q prints no summary line on success.

Sibling sweep

Sibling: src/kiro_crew/dashboard/state_parts/source_links.py -- not affected -- chips from message text; its parser and cache are reused
Sibling: src/kiro_crew/slack/code_channels/wrapup.py -- not affected -- it reads its own workspace record

Pattern harvest

Rule candidate: none
Pattern: a count beside spend comes only from stored text, through the derivation seam, and an unproven event is left out.

Related Issues

None.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 14, 2026 23:27
@javenciu
javenciu enabled auto-merge (squash) November 14, 2026 23:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e89169b1c7c1586f076507e99cc195ae972482ae — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The change is additive and read-only, it goes through the existing derivation seam, it falls back to "no outcome" whenever a lock or read fails, and screenshots of this revision are attached.

Notes

  • Each startup poll now holds up to 256 transcript locks while the response is built. If a session's save or a cron append hits that lock and waits past the acquire ceiling, its own write could time out. Keep the hold window to building the JSON bytes only; that is the current shape, so don't let it widen later.
  • Commit and PR counts come from the stored command text and its output. If git or gh change their output format, the counts drop quietly toward zero. That is the safe failure direction, but nothing would flag the drift.

[DESIGN-REVIEWED] e89169b

Description read

[DESCRIPTION-READ] 25f22fc9b7156dbfed5374f83055f9672fcb5adec30ce6228eb4ccfa3b266711

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e89169b1c7c1586f076507e99cc195ae972482ae — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (8 items) — 8 justified

Intent: let a person see what each session's spend bought (commits, lines, PRs, merges) next to its cost. ADDITION.

  1. Spend table gains Commits, Lines, PRs, Merged and Per-merged columns, shown only when a row delivered something — justified
  2. A session with no delivery reads —, never 0; ≥ marks a merged count with unchecked PRs — justified
  3. Merge figures and spend per merged PR go only to the dashboard owner — justified
  4. Incognito or temporary sessions, unreadable transcripts, and sessions tightened before the response is built get no outcome — justified
  5. Only the 50 costliest sessions in the window are read; outcome_scan tells "not read" apart from "none" — justified
  6. Figures can lag a live session by up to a minute (a memo that rereads at most every 60s, with bounds) — justified
  7. API: optional outcome on each row and outcome_scan on the cost block, both read by the frontend; a window with no delivery returns the same object as before — justified
  8. Seven i18n keys plus updates to metrics.md, the feature map and dashboard.md — justified

[FIRST-PRINCIPLES-REVIEWED] e89169b

Description read

[DESCRIPTION-READ] 421500ca6a00752c3c9a51a8f5e5353e6982f2a7f006bbc053fb0f6ed07b63bb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of e89169b1c7c1586f076507e99cc195ae972482ae — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

No blockers. Every new column and both new tips appear in the screenshots in both themes, and the blind reader read the Commits and Merged tips correctly.

Notes

  • The — cell: the blind reader "can't tell whether a dash means 'zero' or 'unknown'". Add one clause to outcome_tip: "A dash means nothing was found, not zero."
  • merged_tip says "the session's spend", but the column it divides is labelled "Credits" (or Cost, depending on the unit). Interpolate the active unit's column label so the tip names the column the user can see.
  • outcome_tip says every figure is a minimum, while ≥ marks only some Merged cells. The reader called this "confusing". Say that ≥ applies only to merge checks that are still pending.

[UX-REVIEWED] e89169b

Description read

[DESCRIPTION-READ] b14f195cd609385a0bd6aad92e897605e1c5e13ec4e17f96aea58ba6853d55df

That sha256 names everything this verdict read: the PR description, plus the 10 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e89169b1c7c1586f076507e99cc195ae972482ae — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e89169b

Verdict parsed from the review's SHA-scoped output markers for commit e89169b1c7c1586f076507e99cc195ae972482ae.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e89169b1c7c1586f076507e99cc195ae972482ae false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e89169b1c7c1586f076507e99cc195ae972482ae and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e89169b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e89169b1c7c1586f076507e99cc195ae972482ae false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 2193 lines, more than the 400 it may approve on its own.

An additive, read-only feature that shows commits, changed lines and PRs (from transcripts) beside each session's spend. I checked the APIs it calls (derive_messages_chained_full_with_keys, publication_hold with expected_keys, derivation_hold raising TranscriptBusy, reentrant RLock/flock file locks, read_commands never returning an empty list, _parse_row_ts). Privacy checks go through the existing guarded read and are repeated under the locks before the response is built. A refused session is dropped from both spend units, and a busy lock or a large set drops every outcome. The cached cost block is never written into, and a window with no delivery returns the same object as before. Merge state goes only to the owner and comes from the existing chip cache. Backend and frontend tests are extensive. I found nothing that blocks the merge.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit e89169b1c7 · 2193 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from 9f0c808 to 593da7d Compare November 15, 2026 01:15
@javenciu

Copy link
Copy Markdown
Contributor Author

All three GPT findings on 9f0c808 are fixed in 593da7d, which is also rebased onto af94fac.

usage_outcome.py:155, a fallback log proves a failed quiet commit (upheld). A quiet commit now counts only when its first join is && and every later join is && or a pipe. git commit -qm fix >/dev/null 2>&1 && git log --oneline -1 || git log --oneline -1 no longer counts, and neither does the same chain followed by ;, & or a newline. test_a_log_that_runs_after_a_failed_commit_further_on_proves_nothing covers those four tails. && git log --oneline | head -1 still counts.

usage_outcome.py:123, the subject is read as the statistics (upheld). The counts now come only from git's statistics line, the first line after the summary apart from git's own Author: and Date: lines. fix parsing of 999 insertions(+) with a statistics line of 2 now reads 2, and a later git show --stat adds nothing (test_the_lines_come_from_the_statistics_line_never_the_subject, test_a_later_commands_statistics_are_not_the_commits).

usage_outcome.py:221, the bounded memo holds unbounded events (upheld). The memo holds at most 2,048 events for one transcript and 16,384 across its 128 entries, dropping the oldest entry first. A transcript with more events is read again on each poll and never held (test_the_memo_holds_a_bounded_number_of_events).

On the previous code, 7 of the 8 new cases fail. The eighth pins git's Author: and Date: lines and passes on both. On this head the two outcome files give 82 passed, and the 21 files that reach the telemetry handler or this module give 777 passed.

UX punchline, the N+ cell and the open tips. The description now shows the Merged column's 2+ cell in both themes, plus the Commits and Merged tips open in both themes.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from 593da7d to 014f96c Compare November 15, 2026 02:39
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 2 on 593da7d: the GPT finding, the two CI failures from this PR and the UX notes are fixed in 014f96c, rebased onto 2092903.

GPT, usage_outcome.py:199: a fallback PR lookup counts as a new creation (upheld). A pull request now counts only when its create is the command's only gh pr or glab mr call, and only && or a pipe follows it. The new cases fail on the previous code and pass now: gh pr create --fill 2>/dev/null || gh pr view --json url --jq .url, the same with ;, a gh pr view before the create, and a newline then gh pr list (test_a_url_another_pull_request_call_may_have_printed_is_not_a_creation). git push && gh pr create --fill | tee pr.txt still counts.

Backend Tests (3.12, 3), test_transcript_derivation_seam.py::TestTheFence. The outcome read is now named in PLUMBING, with its reason: it counts a session's own tool rows into figures beside its spend, and no row leaves the gateway.

Frontend Tests (1), productName.test.ts. The merged tip no longer names the product. The UX notes reworded both tips: the Commits tip now leads with where the figures come from and that they are minimums, without git's output terms. The merged count's marker is now ≥ instead of a trailing +, which the Lines cell beside it uses for lines added. The screenshots are re-taken in both themes.

The remaining red shards are main-side (main's own tests at af94fac51f), as is the timing assertion in test_denied_commands_security_cost.py.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from 014f96c to 92e5af9 Compare November 15, 2026 04:32
@javenciu

Copy link
Copy Markdown
Contributor Author

First principles, span 13c83ecd4e97 (second spelling of the guarded transcript read). Fixed in 92e5af9. attach_session_outcomes now reads rows through the derivation seam, ConversationLog.derive_messages_chained_full_with_keys, and the PLUMBING row for usage_outcome.py is gone from test/test_transcript_derivation_seam.py. TranscriptWithheld is caught: an incognito, temporary or unreadable transcript gives its session no outcome, and its memo entry is dropped. Tightening a line keeps the file's mtime, so the memo alone would have kept serving the earlier figures; each poll now also asks transcript_withholds_derivation before it serves the memo. These fail on 014f96c: TestAttachSessionOutcomes::test_a_restricted_transcript_gives_no_outcome[incognito|temporary], ::test_a_session_tightened_after_it_was_read_loses_its_outcome, ::test_a_refused_read_drops_what_the_memo_held, and TestTheFence::test_every_plain_read_in_the_tree_is_named_as_plumbing once the row is removed.

Opus, span 369e1a8835cc (src/kiro_crew/docs/dashboard.md:91). Fixed in 92e5af9. The doc now says "a ≥ before the merged count means more may be merged", which matches the UI. A grep of the PR's docs, catalogs, context notes and tooltips finds no other + wording of this marker. All 13 catalogs and the context note already used ≥. This is a docs-only fix, so no test covers it.

GPT, span 0052fa7534cd (usage_outcome.py:196, commit previews). Fixed in 92e5af9. git commit options are now parsed: --dry-run and the formats git documents as implying it (--short, --porcelain, --long, and -z/--null through --porcelain, including bundled -qz and any unambiguous long-option prefix such as --dry) mark the call a preview. A preview is left out of the commit count and of the quiet-commit path, and a real commit in the same row still counts. These fail on 014f96c: TestCommitPreviews::test_a_quiet_preview_joined_to_a_log_is_not_a_commit (8 cases, the trigger --dry-run -qm preview >/dev/null && git log --oneline -1 included) and ::test_a_row_whose_only_commit_is_a_preview_counts_nothing (5 cases).

Maintainer rule, no new regular expression in product code. Fixed in 92e5af9. usage_outcome.py no longer imports re. Every pattern is replaced by plain string methods and loops: a one-pass scanner that splits a command at &&, ||, ;, |, & and newlines, reads quotes and backslashes, and treats 2>&1, >&2 and &> as redirections. Words are read without quotes, a sha is checked against a hex character set, and git's summary and statistics lines are parsed with partition and split. grep -n "import re\|re\.compile" on the file prints nothing. All earlier cases in test/metrics/test_usage_outcome.py and test/metrics/test_telemetry_outcome_unchanged.py pass unchanged. New edges fail on 014f96c: TestCommandScanner::test_an_operator_inside_quotes_does_not_split_the_command, ::test_a_command_with_a_quote_left_open_counts_nothing, ::test_an_abbreviated_amend_is_an_amend, and ::test_a_pull_request_body_with_operators_in_quotes_still_counts. A command longer than 16,384 characters, or one that names neither a commit nor a create, is not scanned, so the per-character scan stays bounded. The website files this PR adds contain no RegExp or regex literal outside tests.

Screenshots. The eight PNGs are removed from the commit (git show --stat HEAD | grep -c png prints 0), and the screenshots are attached to the description instead.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 15, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from 92e5af9 to c3fc2ff Compare November 15, 2026 05:52
@javenciu

Copy link
Copy Markdown
Contributor Author

GPT, span 0052fa7534cd (usage_outcome.py:632, a cache hit skipped the withheld check for older transcripts in the same chain); the code change approver's two items say the same and ask for a test. Fixed in c3fc2ff. read_rows now returns the rows together with the chain the seam validated (derive_messages_chained_full_with_keys returns both), and the memo entry stores that chain. Before a memo entry is served, _events_of runs transcript_withholds_derivation on every transcript in it, not only the requested one. If any of them is withheld, the entry is dropped and the session gets no outcome. On the next poll the read goes through the seam again, and the seam refuses the whole chain. This test fails on 92e5af9: TestAttachSessionOutcomes::test_an_older_transcript_of_the_tab_tightened_after_a_read_takes_its_figures[incognito|temporary]. It writes two transcripts that share a tab id, with the commit in the older one, reads them once so the memo is filled, then tightens only the older line. The requested file's own line and mtime do not change, and on the old head the second poll still returned commits: 1.

First principles PASS punchline: "the PR adds a fourth shell-command splitter to the codebase". Fixed in c3fc2ff. This PR's own scanner is deleted. _split_commands now calls the shared reader, security.shell_commands.read_commands(text, nested=False, heredocs=True), and takes each command's words from shell_normalizer._shell_tokens over its argv. That is how acp/batch_guard.py::_program_commands reads a command line. All that stays here is a loop that keeps each command's words and separator. usage_outcome.py has 598 lines, down from 636. Here is what else exists, and why this PR uses read_commands and not one of them:

  • tool_call_title_shell_parse.py:40 tokenize_shell: it returns None on $, ~, >, (, & and backticks. On the 40 largest local transcripts it refuses 99 of the 252 rows this PR counts.
  • trust_patterns.py:282 split_command_segments: it is regex-based, returns segment strings without the operator between them, and gives up on any $(.
  • security/deny_gate.py:1178 _split_segments: a regex split (_CMD_SPLIT_RE) that does not read quoting.
  • security/shell_normalizer_walk.py:1299 _split_push_command_segments: it is quote-aware but returns strings without their operators, and it does not split at a lone &.
  • security/argv_rules.py:125 _split_commands: it works on a token list that is already flat.

None of these returns which operator joins two commands. This PR needs that, because only && or a pipe may follow a create or a quiet commit. All 118 cases in test_usage_outcome.py pass, including every earlier case unchanged. On the same 40 transcripts the old and new versions give the same events, plus 2 more commits. In those two rows, an apostrophe in a here-document message made the old scanner treat the rest of the command as an open quote, so it counted nothing. These tests fail on 92e5af9: TestCommandScanner::test_a_here_document_before_the_commit_is_data and ::test_a_pull_request_body_read_from_a_here_document_counts. The shared reader costs more per command: 5,000 commit rows take about 1 s, against about 0.25 s before. Each transcript is read at most once a minute, on a worker thread, and only rows that name a commit or a create are read. A 16,384-character command takes 0.003-0.12 s in the shapes tried (quotes, $(, parentheses, here-documents, && chains, ${).

Body. It now has an r4 line, the measured counts (118 / 155 / 2,475 passed) and the proof head c3fc2ff. It is 7,955 bytes and keeps all seven attachment URLs.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from c3fc2ff to aad8df4 Compare November 15, 2026 06:44
@javenciu

Copy link
Copy Markdown
Contributor Author

usage_outcome.py:334, a failed quiet commit counts an earlier log entry. Fixed in aad8df4 by removing the quiet-commit inference entirely rather than patching this case: _quiet_commit_events is deleted, and _commit_call no longer reads -q/--quiet. A commit now counts only when its own tool result shows git's [branch sha] subject header, so git log --oneline -1; git commit -qm update >/dev/null 2>&1 && git log --oneline -1 with nothing staged counts nothing. TestQuietCommitsAreNotCounted::test_a_failed_quiet_commit_does_not_count_an_earlier_log_line fails on c3fc2ff (assert (('commit', ..., '1a2b3c4'),) == ()) and passes now, as do the three spellings in test_a_quiet_commit_joined_to_a_log_is_not_counted. The tests that pinned quiet counting were rewritten to header output; every other case is unchanged. docs/dashboard.md, metrics.md, the module docstring and the body now say a quiet commit is not counted. The path is gone, not patched, because this file drew a finding on it in each of the last four rounds. On the same 40 local transcripts the simpler rule counts 17 commits against 154, so commit figures are a floor, and the body says so.

usage_outcome.py:440, retained transcript chains bypass the memo limits. Fixed in aad8df4: a memo entry may keep a chain of at most 64 transcripts (_MEMO_CHAIN_MAX). A longer chain is not memoised at all, so each poll reads it again through the derivation seam, which applies the privacy rule, and it counts toward no memo limit. TestWithOutcomes::test_an_oversized_chain_is_never_memoised fails on c3fc2ff and passes now. test_a_chain_within_the_cap_is_memoised_and_every_member_is_asked pins that a normal chain still memoises and that every member is asked before a hit.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from aad8df4 to 383cb97 Compare November 15, 2026 08:23
@javenciu

Copy link
Copy Markdown
Contributor Author

telemetry.py:1554, revalidate transcript privacy before publishing outcomes. Fixed in 383cb97 as suggested. The scan now records, for each row that got an outcome, its slot, the transcript it was read for and the exact chain the figures came from (with_outcomes(..., contributors=); _events_of returns the chain from the fresh read or from the memo entry). After both units are scanned and both title passes have run, api_telemetry_startup calls publish_session_outcomes as its last step before web.json_response. Off the loop, it enters the existing ConversationLog.publication_hold(transcript, expected_keys=chain) for each contributing chain and drops the outcome of any refused one (restricted line, unreadable line, busy lock, changed chain) from both the credits and the dollar unit. No second privacy check was written. Each chain is held alone and released before the next, so the response never holds two transcripts at once and takes no lock order against merge-back or the consolidator. If the hold cannot be asked at all, every outcome is dropped. TestPublicationRevalidation tightens a contributing transcript between the scan and the response, from inside the first title pass, and asserts the rows carry no outcome. test_a_tightening_after_the_scan_drops_the_outcome_from_both_units[incognito|temporary] and test_an_older_transcript_of_the_tab_tightened_after_the_scan_is_caught fail on aad8df4 and pass now. test_an_untightened_session_keeps_its_outcome_in_both_units passes on both heads.

CI Backend Tests (3.12, 8) and (Windows) (8), test/test_usage_facade.py:69 and :91. Fixed in 383cb97. The usage facade's owners are the usage_*.py siblings, except sibling handlers that read the facade by attribute (usage_series). usage_outcome imported TimeWindow and _parse_row_ts from the usage_store owner, which made it an outside importer of an owner. It now reads both through the facade (from kiro_crew.dashboard.handlers import usage as _usage), as usage_series does, and the test's _NOT_OWNERS names it beside usage_series. With main's test file and the old source, both tests fail. With the new head, all 6 tests in the file pass.

Shard 6, test/metrics/test_kiro_export_destination.py. Not this PR's. The same 3 TestOffOffer tests fail on a clean origin/main worktree at e9d6223 (3 failed, 74 passed, 1 skipped), and this diff does not touch that file or the code it tests.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from 383cb97 to 185bdaf Compare November 15, 2026 09:49
@javenciu

Copy link
Copy Markdown
Contributor Author

usage_outcome.py:602, privacy locks end before outcomes are published. Not changed to a held lock set in this PR, because this response already publishes main's own transcript-derived field with a wider window. The conversation titles in the same Spend response come from the live slot (telemetry.py:1829) or, for a closed tab, from the transcript's metadata line through conversation_log.get_metadata (telemetry.py:1765, run off the loop at :1838). Neither read takes publication_hold or asks the line's privacy, and nothing else in main's telemetry.py does: git show origin/main:src/kiro_crew/dashboard/handlers/telemetry.py | grep -c -i "incognito\|withheld\|publication_hold\|memory_mode" prints 0. With an incognito transcript whose metadata names a title, main at 2e56b3b gives derive_messages -> TranscriptWithheld and _persisted_titles -> {'chat-1-1': 'Billing audit'}, so that title is published at any time, not only inside a narrow gap. Outcomes leave in the same web.json_response (telemetry.py:1567). Before that, each contributing chain is checked through publication_hold, after both title passes (:1562, :1564, :1566), so the only gap left is between a chain's own check and that one synchronous build. Closing that gap for every transcript-derived field in this response (titles included) means holding all of their chains while the response is built. That belongs in the shared publication layer, not in a feature that adds one field. In 185bdaf, test_outcomes_leave_in_the_same_response_build_as_the_titles pins this path: the check runs after both title passes, and the cost block it returns is exactly what the response serializes. Moving the check ahead of the titles fails it and the three TestPublicationRevalidation tightening tests. metrics.md now names the remaining gap instead of saying a tightened session is never published.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 15, 2026
The Telemetry Spend table ranks sessions by credits but cannot say what a
session produced for them, so a long run that merged three changes and one that
only talked cost the same line. The gateway already holds the answer in each
session's own transcript: the shell tool rows carry the command and its output.

Read the 50 costliest sessions' transcripts through the derivation seam
(memoised by file mtime, reread at most once a minute while a live file keeps
changing, and never memoised past a 64-transcript chain) and attach an outcome
to each row that delivered something: commits and changed lines from git's own
summary line (a quiet commit prints none and is not counted), pull requests
from the URL a create command printed unless it says the pull request already
exists, and merged pull requests from the chip-status cache, never refreshed
here. Commands are read by the shared shell reader the batch guard uses, so
quoting is the shell's. Each row's command is read from the call's own
arguments, since a long title can arrive cut short. The owner alone gets merge
figures. Amended commits, commit previews (--dry-run and the formats that imply
it) and commits whose output is mixed with other text are not counted, so every
figure is a minimum; an incognito or temporary session, or one with such a
transcript earlier in its tab, gives none. The table gains five columns once a
row has an outcome, and a window with none returns the payload as it was.
@javenciu
javenciu force-pushed the feat/session-cost-per-outcome branch from 185bdaf to e89169b Compare November 15, 2026 10:50
@javenciu

Copy link
Copy Markdown
Contributor Author

usage_outcome.py:602, keep privacy holds through response construction. Fixed in e89169b. The per-chain check and the handler's own web.json_response are replaced by respond_with_session_outcomes, which runs off the loop and returns the response the handler sends. It takes the lock stems of every contributing chain in one sorted acquire (the same order as every single acquire), then enters each chain's publication_hold in sorted transcript order through an ExitStack. Those locks are already held at that point, so the second step only revalidates each chain. A refused chain's outcome is dropped from both spend units, and the response bytes are built before any hold is let go. No hold spans an await. A busy lock, or more than 256 transcripts across the chains (_PUBLICATION_HOLD_MAX), sends no outcome rather than holding an unbounded set. test_a_checked_chain_stays_held_while_the_next_one_is_awaited uses your trigger: two contributors, and while the response is entering the second chain's hold, another thread tightens the first if its locks are free. On 185bdaf the first chain had already been let go, so the tightening landed and its outcome was still sent (AssertionError: assert not (True and 'outcome' in {...})). Now the first chain is still held, so the tightening cannot land before the response is built. test_more_chains_than_the_hold_bound_send_no_outcome pins the bound. The round-7 ordering test now checks that the response built under the holds is the one returned, after both title passes. metrics.md describes the held build.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/).

Review result on this head: no blocking finding.

  • Transcript rows are read only through the derivation seam. A session made private between the scan and the response loses its outcome in both units.
  • Merge figures go only to the owner, through the same check the chips use.
  • All transcript and lock work runs off the event loop.
  • A window with no delivery returns the same payload as main.
  • The two new test files pass on this head (117 tests).

The privacy handling is careful and well tested from both sides, through the real route and a real second thread.

Optional:

  • usage_outcome.py:664 and :690: the fallbacks that drop every outcome on TranscriptBusy or on an exception have no test. If either were changed to return the raw payload, all tests would still pass. Two route tests would pin them: one that makes derivation_hold raise TranscriptBusy, and one that makes the off-loop call raise.
  • src/kiro_crew/docs/dashboard.md:91 says commits in mixed output are missed. _commit_events scans every line of the output, so those commits are counted. Only their changed-lines figure can drop to 0. Consider rewording the sentence to match the code.
  • Open fix(dashboard): show the GitHub PRs a chat owns on its card, not every PR it mentions #35327 adds its own detector for "this chat created PR X" from gh pr create tool rows. If both PRs merge, a card chip and a Spend row could disagree for the same session. Whichever PR lands second could import the other's detector and name the merge order in its body.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Read the whole diff on this head: usage_outcome.py, the telemetry.py hook, both test files, the frontend, and every locale.

  • Additive read API: /api/telemetry/startup only gains an optional outcome per row and outcome_scan on the block, and with_outcomes returns the same object when nothing was delivered (pinned by test_telemetry_outcome_unchanged.py, which passes on main too). The only frontend reader is the Spend table, and it shows the five columns only when some row has an outcome. Every failure path (no log, unreadable transcript, busy holds, over 256 transcripts) drops outcomes and answers as main does. Nothing is refused.
  • Privacy: rows come through derive_messages_chained_full_with_keys, so incognito/temporary transcripts give nothing; the memo re-asks every chain member before a hit is served, and the response is built under publication_hold, so a session tightened after the scan is dropped from both units. Merge figures come from the chip cache, owner only, never refreshed here.
  • Parsing reuses the shared shell reader (read_commands, _shell_tokens) and parse_source_url. No regex in product code (the RegExp is in the vitest file).
  • The memoised cost block is copied, never written into.

Non-blocking:

  • A live session's transcript is reread through the derivation seam at most once a minute while the Spend tab is open. For a chain of hundreds of MB, that is one worker thread for seconds every minute. Fine as bounded, but worth a size cap later if the System page shows it.
  • The response now enters derivation_hold over every contributing chain's lock stems while it builds the JSON. The build is short and busy locks drop the outcome rather than wait, so I see no write stall, but a comment saying the acquire is non-blocking would help the next reader.

Not checked: no local run; I relied on the 97 green checks.

@javenciu
javenciu merged commit 3b1ca41 into main Nov 15, 2026
99 checks passed
@javenciu
javenciu deleted the feat/session-cost-per-outcome branch November 15, 2026 17:12
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants