Repository navigation
fix(crew-log): write a turn's tokens and credits only when reported - #16533
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that kiro-cli turns which report credits but no token counts now show a dash for tokens in the panel, while still showing their credits. [DESIGN-REVIEWED] 8e31254 Description read[DESCRIPTION-READ] 12b3924556b7a9e4e602768104d85d47ce2db2516dc4e6aa82c9e850d88dacd5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that #15880's triage really names the credits writer, as the description quotes. The What this change shipsInventory (6 items) — 6 justifiedIntent: when a provider never reported a count, the crew-log usage panel should show a dash instead of a measured
[FIRST-PRINCIPLES-REVIEWED] 8e31254 Description read[DESCRIPTION-READ] 5cacea2d66096c0c1bbd52fa3dd82b3ba4c89da9b6ce03cda8805ee6ec27ff36 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Dispositions for head First Principles — CONCERNS: "dropping zero
Design Review — "a provider that really bills a turn at First Principles (earlier head) — two docstrings stated the retired absence-marks-synthesized rule. Fixed: Design Review (head First Principles (head First Principles (head Design Review (head |
5331c0a to
7d954fa
Compare
|
Evidence for the PR body (moved here so the body's Every writer of a turn record's tokens block, every reader of
|
| site | role | verdict |
|---|---|---|
emit.py on_turn_completed |
writer of turn/completed.tokens and .credits |
defect, fixed (both fields) |
emit.py on_turn_failed |
writer of the synthesized turn/completed |
not a copy: writes neither key (test_the_failed_turn_closer_validates_without_credits_or_tokens) |
emit.py on_background_completed |
writer of background/completed.tokens |
not a copy: already if any(tokens.values()), sparse shape per its schema |
emit.py on_subagent_completed / on_subagent_failed |
credits writers | not a copy: if credits > 0 and math.isfinite(credits), never write tokens |
dashboard/chat_runner.py terminal arm (_crew_log_terminal) |
the only caller of on_turn_completed |
passes event.usage.* through; TurnUsage cannot distinguish absent from zero, so the writer is the right seam |
projection.py _usage_step |
reader of turn/completed.tokens, writer of tokens_turns |
defect, fixed (counts only when a dimension is above zero) |
projection.py _usage_render |
serves tokens_turns as turns.tokens_reported |
correct pass-through, unchanged |
projection.py _bill_credits |
reader of .credits, writer of credits_by_source.turn.reported |
looks like a copy and is not: a present 0.0 is counted on purpose ("Zero passes deliberately"), and a stored 0.0 cannot be told from a turn billed at nothing. Left alone; pinned by test_usage_still_counts_a_recorded_zero_credit_charge_as_reported |
website/src/pages/chat/CrewLogPanel.tsx:358,367-370,836 |
readers of turns.tokens_reported via measured() and the collapsed summary |
correct: dashes when the reporter count is 0; no change needed |
crew_main_contract.py _tokens |
reads usage.tokens.total vs status.turns_completed |
not a reader of tokens_reported; still correct after this fix; left in place |
website/src/pages/OverviewPage.tsx, overview/UsageTab.tsx |
read data.tokens.total |
not readers of the crew-log fold: they read the provider usage store (providerUsageQuery) |
issue_radar/backend/pipeline_fold.py, pipeline-conductor/scripts/credit_spend.py |
read credits off rows |
read usage-store _type == "tokens" rows, not crew-log entries |
Mutation matrix
Each mutation applied alone to the fix, the new tests run by name, source files restored byte-identical after (cmp verified):
- M1 writer always writes
tokens(base behaviour) -> 2 red:omits_tokens_the_provider_never_reported,measured_turn_closer_validates_with_credits_and_no_tokens - M2 writer always writes
credits(base behaviour) -> 3 red:omits_an_unbilled_credits_charge,drops_a_charge_that_is_not_a_finite_positive_number,a_provider_that_bills_in_tokens_alone_folds_as_credits_unreported - M3 writer emits a sparse block (
background/completed's shape) -> 2 red:keeps_all_four_dimensions_once_any_is_measured,omits_an_unbilled_credits_charge - M4 fold counts any dict (base behaviour) -> 2 red:
does_not_count_an_all_zero_tokens_block_as_reported,counts_a_block_reported_once_any_dimension_is_above_zero - M5 fold counts only when
inputis above zero -> 1 red:counts_a_block_reported_once_any_dimension_is_above_zero - M6
usagestate_versionleft at BASE+9 -> 1 red:test_changing_what_a_fold_stores_forces_the_savepoint_version_to_move
Control (unmutated): all 9 named tests green. The existing test_every_emitted_type_matches_the_documented_shape encoded the old behaviour (expected credits/tokens on a closer given no usage); its turn closer now carries a measured usage so the shape row is exercised as documented.
…er reported them `on_turn_completed` wrote `data["tokens"]` on every closer with all four dimensions defaulted to 0, and `data["credits"]` as `float(credits)` whatever the value. `TurnUsage` zero-fills every dimension a provider does not report, so a backend that sent no counts landed as a block of four zeros -- an absence written as a measurement. `_usage_step` then counted any `tokens` dict as a reporting turn, so a session with 4 completed turns and 1.49 credits billed folded to `turns.tokens_reported: 4` beside `tokens.total: 0`, and the CrewLogPanel's `measured()` printed a measured `0 tokens` next to a real bill. Writer: `tokens` is written only when some count is above zero, and then with all four dimensions (the `turn/completed` schema requires each member, unlike `background/completed`'s sparse mapping); `credits` only when positive and finite -- the guard both subagent closers already use. Fold: `tokens_turns` counts a block only when some dimension is above zero, so logs already on disk read correctly too. The usage fold's `state_version` moves (BASE+10): a counting fix that keeps every key must retire savepoints written under the old meaning, per the module's own rule. `credits_reported` is deliberately left alone: a recorded 0.0 cannot be told from a turn billed at nothing, and `_bill_credits` passes zero on purpose. Closes #15880
7d954fa to
8e31254
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Reviewed: turn/completed now writes credits only when positive and finite and tokens only when some dimension is above zero (then all four), since TurnUsage zero-fills unreported dimensions and a written zero was read as a measurement. The usage fold counts an on-disk four-zero block as unreported (sums unchanged), and the fold state version is bumped so old savepoints rebuild. Entry-type notes, specs and reference docs say readers must default both fields. Tests cover emit, projection and types. CI green.
Problem / Motivation
Goal: The crew-log panel shows a dash, not a measured
0 tokens, for a session whose provider never reported token counts.Every
turn/completedcrew-log record carried atokensblock, even when the backend sent no counts.on_turn_completedwrotedata["tokens"]with all four dimensions defaulted to0, anddata["credits"] = float(credits)whatever the value.TurnUsagezero-fills every dimension a provider does not report, so a backend that sent no counts landed as four zeros -- an absence written as a measurement._usage_stepthen counted anytokensdict as a reporting turn. A real session with 4 completed turns and 1.49 credits billed folded toturns.tokens_reported: 4besidetokens.total: 0, andCrewLogPanel'smeasured(tokens.total, turns.tokens_reported)printed0 tokensnext to a real bill.Why it matters
A completed turn cannot cost zero tokens, so the
0is a lie about a measurement nobody made, shown on the most-read line of the usage fold (the collapsed header) and on every token row of the panel. Anyone reading a session's cost sees a bill with no tokens behind it and cannot tell "unreported" from "free". Every sibling writer (background/completed,subagent/completed,subagent/failed) already omits an unreported count, and thebackground/completeddocstring described the rule as "followingturn/completed" -- whichturn/completeddid not do, so the record disagreed with itself about what an unreported count looks like.Not a goal
credits: 0.0on a closer already on disk. A recorded zero cannot be told from a turn a provider genuinely billed at nothing, and_bill_creditslets zero through on purpose; the writer fix stops new ones. Pinned bytest_usage_still_counts_a_recorded_zero_credit_charge_as_reported.crew_main_contract.py_tokensworkaround (a zero total after a completed turn reads "no tokens reported"). It stays correct after this change and is removable separately.CrewLogPanel.tsxalready dashes when the reporter count is 0; it trusted a count the fold inflated.turn/completedschema's four-membertokensshape tobackground/completed's sparse one.What changed (motivation → approach → change)
Symptom:
turns.tokens_reportedequalsturns.completedon sessions whose provider sent no counts, so a measured0 tokenssits beside a real bill.Root cause, one site per layer. Writer:
emit.pyon_turn_completedwrotetokensandcreditsunconditionally. Fold:projection.py_usage_stepcounted anytokensdict as a report. The only caller (chat_runner.py, the terminal arm) passesevent.usage.*through, andTurnUsagecannot distinguish absent from zero, so the writer is the right seam. The same writer has the same fault oncredits, which #15880's triage names ("Credits, same writer ... a harness that does not bill in credits shows a measured0credits per turn"):TurnUsage's contract (acp/types.py:652-654) is "each provider fills the dimensions it bills in and leaves the rest at 0 ... consumers read whichever is non-zero", so aclaude_code/bedrockturn arrives with real token counts andcredits=0.0, every such turn is counted incredits_reported, and the credits tile prints a measured0for a session never billed in credits. The chat footer's gateusage_has_billing(llm_helpers.py:2010) already reads that 0.0 as unbilled; the crew log now agrees with it. This differs from the "Not a goal" bullet about a0.0already on disk: at the writer seam the contract defines the zero as unreported; a stored zero has lost that context.Change:
tokensonly when some count is above zero, then with all four dimensions (the schema's_TOKEN_FIELDSrequires each member; a zero inside a reported block is a real zero).creditsonly whencharge > 0 and math.isfinite(charge)-- the subagent closers' guard, verbatim.tokens_turnsincrements only when some dimension is above zero, so logs already on disk fold correctly; sums unchanged.usagestate_versionBASE+9 → BASE+10 per the module's rule (a counting fix that keeps every key must retire savepoints written under the old meaning); pin updated, digest unchanged.entry_types.pyfield notes;session-types.mdrows, invariants, reader hint;crew-log-emitter.mdrow and the sentence that made absence the synthesized-closer marker (stop_reason: "failed"is);crew-log-projection.md; theon_turn_failedand projection module docstrings.The per-site enumeration of every writer of a turn record's tokens block and every reader of
tokens_reportedis in a PR comment, with the full mutation matrix.Backwards compatibility
Compatible: nothing main accepts is rejected. The
turn/completedschema already declarescreditsandtokensoptional (the crash-repair closer omits both), so a closer without them validates today; a presenttokenskeeps the four-member shape every reader of a present block expects. Entries on disk are not rewritten -- the fold reads an all-zero block as unreported, the only new interpretation, and theusagestate_versionbump retires that fold's savepoints so no resumed state carries the old count.Removes nothing: no key, kind, route or public function is removed or renamed; two fields that were always present on a measured closer are present only when the provider reported them, and every reader of them (enumerated in the PR comment) already reads them with a default.
Tests
Changed test files, run by path with
-n0from the worktree (imports verified from the worktree):test_crew_log_emit.py312,test_crew_log_projection.py175,test_crew_log_types.py216,test_crew_log_checkpoint.py56,test_docs_crew_log_reference.py250 passed.Red on a pristine
origin/maincheckout (imports verified from it), green here -- 7 tests:test_turn_completed_omits_tokens_the_provider_never_reportedtest_turn_completed_omits_an_unbilled_credits_chargetest_turn_completed_drops_a_charge_that_is_not_a_finite_positive_numbertest_a_provider_that_bills_in_tokens_alone_folds_as_credits_unreported-- writer to fold, threeclaude_code-shaped closers; base foldscredits_reported: 3test_usage_does_not_count_an_all_zero_tokens_block_as_reported-- the issue's Seen scenario; base foldstokens_reported: 4test_usage_counts_a_block_reported_once_any_dimension_is_above_zerotest_the_measured_turn_closer_validates_with_credits_and_no_tokensPins, green on base deliberately:
test_turn_completed_keeps_all_four_dimensions_once_any_is_measured,test_usage_still_counts_a_recorded_zero_credit_charge_as_reported.Mutation matrix, six single mutations (writer always writes tokens; always writes credits; writes a sparse block; fold counts any dict; fold counts only
input;state_versionnot bumped): each reddens 1-3 named tests, none passes clean; details in the PR comment.black,isort,flake8,mypyandscripts/check_comment_history.pyclean.Manual verification
N/A — unit coverage sufficient: writer and fold are exercised end to end (the fold tests append real entries through
CrewLogand fold them; the writer-to-fold test folds the emitter's own output), and the only frontend reader already dashes ontokens_reported: 0.Related Issues
Closes #15880
Pattern harvest
Rule candidate: review-prompt
Pattern: "a writer records a zero-filled default from a contract that zero-fills unreported dimensions, so a downstream
reportedcount treats absence as a measurement" -- check every emitter of aTurnUsage-derived field against the sibling that already guards on non-zero.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement