Skip to content

fix(crew-log): write a turn's tokens and credits only when reported - #16533

Merged
NicholasRBowers merged 1 commit into
mainfrom
fix/turn-tokens-unreported-15880
Oct 4, 2026
Merged

NicholasRBowers merged 1 commit into
mainfrom
fix/turn-tokens-unreported-15880

Conversation

@NicholasRBowers

@NicholasRBowers NicholasRBowers commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The crew-log panel shows a dash, not a measured 0 tokens, for a session whose provider never reported token counts.

Every turn/completed crew-log record carried a tokens block, even when the backend sent no counts. on_turn_completed wrote data["tokens"] with all four dimensions defaulted to 0, and data["credits"] = float(credits) whatever the value. TurnUsage zero-fills every dimension a provider does not report, so a backend that sent no counts landed as four zeros -- an absence written as a measurement. _usage_step then counted any tokens dict as a reporting turn. A real session with 4 completed turns and 1.49 credits billed folded to turns.tokens_reported: 4 beside tokens.total: 0, and CrewLogPanel's measured(tokens.total, turns.tokens_reported) printed 0 tokens next to a real bill.

Why it matters

A completed turn cannot cost zero tokens, so the 0 is a lie about a measurement nobody made, shown on the most-read line of the usage fold (the collapsed header) and on every token row of the panel. Anyone reading a session's cost sees a bill with no tokens behind it and cannot tell "unreported" from "free". Every sibling writer (background/completed, subagent/completed, subagent/failed) already omits an unreported count, and the background/completed docstring described the rule as "following turn/completed" -- which turn/completed did not do, so the record disagreed with itself about what an unreported count looks like.

Not a goal

  • Changing how the fold counts a present credits: 0.0 on a closer already on disk. A recorded zero cannot be told from a turn a provider genuinely billed at nothing, and _bill_credits lets zero through on purpose; the writer fix stops new ones. Pinned by test_usage_still_counts_a_recorded_zero_credit_charge_as_reported.
  • Removing the crew_main_contract.py _tokens workaround (a zero total after a completed turn reads "no tokens reported"). It stays correct after this change and is removable separately.
  • Any frontend change. CrewLogPanel.tsx already dashes when the reporter count is 0; it trusted a count the fold inflated.
  • Changing the turn/completed schema's four-member tokens shape to background/completed's sparse one.

What changed (motivation → approach → change)

Symptom: turns.tokens_reported equals turns.completed on sessions whose provider sent no counts, so a measured 0 tokens sits beside a real bill.

Root cause, one site per layer. Writer: emit.py on_turn_completed wrote tokens and credits unconditionally. Fold: projection.py _usage_step counted any tokens dict as a report. The only caller (chat_runner.py, the terminal arm) passes event.usage.* through, and TurnUsage cannot distinguish absent from zero, so the writer is the right seam. The same writer has the same fault on credits, which #15880's triage names ("Credits, same writer ... a harness that does not bill in credits shows a measured 0 credits per turn"): TurnUsage's contract (acp/types.py:652-654) is "each provider fills the dimensions it bills in and leaves the rest at 0 ... consumers read whichever is non-zero", so a claude_code/bedrock turn arrives with real token counts and credits=0.0, every such turn is counted in credits_reported, and the credits tile prints a measured 0 for a session never billed in credits. The chat footer's gate usage_has_billing (llm_helpers.py:2010) already reads that 0.0 as unbilled; the crew log now agrees with it. This differs from the "Not a goal" bullet about a 0.0 already on disk: at the writer seam the contract defines the zero as unreported; a stored zero has lost that context.

Change:

  • Writer: tokens only when some count is above zero, then with all four dimensions (the schema's _TOKEN_FIELDS requires each member; a zero inside a reported block is a real zero). credits only when charge > 0 and math.isfinite(charge) -- the subagent closers' guard, verbatim.
  • Fold: tokens_turns increments only when some dimension is above zero, so logs already on disk fold correctly; sums unchanged. usage state_version BASE+9 → BASE+10 per the module's rule (a counting fix that keeps every key must retire savepoints written under the old meaning); pin updated, digest unchanged.
  • Schema notes and docs: entry_types.py field notes; session-types.md rows, invariants, reader hint; crew-log-emitter.md row and the sentence that made absence the synthesized-closer marker (stop_reason: "failed" is); crew-log-projection.md; the on_turn_failed and projection module docstrings.

The per-site enumeration of every writer of a turn record's tokens block and every reader of tokens_reported is in a PR comment, with the full mutation matrix.

Backwards compatibility

Compatible: nothing main accepts is rejected. The turn/completed schema already declares credits and tokens optional (the crash-repair closer omits both), so a closer without them validates today; a present tokens keeps the four-member shape every reader of a present block expects. Entries on disk are not rewritten -- the fold reads an all-zero block as unreported, the only new interpretation, and the usage state_version bump retires that fold's savepoints so no resumed state carries the old count.

Removes nothing: no key, kind, route or public function is removed or renamed; two fields that were always present on a measured closer are present only when the provider reported them, and every reader of them (enumerated in the PR comment) already reads them with a default.

Tests

Changed test files, run by path with -n0 from the worktree (imports verified from the worktree): test_crew_log_emit.py 312, test_crew_log_projection.py 175, test_crew_log_types.py 216, test_crew_log_checkpoint.py 56, test_docs_crew_log_reference.py 250 passed.

Red on a pristine origin/main checkout (imports verified from it), green here -- 7 tests:

  • test_turn_completed_omits_tokens_the_provider_never_reported
  • test_turn_completed_omits_an_unbilled_credits_charge
  • test_turn_completed_drops_a_charge_that_is_not_a_finite_positive_number
  • test_a_provider_that_bills_in_tokens_alone_folds_as_credits_unreported -- writer to fold, three claude_code-shaped closers; base folds credits_reported: 3
  • test_usage_does_not_count_an_all_zero_tokens_block_as_reported -- the issue's Seen scenario; base folds tokens_reported: 4
  • test_usage_counts_a_block_reported_once_any_dimension_is_above_zero
  • test_the_measured_turn_closer_validates_with_credits_and_no_tokens

Pins, green on base deliberately: test_turn_completed_keeps_all_four_dimensions_once_any_is_measured, test_usage_still_counts_a_recorded_zero_credit_charge_as_reported.

Mutation matrix, six single mutations (writer always writes tokens; always writes credits; writes a sparse block; fold counts any dict; fold counts only input; state_version not bumped): each reddens 1-3 named tests, none passes clean; details in the PR comment. black, isort, flake8, mypy and scripts/check_comment_history.py clean.

Manual verification

N/A — unit coverage sufficient: writer and fold are exercised end to end (the fold tests append real entries through CrewLog and fold them; the writer-to-fold test folds the emitter's own output), and the only frontend reader already dashes on tokens_reported: 0.

Related Issues

Closes #15880

Pattern harvest

Rule candidate: review-prompt
Pattern: "a writer records a zero-filled default from a contract that zero-fills unreported dimensions, so a downstream reported count treats absence as a measurement" -- check every emitter of a TurnUsage-derived field against the sibling that already guards on non-zero.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8e312548912d6308c944211e74e4f229a6d5e584 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that kiro-cli turns which report credits but no token counts now show a dash for tokens in the panel, while still showing their credits.

[DESIGN-REVIEWED] 8e31254

Description read

[DESCRIPTION-READ] 12b3924556b7a9e4e602768104d85d47ce2db2516dc4e6aa82c9e850d88dacd5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8e312548912d6308c944211e74e4f229a6d5e584 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that #15880's triage really names the credits writer, as the description quotes. The credits guard rests on that issue alone, not on the token defect.

What this change ships

Inventory (6 items) — 6 justified

Intent: when a provider never reported a count, the crew-log usage panel should show a dash instead of a measured 0 tokens or 0 credits. FIX (Closes #15880).

  1. A turn closer whose provider reported no token counts no longer writes a block of four zeros. — justified
  2. A turn closer with a 0.0 charge no longer writes credits, so a session billed only in tokens shows its credits as unreported. — justified
  3. A NaN, infinite or negative charge is dropped instead of written. It uses the same guard as the subagent closers at emit.py:5630 and emit.py:5666. — justified
  4. Logs already on disk: an all-zero tokens block no longer counts toward tokens_reported. — justified
  5. The usage fold savepoints are retired (state_version +9 → +10), so a resumed fold cannot carry the inflated count. — justified
  6. The schema notes, session-types.md, the emitter spec and the projection spec now state the "absent means unreported" rule. — justified

[FIRST-PRINCIPLES-REVIEWED] 8e31254

Description read

[DESCRIPTION-READ] 5cacea2d66096c0c1bbd52fa3dd82b3ba4c89da9b6ce03cda8805ee6ec27ff36

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8e312548912d6308c944211e74e4f229a6d5e584 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8e31254

Verdict parsed from the review's SHA-scoped output markers for commit 8e312548912d6308c944211e74e4f229a6d5e584.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8e312548912d6308c944211e74e4f229a6d5e584: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8e312548912d6308c944211e74e4f229a6d5e584 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8e31254

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8e312548912d6308c944211e74e4f229a6d5e584: <one-sentence reason>

@NicholasRBowers

NicholasRBowers commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Dispositions for head 8e3125489 (advisory notes; this comment is edited in place).

First Principles — CONCERNS: "dropping zero credits from the writer is outside the tokens-only Goal; its only stated support is symmetry; it undercuts the 'a recorded zero cannot be told from a free turn' rationale." Rebutted with the evidence the verdict names as clearing it -- both a linked issue and a test failing on base.

  • Linked issue: Turn records always write an all-zero tokens block, so usage.turns.tokens_reported counts unreported turns and CrewLogPanel shows 0 tokens beside a real bill #15880 (the closing reference) diagnoses this field explicitly in its triage: "Credits, same writer. :4103 writes data["credits"] = float(credits) unconditionally too ... A harness that does not bill in credits therefore shows a measured 0 credits per turn," and its prescribed fix step 1 is "Write credits only when positive and finite." The issue's title names the token symptom; its diagnosis scopes both fields of the one writer.
  • Test failing on base: test_a_provider_that_bills_in_tokens_alone_folds_as_credits_unreported folds the writer's own output for three claude_code-shaped closers (real token counts, credits=0.0, which is exactly what chat_runner.py hands the closer for a provider that bills in tokens). On a pristine origin/main checkout it fails with credits_reported == 3; the panel's credits tile then prints a measured 0 for a session never billed in credits. On this head it folds credits_reported: 0, tokens_reported: 3.
  • Not symmetry: the guard's support is the contract. TurnUsage's docstring (src/kiro_crew/acp/types.py:652-654): "Each provider fills the dimensions it bills in and leaves the rest at 0 ... kiro (acp) fills credits. Consumers read whichever is non-zero." The chat footer's persist gate usage_has_billing (llm_helpers.py:2010) already reads that 0.0 as unbilled; the crew log now agrees with the footer instead of contradicting it.
  • Not a contradiction with "Not a goal": that bullet is about a 0.0 ALREADY ON DISK, where the fold has nothing but the stored number and cannot recover whether the provider bills in credits. At the writer seam the contract defines the zero as "not billed here", so the writer can act on it and the fold cannot. The PR body's "What changed" now states this distinction.

Design Review — "a provider that really bills a turn at 0.0 credits now shows a dash on new turns." Accepted as designed; same contract as above. At this seam a 0.0 from a provider that does not bill in credits and a 0.0 from a genuinely free Kiro turn are the same bytes (float(getattr(stats, "credits", 0.0) or 0.0), llm_helpers.py:2004), so writing it as a measurement was the defect. Every sibling writer already takes this posture (subagent/completed, subagent/failed: credits > 0 and math.isfinite(credits); background/completed: if credits:). The panel's credits tile dashes only when NO source reported (creditsReportedBySource), so a session whose background or subagent calls were billed still shows its total.

First Principles (earlier head) — two docstrings stated the retired absence-marks-synthesized rule. Fixed: on_turn_failed says absence does not tell it from a provider-reported closer and stop_reason does; the projection module docstring states the presence rule as it is. grep -rn 'provider-reported one\|synthesized closer from' src/kiro_crew/crew_log/ returns only that corrected negation.

Design Review (head 7d954faf2) — "## Backwards compatibility is cut off by the 8000-byte truncation." Fixed on this head: the body is restructured so every section through ## Checklist sits inside the first 8000 bytes (## Backwards compatibility at byte 4824, ## Tests at 5647, measured on the saved body); the per-site enumeration and the full mutation matrix moved to the evidence comment above. Frozen sections are byte-identical.

First Principles (head 7d954faf2) — "check that no external crew-log reader still uses missing credits/tokens to spot a synthesized closer, which the old reference doc promised." No schema change makes that reader break: both fields were already optional in the declared turn/completed shape (the crash-repair closer omits both, on main today), so an external reader already had to handle their absence. What changes is one inference, and the reference page never offered it as the discriminator on its own: the same page's stop_reason row says "failed is the in-process failed close; interrupted is written only by crash repair; otherwise the provider's terminal reason", which is the documented way to tell a synthesized closer and is unchanged. In this repository no reader draws the inference (measured: grep -rn 'synthesi' src/kiro_crew/crew_log/*.py src/kiro_crew/dashboard/*.py finds only docstrings; the frontend reads the fold's *_reported counts, never a closer's fields). The rewritten reader hint tells readers to use turns.credits_reported / turns.tokens_reported and to read a four-zero legacy block as absent. External readers cannot be enumerated from here; the reference doc is the contract they read, and it now states the rule they need.

First Principles (head 8e3125489) — "check that #15880's triage really names the credits writer, as the description quotes." It does, verbatim, in the triage comment at #15880 (comment): "Credits, same writer. :4103 writes data[\"credits\"] = float(credits) unconditionally too, and _turn_credits defaults to 0.0 ... _bill_credits returns None only when the key is absent ... so an unbilled turn still increments credits_by_source.turn.reported, which is what usage.turns.credits_reported serves. A harness that does not bill in credits therefore shows a measured 0 credits per turn." Its "Fix, mechanically derived" step 1 reads: "give on_turn_completed the same two guards as on_background_completed. Write credits only when positive and finite, and write tokens only when some dimension is non-zero". The later triage comment on the same issue repeats the credits guard as part of the fix. The guard also rests on a test red on base, not on the issue alone: test_a_provider_that_bills_in_tokens_alone_folds_as_credits_unreported (credits_reported == 3 on origin/main).

Design Review (head 8e3125489) — "check that kiro-cli turns which report credits but no token counts now show a dash for tokens while still showing their credits." Measured by three tests on this head. Writer: test_turn_completed_omits_tokens_the_provider_never_reported (credits 1.49 kept, tokens absent) and test_the_measured_turn_closer_validates_with_credits_and_no_tokens (the shape validates). Fold: test_usage_does_not_count_an_all_zero_tokens_block_as_reported is the issue's Seen session -- four Kiro turns at 0.3725 credits with four-zero token blocks -- and asserts credits_reported: 4, credits: 1.49, tokens_reported: 0. The panel reads exactly those fields: the credits tile gates on creditsReportedBySource (4, so it prints 1.49) and every token row on turns.tokens_reported (0, so measured() prints a dash) -- CrewLogPanel.tsx:142,357-370,836, unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@NicholasRBowers
NicholasRBowers force-pushed the fix/turn-tokens-unreported-15880 branch 2 times, most recently from 5331c0a to 7d954fa Compare October 3, 2026 15:31
@NicholasRBowers

Copy link
Copy Markdown
Contributor Author

Evidence for the PR body (moved here so the body's ## Backwards compatibility and ## Tests sit inside the 8000-byte window the review lanes read).

Every writer of a turn record's tokens block, every reader of tokens_reported

site role verdict
emit.py on_turn_completed writer of turn/completed.tokens and .credits defect, fixed (both fields)
emit.py on_turn_failed writer of the synthesized turn/completed not a copy: writes neither key (test_the_failed_turn_closer_validates_without_credits_or_tokens)
emit.py on_background_completed writer of background/completed.tokens not a copy: already if any(tokens.values()), sparse shape per its schema
emit.py on_subagent_completed / on_subagent_failed credits writers not a copy: if credits > 0 and math.isfinite(credits), never write tokens
dashboard/chat_runner.py terminal arm (_crew_log_terminal) the only caller of on_turn_completed passes event.usage.* through; TurnUsage cannot distinguish absent from zero, so the writer is the right seam
projection.py _usage_step reader of turn/completed.tokens, writer of tokens_turns defect, fixed (counts only when a dimension is above zero)
projection.py _usage_render serves tokens_turns as turns.tokens_reported correct pass-through, unchanged
projection.py _bill_credits reader of .credits, writer of credits_by_source.turn.reported looks like a copy and is not: a present 0.0 is counted on purpose ("Zero passes deliberately"), and a stored 0.0 cannot be told from a turn billed at nothing. Left alone; pinned by test_usage_still_counts_a_recorded_zero_credit_charge_as_reported
website/src/pages/chat/CrewLogPanel.tsx:358,367-370,836 readers of turns.tokens_reported via measured() and the collapsed summary correct: dashes when the reporter count is 0; no change needed
crew_main_contract.py _tokens reads usage.tokens.total vs status.turns_completed not a reader of tokens_reported; still correct after this fix; left in place
website/src/pages/OverviewPage.tsx, overview/UsageTab.tsx read data.tokens.total not readers of the crew-log fold: they read the provider usage store (providerUsageQuery)
issue_radar/backend/pipeline_fold.py, pipeline-conductor/scripts/credit_spend.py read credits off rows read usage-store _type == "tokens" rows, not crew-log entries

Mutation matrix

Each mutation applied alone to the fix, the new tests run by name, source files restored byte-identical after (cmp verified):

  • M1 writer always writes tokens (base behaviour) -> 2 red: omits_tokens_the_provider_never_reported, measured_turn_closer_validates_with_credits_and_no_tokens
  • M2 writer always writes credits (base behaviour) -> 3 red: omits_an_unbilled_credits_charge, drops_a_charge_that_is_not_a_finite_positive_number, a_provider_that_bills_in_tokens_alone_folds_as_credits_unreported
  • M3 writer emits a sparse block (background/completed's shape) -> 2 red: keeps_all_four_dimensions_once_any_is_measured, omits_an_unbilled_credits_charge
  • M4 fold counts any dict (base behaviour) -> 2 red: does_not_count_an_all_zero_tokens_block_as_reported, counts_a_block_reported_once_any_dimension_is_above_zero
  • M5 fold counts only when input is above zero -> 1 red: counts_a_block_reported_once_any_dimension_is_above_zero
  • M6 usage state_version left at BASE+9 -> 1 red: test_changing_what_a_fold_stores_forces_the_savepoint_version_to_move

Control (unmutated): all 9 named tests green. The existing test_every_emitted_type_matches_the_documented_shape encoded the old behaviour (expected credits/tokens on a closer given no usage); its turn closer now carries a measured usage so the shape row is exercised as documented.

…er reported them

`on_turn_completed` wrote `data["tokens"]` on every closer with all four
dimensions defaulted to 0, and `data["credits"]` as `float(credits)` whatever
the value. `TurnUsage` zero-fills every dimension a provider does not report,
so a backend that sent no counts landed as a block of four zeros -- an absence
written as a measurement. `_usage_step` then counted any `tokens` dict as a
reporting turn, so a session with 4 completed turns and 1.49 credits billed
folded to `turns.tokens_reported: 4` beside `tokens.total: 0`, and the
CrewLogPanel's `measured()` printed a measured `0 tokens` next to a real bill.

Writer: `tokens` is written only when some count is above zero, and then with
all four dimensions (the `turn/completed` schema requires each member, unlike
`background/completed`'s sparse mapping); `credits` only when positive and
finite -- the guard both subagent closers already use. Fold: `tokens_turns`
counts a block only when some dimension is above zero, so logs already on
disk read correctly too. The usage fold's `state_version` moves (BASE+10): a
counting fix that keeps every key must retire savepoints written under the
old meaning, per the module's own rule. `credits_reported` is deliberately
left alone: a recorded 0.0 cannot be told from a turn billed at nothing, and
`_bill_credits` passes zero on purpose.

Closes #15880
@NicholasRBowers
NicholasRBowers force-pushed the fix/turn-tokens-unreported-15880 branch from 7d954fa to 8e31254 Compare October 3, 2026 15:42
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 3, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: turn/completed now writes credits only when positive and finite and tokens only when some dimension is above zero (then all four), since TurnUsage zero-fills unreported dimensions and a written zero was read as a measurement. The usage fold counts an on-disk four-zero block as unreported (sums unchanged), and the fold state version is bumped so old savepoints rebuild. Entry-type notes, specs and reference docs say readers must default both fields. Tests cover emit, projection and types. CI green.

@NicholasRBowers
NicholasRBowers merged commit 5b97ef3 into main Oct 4, 2026
101 of 107 checks passed
@NicholasRBowers
NicholasRBowers deleted the fix/turn-tokens-unreported-15880 branch October 4, 2026 16:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants