Skip to content

feat: report subagent credit usage alongside elapsed time - #8003

Closed
welikoiwanenko wants to merge 1 commit into
kirodotdev:mainfrom
welikoiwanenko:codex/subagent-credit-usage
Closed

welikoiwanenko wants to merge 1 commit into
kirodotdev:mainfrom
welikoiwanenko:codex/subagent-credit-usage

Conversation

@welikoiwanenko

@welikoiwanenko welikoiwanenko commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Show the credits consumed by each managed subagent alongside its terminal elapsed time.

Managed subagent completion reports elapsed time but not the credits consumed by that run. Parents and users cannot attribute the cost of a successful, failed, or stopped subagent without comparing noisy account-level totals.

For maintainer approval: cumulative positive reported credits appear in expanded Subagents activity cards, with elapsed time retained in compact headers and usage included in completion messages and spawn_status.

Why it matters

A failed or stopped attempt can consume credits even when it produces no final answer. Per-run attribution makes those costs visible, including billed transient retries, without relying on unrelated activity in the same account.

Not a goal

  • Estimate dollar costs or invent usage for providers that do not report credit billing.
  • Attribute native-harness subagents whose billing remains part of the parent turn.
  • Add a new billing ledger, dependency, configuration option, or schema migration.

What changed (motivation → approach → change)

  • Accumulate the shared TurnUsage representation per attempted provider turn. Completion events supply usage on success; the shared billing resolver supplies fresh usage on interrupted or failed attempts. Each attempt settles once, including cancellation during event handling and force-reaping.
  • Settle active usage before force-reap reporting so stopped runs cannot publish partial totals.
  • Preserve terminal credits and elapsed time in existing delivery/failure tombstones, then expose them through completion notifications and full or paged spawn_status responses. Parent delivery does not wait for a new post-completion state write.
  • Carry credits in live and replayed subagent_done frames so dashboard reconnects retain terminal usage. Legacy retained records remain readable without invented fields.
  • Display positive credits in expanded completed, failed, and stopped cards as the run-scoped copy Used … credits. Compact headers continue to show elapsed time. Backend summaries switch to minutes and seconds at 60 seconds and retain the existing credit precision.
  • Keep the credit formatter in the shared i18n formatting module so the activity chunk does not pull in the assistant-message component tree.

No dependency, configuration, schema migration, model-default, or changelog changes. Native harness subagents remain unattributed because they share the parent's billed turn. Providers that do not report credit billing return zero; displays omit that credit label rather than claim the run was free. This feature does not estimate charges or add dollar billing. Run totals are completion summaries, not additional billing-ledger entries; failed-attempt credits can make them larger than successful-turn records alone.

Backwards compatibility

Compatible: existing providers and retained records without credit data continue to render elapsed time without inventing usage.

Tests

  • 1,458 focused backend tests passed after the latest rebase, covering subagent execution, persistence, reaping, delivery, messaging, scale, and session behavior.
  • Earlier frontend validation covered the activity viewer, assistant messages, and chat state; CI is the authoritative validation after the latest rebase.
  • Flake8, isort, and diff checks passed for the conflict resolution.

Manual verification

Rendered the current production ActivityViewer component with fixture-only state and captured expanded successful, failed, and stopped cards in dark and light themes. Verified credit-only expanded copy, elapsed-only compact headers, minute formatting, and narrow activity-rail layout. No live gateway or paid subagent calls were used. Live provider billing verification remains for maintainers.

Screenshots / video

Fixture data rendered by the current component; each card shows its own terminal usage.

kirocrew-credit-usage-dark-456x760 kirocrew-credit-usage-light-456x760

Related Issues

Implements the credit-attribution request in #4721. The issue has an earlier automated claim; this PR is for coordination and maintainer approval, not an assertion that design approval has already been given.

Fixes #4721

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

The repository's contribution terms remain unchanged; no additional declaration is made here.

@iamwhatever

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@welikoiwanenko
welikoiwanenko marked this pull request as ready for review September 2, 2026 22:05
@welikoiwanenko
welikoiwanenko requested review from a team and krishdhasmana September 2, 2026 22:05
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 2, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 1dcffc7 to 6bb4677 Compare September 2, 2026 23:10
@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5, fork) — 🟡 CONCERNS

UX-level review of 1484bb0cf3e89af77511f5d13a520a61831a58d0 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

"Credit usage not reported" reads as a fault line on every expanded card for providers that never bill credits, and no first-time reader has seen either new string.

Watch

  • credits_used/credits_not_reported (en.manual.json): providers that don't bill credits report 0, so every expanded completed managed card permanently opens with "Credit usage not reported" — a warning-shaped sentence with nothing to act on, while chat notices render the same unknown state as silence (Usage: 13s). High frequency for those users × mild anxiety × every expansion. Smallest fix: name the cause ("This provider doesn't report credit usage") or omit the line, matching the message surfaces.
  • Both new strings are shown in both themes but unread cold — this lane cannot confirm a first-time user parses "credits" as billing rather than quota; a maintainer can push the branch to this repository to get the blind read.

Suggestions

  • ActivityViewer.tsx terminalCredits gate (a.credits > 0): a 0.004-credit run renders "Used 0.00 credits"; route values under 0.005 into the unreported branch or render "<0.01".

[UX-REVIEWED] 1484bb0

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 1484bb0cf3e89af77511f5d13a520a61831a58d0 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the two attached screenshots actually show the expanded managed-card usage line in both states — "Used … credits" and "Credit usage not reported" — since that expanded row is the only new visual surface.

[DESIGN-REVIEWED] 1484bb0

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 1484bb0cf3e89af77511f5d13a520a61831a58d0 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the accumulated credit total matches real provider billing on a live paid run — the author explicitly deferred that ("Live provider billing verification remains for maintainers"; no paid subagent calls were used).

Every inventory item has a named harm (per-run cost attribution, issue #4721), a live consumer count, and sits at cause level: credits are accumulated at the provider-billing source per attempted turn rather than patched onto the success path. The SubagentDelivery refactor and _RunCreditAccounting complexity are load-bearing for the "billed failed/retried attempts" clause the Goal's motivation names, not surface riding along. Not product-shape: elapsed time is retained and credits are added beside existing output, so no default is replaced and no capability removed — lens 9 does not fire, and the injected-messages/subagent specs are updated in the same commit.

Inventory (10 items) — 10 justified

Intent: Show credits consumed by each managed subagent beside its terminal elapsed time (ADDITION).

  1. Subagent completion event carries a Usage: <credits> · <elapsed> line — justified
  2. spawn_status MCP output prefixed with [usage: …] (full, paged, and error responses) — justified
  3. Expanded dashboard card shows localized "Used … credits" — justified
  4. Expanded card shows "Credit usage not reported" when credits are zero/absent — justified
  5. Wave-digest and delivery-failure messages carry per-member usage — justified
  6. GET /api/spawn/{id} returns elapsed+credits; delivery/failure tombstones persist them; legacy records omit — justified
  7. SubagentDelivery replaces list[str] in the delivery-debt ledger so settlement uses an immutable snapshot after manager eviction — justified
  8. _RunCreditAccounting settles each attempted turn once across stream/consumer/retry/cancel/reap — justified
  9. fmtCredits moved from AssistantMessage.tsx to shared i18n/format.ts — justified (avoids pulling the assistant-message tree into the activity chunk)
  10. Credit precision formatting (2 decimals under 10, 1 at/above), elapsed rounds to whole seconds — justified

[FIRST-PRINCIPLES-REVIEWED] 1484bb0

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 1484bb0cf3e89af77511f5d13a520a61831a58d0 via the fork AI-review pipeline; updated in place on each push.

Review details

Two spec/output-contract mismatches the PR's own added prose contradicts.

FINDING — docs/system-specs/modules/subagent.md:782 — the owning spec still documents _digest_settle_ids (also line 837) and held_ok_ids (line 1766) after this PR renames both to _digest_settle_deliveries / held_ok_deliveries in subagent.py's DELIVERY_ROUTING_FIELDS and gateway.py's chunk buffers, so the documented parked-state table names attributes that no longer exist in the tree → Fix: rename the three references in the same commit and note at 782 that the list now holds SubagentDelivery records rather than bare ids.

FINDING — src/kiro_crew/mcp_tools/spawn.py:1135 — return f"Error: {meta['grep_error']}" returns before the new [usage: …] prefix, so spawn_status(agent_id, grep="(") on a terminal record with recorded credits/elapsed yields an error with no usage, contradicting the sentence this PR adds to the tool schema (Error responses also carry this prefix when terminal usage is available.) → Fix: prefix that return like the d.get("error") branch, or narrow the added sentence to the backend-failure error.

[OPUS-REVIEWED] 1484bb0

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 1484bb0cf3e89af77511f5d13a520a61831a58d0 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1484bb0

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Sep 3, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 6bb4677 to e3d8211 Compare September 3, 2026 08:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 3, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from a1aeb5b to 63a6148 Compare September 7, 2026 10:14
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 7, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch 2 times, most recently from 845224a to 1bfa611 Compare September 8, 2026 10:13
@bolichen97 bolichen97 mentioned this pull request Sep 8, 2026
5 tasks done
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

@welikoiwanenko Thanks for this. Keeping it: it is the only implementation of #4721, and nothing on main covers any part of it. Two open PRs touch the same code, so we should agree a landing order first.

#9008 (@javenciu) inserts at the exact line this PR does: TerminalCoordinator._report_terminal_impl in src/kiro_crew/subagent_manager/terminal.py, immediately after info.done = True. #9008 moves the whole report body into a new _report_terminal_guarded_impl; this PR adds info._credit_accounting.settle(), the credits key on the subagent_done payload, and a bounded terminal-usage state write. Both also add lines at the child-escalation and turn-limit bails in src/kiro_crew/subagent_manager/run.py, and both edit the inspect.getsource pin in test/test_subagent_scale.py. No behaviour is duplicated. #9008 is further along in that region because it already relocates that pin, so please let #9008 land first, then re-place the settle() call and the payload key inside _report_terminal_guarded_impl.

#9101 (@Premshay) is much smaller (5 files against 27 here) and collides only textually: both add a field to the SubagentInfo dataclass in src/kiro_crew/subagent.py, both add statements around _run_inner_impl in run.py (this PR changes the signature at line 620, #9101 sets _first_stream_started at 629 and 1167), and both edit neighbouring bullets in docs/system-specs/modules/subagent.md. Letting #9101 go first costs you a rebase and nothing else.

Please rebase either way: at audit time the branch was 46 commits behind main, and #7195 rewrites the same subagent.md timeout bullet you reword here.

Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 10, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 1bfa611 to 459187c Compare September 10, 2026 11:56
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 10, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 459187c to 5bac35e Compare September 10, 2026 17:47
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 10, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch 3 times, most recently from a7b5f0f to 318c9ff Compare September 14, 2026 20:34
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 14, 2026
@welikoiwanenko

Copy link
Copy Markdown
Contributor Author

@welikoiwanenko Thanks for this. Keeping it: it is the only implementation of #4721, and nothing on main covers any part of it. Two open PRs touch the same code, so we should agree a landing order first.

#9008 (@javenciu) inserts at the exact line this PR does: TerminalCoordinator._report_terminal_impl in src/kiro_crew/subagent_manager/terminal.py, immediately after info.done = True. #9008 moves the whole report body into a new _report_terminal_guarded_impl; this PR adds info._credit_accounting.settle(), the credits key on the subagent_done payload, and a bounded terminal-usage state write. Both also add lines at the child-escalation and turn-limit bails in src/kiro_crew/subagent_manager/run.py, and both edit the inspect.getsource pin in test/test_subagent_scale.py. No behaviour is duplicated. #9008 is further along in that region because it already relocates that pin, so please let #9008 land first, then re-place the settle() call and the payload key inside _report_terminal_guarded_impl.

#9101 (@Premshay) is much smaller (5 files against 27 here) and collides only textually: both add a field to the SubagentInfo dataclass in src/kiro_crew/subagent.py, both add statements around _run_inner_impl in run.py (this PR changes the signature at line 620, #9101 sets _first_stream_started at 629 and 1167), and both edit neighbouring bullets in docs/system-specs/modules/subagent.md. Letting #9101 go first costs you a rebase and nothing else.

Please rebase either way: at audit time the branch was 46 commits behind main, and #7195 rewrites the same subagent.md timeout bullet you reword here.

Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong.

I see one MR already landed, any updates on the second one?

@Premshay

Copy link
Copy Markdown
Contributor

#9101 has now merged. #9008 remains open and blocked on its own terminal-report hold fixes, so it has not reached the point where #8003 can safely rebase onto its guarded reporter. Once #9008 lands, I’ll rebase #8003 and re-place the credit settlement and subagent_done payload changes inside that structure.

Attribute reported credits to managed subagent runs, including failed and
cancelled attempts, so users and parent agents can compare their costs.

Reuse shared turn billing, preserve terminal usage in delivery/failure
tombstones, and show it across completion events, status queries, and
dashboard cards without delaying parent delivery.
@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks for this contribution, @welikoiwanenko! Your work on reporting subagent credit usage alongside elapsed time is landing.

To let a clean same-repo CI wave run (fork PRs cannot exercise every review lane), your commit has been carried into an in-repo replacement PR — #14913 — with your authorship preserved (Co-authored-by). That PR is now all-green: GPT 5.6, Opus 5, Design, First-Principles, Security Scope, and UX review lanes are all passing, along with the full backend/frontend/desktop/E2E suite.

Two review findings surfaced during drive-to-green and were addressed in the replacement, both preserving the intent of your change:

  • Failure/reap tombstones persisted a pre-finalization elapsed value, so a replayed spawn_status could disagree with the terminal subagent_done event. elapsed is now finalized once before any abnormal-exit tombstone write, and both writers read that single value.
  • A delivery-ledger test assertion was restored to a whole-ledger comparison so the "exactly one key" pin the successor-dequeue race depends on is preserved.

Closing this in favor of #14913. Full credit for the feature is yours — thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface per-session credit usage in subagent completion events

4 participants