Skip to content

feat(chat): show the prompt-cache hit rate in the context popover - #31501

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/17985-cache-hit-ratio
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/17985-cache-hit-ratio

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let a user see, in the chat composer's context popover, how much of the session's prompt the model read from its prompt cache, and show nothing at all for a backend that does not report cache counts.

The context popover shows the used, remaining and total window tokens, the model and the auto-compact threshold. It says nothing about the prompt cache, so a user whose session costs more than expected cannot tell whether the prompt is reused from the cache or rebuilt on every turn. Backends that report cache counts already send them, and the session's crew log already sums them, but no chat surface turns them into a rate.

Builds on #20033 by @javenciu.

Why it matters

A low hit rate on a long session is the sign that something in the prompt changes every turn and defeats the cache, which costs tokens. The popover is where a user already checks what the session holds, so the rate belongs beside those numbers rather than on a page they have to go looking for.

Not a goal

  • A per-turn or historical view of the rate. The row is the session total.
  • Estimating a rate for a backend that reports no cache counts, such as kiro-cli. Those sessions show no row.
  • Changing how the crew log sums token counts, or which backends report them.
  • An explainer or tooltip on the row beyond its one-line caption.

What changed (motivation → approach → change)

Which backends report it. The counts come from the ACP prompt response's inputTokens, cachedReadTokens and cachedWriteTokens. claude-agent-acp, codex-acp and opencode send them as separate counts; kiro-cli's response carries only a stop reason. Copilot CLI is the exception: its inputTokens already holds its cache reads and writes (a recorded turn: 12690 = 12629 read + 58 written + 3 uncached), so a rate computed from what Crew records today would show Copilot about half its real hit rate.

Approach. Make the four counts mean the same thing on every harness, then read them where they already live. On a Copilot session the client takes the cache counts back out of the input count before recording the turn (acp/client_usage.py, a positive _is_copilot test). The popover reads the crew log's usage fold through the read the Crew log panel already makes (GET /api/sessions/{id}/crew-log/projections), under the panel's query key, so a pushed session_projection frame keeps an open popover current. The rate is cache_read / (cache_read + cache_write + input); output tokens are not part of the prompt.

Change.

Where What
ContextShelf.tsx a "Cache hit rate" group under the window counts, with a caption naming its basis. It appears only when the fold holds a cache count above zero: a backend with none or a Shared Crew guest the owner-only log refuses gets no row, never a 0%. Any other failed read, a re-read over an earlier rate included, shows an inline error instead of the rate, with no hand-off beside the unsaved draft
ChatInput.tsx a session that lives on a peer crew reads nothing, the rule it already applies to the auto-compact threshold
client_usage.py, session_handle_compaction.py Copilot's input count without its cache tokens; the handle's docstring says why it has no such branch
website/src/i18n, docs the two strings in every locale with translator notes (en.context.json); acp-client.md states the disjoint rule; the feature map names the popover as a reader of the usage fold

Backwards compatibility

Compatible: one added popover row, and nothing main accepts is refused, renamed or removed. A Copilot turn now records its input count without its cache tokens, the meaning every other harness's count already has, so the Crew log panel's token total and the usage telemetry stop counting Copilot's cache tokens twice. Rows written before this change keep the count they were written with.

Removes nothing: every existing popover row stays, and the crew-log read is an added reader of a route the Crew log panel already uses.

Tests

  • website/src/test/ChatInputCoverage.test.tsx: the popover shows 75% for 750 cache reads, 150 cache writes and 100 uncached input tokens, with its caption. It shows no row and no 0% for four zero counts (how kiro-cli folds), for input tokens with no cache count, and for a Shared Crew guest's owner_only refusal; each case then seeds a reporting fold into the same open popover and sees the row, so the absence is the data's doing. Any other failed read shows the inline error and no "ask the agent" button, on a first read and on a re-read after the popover was closed over a drawn rate (that case is red with the error hidden while earlier data exists).
  • website/src/test/CrewChatWindow.test.tsx: a peer session's open popover never reads this machine's crew log.
  • test/test_acp_usage_cost.py: a Copilot prompt response with the recorded counts lands as 3 input, 6 output, 12629 read and 58 written.
  • Determinism: the popover tests ran 5 times in a row and in shuffled order (seed 1793967707129), all green. The seven cache-rate tests, including both failed-read notices, then ran 3 more times, shuffled on seeds 1794000001-1794000003, all green. Shuffling the whole ChatInputCoverage.test.tsx file on that seed fails two paste-token caret tests the same way on the base commit, without this change.

Proof: PROVEN test/test_acp_usage_cost.py::TestClientTracking::test_copilot_prompt_usage_keeps_cache_out_of_input (head f3071c6)

Manual verification

Shot from the real built SPA with every /api/** call answered from fixtures (screenshots below): the row with a reporting fold in dark and light, and the popover with no row for a fold of zero counts. The codex-acp 2.3.0 package the adapter tests pin was read to confirm it takes the cached tokens out of inputTokens itself.

Screenshots / video

Reporting backend (dark):

Context popover with a 75% cache hit rate, dark theme

Reporting backend (light):

Context popover with a 75% cache hit rate, light theme

Backend with no cache counts: no row.

Context popover without a cache hit rate row

Related Issues

Closes #17985

Sibling sweep

N/A: a feature, not a fix.

Pattern harvest

N/A: a feature, not a fix.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 15:28
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 15:28
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a474cd2269d6ab4914b64d2e96559ca36491d824 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The Copilot input-count correction is what makes the rate true on Copilot sessions, and the recorded turn backs it: 12690 = 12629 + 58 + 3.

What this change ships

Inventory (8 items) — 8 justified

Intent: Let a user see how much of a session's prompt came from the prompt cache, so they can spot a session that rebuilds its prompt every turn. This is an ADDITION.

No prompt-cache hit rate exists anywhere yet. I grepped cache.?hit|hit_rate|hitRate across website/src and src/kiro_crew. The only rate found is the MCP warm-pool rate in McpGatewayCard.tsx, which measures something else. No other code removes Copilot's cache tokens from its input count either (grepped copilot.*input_tokens|cache_read: 0 hits outside this diff).

  1. A "Cache hit rate" row with a one-line caption under the popover's window counts — justified
  2. No row at all for a backend with no cache counts, or for a Shared Crew guest the owner-only log refuses — justified
  3. An inline error in place of the rate when any other crew-log read fails, including a re-read — justified
  4. The popover re-reads the crew log each time it opens, and a pushed frame updates it while open — justified
  5. A session on a peer crew reads nothing from this machine's crew log — justified
  6. Copilot turns now record their input count without cache tokens, so Copilot's Crew log token total and usage telemetry drop — justified
  7. Three new strings in every locale, with translator notes — justified
  8. acp-client.md states that the four counts are disjoint, and the feature map lists the popover as a reader of the usage fold — justified

[FIRST-PRINCIPLES-REVIEWED] a474cd2

Description read

[DESCRIPTION-READ] cf840d972d5b81acb351dcb109da6a4f56ca331ef8bed4a555ea3588268806b2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a474cd2269d6ab4914b64d2e96559ca36491d824 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the Copilot fix changes what gets recorded at the place the counts are parsed, and the popover reads an existing projection under the panel's own query key, so updates pushed to the panel reach the popover too.

Notes

  • A Copilot session that started before the upgrade mixes rows that count cache tokens inside inputTokens with rows that don't. Its total and its hit rate stay understated until those older rows are no longer counted. This reverses on its own, but the ## Backwards compatibility line about old rows could say that the rate is affected too.

[DESIGN-REVIEWED] a474cd2

Description read

[DESCRIPTION-READ] e2d42d18ec86e17cfa0d937e74d113acb2e802c26755b7633ae804424e86bb79

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

UX-level review of a474cd2269d6ab4914b64d2e96559ca36491d824 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

cannot evaluate: missing the inline error state ("Couldn't load the cache hit rate") that the diff adds to the popover.

Evidence gaps

  • cache-hit-rate-load-error: the ErrorNotice variant="inline" that CacheHitRate renders when a read fails, including a failed re-read over a rate that was already drawn. The three screenshots show the row filled in dark, filled in light, and absent; none shows the error. To close this, attach a still of the open popover with the read failing.

Notes

  • The blind reader understood the row but said "I don't know if 75% is good or bad" and "no idea what it means for me". The PR's stated reason for the row is spotting a low rate on a long session, so the caption "Share of the prompt reused from cache" could say what it means, e.g. "Higher means cheaper turns".
  • The row only mounts once the query resolves, so the popover grows by one group after it opens. Holding the row's height with a placeholder while the first read is in flight would stop the Model and Auto-compact rows from jumping down.

[UX-REVIEWED] a474cd2

Description read

[DESCRIPTION-READ] d1bcba98b26a6ae90d082f246627d396c6a45d54ae22dd8da76994a9bc18a7e6

That sha256 names everything this verdict read: the PR description, plus the 6 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a474cd2269d6ab4914b64d2e96559ca36491d824 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a474cd2

Verdict parsed from the review's SHA-scoped output markers for commit a474cd2269d6ab4914b64d2e96559ca36491d824.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a474cd2269d6ab4914b64d2e96559ca36491d824 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a474cd2269d6ab4914b64d2e96559ca36491d824 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a474cd2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a474cd2269d6ab4914b64d2e96559ca36491d824 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 15:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/17985-cache-hit-ratio branch from f3071c6 to e91049d Compare November 6, 2026 20:43
@bolichen97

bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

span=9b0949fe2799
self-added: no
mechanism: inline ErrorNotice for a failed crew-log read in the popover

  • Cache-usage request failures silently hide the readout

fixed in e91049d: CacheHitRate now renders an inline ErrorNotice ("Couldn't load the cache hit rate") for any failed read, a re-read over an earlier rate included, with no hand-off beside the unsaved composer draft.
The one refusal that stays silent is the owner-only log's 403 owner_only for a Shared Crew guest (isOwnerOnlyRefusal), which is the expected answer for that caller, not a failure.
Red/green: the two "says so when ... of the crew log fails" cases in ChatInputCoverage.test.tsx; the re-read case is red while earlier data hides the error.

@bolichen97

bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

span=9b0949fe2799
self-added: no
mechanism: none added

  • Resumed Copilot logs mix incompatible input-token counts

rebutted (disproportionate; the lane's own adjudication downgraded it to LOW): only a Copilot session recorded before this change and then resumed is affected, and only the optional on-screen figure, which moves toward the right value as new turns land.
No stored data is changed. Telling old rows from new needs a provenance field in the crew-log turn schema plus a legacy branch in the usage fold, kept forever for a one-time upgrade state.
The rate stays correct for every session recorded after this change, which is the goal of the PR.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026
The context popover now shows the session's prompt-cache hit rate,
read from the crew log's usage fold under the Crew log panel's query
key: cache reads over cache reads, cache writes and uncached input.
The row appears only when the session has a cache count, so a backend
that reports none (kiro-cli), a Shared Crew guest the owner-only log
refuses and a peer session show nothing rather than 0%. Any other
failed read shows an inline error with no hand-off, beside the unsaved
composer draft.

Copilot CLI counts its cache reads and writes inside inputTokens, so a
Copilot turn now records its input count without them, the meaning the
other harnesses' counts already have. That keeps the rate right on
Copilot and stops the usage totals counting its cache tokens twice.

Closes #17985
@bolichen97
bolichen97 force-pushed the fix/17985-cache-hit-ratio branch from e91049d to a474cd2 Compare November 6, 2026 22:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on a474cd2, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit 5c1fc97 into main Nov 7, 2026
97 of 99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/17985-cache-hit-ratio branch November 7, 2026 00:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Context popover: show the prompt-cache hit ratio for the session

2 participants