Repository navigation
feat(chat): show the prompt-cache hit rate in the context popover - #31501
Conversation
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The Copilot input-count correction is what makes the rate true on Copilot sessions, and the recorded turn backs it: 12690 = 12629 + 58 + 3. What this change shipsInventory (8 items) — 8 justifiedIntent: Let a user see how much of a session's prompt came from the prompt cache, so they can spot a session that rebuilds its prompt every turn. This is an ADDITION. No prompt-cache hit rate exists anywhere yet. I grepped
[FIRST-PRINCIPLES-REVIEWED] a474cd2 Description read[DESCRIPTION-READ] cf840d972d5b81acb351dcb109da6a4f56ca331ef8bed4a555ea3588268806b2 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the Copilot fix changes what gets recorded at the place the counts are parsed, and the popover reads an existing projection under the panel's own query key, so updates pushed to the panel reach the popover too. Notes
[DESIGN-REVIEWED] a474cd2 Description read[DESCRIPTION-READ] e2d42d18ec86e17cfa0d937e74d113acb2e802c26755b7633ae804424e86bb79 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)UX-level review of UX-Verdict: PASS cannot evaluate: missing the inline error state ("Couldn't load the cache hit rate") that the diff adds to the popover. Evidence gaps
Notes
[UX-REVIEWED] a474cd2 Description read[DESCRIPTION-READ] d1bcba98b26a6ae90d082f246627d396c6a45d54ae22dd8da76994a9bc18a7e6 That sha256 names everything this verdict read: the PR description, plus the 6 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
f3071c6 to
e91049d
Compare
|
span=9b0949fe2799
|
|
span=9b0949fe2799
|
The context popover now shows the session's prompt-cache hit rate, read from the crew log's usage fold under the Crew log panel's query key: cache reads over cache reads, cache writes and uncached input. The row appears only when the session has a cache count, so a backend that reports none (kiro-cli), a Shared Crew guest the owner-only log refuses and a peer session show nothing rather than 0%. Any other failed read shows an inline error with no hand-off, beside the unsaved composer draft. Copilot CLI counts its cache reads and writes inside inputTokens, so a Copilot turn now records its input count without them, the meaning the other harnesses' counts already have. That keeps the rate right on Copilot and stops the usage totals counting its cache tokens twice. Closes #17985
e91049d to
a474cd2
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved for the on-call. Review lanes pass on a474cd2, no conflict, no open change request.
Problem / Motivation
Goal: Let a user see, in the chat composer's context popover, how much of the session's prompt the model read from its prompt cache, and show nothing at all for a backend that does not report cache counts.
The context popover shows the used, remaining and total window tokens, the model and the auto-compact threshold. It says nothing about the prompt cache, so a user whose session costs more than expected cannot tell whether the prompt is reused from the cache or rebuilt on every turn. Backends that report cache counts already send them, and the session's crew log already sums them, but no chat surface turns them into a rate.
Builds on #20033 by @javenciu.
Why it matters
A low hit rate on a long session is the sign that something in the prompt changes every turn and defeats the cache, which costs tokens. The popover is where a user already checks what the session holds, so the rate belongs beside those numbers rather than on a page they have to go looking for.
Not a goal
What changed (motivation → approach → change)
Which backends report it. The counts come from the ACP prompt response's
inputTokens,cachedReadTokensandcachedWriteTokens. claude-agent-acp, codex-acp and opencode send them as separate counts; kiro-cli's response carries only a stop reason. Copilot CLI is the exception: itsinputTokensalready holds its cache reads and writes (a recorded turn: 12690 = 12629 read + 58 written + 3 uncached), so a rate computed from what Crew records today would show Copilot about half its real hit rate.Approach. Make the four counts mean the same thing on every harness, then read them where they already live. On a Copilot session the client takes the cache counts back out of the input count before recording the turn (
acp/client_usage.py, a positive_is_copilottest). The popover reads the crew log'susagefold through the read the Crew log panel already makes (GET /api/sessions/{id}/crew-log/projections), under the panel's query key, so a pushedsession_projectionframe keeps an open popover current. The rate iscache_read / (cache_read + cache_write + input); output tokens are not part of the prompt.Change.
ContextShelf.tsxChatInput.tsxclient_usage.py,session_handle_compaction.pywebsite/src/i18n, docsen.context.json);acp-client.mdstates the disjoint rule; the feature map names the popover as a reader of theusagefoldBackwards compatibility
Compatible: one added popover row, and nothing main accepts is refused, renamed or removed. A Copilot turn now records its input count without its cache tokens, the meaning every other harness's count already has, so the Crew log panel's token total and the usage telemetry stop counting Copilot's cache tokens twice. Rows written before this change keep the count they were written with.
Removes nothing: every existing popover row stays, and the crew-log read is an added reader of a route the Crew log panel already uses.
Tests
website/src/test/ChatInputCoverage.test.tsx: the popover shows 75% for 750 cache reads, 150 cache writes and 100 uncached input tokens, with its caption. It shows no row and no 0% for four zero counts (how kiro-cli folds), for input tokens with no cache count, and for a Shared Crew guest'sowner_onlyrefusal; each case then seeds a reporting fold into the same open popover and sees the row, so the absence is the data's doing. Any other failed read shows the inline error and no "ask the agent" button, on a first read and on a re-read after the popover was closed over a drawn rate (that case is red with the error hidden while earlier data exists).website/src/test/CrewChatWindow.test.tsx: a peer session's open popover never reads this machine's crew log.test/test_acp_usage_cost.py: a Copilot prompt response with the recorded counts lands as 3 input, 6 output, 12629 read and 58 written.ChatInputCoverage.test.tsxfile on that seed fails two paste-token caret tests the same way on the base commit, without this change.Proof: PROVEN test/test_acp_usage_cost.py::TestClientTracking::test_copilot_prompt_usage_keeps_cache_out_of_input (head f3071c6)
Manual verification
Shot from the real built SPA with every
/api/**call answered from fixtures (screenshots below): the row with a reporting fold in dark and light, and the popover with no row for a fold of zero counts. The codex-acp 2.3.0 package the adapter tests pin was read to confirm it takes the cached tokens out ofinputTokensitself.Screenshots / video
Reporting backend (dark):
Reporting backend (light):
Backend with no cache counts: no row.
Related Issues
Closes #17985
Sibling sweep
N/A: a feature, not a fix.
Pattern harvest
N/A: a feature, not a fix.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.