Repository navigation
feat(chat): estimate a cache rebuild from the idle gap behind a setting - #39319
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the estimate is opt-in and labeled "likely", and with the setting off the footer is the same as before. It removes nothing, and screenshots of this revision are attached. [DESIGN-REVIEWED] 99603c7 Description read[DESCRIPTION-READ] 2e764dbf1088cd2e9337af3324ca22a06b57aabef2f10ca5139d40524137c4b2 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The new setting is off by default, so nobody gets the change without choosing it. The idle-gap rule reuses the existing Notes
What this change shipsInventory (8 items) — 7 justifiedIntent: let users of a backend that reports no cache counts see when an idle pause probably rebuilt the prompt cache. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 99603c7 Description read[DESCRIPTION-READ] 1a8280eed2e5abf45c0adafaa56483e36eb75a0a33f714e852788789ecafd3bb That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing blocks. The blind reader read the new footer note, its info tip and the setting toggle correctly, and said they would use them. Notes
[UX-REVIEWED] 99603c7 Description read[DESCRIPTION-READ] f94c0b8fb463891d08539aa5898d844242ab91a4eeb22ce21d1cb654783c59a5 That sha256 names everything this verdict read: the PR description, plus the 10 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
5a4a039 to
4ba7a1f
Compare
|
Fixed in 4ba7a1f. |
A backend that reports no cache counts could never show the footer's cache-rebuilt marker. Add dashboard.turn_cache_estimate (off by default). With it on, a turn that starts at least the backend's cache lifetime after the previous reply ended, on a prefix of 20,000 tokens or more, is marked "Cache likely rebuilt", worded as an estimate. With it off, the footer is exactly as before.
4ba7a1f to
99603c7
Compare
|
5dfa726d6183 (Opus, |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: 32 files, more than 20. An opt-in "Cache likely rebuilt" estimate based on the idle gap, behind dashboard.turn_cache_estimate (off by default). With the setting off, the measured verdict is unchanged; the only new effect is that _prev_turn_ended_at is reset, which makes the estimate more cautious. Both clock readings are monotonic. The lifetime lookup resolves through the runner's composed names. The slot field is keyed to the same backend session as the stored context size, compaction and synthesized completions stay unflagged, and both footer paths carry the flag. Tests cover the rule, the setting, a real idle turn with the setting on and off, cut-turn footers and the frontend note. CI shards passed. Tier: 2, held for a person: 32 files, more than 20. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. The review checked this head and found:
- The setting is off by default (sections_dashboard.py, loader_builders.py, config-baseline.json). With it off,
note_turn_cachegives the same measured verdict as before. - With it on, a measured rewrite always wins. Any reported cache count turns the estimate off. The existing reset, restart,
/compact, synthetic-completion and mid-turn-compaction exclusions still apply, and a context that shrank is skipped. - Backends with no known lifetime (from
provider_prompt_cache_lifetime) are never estimated. - The PR's 47 tests in test/test_turn_cache_rewrite.py pass on an extracted copy of the head.
Well done: the estimate uses the same per-slot baseline as the measured rule, so every existing exclusion covers it without a second set of checks.
Optional:
- The real-turn test sets
_prev_turn_ended_atbefore each turn and reports zero counts. Changingended_at=time.monotonic()orcache_read=...in_on_complete(turn_stats.py:986-988) still passes. Asserting0 < slot._prev_turn_ended_atafter the run, and adding a turn that reports cache reads after an idle gap, would catch both changes. time.monotonic()pauses while the host is suspended, so after a laptop sleep the estimate can miss a rebuild. It only under-reports. The idle-compaction timers already track suspended time if you want to add it.- The UX lane reports that the "likely" screenshots show an older tip string than
turn_cache_likely_hinton this head. Re-shooting them would keep the evidence current.
Problem / Motivation
Goal: the "Cache rebuilt" footer note can appear on a backend that reports no cache counts, as an estimate worded "Cache likely rebuilt", taken from the time since the previous reply. It is behind a new setting,
dashboard.turn_cache_estimate, which is off by default.The marker for backends that report counts (
cache_rewritten,dashboard/cache_rewrite.py) needs a cache-write count. The default backend's turn-complete event carries none, so the rule is false for every turn there. On one install no stored usage row (0 of 41,514) held a cache count above zero, so the note could never show on the backend most people run.The cause is still visible without counts. A provider keeps a conversation's cached prefix for a known time after the last request, and Crew already holds that time per backend (
PROMPT_CACHE_LIFETIMES, used by idle compaction). On the same install, over one week, 511 of 1,062 dashboard turns that had a predecessor started 300 s or more after the previous reply, which is that backend's lifetime.Why it matters
A rebuilt prefix is the largest cost a user can change by replying sooner, compacting before a break, or turning on idle compaction. Today such a turn carries no explanation.
Not a goal
What changed (motivation → approach → change)
Symptom → cause: the default backend sends no counts, so the measured rule is blind. Approach: judge by the clock, label the result "likely", and gate it.
src/kiro_crew/config/sections_dashboard.py,loader_builders.py,docs/configuration.md,config-baseline.json: the booleandashboard.turn_cache_estimate, defaultfalse.src/kiro_crew/dashboard/cache_rewrite.py:judge_turn_cachereturns"rewrite"(measured, unchanged),"likely"or""."likely"needs both cache counts zero, a previous turn in the same backend session, a context that did not shrink since then, the smaller context at least 20,000 tokens, a known positive lifetime, and a gap from the previous reply's end to this turn's start of at least that lifetime. A context that shrank means the chat was compacted while quiet, so the cache's age is unknown and nothing is flagged.note_turn_cachekeeps its signature as the measured-only form.chat_turn/turn_stats.py:_on_completereads the setting from the live-config snapshot (no file read on the loop; unprimed means off). Off, it callsnote_turn_cacheexactly as before. On, it callsjudge_turn_cache. A synthesized completion and a turn that compacted stay unflagged.turn_stats_metaand_attach_turn_statswrite"cache": "likely"; a measured rewrite outranks it.state_parts/chat_slot.py,slot_ladder.py: the slot remembers in memory when the previous turn ended (_prev_turn_ended_at, 0.0 when unknown).chat_runner.pybindsprovider_prompt_cache_lifetime, the patch surface the composition contract requires.stop_recovery.py,turn_dispatch_gates.py,turn_locals.pycarry the new flag on the normal end-of-turn path, andturn_reply_seams.py(_attach_abnormal_turn_stats) carries it for a turn cut after its completion, so both footers agree.website/src/pages/chat/AssistantMessage.tsxand 13 locale catalogs: a "Cache likely rebuilt" note whose info tip names the cause and ends "This is an estimate."docs/system-specs/modules/turn-stats-footer.md: the "Prompt-cache rewrite flag" section now covers the setting, the idle-gap rule,_prev_turn_ended_atand thelikelykey, and its "never flagged" sentence holds with the setting off.AGENTS.mdasks for the owning spec in the same commit.Backwards compatibility
With the setting off (the default) the footer is exactly as before: the turn takes the same measured call it took, and a new test pins that a real idle turn carries no
likelykey. Additive otherwise: an unflagged turn has the payload it had before; the footer helpers gain one optional keyword that defaults to false. An older dashboard that receives"cache": "likely"draws nothing, because it matches only"rewrite". No stored data changes.Tests
test/test_turn_cache_rewrite.pyfail (nojudge_turn_cache, no setting, no slot clock, and a cut turn dropslikely); the other 25 pass.flake8,isort,mypy, the black, comment-history, sync-IO and Static Ratchet gates,npm run i18n:checkand the i18n style suites (with the base ref) pass.python -m pytest test/test_turn_cache_rewrite.py -n0andcd website && npx vitest run src/test/AssistantMessage.test.tsx -t "cache"Manual verification
The real footer on loopback with the setting on, light and dark. Likely rebuilt, tip open:
A measured rebuild (unchanged) and a warm cache (unchanged):
The setting's row in Settings → Advanced:
Sibling sweep
Sibling:
turn_reply_seams.py(_attach_abnormal_turn_stats) -- fixed here -- a turn cut after its completion now keeps the "likely" flag as well as the measured oneSibling: cron result rows (
turn_stats_metacallers without a turn) -- not affected -- neither flag is passedSibling:
src/kiro_crew/session_idle_compaction.py-- not affected -- its lifetime table is read throughprovider_prompt_cache_lifetimePattern harvest
Rule candidate: review-prompt
Pattern: an owner module of the chat runner reads a name that tests patch, so the name is bound on the runner and imported by the owner only for type checking.
Related Issues
Builds on the merged cache-rebuilt footer (#35168), the hit-rate display (#31501) and opt-in idle compaction (#24384). No open issue or PR covers inference without counts.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement