Skip to content

feat(chat): estimate a cache rebuild from the idle gap behind a setting - #39319

Merged
javenciu merged 1 commit into
mainfrom
feat/cache-likely-flag
Nov 19, 2026
Merged

javenciu merged 1 commit into
mainfrom
feat/cache-likely-flag

Conversation

@javenciu

@javenciu javenciu commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the "Cache rebuilt" footer note can appear on a backend that reports no cache counts, as an estimate worded "Cache likely rebuilt", taken from the time since the previous reply. It is behind a new setting, dashboard.turn_cache_estimate, which is off by default.

The marker for backends that report counts (cache_rewritten, dashboard/cache_rewrite.py) needs a cache-write count. The default backend's turn-complete event carries none, so the rule is false for every turn there. On one install no stored usage row (0 of 41,514) held a cache count above zero, so the note could never show on the backend most people run.

The cause is still visible without counts. A provider keeps a conversation's cached prefix for a known time after the last request, and Crew already holds that time per backend (PROMPT_CACHE_LIFETIMES, used by idle compaction). On the same install, over one week, 511 of 1,062 dashboard turns that had a predecessor started 300 s or more after the previous reply, which is that backend's lifetime.

Why it matters

A rebuilt prefix is the largest cost a user can change by replying sooner, compacting before a break, or turning on idle compaction. Today such a turn carries no explanation.

Not a goal

  • Changing the measured rule or its thresholds. A backend that reports any cache count is judged by the counts alone, as before.
  • Other causes of a rebuild (a model switch, an edited prefix). Only the idle gap is inferred, and the note says so.
  • Turning the setting on by default. That is a separate follow-up.

What changed (motivation → approach → change)

Symptom → cause: the default backend sends no counts, so the measured rule is blind. Approach: judge by the clock, label the result "likely", and gate it.

  • src/kiro_crew/config/sections_dashboard.py, loader_builders.py, docs/configuration.md, config-baseline.json: the boolean dashboard.turn_cache_estimate, default false.
  • src/kiro_crew/dashboard/cache_rewrite.py: judge_turn_cache returns "rewrite" (measured, unchanged), "likely" or "". "likely" needs both cache counts zero, a previous turn in the same backend session, a context that did not shrink since then, the smaller context at least 20,000 tokens, a known positive lifetime, and a gap from the previous reply's end to this turn's start of at least that lifetime. A context that shrank means the chat was compacted while quiet, so the cache's age is unknown and nothing is flagged. note_turn_cache keeps its signature as the measured-only form.
  • chat_turn/turn_stats.py: _on_complete reads the setting from the live-config snapshot (no file read on the loop; unprimed means off). Off, it calls note_turn_cache exactly as before. On, it calls judge_turn_cache. A synthesized completion and a turn that compacted stay unflagged. turn_stats_meta and _attach_turn_stats write "cache": "likely"; a measured rewrite outranks it.
  • state_parts/chat_slot.py, slot_ladder.py: the slot remembers in memory when the previous turn ended (_prev_turn_ended_at, 0.0 when unknown). chat_runner.py binds provider_prompt_cache_lifetime, the patch surface the composition contract requires. stop_recovery.py, turn_dispatch_gates.py, turn_locals.py carry the new flag on the normal end-of-turn path, and turn_reply_seams.py (_attach_abnormal_turn_stats) carries it for a turn cut after its completion, so both footers agree.
  • website/src/pages/chat/AssistantMessage.tsx and 13 locale catalogs: a "Cache likely rebuilt" note whose info tip names the cause and ends "This is an estimate."
  • docs/system-specs/modules/turn-stats-footer.md: the "Prompt-cache rewrite flag" section now covers the setting, the idle-gap rule, _prev_turn_ended_at and the likely key, and its "never flagged" sentence holds with the setting off. AGENTS.md asks for the owning spec in the same commit.

Backwards compatibility

With the setting off (the default) the footer is exactly as before: the turn takes the same measured call it took, and a new test pins that a real idle turn carries no likely key. Additive otherwise: an unflagged turn has the payload it had before; the footer helpers gain one optional keyword that defaults to false. An older dashboard that receives "cache": "likely" draws nothing, because it matches only "rewrite". No stored data changes.

Tests

  • Before: on main, 22 of the new and changed cases in test/test_turn_cache_rewrite.py fail (no judge_turn_cache, no setting, no slot clock, and a cut turn drops likely); the other 25 pass.
  • After: all 47 pass, plus the web test for the note.
  • Related backend tests (186 files: the cache, config, turn-stats and runner-surface tests): 6,382 passed, 6 skipped.
  • Full backend suite (round-1 head, same base): 281,373 passed; the 209 failures also fail on plain main here. No new failures.
  • flake8, isort, mypy, the black, comment-history, sync-IO and Static Ratchet gates, npm run i18n:check and the i18n style suites (with the base ref) pass.
  • Reproduce: python -m pytest test/test_turn_cache_rewrite.py -n0 and cd website && npx vitest run src/test/AssistantMessage.test.tsx -t "cache"
  • Proof: PROVEN test/test_turn_cache_rewrite.py::test_a_real_idle_turn_carries_the_likely_key_only_with_the_setting_on, test/test_turn_cache_rewrite.py::test_a_turn_cut_after_its_completion_keeps_the_flag_in_its_footer, test/test_turn_cache_rewrite.py::test_the_gap_rule_over_the_cases, test/test_turn_cache_rewrite.py::test_the_setting_is_off_by_default_and_read_from_config (head 99603c7)

Manual verification

The real footer on loopback with the setting on, light and dark. Likely rebuilt, tip open:

likely, light
likely, dark

A measured rebuild (unchanged) and a warm cache (unchanged):

measured, light
measured, dark
warm, light
warm, dark

The setting's row in Settings → Advanced:

settings row, light

Sibling sweep

Sibling: turn_reply_seams.py (_attach_abnormal_turn_stats) -- fixed here -- a turn cut after its completion now keeps the "likely" flag as well as the measured one
Sibling: cron result rows (turn_stats_meta callers without a turn) -- not affected -- neither flag is passed
Sibling: src/kiro_crew/session_idle_compaction.py -- not affected -- its lifetime table is read through provider_prompt_cache_lifetime

Pattern harvest

Rule candidate: review-prompt
Pattern: an owner module of the chat runner reads a name that tests patch, so the name is bound on the runner and imported by the owner only for type checking.

Related Issues

Builds on the merged cache-rebuilt footer (#35168), the hit-rate display (#31501) and opt-in idle compaction (#24384). No open issue or PR covers inference without counts.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 18, 2026 23:53
@javenciu
javenciu requested a review from iamwhatever November 18, 2026 23:53
@javenciu
javenciu enabled auto-merge (squash) November 18, 2026 23:53
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 99603c74743aa4ca4b916f1907b25c49e1107139 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the estimate is opt-in and labeled "likely", and with the setting off the footer is the same as before. It removes nothing, and screenshots of this revision are attached.

[DESIGN-REVIEWED] 99603c7

Description read

[DESCRIPTION-READ] 2e764dbf1088cd2e9337af3324ca22a06b57aabef2f10ca5139d40524137c4b2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 99603c74743aa4ca4b916f1907b25c49e1107139 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The new setting is off by default, so nobody gets the change without choosing it. The idle-gap rule reuses the existing provider_prompt_cache_lifetime. One leftover second spelling is a note.

Notes

  • Item 6: one consumer, generalized. The diff keeps note_turn_cache as "the measured-only form", but it is only judge_turn_cache(...) == CACHE_REWRITE. Its one production caller is the elif in _on_complete (turn_stats.py:991; grepped note_turn_cache in src/: 1 call). judge_turn_cache with started_at=None already returns only the measured verdict.
    Subtraction: delete note_turn_cache and the if/elif/else in _on_complete. Make one judge_turn_cache call that passes the gap keywords only when _cache_estimate_on() is true.

What this change ships

Inventory (8 items) — 7 justified

Intent: let users of a backend that reports no cache counts see when an idle pause probably rebuilt the prompt cache. ADDITION.

  1. New Settings → Advanced toggle "Estimate cache rebuilds" (dashboard.turn_cache_estimate), off by default — justified
  2. With it on, the footer shows "Cache likely rebuilt" plus a tip that says it is an estimate — justified
  3. Turn payload can carry "cache": "likely", and a measured rewrite outranks it — justified
  4. Slot remembers in memory when the previous turn ended (_prev_turn_ended_at) — justified
  5. A turn cut after its completion keeps the likely flag in its footer — justified
  6. Public judge_turn_cache, with note_turn_cache kept as a wrapper — one consumer, generalized: the wrapper repeats judge_turn_cache
  7. Two new strings in each of 13 locale catalogs — justified
  8. chat_runner binds provider_prompt_cache_lifetime as a patch surface — justified

[FIRST-PRINCIPLES-REVIEWED] 99603c7

Description read

[DESCRIPTION-READ] 1a8280eed2e5abf45c0adafaa56483e36eb75a0a33f714e852788789ecafd3bb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 99603c74743aa4ca4b916f1907b25c49e1107139 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing blocks. The blind reader read the new footer note, its info tip and the setting toggle correctly, and said they would use them.

Notes

  • Placement: the "Estimate cache rebuilds" toggle sits under "DASHBOARD" in the Advanced settings list. The blind reader said: "It is strange that a setting about chat replies sits under 'Dashboard'… I would not have looked for it there." Its sibling, showTurnStats, is in ChatSettings.tsx. Link the toggle from Chat settings, next to "Show turn stats".
  • The tip in shot-01/02 reads "…This is an estimate: the backend does not report cache counts.", but turn_cache_likely_hint in the diff ends "This is an estimate." Re-shoot so the screenshots match the shipped string.
  • The setting's help text uses words the blind reader called jargon: "backend reports no cache counts". Say it in the user's terms instead, for example: "Mark a reply 'Cache likely rebuilt' when it came after a long pause and your model doesn't report cache use."

[UX-REVIEWED] 99603c7

Description read

[DESCRIPTION-READ] f94c0b8fb463891d08539aa5898d844242ab91a4eeb22ce21d1cb654783c59a5

That sha256 names everything this verdict read: the PR description, plus the 10 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 99603c74743aa4ca4b916f1907b25c49e1107139 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 99603c7

Verdict parsed from the review's SHA-scoped output markers for commit 99603c74743aa4ca4b916f1907b25c49e1107139.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 99603c74743aa4ca4b916f1907b25c49e1107139 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 99603c74743aa4ca4b916f1907b25c49e1107139 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 99603c7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 99603c74743aa4ca4b916f1907b25c49e1107139 false-positive: <reason, 20 to 500 characters>

@javenciu
javenciu force-pushed the feat/cache-likely-flag branch from 5a4a039 to 4ba7a1f Compare November 19, 2026 01:16
@javenciu

Copy link
Copy Markdown
Contributor Author

Fixed in 4ba7a1f. _attach_abnormal_turn_stats now takes cache_likely: bool = False and passes it to turn_stats_meta, and _stamp_abnormal_turn_footer passes cache_likely=turn._turn_cache_likely, so a turn cut after EVENT_COMPLETE keeps "cache": "likely" the same way it keeps "rewrite". test_a_turn_cut_after_its_completion_keeps_the_flag_in_its_footer is now parametrized over rewrite, likely and none; the likely case fails on the previous head (assert None == 'likely') and passes here. The design-document paragraph is left for a separate docs change.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
A backend that reports no cache counts could never show the footer's cache-rebuilt marker. Add dashboard.turn_cache_estimate (off by default). With it on, a turn that starts at least the backend's cache lifetime after the previous reply ended, on a prefix of 20,000 tokens or more, is marked "Cache likely rebuilt", worded as an estimate. With it off, the footer is exactly as before.
@javenciu
javenciu force-pushed the feat/cache-likely-flag branch from 4ba7a1f to 99603c7 Compare November 19, 2026 02:50
@javenciu

Copy link
Copy Markdown
Contributor Author

5dfa726d6183 (Opus, docs/system-specs/modules/turn-stats-footer.md:58): fixed in 99603c7. The "Prompt-cache rewrite flag" section now names judge_turn_cache beside note_turn_cache, and its "never flagged" sentence holds with dashboard.turn_cache_estimate off (the default). A new paragraph describes the setting-on path: cache_likely_rewritten (previous context of the same session at least 20,000 tokens, no shrink, a turn start at _turn_t0 at least provider_prompt_cache_lifetime after the previous reply ended), the in-memory _prev_turn_ended_at baseline, the "likely" key, and that a measured "rewrite" outranks it. Code and tests are unchanged from 4ba7a1f; scripts/docs_lint.py and the comment-history gate pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: 32 files, more than 20.

An opt-in "Cache likely rebuilt" estimate based on the idle gap, behind dashboard.turn_cache_estimate (off by default). With the setting off, the measured verdict is unchanged; the only new effect is that _prev_turn_ended_at is reset, which makes the estimate more cautious. Both clock readings are monotonic. The lifetime lookup resolves through the runner's composed names. The slot field is keyed to the same backend session as the stored context size, compaction and synthesized completions stay unflagged, and both footer paths carry the flag. Tests cover the rule, the setting, a real idle turn with the setting on and off, cut-turn footers and the frontend note. CI shards passed.

Tier: 2, held for a person: 32 files, more than 20.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 99603c7474 · 368 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The review checked this head and found:

  • The setting is off by default (sections_dashboard.py, loader_builders.py, config-baseline.json). With it off, note_turn_cache gives the same measured verdict as before.
  • With it on, a measured rewrite always wins. Any reported cache count turns the estimate off. The existing reset, restart, /compact, synthetic-completion and mid-turn-compaction exclusions still apply, and a context that shrank is skipped.
  • Backends with no known lifetime (from provider_prompt_cache_lifetime) are never estimated.
  • The PR's 47 tests in test/test_turn_cache_rewrite.py pass on an extracted copy of the head.

Well done: the estimate uses the same per-slot baseline as the measured rule, so every existing exclusion covers it without a second set of checks.

Optional:

  • The real-turn test sets _prev_turn_ended_at before each turn and reports zero counts. Changing ended_at=time.monotonic() or cache_read=... in _on_complete (turn_stats.py:986-988) still passes. Asserting 0 < slot._prev_turn_ended_at after the run, and adding a turn that reports cache reads after an idle gap, would catch both changes.
  • time.monotonic() pauses while the host is suspended, so after a laptop sleep the estimate can miss a rebuild. It only under-reports. The idle-compaction timers already track suspended time if you want to add it.
  • The UX lane reports that the "likely" screenshots show an older tip string than turn_cache_likely_hint on this head. Re-shooting them would keep the evidence current.

@javenciu
javenciu merged commit e595eba into main Nov 19, 2026
107 of 172 checks passed
@javenciu
javenciu deleted the feat/cache-likely-flag branch November 19, 2026 11:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants