Skip to content

perf(dashboard): cache clean redaction verdicts for slot detail - #19624

Merged
buluoray merged 1 commit into
mainfrom
perf/slot-detail-redact-verdict-cache
Oct 13, 2026
Merged

buluoray merged 1 commit into
mainfrom
perf/slot-detail-redact-verdict-cache

Conversation

@buluoray

@buluoray buluoray commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Reopening a long chat session renders fast because already-checked text is not redacted again.

slot_detail runs every transcript string through the display-redaction battery (exfiltration URLs, then credentials) before it is shown. A cache already existed, but it stored every OUTPUT under an 8,000-entry / 16 MiB cap. One render of a long session redacts hundreds of thousands of strings, so each render evicted the previous one and the cache effectively never hit. Reopening a long session redid all the work every time.

Why it matters

Opening or switching to a long session is one of the most common dashboard actions. Under load, slot_detail p95 was 724-808 ms and it competed with tool calls for the same CPU; a repeat pass over 39 real transcripts took 34-37 s.

Not a goal

  • Changing what is redacted or how. The battery, its order and its patterns are untouched.
  • Redacting at write time. That is a follow-up PR stacked on this one.
  • Persisting any cache across restarts.

What changed (motivation → approach → change)

Symptom: repeat renders of long sessions are as slow as cold ones. Root cause: nearly every transcript string comes back from the battery unchanged (110 of 708k on a real corpus), yet each was stored as a full output entry, so the byte/entry caps were exhausted by a few thousand rows.

Change, all in src/kiro_crew/dashboard/chat_utils.py:

  • For a string the battery left unchanged, remember only a verdict: the first 128 bits of the same per-process HMAC-SHA256 key the output cache already uses (keyed by a random 32-byte salt and covering the exempt-host set), stored as one int together with the input byte length. The length is required: the MAC input text + NUL + hosts does not mark where the text ends, and the output cache already relies on its length component to tell such inputs apart.
  • Verdicts live in two bounded generations of 200,000 entries each. A hit in the old generation is promoted; when the young one fills it becomes the old one and the previous old one is dropped. Two full generations measure about 34 MB (86 B per verdict including its set slot).
  • Strings the battery does change still go through the existing output cache. Its entry cap rises to 40,000; the 16 MiB byte cap is unchanged.
  • A returned value is therefore always either the cached output for this exact key or the caller's own string that the battery already returned unchanged under the same key.

Why this over a bigger output cache: an output cache pays for the output bytes, which is exactly what made it thrash. A verdict costs a fixed small amount per string.

Measured

Gateway benchmark at a fixed offered request rate (so both sides do the same work), plus an offline replay of 39 real dashboard transcripts:

Metric Before After
slot_detail p95 724-808 ms 215-220 ms (−72%)
MCP tool-call p95 baseline 14-22% lower
Gateway RSS at end of load baseline flat
Repeat pass over 39 transcripts 34-37 s 6.7-7.3 s
Battery calls on repeat pass 196,927 0
Verdict memory held (replay) — ~19 MB (ceiling ~34 MB)

Byte-for-byte oracle against the previous code: 0 diffs over 773 transcripts, rendered cold and warm (1,546 renders).

Security notes

This touches display redaction, so the risk is a secret shown unredacted.

  • A verdict hit only skips the battery if the input's keyed 128-bit digest matches a string the battery previously returned unchanged in this process. The salt is secret and per process, so a forged or accidental collision is not practical.
  • The key covers the exempt-host set and the input byte length, so a change to allowed hosts produces new keys rather than reusing old verdicts, including a text crafted to end in NUL + host.
  • Changed content is a new key; nothing is keyed by message id.
  • Nothing is written to disk.

Backwards compatibility

Compatible: same function, same outputs (0-diff oracle); only internal caching changed.

Removes nothing: no key, field, function or record is removed; the cache only adds a verdict store.

Tests

In test/test_chat_utils.py:

  • test_long_session_renders_warm_with_zero_redactions: a 12,000-row session rendered twice must run zero redactors on the second pass, give identical output, and still hide the secret in the rows that leak one; editing a row to add a secret must redact again. Fails on the previous code (checked by restoring main's chat_utils.py: "a repeat render of the session must not redact again").
  • test_clean_verdicts_are_bounded_by_two_generations: with a small generation size, the verdict store never exceeds two generations.
  • test_clean_verdict_does_not_carry_over_a_narrowed_host_set: a clean text under hosts {a, b}, then the same text + NUL a.example.com under {b}, must still be redacted. Fails if the verdict key drops the input length (checked by stashing the fix).
  • The existing full-page cache test now counts verdicts alongside output entries.
  • test/test_session_transfer.py::test_the_import_prefix_is_streamed_not_joined now starts the verdict store empty. The import prefix write runs display redaction, and a store that earlier tests filled can double its hash table inside the traced window (4.4 MB peak with 75k verdicts held, against a 2.1 MB limit; reproduced locally). That is the bounded cache growing, not a copy of the prefix, which is what the test measures.

Local runs: every test file that references chat_utils / display redaction (2,467 passed, 25 skipped), black gate, flake8, isort, mypy on the changed module.

Manual verification

N/A — behavior is covered by unit tests and the byte-for-byte oracle above; no UI change.

Related Issues

Part of the gateway performance work. A follow-up PR (redact at write time) is built on top of this one and should merge after it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 12, 2026 23:59
@buluoray
buluoray requested a review from mbriones98 October 12, 2026 23:59
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f77384a82e5abbf7bb70309eb1a190f40067821f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of f77384a82e5abbf7bb70309eb1a190f40067821f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The output-cache cap raise to 40,000 rests on "35.6k changed strings", which contradicts the description's "110 of 708k" redacted. If only ~110 strings change, the old 8,000 cap is enough.

Not justified as shipped

  • Item 2 — inherited: the code comment says "39 real dashboard transcripts held 35.6k of them" (strings the battery changes). The description says "nearly every transcript string comes back from the battery unchanged (110 of 708k on a real corpus)". Both figures can't describe the strings the battery changes on comparable corpora. If 110 is right, an 8,000-entry cap never evicts them, and the verdict store (item 1) is the whole fix.
    Subtraction: change _DISPLAY_REDACTION_CACHE_MAX_ENTRIES back to SLOT_DETAIL_MAX_LIMIT * _DISPLAY_REDACTION_STRINGS_PER_ROW * 2.
    Clears when: the description says which corpus each number counts, and a repeat pass at the old 8,000 cap is shown to still call the battery.

What this change ships

Inventory (4 items) — 3 justified

Intent: reopening a long chat session renders fast because text already checked isn't redacted again. FIX (perf).

  1. Strings the battery left unchanged are remembered as a small keyed verdict in two bounded generations, so repeat renders skip redaction — justified
  2. The redacted-output cache holds 40,000 entries instead of 8,000 — inherited: rests on a figure the description contradicts
  3. The test cache-reset helper also clears the verdict store — justified
  4. The session-transfer memory test starts with an empty verdict store — justified

[FIRST-PRINCIPLES-REVIEWED] f77384a

Description read

[DESCRIPTION-READ] 1ae70a24c867da19d824f69f5e97cdd70c93aa60e7aeb66a9b96ff5cabb562dc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f77384a82e5abbf7bb70309eb1a190f40067821f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] f77384a

Description read

[DESCRIPTION-READ] 7d5fb1e2894daa7f7958feef830087c46468838761c702c38b7d744209dd9502

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f77384a82e5abbf7bb70309eb1a190f40067821f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f77384a

Verdict parsed from the review's SHA-scoped output markers for commit f77384a82e5abbf7bb70309eb1a190f40067821f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f77384a82e5abbf7bb70309eb1a190f40067821f: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f77384a82e5abbf7bb70309eb1a190f40067821f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f77384a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f77384a82e5abbf7bb70309eb1a190f40067821f: <one-sentence reason>

@buluoray
buluoray force-pushed the perf/slot-detail-redact-verdict-cache branch from c63cb29 to 10c1a42 Compare October 13, 2026 00:49
@buluoray

Copy link
Copy Markdown
Contributor Author

GPT 6.1 finding at chat_utils.py:2406 (clean verdict drops input length): confirmed and fixed in 10c1a42. The verdict key is now (length << 128) | digest[:16], matching the output cache, whose length component already separates T under {a,b} from T + NUL + a under {b}. Regression test test_clean_verdict_does_not_carry_over_a_narrowed_host_set reproduces the reported case and fails without the fix. Still one commit.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@buluoray
buluoray force-pushed the perf/slot-detail-redact-verdict-cache branch from 10c1a42 to 4a310ac Compare October 13, 2026 01:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 13, 2026
…ons render warm

The display-redaction cache stored every OUTPUT, so its 8,000-entry / 16 MiB
caps held a few thousand strings while one slot-detail render of a long session
redacts hundreds of thousands: each render evicted the last and nothing hit.

Nearly every transcript string comes back unchanged (110 of 708k on a real
corpus), and for those the input is the output. They are now remembered as a
128-bit slice of the same keyed digest (which already covers the exempt-host
set) plus the input byte length, in two bounded generations of 200k ints
(~34 MB ceiling); strings the battery changes still use the output cache, whose
entry cap rises to 40k with the 16 MiB byte cap unchanged. Content that changes
is a new key.

The verdict key keeps the input length because the MAC input text + NUL + hosts
does not mark where the text ends: a text ending in NUL + host under a narrower
host set would otherwise reuse a clean verdict and show a no-longer-exempt URL.

Replay of 39 real dashboard transcripts (the set the gateway bench views):
repeat pass 34-37 s -> 6.7-7.3 s, 196,927 -> 0 battery calls, ~19 MB held.
Byte-for-byte oracle vs the previous code: 0 diffs over 773 transcripts, cold
and warm (1,546 renders). test_long_session_renders_warm_with_zero_redactions
and test_clean_verdict_does_not_carry_over_a_narrowed_host_set fail on the
previous code.

test_the_import_prefix_is_streamed_not_joined starts the verdict store empty:
the import prefix write runs display redaction, and a store other tests filled
can double its table inside the traced window (4.4 MB peak at 75k verdicts),
which is the cache's growth, not a copy of the prefix.
@buluoray
buluoray force-pushed the perf/slot-detail-redact-verdict-cache branch from 4a310ac to f77384a Compare October 13, 2026 02:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 13, 2026
@buluoray
buluoray enabled auto-merge (squash) October 13, 2026 03:07
@buluoray
buluoray merged commit 2c79516 into main Oct 13, 2026
104 of 112 checks passed
@buluoray
buluoray deleted the perf/slot-detail-redact-verdict-cache branch October 13, 2026 05:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
buluoray added a commit that referenced this pull request Oct 13, 2026
#19624 caches clean display-redaction verdicts in memory, so a long session
renders warm only until the gateway restarts. This stores each row's verdict
(clean, or the redacted text) in a per-session sidecar under
<data>/display-verdicts/ once the transcript save commits, and reads it back on
the next open. The transcript itself is never rewritten.

- Tag: HMAC over (rules version, text length, text, exempt hosts) under a subkey of
  token_signing.key; a verdict whose tag does not match is ignored. While the
  owner-view switch bypasses the credential pass, nothing is stored or read.
- Rules version: sha256 of security.display_battery.RULE_SOURCES (the display
  battery and the files its output depends on), the native gate binary when
  one is loaded, and the Python and Unicode versions. A release that leaves
  the redaction rules alone keeps every stored verdict; a process whose rule
  files changed on disk since import stores and reads nothing.
- Fail closed: any read error, bad MAC, version mismatch or symlink
  (O_NOFOLLOW) means "not scanned"; the string goes through the battery.
- Bounds: one sidecar is cleared past 16 MiB, the directory past 512 MiB.
- A render reads the sidecar after 4 misses on a cold open (more misses than
  hits), or when the sidecar is at most 1 KiB per miss so far; a live render
  with a few new rows scans them rather than parse a long file. Verdicts the
  sidecar already holds are not appended again, so reopening does not grow it.
buluoray added a commit that referenced this pull request Oct 13, 2026
#19624 caches clean display-redaction verdicts in memory, so a long session
renders warm only until the gateway restarts. This stores each row's verdict
(clean, or the redacted text) in a per-session sidecar under
<data>/display-verdicts/ once the transcript save commits, and reads it back on
the next open. The transcript itself is never rewritten.

- Tag: HMAC over (rules version, text length, text, exempt hosts) under a subkey of
  token_signing.key; a verdict whose tag does not match is ignored. While the
  owner-view switch bypasses the credential pass, nothing is stored or read.
- Rules version: sha256 of security.display_battery.RULE_SOURCES (the display
  battery and the files its output depends on), the native gate binary when
  one is loaded, and the Python and Unicode versions. A release that leaves
  the redaction rules alone keeps every stored verdict; a process whose rule
  files changed on disk since import stores and reads nothing.
- Clean verdicts are stored under the second half of the 32-byte MAC and
  redacted ones under the first, so a tag copied between the files names
  nothing.
- Fail closed: any read error, bad MAC, version mismatch, symlink
  (O_NOFOLLOW) or FIFO (O_NONBLOCK) means "not scanned"; the string goes
  through the battery. Storage stays off where there is no uid (Windows).
- Bounds: the save-to-worker queue holds at most 256 batches and 32 Mi
  characters of rows; one sidecar is cleared past 16 MiB, the directory past
  512 MiB.
- A render reads the sidecar after 4 misses on a cold open (more misses than
  hits), or when the sidecar is at most 1 KiB per miss so far; a live render
  with a few new rows scans them rather than parse a long file. Verdicts the
  sidecar already holds are not appended again, so reopening does not grow it.
buluoray added a commit that referenced this pull request Oct 13, 2026
#19624 caches clean display-redaction verdicts in memory, so a long session
renders warm only until the gateway restarts. This stores each row's verdict
(clean, or the redacted text) in a per-session sidecar under
<data>/display-verdicts/ once the transcript save commits, and reads it back on
the next open. The transcript itself is never rewritten.

- Tag: HMAC over (rules version, text length, text, exempt hosts) under a subkey of
  token_signing.key; a verdict whose tag does not match is ignored. While the
  owner-view switch bypasses the credential pass, nothing is stored or read.
- Rules version: sha256 of dashboard.display_battery.RULE_SOURCES (the display
  battery and the files its output depends on), the native gate binary when
  one is loaded, and the Python and Unicode versions. A release that leaves
  the redaction rules alone keeps every stored verdict; a process whose rule
  files changed on disk since import stores and reads nothing.
- Clean verdicts are stored under the second half of the 32-byte MAC and
  redacted ones under the first, so a tag copied between the files names
  nothing.
- Fail closed: any read error, bad MAC, version mismatch, symlink
  (O_NOFOLLOW) or FIFO (O_NONBLOCK) means "not scanned"; the string goes
  through the battery. Storage stays off where there is no uid (Windows).
- Bounds: the save-to-worker queue holds at most 256 batches and 32 Mi
  characters of rows; one sidecar is cleared past 16 MiB, the directory past
  512 MiB.
- A render reads the sidecar after 4 misses on a cold open (more misses than
  hits), or when the sidecar is at most 1 KiB per miss so far; a live render
  with a few new rows scans them rather than parse a long file. Verdicts the
  sidecar already holds are not appended again, so reopening does not grow it.
buluoray added a commit that referenced this pull request Oct 13, 2026
#19624 caches clean display-redaction verdicts in memory, so a long session
renders warm only until the gateway restarts. This stores each row's verdict
(clean, or the redacted text) in a per-session sidecar under
<data>/display-verdicts/ once the transcript save commits, and reads it back on
the next open. The transcript itself is never rewritten.

- Tag: HMAC over (rules version, text length, text, exempt hosts) under a subkey of
  token_signing.key; a verdict whose tag does not match is ignored. While the
  owner-view switch bypasses the credential pass, nothing is stored or read.
- Rules version: sha256 of dashboard.display_battery.RULE_SOURCES (the display
  battery and the files its output depends on), the native gate binary when
  one is loaded, and the Python and Unicode versions. A release that leaves
  the redaction rules alone keeps every stored verdict; a process whose rule
  files changed on disk since import stores and reads nothing.
- Clean verdicts are stored under the second half of the 32-byte MAC and
  redacted ones under the first, so a tag copied between the files names
  nothing.
- Fail closed: any read error, bad MAC, version mismatch, symlink
  (O_NOFOLLOW) or FIFO (O_NONBLOCK) means "not scanned"; the string goes
  through the battery. Storage stays off where there is no uid (Windows).
- Bounds: the save-to-worker queue holds at most 256 batches and 32 Mi
  characters of rows; one sidecar is cleared past 16 MiB, the directory past
  512 MiB.
- A render reads the sidecar after 4 misses on a cold open (more misses than
  hits), or when the sidecar is at most 1 KiB per miss so far; a live render
  with a few new rows scans them rather than parse a long file. Verdicts the
  sidecar already holds are not appended again, so reopening does not grow it.
buluoray added a commit that referenced this pull request Oct 13, 2026
#19624 caches clean display-redaction verdicts in memory, so a long session
renders warm only until the gateway restarts. This stores each row's verdict
(clean, or the redacted text) in a per-session sidecar under
<data>/display-verdicts/ once the transcript save commits, and reads it back on
the next open. The transcript itself is never rewritten.

- Tag: HMAC over (rules version, text length, text, exempt hosts) under a subkey of
  token_signing.key; a verdict whose tag does not match is ignored. While the
  owner-view switch bypasses the credential pass, nothing is stored or read.
- Rules version: sha256 of dashboard.display_battery.RULE_SOURCES (the display
  battery and the files its output depends on), the native gate binary when
  one is loaded, and the Python and Unicode versions. A release that leaves
  the redaction rules alone keeps every stored verdict; a process whose rule
  files changed on disk since import stores and reads nothing.
- Clean verdicts are stored under the second half of the 32-byte MAC and
  redacted ones under the first, so a tag copied between the files names
  nothing.
- Fail closed: any read error, bad MAC, version mismatch, symlink
  (O_NOFOLLOW) or FIFO (O_NONBLOCK) means "not scanned"; the string goes
  through the battery. Storage stays off where there is no uid (Windows).
- Bounds: the save-to-worker queue holds at most 256 batches and 32 Mi
  characters of rows; one sidecar is cleared past 16 MiB, the directory past
  512 MiB.
- A render reads the sidecar after 4 misses on a cold open (more misses than
  hits), or when the sidecar is at most 1 KiB per miss so far; a live render
  with a few new rows scans them rather than parse a long file. Verdicts the
  sidecar already holds are not appended again, so reopening does not grow it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants