Repository navigation
perf(dashboard): re-plan a slot page when the memo evicts its rows - #21110
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check one case: rows shared by two scopes of one session (newest page and whole history). An evicted row is charged only to the scope that read it last, so the other scope stays "filled" and misses that row. [DESIGN-REVIEWED] 28d4fc9 Description read[DESCRIPTION-READ] 4d08566fb1f631b9d067b16d04172f7bb0bc8c9ab061a664afb25f0ceb33eacf That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that the 8-read plain cooldown pays for its two extra states: the bench has no arm that just forgets an evicted page. Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: make plain slot-detail reads of long sessions stop paying for row-memo lookups that mostly miss. This is a FIX (perf).
[FIRST-PRINCIPLES-REVIEWED] 28d4fc9 Description read[DESCRIPTION-READ] 4720fbd469b7eec011ba38221677aecc7c68e87791cd6baacab085f76a025acc That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
aeed883 to
e406a44
Compare
e406a44 to
d03b0ac
Compare
d03b0ac to
08fd4d5
Compare
Zedmor
left a comment
There was a problem hiding this comment.
Measured end to end on today's main: approving
This fills in the end-to-end figures the PR body lists as not measured. I re-ran the part-2 gateway benchmark (isolated gateway, packaged fake ACP backend, fixed 12 req/s mix over 9 endpoints for 240 s, every request traced) on three trees side by side:
- part-2 main
5e2f1f7485 - main
8fd2af91d9(includes #21108) - main + this PR:
git merge-treeof main and08fd4d5638. The merge is clean and only this PR's 4 source files differ from main.
Each tree got 3 rotated rounds on the part-2 home, 1 round on the steady-state home, a py-spy --gil run, and in-process micros. Host load was 2.6–6 on 16 cores, with no errors, probe failures or loop stalls.
Slot detail, end to end
| main | main + #21110 | |
|---|---|---|
| Steady state, t ≥ 30 s, 1,647 requests pooled, p50 / p95 / p99 | 42 / 198 / 284 ms | 45 / 85 / 133 ms |
| Whole 240 s window, mean of 3 rounds, p95 / p99 | 218 / 419 ms | 102 / 227 ms |
| Steady-state home, 1 round, p95 / p99 | 230 / 365 ms | 112 / 278 ms |
| Gateway CPU, 240 s load | 330 s | 313 s (−5%) |
GIL held by slot-detail work per request (py-spy --gil) |
29.6 ms | 27.7 ms (−6%) |
This PR has the lowest slot-detail tail in every round. At steady state, other endpoints are flat or slightly faster: session list p95 65 → 46 ms, memory recall p99 100 → 70 ms. Nothing gets slower beyond noise.
Why the tail halves while GIL hold barely moves
Under the mix, main's memo still hits often, so average GIL time is close. What changes is that a plain read no longer serializes and BLAKE2b-hashes every source row. hashlib drops the GIL for inputs over 2 KB, so a long read gives the GIL away thousands of times and waits to win it back each time.
In-process measurement on the part-2 home's 6 long sessions, whole history, median of 3 rounds:
| main (memo path) | this PR (plain) | |
|---|---|---|
| Repeat read of one session (memo hit), CPU | 31.3 ms | 34.5 ms |
| Cycling the 6 sessions (memo evicting), CPU | 78.8 ms | 27.2 ms |
| Same cycle with 2 busy Python threads, wall | 5.68 s | 2.60 s |
Main's own plain path measures 27.5 ms on the cycle, so the display-key change is worth about 1–6% on real sessions.
Your bench on this host
test/perf/slot_render_bench.py on the merged tree, 3000 rows, p50 render CPU (verdicts off / on):
memo_hit: 33 / 34 msmemo_evict: 267 / 240 msplain: 91 / 83 msplain_old: 189 / 164 ms
Oracle differences: 0. Same direction as your table.
Tests
test_slot_detail_render_memo.py and test_slot_render_context_key.py: 23 passed on the PR merged into 8fd2af91d9. CI is green.
Notes (none blocking)
- Trade-off, fine: a repeat read of an unchanged long session costs ~10% more CPU than a memo hit did, and slot p50 moves 42 → 45 ms. The tail gain is much larger.
wire_render.prepare_messages_jsonhas no production caller left; only tests and the bench use it. Either drop it or mark it as a test seam.prepare_message_rows(use_memo=...): no caller passesuse_memoany more.- Follow-up: the 16 MiB / 60k-entry row memo now serves only cursor reads (the embed's 200-row pages), so it can be much smaller.
- Bench:
plain_oldgoes throughunittest.mock.patch, which adds a call layer per string. Here it reads ~2× slower thanplain. The 3–9% in the PR body matches what I see on real sessions (1–6%); the bench's own gap overstates it. - Security: on a cursor read whose redaction context moved, the PR re-renders every memo hit, stores nothing and issues no identities. Misses use the live context, as an unmemoized render would. Looks right.
Nice work, approving.
|
HOLD at 08fd4d5. A human needs to review this one. The perf change looks correct, and Zedmor's end-to-end numbers back it up. However, it changes when memoized redacted output is served or stored ( ), and it changes how the display-redaction cache key is built (chat_utils_display.py L372). Redaction-path changes need a human reviewer. I found no defect: the exempt sets are frozensets, so the cached key suffix cannot go stale.Other blockers:
— crew-pr-reviewer (r111) |
08fd4d5 to
02f2d41
Compare
02f2d41 to
cf4b41c
Compare
|
|
|
cf4b41c to
b2b2058
Compare
|
|
Note on the description: at Ray's request, the maintainer's crewmate replaced an internal link in Problem / Motivation with plain text ("an internal performance review"), because this is a public repo. Nothing else in that section changed. |
A slot page stayed marked filled after the row memo's cap evicted its rows, so every later plain read looked each row up, missed, and paid the row's serialization and hash, a re-parse and the render: about 108 ms at 3000 rows against 60 ms for the plain render. A reader cycling through more history than the memo holds paid that on every read. Each memo entry now records the page that last stored or read it. When the cap evicts an entry, a filled page a read has used is forgotten, so its next read is the plain render and schedules a fresh fill. A fill the cap undoes before any read used it marks the page plain for eight reads before it is filled again, so a cycling reader does not pay a fill per read either. Pages that stay whole keep their memo hits. A memo read digests the redaction context before and after its rows. When the context moved, every row the memo answered is rendered again against the live context, nothing from that read is stored and no row identity is issued, so an exemption withdrawn mid-read is never served from an entry filed under it. The row key is unchanged: no cheaper identity covers every field of a stored row, which can be edited in place.
b2b2058 to
28d4fc9
Compare
|
|
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Every review lane on the head is clean (Design PASS, First Principles PASS or skipped, UX skipped; no findings); held by the triage cron on category or path rules alone. Approved on the watch holder's request.
|
Post-merge stress review found a performance case the static benchmark misses. With a sliding 50-row page, memo pressure from another session, and a cap near 2.5 pages, evicting rows that already slid off the current page resets the page scope even while all current rows remain present. A local 40-round comparison observed 40 memo-backed reads on main versus 13 here, about 37% higher request-path CPU, and eventual decay to 12 of 50 cached rows. Please reproduce this moving-page case under pressure. Scope ownership likely needs a read epoch, so only eviction of an entry from the latest read can reset that scope. |
Problem / Motivation
Goal: A plain slot-detail read renders its rows directly again instead of paying for a row-memo lookup that mostly misses.
Follow-up to #20153, from an internal performance review ("Slot render is now the largest GIL consumer", Figure 5.7). Since #20153, a plain slot-detail read serializes and hashes every row to look it up in the row memo. Under the report's 12 req/s mix the long sessions a reader cycles through outgrow the memo's cap, so most lookups miss and pay identity + re-parse + render. Pages whose rows were evicted also stayed marked "filled" and kept taking that path.
Why it matters
Slot render is the largest GIL consumer in the report. On main, a plain read of a page whose rows were evicted (the common case under the report's mix) costs 108.1 ms p50 render-thread CPU at 3000 rows, against 59.2 ms with this change (see Numbers below).
Not a goal
What changed (motivation → approach → change)
Root cause: a page stayed marked filled after the memo's cap evicted its rows, so every later plain read of it looked each row up, missed, and paid the row's serialization, hash and re-parse on top of the render. The memo itself stays: a page that is still whole (one open session refetched after each turn) keeps its hits.
src/kiro_crew/dashboard/wire_render.py: each memo entry records the page (plan_readscope) that last stored or read it. When the cap evicts an entry:src/kiro_crew/dashboard/chat_api/slot_detail.pypasses the page scope to the plain and the cursor render.test_an_exemption_dropped_mid_read_is_not_served_from_the_memofails there.docs/system-specs/modules/history.mddescribes the eviction rule and the context check.Numbers
test/perf/slot_render_bench.pydrives each read the way the handler does (plan_read, then the memo or the plain render, then a first read's fill) against main'swire_render.pyloaded side by side. Render-thread CPU (= GIL hold) per read in ms; "fill" is the mean fill cost per read, which the gateway runs in a worker thread. 3 rounds, arms alternated, host load 6.8–7.9, every output byte-identical to the plain render.Mean CPU per read including fills, 6-session case at 3000 rows: 109.4 ms on main, 50.3 ms here. The evicted row measures the read right after another page's fill evicted part of a page no read had used yet, which is the plain render here.
Backwards compatibility
Compatible: every read returns the same bytes as
_prepare_messagesof the stored page; the display key and the row key are unchanged;prepare_messages_jsonandprepare_message_rowsgain an optionalscopekeyword and keep every existing argument.Removes nothing: no function, field or key is removed or renamed.
Tests
test/test_slot_detail_render_memo.py(27 tests):TestEvictionForgetsTheFill: a filled, used page whose row was evicted is read plain and refilled; a page that cannot stay whole through its fill is read plain for 8 reads, then refilled; a fill evicted before any read used it waits the same way; an evicted row is charged to the page that read it last; eviction never changes the bytes. The first four fail on main ((True, False)where the plain render is expected).TestContextMovedMidRender: a context change mid-render stores nothing and issues no identity; a withdrawn exemption is not served from the memo. Both fail on main.-n 4) and one run in reversed node order, all green.Related Issues
Follow-up to #20153. Source: an internal performance review (part 2).