Repository navigation
perf(context): read a new session's memory beside its startup - #23269
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the early read is used only on an exact input match and falls back to today's read on any miss, so a revert removes it cleanly. Notes
[DESIGN-REVIEWED] 8679419 Description read[DESCRIPTION-READ] 50773988f621b44398ab9318d4f5e4d223362562f19ff3853289dedaeacd1484 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each item answers the measured first-turn wait (memory is 85–94% of a 0.79–4.6 s build), and any mismatch falls back to today's read. What this change shipsInventory (7 items) — 7 justifiedIntent: a new dashboard chat sends its first prompt sooner, because memory is read while the session is still starting. This is an ADDITION (perf).
[FIRST-PRINCIPLES-REVIEWED] 8679419 Description read[DESCRIPTION-READ] 48c4c06371374184cd679788c74255790897f4ee0a87bc57b2a1ae3c621d986a That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Needs changes. The input match is sound and the tests are strong, but one case the PR targets gets slower than main.
Head reviewed: 5ea5234. Bots read: Design PASS, First Principles PASS, UX skipped, Opus no findings, GPT changes requested. Readiness is still checking.
What I checked on this head:
test/test_session_memory_prefetch.py: 22 passed. Withtest_chat_runner_startup_recall.py,test_default_context_admission.pyand the omission tests: 141 passed.- Ordering: the reserve on the loop, the
startedanddoneevents, the pop in the take, and the discard on a failed or busy start. Two cold starts with the same text in one workspace only lose the head start, because the result is used only on an exact input match. context_omissions.replay: the memory reads only callrecord, neverclear, soupdatekeeps today's last-write semantics.- The branch merges clean with current main.
🟡 1: a prefetch still running past the recall bound is thrown away, and the build reads again from zero. See the inline comment.
The GPT lane finding on store_admission.py:551 holds on this head. get_context returns the whole V1 preferences file, and each of up to 16 entries keeps that string for up to 60 s. Only the entry count is bounded. A size check before entry.reads = reads that reuses the existing protected content ceiling, with the current fallback read on a miss, fixes it. I do not repeat it inline.
5ea5234 to
ca25add
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the quick turn. Both points from my last review are fixed on this head, and each fix has a test that fails when it is reverted.
Head reviewed: ca25add. Bots read: Design PASS, First Principles PASS, UX skipped, Opus changes requested, GPT changes requested. Readiness is still checking.
What I checked on this head:
test/test_session_memory_prefetch.py: 25 passed.- Restoring the old join bound makes 2 of the new tests fail. Removing the size check makes
test_a_result_over_the_size_bound_is_not_heldfail. - The slow read case from my last review no longer restarts the reads.
The Opus lane finding on store_admission.py:596 reproduces, and my earlier suggestion helped cause it. See the inline comment for numbers and a fix that keeps both cases fast.
The GPT lane finding on store_admission.py:521 also holds. The full message is kept as the key of a reservation with no length check, so 16 reservations of large pastes kept about 88M characters in my run. The build of the same turn usually takes the entry, so the cost is limited to entries nobody takes, for up to 60 s. Skipping the prefetch when the message is longer than _MEMORY_PREFETCH_MAX_CHARS, before the reservation, closes it. I agree with the adjudication that the history cache finding is low.
ca25add to
1e859ac
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the update. Both points from my last review are fixed on this head, and each fix has a test that fails when it is reverted.
Head reviewed: 1e859ac. Bots read: Design PASS, First Principles PASS, UX skipped, Opus PASS, GPT changes requested. Readiness is action required.
What I checked on this head:
test/test_session_memory_prefetch.py: 28 passed.- Removing the deadline cut in
_take_memory_prefetchmakestest_the_build_cuts_the_prefetch_embed_deadline_to_its_ownfail. Not recording the work also makestest_the_prefetch_records_the_embed_budget_it_runs_underfail. - Slow first embed repro (embed 0.8 s, recall bound 0.2 s): the build is ready at 0.24 s with the prefetch, 0.77 s with the cut disabled, and 0.21 s with no prefetch. The regression from my last review is gone.
- The table now keys the message by its SHA-256, so a large paste is no longer held as a key.
One point blocks: the GPT lane finding on store_admission.py:457 reproduces. See the inline comment.
I agree with the adjudication on the other two GPT findings. I reproduced the timed out embed case: the build takes a keyword only result after the vector is cached. It costs one prompt and the next turn corrects it.
1e859ac to
21667dd
Compare
Withdrawing my request for changes: the link following stat is fixed on 21667dd. Details in my new review.
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the update. The point from my last review is fixed on this head, so I am withdrawing my request for changes.
Head reviewed: 21667dd. Bots read: Design PASS, First Principles PASS, UX skipped, Opus PASS, GPT PASS after adjudication.
What I checked on this head:
test/test_session_memory_prefetch.py: 32 passed.- My earlier repro, a link inside the history directory to a file outside the tree plus a dangling link: the outside file's size is no longer counted and the result is no longer
None. Nothing is followed. - A full build with the prefetch and a planted history link follows no link, the same as main.
- With the previous
store_admission.pyrestored, four of the new tests fail.
One small gap, see the inline comment: the new tests do not pin the in directory link case.
On the lanes: I reproduced the GPT finding (the prefetch warms the five second history cache, an edit through another handle within that window shows old history in one prompt) and the Opus note (a held entry older than the 60 second TTL is still taken). I agree with both being low: main shows the first with any read in the same window, and the second only differs in day level decay because every other input is compared again.
This is a performance change, so it needs a maintainer's approval.
|
|
|
|
|
|
|
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head cc811d4 (force push of one squashed commit; the tree delta over 21667dd is the join bound, the take time TTL check, the in directory link test and the link screen site entry).
Fixed since my last review:
- The in directory link case is now pinned. Reverting
entry.stat(follow_symlinks=False)toentry.stat()in_dir_lstatturnstest_a_link_inside_the_history_directory_is_never_followedred (1 failed, 34 passed). - The join now ends at the build's own embed deadline. Removing the
bound = min(...)line turnstest_the_join_ends_at_the_build_embed_deadlinered. - A prefetch past its TTL is not taken. Removing that check turns
test_a_prefetch_older_than_its_ttl_is_not_takenred.
The prefetch suite passes on head (35 passed).
One blocking issue remains, and it matches the Opus lane finding on this head, which I reproduced: with memory.startup_recall_timeout_secs: 0 the cold start prewarm still embeds the first message beside session start, through the new memory prefetch. Details and a verified one line fix are inline.
This is a Tier 2 perf change, so I am leaving a comment for the maintainers rather than approving.
| # within its own deadline. A pool too busy to start it first leaves the build to | ||
| # embed for itself, as it does without this. | ||
| await asyncio.sleep(0) | ||
| if not prefetch_memory: |
There was a problem hiding this comment.
The zero recall timeout opt out no longer keeps the first message embed out of session start
embed_startup_query returns early when _startup_recall_timeout_secs() <= 0, and the config spec says 0 turns the early embedding off. The new branch below it does not check that setting, so reserve_memory_prefetch and _prefetch_session_memory_off_loop still run. prefetch_session_memory then calls read_session_memory, whose activity ranking embeds the message under a 30 s interactive deadline while the session is still starting.
if not prefetch_memory:
return
# Reserved on the loop, before the thread exists: ...
reserve_memory_prefetch(workspace, memory_store, text)Repro: a real MemoryStore with a VectorMemoryStore whose embed_fn counts calls, then _prewarm_startup_query(..., prefetch_memory=True) with the timeout patched to 0. Head: 2 embeds of the first message during prewarm. Main with the same setup: 0. With the timeout at 1 the early embed runs on both, as intended.
Suggestion, verified on head (the repro drops to 0 embeds at timeout 0 and still embeds at 1; test_session_memory_prefetch.py and test_chat_runner_startup_recall.py pass, 53 passed):
if not prefetch_memory or _budgets._startup_recall_timeout_secs() <= 0:
returnPutting the same check inside prefetch_session_memory instead breaks test_a_prefetch_running_past_the_recall_bound_is_joined_not_restarted, which patches the timeout to 0, so the call site is the better place. A test that drives _prewarm_startup_query with the timeout at 0 and asserts no prefetch is reserved would pin it.
Rule: a documented config opt out must keep covering every path that does the work it turns off.
cc811d4 to
8679419
Compare
|
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head 8679419 (force push of one squashed commit; the tree delta over cc811d4 is the new timeout check in prefetch_session_memory, its test, the adjusted join test and the two config doc lines).
Fixed since my last review: with memory.startup_recall_timeout_secs: 0 the cold start prewarm no longer embeds the first message early. I reran the same repro as last time (a real MemoryStore with a vector store whose embed function counts calls, then _prewarm_startup_query(..., prefetch_memory=True)):
- Timeout 0: head 0 embeds during prewarm, main 0. Last head was 2.
- Timeout 1: head still embeds early, as intended.
The check sits inside prefetch_session_memory rather than at the call site I suggested, and that works: the entry is reserved but never marked started, so _take_memory_prefetch does not wait for it and the build reads for itself. The join test now uses a 0.001 s bound instead of 0, which still keeps it far below the held reads, so it tests the same join. Removing the new check turns test_turning_the_early_embed_off_turns_the_early_reads_off red (1 failed, 35 passed) and brings the repro back to 2 embeds. On head, test_session_memory_prefetch.py and test_chat_runner_startup_recall.py pass (54 passed). The head also merges cleanly onto current main, and the prewarm wiring survives the recent chat runner split.
Bots on this head: Design PASS, First Principles PASS, UX skipped, Opus no findings, GPT no findings. Several required checks were still running when I read it.
No blocking issues from me. This is a Tier 2 perf change, so I am leaving a comment for the maintainers rather than approving; the remaining step is maintainer approval once readiness passes on this head.
buluoray
left a comment
There was a problem hiding this comment.
Approving at 86794191986a2f6fa35a6c8bd6b804f031025612.
What it fixes: a new dashboard chat that starts its session on the send path no longer waits for KAS startup before reading its first-turn memory; the reads run beside the startup and the build takes them only on an exact input match.
Checked against source:
- Same blocks, same order:
read_session_memorymakes the sameget_context/activity_index/get_activity_context/ranks_activity_againstcalls with the same arguments,session_memory_partsappends them where it did, and omissions reach the build throughcontext_omissions.replay. - The compared inputs cover every source those readers touch today (prefs and projects via lstat, the history directory via a no-follow scandir, the vector store's scoring generations plus
PRAGMA data_versionon its single locked connection). - Every wait has a bound: a never-started prefetch is not waited on, the join ends at
min(_PROMPT_BUILD_EMBED_TIMEOUT_SECS, own deadline), and the prefetch's own embed deadline is cut to the build's. The reads run on the recall pool and the build on the embed pool, so the join cannot starve itself. - The loop only does reserve/discard (a lock, a dict of at most 16 entries, one sha256); no file IO on it.
- Any prefetch failure leaves
reads=Noneand setsdone, so the build reads for itself; member, private, temporary and blocked-reads turns never prefetch.
Tests: test/test_session_memory_prefetch.py 36 passed; with test_chat_runner_startup_recall, test_chat_runner_composition_contract, test_context_omissions, test_memory_v1_golden, test_memory_on_demand, test_default_context_admission, test_start_priority and the other files that touch the prewarm path, 862 passed. Full suite not run.
Mutations, one at a time, each restored before the next:
- waiting on a never-started prefetch: red
test_a_prefetch_that_never_started_is_not_waited_for - unbounded join: red
test_a_prefetch_running_past_the_recall_bound_is_joined_not_restarted,test_a_stuck_prefetch_is_given_up_on_at_the_build_embed_deadline,test_the_join_ends_at_the_build_embed_deadline - prefetch embed deadline not cut: red
test_the_build_cuts_the_prefetch_embed_deadline_to_its_own - join not cut to the build's own deadline: red
test_the_join_ends_at_the_build_embed_deadline - write-during-reads check removed: red
test_a_preferences_write_during_the_reads_drops_the_prefetch - take-time input compare removed: red
test_a_memory_file_written_after_the_prefetch_is_read_again[prefs|projects|history] startednever set: redtest_a_build_from_a_prefetch_is_byte_for_byte_the_build_without_oneand the two join tests- TTL not checked on take: red
test_a_prefetch_older_than_its_ttl_is_not_taken - recall-off opt-out ignored: red
test_turning_the_early_embed_off_turns_the_early_reads_off - missing
data_versionmatches: redtest_a_store_that_cannot_see_other_writers_never_matches - failed start keeps its prefetch: red
test_a_failed_session_start_drops_its_prefetch prefetch_memoryflag ignored: redtest_a_cold_start_reads_memory_while_the_session_starts,test_a_prewarmed_session_starts_no_memory_prefetch
Three mutants survive, all on cleanup or ordering paths whose cost is bounded by the 60 s TTL and the 16-entry cap: dropping the discard on the SessionBusyError re-raise, dropping the discard in _prefetch_session_memory_off_loop's except, and moving reserve_memory_prefetch after the prefetch task is created. Worth a test each, not blocking.
CI on this head: 87 success, 16 skipped, 0 failing (latest run per name); legacy statuses success, PR Readiness passed. Merges cleanly onto current main with the prefetch wiring intact.
Non-blocking follow-ups:
- Two cold starts with the same workspace, store and first message close together share one table key: the second reserve replaces the first entry, both prefetch threads can write that one entry, and
readsandinputsare assigned separately, so a build has a very narrow window to pair one thread's result with the other's inputs. Handing the reserved entry object to the thread (or assigning(reads, inputs)as one tuple) closes it. Found by reading the code, not reproduced. - When the reads are slow for a reason other than the embed (disk, SQLite lock), the build joins up to its own deadline and then reads again, so the worst case is about one embed deadline above main. Bounded, fine as is.
- Design's two notes are still open: pin
_memory_read_inputsto the readers it mirrors with a test, and consider moving_prefetch_session_memory_off_loopnext toprefetch_session_memoryinstead of growingchat_runner.py. - The bench is single-session; the motivating case is several new sessions at once. A run with that concurrency would show whether the recall pool holds up.
bolichen97
left a comment
There was a problem hiding this comment.
Approving head 8679419. Rechecked on this head: readiness passed, PR Readiness green, every required check green, all AI review lanes pass, no open change requests, mergeable against main. The early memory read falls back to the original read path whenever an input moved, and the new prefetch test pins that behavior. Thanks.
Problem / Motivation
Goal: A new dashboard chat that starts its session on the send path reads its first-turn memory while the session starts, so the prompt leaves soon after the session is ready.
Today the first-turn context is built only after KAS startup ends (spawn,
session/new, managed MCP wait: ~2–7 s). Memory reads are most of that build. On the R8 gateway bench (6 new sessions,7d9673ba7b), the build took 0.79 s in a normal run and up to 4.6 s under load. Memory took 0.68 s and up to 3.3 s of that, which is 85–94%. #21923 already starts the first message's embedding early. The memory reads that use it still wait for the session.Why it matters
Every new chat sent at once pays this wait before its first word: 0.5–0.8 s on an idle host and 3–4.6 s on a busy one. It is the largest part of the first-turn build. It sits on the same path as the 8.95 s send → prompt-at-KAS gap that R8 measured.
Not a goal
What changed (motivation → approach → change)
The build waits for the session, but its memory reads need nothing from the session. So a cold-start dashboard turn now starts those reads at the same time as the early embed, beside the session start. The build uses their result only if nothing they read has changed. If anything did, it reads again, as today.
dashboard/chat_turn/turn_admission.pydashboard/chat_runner.pycontext_assembly/store_admission.pyread_session_memoryholds today's reads in today's order. The build takes the early result only on an exact input match: same store and vector store, message, caps, activity switch, date, thelstatof the prefs, projects and history files (read only after the memory reader's own root and leaf link checks pass; otherwise the prefetch is off), and the vector store's validity token (scoring generations +PRAGMA data_version). These are taken before and after the early reads, and a move during them drops the result. Otherwise it reads for itself. Reads still running started first, so the build joins them instead of reading twice. It first cuts their embedding deadline down to its own, and the join ends at that same deadline, never later than_PROMPT_BUILD_EMBED_TIMEOUT_SECS. So neither waits past what the build's own read would have spent. It never waits for reads that have not started. A result over 512K characters is not held. The message is kept only as a SHA-256 digest, and a result older than 60 s is neither held nor taken.context_omissions.pyreplay()hands the early reads' cut records to the build that takes them.context.py, docscontext-management.mdandmemory-skills-hooks.mddescribe the early reads.Same blocks, same order, same budget: the build calls the same reads with the same arguments, and only where they run changes. The memory caps do not depend on the model window (
_resolve_caps_cached), and a test pins that.Backwards compatibility
Compatible: same prompt text and omission records. A miss of any kind falls back to today's reads, and no config, API or stored data changes.
Removes nothing: the change adds an early read path and keeps today's read as the fallback.
Tests
test/test_session_memory_prefetch.py, realContextBuilderandMemoryStore. Invariant: a build from the early reads is byte-for-byte the build without them, and any miss reads again.data_version), or nodata_versionat all each mean no match. A real store's token holds across its own reads.memory.startup_recall_timeout_secs: 0turns the early reads off with the early embed, since they would embed the message just as early.src/kiro_crew/docs/configuration.mdand the config spec say so.get_or_createreturns. A failed start drops them. A pre-spawned session starts none.Red on
origin/main: every test that touches the new path fails. The caps test passes on main as a guard.Mutations: 22, each run against the test that names it, and all of them turn it red. They are:
data_versionmatchesDeterminism: two early reds had one cause. Two compared builds straddled a minute boundary, so their
[CURRENT DATE]lines differed. The test helper now blanks only that line, and it asserts that the line appears exactly once. After the fix, 40 of 40 repeats pass at-n 0, and the file passes at-n 2. ContextVar budgets in the tests are restored by value, as the repo's Semgrep rule on ContextVar resets requires. Reversed order green. The flake ratchet is green. Related tests:test_context*,test_memory*,test_lesson*,test_chat_runner_*,test_default_context_admissionand others, 3,792 passed with-n 2.Manual verification
Before/after bench:
research/crewmate-perf/g3/bench_g3.py. It uses a copy of the live home's memory (135 MBmemory.db, real embedding model) and the real dashboard code. That covers the early embed, the early reads (after only) and the realbuild_messageon the embed pool. KAS startup is a sleep of S. New session sent at once, fresh message per run, arms interleaved, n 5 each, 1-min load 6–8:The 0.1 s and 0.3 s rows are startups shorter than the memory reads: the build joins the running reads instead of starting over, so it is still never slower than main. The ~100 ms left is the rest of the build (identity, skills, lessons). That is option A's share. Under the loaded host case R8 saw (memory 2.4–3.3 s), the saving grows with it, as long as startup is longer than the reads.
Related Issues
Refs #23258
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)