Repository navigation
perf(memory): read a new chat's history while the person types - #37595
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each item has a measured harm or is a deletion. The prefetch's count/mtime/size What this change shipsInventory (7 items) — 7 justifiedIntent: shorten the wait for the first reply in a new chat by reading the chat's recent history while the person is still typing. ADDITION (perf).
[FIRST-PRINCIPLES-REVIEWED] 36fc6cc Description read[DESCRIPTION-READ] 61b0ea24d7bc9eb2d5869e0affb6c075778d327653ffbccf1ede0e40b71ec380 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the warm read is reused only on a matching lstat signature, at most once, within 60 s, and only on POSIX. Every other read keeps the 5 s TTL. Notes
[DESIGN-REVIEWED] 36fc6cc Description read[DESCRIPTION-READ] 5a5eccf95e05a4b1f55070cb929d72ffe53bfae12cdb0a0e7b0b82bb99ab3bf8 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 421 lines, more than the 400 it may approve on its own. A fresh pre-warm now reads the 14-day history in the background. The first build reuses that read once, within 60 s, and only when the history directory's lstat signature (taken before and after the warm, POSIX-only) is unchanged. Nothing else changes: the TTL path stays as it was, append, prune and cache reset clear the reuse, and V2, temporary and member slots are skipped. The early An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
aa9ea20 to
85b4009
Compare
85b4009 to
f0bcf06
Compare
|
Fixed on f0bcf06, all four findings on aa9ea20:
|
f0bcf06 to
dbdf5f7
Compare
|
dbdf5f7, for f0bcf06: GPT's unbounded-signature finding is fixed with a 512-entry cap (_HISTORY_SIGNATURE_MAX_ENTRIES). A fuller directory gets no signature and no reuse, and the scan stops at the cap. New test test_a_crowded_directory_gets_no_signature fails without it. Flake Pattern Ratchet K19: the 3600 s TTL in the new test is never waited, so it now carries flake-ok. Rebased on bdff21a. |
dbdf5f7 to
e0b9f96
Compare
|
e0b9f96, for the First Principles BLOCK on dbdf5f7:
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Thanks for this. The memory-side reuse tests are solid. One item in the new eager-spawn test needs a fix before this can merge.
test/test_memory_history_reuse.py:200:test_a_fresh_prewarm_reads_the_history_earlyimports_bindings, _cfg, _mock_statefromeager_spawn_helpersinside the function body. Because of that, the autouse fixtures_pin_prewarm_allowanceand_isolate_armed_registryare never registered for this module._eager_spawnthen reads the real host memory throughresource_status.prewarm_allowance(). On a host with little free memory (allowance 0),_admit_prefetchrefuses the prewarm and the new warm block is never reached. The(True, False, True)case then fails withwarm_recent_historycalled 0 times. The test also leaves at1entry in the module-global_armed_prefetchesregistry for the rest of the worker. With the allowance pinned to 0, this test failed and thetest_eager_spawn.pytests passed. Smallest fix: addfrom eager_spawn_helpers import _isolate_armed_registry, _pin_prewarm_allowance # noqa: F401 -- fixtures are used by nameat module scope, as the siblingtest_eager_spawn*.pymodules do. You can also move this test intotest_eager_spawn.py, which already has both fixtures.
Done well: the reuse is kept only when the signature taken before the read matches the one taken after. The symlink-swap test asserts that neither the outside file nor the old cached text comes back.
Optional:
docs/system-specs/modules/memory-skills-hooks.md("History reads", around line 359) still describes only the 5 s TTL. Could you add the 60 s signature reuse, the 2048-entry cap, the POSIX-only gate, thewarm_recent_historycaller, and the shared_history_signature?- The 60 s reuse applies to every
read_recent_historycaller (suggestions, tips, the memory graph,read_history), not only the warmed first build. Is that intended? If it is not, letting only an entry written bywarm_recent_historybe reused would keep the 5 s bound for the other readers. - In the warmed case, asserting
get_memory_forcall args would pin which store gets warmed.
A message sent in a chat whose session was pre-warmed still waited for its prompt build, and about a third of that build was reading every recent-history file through the hardened reader. None of it depends on the message. The pre-warm of a fresh session now calls the new MemoryStore.warm_recent_history in the background. That read writes no plain 5 s TTL entry, so the build cannot take a TTL hit that checks nothing. It is kept with the history directory's lstat signature (name, mode, inode, size, mtime, ctime, link count of every entry, plus the directory's own), and the next read_recent_history of that window reuses it once, within 60 s, only while the signature is unchanged. Any rewrite, new or removed day, relink or symlink swap reads every file again through the same reader, and append/prune drop it as they drop the cache. Reads nobody warmed keep the plain TTL, and a TTL hit lists nothing. MemoryStore._history_signature is now the one signature of the history directory: store_admission._admitted_file_stats compares it too, in place of its count/max-mtime/size _dir_lstat, which is deleted. It holds at most 2048 entries; a fuller directory gets no signature, so no reuse and no prefetch match. The memory spec describes the warm, the reuse and the signature. Windows gets no reuse and no warm read: DirEntry.stat reports 0 for inode and link count there, and an lstat through a linked ancestor is the SMB probe that LocalMemoryFiles._read_root_guard exists to refuse, so the reuse path, which runs no root guard, never takes the signature. The prefetch takes it only behind that guard. On a pod with a copy of a real store (lessons 36.9k, memory 30.3k chars), sending 7 s after "New chat": build_message median 939 -> 662 ms, paired difference -207 ms over 10 interleaved pairs (all ten negative), against an A/A paired difference of -9 ms.
e0b9f96 to
36fc6cc
Compare
|
36fc6cc, for KiroCrewCaptain's review on e0b9f96:
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving on 36fc6cc. I checked these on this head. A warmed read writes no TTL entry. It is kept only when the directory signature taken before the read matches the one taken after. The next read of the same window uses it at most once, within 60 s, and only while the signature, the day and the bound all still hold. Every other outcome reads every file again through the unchanged hardened reader. The warm runs off the event loop through asyncio.to_thread, only for a new, non-resumed, non-temporary chat. Windows skips both the warm and the reuse.
Earlier review: the required item is fixed, since _isolate_armed_registry and _pin_prewarm_allowance are now imported at module scope (test/test_memory_history_reuse.py:17-20). The spec note is added in memory-skills-hooks.md. Only warm_recent_history now writes the reuse entry. The store pin is still partial; see below.
Done well: the prefetch now compares the same _history_signature, which removes the second, weaker signature. The symlink-swap test checks that neither the outside file nor the old cached text comes back.
Security: the sensitive-path and realpath checks can apply up to 60 s late (was 5 s) for one warmed read on POSIX. For that to matter, the crew's own memory root would have to become sensitive inside that window. The PR body discloses this.
Optional:
- The body says the memory graph never reuses the warmed read. In fact the first read of the same window (days=14) on the same store takes it, whichever reader comes first (memory.py:724). Either correct the body or limit the take to the build's
_history_section. - Pin which store gets warmed: assert
get_memory_for.call_args.args == (None, "default")at test/test_memory_history_reuse.py:230, or add a case with a named store. - Share one constant for the history window (days=14) between eager_spawn.py:1065 and the build's read at memory.py:1206, so the warm and the build cannot drift apart.
Problem / Motivation
Goal: a message sent in a new chat whose session is already warm gets its first reply sooner, with no stale memory in it.
When a person opens a new chat and types for a few seconds, the session is already started by the time they press send. The prompt build then reads the chat's memory, and on a store with about 75 recent-history files that read alone took about 0.29 s, all of it on the send path, though none of it depends on the message.
Why it matters
This is the common way to begin a chat, and the wait sits between the send press and the first word. On a pod with a copy of a real store (lessons 36.9k, memory 30.3k chars) the whole pre-prompt work was about 1 s, and the history read was the largest single part of it.
Not a goal
What changed (motivation → approach → change)
The history read is message-independent, so it can run while the person types, as long as a file that changes afterwards is never answered from the earlier read.
dashboard/chat_turn/eager_spawn.pyMemoryStore.warm_recent_history(days=14)in the background, the window the build uses. A failure only costs the head start.MemoryStore.warm_recent_history(memory.py)MemoryStore.read_recent_history(memory.py)_HISTORY_REUSE_MAX_SECS), only while the history directory'slstatsignature is unchanged. The signature covers every entry's name, mode, inode, size, mtime, ctime and link count, plus the directory's own inode, mtime and ctime, and is kept only when it is the same before and after the warm read.append_historyandprune_historydrop it with the existing cache. Reads nobody warmed (suggestions, tips, the memory graph) keep the plain TTL and never reuse; a TTL hit lists nothing.memory.py,context_assembly/store_admission.py)MemoryStore._history_signatureis the only signature of the history directory:_admitted_file_stats(the session memory prefetch) compares it too, in place of its count/max-mtime/size_dir_lstat, which is deleted. It holds at most 2048 entries (_HISTORY_SIGNATURE_MAX_ENTRIES; pruning keeps a year of dated files); a fuller directory gets no signature, so no reuse and no prefetch match. A missing directory has its own fixed signature.HISTORY_REUSE_SUPPORTED)DirEntry.statreports 0 for inode and link count there, and anlstatthrough a linked ancestor is the SMB probe thatLocalMemoryFiles._read_root_guardexists to refuse. The reuse path runs no root guard, so it never takes the signature there; the prefetch takes it only behind that guard.The memory spec (
docs/system-specs/modules/memory-skills-hooks.md) describes the warm, the reuse and the signature.A rewritten, added, removed or relinked day file, or a day swapped for a symlink, changes the signature, so the next read goes through the same hardened reader again.
Measured on a pod, sending 7 s after "New chat session", 10 interleaved rounds per arm, every counted round at host load under 15:
build_messagemedianbuild_prompt_msmedian moved 1038 → 784 ms. These numbers come from a probe build implementing the same mechanism. This branch was checked by its tests andprove.py, not re-measured on the pod.Residual risk, for a maintainer to accept: a stat signature cannot see a change in the sensitive-path policy itself (a protected directory added, a home symlink repointed). Today the 5 s TTL bounds how long a history read can predate such a change; this PR stretches that bound to 60 s for the one warmed read a new chat's first build takes, and only for an unchanged directory. Every other reader keeps the 5 s bound.
test_memory.py::TestNamedV1StoreUnderFencedTree::test_default_store_history_reads_empty_under_the_same_fencestill passes.Backwards compatibility
Compatible: no stored format, key or signature changes outside a private keyword
store_to_warmon_spawn_admitted_prefetch.Removes nothing: the TTL cache, the hardened reader and every read check stay as they were; the change adds a reuse path in front of them.
Tests
test/test_memory_history_reuse.py:test/test_session_memory_prefetch.py: a link inside the history directory is still never followed, now read from the shared signature.test/test_chat_runner_base_surface.py: the recorded signature of_spawn_admitted_prefetch.Proof: PROVEN test/test_chat_runner_base_surface.py::test_a_moved_name_keeps_its_base_shape_and_owner[eager_spawn:_spawn_admitted_prefetch] (head 36fc6cc)
Also green locally:
test_memory.py,test_memory_markdown_read.py,test_memory_files_parity.py,test_eager_spawn*.py,test_chat_runner_composition_contract.py,test_flake_pattern_ratchet.py. black, flake8, isort and the repository semgrep rules are clean on the diff.Manual verification
On a Dev Fleet pod with a read-only copy of a real memory database and history folder, with the embedding model off: the table above, with
KIROCREW_TURN_TIMING=1and section timers in the build.Related Issues
N/A — follow-up to the first-message wait investigated in #37554 and #37575.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)