Skip to content

perf(memory): read a new chat's history while the person types - #37595

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/kiro-crew-rust/prewarm-memory-read
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/kiro-crew-rust/prewarm-memory-read

Conversation

@buluoray

@buluoray buluoray commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a message sent in a new chat whose session is already warm gets its first reply sooner, with no stale memory in it.

When a person opens a new chat and types for a few seconds, the session is already started by the time they press send. The prompt build then reads the chat's memory, and on a store with about 75 recent-history files that read alone took about 0.29 s, all of it on the send path, though none of it depends on the message.

Why it matters

This is the common way to begin a chat, and the wait sits between the send press and the first word. On a pod with a copy of a real store (lessons 36.9k, memory 30.3k chars) the whole pre-prompt work was about 1 s, and the history read was the largest single part of it.

Not a goal

  • Lessons, semantic and episodic recall. They depend on the message or need their own design.
  • Sending the prompt before memory is read.
  • Changing the hardened reader or the sensitive-path checks.

What changed (motivation → approach → change)

The history read is message-independent, so it can run while the person types, as long as a file that changes afterwards is never answered from the earlier read.

Part Change
dashboard/chat_turn/eager_spawn.py After a fresh pre-warm registers its session, it calls MemoryStore.warm_recent_history(days=14) in the background, the window the build uses. A failure only costs the head start.
MemoryStore.warm_recent_history (memory.py) Reads the window and keeps it with the directory's signature. It writes no plain 5 s TTL entry, so the build never takes a TTL hit that checks nothing.
MemoryStore.read_recent_history (memory.py) Reuses a warmed window once, within 60 s (_HISTORY_REUSE_MAX_SECS), only while the history directory's lstat signature is unchanged. The signature covers every entry's name, mode, inode, size, mtime, ctime and link count, plus the directory's own inode, mtime and ctime, and is kept only when it is the same before and after the warm read. append_history and prune_history drop it with the existing cache. Reads nobody warmed (suggestions, tips, the memory graph) keep the plain TTL and never reuse; a TTL hit lists nothing.
One signature (memory.py, context_assembly/store_admission.py) MemoryStore._history_signature is the only signature of the history directory: _admitted_file_stats (the session memory prefetch) compares it too, in place of its count/max-mtime/size _dir_lstat, which is deleted. It holds at most 2048 entries (_HISTORY_SIGNATURE_MAX_ENTRIES; pruning keeps a year of dated files); a fuller directory gets no signature, so no reuse and no prefetch match. A missing directory has its own fixed signature.
Windows (HISTORY_REUSE_SUPPORTED) No reuse and no warm read; the plain 5 s TTL as before. DirEntry.stat reports 0 for inode and link count there, and an lstat through a linked ancestor is the SMB probe that LocalMemoryFiles._read_root_guard exists to refuse. The reuse path runs no root guard, so it never takes the signature there; the prefetch takes it only behind that guard.

The memory spec (docs/system-specs/modules/memory-skills-hooks.md) describes the warm, the reuse and the signature.

A rewritten, added, removed or relinked day file, or a day swapped for a symlink, changes the signature, so the next read goes through the same hardened reader again.

Measured on a pod, sending 7 s after "New chat session", 10 interleaved rounds per arm, every counted round at host load under 15:

arm build_message median history files read paired vs main
main 939 ms 289 ms —
main again (A/A) 910 ms 295 ms −9 ms (−248..256)
read early 662 ms 4 ms −207 ms (−503..−121, all 10 pairs faster)

build_prompt_ms median moved 1038 → 784 ms. These numbers come from a probe build implementing the same mechanism. This branch was checked by its tests and prove.py, not re-measured on the pod.

Residual risk, for a maintainer to accept: a stat signature cannot see a change in the sensitive-path policy itself (a protected directory added, a home symlink repointed). Today the 5 s TTL bounds how long a history read can predate such a change; this PR stretches that bound to 60 s for the one warmed read a new chat's first build takes, and only for an unchanged directory. Every other reader keeps the 5 s bound. test_memory.py::TestNamedV1StoreUnderFencedTree::test_default_store_history_reads_empty_under_the_same_fence still passes.

Backwards compatibility

Compatible: no stored format, key or signature changes outside a private keyword store_to_warm on _spawn_admitted_prefetch.

Removes nothing: the TTL cache, the hardened reader and every read check stay as they were; the change adds a reuse path in front of them.

Tests

  • test/test_memory_history_reuse.py:
    • after a warm, an unchanged directory is not read again, and the warmed read is used once;
    • reads nobody warmed never reuse;
    • after a warm, a rewritten day, a new day file, a new hard link, and the end of the 60 s bound each read again;
    • an append drops the reuse;
    • after a warm, a day rewritten by another process is read again even inside the TTL;
    • an explicit read inside the TTL lists nothing;
    • a directory with more entries than the cap gets no signature, so no reuse;
    • without reuse support (Windows), the warm reads nothing, the signature is never taken, and every read goes to the files;
    • a day swapped for a symlink to an outside secret is refused, not reused;
    • a fresh pre-warm warms the chat's own store, while a resumed or already-claimed one does not (prewarm allowance pinned by the shared fixtures).
  • test/test_session_memory_prefetch.py: a link inside the history directory is still never followed, now read from the shared signature.
  • test/test_chat_runner_base_surface.py: the recorded signature of _spawn_admitted_prefetch.

Proof: PROVEN test/test_chat_runner_base_surface.py::test_a_moved_name_keeps_its_base_shape_and_owner[eager_spawn:_spawn_admitted_prefetch] (head 36fc6cc)

Also green locally: test_memory.py, test_memory_markdown_read.py, test_memory_files_parity.py, test_eager_spawn*.py, test_chat_runner_composition_contract.py, test_flake_pattern_ratchet.py. black, flake8, isort and the repository semgrep rules are clean on the diff.

Manual verification

On a Dev Fleet pod with a read-only copy of a real memory database and history folder, with the embedding model off: the table above, with KIROCREW_TURN_TIMING=1 and section timers in the build.

Related Issues

N/A — follow-up to the first-message wait investigated in #37554 and #37575.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 15, 2026 11:25
@buluoray
buluoray requested a review from CrysisDeu November 15, 2026 11:25
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item has a measured harm or is a deletion. The prefetch's count/mtime/size _dir_lstat is gone, replaced by one shared signature.

What this change ships

Inventory (7 items) — 7 justified

Intent: shorten the wait for the first reply in a new chat by reading the chat's recent history while the person is still typing. ADDITION (perf).

  1. A fresh new-chat pre-warm reads the 14-day history in the background — justified
  2. The first send in that chat reuses the early read once, within 60 s, if the history folder is unchanged — justified
  3. A file history read can now be up to 60 s older than a sensitive-path policy change, up from 5 s; this applies only to the warmed read — justified
  4. The session-memory prefetch uses the same history signature as the reuse; _dir_lstat is deleted — justified
  5. A history folder with more than 2048 entries gets no signature, so no reuse and no prefetch match — justified
  6. Windows: no early read and no reuse, the plain 5 s TTL as before — justified
  7. The private store_to_warm keyword on _spawn_admitted_prefetch, with its one caller, and the updated surface pin — justified

[FIRST-PRINCIPLES-REVIEWED] 36fc6cc

Description read

[DESCRIPTION-READ] 61b0ea24d7bc9eb2d5869e0affb6c075778d327653ffbccf1ede0e40b71ec380

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the warm read is reused only on a matching lstat signature, at most once, within 60 s, and only on POSIX. Every other read keeps the 5 s TTL.

Notes

  • _admitted_file_stats now returns None (no prefetch match) when the history directory can't be listed or holds more than 2048 entries. The old _dir_lstat tuple still matched in both cases. This is a small, reversible prefetch change, and the description doesn't mention it.
  • The warm thread and the build thread share _history_reuse without a lock. If the build pops first, the warm stores a stale-in-time window that some later read could take. The signature check keeps that correct, but a short comment saying so would stop a later reviewer from adding a lock.

[DESIGN-REVIEWED] 36fc6cc

Description read

[DESCRIPTION-READ] 5a5eccf95e05a4b1f55070cb929d72ffe53bfae12cdb0a0e7b0b82bb99ab3bf8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 36fc6cc

Verdict parsed from the review's SHA-scoped output markers for commit 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 36fc6cc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 36fc6cca369270b1e2ff250cad71f43cfbf0ea5d false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 421 lines, more than the 400 it may approve on its own.

A fresh pre-warm now reads the 14-day history in the background. The first build reuses that read once, within 60 s, and only when the history directory's lstat signature (taken before and after the warm, POSIX-only) is unchanged. Nothing else changes: the TTL path stays as it was, append, prune and cache reset clear the reuse, and V2, temporary and member slots are skipped. The early get_memory_for checks the store directory (via require_memory_store) before anything can be created, so a deleted store can't be recreated. The prefetch's directory check is replaced by a stronger shared signature. The tests cover reuse, the ways reuse is turned off, a symlink swap and the eager-spawn wiring. The only notable risks are the stated 60 s policy-staleness window and a slight over-claim about memory-graph reuse; neither is blocking.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 36fc6cca36 · 421 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@buluoray
buluoray force-pushed the crew/kiro-crew-rust/prewarm-memory-read branch from aa9ea20 to 85b4009 Compare November 15, 2026 14:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 15, 2026
@buluoray
buluoray force-pushed the crew/kiro-crew-rust/prewarm-memory-read branch from 85b4009 to f0bcf06 Compare November 15, 2026 14:11
@buluoray

Copy link
Copy Markdown
Contributor Author

Fixed on f0bcf06, all four findings on aa9ea20:

  • mypy union-attr at eager_spawn.py:1066: the builder is passed into the worker function.
  • Opus (Windows hard links) and GPT F1 (probe ahead of the Windows root guard): HISTORY_REUSE_SUPPORTED is False on Windows, so _history_signature returns None before any lstat/scandir; no reuse and no pre-warm read there, the plain TTL as before.
  • GPT F2 (prewarmed hit inside the 5 s TTL): with a signature, a TTL hit is served only through the signature check. New test test_a_day_rewritten_inside_the_ttl_is_read_again fails without it.

@buluoray
buluoray force-pushed the crew/kiro-crew-rust/prewarm-memory-read branch from f0bcf06 to dbdf5f7 Compare November 15, 2026 14:47
@buluoray

Copy link
Copy Markdown
Contributor Author

dbdf5f7, for f0bcf06: GPT's unbounded-signature finding is fixed with a 512-entry cap (_HISTORY_SIGNATURE_MAX_ENTRIES). A fuller directory gets no signature and no reuse, and the scan stops at the cap. New test test_a_crowded_directory_gets_no_signature fails without it. Flake Pattern Ratchet K19: the 3600 s TTL in the new test is never waited, so it now carries flake-ok. Rebased on bdff21a.

@buluoray
buluoray force-pushed the crew/kiro-crew-rust/prewarm-memory-read branch from dbdf5f7 to e0b9f96 Compare November 15, 2026 15:13
@buluoray

Copy link
Copy Markdown
Contributor Author

e0b9f96, for the First Principles BLOCK on dbdf5f7:

  • One signature: MemoryStore._history_signature now feeds store_admission._admitted_file_stats as well, and _dir_lstat is deleted. The prefetch match is now at least as strict: it also sees inode, ctime and link count, and an unreadable or over-cap directory means no match.
  • Ride-along TTL listing removed: explicit readers' TTL hits list nothing again (new test test_a_read_inside_the_ttl_lists_nothing). GPT's F2 stays fixed a different way: the pre-warm calls the new warm_recent_history, which drops the TTL entry its read leaves, so the build goes through the signature check (test_a_day_rewritten_after_the_warm_is_read_again fails without the drop).
  • Cap raised to 2048, because pruning keeps a year of dated files and the prefetch now depends on the signature.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The memory-side reuse tests are solid. One item in the new eager-spawn test needs a fix before this can merge.

  1. test/test_memory_history_reuse.py:200: test_a_fresh_prewarm_reads_the_history_early imports _bindings, _cfg, _mock_state from eager_spawn_helpers inside the function body. Because of that, the autouse fixtures _pin_prewarm_allowance and _isolate_armed_registry are never registered for this module. _eager_spawn then reads the real host memory through resource_status.prewarm_allowance(). On a host with little free memory (allowance 0), _admit_prefetch refuses the prewarm and the new warm block is never reached. The (True, False, True) case then fails with warm_recent_history called 0 times. The test also leaves a t1 entry in the module-global _armed_prefetches registry for the rest of the worker. With the allowance pinned to 0, this test failed and the test_eager_spawn.py tests passed. Smallest fix: add from eager_spawn_helpers import _isolate_armed_registry, _pin_prewarm_allowance # noqa: F401 -- fixtures are used by name at module scope, as the sibling test_eager_spawn*.py modules do. You can also move this test into test_eager_spawn.py, which already has both fixtures.

Done well: the reuse is kept only when the signature taken before the read matches the one taken after. The symlink-swap test asserts that neither the outside file nor the old cached text comes back.

Optional:

  • docs/system-specs/modules/memory-skills-hooks.md ("History reads", around line 359) still describes only the 5 s TTL. Could you add the 60 s signature reuse, the 2048-entry cap, the POSIX-only gate, the warm_recent_history caller, and the shared _history_signature?
  • The 60 s reuse applies to every read_recent_history caller (suggestions, tips, the memory graph, read_history), not only the warmed first build. Is that intended? If it is not, letting only an entry written by warm_recent_history be reused would keep the 5 s bound for the other readers.
  • In the warmed case, asserting get_memory_for call args would pin which store gets warmed.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 15, 2026
A message sent in a chat whose session was pre-warmed still waited for
its prompt build, and about a third of that build was reading every
recent-history file through the hardened reader. None of it depends on
the message.

The pre-warm of a fresh session now calls the new
MemoryStore.warm_recent_history in the background. That read writes no
plain 5 s TTL entry, so the build cannot take a TTL hit that checks
nothing. It is kept with the history directory's lstat signature (name,
mode, inode, size, mtime, ctime, link count of every entry, plus the
directory's own), and the next read_recent_history of that window reuses
it once, within 60 s, only while the signature is unchanged. Any
rewrite, new or removed day, relink or symlink swap reads every file
again through the same reader, and append/prune drop it as they drop the
cache. Reads nobody warmed keep the plain TTL, and a TTL hit lists
nothing.

MemoryStore._history_signature is now the one signature of the history
directory: store_admission._admitted_file_stats compares it too, in
place of its count/max-mtime/size _dir_lstat, which is deleted. It holds
at most 2048 entries; a fuller directory gets no signature, so no reuse
and no prefetch match. The memory spec describes the warm, the reuse and
the signature.

Windows gets no reuse and no warm read: DirEntry.stat reports 0 for
inode and link count there, and an lstat through a linked ancestor is
the SMB probe that LocalMemoryFiles._read_root_guard exists to refuse,
so the reuse path, which runs no root guard, never takes the signature.
The prefetch takes it only behind that guard.

On a pod with a copy of a real store (lessons 36.9k, memory 30.3k
chars), sending 7 s after "New chat": build_message median 939 -> 662
ms, paired difference -207 ms over 10 interleaved pairs (all ten
negative), against an A/A paired difference of -9 ms.
@buluoray
buluoray force-pushed the crew/kiro-crew-rust/prewarm-memory-read branch from e0b9f96 to 36fc6cc Compare November 15, 2026 17:09
@buluoray

Copy link
Copy Markdown
Contributor Author

36fc6cc, for KiroCrewCaptain's review on e0b9f96:

  • Fixed the required item: test_memory_history_reuse.py now imports _isolate_armed_registry and _pin_prewarm_allowance at module scope, so a host with little free memory no longer skips the warm, and no t1 entry leaks into _armed_prefetches.
  • Optional 1: memory-skills-hooks.md (History reads) now describes the warm, the 60 s one-shot reuse, the 2048-entry cap, the POSIX-only gate and the shared _history_signature.
  • Optional 2: not intended, so narrowed. Only a read made by warm_recent_history is kept, and the next read uses it once. Suggestions, tips, the memory graph and read_history keep the 5 s TTL. New test test_reads_nobody_warmed_never_reuse fails on the previous head.
  • Optional 3: the warmed case asserts get_memory_for's workspace argument.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 15, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 15, 2026 18:50

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on 36fc6cc. I checked these on this head. A warmed read writes no TTL entry. It is kept only when the directory signature taken before the read matches the one taken after. The next read of the same window uses it at most once, within 60 s, and only while the signature, the day and the bound all still hold. Every other outcome reads every file again through the unchanged hardened reader. The warm runs off the event loop through asyncio.to_thread, only for a new, non-resumed, non-temporary chat. Windows skips both the warm and the reuse.

Earlier review: the required item is fixed, since _isolate_armed_registry and _pin_prewarm_allowance are now imported at module scope (test/test_memory_history_reuse.py:17-20). The spec note is added in memory-skills-hooks.md. Only warm_recent_history now writes the reuse entry. The store pin is still partial; see below.

Done well: the prefetch now compares the same _history_signature, which removes the second, weaker signature. The symlink-swap test checks that neither the outside file nor the old cached text comes back.

Security: the sensitive-path and realpath checks can apply up to 60 s late (was 5 s) for one warmed read on POSIX. For that to matter, the crew's own memory root would have to become sensitive inside that window. The PR body discloses this.

Optional:

  • The body says the memory graph never reuses the warmed read. In fact the first read of the same window (days=14) on the same store takes it, whichever reader comes first (memory.py:724). Either correct the body or limit the take to the build's _history_section.
  • Pin which store gets warmed: assert get_memory_for.call_args.args == (None, "default") at test/test_memory_history_reuse.py:230, or add a case with a named store.
  • Share one constant for the history window (days=14) between eager_spawn.py:1065 and the build's read at memory.py:1206, so the warm and the build cannot drift apart.

@KiroCrewCaptain
KiroCrewCaptain merged commit 2166993 into main Nov 15, 2026
103 of 113 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/kiro-crew-rust/prewarm-memory-read branch November 15, 2026 18:51
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants