Skip to content

perf(context): read a new session's memory beside its startup - #23269

Merged
bolichen97 merged 1 commit into
mainfrom
perf/g3-context-overlap-startup
Oct 23, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
perf/g3-context-overlap-startup

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new dashboard chat that starts its session on the send path reads its first-turn memory while the session starts, so the prompt leaves soon after the session is ready.

Today the first-turn context is built only after KAS startup ends (spawn, session/new, managed MCP wait: ~2–7 s). Memory reads are most of that build. On the R8 gateway bench (6 new sessions, 7d9673ba7b), the build took 0.79 s in a normal run and up to 4.6 s under load. Memory took 0.68 s and up to 3.3 s of that, which is 85–94%. #21923 already starts the first message's embedding early. The memory reads that use it still wait for the session.

Why it matters

Every new chat sent at once pays this wait before its first word: 0.5–0.8 s on an idle host and 3–4.6 s on a busy one. It is the largest part of the first-turn build. It sits on the same path as the 8.95 s send → prompt-at-KAS gap that R8 measured.

Not a goal

  • Starting the rest of the session-start build early (option A: identity, skills, lessons, history). That is the last 6–15% of the build: ~0.1 s of 0.79 s idle, ~1.3 s of 4.6 s loaded. Many of its inputs come from the started session (provider type, model window, member mount, V2 essentials), so it is a bigger change. Follow-up if that part ever matters.
  • Member (V2) chats, a warm or pre-spawned session, a resumed session, Slack/cron/other channels. They keep today's path.
  • Making KAS startup or the managed MCP wait faster.

What changed (motivation → approach → change)

The build waits for the session, but its memory reads need nothing from the session. So a cold-start dashboard turn now starts those reads at the same time as the early embed, beside the session start. The build uses their result only if nothing they read has changed. If anything did, it reads again, as today.

Part What it does
dashboard/chat_turn/turn_admission.py On a cold start, also starts the memory reads. Drops them if the session start fails.
dashboard/chat_runner.py Runs the reads on the recall pool, after the early embed, so they join it.
context_assembly/store_admission.py read_session_memory holds today's reads in today's order. The build takes the early result only on an exact input match: same store and vector store, message, caps, activity switch, date, the lstat of the prefs, projects and history files (read only after the memory reader's own root and leaf link checks pass; otherwise the prefetch is off), and the vector store's validity token (scoring generations + PRAGMA data_version). These are taken before and after the early reads, and a move during them drops the result. Otherwise it reads for itself. Reads still running started first, so the build joins them instead of reading twice. It first cuts their embedding deadline down to its own, and the join ends at that same deadline, never later than _PROMPT_BUILD_EMBED_TIMEOUT_SECS. So neither waits past what the build's own read would have spent. It never waits for reads that have not started. A result over 512K characters is not held. The message is kept only as a SHA-256 digest, and a result older than 60 s is neither held nor taken.
context_omissions.py replay() hands the early reads' cut records to the build that takes them.
context.py, docs Re-export the three names. context-management.md and memory-skills-hooks.md describe the early reads.

Same blocks, same order, same budget: the build calls the same reads with the same arguments, and only where they run changes. The memory caps do not depend on the model window (_resolve_caps_cached), and a test pins that.

Backwards compatibility

Compatible: same prompt text and omission records. A miss of any kind falls back to today's reads, and no config, API or stored data changes.

Removes nothing: the change adds an early read path and keeps today's read as the fallback.

Tests

test/test_session_memory_prefetch.py, real ContextBuilder and MemoryStore. Invariant: a build from the early reads is byte-for-byte the build without them, and any miss reads again.

  • Same text and omissions, with the early read used and no second read.
  • Prefs, projects or history written afterwards: read again, same text as a fresh build.
  • A preferences Save landing during the early reads: result dropped, the build reads the new file.
  • Vector store: a semantic or episodic write, another process's commit (data_version), or no data_version at all each mean no match. A real store's token holds across its own reads.
  • Another message, never started, failed: read again. Never started is not waited on.
  • Still running past the embed recall bound: joined, not restarted, with no second read. Stuck: one wait of exactly the build's embed deadline, then read again.
  • Over the size bound: not held. The bound is at least the largest protected ceiling a build admits.
  • A linked preferences, projects or history source, or a store on another file provider: no prefetch, and no link followed.
  • A link inside the history directory (to an outside file, and dangling): the metadata stays readable and the outside file is never counted.
  • With the embed in flight, the join keeps to the build's short deadline. A prefetch past its TTL is not taken.
  • memory.startup_recall_timeout_secs: 0 turns the early reads off with the early embed, since they would embed the message just as early. src/kiro_crew/docs/configuration.md and the config spec say so.
  • The table holds no copy of the message, and a 200K paste leaves under 10K of repr.
  • The build cuts the prefetch's embedding deadline to its own, and the prefetch records the budget it runs under.
  • Omission records reach the build.
  • Caps are the same at every model window.
  • Expiry after the TTL.
  • Runner: a cold start begins the reads before get_or_create returns. A failed start drops them. A pre-spawned session starts none.

Red on origin/main: every test that touches the new path fails. The caps test passes on main as a guard.

Mutations: 22, each run against the test that names it, and all of them turn it red. They are:

  • take never used, inputs not compared
  • root admission skipped, leaf link check skipped, history entries followed
  • join not bounded by the build deadline, TTL not checked on take, recall-off opt-out ignored
  • write during the reads not checked, vector token dropped, missing data_version matches
  • not-started waited for, stuck join unbounded, running reads restarted at the recall bound
  • deadline not cut, budget not recorded
  • size bound removed, query held as text
  • omissions not replayed
  • cold start does not start reads, failed start keeps them
  • TTL prune removed

Determinism: two early reds had one cause. Two compared builds straddled a minute boundary, so their [CURRENT DATE] lines differed. The test helper now blanks only that line, and it asserts that the line appears exactly once. After the fix, 40 of 40 repeats pass at -n 0, and the file passes at -n 2. ContextVar budgets in the tests are restored by value, as the repo's Semgrep rule on ContextVar resets requires. Reversed order green. The flake ratchet is green. Related tests: test_context*, test_memory*, test_lesson*, test_chat_runner_*, test_default_context_admission and others, 3,792 passed with -n 2.

Manual verification

Before/after bench: research/crewmate-perf/g3/bench_g3.py. It uses a copy of the live home's memory (135 MB memory.db, real embedding model) and the real dashboard code. That covers the early embed, the early reads (after only) and the real build_message on the embed pool. KAS startup is a sleep of S. New session sent at once, fresh message per run, arms interleaved, n 5 each, 1-min load 6–8:

Startup S wait after startup p50 (max) before after send → prompt ready p50 before after
2.0 s 515 ms (1,507) 99 ms (101) 2.52 s 2.10 s
5.3 s 518 ms (582) 98 ms (100) 5.82 s 5.40 s
0.1 s 244 ms (1,459) 144 ms (171) 0.34 s 0.25 s
0.3 s 320 ms (499) 94 ms (96) 0.62 s 0.39 s

The 0.1 s and 0.3 s rows are startups shorter than the memory reads: the build joins the running reads instead of starting over, so it is still never slower than main. The ~100 ms left is the rest of the build (identity, skills, lessons). That is option A's share. Under the loaded host case R8 saw (memory 2.4–3.3 s), the saving grows with it, as long as startup is longer than the reads.

Related Issues

Refs #23258

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 23, 2026 00:58
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 86794191986a2f6fa35a6c8bd6b804f031025612 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 86794191986a2f6fa35a6c8bd6b804f031025612 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the early read is used only on an exact input match and falls back to today's read on any miss, so a revert removes it cleanly.

Notes

  • The match key is a list of the stores the reads touch (prefs/projects/history lstat plus the vector token). If MemoryStore.get_context or get_activity_context later reads a new source, a stale prefetch would still match, and nothing in the build would notice. Pin that list next to the reader it mirrors, for example with a test that fails when get_context gains a source not in _memory_read_inputs.
  • chat_runner.py is on the line-cap baseline, and this PR adds about 40 lines to it. _prefetch_session_memory_off_loop could live beside prefetch_session_memory instead of growing a file that is already over the cap.

[DESIGN-REVIEWED] 8679419

Description read

[DESCRIPTION-READ] 50773988f621b44398ab9318d4f5e4d223362562f19ff3853289dedaeacd1484

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 86794191986a2f6fa35a6c8bd6b804f031025612 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item answers the measured first-turn wait (memory is 85–94% of a 0.79–4.6 s build), and any mismatch falls back to today's read.

What this change ships

Inventory (7 items) — 7 justified

Intent: a new dashboard chat sends its first prompt sooner, because memory is read while the session is still starting. This is an ADDITION (perf).

  1. A cold-start dashboard chat now reads its memory (preferences, activity index, [Memory activity]) while the session starts, not after — justified
  2. The build uses the early result only if the store, vector token, file stats, date, caps and message all still match; otherwise it reads again — justified
  3. A build that finds the early read still running waits for it, but no longer than its own embedding deadline — justified
  4. memory.startup_recall_timeout_secs: 0 now also turns the early reads off (documented in configuration.md and config.md) — justified
  5. A failed or busy session start drops its early read — justified
  6. An in-process table of early reads, capped at 60 s TTL, 16 entries and 512K chars, keyed by a SHA-256 of the message — justified
  7. Exports reserve/discard/prefetch_session_memory and context_omissions.replay, each with one or more real consumers (chat_runner, turn_admission, store_admission) — justified

[FIRST-PRINCIPLES-REVIEWED] 8679419

Description read

[DESCRIPTION-READ] 48c4c06371374184cd679788c74255790897f4ee0a87bc57b2a1ae3c621d986a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 86794191986a2f6fa35a6c8bd6b804f031025612 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8679419

Verdict parsed from the review's SHA-scoped output markers for commit 86794191986a2f6fa35a6c8bd6b804f031025612.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 86794191986a2f6fa35a6c8bd6b804f031025612: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 86794191986a2f6fa35a6c8bd6b804f031025612 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8679419

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 86794191986a2f6fa35a6c8bd6b804f031025612: <one-sentence reason>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs changes. The input match is sound and the tests are strong, but one case the PR targets gets slower than main.

Head reviewed: 5ea5234. Bots read: Design PASS, First Principles PASS, UX skipped, Opus no findings, GPT changes requested. Readiness is still checking.

What I checked on this head:

  • test/test_session_memory_prefetch.py: 22 passed. With test_chat_runner_startup_recall.py, test_default_context_admission.py and the omission tests: 141 passed.
  • Ordering: the reserve on the loop, the started and done events, the pop in the take, and the discard on a failed or busy start. Two cold starts with the same text in one workspace only lose the head start, because the result is used only on an exact input match.
  • context_omissions.replay: the memory reads only call record, never clear, so update keeps today's last-write semantics.
  • The branch merges clean with current main.

🟡 1: a prefetch still running past the recall bound is thrown away, and the build reads again from zero. See the inline comment.

The GPT lane finding on store_admission.py:551 holds on this head. get_context returns the whole V1 preferences file, and each of up to 16 entries keeps that string for up to 60 s. Only the entry count is bounded. A size check before entry.reads = reads that reuses the existing protected content ceiling, with the current fallback read on a miss, fixes it. I do not repeat it inline.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever
iamwhatever force-pushed the perf/g3-context-overlap-startup branch from 5ea5234 to ca25add Compare October 23, 2026 02:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turn. Both points from my last review are fixed on this head, and each fix has a test that fails when it is reverted.

Head reviewed: ca25add. Bots read: Design PASS, First Principles PASS, UX skipped, Opus changes requested, GPT changes requested. Readiness is still checking.

What I checked on this head:

  • test/test_session_memory_prefetch.py: 25 passed.
  • Restoring the old join bound makes 2 of the new tests fail. Removing the size check makes test_a_result_over_the_size_bound_is_not_held fail.
  • The slow read case from my last review no longer restarts the reads.

The Opus lane finding on store_admission.py:596 reproduces, and my earlier suggestion helped cause it. See the inline comment for numbers and a fix that keeps both cases fast.

The GPT lane finding on store_admission.py:521 also holds. The full message is kept as the key of a reservation with no length check, so 16 reservations of large pastes kept about 88M characters in my run. The build of the same turn usually takes the entry, so the cost is limited to entries nobody takes, for up to 60 s. Skipping the prefetch when the message is longer than _MEMORY_PREFETCH_MAX_CHARS, before the reservation, closes it. I agree with the adjudication that the history cache finding is low.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever
iamwhatever force-pushed the perf/g3-context-overlap-startup branch from ca25add to 1e859ac Compare October 23, 2026 03:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Both points from my last review are fixed on this head, and each fix has a test that fails when it is reverted.

Head reviewed: 1e859ac. Bots read: Design PASS, First Principles PASS, UX skipped, Opus PASS, GPT changes requested. Readiness is action required.

What I checked on this head:

  • test/test_session_memory_prefetch.py: 28 passed.
  • Removing the deadline cut in _take_memory_prefetch makes test_the_build_cuts_the_prefetch_embed_deadline_to_its_own fail. Not recording the work also makes test_the_prefetch_records_the_embed_budget_it_runs_under fail.
  • Slow first embed repro (embed 0.8 s, recall bound 0.2 s): the build is ready at 0.24 s with the prefetch, 0.77 s with the cut disabled, and 0.21 s with no prefetch. The regression from my last review is gone.
  • The table now keys the message by its SHA-256, so a large paste is no longer held as a key.

One point blocks: the GPT lane finding on store_admission.py:457 reproduces. See the inline comment.

I agree with the adjudication on the other two GPT findings. I reproduced the timed out embed case: the build takes a keyword only result after the vector is cached. It costs one prompt and the next turn corrects it.

@iamwhatever
iamwhatever force-pushed the perf/g3-context-overlap-startup branch from 1e859ac to 21667dd Compare October 23, 2026 04:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@bolichen97
bolichen97 dismissed their stale review October 23, 2026 05:10

Withdrawing my request for changes: the link following stat is fixed on 21667dd. Details in my new review.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The point from my last review is fixed on this head, so I am withdrawing my request for changes.

Head reviewed: 21667dd. Bots read: Design PASS, First Principles PASS, UX skipped, Opus PASS, GPT PASS after adjudication.

What I checked on this head:

  • test/test_session_memory_prefetch.py: 32 passed.
  • My earlier repro, a link inside the history directory to a file outside the tree plus a dangling link: the outside file's size is no longer counted and the result is no longer None. Nothing is followed.
  • A full build with the prefetch and a planted history link follows no link, the same as main.
  • With the previous store_admission.py restored, four of the new tests fail.

One small gap, see the inline comment: the new tests do not pin the in directory link case.

On the lanes: I reproduced the GPT finding (the prefetch warms the five second history cache, an edit through another handle within that window shows old history in one prompt) and the Opus note (a held entry older than the 60 second TTL is still taken). I agree with both being low: main shows the first with any read in the same window, and the second only differs in day level decay because every other input is compared again.

This is a performance change, so it needs a maintainer's approval.

@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c64e94adf392 The retained query bypasses the prefetch size bound

fixed in 1e859ac (still in cc811d4): the prefetch table and its inputs now hold the message only as a SHA-256 digest (_query_digest), never the text, so a large paste is not retained by the table.
Pinned by test_the_table_holds_no_copy_of_the_message: a 200K paste leaves under 10K of table repr. Putting the raw query back turns it red.

@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c64e94adf392 Prefetch validity ignores history-cache expiry

rebutted (adjudication downgraded it to LOW): the history block may lag a write by one cache window. This happens only when another process appends history while that cache is still warm. It is the 5 s staleness _HISTORY_CACHE_TTL_SECS already accepts for every read, and main shows it the same way inside that window.
The block is background context that the admission loop may drop. Nothing is lost: the next session start reads fresh.
Carrying a cache-expiry token through read_session_memory → get_activity_context → read_recent_history for this one path costs more than the harm.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c64e94adf392 Prefetch metadata follows links before memory admission

fixed in 21667dd (still in cc811d4): _admitted_file_stats runs the reader's own admission first: the LocalMemoryFiles._read_root_guard UNC, linked-ancestor and linked-root gate, then the leaf link check.
After that it uses only os.lstat and entry.stat(follow_symlinks=False). A refused root, a linked source or a non-local file provider turns the prefetch off.
Pinned by test_a_linked_memory_source_turns_the_prefetch_off, test_a_store_on_another_file_provider_turns_the_prefetch_off and test_a_link_inside_the_history_directory_is_never_followed. Removing each guard turns its test red.

@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c64e94adf392 Equal-length saves can reuse obsolete preferences

rebutted (adjudication downgraded it to LOW): it needs a filesystem with coarse mtimes and a Save that writes text of exactly the same length within one mtime tick, between the prefetch read and the take.
The cost is one first prompt with the old preferences. The next build compares again and reads fresh.
A content hash on every prefetch and every take would add a mechanism only FAT-class mtimes need. That is more than the harm.

@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c64e94adf392 Timed-out keyword results remain reusable after embedding completes

rebutted (adjudication downgraded it to LOW): the case needs an early embed of more than 30 s that then finishes before a session start of more than 30 s ends. Only a cold model load does that.
The cost is lower recall in one first prompt. Later turns use the cached vector. Since 1e859ac the build also cuts the prefetch's embed deadline to its own.
Carrying a "ranked without vector" flag from the embedding layer into SessionMemoryReads is new state across modules for one rare cold path.

@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c64e94adf392 History invalidation falls back to the cache populated by prefetch

rebutted (adjudication downgraded it to LOW): it needs a named V1 store, a prefetch that warms its 5 s history cache, and a history save through a separate handle inside that window. Then the fallback build reads the cached history for one prompt.
The same staleness shows on main for any read in that window. The file is correct, and the cache expires on its own.
Clearing MemoryStore's private cache from this module only on the prefetch path would add cross-module coupling for a self-correcting, documented window.

@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=05ade5394cfa Session-start build waits up to 5 s on the early memory reads behind a slow embed

fixed in 1e859ac and cc811d4: _take_memory_prefetch cuts the prefetch's EmbeddingWork.deadline to the build's own. A joined embed polls expired(), so the reads give up on a slow embed when the build's read would.
The join itself now ends at that same deadline, never later than _PROMPT_BUILD_EMBED_TIMEOUT_SECS.
Pinned by test_the_build_cuts_the_prefetch_embed_deadline_to_its_own, test_the_prefetch_records_the_embed_budget_it_runs_under and test_the_join_ends_at_the_build_embed_deadline. Removing each line turns its test red.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head cc811d4 (force push of one squashed commit; the tree delta over 21667dd is the join bound, the take time TTL check, the in directory link test and the link screen site entry).

Fixed since my last review:

  1. The in directory link case is now pinned. Reverting entry.stat(follow_symlinks=False) to entry.stat() in _dir_lstat turns test_a_link_inside_the_history_directory_is_never_followed red (1 failed, 34 passed).
  2. The join now ends at the build's own embed deadline. Removing the bound = min(...) line turns test_the_join_ends_at_the_build_embed_deadline red.
  3. A prefetch past its TTL is not taken. Removing that check turns test_a_prefetch_older_than_its_ttl_is_not_taken red.
    The prefetch suite passes on head (35 passed).

One blocking issue remains, and it matches the Opus lane finding on this head, which I reproduced: with memory.startup_recall_timeout_secs: 0 the cold start prewarm still embeds the first message beside session start, through the new memory prefetch. Details and a verified one line fix are inline.

This is a Tier 2 perf change, so I am leaving a comment for the maintainers rather than approving.

# within its own deadline. A pool too busy to start it first leaves the build to
# embed for itself, as it does without this.
await asyncio.sleep(0)
if not prefetch_memory:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The zero recall timeout opt out no longer keeps the first message embed out of session start

embed_startup_query returns early when _startup_recall_timeout_secs() <= 0, and the config spec says 0 turns the early embedding off. The new branch below it does not check that setting, so reserve_memory_prefetch and _prefetch_session_memory_off_loop still run. prefetch_session_memory then calls read_session_memory, whose activity ranking embeds the message under a 30 s interactive deadline while the session is still starting.

    if not prefetch_memory:
        return
    # Reserved on the loop, before the thread exists: ...
    reserve_memory_prefetch(workspace, memory_store, text)

Repro: a real MemoryStore with a VectorMemoryStore whose embed_fn counts calls, then _prewarm_startup_query(..., prefetch_memory=True) with the timeout patched to 0. Head: 2 embeds of the first message during prewarm. Main with the same setup: 0. With the timeout at 1 the early embed runs on both, as intended.

Suggestion, verified on head (the repro drops to 0 embeds at timeout 0 and still embeds at 1; test_session_memory_prefetch.py and test_chat_runner_startup_recall.py pass, 53 passed):

    if not prefetch_memory or _budgets._startup_recall_timeout_secs() <= 0:
        return

Putting the same check inside prefetch_session_memory instead breaks test_a_prefetch_running_past_the_recall_bound_is_joined_not_restarted, which patches the timeout to 0, so the call site is the better place. A test that drives _prewarm_startup_query with the timeout at 0 and asserts no prefetch is reserved would pin it.

Rule: a documented config opt out must keep covering every path that does the work it turns off.

@iamwhatever
iamwhatever force-pushed the perf/g3-context-overlap-startup branch from cc811d4 to 8679419 Compare October 23, 2026 06:52
@iamwhatever

iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=05ade5394cfa Setting memory.startup_recall_timeout_secs: 0 no longer stops the first message from being embedded early

fixed in 8679419: prefetch_session_memory returns before it starts when _budgets._startup_recall_timeout_secs() <= 0. The entry is never marked started, so the build does not wait for it and reads and embeds for itself, the same as with the early embed off.
src/kiro_crew/docs/configuration.md and the config spec now say 0 turns off the early embed and the early memory reads together.
Pinned by test_turning_the_early_embed_off_turns_the_early_reads_off; removing the check turns it red.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 8679419 (force push of one squashed commit; the tree delta over cc811d4 is the new timeout check in prefetch_session_memory, its test, the adjusted join test and the two config doc lines).

Fixed since my last review: with memory.startup_recall_timeout_secs: 0 the cold start prewarm no longer embeds the first message early. I reran the same repro as last time (a real MemoryStore with a vector store whose embed function counts calls, then _prewarm_startup_query(..., prefetch_memory=True)):

  1. Timeout 0: head 0 embeds during prewarm, main 0. Last head was 2.
  2. Timeout 1: head still embeds early, as intended.

The check sits inside prefetch_session_memory rather than at the call site I suggested, and that works: the entry is reserved but never marked started, so _take_memory_prefetch does not wait for it and the build reads for itself. The join test now uses a 0.001 s bound instead of 0, which still keeps it far below the held reads, so it tests the same join. Removing the new check turns test_turning_the_early_embed_off_turns_the_early_reads_off red (1 failed, 35 passed) and brings the repro back to 2 embeds. On head, test_session_memory_prefetch.py and test_chat_runner_startup_recall.py pass (54 passed). The head also merges cleanly onto current main, and the prewarm wiring survives the recent chat runner split.

Bots on this head: Design PASS, First Principles PASS, UX skipped, Opus no findings, GPT no findings. Several required checks were still running when I read it.

No blocking issues from me. This is a Tier 2 perf change, so I am leaving a comment for the maintainers rather than approving; the remaining step is maintainer approval once readiness passes on this head.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 86794191986a2f6fa35a6c8bd6b804f031025612.

What it fixes: a new dashboard chat that starts its session on the send path no longer waits for KAS startup before reading its first-turn memory; the reads run beside the startup and the build takes them only on an exact input match.

Checked against source:

  • Same blocks, same order: read_session_memory makes the same get_context / activity_index / get_activity_context / ranks_activity_against calls with the same arguments, session_memory_parts appends them where it did, and omissions reach the build through context_omissions.replay.
  • The compared inputs cover every source those readers touch today (prefs and projects via lstat, the history directory via a no-follow scandir, the vector store's scoring generations plus PRAGMA data_version on its single locked connection).
  • Every wait has a bound: a never-started prefetch is not waited on, the join ends at min(_PROMPT_BUILD_EMBED_TIMEOUT_SECS, own deadline), and the prefetch's own embed deadline is cut to the build's. The reads run on the recall pool and the build on the embed pool, so the join cannot starve itself.
  • The loop only does reserve/discard (a lock, a dict of at most 16 entries, one sha256); no file IO on it.
  • Any prefetch failure leaves reads=None and sets done, so the build reads for itself; member, private, temporary and blocked-reads turns never prefetch.

Tests: test/test_session_memory_prefetch.py 36 passed; with test_chat_runner_startup_recall, test_chat_runner_composition_contract, test_context_omissions, test_memory_v1_golden, test_memory_on_demand, test_default_context_admission, test_start_priority and the other files that touch the prewarm path, 862 passed. Full suite not run.

Mutations, one at a time, each restored before the next:

  • waiting on a never-started prefetch: red test_a_prefetch_that_never_started_is_not_waited_for
  • unbounded join: red test_a_prefetch_running_past_the_recall_bound_is_joined_not_restarted, test_a_stuck_prefetch_is_given_up_on_at_the_build_embed_deadline, test_the_join_ends_at_the_build_embed_deadline
  • prefetch embed deadline not cut: red test_the_build_cuts_the_prefetch_embed_deadline_to_its_own
  • join not cut to the build's own deadline: red test_the_join_ends_at_the_build_embed_deadline
  • write-during-reads check removed: red test_a_preferences_write_during_the_reads_drops_the_prefetch
  • take-time input compare removed: red test_a_memory_file_written_after_the_prefetch_is_read_again[prefs|projects|history]
  • started never set: red test_a_build_from_a_prefetch_is_byte_for_byte_the_build_without_one and the two join tests
  • TTL not checked on take: red test_a_prefetch_older_than_its_ttl_is_not_taken
  • recall-off opt-out ignored: red test_turning_the_early_embed_off_turns_the_early_reads_off
  • missing data_version matches: red test_a_store_that_cannot_see_other_writers_never_matches
  • failed start keeps its prefetch: red test_a_failed_session_start_drops_its_prefetch
  • prefetch_memory flag ignored: red test_a_cold_start_reads_memory_while_the_session_starts, test_a_prewarmed_session_starts_no_memory_prefetch

Three mutants survive, all on cleanup or ordering paths whose cost is bounded by the 60 s TTL and the 16-entry cap: dropping the discard on the SessionBusyError re-raise, dropping the discard in _prefetch_session_memory_off_loop's except, and moving reserve_memory_prefetch after the prefetch task is created. Worth a test each, not blocking.

CI on this head: 87 success, 16 skipped, 0 failing (latest run per name); legacy statuses success, PR Readiness passed. Merges cleanly onto current main with the prefetch wiring intact.

Non-blocking follow-ups:

  1. Two cold starts with the same workspace, store and first message close together share one table key: the second reserve replaces the first entry, both prefetch threads can write that one entry, and reads and inputs are assigned separately, so a build has a very narrow window to pair one thread's result with the other's inputs. Handing the reserved entry object to the thread (or assigning (reads, inputs) as one tuple) closes it. Found by reading the code, not reproduced.
  2. When the reads are slow for a reason other than the embed (disk, SQLite lock), the build joins up to its own deadline and then reads again, so the worst case is about one embed deadline above main. Bounded, fine as is.
  3. Design's two notes are still open: pin _memory_read_inputs to the readers it mirrors with a test, and consider moving _prefetch_session_memory_off_loop next to prefetch_session_memory instead of growing chat_runner.py.
  4. The bench is single-session; the motivating case is several new sessions at once. A run with that concurrency would show whether the recall pool holds up.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head 8679419. Rechecked on this head: readiness passed, PR Readiness green, every required check green, all AI review lanes pass, no open change requests, mergeable against main. The early memory read falls back to the original read path whenever an input moved, and the new prefetch test pins that behavior. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants