Skip to content

perf(chat): read a prewarmed new chat's memory beside its claim - #26970

Merged
bolichen97 merged 1 commit into
mainfrom
perf/new-chat-early-context
Nov 2, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
perf/new-chat-early-context

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new chat whose session an eager spawn already started reaches its first token sooner, because its session-start memory is read while the turn claims the session instead of after.

A user opens a new sidebar chat, types for a few seconds, and sends. By then the eager spawn has started the session and registered it. The early memory read from #23269 and #24394 runs only when no session is registered (src/kiro_crew/dashboard/chat_turn/turn_admission.py:862-884 on main). So this turn reads its session-start memory only at the build, after the claim, model settle, set mode, replay and store preparation. The prewarm's own registration also maps its session id, so the resumable-hint check skips it a second time, although its first real turn builds a fresh session-start context.

Why it matters

Session-start memory is ~0.6 s of a new chat's first-token wait on a quiet host and several seconds on a busy one, where the read queues behind other embeds (R15 research: memory 0.77 s p50 quiet, 4.86 s p50 busy). The steps between the claim and the build take ~1.1 s quiet, so the read fits under them.

Not a goal

  • A send that arrives while the eager start is still running. That case already gets the early read (nothing is registered yet), but the turn starts a second process instead of joining the eager one. See "Found on the way" below; it is tracked separately.
  • Building the message-free session-start blocks (skills index, member identity, lessons) before the send. On sidebar chats they are ~0.1 s (R8 gateway INFO, 10 new sidebar chats: skills 0.04 s p50, lessons 0.05 s, member/preamble ~2 ms), and V1 lessons are ranked by the message. The larger figures (member 585 ms / 1078 ms, R12, n 2) are member chats, which already start their session before the user types (welcome, scheduled turns), so a member-only version buys little.
  • Any change to what the prompt contains.

What changed (motivation → approach → change)

  • The read is gated on "will this turn build a fresh session-start context". For a turn that starts its own session, that is still "not resumable". For a registered session, it is now "a prewarm no real turn has claimed": first turn still armed FRESH.
  • SessionManager.fresh_start_unclaimed(key) (src/kiro_crew/session_parts/claims.py, bound in src/kiro_crew/session.py) answers that from the live session's first_turn, and consumes nothing.
  • The read itself is the existing one. The build takes it only when every input still matches, and reads for itself otherwise, so the prompt sent is byte-for-byte the same.

Found on the way (not changed here)

A send during an in-flight eager start does not join it. The start registers only after provider.start() returns (src/kiro_crew/session_allocation_parts/get_or_create.py:1045 start, :1197 register), so has_session answers no and the turn cold-starts its own provider. When that finishes, the won-race branch (get_or_create.py:1131-1141, :1263) shuts the duplicate down and claims the eager session. Tracked apart from this PR.

Backwards compatibility

Compatible: one new read-only SessionManager method; a claimed, resumed or unregistered session keeps today's path.

Removes nothing: the change only widens when an existing early read starts.

Tests

  • test_an_unclaimed_prewarm_reads_memory_before_the_claim_returns: the claim returns only once the reads have begun. Red on main (assert [] == ['Summarize the deploy checklist.']).
  • test_a_cold_start_build_takes_its_prefetch_and_sends_the_same_context[unclaimed-prewarm]: through the real runner and real build_message, the prompt sent with the early read is byte-for-byte the prompt sent without it, and the build reads no memory itself. Red on main (the reads never start).
  • test_a_claimed_session_starts_no_memory_prefetch (renamed from ..._prewarmed_...): a registered, already-claimed session reads nothing early.
  • test_only_an_unclaimed_fresh_session_answers_fresh_start_unclaimed: FRESH yes; RESUMED, claimed and absent no; nothing consumed.
  • test_session_bindings.py inventory and turn_harness.py answers updated for the new method.

Red-before run: reverting only turn_admission.py fails the two new runner tests above (2 failed, 4 passed in the -k unclaimed set).

Local: test_session_memory_prefetch.py, test_chat_runner_startup_recall.py, test_session_bindings.py, test_chat_runner_turn_locks.py, test_chat_runner_turn_sequence.py: 495 passed. The 165 test files that reference the turn path or the harness: 7389 passed before the inventory/harness updates; the 4 failures were those two inventories, now fixed.

Manual verification

In-process timing (not a live gateway). The real runner and real build_message on a prewarmed, unclaimed session, with memory = a copy of a real default store's markdown memory (~1 MB history). The claim is a mock that returns at once. Main = this branch with turn_admission.py from origin/main. n = 12 per row, two interleaved rounds, serial, under the test lock, 1-min load 14.6–17.2.

Post-claim steps Side claim → memory read done (p50, range) claim → prompt sent (p50, range)
none main 0.746 s (0.676–0.875) 0.813 s (0.738–0.946)
none branch 0.792 s (0.707–0.892) 0.852 s (0.767–0.956)
1.1 s stand-in main 1.877 s (1.813–2.078) 1.952 s (1.883–2.146)
1.1 s stand-in branch 0.774 s (0.687–0.835) 1.349 s (1.224–1.374)

The 1.1 s stand-in is a sleep before model settle, sized to R15's measured post-claim total on a quiet gateway (set mode, model settle, replay, store prep, dispatch gates). The read itself is 0.66–0.76 s p50 on both sides. With no post-claim work there is nothing to overlap, so no gain. With it, the prompt goes out 0.60 s sooner. A live send → first-token run was not done: it needs a gateway on this branch.

Related Issues

Closes #26959

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 1, 2026 07:39
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b2c3a93b854d9fbd05364eb184d9cf8bc08d872c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b2c3a93b854d9fbd05364eb184d9cf8bc08d872c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The change adds one read-only check, which has one caller, and widens an existing early read. The prompt sent is byte-for-byte unchanged, and the defect is shown by a test that fails on main.

What this change ships

Inventory (3 items) — 3 justified

Intent: a new chat whose session the eager start already began reaches its first token sooner. This is a FIX (latency).

  1. On a session the eager start registered but no turn has used yet, the first send's memory read now runs during the claim instead of after it. — justified
  2. A new read-only check on SessionManager, fresh_start_unclaimed, has one caller (turn_admission.py) and does not overlap first_turn_history_owed_pending, which reads a different flag. — justified
  3. The test inventory and the harness answers now include the new method. — justified

[FIRST-PRINCIPLES-REVIEWED] b2c3a93

Description read

[DESCRIPTION-READ] 40b57ba0e483565c0843b4d897ebe7ce22095fb2c9be0bdfdcc485d8238680df

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b2c3a93b854d9fbd05364eb184d9cf8bc08d872c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The change reuses the existing early memory read under one more read-only condition, and the build still checks that every input matches before it uses the result, so the prompt sent does not change.

Notes

  • If another turn claims the prewarm between fresh_start_unclaimed and this turn's claim, the early read is wasted. That costs time but does not change the result. The only evidence is in-process timing; a live send → first-token measurement on a gateway would confirm the 0.6 s gain.

[DESIGN-REVIEWED] b2c3a93

Description read

[DESCRIPTION-READ] 20100e60a8eed792d0566b560a24e99743b53860bd78ecad9f26bdd2486a0eaf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b2c3a93b854d9fbd05364eb184d9cf8bc08d872c — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • The memory-reads wording in docs/system-specs/modules/memory-skills-hooks.md:1768 and docs/architecture/context-management.md:180 still covers only the cold start. It should also name the claim of an unclaimed FRESH prewarm.

[OPUS-REVIEWED] b2c3a93

Verdict parsed from the review's SHA-scoped output markers for commit b2c3a93b854d9fbd05364eb184d9cf8bc08d872c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b2c3a93b854d9fbd05364eb184d9cf8bc08d872c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b2c3a93b854d9fbd05364eb184d9cf8bc08d872c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b2c3a93

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b2c3a93b854d9fbd05364eb184d9cf8bc08d872c false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the perf/new-chat-early-context branch from 7d0cc05 to b2c3a93 Compare November 1, 2026 09:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 1, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 8) red on head 7d0cc05: test_sandbox_protected_name_holds.py::TestLeafOnlyPopulationIsRecorded::test_leaf_only_count_has_not_grown[cc|standard|strict] (leaf-only count one over). Unrelated to this diff: it fails on clean origin/main 7e049c0, tracked in #26982, fix in #26983. Coverage Gate is red because that shard is. I will rerun the failed jobs once after #26983 merges.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 1, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 5) red on head b2c3a93: test_warm_task_brief.py::test_old_naive_rows_read_as_local_time[600-False]. Unrelated to this diff: it also fails locally with origin/main's sources, and on a docs-only PR (#26966). Tracked in #26991 (cause: DST fall-back hour, details there). Coverage Gate is red because that shard is. I will rerun the failed jobs once after 10:10 UTC, when the ambiguous hour has passed.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 1, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:

Approving. A new chat whose session the eager spawn already started now begins its first-message embed and session-start memory reads beside the claim instead of after it, which the in-process numbers put at about 0.6 s off the prompt send when the post-claim steps take about 1.1 s.

What I checked:

  • turn_admission.py:881-886 keeps main's rule for an unregistered key and, for a registered one, reads only when the live session's first turn is still FRESH. claims.py fresh_start_unclaimed reads first_turn and consumes nothing, so the claim still takes the one-shot observation and the build decision is unchanged.
  • The prompt cannot drift: the build uses the prefetch only when every input still matches, and the byte-for-byte test now covers the prewarm case.
  • Mutations, run locally on this head: replacing the fresh_start_unclaimed call with False reddens test_an_unclaimed_prewarm_reads_memory_before_the_claim_returns and test_a_cold_start_build_takes_its_prefetch_and_sends_the_same_context[unclaimed-prewarm]. Widening fresh_start_unclaimed to anything but NOTHING_ARMED reddens test_only_an_unclaimed_fresh_session_answers_fresh_start_unclaimed[unclaimed-resumed]. Both restored.

Follow-ups, not blocking:

  1. The check runs before the claim. If the FRESH prewarm is reset, evicted or recycling in between, or two sends race on it, the claim can end up resuming or not owning the first turn, and the early reads are thrown away. That costs one embed plus the V1 reads on the recall pool, bounded and rare, so I am fine with it; a comment at the gate saying so would help.
  2. memory-skills-hooks.md (around line 1767) and context-management.md (around line 180) still say the early memory reads start only when the turn starts the session itself. They now also start for an eager prewarm no turn has claimed yet; a clause in each would keep the specs matching.

@CrysisDeu CrysisDeu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at b2c3a93b. The store comes from the turn's own member before the early read starts, a private member never reads early, and saved reads are filed by store, so one store's reads cannot feed another's build. If another turn claims the session first, this turn's early read is simply unused. 0.60 s first-turn gain measured in-process with a mock claim. CI green.

Nit: drop the early read when the claim returns is_new=False instead of holding it for up to 60 s.

Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b2c3a93. No blocking findings; LGTM.

Non-blocking notes (follow-up is fine):

  • rc/kiro_crew/dashboard/chat_turn/turn_admission.py:881 - When a session is registered, the gate reads fresh_start_unclaimed before this turn holds the lease. If another turn on that key (a sibling alias or a channel-side turn) claims the FRESH prewarm first, this turn has already started a memory read the build will
  • rc/kiro_crew/session_parts/claims.py:347 - FirstTurnState is imported inside the function. If no import cycle forces this, a module-level import would match the rest of claims.py. Cosmetic.
  • est/test_session_memory_prefetch.py:460 - No test covers a registered session where fresh_start_unclaimed is True and resumable_hint is True, but the claim comes back resumed=True or not new (the prewarm was claimed or resumed between the gate and the claim). A test that asserts the reserved prefetch is dropped

@bolichen97
bolichen97 merged commit 59c7a87 into main Nov 2, 2026
151 of 154 checks passed
@bolichen97
bolichen97 deleted the perf/new-chat-early-context branch November 2, 2026 03:36
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(chat): read a prewarmed new chat's memory beside its claim

4 participants