Skip to content

perf(chat): embed a new chat's first message beside session startup - #21923

Merged
chenmingwei23 merged 1 commit into
mainfrom
perf/first-turn-prep-off-send-path
Oct 19, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
perf/first-turn-prep-off-send-path

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new chat's first message reaches the agent sooner: the embedding its session-start memory needs runs beside session startup instead of after it, and the first turn waits for that embedding no longer than a short, configurable time.

A new chat's first message waits for session startup (4.4 s p50: process spawn plus session/new), then for the session-start build, which starts only after startup returns. 80 to 98% of that build is one embedding of the first message. That embedding ranks the task facts, past episodes and lessons the session starts with. One host measured the build at 0.63 to 14.8 s, the memory part at 0.55 to 14.76 s. The 5 s prompt-build deadline did not bound it: it limits the wait in the embedding queue, and a build whose own inference is already running waits for it to finish (#21905).

Why it matters

Every new chat pays the build after startup. A message that arrives with its chat (a dispatched worker, session_send, cron, Slack) pays startup and build one after the other. A matched benchmark measured a new chat's first reply at 10.7 s against 5.4 s for bare kiro-cli at the same model and effort, and one dispatched worker's first message spent 14.8 s on the embedding alone.

Not a goal

  • Precomputing the query-free blocks (agent prompt, preferences, skills index) at eager-spawn or slot-create time. On a copy of a real store they cost 40 to 60 ms warm: agent prompt 13 ms, preference and index reads 23 ms, skills 4 ms, member and workspace 4 ms. V1 lessons are ranked against the first message, so they are not query-free. A cache of prebuilt blocks would risk sending stale content to save under 60 ms, so it is left out.
  • Changing the session.pool_size default. The configuration page now recommends 1 for hosts whose first messages arrive with their chat.
  • Moving the ranked reads off the send path. Only the embedding moves; scoring and rendering (about 0.3 s on that store) stay in the build.
  • Private crew members. Their session start runs no embedding search.
  • Other surfaces (Slack, cron, task runner, sub-agents). They start no early embed and keep the five-second deadline.
  • Cancelling the early embed of an aborted turn. It finishes one inference, and a retry of the same message reads the cached vector.

What changed (motivation → approach → change)

The slow step needs only the message text, so it can start when the turn starts. A wait can be bounded only for a caller that joins a running inference, never for the caller running it, since native inference cannot be stopped. So the dashboard runs the inference, and the build joins it.

Part What it does
embed_startup_query in context_assembly/store_admission.py, on the context.py facade Embeds the first message through the store the activity block reads, behind the same gates (inject_activity, the memory group, a vector store), and only while memory.startup_recall_timeout_secs is above 0
dashboard/chat_runner.py Runs it on the recall pool: before get_or_create when no session is registered, so it runs beside session/new, and again just before every session-start build, where it is a cache hit or joins the running inference. The gates in dashboard/chat_turn/turn_context.py skip temporary chats and private members, and keep a message the turn rewrites before its build (an @prompt body, drained sub-agent notices, pending context) for the second start only
embedding_in_flight in embeddings_shared.py, served by the embeddings.py facade Says whether the shared embedder is running the inference for a text
context_assembly/budget.py, called from context_assembly/turn_message.py A build that finds its first message's embedding running waits memory.startup_recall_timeout_secs (new in config/memory_sections.py, read by config/section_builders.py: default 1.0 s, clamped to 0–10). Every other build keeps _PROMPT_BUILD_EMBED_TIMEOUT_SECS, 5.0 s. Past the wait, task facts, episodes and lessons rank by keyword, the path a missing vector already takes
Docs src/kiro_crew/docs/configuration.md gains the key and a session.pool_size: 1 note for worker, session_send, cron and Slack traffic. The context-management page, the config and memory specs and config-baseline.json follow

The shared embed cache is keyed by model and text, and a second caller of the same text joins the inference that is running. So the build reads the vector it would have computed, and the first turn is byte-identical whenever the embedding finishes in time. Only the vector is cached: memory and lessons are read when the build runs, so a lesson written after the early embed still reaches the first turn. The early embed may wait 30 s for the embedder, longer than the 10 s ceiling on the wait a build joins it for. The build looks for the running embed as it starts; if the recall pool is too busy to have started it, the build embeds for itself, as today.

This bounds the wait at the mechanism level, for the dashboard's first turn only. The cause-level fix is to run every build's own inference on the recall pool inside its deadline, so that any build_message caller can stop waiting. That would make the wait a hard bound on every surface, which changes ranking on Slack, cron, sub-agents and the task runner, so it is left out here and #21905 stays open for it.

Backwards compatibility

Compatible: the new key is optional, and 0 gives today's behavior (no early embed; the build embeds for itself). The first turn's content is unchanged whenever the embedding finishes inside the wait. A dashboard session-start build stops waiting for its embedding after 1 s, where today it waits for the inference to finish. On a host whose first-message embedding takes longer, that session starts with task facts, episodes and lessons ranked by keyword instead of by meaning. Other surfaces keep the five-second deadline. One gets the shorter wait only when the same text is already being embedded, and then it joins that inference.

Removes nothing: no key, path, record or public function is removed; a session-start dashboard turn gains one or two background embeds of the same text on the recall pool.

Tests

Test Locks in
test_chat_runner_startup_recall.py cold start With nothing registered, the embed starts before get_or_create returns (the fake session start completes only once it has), and again before the build
test_chat_runner_startup_recall.py rewritten message An @prompt cold start embeds only the expanded text, before the build. The rewrite check covers @, drained notices and pending context
test_chat_runner_startup_recall.py pre-warmed session The build of an eager-spawned session runs with an early embed of the same text
test_chat_runner_startup_recall.py no session-start memory Warm turn, native resume, resumable cold start: no embed
test_chat_runner_startup_recall.py gate Temporary chat, temporary execution, private member, blank message, no builder: no embed
test_chat_runner_startup_recall.py allowance The early embed's allowance outlasts the longest wait a build can join it for
test_memory_v1_golden.py byte-identical Three fixed queries: the first turn is byte-identical to one built without the early embed, with one inference in total
test_memory_v1_golden.py late memory A lesson and an episode written after the early embed reach the first turn
test_memory_v1_golden.py declines Activity off, memory off, persistence off, a blank message, a store with no vectors: neither the early embed nor the build runs an inference. A zero wait: no early embed, and the build runs its own one inference
test_memory_v1_golden.py wait runs out The build joins a held inference; moving one manual clock by the configured 1 s returns the keyword-ranked turn
test_startup_recall_wait.py The key's clamping. A build waits 5 s unless its embedding is in flight, then the configured wait (0 keeps 5 s). embedding_in_flight holds only while the inference runs

Changed: turn_harness.py answers resumable_hint (False, a healthy fresh session); test_chat_runner_config_load_failure.py fails only the read the turn's loop makes, since the early embed reads the config from a worker inside the same window; test_context_management_doc.py pins the five-second deadline and the one-second joined wait.

Before the fix: prove.py's whole-diff revert is inconclusive, because the tests then fail at import on the names this change adds. So each behavior was reverted by hand in a scratch worktree, and each revert turns its tests red: no embed beside session/new; no pre-build embed after a cold-start one; embedding a rewritten message's claim-time text; the short wait for every build; no short wait for a joined embed; an early embed that embeds nothing; an allowance below the wait ceiling; an in-flight check that is always false.

Determinism: the 48 new and changed tests pass 20 runs at -n0, 10 at -n 2 --dist loadgroup beside test_lessons_context_embedding_deadline.py and test_embeddings_resource_budget.py (xdist is capped at 2 on this host), and one shuffled run (seed 16078). Clocks are manual and each wait ends on a signal; the timeouts only name a hang. Since main split embeddings.py into owners, the wait-runs-out test puts its clock on embeddings_settings, whose EmbeddingWork.expired ends the joined wait (the facade binds no clock). On the rebased head that test passes 20 runs at -n0 and 10 at -n 2 beside the same two files, and the 67 tests of test_memory_v1_golden.py and the two new test files pass one shuffled run (seed 29414).

Nearby inputs:

Input Path through this change Tested by
Crewmate DM first turn after a reset Private member: the gate skips it; a V2 start runs no embedding search gate, private-member
Worker started by session_create No session registered: embed beside session/new, joined before the build cold start
Normal chat, eager spawn finished Embed just before the build pre-warmed session
Chat with memory off or no vector store No embed golden declines
Temporary chat No embed gate, temporary-chat
First message that is an @prompt Embed of the expanded text, before the build rewritten message

Manual verification

Measured locally, without a dashboard token: an isolated home holding a copy of a real 114 MB store (7,979 facts, 364 lessons), the real Qwen3 model on llama.cpp, host load 10 to 14, and session startup simulated as 4.4 s. The figure is the time from startup done to build done, p50 of 5 to 8 runs. It was measured on this PR's first revision, whose waits in these three cases match the final one's.

First message Today Startup on the send path Session already started
R3 P1, 34 chars 778 ms 292 ms 396 ms
R3 P2, 128 chars 1,253 ms 504 ms 873 ms
R3 P3, 236 chars 1,641 ms 452 ms 1,235 ms
Worker brief, 3,000 chars 14,073 ms 1,264 ms 1,278 ms

How often the 1 s wait runs out, from this host's gateway log (Context timings [build_session_context], all 12 session-start builds, 2026-10-18 07:31–07:59Z). The memory section is the embedding plus about 0.25 s of ranking and rendering (measured on the store copy with the vector ready).

Memory section Builds
0.55–0.78 s 7
1.05–1.30 s 3
3.11 s 1
14.76 s (3,000-char worker brief) 1

p50 is 0.62 s. With the embed started just before the build (a pre-warmed session), 3 to 5 of these 12 would have ranked by keyword at the 1 s default. With it started beside a 4.4 s startup, 1 would: the worker brief.

R3's success measure needs a dashboard token and is still to run: B P1 cold at opus-5.5 · medium, n ≥ 5 per case. Startup on the send path: "send → prompt at KAS" p50 from 7.1 s to ≤ 4.5 s, and to ≤ 1.5 s with pool_size: 1. Eager spawn finished: from 1.1 s to ≤ 0.4 s. On a fixed query set, the recall selects the same memory items whenever it finishes inside the time box. The table above suggests the eager-spawn target will be missed: the embedding itself takes 0.1 to 1 s on this host, and only the build's own 0.3 s of work runs beside it.

Related Issues

Refs #21905

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision df26659dc948051d7da903ee192c9e8fb1063923 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of df26659dc948051d7da903ee192c9e8fb1063923 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check the 1.0 s default on a slow-embedding host: a pre-warmed session starts its embed just before the build, so its first turn ranks by keyword.

[DESIGN-REVIEWED] df26659

Description read

[DESCRIPTION-READ] ea9eecf29268c8ae2f538ea6d1ac0606875239ef085c3d6acf0115945e14d571

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of df26659dc948051d7da903ee192c9e8fb1063923 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Pre-warmed dashboard chats (session.eager_spawn defaults on) get no head start. They only get a 1 s cutoff on their embedding, and nothing supports the 1.0 default.

Not justified as shipped

  • Item 3 — inherited: the default 1.0 has no measurement behind it. The description measured embeddings at "0.55 to 14.76 s", and on a pre-warmed session the second _prewarm_startup_query starts only one asyncio.sleep(0) ahead of the build. So on that path the change never makes the embedding earlier. Any embedding over 1 s now ranks by keyword, where before the build waited for it.
    Clears when: a measurement on the eager-spawn path shows how often first turns embed within 1 s, or the default changes to the value that data supports.
  • Item 6 — rides along: the session.pool_size: 1 advice is about Slack, cron and worker startup, which the Goal leaves out ("Other surfaces … start no early embed").

What this change ships

Inventory (6 items) — 4 justified

Intent: make a new dashboard chat's first reply arrive sooner by running its embedding beside session startup and capping the wait. This is an ADDITION (perf).

  1. A cold-start dashboard chat embeds its first message while session/new runs — justified
  2. Every session-start build starts or joins that embedding just before it runs — justified
  3. A dashboard first turn waits at most 1 s for an embedding that is already running, then ranks by keyword — inherited: the default value has no measurement
  4. New key memory.startup_recall_timeout_secs (0–10, 0 turns it off) — justified
  5. New embedding_in_flight and embed_startup_query helpers, one consumer each — justified
  6. The configuration page now recommends session.pool_size: 1 for traffic whose first message arrives with its chat — rides along

[FIRST-PRINCIPLES-REVIEWED] df26659

Description read

[DESCRIPTION-READ] 799453cb2f9e15fc27a8f9276aa4b5a8e4875f115400a0ccd6d0b01f871b7d32

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of df26659dc948051d7da903ee192c9e8fb1063923 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] df26659

False positive or not applicable? A repository writer can comment:
/ai-review override gpt df26659dc948051d7da903ee192c9e8fb1063923: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed df26659dc948051d7da903ee192c9e8fb1063923 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] df26659

Verdict parsed from the review's SHA-scoped output markers for commit df26659dc948051d7da903ee192c9e8fb1063923.

False positive or not applicable? A repository writer can comment:
/ai-review override fable df26659dc948051d7da903ee192c9e8fb1063923: <one-sentence reason>

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Pre-warmed sessions get a shorter wait and no head start span=5972f434c893

Disposition: rebutted. The 1 s wait on a pre-warmed session's first turn is this PR's goal ("waits for that embedding no longer than a short, configurable time"). The latency gap it follows counts every new chat, also after eager spawn, and Backwards compatibility states the trade.
The measurement the item asks for is now in the description's Manual verification. This host's gateway log holds 12 session-start builds. Their memory section, the embedding plus about 0.25 s of ranking, took 0.55–0.78 s in 7, 1.05–1.30 s in 3, then 3.11 s and 14.76 s.
With the embed started just before the build (a pre-warmed session), 3 to 5 of those 12 would have ranked by keyword at the 1 s default. With it started beside a 4.4 s startup, 1 would.
memory.startup_recall_timeout_secs: 0 keeps today's wait on a host that prefers meaning ranking at any latency.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Record whether the cold-start prewarm ran instead of inferring it from embedding_in_flight(query) span=8a417cfd0ae1

Disposition: rebutted. A build that finds its text in flight joins that inference whoever started it, and a joined wait can end on time. The shorter wait follows from joining, so it fits any joiner.
Recording "the cold-start prewarm ran" would also take pre-warmed sessions out of the time box, which this PR's goal keeps (see the disposition on the Watch item).
Another surface gets the shorter wait only while the same text is already being embedded. The description's Backwards compatibility now says so.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 3: the default startup_recall_timeout_secs: 1.0 has no measurement behind it span=68ee334874d4

Disposition: rebutted, with the measured distribution the item asks for, now in the description's Manual verification.
This host's gateway log holds 12 session-start builds (2026-10-18 07:31–07:59Z). The memory section, the embedding plus about 0.25 s of ranking, took 0.55–0.78 s in 7, 1.05–1.30 s in 3, then 3.11 s and 14.76 s. The p50 is 0.62 s.
So most first turns finish within 1 s; "0.55 to 14.76 s" is the two ends of that range. A cold start's embed also runs beside the ~4.4 s session startup, so there only the 14.76 s worker brief would outlast the wait.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 2: the second early embed is symptom-level, since the deadline cannot bound a build's own inference span=cb5d2dbf6b88

Disposition: rebutted as out of proportion for this PR. The description now says this sits at mechanism level and names the cause-level fix.
The subtraction, running every build's own inference on the recall pool inside its deadline, makes the wait a hard bound for every build_message caller. Slack, cron, sub-agents and the task runner would then rank by keyword past 5 s, a ranking change on surfaces outside this PR's goal.
This PR bounds only the dashboard's first turn. #21905 stays open for the cause-level fix: the PR links it with Refs, not Fixes.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the perf/first-turn-prep-off-send-path branch from 80a1c94 to 72ad463 Compare October 18, 2026 16:48
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_runner.py and docs/system-specs/modules/memory-skills-hooks.md (plus files beside them) since this head went green (900c33f -> a9898a0); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • The default trades ranking quality for latency, and the change gives no measurement of what that costs span=9c699d6b3091

Disposition: rebutted. The trade is this PR's goal: the first turn waits for its embedding "no longer than a short, configurable time". The description's Manual verification measures it for both starts.
This host's gateway log holds 12 session-start builds. Their memory section took 0.55–0.78 s in 7, 1.05–1.30 s in 3, then 3.11 s and 14.76 s (p50 0.62 s).
Pre-warmed start: 7 to 9 of the 12 finish inside the 1 s wait. A pooled session (session.pool_size: 1) has little head start, so it gets about that share. Cold start beside the 4.4 s session/new: 11 of 12; only the 3,000-char worker brief misses.
A miss ranks by keyword, the path a missed embed already takes, and cuts that turn's wait to 1 s from up to 14.8 s. A host that prefers meaning ranking sets memory.startup_recall_timeout_secs up to 10, or 0 for today's behaviour.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • One probe at the start of the build sets the deadline for every embed the build makes span=4bb5697ce84f

Disposition: rebutted. The session-start build makes no other embed under that deadline. _prompt_build_embedding_deadline has one caller, context_assembly/turn_message.py:297, and its with block wraps only build_session_context.
Every embed in that call is of the first message: three calls, one inference (test_startup_embeds_the_request_once_and_explicit_readers_share_it pins embed_fn_calls == [QUERY] * 3). The skills, member and workspace sections embed nothing.
So each embed under the shortened deadline joins the probed inference or reads its cached vector, as the docstring says. A build whose probe finds nothing running keeps 5 s.
One exception: if the joined inference ends with no vector, the next reader embeds the same text again within what is left of the short wait. That needs the early embed's backend call to fail, and the turn then ranks by keyword unless the second inference finishes in time.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 4: memory.startup_recall_timeout_secs is a config key with no named operator who needs it span=f980e927f9a8

Disposition: rebutted. The operator is one on a host whose first-message embedding often takes over a second, measured in the description: on this host 5 of 12 session-start memory sections ran 1.05–14.76 s.
There the 1 s default ranks those first turns by keyword. An operator who prefers meaning-ranked memory needs a longer wait (up to 10 s), or 0 for today's behaviour. The Design and Opus lanes argue for exactly that operator on this head.
As a constant, that choice would be a code edit. _PROMPT_BUILD_EMBED_TIMEOUT_SECS differs in kind: it is a safety bound under the gateway's 25 s loop-stall budget, not a latency-versus-ranking preference.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • On an eager-spawned session the pre-build embed gives no head start but cuts the build's wait to 1 s span=e48125549479

Disposition: rebutted. The short wait on that path is this PR's goal: the first turn waits for its embedding no longer than a short, configurable time.
The pre-build embed is what bounds that path. Without it the build runs its own inference and waits for it to finish (#21905; 14.8 s on this host). Shortening by elapsed time, max(1, 5 − elapsed), keeps the full 5 s there, so the default path never gets the short wait.
The cost is measured in the description: on that path 7 to 9 of this host's 12 session-start builds finish inside 1 s. A miss ranks by keyword, the path a missed embed already takes. Backwards compatibility states the trade, and memory.startup_recall_timeout_secs (up to 10, or 0) keeps meaning ranking where a host prefers it.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 18, 2026 18:02
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the perf/first-turn-prep-off-send-path branch from 72ad463 to 4cf3287 Compare October 19, 2026 03:05
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 2375643f7d (was a9898a0e21): still one commit, now 4cf3287b04. Two files conflicted, both because main moved code into owner modules. In both, this PR's only change inside the moved code was new code, so I kept main's side and put the new code where main now keeps its neighbours.

File Main's change How it was resolved
src/kiro_crew/dashboard/chat_runner.py #21918 moved the block after _turn_start_priority into chat_turn owners Kept this PR's three new definitions (_STARTUP_QUERY_PREWARM_SECS, _prewarm_startup_query, _embed_startup_query_off_loop) right after _turn_start_priority, which stays in the runner. Took main's side for the rest of the block. The imports and the two call sites in _run_chat merged cleanly, and the code around them is the same as before
src/kiro_crew/embeddings.py #21946 split the module into owners behind a mirror facade embedding_in_flight moved to embeddings_shared.py, next to _shared_sync_embed and the cache it reads. The facade's TYPE_CHECKING list names it. Callers and tests still reach it as kiro_crew.embeddings.embedding_in_flight

Main's other changes stay as main has them, including #21907 and #21937 (throttle retries) in chat_runner.py and #21913's skill_trigger_text and unusable set in context_assembly/turn_message.py. The one-line change this PR makes in turn_message.py (query=text on the embedding deadline) merged cleanly beside them.

One follow-on edit, not a conflict: test_memory_v1_golden.py::test_a_build_ranks_by_keyword_once_its_wait_runs_out put its manual clock on kiro_crew.embeddings. The facade now binds no clock, so ManualClock.install would raise there. The test now puts the clock on kiro_crew.embeddings_settings, where EmbeddingWork.expired ends the joined wait.

The diff went from 20 files (+847/−20) to 21 files (+851/−20): embedding_in_flight's 15 lines moved into embeddings_shared.py, plus one line in the facade list and three in the golden test. Every function main defines in a touched file is still there.

Checked locally: isort, flake8, black and mypy on the touched files; the line-cap, comment-history, sync-IO, loop-lock, memory-store-seam and docs-lint gates; and the related tests (this PR's tests, the embeddings facade and mirror guards, the runner composition and base-surface contracts, config baseline and docs, skill-trigger and lesson-injection tests). They passed: 850 tests after the first rebase and 538 on the final head. The changed test passed 20 runs at -n0 and 10 at -n 2 beside two other files, and a shuffled run of 67 tests passed with seed 29414.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (1) is red on 4cf3287b04 with one test: test/test_dashboard_chat_session_meta.py::TestSessionRename::test_resumed_session_restores_tags_and_auto_tagged raised PermissionError: [WinError 5] when the conversation log renamed its temp file onto dashboard_s1.jsonl (atomic_write.py:437). This PR does not touch the conversation log, atomic_write.py, the dashboard state or that test, and the test passes on Linux on this head. It is the known Windows tmp+rename flake (#18460, #20697, #21475, #21548, #21800), now recorded for this test as #22047. Rerunning that one job once.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • The default trades quality for latency without telling anyone span=f7de07492c4b

Disposition: rebutted. The description's Manual verification meets the item's second clearing condition: on the measured host the 1 s default keeps meaning-ranked recall for most first turns.
That host's gateway log holds 12 session-start builds. Their memory section, the embedding plus about 0.25 s of ranking, took 0.55–0.78 s in 7, 1.05–1.30 s in 3, then 3.11 s and 14.76 s (p50 0.62 s).
Cold start, with the embed beside the 4.4 s session/new: 11 of 12 keep meaning ranking; only the 3,000-char worker brief misses. Pre-warmed session: 7 to 9 of 12.
The trade is written where an operator reads it: the configuration page's memory.startup_recall_timeout_secs row says that past the wait those blocks rank by keyword, and Backwards compatibility says the same. A miss takes the path a missed embedding already takes.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • The early embed's gates are a hand-kept copy of the build's span=755ec465d258

Disposition: rebutted. Tests already fail when the build's embed text differs from the early embed's, the item's second clearing condition.
test_chat_runner_startup_recall.py drives the real _run_chat and asserts the early embed got exactly the text build_message was called with: [MESSAGE, MESSAGE] on a cold start, [expanded] for an @prompt body.
test_memory_v1_golden.py::test_an_early_embed_leaves_the_first_turn_byte_identical (three queries) asserts the build adds no inference after the early embed, so a different cache key fails it. test_an_early_embed_runs_only_where_the_build_embeds pins the gates: activity, memory and persistence off decline on both sides.
Drift cannot put stale memory in the first turn: only the query's vector is cached, keyed by model and text, and the store is read when the build runs. The embed just ahead of the build takes the same message the build is called with.
A rewrite missing from _message_rewritten_before_build costs one wasted inference ahead of that embed, which the build's wait bounds; past the wait the turn ranks by keyword, the path a missed embedding already takes.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 3: no issue, RFC or measurement supports the 1.0 s default over 5.0 s span=09b06767bda7

Disposition: rebutted. The measurement is in the description's Manual verification, as answered on an earlier head (span 68ee334874d4).
This host's gateway log holds 12 session-start builds (2026-10-18 07:31–07:59Z). The memory section, the embedding plus about 0.25 s of ranking, took 0.55–0.78 s in 7, 1.05–1.30 s in 3, then 3.11 s and 14.76 s (p50 0.62 s). "0.55–14.76 s" is the two ends of that range.
At 1 s, 11 of 12 cold starts and 7 to 9 of 12 pre-warmed first turns keep meaning ranking. A 5.0 s default would let a first turn wait five times as long to keep it for 2 to 4 more pre-warmed turns of those 12, and for no cold start.
The 1.0 s default is the trade the PR states in its Goal ("no longer than a short, configurable time") and in Backwards compatibility.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 4: memory.startup_recall_timeout_secs has no named operator need for a non-default value span=99829184bbb3

Disposition: rebutted, as on the earlier head (span f980e927f9a8). The operator is one on a host whose first-message embedding often takes over a second: on this host 5 of 12 session-start memory sections ran 1.05–14.76 s.
There the 1 s default ranks some first turns by keyword. An operator who prefers meaning-ranked memory needs a longer wait (up to 10 s), and one who wants today's behaviour sets 0. The Design lane's Watch item on this head speaks for that operator.
As a constant, that choice would be a code edit. _PROMPT_BUILD_EMBED_TIMEOUT_SECS differs in kind: it is a safety bound under the gateway's 25 s loop-stall budget, not a latency-versus-ranking preference.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 19, 2026
A new chat's first turn waited for session startup and then for the
session-start build, whose memory section is one embedding of the first
message (80-98% of the build). The dashboard now starts that embedding on
the recall pool before the build: beside get_or_create when the turn
starts the session itself, and again just ahead of every session-start
build, where it is a cache hit or joins the running inference. A message
the turn rewrites before its build (an @prompt body, drained notices or
pending context) is embedded only there. Only the query's vector is kept,
so a first turn carries exactly what it carried before whenever the
embedding finishes in time.

A build that finds its first message's embedding already running joins it
and waits memory.startup_recall_timeout_secs (default 1.0 s, clamped to
0-10; 0 turns the early embed off), then ranks by keyword. Every other
build keeps _PROMPT_BUILD_EMBED_TIMEOUT_SECS, 5.0 s. The early embed's own
allowance (30 s) outlasts the longest wait a build can join it for. The
configuration page recommends session.pool_size 1 for hosts whose first
messages arrive with their chat.

Refs #21905
@iamwhatever
iamwhatever force-pushed the perf/first-turn-prep-off-send-path branch from 4cf3287 to df26659 Compare October 19, 2026 10:57
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 16779f653a (was 2375643f7d). New head df26659dc9.

What conflicted. #22057 (20fcbd54e8) hoisted the closures of _run_chat into chat_turn owners. Git reported two textual conflicts:

  • the turn_context import list in chat_runner.py: main added _restore_skills_context_after_compaction next to this PR's _startup_query_prewarm_applies;
  • the module docstring of chat_turn/turn_context.py: both sides appended a clause.

How it was resolved. Both import names kept, both docstring clauses joined. The two _prewarm_startup_query call sites merged cleanly and stay in the body of _run_chat (before get_or_create, and just before the session-start build). Every name they read is still bound there after the hoist. Nothing in the refactor was undone.

range-diff. git range-diff 2375643f7d..4cf3287b04 16779f653a..df26659dc9 shows only those two context hunks. No behaviour hunk changed: the parallel embed prewarm, the memory.startup_recall_timeout_secs time box (1.0 s) with keyword fallback, and the session.pool_size docs note are byte-identical. Shape: 21 files, +852/-21 (was +851/-20, the docstring wrap).

Local checks. 1257 tests pass at -n 2: this PR's own tests, the chat_runner composition, base-surface, turn-plumbing and shared-state guards, and the config and embeddings tests. isort, flake8, mypy and the flake-pattern ratchet are clean.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 3: the 1.0 s default has no measurement on the eager-spawn path span=76126b08d0ff

Disposition: rebutted. The measurement the clearing condition asks for is in the description's Manual verification, as answered on earlier heads (spans 68ee334874d4, 09b06767bda7). This head changes no behaviour hunk of 4cf3287; it only rebases onto #22057.
This host's gateway log holds all 12 session-start builds of 2026-10-18 07:31–07:59Z. The memory section (the embedding plus about 0.25 s of ranking) took 0.55–0.78 s in 7, 1.05–1.30 s in 3, then 3.11 s and 14.76 s.
The item is right that a pre-warmed session's embed starts just ahead of its build, so its embedding takes that section's time less the ranking: 7 to 9 of the 12 finish inside 1 s, and 3 to 5 rank by keyword. The description states this, and that the eager-spawn latency target will likely be missed.
The 1.0 s default is the trade the frozen Goal names ("no longer than a short, configurable time"). A miss ranks by keyword, the path a missed embedding already takes, and caps that turn's wait at 1 s rather than up to 14.8 s. A slower host sets memory.startup_recall_timeout_secs up to 10, or 0 for today's behaviour.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Item 6: the session.pool_size: 1 advice rides along span=84119b3a8d6d

Disposition: rebutted. The advice changes no code and no default. It is one note in the session.pool_size row of the configuration page, and the description declares it under Not a goal ("Changing the session.pool_size default").
It serves the frozen Goal, "A new chat's first message reaches the agent sooner", on the surfaces this PR gives no early embed. On worker, session_send, cron and Slack traffic the first message arrives with its chat, so session/new (4.4 s p50) stays on its send path; a pooled process is the existing setting that takes it off.
"Other surfaces … start no early embed" scopes the code change, not the docs. The note tells those operators which existing key moves their first-message latency, and what each pooled process costs (about 110 MB).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Check the 1.0 s default on a slow-embedding host, where a pre-warmed session's first turn ranks by keyword

Disposition: rebutted. The check ran on a slow-embedding host, and the description's Manual verification reports it: this host runs the real Qwen3 model on llama.cpp at load 10–14, and one of its session-start memory sections took 14.76 s.
Of that host's 12 session-start builds, a pre-warmed session keeps meaning ranking in 7 to 9 at the 1 s default and ranks by keyword in 3 to 5. Started beside a cold session/new, 11 of 12 keep it.
The trade is written where an operator reads it: the memory.startup_recall_timeout_secs row says that past the wait those blocks rank by keyword, that 0 turns the early embed off, and that the value is clamped to 0–10. Backwards compatibility says the same.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 merged commit 8e686e5 into main Oct 19, 2026
103 of 104 checks passed
@chenmingwei23
chenmingwei23 deleted the perf/first-turn-prep-off-send-path branch October 19, 2026 14:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants