Repository navigation
perf(chat): embed a new chat's first message beside session startup - #21923
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check the 1.0 s default on a slow-embedding host: a pre-warmed session starts its embed just before the build, so its first turn ranks by keyword. [DESIGN-REVIEWED] df26659 Description read[DESCRIPTION-READ] ea9eecf29268c8ae2f538ea6d1ac0606875239ef085c3d6acf0115945e14d571 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Pre-warmed dashboard chats ( Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: make a new dashboard chat's first reply arrive sooner by running its embedding beside session startup and capping the wait. This is an ADDITION (perf).
[FIRST-PRINCIPLES-REVIEWED] df26659 Description read[DESCRIPTION-READ] 799453cb2f9e15fc27a8f9276aa4b5a8e4875f115400a0ccd6d0b01f871b7d32 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
|
|
|
80a1c94 to
72ad463
Compare
|
|
|
|
72ad463 to
4cf3287
Compare
|
Rebased onto main
Main's other changes stay as main has them, including #21907 and #21937 (throttle retries) in One follow-on edit, not a conflict: The diff went from 20 files (+847/−20) to 21 files (+851/−20): Checked locally: isort, flake8, black and mypy on the touched files; the line-cap, comment-history, sync-IO, loop-lock, memory-store-seam and docs-lint gates; and the related tests (this PR's tests, the embeddings facade and mirror guards, the runner composition and base-surface contracts, config baseline and docs, skill-trigger and lesson-injection tests). They passed: 850 tests after the first rebase and 538 on the final head. The changed test passed 20 runs at |
|
Backend Tests (Windows) (1) is red on |
|
|
|
|
A new chat's first turn waited for session startup and then for the session-start build, whose memory section is one embedding of the first message (80-98% of the build). The dashboard now starts that embedding on the recall pool before the build: beside get_or_create when the turn starts the session itself, and again just ahead of every session-start build, where it is a cache hit or joins the running inference. A message the turn rewrites before its build (an @prompt body, drained notices or pending context) is embedded only there. Only the query's vector is kept, so a first turn carries exactly what it carried before whenever the embedding finishes in time. A build that finds its first message's embedding already running joins it and waits memory.startup_recall_timeout_secs (default 1.0 s, clamped to 0-10; 0 turns the early embed off), then ranks by keyword. Every other build keeps _PROMPT_BUILD_EMBED_TIMEOUT_SECS, 5.0 s. The early embed's own allowance (30 s) outlasts the longest wait a build can join it for. The configuration page recommends session.pool_size 1 for hosts whose first messages arrive with their chat. Refs #21905
4cf3287 to
df26659
Compare
|
Rebased onto main What conflicted. #22057 (
How it was resolved. Both import names kept, both docstring clauses joined. The two range-diff. Local checks. 1257 tests pass at |
|
|
|
Problem / Motivation
Goal: A new chat's first message reaches the agent sooner: the embedding its session-start memory needs runs beside session startup instead of after it, and the first turn waits for that embedding no longer than a short, configurable time.
A new chat's first message waits for session startup (4.4 s p50: process spawn plus
session/new), then for the session-start build, which starts only after startup returns. 80 to 98% of that build is one embedding of the first message. That embedding ranks the task facts, past episodes and lessons the session starts with. One host measured the build at 0.63 to 14.8 s, the memory part at 0.55 to 14.76 s. The 5 s prompt-build deadline did not bound it: it limits the wait in the embedding queue, and a build whose own inference is already running waits for it to finish (#21905).Why it matters
Every new chat pays the build after startup. A message that arrives with its chat (a dispatched worker,
session_send, cron, Slack) pays startup and build one after the other. A matched benchmark measured a new chat's first reply at 10.7 s against 5.4 s for bare kiro-cli at the same model and effort, and one dispatched worker's first message spent 14.8 s on the embedding alone.Not a goal
session.pool_sizedefault. The configuration page now recommends1for hosts whose first messages arrive with their chat.What changed (motivation → approach → change)
The slow step needs only the message text, so it can start when the turn starts. A wait can be bounded only for a caller that joins a running inference, never for the caller running it, since native inference cannot be stopped. So the dashboard runs the inference, and the build joins it.
embed_startup_queryincontext_assembly/store_admission.py, on thecontext.pyfacadeinject_activity, the memory group, a vector store), and only whilememory.startup_recall_timeout_secsis above 0dashboard/chat_runner.pyget_or_createwhen no session is registered, so it runs besidesession/new, and again just before every session-start build, where it is a cache hit or joins the running inference. The gates indashboard/chat_turn/turn_context.pyskip temporary chats and private members, and keep a message the turn rewrites before its build (an@promptbody, drained sub-agent notices, pending context) for the second start onlyembedding_in_flightinembeddings_shared.py, served by theembeddings.pyfacadecontext_assembly/budget.py, called fromcontext_assembly/turn_message.pymemory.startup_recall_timeout_secs(new inconfig/memory_sections.py, read byconfig/section_builders.py: default 1.0 s, clamped to 0–10). Every other build keeps_PROMPT_BUILD_EMBED_TIMEOUT_SECS, 5.0 s. Past the wait, task facts, episodes and lessons rank by keyword, the path a missing vector already takessrc/kiro_crew/docs/configuration.mdgains the key and asession.pool_size: 1note for worker,session_send, cron and Slack traffic. The context-management page, the config and memory specs andconfig-baseline.jsonfollowThe shared embed cache is keyed by model and text, and a second caller of the same text joins the inference that is running. So the build reads the vector it would have computed, and the first turn is byte-identical whenever the embedding finishes in time. Only the vector is cached: memory and lessons are read when the build runs, so a lesson written after the early embed still reaches the first turn. The early embed may wait 30 s for the embedder, longer than the 10 s ceiling on the wait a build joins it for. The build looks for the running embed as it starts; if the recall pool is too busy to have started it, the build embeds for itself, as today.
This bounds the wait at the mechanism level, for the dashboard's first turn only. The cause-level fix is to run every build's own inference on the recall pool inside its deadline, so that any
build_messagecaller can stop waiting. That would make the wait a hard bound on every surface, which changes ranking on Slack, cron, sub-agents and the task runner, so it is left out here and #21905 stays open for it.Backwards compatibility
Compatible: the new key is optional, and
0gives today's behavior (no early embed; the build embeds for itself). The first turn's content is unchanged whenever the embedding finishes inside the wait. A dashboard session-start build stops waiting for its embedding after 1 s, where today it waits for the inference to finish. On a host whose first-message embedding takes longer, that session starts with task facts, episodes and lessons ranked by keyword instead of by meaning. Other surfaces keep the five-second deadline. One gets the shorter wait only when the same text is already being embedded, and then it joins that inference.Removes nothing: no key, path, record or public function is removed; a session-start dashboard turn gains one or two background embeds of the same text on the recall pool.
Tests
test_chat_runner_startup_recall.pycold startget_or_createreturns (the fake session start completes only once it has), and again before the buildtest_chat_runner_startup_recall.pyrewritten message@promptcold start embeds only the expanded text, before the build. The rewrite check covers@, drained notices and pending contexttest_chat_runner_startup_recall.pypre-warmed sessiontest_chat_runner_startup_recall.pyno session-start memorytest_chat_runner_startup_recall.pygatetest_chat_runner_startup_recall.pyallowancetest_memory_v1_golden.pybyte-identicaltest_memory_v1_golden.pylate memorytest_memory_v1_golden.pydeclinestest_memory_v1_golden.pywait runs outtest_startup_recall_wait.py0keeps 5 s).embedding_in_flightholds only while the inference runsChanged:
turn_harness.pyanswersresumable_hint(False, a healthy fresh session);test_chat_runner_config_load_failure.pyfails only the read the turn's loop makes, since the early embed reads the config from a worker inside the same window;test_context_management_doc.pypins the five-second deadline and the one-second joined wait.Before the fix:
prove.py's whole-diff revert is inconclusive, because the tests then fail at import on the names this change adds. So each behavior was reverted by hand in a scratch worktree, and each revert turns its tests red: no embed besidesession/new; no pre-build embed after a cold-start one; embedding a rewritten message's claim-time text; the short wait for every build; no short wait for a joined embed; an early embed that embeds nothing; an allowance below the wait ceiling; an in-flight check that is always false.Determinism: the 48 new and changed tests pass 20 runs at
-n0, 10 at-n 2 --dist loadgroupbesidetest_lessons_context_embedding_deadline.pyandtest_embeddings_resource_budget.py(xdist is capped at 2 on this host), and one shuffled run (seed 16078). Clocks are manual and each wait ends on a signal; the timeouts only name a hang. Since main splitembeddings.pyinto owners, the wait-runs-out test puts its clock onembeddings_settings, whoseEmbeddingWork.expiredends the joined wait (the facade binds no clock). On the rebased head that test passes 20 runs at-n0and 10 at-n 2beside the same two files, and the 67 tests oftest_memory_v1_golden.pyand the two new test files pass one shuffled run (seed 29414).Nearby inputs:
private-membersession_createsession/new, joined before the buildtemporary-chat@promptManual verification
Measured locally, without a dashboard token: an isolated home holding a copy of a real 114 MB store (7,979 facts, 364 lessons), the real Qwen3 model on llama.cpp, host load 10 to 14, and session startup simulated as 4.4 s. The figure is the time from startup done to build done, p50 of 5 to 8 runs. It was measured on this PR's first revision, whose waits in these three cases match the final one's.
How often the 1 s wait runs out, from this host's gateway log (
Context timings [build_session_context], all 12 session-start builds, 2026-10-18 07:31–07:59Z). The memory section is the embedding plus about 0.25 s of ranking and rendering (measured on the store copy with the vector ready).p50 is 0.62 s. With the embed started just before the build (a pre-warmed session), 3 to 5 of these 12 would have ranked by keyword at the 1 s default. With it started beside a 4.4 s startup, 1 would: the worker brief.
R3's success measure needs a dashboard token and is still to run: B P1 cold at opus-5.5 · medium, n ≥ 5 per case. Startup on the send path: "send → prompt at KAS" p50 from 7.1 s to ≤ 4.5 s, and to ≤ 1.5 s with
pool_size: 1. Eager spawn finished: from 1.1 s to ≤ 0.4 s. On a fixed query set, the recall selects the same memory items whenever it finishes inside the time box. The table above suggests the eager-spawn target will be missed: the embedding itself takes 0.1 to 1 s on this host, and only the build's own 0.3 s of work runs beside it.Related Issues
Refs #21905
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)