Skip to content

perf(chat): start a new chat's session at once, with no debounce - #37575

Merged
bolichen97 merged 1 commit into
mainfrom
crew/kiro-crew-rust/new-chat-prewarm-now
Nov 15, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/kiro-crew-rust/new-chat-prewarm-now

Conversation

@buluoray

@buluoray buluoray commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a message sent soon after opening a new chat gets its first reply sooner.

When the owner opens a new chat, the gateway starts that chat's session ahead of the first message, but only after a 1.5 s debounce. A cold start takes about 3 s on top of that (engine start, session/new, the wait for Kiro Crew's own MCP servers). So a message sent a couple of seconds after "New chat" waits for most of that start.

Why it matters

The common way to begin is to open a new chat and type a short question right away. That first reply is the one where the wait is most visible, and the 1.5 s debounce is spent while the person is already waiting.

Not a goal

What changed (motivation → approach → change)

The debounce exists to absorb rapid consecutive signals (a user re-picking the project or the agent) so that only the settled state spawns a session. A brand-new chat created by its owner has nothing to settle yet, and the person is about to type. The slot-create handler (dashboard/chat_api/slot_create.py) now passes debounce=0.0 to schedule_eager_spawn for the owner's own create; schedule_eager_spawn and _eager_spawn (dashboard/chat_turn/eager_spawn.py) take an optional debounce that defaults to _EAGER_SPAWN_DEBOUNCE_SECS.

The trade-off: a create whose client sets the project right after it keeps the 1.5 s debounce. Every dashboard create that does so (store/chat/lifecycle.ts, pages/chat/page/composerSend.ts, pages/chat/startFromInitialPrompt.ts) sends no_session_folder: true, and the handler reads that flag. Without it, the project change would find the immediate start's reservation in session_keys(), arm a deferred reset that main never arms, and the replacement pre-warm would stand down on the armed flag. The same holds while dashboard.new_chat_crew_mode is on: the client then switches the new chat to the conductor right after the create (lifecycle.ts startNewChatInCrewMode), which would cancel a start already under way, so those creates keep the debounce too. So those chats behave exactly as on main, and only a plain "New chat" outside Crew Mode starts at once. The focus frame that follows a create does not cancel the start: ws._handle_slot_focused already returns before schedule_eager_spawn for a slot with nothing to resume.

Measured on a Dev Fleet pod, sending 2 s after "New chat session" (server-side turn_timing, five rounds per arm, interleaved, host load under 15 for every counted round):

arm median send to first output median session claim
main 4489 ms 2295 ms
main again (A/A) 4318 ms 2261 ms
no debounce 2965 ms 806 ms

Every no-debounce run (2834 to 3386 ms) was faster than every main run (4066 to 5070 ms).

Backwards compatibility

Compatible: the new keyword is optional and defaults to the old value, so every existing caller behaves as before.

Removes nothing: no key, field or behaviour is taken away; only the owner's create starts sooner.

Tests

  • test/test_new_chat_prewarm_debounce.py::test_only_the_owners_plain_create_skips_the_debounce: the owner's create passes debounce=0.0; a create from anyone else, or one that a binding change follows (sent with no_session_folder, or while dashboard.new_chat_crew_mode is on), passes nothing and keeps the default. The project side of that case, no reset armed while only a debouncing start is scheduled, is pinned on main by test_chat_slot_project.py::test_change_with_only_scheduled_eager_spawn_does_not_arm_reset.
  • test_schedule_hands_the_debounce_to_the_start: with the module debounce set to an hour, debounce=0.0 still reaches get_or_create.
  • test/test_chat_runner_base_surface.py: the recorded signatures of schedule_eager_spawn and _eager_spawn now include debounce.

Proof: PROVEN test/test_chat_runner_base_surface.py::test_a_moved_name_keeps_its_base_shape_and_owner[eager_spawn:schedule_eager_spawn], test/test_chat_runner_base_surface.py::test_a_moved_name_keeps_its_base_shape_and_owner[eager_spawn:_eager_spawn], test/test_new_chat_prewarm_debounce.py::test_only_the_owners_plain_create_skips_the_debounce[True-body0-False-0.0] (head 96f4f6e)

Also run locally and green: test_chat_slot*.py, test_eager_spawn*.py, test_slot_create*.py, test_chat_handlers*.py, test_new_chat_prewarm_debounce.py, test_chat_runner_base_surface.py ; every test file that posts to /api/chat/slots (88 files, 3435 tests). black, flake8 and isort are clean on the changed files.

Manual verification

On pods built from main and from this branch (Playwright, real server and model, no fixtures): New chat session, wait 2 s, send "说个笑话". The timer below starts at the send press and stops at the first word.

Screenshots / video

Left: main, first word 4.75 s after send. Right: this PR, 2.97 s.

new chat prewarm before and after

Related Issues

N/A — follow-up to the first-message wait investigated in #37554. The draft warm-pool RFC #23822 lists a shorter debounce among the alternatives it considered.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 15, 2026 09:05
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the timing change has a measured harm and a single caller. The one note is a parameter more general than its only use.

Notes

  • Item 2 — one consumer, generalized: debounce: float | None = None is used at exactly 1 call site (slot_create.py:1309, grepped debounce= across src/kiro_crew). That caller only ever passes 0.0 or None, so no length other than zero or the default is ever chosen. Subtraction: shrink it to a bool (for example immediate: bool = False) that skips the asyncio.sleep, and drop the float | None override sentence from the docstring.

What this change ships

Inventory (4 items) — 3 justified

Intent: when someone opens a new chat and types right away, the first reply arrives sooner. This is an ADDITION (perf tuning).

  1. A plain owner "New chat" outside Crew Mode now starts its session at once, not 1.5 s later — justified
  2. schedule_eager_spawn and _eager_spawn take a new optional debounce keyword — one consumer, generalized
  3. A create that sends no_session_folder, runs in Crew Mode, or has no config keeps the 1.5 s debounce — justified
  4. A non-owner create keeps the 1.5 s debounce — justified

[FIRST-PRINCIPLES-REVIEWED] 96f4f6e

Description read

[DESCRIPTION-READ] 4f14c8f15ab2b57f952723f1da450901fd6fcc31a2db973bca932de04f9d2826

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change adds one optional keyword that defaults to the old value. Its fallback cases (a binding change after the create, Crew Mode) keep the debounce, and if one is misjudged the cost is a cancelled pre-warm that a revert undoes.

Notes

  • The binding_follows check depends on clients sending no_session_folder whenever they set the project after creating the chat. The handler can't verify that; it relies on the client keeping to it. If a future owner-side create path sets the project without sending the flag, the server shows nothing: a reset gets armed that main never arms, and the pre-warm quietly stands down. A server-side pin on that pairing, or a comment at createChatSlot, would keep the two sides from drifting apart.

[DESIGN-REVIEWED] 96f4f6e

Description read

[DESCRIPTION-READ] 7cc23cb45a348a4c9a090ee4a362665ef5c302d18062950de651011b8dd931c8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 96f4f6e

Verdict parsed from the review's SHA-scoped output markers for commit 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 96f4f6e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 96f4f6ee4371e6a5a7a2f101e1097923b0d995d1 false-positive: <reason, 20 to 500 characters>

@buluoray
buluoray force-pushed the crew/kiro-crew-rust/new-chat-prewarm-now branch from 5f9488c to 52242f5 Compare November 15, 2026 09:14
@buluoray

Copy link
Copy Markdown
Contributor Author

On the two review notes:

  • Design note (does the new-chat flow send a project or agent set as part of the create?). In the pod runs behind the table, the "no debounce" arm's session claim was 636 to 1018 ms when the message went out 2 s after "New chat session". A cancelled and restarted start would have cost the full start again (about 3 s), so the dashboard's own create flow did not discard a start there. A project or agent set after the create still does, which the description already names as the trade-off.
  • First Principles note (debounce: float | None with one caller passing 0.0). Agreed that only 0.0 is used today. I kept the float because it is the value _eager_spawn already sleeps on, so the keyword reads as "the wait" instead of a second flag with its own meaning. Happy to switch it to a bool if a maintainer prefers.

The Design Review run on 52242f5046e2 failed on a GitHub API rate limit before it produced a verdict; I re-ran it.

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Small, well-scoped perf change: the owner's plain New-chat create passes debounce=0.0 to schedule_eager_spawn; every other caller keeps the 1.5 s default. Verified the client-side claim that every dashboard create followed by a project set sends no_session_folder (lifecycle.ts via project/sessionFolder:false, composerSend pendingProject, startFromInitialPrompt, memoryModeSwitch), and that Crew Mode's follow-up conductor switch is gated by the same new_chat_crew_mode config the server checks. The focus frame does not cancel the start for non-resumable slots (ws._handle_slot_focused). A send, agent switch or project pick during the in-flight start goes through the same path main already takes for actions 1.5–4.5 s after create, only earlier. Existing pins (test_start_priority source pin, assert_called_once_with on the agent/project handlers) are unaffected. The new tests fail without the change. Backend/E2E/Integration CI results were still pending at review time.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 96f4f6ee43 · 152 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@buluoray
buluoray force-pushed the crew/kiro-crew-rust/new-chat-prewarm-now branch from 52242f5 to 9085c0b Compare November 15, 2026 10:22
@buluoray

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_api/ and src/kiro_crew/dashboard/chat_turn/ (same-dir overlap, 17 commits) since this head went green (2e56b3b -> c970398); CI re-running.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the narrow change and the interleaved timing table. One gap needs fixing before this can merge.

  1. src/kiro_crew/dashboard/chat_turn/eager_spawn.py:58-60 and session_relaunch.py:916-919: a project set right after the owner's create is not tested, and that is the one path where removing the debounce changes behaviour.
    • Trigger: the dashboard sends POST /api/chat/slots, then POST /api/chat/slots/<k>/project right after it (website/src/store/chat/lifecycle.ts:219, website/src/pages/chat/page/composerSend.ts:749, website/src/pages/chat/startFromInitialPrompt.ts:72). With debounce=0.0, the create's start can already hold a get_or_create reservation when the project request arrives.
    • Outcome: arm_reset counts session_key in state.sessions.session_keys(), which includes that reservation, so the project handler arms a deferred reset that main never arms. The replacement eager task runs _consume_pending_reset. Nothing is registered yet, so the flag stays armed, and the pre-warm returns at eager_spawn.py:652. A later consume can then tear down the correctly rooted session after the first turn. On this path the cost is more than "one discarded start", and project-scoped new chats can end up slower than on main. On main, test_change_with_only_scheduled_eager_spawn_does_not_arm_reset pins the opposite. The reviewer comment on this head raises the same point and has no reply yet.
    • Smallest fix: add a test that creates as the owner with debounce=0.0, lets the start reach a stubbed get_or_create that holds a reservation in session_keys(), then posts a project change. Assert that the replacement start reaches get_or_create rooted at the new project and that no reset flag stays armed. If that cannot hold, there are two options. You can stop the slot's own cancelled speculative start from arming the reset (its token is in state.speculative_starts). Or you can keep the debounce when a project follow-up is expected. Either way, please correct the comment and the trade-off paragraph in the body to match what the code does.

Optional:

  • debounce: float | None has one caller, and it passes only 0.0 or None. An immediate: bool would say the same thing. Your call.
  • test_a_zero_debounce_starts_at_once overlaps test_schedule_hands_the_debounce_to_the_start and could be dropped.
  • The draft warm-pool RFC (#23822) lists a shorter debounce as an alternative it considered. A mention under Related Issues would help readers.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 15, 2026
@buluoray
buluoray force-pushed the crew/kiro-crew-rust/new-chat-prewarm-now branch from 9085c0b to 963ee17 Compare November 15, 2026 14:17
@buluoray

Copy link
Copy Markdown
Contributor Author

Re the review on 9085c0b: agreed, the project-follow-up path did change behaviour. Fixed on 963ee17 by taking the second option: a create sent with no_session_folder (every dashboard create that sets the project right after it: lifecycle.ts, composerSend.ts, startFromInitialPrompt.ts) keeps the 1.5 s debounce, so the project change never lands on a start holding the key and arms no reset, as on main (test_chat_slot_project.py::test_change_with_only_scheduled_eager_spawn_does_not_arm_reset). Only a plain New chat starts at once. New case in test_only_the_owners_plain_create_skips_the_debounce fails without the guard. Comment and trade-off paragraph corrected. Optional items: dropped test_a_zero_debounce_starts_at_once, added #23822 under Related Issues, kept the float parameter.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 15, 2026
@buluoray
buluoray force-pushed the crew/kiro-crew-rust/new-chat-prewarm-now branch from 963ee17 to 7a664b9 Compare November 15, 2026 14:47
@buluoray

Copy link
Copy Markdown
Contributor Author

7a664b9, for 963ee17: Opus's Crew Mode finding is fixed. While dashboard.new_chat_crew_mode is on, or the config can't be read, the create keeps the 1.5 s debounce, because the client switches the agent to the conductor right after the create. New case (True, {}, True, None) fails without the guard. Rebased on bdff21a.

When the owner opens a new chat, the gateway starts its session ahead of
the first message, but only after a 1.5 s debounce. A message sent a
couple of seconds after "New chat" then waits for the rest of that start.
On a pod, sending 2 s after "New chat" took a median 4.3-4.5 s to the
first output (two runs of five) with the debounce and 3.0 s without it.

The slot-create handler now passes debounce=0.0 for the owner's own
create. Every other signal (agent or project switch, focus, reload, app
callers) keeps the 1.5 s debounce.

A create that a binding change follows keeps the debounce: one sent with
no_session_folder (the client sets the project right after it), and any
create while dashboard.new_chat_crew_mode is on (the client switches the
agent to the conductor right after it). Without it, that change would
cancel a start already reserving the key; for the project it would also
find the reservation in session_keys(), arm a deferred reset main never
arms, and the replacement pre-warm would stand down on the armed flag.
@buluoray
buluoray force-pushed the crew/kiro-crew-rust/new-chat-prewarm-now branch from 7a664b9 to 96f4f6e Compare November 15, 2026 15:33
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix to the dashboard's own create flows and the new parametrized cases. One caller of the same class is still not covered.

  1. chat_api/slot_create.py:1300-1310: the Design Tweak panel creates its slot and sets the project right after, with no no_session_folder.
    • Trigger: ensureSlot calls createChatSlot(want, title) and then await setChatSlotProject(key, path) (website/src/apps/design-tweak/useDesignTweakPageState.ts:316-326). The create body is { name, agent: '', title } (website/src/apps/design-tweak/api.ts:239). It is a same-origin dashboard fetch, so is_owner_dashboard_request is true and binding_follows is False.
    • Outcome: the create passes debounce=0.0, so the start can already hold a get_or_create reservation when the project request arrives. session_relaunch.py:916-920 then arms a deferred reset that main never arms here. The replacement pre-warm stands down, and the seed turn can pay a teardown and a second cold start. This is the sequence the body gives as the reason to keep the debounce. It also shows that two sentences in the body are not accurate: "every dashboard create that does so sends no_session_folder" and "those chats behave exactly as on main". I found this by reading the code and did not reproduce the race.
    • Smallest fix: have Design Tweak's createChatSlot send no_session_folder: true. It always sends a name, so the flag changes nothing else (slot_create.py:964-970). Add a test for it, and correct the body sentence. Note that Design Tweak is now adopted from its own repository (KiroCrewCaptain/kirocrew-app-design-tweak, pinned in #37450), so an edit to the wheel copy alone does not reach adopted installs, and any other installed app could send the same create. The sturdier fix is therefore core-side: make the immediate start opt-in: only the plain New chat path sends a positive flag, so any caller that does not know about it keeps 1.5 s.

Earlier review on 9085c0b:

  • Project set after an owner create: fixed for lifecycle.ts, composerSend.ts and startFromInitialPrompt.ts. It still stands for Design Tweak (item 1 above).
  • Optional items: the duplicate test is dropped, #23822 is listed, and the float parameter was your call. All resolved.

Optional:

  • The gate also reaches an owner adopt of an existing slot and in-bundle app pages that send no no_session_folder, but ## Not a goal says app callers keep 1.5 s. The opt-in flag above would also close this.
  • No test pins the cfg is None guard. A case where KiroCrewConfig.load raises, asserting the create still succeeds with debounce None, would catch a reorder that dereferences cfg.dashboard first.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Only the owner's own plain new chat drops the 1.5 s eager-spawn debounce (debounce=0.0); every other caller still passes None and keeps _EAGER_SPAWN_DEBOUNCE_SECS.

The earlier change request (a project set right after the create landing on a start that already holds the key) looks addressed on this head. A create keeps the debounce when no_session_folder is true, when there is no config, or when new chats open in Crew Mode. I checked on main that every create followed by a project set sends no_session_folder:

  • store/chat/lifecycle.ts: skipSessionFolder = !!project || ..., passed to createChatSlot, which sends no_session_folder: true (api/client/chat.ts:297), before chatSlotProject.
  • pages/chat/page/composerSend.ts:581: a pending project passes sessionFolder: false.
  • pages/chat/startFromInitialPrompt.ts: the agent-then-project path runs only when project is set (line 30), so the create carries the project and sends the flag.

No new refusal: nothing is refused that main allows; only a wait gets shorter. No regex. The five red checks are cancelled duplicates, each with a newer success on this head.

I read the whole diff on 96f4f6e, including both tests.

Not checked: I did not measure the start time myself or run the tests locally.

@bolichen97
bolichen97 enabled auto-merge (squash) November 15, 2026 17:12
@bolichen97
bolichen97 merged commit 4a89680 into main Nov 15, 2026
105 of 110 checks passed
@bolichen97
bolichen97 deleted the crew/kiro-crew-rust/new-chat-prewarm-now branch November 15, 2026 17:23
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants