Repository navigation
fix(gateway): stop event-loop starvation under load - #25928
bolichen97 wants to merge 1 commit into
Conversation
418e5b4 to
b63e0df
Compare
5354251 to
986f5bf
Compare
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Each of the four starvation causes is fixed where it starts, and every cron transcript reader I found outside the patch is listed. Notes
[DESIGN-REVIEWED] 4713abb Description read[DESCRIPTION-READ] e058a6f5206188bbed363a5715479aa1ffc39bc8f529237f9edf8826e856084c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)UX-level review of UX-Verdict: PASS cannot evaluate: missing a screenshot of the new "gateway is busy" read-failure state. The PR attached no image and committed none, so the blind read was not performed. Evidence gaps
Notes
[UX-REVIEWED] 4713abb Description read[DESCRIPTION-READ] d811dee15482197b1c4605c07cd158aa63f1b0761e47e134f9538e2239f944f1 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK product-shape change without accepted RFC: the cron reaper now deletes every stateless job's per-run transcripts after 7 days, by default, with no way to turn it off. No RFC on the base records that decision. Notes
What this change shipsInventory (6 items) — 3 justifiedIntent: keep a busy gateway responsive without restarts (#25828). FIX.
Blockersproduct-shape change without accepted RFC: automatic deletion of cron run transcripts — product-shape [FIRST-PRINCIPLES-REVIEWED] 4713abb Description read[DESCRIPTION-READ] 5aef4d1eda1c5732b01a0d538e02f433337cf8e883dd3104e2523a763bf5f72a That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
986f5bf to
52dcf83
Compare
|
52dcf83 to
c50874e
Compare
A busy gateway slowed every request by seconds every few hours. Five fixes: - Run one memory consolidation pass at a time per gateway (consolidation_gate). Passes for different sessions used to run side by side for minutes and hold the GIL. - Reuse a whole-directory session listing for up to 2 seconds (history_listing_memo) instead of statting every transcript on every call. In-process writes are seen at once. - Prune a live stateless cron job's per-run transcripts once they are 7 days old, hourly from the cron reaper (cron_service.run_residue). - Warn from the loop watchdog's own thread when heartbeats run 2s+ late three times in a row, and report it on local /api/health. - Say "the gateway is busy" instead of "could not read this project's AWS profile" when the profile read times out or the connection drops. Closes #25828
c50874e to
4713abb
Compare
|
Opus finding on c50874e (recovery log under-reports the worst lag): fixed in 4713abb. |
|
Superseded by #26057, which carries all six fixes in one PR. |
Pull request was closed
Problem / Motivation
Goal: A gateway with many sessions and cron jobs stays responsive, instead of slowing every request by seconds every few hours until someone restarts it.
On a busy host (about 70 sidebar sessions, a store of about 7,500 transcript files, many stateless cron jobs),
/api/healthclimbs to 2 to 5 seconds or times out, the composer spins, and in-flight turns die. Host load stays low: the slowdown is inside the gateway process. The project chip then shows "Could not read this project's AWS profile", which sends the user to fix AWS credentials that are fine.Why it matters
Everyone on a busy gateway is blocked until a restart, and every restart kills every running turn. The gateway's own alerts ride the same slow event loop, so nobody is told; users find out when the UI freezes.
Not a goal
session.watchdog_rss_max_mbdefault; that product decision is its own PR.What changed (motivation → approach → change)
Four causes slow the event loop, and one message hides it. Each gets the smallest fix:
consolidation_gate.py). A queued pass waits before taking its lease, so it blocks no other processhistory_listing_memo.py). In-process writes are seen at once; another process's write within 2 secondscron_<job>_<run>.jsonlevery fire, and nothing removes itcron_service/run_residue.py)/api/healthreportsloop.laggingProjectAwsProfilesays the gateway is busy on a timeout, a dropped connection or a 502/503/504; a 500 keeps the old textSpecs updated in
history.md,memory-skills-hooks.md,learn-cron-dashboard.mdandslack-gateway.md; every locale underwebsite/src/i18ncarries the new string.Backwards compatibility
Compatible: no input main accepts is refused. Old per-run cron transcripts of a live stateless job are removed after 7 days.
Reader: src/kiro_crew/dashboard/handlers/cron_runs.py:api_cron_open_run -- crew page -- reads only the stable
cron:<id>transcript, which the prune never matches (test_the_jobs_stable_transcript_is_kept)Reader: src/kiro_crew/dashboard/handlers/history_reads.py:api_sessions -- chat -- an old per-run transcript drops out of Older Sessions after 7 days; newer runs still list
Reader: src/kiro_crew/mcp_tools/sessions.py:search_chat_history -- subagent -- same: runs older than 7 days are no longer searchable, newer ones are
Reader: src/kiro_crew/cron_history.py:CronHistoryStore -- cron -- run results and traces live in their own store and are not touched
Reader: src/kiro_crew/cron_service/identity.py:cron_session_key_is_stable -- cron -- a multi-agent
agent_sequencejob's durable three-part key is kept (test_an_agent_sequence_jobs_durable_key_is_kept)Tests
test_history_consolidation_pass_gate.py: two sessions' passes never overlap and both run; a raising pass and a pass cancelled while queued leave the gate usable.test_list_sessions_listing_memo.py: a repeat listing stats nothing; another log shares the memo; an in-process write or new session is seen at once; another process's append after the TTL; callers cannot change the memo.test_cron_run_residue.py: an old run transcript is removed; kept when its job is gone, or when recent, stable, persistent,agent_sequence, still held, pinned or not a cron key; the per-pass cap holds.test_loop_watchdog.py: 3 late beats open one episode; 2 do not; an on-time beat ends it and a new one counts; a poll with no new beat takes no sample.test_api_health.py: the lag field is local-only.ProjectAwsProfile.test.tsx: busy text on our deadline, a dropped connection and a 504.Proof: not provable -- other: the fixes add new modules, so prove.py's whole revert fails the new tests at import and its per-hunk mode crashed; reverting
history_consolidation_pass.pyandhistory_search.pyby hand turnstest_two_sessions_passes_never_overlap_and_both_run,test_a_pass_cancelled_while_queued_leaves_the_gate_to_the_holder,test_a_repeat_listing_stats_no_transcriptandtest_another_log_over_the_same_directory_shares_the_memoredDeterminism: the five new or changed backend test files (130 tests) ran 20 times green, then green once in a shuffled order (seed 4711).
Manual verification
N/A: the causes are covered at their seams by unit tests; reproducing a multi-hour starvation needs a loaded host.
Screenshots / video
Why no screenshot: the chip renders the same
ErrorNoticewith one different sentence, which the component tests assert.Related Issues
Closes #25828
Sibling sweep
Swept: git grep -n "list_sessions_status|def list_sessions" -- src/kiro_crew
Sibling: src/kiro_crew/apps/builtins/auto_improvement/backend/store.py -- not affected -- lists its own small app store, not the session directory
Swept: git grep -n "_consolidate(" -- src/kiro_crew
Sibling: src/kiro_crew/cli_knowledge.py -- not affected -- the CLI runs in its own process, and the per-session lease already excludes it
Swept: git grep -n "read_failed" -- website/src/components
Sibling: website/src/pages/settings/remoteCrewPanelModel.ts -- not affected -- already tells a busy or unreachable remote gateway apart
Pattern harvest
Rule candidate: review-prompt
Pattern: background work started once per session (passes, scans, sweeps) with no process-wide concurrency bound, running inside the process that serves the event loop
Checklist