Skip to content

fix(gateway): stop event-loop starvation under load - #25928

Closed
bolichen97 wants to merge 1 commit into
mainfrom
fix/gateway-starvation
Closed

bolichen97 wants to merge 1 commit into
mainfrom
fix/gateway-starvation

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway with many sessions and cron jobs stays responsive, instead of slowing every request by seconds every few hours until someone restarts it.

On a busy host (about 70 sidebar sessions, a store of about 7,500 transcript files, many stateless cron jobs), /api/health climbs to 2 to 5 seconds or times out, the composer spins, and in-flight turns die. Host load stays low: the slowdown is inside the gateway process. The project chip then shows "Could not read this project's AWS profile", which sends the user to fix AWS credentials that are fine.

Why it matters

Everyone on a busy gateway is blocked until a restart, and every restart kills every running turn. The gateway's own alerts ride the same slow event loop, so nobody is told; users find out when the UI freezes.

Not a goal

  • Moving memory consolidation or embeddings into a separate process.
  • An incremental sidebar protocol (sending diffs instead of listings).
  • Automatic restarts on sustained lag, or new alert channels.
  • Changing the session.watchdog_rss_max_mb default; that product decision is its own PR.

What changed (motivation → approach → change)

Four causes slow the event loop, and one message hides it. Each gets the smallest fix:

Cause Fix
Each idle or ending session starts its own multi-minute consolidation pass; 3 or more run side by side and hold the GIL One pass at a time per gateway (consolidation_gate.py). A queued pass waits before taking its lease, so it blocks no other process
Every full session listing stats every transcript, for each sidebar, palette, folder and restore reader Reuse a whole-directory listing for up to 2 seconds (history_listing_memo.py). In-process writes are seen at once; another process's write within 2 seconds
A stateless cron job writes a new cron_<job>_<run>.jsonl every fire, and nothing removes it The cron reaper removes a live stateless job's per-run transcripts older than 7 days, once an hour (cron_service/run_residue.py)
The loop answers seconds late but never goes silent long enough for the stall dump The loop watchdog warns from its own thread after 3 heartbeats in a row run 2s or more late, and local /api/health reports loop.lagging
A timed-out profile read says the AWS profile is unreadable ProjectAwsProfile says the gateway is busy on a timeout, a dropped connection or a 502/503/504; a 500 keeps the old text

Specs updated in history.md, memory-skills-hooks.md, learn-cron-dashboard.md and slack-gateway.md; every locale under website/src/i18n carries the new string.

Backwards compatibility

Compatible: no input main accepts is refused. Old per-run cron transcripts of a live stateless job are removed after 7 days.

Reader: src/kiro_crew/dashboard/handlers/cron_runs.py:api_cron_open_run -- crew page -- reads only the stable cron:<id> transcript, which the prune never matches (test_the_jobs_stable_transcript_is_kept)
Reader: src/kiro_crew/dashboard/handlers/history_reads.py:api_sessions -- chat -- an old per-run transcript drops out of Older Sessions after 7 days; newer runs still list
Reader: src/kiro_crew/mcp_tools/sessions.py:search_chat_history -- subagent -- same: runs older than 7 days are no longer searchable, newer ones are
Reader: src/kiro_crew/cron_history.py:CronHistoryStore -- cron -- run results and traces live in their own store and are not touched
Reader: src/kiro_crew/cron_service/identity.py:cron_session_key_is_stable -- cron -- a multi-agent agent_sequence job's durable three-part key is kept (test_an_agent_sequence_jobs_durable_key_is_kept)

Tests

  • test_history_consolidation_pass_gate.py: two sessions' passes never overlap and both run; a raising pass and a pass cancelled while queued leave the gate usable.
  • test_list_sessions_listing_memo.py: a repeat listing stats nothing; another log shares the memo; an in-process write or new session is seen at once; another process's append after the TTL; callers cannot change the memo.
  • test_cron_run_residue.py: an old run transcript is removed; kept when its job is gone, or when recent, stable, persistent, agent_sequence, still held, pinned or not a cron key; the per-pass cap holds.
  • test_loop_watchdog.py: 3 late beats open one episode; 2 do not; an on-time beat ends it and a new one counts; a poll with no new beat takes no sample. test_api_health.py: the lag field is local-only.
  • ProjectAwsProfile.test.tsx: busy text on our deadline, a dropped connection and a 504.
  • Existing tests that edit transcripts outside the log expire the listing memo first.

Proof: not provable -- other: the fixes add new modules, so prove.py's whole revert fails the new tests at import and its per-hunk mode crashed; reverting history_consolidation_pass.py and history_search.py by hand turns test_two_sessions_passes_never_overlap_and_both_run, test_a_pass_cancelled_while_queued_leaves_the_gate_to_the_holder, test_a_repeat_listing_stats_no_transcript and test_another_log_over_the_same_directory_shares_the_memo red

Determinism: the five new or changed backend test files (130 tests) ran 20 times green, then green once in a shuffled order (seed 4711).

Manual verification

N/A: the causes are covered at their seams by unit tests; reproducing a multi-hour starvation needs a loaded host.

Screenshots / video

Why no screenshot: the chip renders the same ErrorNotice with one different sentence, which the component tests assert.

Related Issues

Closes #25828

Sibling sweep

Swept: git grep -n "list_sessions_status|def list_sessions" -- src/kiro_crew
Sibling: src/kiro_crew/apps/builtins/auto_improvement/backend/store.py -- not affected -- lists its own small app store, not the session directory
Swept: git grep -n "_consolidate(" -- src/kiro_crew
Sibling: src/kiro_crew/cli_knowledge.py -- not affected -- the CLI runs in its own process, and the per-session lease already excludes it
Swept: git grep -n "read_failed" -- website/src/components
Sibling: website/src/pages/settings/remoteCrewPanelModel.ts -- not affected -- already tells a busy or unreachable remote gateway apart

Pattern harvest

Rule candidate: review-prompt
Pattern: background work started once per session (passes, scans, sweeps) with no process-wide concurrency bound, running inside the process that serves the event loop

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 29, 2026 20:06
@bolichen97
bolichen97 requested a review from CrysisDeu October 29, 2026 20:06
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 29, 2026
@bolichen97
bolichen97 force-pushed the fix/gateway-starvation branch from 418e5b4 to b63e0df Compare October 29, 2026 20:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 29, 2026 20:09
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in website/src/i18n/locales/*.json since this head went green (e598df5 -> c38fcb1); CI re-running. Same push fixes the comment-history gate (two docstrings said "used to").

@bolichen97
bolichen97 force-pushed the fix/gateway-starvation branch 2 times, most recently from 5354251 to 986f5bf Compare October 29, 2026 20:50
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4713abb441bd412c8833a64fc1247eac332ba0d9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Each of the four starvation causes is fixed where it starts, and every cron transcript reader I found outside the patch is listed.

Notes

  • The process-wide gate (await waiting.acquire()) is held for a whole multi-minute pass. If one pass hangs, for example on an LLM call with no deadline, every later consolidation in the gateway stops with no warning. Check that the pass has an overall deadline, or log how long a pass waited in the queue.
  • The 2s listing memo means another process's append can show up to 2s late. The spec documents this, and reverting it is just removing the memo, so I'm leaving it as a note.

[DESIGN-REVIEWED] 4713abb

Description read

[DESCRIPTION-READ] e058a6f5206188bbed363a5715479aa1ffc39bc8f529237f9edf8826e856084c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

UX-level review of 4713abb441bd412c8833a64fc1247eac332ba0d9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

cannot evaluate: missing a screenshot of the new "gateway is busy" read-failure state. The PR attached no image and committed none, so the blind read was not performed.

Evidence gaps

  • New read_unreachable notice in the project chip (ProjectAwsProfileStatus, inline ErrorNotice with messageClassName="truncate"): no screenshot shows it. Close the gap with a still of the shelf chip after a timed-out or 503 profile read, in both the normal and the shelfCompact widths.
  • The same notice as the section title (ProjectAwsProfileSection, title={readFailureText(error)}): no screenshot shows it. Close the gap with a still of the AWS profile section in the same failure state.

Notes

  • In the chip, "The gateway is busy and did not answer. Try again in a moment." is cut off by truncate, so a narrow chip likely shows only "The gateway is busy and did n…". Put the meaning in the first words, e.g. "Gateway busy — retry shortly".
  • "Try again in a moment" asks the user to retry, but neither notice offers a retry button. Add a retry action that refetches the profile, or say the chip will retry by itself if it does.

[UX-REVIEWED] 4713abb

Description read

[DESCRIPTION-READ] d811dee15482197b1c4605c07cd158aa63f1b0761e47e134f9538e2239f944f1

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of 4713abb441bd412c8833a64fc1247eac332ba0d9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

product-shape change without accepted RFC: the cron reaper now deletes every stateless job's per-run transcripts after 7 days, by default, with no way to turn it off. No RFC on the base records that decision.

Notes

  • Item 4 — duplicate of src/kiro_crew/dashboard/server_runtime/heartbeat.py:139: that line already logs a WARNING on every beat with lag > 1.0. That breaks the premise that "the loop answers seconds late but never goes silent long enough for the stall dump", because the slowness is already logged. Subtraction: delete _note_sustained_lag and its state, or keep only the episode counter.
  • Item 5 — zero consumers: grep lagging|sustained_lag_episodes over src/ and website/src outside tests finds 0 readers of /api/health loop. Subtraction: delete the payload["loop"] block in handlers/core.py.

What this change ships

Inventory (6 items) — 3 justified

Intent: keep a busy gateway responsive without restarts (#25828). FIX.

  1. Memory consolidation passes run one at a time per gateway — justified
  2. Whole-directory session listings are reused for up to 2s — justified
  3. Old per-run cron transcripts disappear from Older Sessions and search after 7 days — product-shape, no recorded decision
  4. A new "event loop lagging" warning after 3 late heartbeats — duplicate of heartbeat.py:139 lag warning
  5. /api/health reports loop.lagging locally — zero consumers, rides along
  6. The AWS profile chip says "gateway busy" on a timeout or 502/503/504 — justified

Blockers

product-shape change without accepted RFC: automatic deletion of cron run transcripts — product-shape
Trigger: RUN_TRANSCRIPT_TTL_SECS = 7 * 86400 runs hourly with no setting, and the description says "runs older than 7 days are no longer searchable". It also skips session.archive_retention_days, whose -1 means "the user manages archive deletion manually". rfc-status.txt has no RFC on cron transcript retention.
Outcome: on every install, users lose transcripts they could open and search before, and nobody recorded deciding that.
Clears when: an RFC covering automatic per-run cron transcript retention is merged non-draft on the base, followed by a push or rebase; or a maintainer posts /ai-review override first-principles 4713abb441bd412c8833a64fc1247eac332ba0d9 product-shape: <decision>.

[FIRST-PRINCIPLES-REVIEWED] 4713abb

Description read

[DESCRIPTION-READ] 5aef4d1eda1c5732b01a0d538e02f433337cf8e883dd3104e2523a763bf5f72a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4713abb441bd412c8833a64fc1247eac332ba0d9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4713abb

Verdict parsed from the review's SHA-scoped output markers for commit 4713abb441bd412c8833a64fc1247eac332ba0d9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4713abb441bd412c8833a64fc1247eac332ba0d9 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4713abb441bd412c8833a64fc1247eac332ba0d9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4713abb

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4713abb441bd412c8833a64fc1247eac332ba0d9 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Frontend Tests (1) red is main's, not this PR's: src/test/kiroUsageRefetchInterval.test.ts > maps an explicit unavailable answer to a terminal sentinel fails the same way on a clean origin/main (df72ee4). Tracked in #25941, fix in #25944; this PR will rebase onto it once it lands.

@bolichen97
bolichen97 force-pushed the fix/gateway-starvation branch from 986f5bf to 52dcf83 Compare October 30, 2026 00:12
@bolichen97

bolichen97 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=7eeda5878368 Deleted sequence jobs lose their durable session records

fixed in 52dcf83: prune_stale_run_transcripts keeps a transcript whose job record is gone, so a deleted agent_sequence job's durable cron:<job>:<agent> transcript is never pruned.
Only a live job whose key is not stable has its per-run transcripts removed; test_a_deleted_jobs_old_three_part_transcript_is_kept pins it.

@bolichen97
bolichen97 force-pushed the fix/gateway-starvation branch from 52dcf83 to c50874e Compare October 30, 2026 01:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 30, 2026
A busy gateway slowed every request by seconds every few hours. Five fixes:

- Run one memory consolidation pass at a time per gateway
  (consolidation_gate). Passes for different sessions used to run side by
  side for minutes and hold the GIL.
- Reuse a whole-directory session listing for up to 2 seconds
  (history_listing_memo) instead of statting every transcript on every call.
  In-process writes are seen at once.
- Prune a live stateless cron job's per-run transcripts once they are 7 days
  old, hourly from the cron reaper (cron_service.run_residue).
- Warn from the loop watchdog's own thread when heartbeats run 2s+ late three
  times in a row, and report it on local /api/health.
- Say "the gateway is busy" instead of "could not read this project's AWS
  profile" when the profile read times out or the connection drops.

Closes #25828
@bolichen97
bolichen97 force-pushed the fix/gateway-starvation branch from c50874e to 4713abb Compare October 30, 2026 01:19
@bolichen97

Copy link
Copy Markdown
Contributor Author

Opus finding on c50874e (recovery log under-reports the worst lag): fixed in 4713abb. _note_sustained_lag keeps the maximum across the whole late streak and resets it where the streak resets; test_recovery_names_the_worst_beat_of_the_opening_streak pins it (fails without the fix). Same push rebases onto current main.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 30, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Superseded by #26057, which carries all six fixes in one PR.

@bolichen97 bolichen97 closed this Oct 30, 2026
auto-merge was automatically disabled October 30, 2026 02:30

Pull request was closed

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 30, 2026
@bolichen97
bolichen97 deleted the fix/gateway-starvation branch November 20, 2026 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway event loop starves under load: parallel consolidation, full-store listing scans, unbounded cron run transcripts

1 participant