Skip to content

fix(session): bound the background runtime's RSS growth, not its MCP servers - #32030

Open
rinbn wants to merge 1 commit into
kirodotdev:mainfrom
rinbn:fix/bg-rss-growth-budget
Open

rinbn wants to merge 1 commit into
kirodotdev:mainfrom
rinbn:fix/bg-rss-growth-budget

Conversation

@rinbn

@rinbn rinbn commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Keep the background runtime warm across turns on hosts whose MCP servers alone put its process tree above 1536 MiB, so session titles and status cards stop timing out.

On the claude backend the background runtime launches every configured MCP server. With about 15-20 servers its process tree is around 3.7 GB the moment it starts. The RSS recycle compared that whole tree against a 1536 MiB ceiling, so it fired after every background turn. Each turn then started a cold conversation: status cards written from scratch ran past their 45 s timeout, and the title requested when a chat's first message is sent timed out behind them. A new chat stayed "Untitled session" until its whole first turn ended. Numbers and the full loop are in #31900.

Why it matters

Anyone on the claude backend with a typical MCP setup is hit on every gateway start since the 1536 MiB cap began to override session.watchdog_rss_max_mb. Titles arrive late or never, dynamic status cards mostly time out (77 of 106 in one measurement), and every background turn pays a cold prompt of about 50k uncached tokens.

Not a goal

What changed (motivation → approach → change)

The 1536 MiB bound exists to catch a background transcript of oversized records growing the runtime. That is growth, so the check in BackgroundSessionRuntime._rss_recycle_reason (src/kiro_crew/session_background.py) measures growth.

Before After
What is compared with the limit the whole tree RSS tree RSS minus the provider's baseline
Readings after a provider's first turn, up to the first one at least 30 s later recycle if at or above the limit set and raise the baseline, and the last of them fixes it; none recycles. MCP servers keep starting for seconds after the provider is up, so neither spawn nor a short first turn sees the full tree
After a recycle adopts a replacement n/a the baseline is dropped, so the replacement re-baselines even if it reports the same pid
A 0 reading (a host that cannot measure the tree) never recycles sets no baseline
The limit itself watchdog_rss_max_mb when lower than 1536, else 1536 unchanged

A tree that rests at 3.7 GB is kept turn after turn, and still recycles once it grows by the limit. The recycle log line names the growth and the baseline. The help text for session.watchdog_rss_max_mb (src/kiro_crew/config/sections_session.py, regenerated into config-baseline.json), src/kiro_crew/docs/configuration.md, docs/system-specs/modules/config.md and the background-recycle section of docs/system-specs/modules/session.md describe the growth limit.

Backwards compatibility

Compatible: no config key, API or record changes; the same knob and fallback set the limit, and a runtime whose tree starts below the limit recycles at the same point as before, plus its first reading.

Removes nothing: the RSS criterion still exists and still bounds the runtime; it measures growth instead of the absolute size.

Tests

test/test_background_rss_recycle.py pins the growth semantics: growth at and over the limit recycles and names the baseline, growth under it keeps the session, a 3.7 GB starting tree survives several turns and still recycles on growth, servers still starting after a short first turn raise the baseline instead of counting as growth, a start-up reading followed by a quiet gap fixes the baseline instead of recycling, the window counts from the first reading rather than the latest raise, the replacement re-baselines on the same pid, a new pid is a new baseline, a 0 reading (a host that cannot measure the tree) sets no baseline, and the 0 and shipped-default fallbacks apply to growth. The permit-handoff and unreadable-tree cases are unchanged apart from recording a baseline first.

Proof: PROVEN test/test_background_rss_recycle.py::TestTreeRssGrowthRecyclesTheBackgroundSession::test_growth_at_the_limit_recycles_and_names_the_rss_reason, test/test_background_rss_recycle.py::TestTreeRssGrowthRecyclesTheBackgroundSession::test_growth_far_over_the_limit_recycles, test/test_background_rss_recycle.py::TestTreeRssGrowthRecyclesTheBackgroundSession::test_growth_below_the_limit_keeps_the_session, test/test_background_rss_recycle.py::TestTheBaseline::test_a_tree_that_starts_above_the_limit_survives_its_turns, test/test_background_rss_recycle.py::TestTheBaseline::test_a_tree_that_starts_above_the_limit_still_recycles_on_growth (head 6aa5939)

Determinism: 20/20 runs at -n0; 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 with test/test_artifact_companion.py and test/test_background_turn_accounting.py as co-tenants; a shuffled order (seed 29379) passes.

Manual verification

On a host with this MCP footprint (60 processes, about 3.7 GB tree at rest), raising the fallback to 8192 on 0.12.0.9 and restarting the gateway reproduced the intended behavior: one background conversation served 7+ turns, the first from-scratch card took 57 s and the next ones 16-18 s, and titles took 2 s. This branch was not run live; the growth check is covered by the unit tests above.

Related Issues

Closes #31900

Sibling sweep

Swept: git grep -nE "tree_rss_mb|get_session_rss_mb(|rss_max_mb(" -- src
Sibling: src/kiro_crew/session_cleanup.py -- not affected -- the chat-session sweep skips persistent keys and its 4096 MiB default is sized above a healthy MCP tree

Pattern harvest

Not generalizable: the only absolute tree-RSS ceiling that can sit below the tree's resting size is this one; the chat-session ceiling is user-set and sized above it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@rinbn
rinbn requested review from a team and KiroCrewCaptain as code owners November 7, 2026 03:17
@rinbn
rinbn requested a review from tim-finnigan November 7, 2026 03:17
@rinbn

rinbn commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Local review before opening, on the extracted CI contracts (opus on claude-opus-5.5; gpt at reduced fidelity on gpt-5.6-sol, because gpt-6.x is not served on this host), plus one focused verifier.

opus, blocking: docs/system-specs/modules/config.md:3363 still states the absolute 1536 MiB background ceiling.

fixed: that copy says the growth limit, like the other four.

opus, note: get_session_rss_mb returns 0 when it cannot measure, and a 0 first reading became the baseline.

fixed: a 0 reading is no reading and sets no baseline; test_a_zero_reading_is_not_a_baseline pins it.

gpt, High: "the first turn bypasses the RSS-growth limit" — record the baseline after startup but before the first prompt.

rebutted: a spawn-time baseline counts MCP start-up as growth, which is the loop this PR fixes. Measured on the reporting host: the runtime spawned at +0 s, its first prompt was queued at +3 s with the tree at about 1.0 GB, and MCP servers kept starting until +15 s, when the tree reached 3.8 GB. A baseline before the first prompt would read the 1.0 GB, so the next turn would see about 2.8 GB of "growth" and recycle every replacement after its first turn. The exemption this leaves is one background one-liner's growth plus readings inside the settle window, and the 40-prompt blind backstop still bounds the runtime's lifetime.
mechanism: baseline taken after the first turn, not at spawn

verifier, Medium: a first turn shorter than MCP start-up takes the baseline mid-start-up, so the servers that come up afterwards count as growth on turn 2.

fixed: the baseline is provisional until the first reading at least BACKGROUND_RSS_SETTLE_SECS (30 s) after the first one; every reading until then raises it, that reading fixes it, and none of them recycles. test_servers_still_starting_after_a_short_first_turn_raise_the_baseline pins the in-window raise.
self-added: yes
mechanism: provisional baseline fixed by the first reading past the settle window

verifier, residual: a reading taken during MCP start-up, followed by the next one 30 s or more later (burst-then-gap traffic), still counted the start-up as growth.

fixed: the first reading past the window fixes the baseline instead of being compared with it; test_a_start_up_reading_followed_by_a_gap_fixes_the_baseline pins it, and test_the_window_counts_from_the_first_reading pins that a raise does not restart the window.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 9, 2026
@rinbn
rinbn force-pushed the fix/bg-rss-growth-budget branch from ebfb9d5 to 6aa5939 Compare November 11, 2026 16:58
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 11, 2026
@rinbn

rinbn commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Five of my fork PRs are blocked on "Approve and run" for their workflows. KiroCrewCaptain has not approved fork CI on these heads, so each needs a manual approval:

PR head CI waiting since (UTC) note
#30797 8ed8287 2026-11-09 17:54 approved by @bolichen97 at ee1dc5e, then rebased onto main with no conflicts. The three red checks on the old head (the security-gate line-budget test, the codex sandbox adapter contract tests, and Coverage Gate failing closed on them) touch no file in this PR
#32030 6aa5939 2026-11-11 16:58 opened 2026-11-07, CI never approved; rebased today to clear a docs-only conflict
#30786 8cd6bf7 2026-11-11 17:10 first pushed 2026-11-05, CI never approved; rebased today to clear a conflict in groupDisplayItems.ts
#26459 d3f5c1c 2026-11-05 18:58 no conflicts
#31847 a3f257d 2026-11-09 17:54 rebased on 2026-11-09, conflicts with main again since; I will rebase it once more

On #32030, #30786 and #26459 the PR Readiness workflow also failed in Publish readiness signal ("Failed to publish the readiness status for ... re-run this workflow to re-evaluate and republish"). That needs a rerun from someone with write access; I cannot rerun it from the fork.

Main takes about 350 commits a day, so a rebased head picks up new conflicts within a day or two while it waits. An approval soon after a push would save those rounds.

@bolichen97 @iamwhatever @buluoray @CrysisDeu could one of you approve these workflows? Thanks.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own.

The background runtime's RSS recycle now measures growth past a per-pid baseline instead of the absolute tree size. The baseline is set after the provider's first turn, can only rise during a 30 s settle window, and is then fixed. A 0 reading sets none, and the baseline is dropped when a recycle adopts its replacement. State is written only in _rss_recycle_reason and at adoption; a pid change also forces a fresh baseline. The earlier context, compaction and 40-prompt criteria are unchanged, so the runtime still has a lifetime bound. The new tests would fail on main, and the docs and config help are updated consistently. The looser bound for small trees is intended and stated in the PR. No blocking issues found.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 6aa59395ca · 384 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
…servers

The background runtime was recycled when its whole process tree reached
1536 MiB. On the claude backend it launches every configured MCP server, so
a healthy tree can start above that and was recycled after every background
turn: the conversation never warmed, status cards written from scratch ran
past their 45 s timeout, and the on-send title attempt timed out behind them.

The limit now applies to growth past the provider's first reading, which
keeps the bound on transcript growth the value was chosen for. A recycle
that adopts a replacement drops the reading, so the replacement re-baselines.

Closes kirodotdev#31900
@iamwhatever
iamwhatever force-pushed the fix/bg-rss-growth-budget branch from 6aa5939 to f8af7e0 Compare November 18, 2026 01:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision f8af7e0cda68a57e6a1ffc4f5c0686e01084f291 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of f8af7e0cda68a57e6a1ffc4f5c0686e01084f291 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The limit now measures growth from a baseline taken after the settle window, which still catches the oversized-transcript leak and no longer fires on every turn because of a large MCP footprint.

Notes

  • rss_baselines is cleared only when a recycle adopts a replacement. A reset or remove of the background key leaves its entry behind, and only the pid check keeps a reused pid from inheriting a stale baseline. Popping the entry on those teardown paths too would make the provider's lifetime the one thing that owns the baseline.

[DESIGN-REVIEWED] f8af7e0

Description read

[DESCRIPTION-READ] 89afeb9fa198b2931cb140b0580a672f39e0ef232d253ebfec703479d4ef28c4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🔴 BLOCK

Premise-level review of f8af7e0cda68a57e6a1ffc4f5c0686e01084f291 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

The new background RSS baseline duplicates the per-pid start-up footprint that session_rss_rebound.judge_rss_reading already keeps for the same problem: MCP servers make a healthy tree start above its ceiling.

Notes

  • Item 2 — symmetry risk: BACKGROUND_RSS_SETTLE_SECS = 30.0 is counted in seconds, while the existing settle rule (RSS_FOOTPRINT_SETTLE_TICKS = 2.5) is counted in ticks. The two start-up rules can now drift apart.

What this change ships

Inventory (6 items) — 3 justified

Intent: keep the background runtime warm on hosts with many MCP servers, so titles and status cards stop timing out. FIX.

  1. The background runtime recycles on growth past its start-up size, not on absolute size — duplicate of src/kiro_crew/session_rss_rebound.py judge_rss_reading
  2. A 30 s settle window after the first reading raises the baseline — duplicate of RSS_FOOTPRINT_SETTLE_TICKS
  3. A recycle's replacement takes a fresh baseline — justified
  4. A 0 reading sets no baseline — duplicate (judge_rss_reading: "A reading of 0 … settles nothing")
  5. The recycle log line names the growth and the baseline — justified
  6. The help text and docs describe a growth limit — justified

Blockers

Second spelling of the RSS start-up footprint — duplicate of session_rss_rebound.py
Trigger: the diff adds rss_baselines: dict[str, tuple[int, int, float, bool]] and BACKGROUND_RSS_SETTLE_SECS. I grepped RssFootprint|footprint|baseline across the session_cleanup* modules and found one existing mechanism with the same job: RssFootprint(seen_at, seen_mb, held_mb, settled) with judge_rss_reading, keyed per pid, with a settle window, skipping 0, and written for "many MCP servers". The description's sibling sweep never names it.
Outcome: two start-up footprint rules for one problem, with different units and different growth tests, each maintained separately.
Subtraction: delete rss_baselines and BACKGROUND_RSS_SETTLE_SECS, and judge the background reading through judge_rss_reading with its own RssRecords.
Clears when: the background path reuses RssFootprint/judge_rss_reading, or the PR shows a reported case the held-footprint rule fails for the background runtime.

[FIRST-PRINCIPLES-REVIEWED] f8af7e0

Description read

[DESCRIPTION-READ] 90aca8167c0bb86e1c72e6b8f376c1b039741faba6c6a19e62be3bacc938e5d2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed f8af7e0cda68a57e6a1ffc4f5c0686e01084f291 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f8af7e0

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed f8af7e0cda68a57e6a1ffc4f5c0686e01084f291 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f8af7e0

@iamwhatever

Copy link
Copy Markdown
Contributor

Stale-PR cleanup note. I rebased this onto main (f8af7e0; only the help-text wording conflicted, I kept main's text and applied your growth wording) and approved the fork runs. One blocker is left for @rinbn: the First Principles Review says the new rss_baselines / BACKGROUND_RSS_SETTLE_SECS duplicate session_rss_rebound.judge_rss_reading and RssFootprint. It clears if the background path reuses that rule with its own RssRecords, or if the PR shows a case where the held-footprint rule fails for the background runtime. That rule recycles at twice the held size, not at a fixed growth budget, so this is a design choice for you to make, not a mechanical fix.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

claude backend: the background runtime is recycled after every turn because its MCP servers alone exceed the 1536 MiB RSS ceiling

3 participants