Skip to content

fix(resources): admit and hand-shake against the agents slice ceiling - #11807

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
billygerhard:fix/slice-aware-admission
Sep 23, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
billygerhard:fix/slice-aware-admission

Conversation

@billygerhard

@billygerhard billygerhard commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

On a bare Linux host (no container cgroup), the sandbox places every agent process in kirocrew-agents.slice with MemoryHigh at 75% of RAM, but the spawn-admission probe (resource_status, _probe_resources) never reads that slice. Its "cgroup-clamped" branch reads /sys/fs/cgroup/memory.max (the container root), which does not exist on a bare host, so it falls back to host-wide MemAvailable. With plenty of host memory free the probe reported ample and admitted new spawns while the kernel was already throttling the entire agent subtree.

Observed on a 32 GB host: the slice's memory.events high counter climbed from 0 to 3.7M over 40 minutes (9.6 min of accumulated stall) with zero admission refusals logged. Under that throttle, kiro-cli could not answer initialize inside the fixed 30 s handshake budget; the runtime killed it (process_state=running on every one of nine consecutive kills, i.e. a healthy but slow process) and respawned into the same throttled slice, deepening the throttle. That spiral killed a running subagent mid-prompt.

Why it matters

Anyone running the gateway on a bare Linux host with the systemd slice active. The gate that exists to prevent overload is blind to the one ceiling the product itself imposes, so it admits work into an already-throttled slice, and the handshake reaper then converts "slow" into "dead" and retries, making the overload worse instead of backing off. The user sees agents and chat sessions dying with AcpProcessDied ... killed and no explanation.

What changed (motivation → approach → change)

Symptom: resource_status says ample while the agents slice is throttling; healthy runtimes are killed at 30.0 s and respawned.
Root cause: (1) _cgroup_available_gb consults only the container root cgroup; (2) the init-handshake budget is fixed and any timeout is treated as a dead process.

Two changes, kept separate:

  1. Slice-aware admission (sandbox.py, subagent.py). New _agents_slice_available_gb() in subagent reads min(memory.high, memory.max) - memory.current from the existing sandbox._agents_slice_cgroup_dir(). _cgroup_available_gb now takes the minimum of the process-ancestry figure (main's _container_cgroup_available_gb walker) and the slice figure, so posture flips to tight/critical when the slice is the binding limit and spawns are refused there (resource_status, admission_check, compute_max_subagents all sit behind that probe). The slice is a sibling of the gateway's cgroup, not an ancestor, and the walker never reads memory.high, so the slice term is not subsumed by it. subagent keeps its own _read_int_file (main's sizing tests isolate the kernel by patching that module's open); sandbox.read_cgroup_int serves the sandbox-side probes, and a test pins both readers to the same sentinel contract.

  2. Throttle-aware handshake (acp/runtime.py, acp/session_handle.py). New typed AcpRuntimeOverloaded(AcpRequestTimeout) with transient = False. The handshake helper checks sandbox.agents_slice_throttling() (memory.current >= memory.high, or the memory.events high counter climbing between probes): when the slice is throttling, the init budget is extended to 120 s; if the process is still alive at the deadline, the failure is raised as AcpRuntimeOverloaded. The uninitialized cold-start process is still reaped by the existing anti-leak guard (it holds no session and no prompt), but because the verdict is non-transient the retry layers that read .transient structurally (llm_helpers.acp_error_is_transient, the taskq acp_provider classifier) do NOT respawn a fresh cold start into the same throttle -- the error surfaces to the caller naming the remedy (free agent memory). When the slice is not throttling, behaviour is unchanged (30 s, plain AcpRequestTimeout, retried as before).

Related: #11483 widens the handshake timeout to 120 s unconditionally. This PR raises it only while the slice is throttling and adds the typed refusal, so it is complementary; if #11483 lands first this rebases down to the throttle-aware refusal on top of its budget.

Backwards compatibility

Compatible: on hosts with no agents slice (macOS, containers, systemd unavailable) both readers return None and every code path falls through to today's behaviour. No public signature, config key or error code is removed or renamed; AcpRuntimeOverloaded is additive.

Tests

test/test_agents_slice_admission.py (27 new tests; 760 passed with the existing runtime/sandbox/subagent-sizing neighbours at -n 2):

  • TestAgentsSliceHeadroom: lower of memory.high/memory.max binds; usage above the soft ceiling clamps to 0 not negative; max sentinel on high falls back to the hard ceiling; no ceiling at all does not constrain.
  • Probe: slice headroom lowers available_gb when it is the binding limit; container-root figure still wins when it is lower; missing slice leaves the probe unchanged; posture reaches critical from the slice alone.
  • Handshake: throttling slice extends the budget and a still-alive process at deadline raises AcpRuntimeOverloaded; AcpRuntimeOverloaded.transient is False while AcpRequestTimeout.transient is True, and acp_error_is_transient returns the matching verdict for each (the no-respawn pin); non-throttling slice keeps the 30 s path; dead process at deadline still reports the death, not overload; error exported from the runtime module.
  • Both cgroup readers (subagent._read_int_file, sandbox.read_cgroup_int) agree on every sentinel: max, garbage and an absent file all read as None.
  • Concurrent cold starts racing one throttle tick all read the same verdict: an observed memory.events high advance is held for 60 s (_SLICE_THROTTLE_EDGE_AT) instead of being consumed by the first probe.

Revert-mutation probes were run for both headline claims (dropping the slice term from the probe; replacing the overload conversion with a plain re-raise) and each fails exactly the named tests.

Manual verification

Reproduced the original failure signature from the journal on the affected host (40 min continuous memory.events high climb, nine failed init handshake kills all with process_state=running, zero admission refusals). With the patch, the live slice reads headroom_gb correctly against the 23 GB MemoryHigh. A full re-trigger of the throttle spiral was not re-run on the shared host; the handshake path is covered by the unit tests above.

Related Issues

Related: #11483, #10166, #10521.

Pattern harvest

Rule candidate: review-prompt
Pattern: "admission/health probe reads a ceiling the product does not set while ignoring the ceiling the product itself imposes" -- any resource gate should read the same cgroup/limit that the sandbox writes.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@billygerhard
billygerhard requested review from a team and smeyffret September 18, 2026 14:37
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from 5f2d84c to ee4ebf3 Compare September 18, 2026 15:01
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 6fa85f633262b7158dc41da7809d381913737a84 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The non-transient overload verdict leaks: type-based handlers still classify AcpRuntimeOverloaded as retryable because it subclasses AcpRequestTimeout.

Watch

  • The PR's no-respawn claim ("every retry layer reads it structurally") relies on handlers reading .transient, but _author_startup_retryable (src/kiro_crew/workflows/service.py:81) returns True on bare isinstance(exc, (AcpRequestTimeout, ...)), so the workflow-author startup loop retries up to 3 fresh cold starts into the same throttled slice — bounded, but exactly the behavior the subclass exists to stop, and every future type-dispatching handler inherits the same hole.
    Clears when: type-based catch sites (workflows/service.py:81; audit run.py:1151's _StartTimeout path) consult the transient flag, or a test pins that no retry classifier treats AcpRuntimeOverloaded as retryable.
  • The description says the throttled budget is "extended to 120 s"; the diff ships 90 s, pinned below the 120 s subagent startup watchdog. The stale number matters because sibling PR fix: widen ACP initialize handshake timeout (env-configurable, 120s) #11483 proposes an unconditional 120 s budget, which would violate the watchdog ordering this PR's own test (test_extended_budget_expires_before_the_startup_watchdog_reaps) establishes.
    Clears when: the description states 90 s and the #11483 coordination note acknowledges the watchdog ceiling.

[DESIGN-REVIEWED] 6fa85f6

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 6fa85f633262b7158dc41da7809d381913737a84 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/acp/runtime.py:2177 -- "being throttled" is classified as retryable before transient=False, so taskq respawns into the overloaded slice -> Fix: avoid provider-throttle vocabulary in this message.
FINDING -- src/kiro_crew/acp/runtime.py:2152 -- "throttled_at_spawn or" reports overload after pressure has cleared, suppressing a valid retry -> Fix: require a fresh throttle reading at the deadline.
[GPT-REVIEWED] 6fa85f6

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 6fa85f633262b7158dc41da7809d381913737a84 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Description twice says the throttled init budget is "120 s"; the diff ships 90 s (watchdog-bounded) — reconcile the numbers, especially against #11483's unconditional 120 s.

Not justified as shipped

  • Item 4 — undeclared: _send_and_await now keeps any timed-out initialize future registered (runtime.py adopt hunk) so a late-detected throttle can keep waiting on the same request; the description never mentions this transport-level change.
Inventory (7 items) — 6 justified

Intent: FIX — stop admitting spawns into, and killing healthy-but-slow cold starts under, the memory throttle the sandbox itself imposes (reproduction in description; accepted rfc-overload-resilience records "overload is not death" and admission-reads-pressure).

  1. Spawn admission refuses (posture tight/critical) when the agents slice throttles, despite ample host RAM — justified
  2. initialize budget rises 30 s→90 s while the slice throttles — justified
  3. Still-alive process at the deadline under throttle fails as non-retried AcpRuntimeOverloaded naming the remedy, instead of kill-and-respawn — justified
  4. An initialize timeout keeps the request pending so a late-detected throttle waits it out — undeclared
  5. New sandbox.agents_slice_throttling() probe with 60 s shared edge-hold — justified
  6. New sandbox.read_cgroup_int beside subagent._read_int_file, agreement pinned by test; constraint verified (sizing tests patch builtins.open, which Path.read_text bypasses) — justified
  7. Both owning specs updated in the same commit — justified

Verified on base: llm_helpers.acp_error_is_transient and taskq/adapters/acp_provider.py:193 read .transient structurally; _STARTUP_TIMEOUT_SECS = 120 (subagent.py:542) so 90 s clears the reaper; _agents_slice_cgroup_dir exists (sandbox.py:12358).

[FIRST-PRINCIPLES-REVIEWED] 6fa85f6

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 6fa85f633262b7158dc41da7809d381913737a84 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6fa85f6

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from ee4ebf3 to 5cd5bf8 Compare September 18, 2026 15:18
@billygerhard

billygerhard commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Review response for 5cd5bf8c2 (addresses the GPT 5.6, First Principles and Design lanes on ee4ebf37b).

First Principles item 4 (zero consumers of AcpRuntimeOverloaded, inherited transient = True) -- accepted, this was a real gap. The description promised "refused instead of respawning" but an inherited transient = True meant every retry layer that reads the verdict structurally would respawn straight into the throttle, so only the 120 s budget actually shipped. Fixed by making the verdict transient = False on the subclass: llm_helpers.acp_error_is_transient and the taskq acp_provider classifier both read .transient via getattr, so both now decline to respawn without any new except site. Pinned by test_overload_is_not_a_retry_verdict (parent True, subclass False, and acp_error_is_transient returning the matching verdict for each). The class docstring now states why overload is the one timeout that must not be retried.

First Principles item 6 (duplicate cgroup reader) -- accepted. One reader remains: sandbox.read_cgroup_int (public, str | Path), and subagent binds it at module scope under its historical _read_int_file name so the existing sizing/coverage tests that patch that attribute keep working. Pinned by test_one_cgroup_reader_serves_both_probes (sa._read_int_file is sb.read_cgroup_int).

GPT finding (local import in subagent.py) -- accepted. Both sandbox imports are module-scope now; sandbox does not import subagent, so there is no cycle.

Design review (description drift) -- accepted. The description named a helper that does not exist (agents_slice_headroom_gb; the real one is _agents_slice_available_gb) and claimed resource_status reports which ceiling bound, which no hunk shipped. Both corrected in the description; the behaviour list now matches the diff.

GPT BLOCKING F1 ("overload still kills the live runtime") -- rebutted; requesting the fence be lifted. The reaped process is a cold-start runtime that never completed initialize: spawn refuses when _process is not None, so _initialize_handshake only ever runs against a fresh process with no session, no in-flight prompt and no data (_send_and_await carries control-plane requests only, never the prompt stream). The except BaseException guard's kill() is the designed anti-leak cleanup -- leaving that process behind leaks the process, its reader/stderr tasks, its PID-file entry and its _PROTECTED_PIDS shield for the life of the host, which is the very leak class this repo just fixed elsewhere. The lane's own adjudicator reached the same conclusion (FLAG, "nothing is lost by reaping an uninitialized cold-start process, and the alternative is the actual leak bug"). What the finding's premise actually points at -- "the kill is followed by an immediate respawn into the same throttle" -- is the item-4 gap above, and it is now closed by the non-transient verdict rather than by preserving a half-initialized process. test_the_spawn_path_still_kills_on_a_failed_handshake deliberately pins the reap.

Local gates on 5cd5bf8c2: 760 tests passed (-n 2, targeted runtime/sandbox/subagent files), flake8/isort/mypy clean on changed files, brand + comment-history + docs-lint gates pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from 5cd5bf8 to 1dd20cf Compare September 18, 2026 16:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from 1dd20cf to e7b493d Compare September 18, 2026 18:49
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from 1492109 to 1588e2a Compare September 21, 2026 18:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 21, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from 1588e2a to 4ea6fdf Compare September 21, 2026 18:43
@billygerhard

Copy link
Copy Markdown
Contributor Author

Review response for 4ea6fdf317 (addresses the GPT 5.6 lane on 1588e2aac3).

GPT finding (late throttle detection still reaps at the plain deadline) -- accepted, fixed. On a fresh gateway agents_slice_throttling() can only baseline the memory.events counter on its first call, so a throttle already under way with usage just under memory.high is invisible at spawn, the plain 30 s budget applies, and at the deadline the handshake detected the throttle but still raised AcpRuntimeOverloaded -- the healthy process was reaped without ever receiving the extended budget the PR promises.

Fix: _send_and_await keeps a timed-out initialize request registered (the same mechanism session/new uses for its StartCollector adoption) and hands the pending future back on the exception. _initialize_handshake, on a plain-budget timeout where the process is alive and throttle is first seen at the deadline, awaits that same request for the remainder of _INIT_TIMEOUT_UNDER_THROTTLE instead of refusing; only if the extended total also passes does it raise overload, after unregistering the request. Pinned by test_late_detected_throttle_keeps_the_request_pending (late reply arriving inside the extension is the handshake result), test_late_detected_throttle_still_overloads_at_the_extended_deadline (message names the full budget, request unregistered) and test_send_and_await_keeps_a_timed_out_initialize_registered. Mutation-probed: disabling the extension branch fails both handshake tests.

Also on this head, not from the diff: Backend Tests (Windows) (8) is two test_subagent_sizing.py drain-pump timing tests (#12596, Windows runner clock; green on every Linux shard and on prior Windows runs of this diff), and the hosted private-member E2E lane spent its 300 s in harness teardown waiting for the gateway process group to exit (#12597; green on other CI runs this hour).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 21, 2026
@billygerhard
billygerhard force-pushed the fix/slice-aware-admission branch from 4ea6fdf to f68dc74 Compare September 22, 2026 13:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants