Repository navigation
fix(adaptive): backport idle cap recovery to release/0.8.0 - #15544
Conversation
|
Local test evidence for this backport (head Reminder: do not merge before #15541. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm #15541 merged on [DESIGN-REVIEWED] 9be61d0 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Backport precondition is the only open question: confirm #15541 landed on What this change shipsInventory (8 items) — 8 justifiedIntent: FIX — on the 0.8.0 line, a concurrency cap the adaptive controller cut under pressure never recovers on an idle host; restore it and say why the cap is low. (Reproduction in the description, mutation-tested, and covered by RFC overload-resilience,
[FIRST-PRINCIPLES-REVIEWED] 9be61d0 |
8636c32 to
2706423
Compare
|
Re-cut from the amended main-PR commit (#15541 round 1: idle-resume clamp, Still: do not merge before #15541. |
After overlapping long workflows met timeouts and memory pressure, the adaptive controller cut the execution cap to 1 (or paused and probed at 1) and it stayed there for the life of the gateway: workflow ctx.parallel() calls ran one at a time on a clear, idle host. Every increase rule is earned on the exec track's own evidence -- demand at the cap and completions diffed from the sub-agent manager. An idle process produces neither, and neither does load on the runner lane (workflow ctx.agent() calls, TaskRunner steps): the lane is bounded by the effective cap and its session starts feed the timeout signal, so it can cut the cap, but its occupancy and completions are not the manager's. A pause probe taken by such work never reports, so the cap sat at 1; one sub-agent completion resumed it at floor + 1. Add idle recovery: after 60 s with no exec demand and no pressure signal of any kind, a cap below the fresh-start value min(user_max, adaptive_initial) climbs +1 per clean window back to it, and an unreported probe resumes at floor + 1 (never above the fresh-start value; the spawn gate stays at its floor and earns on inits). The bound means a drained process never holds more untested concurrency than a restart would; growth past it is still earned, and the next corroborated pressure cuts as before. Say why a cap is low: the policy snapshot keeps last_cut (the decision that last lowered the exec cap, with its signals, and its age in controller.state()), resource_status prints it as "Last pressure cut: ..." while the cap is below the ceiling, and a clear hold names what the next step waits for (the idle clock, demand at the cap, or the completions still owed). (cherry picked from commit 9a56254)
2706423 to
9be61d0
Compare
|
Re-cut from #15541 round 2 (an idle resume leaves the spawn gate at its floor). Head Still: do not merge before #15541. |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal: On the 0.8.0 release line, a concurrency cap the adaptive controller cut under pressure climbs back once the gateway sits idle and clear, and the status report says why the cap is low.
A user ran overlapping long workflows that hit timeouts, provider-process failures and CPU/memory pressure. The controller cut the execution cap to 1 of 9 slots. After every workflow finished and the pressure cleared, the cap stayed low ("idle time does not restore capacity"). It later reached only 2/9, so
ctx.parallel()work ran mostly one call at a time.Root cause (
src/kiro_crew/adaptive/policy.pyonmain@ c1a02ce):_maybe_increase. It needs demand at the cap (sample.demand >= self._exec_cap) and completions since the last change (or a progress probe, which needsqueued > 0andrunning >= cap). The module doc says so: "An idle track never drifts up." An idle process has neither.demand,completionsandprogressingall come fromSubagentManager(controller.pybuild_sample/_ingest_manager_runs). Workflowctx.agent()calls and TaskRunner steps run on the runner lane (taskq/adapters/runner.pyRunnerLane). That lane is bounded by the same effective cap, and itssession/newstarts feedrecord_start(acp/runtime_start.py), so it can cut the cap. It can never earn it back, because its occupancy and completions are not the manager's.sample.completionsadvances past the probe base (_while_paused). A probe slot taken by workflow work never reports, so the cap sat at 1 in probing indefinitely. One sub-agent completion resumes it atfloor + 1= 2, which matches the "only 2/9" the user saw.Reproduced against unchanged
mainwith the pure policy: severe memory for two samples, then an hour of clear idle samples, left the cap at 1 ("probe in flight"). A loop-lag cut 4 -> 2 -> 1 followed by an hour of clear idle samples also stayed at 1 ("clear; increase not yet earned or no demand").Why it matters
Anyone whose workflows or sub-agents once met pressure keeps a crippled cap until they restart the gateway, even on an idle, healthy host. Workflow fan-out (
ctx.parallel) is hit hardest because its own work can never earn the cap back. The report gives no hint why the cap is 1 of 9. RFC overload-resilience (statuspartial) sets the goal "recover from transient failures without a human restarting the Gateway", and its §5 "recovery up-scaling" row expects "pressure removed -> +1 per window".Not a goal
compute_max_subagents/_host_mem_termholding the auto cap at its floor of 3; Auto subagent cap priced every slot at the heaviest agent's p90 peak #14013, fix(subagent): size the auto cap from a typical run, not the heaviest agent's peak #14015, fix(subagent): size the auto cap from the agent's own processes, not the whole-tree peak #15472). That is a different mechanism: it sizes the user ceiling once at startup, whereas this PR is about the AIMD cap beneath it.What changed (motivation → approach → change)
Symptom: the cap stays cut on an idle, clear host. Cause: every growth path needs exec-track evidence that an idle track, or runner-lane load, never produces. Change:
AdaptivePolicy): once the exec track has had no demand (running + queued == 0) and the host no signal of any kind forDEFAULT_IDLE_RECOVERY_SECS(60 s, two congestion-avoidance windows), a cap below the fresh-start valuemin(user_max, adaptive_initial)climbs+1per clean window back to it. While paused and probing, a probe that never reported resumes atfloor + 1, clamped to the fresh-start value (soadaptive_floor >= adaptive_initialcannot resume above it); the spawn gate stays at its floor, since idle is no backend-init evidence. Demand or any signal restarts the idle clock.last_cut: the decision that last lowered the exec cap (action, reason, signals, sample time). An increase never clears it.AdaptiveController.state()addslast_cut.age_secs.resource_statusprintsLast pressure cut: decrease 42s ago (corroborated pressure: ...)while the cap is below the ceiling. It is labelled as the last cut rather than as the cause, because a cap that has recovered to its fresh-start value is still below the ceiling.idle; restoring toward 4 in 25s), demand at the cap, or the completions still owed. Before, the reason was always the generic "increase not yet earned or no demand".docs/system-specs/modules/adaptive-concurrency.mdis updated in the same commit: the decision-rule row, a new "Idle recovery" section, Visibility, Fixed tuning and Tests.Backwards compatibility
Compatible: no config key, schema, API field or action enum is removed or renamed. The new snapshot/state keys are additive,
PolicyParams.idle_recovery_secshas a default, andADAPTIVE_DECISIONSstill uses the existingincrease/resumeactions. Only holdreasonstrings changed text; no code or test onmainmatches them.Tests
test_adaptive_policy.py::TestIdleRecovery:last_cutnames the cut (decrease, then pause) and survives an increase.floor == initial.idle_recovery_secs <= 0is refused.test_adaptive_controller.py::test_an_idle_controller_restores_the_cap_and_names_the_cut: end to end throughtick()on a virtual clock. A lag cut to 1, then an idle gateway, returns the manager's effective cap to 4.resource_statusprintsLast pressure cut: decrease 10s ago (corroborated pressure: loop_lag)while the cap is below the ceiling and drops the line at the ceiling.src/tomainwhile keeping the new tests fails 7 of them (the no-drift guard passes both ways by design). Dropping only the idle-resume clamp failstest_an_idle_resume_never_passes_the_fresh_start_cap.Manual verification
N/A. The controller is a pure state machine driven by samples. The e2e test drives the real
AdaptiveController.tick()and the realresource_statusrenderer on a virtual clock, which is exactly the path a live gateway takes every 5 s.Local gates on the head:
black --target-version py310/isort/flake8are clean on the touched files.mypy src/kiro_crew: "Success: no issues found in 1997 source files".scripts/docs-lint.shpassed.adaptiveorresource_status(60 files): 5066 passed. The 1 failure,test_security_path_resolve_bounded.py::test_the_resolver_pool_knob_reaches_the_pool_in_a_fresh_interpreter, is a host artifact: its fresh interpreter imports a stale editable install, and the test passes withPYTHONPATH=<worktree>/src.Related Issues
Backport of #15541. Follow-up for the deferred runner-lane earn evidence: #15556. Related: #14013 (the learned-cost ceiling issue, a separate mechanism), #12710 (an earlier loop-lag source that pinned the cap at 1).
Pattern harvest
Rule candidate: review-prompt
Pattern: "a controller whose decrease reads a signal source (runner-lane session starts) that its increase cannot observe" — every AIMD input that can cut must either be able to earn, or be covered by a time-based retirement of the cut.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement