Skip to content

fix(adaptive): backport idle cap recovery to release/0.8.0 - #15544

Merged
bolichen97 merged 1 commit into
release/0.8.0from
backport/0.8.0-adaptive-idle-recovery
Sep 30, 2026
Merged

bolichen97 merged 1 commit into
release/0.8.0from
backport/0.8.0-adaptive-idle-recovery

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backport of #15541 to release/0.8.0. Do NOT merge before #15541 merges on main. If #15541 changes before it lands, this branch is re-cut from its final commit. One commit, git cherry-pick -x 9a56254f8f, applied cleanly (no conflicts) on release/0.8.0 @ 556bf8e. adaptive/{policy,controller,signals}.py are byte-identical between main and release/0.8.0, so the 0.8.0.x line carries the same defect.

Goal: On the 0.8.0 release line, a concurrency cap the adaptive controller cut under pressure climbs back once the gateway sits idle and clear, and the status report says why the cap is low.

A user ran overlapping long workflows that hit timeouts, provider-process failures and CPU/memory pressure. The controller cut the execution cap to 1 of 9 slots. After every workflow finished and the pressure cleared, the cap stayed low ("idle time does not restore capacity"). It later reached only 2/9, so ctx.parallel() work ran mostly one call at a time.

Root cause (src/kiro_crew/adaptive/policy.py on main @ c1a02ce):

  • The only growth path is _maybe_increase. It needs demand at the cap (sample.demand >= self._exec_cap) and completions since the last change (or a progress probe, which needs queued > 0 and running >= cap). The module doc says so: "An idle track never drifts up." An idle process has neither.
  • demand, completions and progressing all come from SubagentManager (controller.py build_sample / _ingest_manager_runs). Workflow ctx.agent() calls and TaskRunner steps run on the runner lane (taskq/adapters/runner.py RunnerLane). That lane is bounded by the same effective cap, and its session/new starts feed record_start (acp/runtime_start.py), so it can cut the cap. It can never earn it back, because its occupancy and completions are not the manager's.
  • A pause resumes only when sample.completions advances past the probe base (_while_paused). A probe slot taken by workflow work never reports, so the cap sat at 1 in probing indefinitely. One sub-agent completion resumes it at floor + 1 = 2, which matches the "only 2/9" the user saw.

Reproduced against unchanged main with the pure policy: severe memory for two samples, then an hour of clear idle samples, left the cap at 1 ("probe in flight"). A loop-lag cut 4 -> 2 -> 1 followed by an hour of clear idle samples also stayed at 1 ("clear; increase not yet earned or no demand").

Why it matters

Anyone whose workflows or sub-agents once met pressure keeps a crippled cap until they restart the gateway, even on an idle, healthy host. Workflow fan-out (ctx.parallel) is hit hardest because its own work can never earn the cap back. The report gives no hint why the cap is 1 of 9. RFC overload-resilience (status partial) sets the goal "recover from transient failures without a human restarting the Gateway", and its §5 "recovery up-scaling" row expects "pressure removed -> +1 per window".

Not a goal

What changed (motivation → approach → change)

Symptom: the cap stays cut on an idle, clear host. Cause: every growth path needs exec-track evidence that an idle track, or runner-lane load, never produces. Change:

  • Idle recovery (AdaptivePolicy): once the exec track has had no demand (running + queued == 0) and the host no signal of any kind for DEFAULT_IDLE_RECOVERY_SECS (60 s, two congestion-avoidance windows), a cap below the fresh-start value min(user_max, adaptive_initial) climbs +1 per clean window back to it. While paused and probing, a probe that never reported resumes at floor + 1, clamped to the fresh-start value (so adaptive_floor >= adaptive_initial cannot resume above it); the spawn gate stays at its floor, since idle is no backend-init evidence. Demand or any signal restarts the idle clock.
  • Bound = fresh-start value, not the user ceiling. A drained process never holds more untested concurrency than a restart would give it, and every step above that is still earned under the existing rules. The next corroborated pressure cuts exactly as before. Alternatives considered: restoring to the ceiling (this would admit a full untested burst on the next fan-out), and restoring to the pre-cut cap (the cap the pressure fired under).
  • Why the cap is low (telemetry):
    • The policy snapshot gains last_cut: the decision that last lowered the exec cap (action, reason, signals, sample time). An increase never clears it.
    • AdaptiveController.state() adds last_cut.age_secs.
    • resource_status prints Last pressure cut: decrease 42s ago (corroborated pressure: ...) while the cap is below the ceiling. It is labelled as the last cut rather than as the cause, because a cap that has recovered to its fresh-start value is still below the ceiling.
    • A clear hold names what the next step waits for: the ceiling, the idle clock (idle; restoring toward 4 in 25s), demand at the cap, or the completions still owed. Before, the reason was always the generic "increase not yet earned or no demand".
  • Spec docs/system-specs/modules/adaptive-concurrency.md is updated in the same commit: the decision-rule row, a new "Idle recovery" section, Visibility, Fixed tuning and Tests.

Backwards compatibility

Compatible: no config key, schema, API field or action enum is removed or renamed. The new snapshot/state keys are additive, PolicyParams.idle_recovery_secs has a default, and ADAPTIVE_DECISIONS still uses the existing increase / resume actions. Only hold reason strings changed text; no code or test on main matches them.

Tests

  • test_adaptive_policy.py::TestIdleRecovery:
    • A cut cap climbs back one step per 30 s window, only after 60 s idle and clear, to the fresh-start value (4) and not to the ceiling (9).
    • An unreported probe resumes once idle and clear, and the gate stays at its floor.
    • Demand, or a lone uncorroborated signal, restarts the idle clock.
    • Work below the cap is not idle, and its hold says "no demand at the cap".
    • A cap already at the fresh-start value never drifts.
    • last_cut names the cut (decrease, then pause) and survives an increase.
    • An idle resume never passes the fresh-start cap, even with floor == initial.
    • idle_recovery_secs <= 0 is refused.
  • test_adaptive_controller.py::test_an_idle_controller_restores_the_cap_and_names_the_cut: end to end through tick() on a virtual clock. A lag cut to 1, then an idle gateway, returns the manager's effective cap to 4. resource_status prints Last pressure cut: decrease 10s ago (corroborated pressure: loop_lag) while the cap is below the ceiling and drops the line at the ceiling.
  • Mutation: reverting src/ to main while keeping the new tests fails 7 of them (the no-drift guard passes both ways by design). Dropping only the idle-resume clamp fails test_an_idle_resume_never_passes_the_fresh_start_cap.

Manual verification

N/A. The controller is a pure state machine driven by samples. The e2e test drives the real AdaptiveController.tick() and the real resource_status renderer on a virtual clock, which is exactly the path a live gateway takes every 5 s.

Local gates on the head:

  • black --target-version py310 / isort / flake8 are clean on the touched files.
  • mypy src/kiro_crew: "Success: no issues found in 1997 source files".
  • scripts/docs-lint.sh passed.
  • Every test file that references adaptive or resource_status (60 files): 5066 passed. The 1 failure, test_security_path_resolve_bounded.py::test_the_resolver_pool_knob_reaches_the_pool_in_a_fresh_interpreter, is a host artifact: its fresh interpreter imports a stale editable install, and the test passes with PYTHONPATH=<worktree>/src.

Related Issues

Backport of #15541. Follow-up for the deferred runner-lane earn evidence: #15556. Related: #14013 (the learned-cost ceiling issue, a separate mechanism), #12710 (an earlier loop-lag source that pinned the cap at 1).

Pattern harvest

Rule candidate: review-prompt
Pattern: "a controller whose decrease reads a signal source (runner-lane session starts) that its increase cannot observe" — every AIMD input that can cut must either be able to earn, or be covered by a time-based retirement of the cut.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team as a code owner September 30, 2026 19:01
@bolichen97
bolichen97 requested review from dodgeblaster and removed request for a team September 30, 2026 19:01
@bolichen97

Copy link
Copy Markdown
Contributor Author

Local test evidence for this backport (head 8636c3233d, release/0.8.0 @ 556bf8e + cherry-pick -x d03eb532f6, CPython 3.12.13, Linux):

# every test file that references adaptive / resource_status (58 files on release/0.8.0)
python3 -m pytest -q -n 8 --dist loadgroup --max-worker-restart=2 <58 files>
4740 passed, 141 warnings in 72.56s

# the two owning suites
python3 -m pytest -q test/test_adaptive_policy.py test/test_adaptive_controller.py
133 passed

# mutation: src/kiro_crew/adaptive/{policy,controller}.py + resource_status.py reverted to release/0.8.0, new tests kept
python3 -m pytest -q test/test_adaptive_policy.py test/test_adaptive_controller.py -k "Idle or idle_controller"
7 failed, 1 passed   # the 1 = no-drift-at-fresh-start guard, passes both ways by design

flake8 (touched files): clean
black --check --target-version py310 (touched files): 5 files would be left unchanged
mypy src/kiro_crew: Success: no issues found in 1860 source files

Reminder: do not merge before #15541.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 9be61d09c311da5f10699d39eeee6c16c44d5238 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm #15541 merged on main unchanged before this backport merges; if it was re-cut, this branch must be re-cut from its final commit.

[DESIGN-REVIEWED] 9be61d0

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 9be61d09c311da5f10699d39eeee6c16c44d5238 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9be61d0

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9be61d09c311da5f10699d39eeee6c16c44d5238: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 9be61d09c311da5f10699d39eeee6c16c44d5238 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Backport precondition is the only open question: confirm #15541 landed on main unchanged, since this branch is a verbatim cherry-pick of it.

What this change ships

Inventory (8 items) — 8 justified

Intent: FIX — on the 0.8.0 line, a concurrency cap the adaptive controller cut under pressure never recovers on an idle host; restore it and say why the cap is low. (Reproduction in the description, mutation-tested, and covered by RFC overload-resilience, partial on base, whose recovery-up-scaling row expects "pressure removed → +1 per window".)

  1. A cut cap climbs +1 per clean window back to the fresh-start value after 60s idle and clear — justified
  2. A pause whose probe never reported resumes at floor+1, clamped to fresh-start — justified
  3. resource_status prints "Last pressure cut: …" while the cap is below the ceiling — justified
  4. Snapshot/state gain last_cut and age_secs — justified
  5. Hold reasons now name what the next increase waits for (grepped old string: 0 consumers) — justified
  6. PolicyParams.idle_recovery_secs tuning field, >0 enforced, not a config key — justified
  7. adaptive-concurrency.md spec updated in the same commit — justified
  8. Module-doc pin "An idle track never drifts up" rewritten to "…PAST the fresh-start cap", quoted and evidenced in the description — justified

[FIRST-PRINCIPLES-REVIEWED] 9be61d0

@bolichen97
bolichen97 force-pushed the backport/0.8.0-adaptive-idle-recovery branch from 8636c32 to 2706423 Compare September 30, 2026 20:06
@bolichen97

Copy link
Copy Markdown
Contributor Author

Re-cut from the amended main-PR commit (#15541 round 1: idle-resume clamp, Last pressure cut label, unused snapshot keys removed). Head is now 2706423c14 = release/0.8.0 @ 556bf8e + cherry-pick -x 907c31b631 (clean).

# every test file referencing adaptive / resource_status (58 files on release/0.8.0)
4741 passed, 142 warnings in 62.22s

# mutation: adaptive/{policy,controller}.py + resource_status.py reverted to release/0.8.0, new tests kept
-k "Idle or idle_controller": 8 failed, 1 passed   # the 1 = no-drift guard, passes both ways by design

flake8 / black --check (touched files): clean
mypy src/kiro_crew: Success: no issues found in 1860 source files

Still: do not merge before #15541.

After overlapping long workflows met timeouts and memory pressure, the
adaptive controller cut the execution cap to 1 (or paused and probed at
1) and it stayed there for the life of the gateway: workflow
ctx.parallel() calls ran one at a time on a clear, idle host.

Every increase rule is earned on the exec track's own evidence --
demand at the cap and completions diffed from the sub-agent manager.
An idle process produces neither, and neither does load on the runner
lane (workflow ctx.agent() calls, TaskRunner steps): the lane is
bounded by the effective cap and its session starts feed the timeout
signal, so it can cut the cap, but its occupancy and completions are
not the manager's. A pause probe taken by such work never reports, so
the cap sat at 1; one sub-agent completion resumed it at floor + 1.

Add idle recovery: after 60 s with no exec demand and no pressure
signal of any kind, a cap below the fresh-start value
min(user_max, adaptive_initial) climbs +1 per clean window back to it,
and an unreported probe resumes at floor + 1 (never above the
fresh-start value; the spawn gate stays at its floor and earns on
inits). The bound means a drained
process never holds more untested concurrency than a restart would;
growth past it is still earned, and the next corroborated pressure
cuts as before.

Say why a cap is low: the policy snapshot keeps last_cut (the decision
that last lowered the exec cap, with its signals, and its age in
controller.state()), resource_status prints it as "Last pressure cut: ..."
while the cap is below the ceiling, and a clear hold names what the
next step waits for (the idle clock, demand at the cap, or the
completions still owed).

(cherry picked from commit 9a56254)
@bolichen97
bolichen97 force-pushed the backport/0.8.0-adaptive-idle-recovery branch from 2706423 to 9be61d0 Compare September 30, 2026 21:00
@bolichen97

Copy link
Copy Markdown
Contributor Author

Re-cut from #15541 round 2 (an idle resume leaves the spawn gate at its floor). Head 9be61d09c3 = release/0.8.0 @ 556bf8e + cherry-pick -x 9a56254f8f (clean).

58 test files referencing adaptive / resource_status: 4741 passed, 141 warnings in 65.69s
mutation (adaptive/{policy,controller}.py + resource_status.py reverted, new tests kept): 8 failed, 1 passed (no-drift guard)
flake8 / black --check (touched files): clean
mypy src/kiro_crew: Success: no issues found in 1860 source files

Still: do not merge before #15541.

@bolichen97
bolichen97 merged commit e1a248b into release/0.8.0 Sep 30, 2026
20 of 26 checks passed
@bolichen97
bolichen97 deleted the backport/0.8.0-adaptive-idle-recovery branch September 30, 2026 21:06
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 9be61d09c311da5f10699d39eeee6c16c44d5238 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9be61d0

Verdict parsed from the review's SHA-scoped output markers for commit 9be61d09c311da5f10699d39eeee6c16c44d5238.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9be61d09c311da5f10699d39eeee6c16c44d5238: <one-sentence reason>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant