Skip to content

fix(subagent): capture each run's deadline at start and forward it to the ACP transport - #7195

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
rubencu:feat/adaptive-subagent-timeouts
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
rubencu:feat/adaptive-subagent-timeouts

Conversation

@rubencu

@rubencu rubencu commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A config reload no longer changes or cuts short a subagent run that is already executing, because the run's deadline and its ACP transport budget are both fixed when it starts.

A subagent's deadline was read from the manager default at each point of use. A reload while a run executed changed the deadline the reaper compared it against, the deadline its follow-up watcher timed itself on, the deadline a cancel-recovery respawn of the same run started with, and the figure its timeout record named. The reaper also measured from registration, so time spent waiting for spawn approval counted against the run's deadline before the run started. Every prompt resolved the ACP transport wait from live config, so a reload that lowered agent.subagent_timeout_secs from above the 4-hour transport floor could cut a later retry or recovery prompt below the deadline its run still enforced, and the run then ended on a raw transport timeout instead of its own deadline.

Why it matters

An operator who lowers agent.subagent_timeout_secs while subagents run should change the next runs, not cut the work already in flight. A run that times out should name the deadline it actually enforced, and the transport layer should never be the first thing to expire.

Not a goal

  • Learning a longer deadline from observed runs: an earlier revision carried an opt-in learner that raised one shared deadline after any timed-out run with stream activity. That rewards a busy wedge exactly like a healthy long run, so it is removed, not deferred. Raising a deadline stays an operator decision via agent.subagent_timeout_secs.
  • Making agent.subagent_timeout_secs editable from the dashboard: it is a resource-protection policy decision for a maintainer, so it stays known gap 1 in docs/architecture/resource-protection.md.
  • Raising a deadline on a run that is already executing: each run keeps the deadline captured at its start.
  • Changing how legacy providers are called: an undeclared provider keeps the exact stream(message) call.
  • Changing stall detection or the startup watchdog.

What changed (motivation → approach → change)

Motivation: a run's deadline and its transport wait both followed live config, so neither was fixed for the life of the run. Approach: capture both once, when execution starts, and enforce only the captured values wherever the run is timed. Change:

  • _run() captures agent.subagent_timeout_secs on SubagentInfo.timeout_secs while the record still carries the zero sentinel. Unexpected-cancel recovery respawns the same record and keeps its nonzero value. asyncio.wait_for, the reaper and the follow-up watcher all enforce that value. The reaper measures an executing run from _exec_started. A run that is not executing is measured on its registration age. Before its first start (parked at spawn approval, or approved and waiting for admission) it has captured nothing and keeps the live default as its bound. Between the two attempts of a context-overflow cancel recovery, which clears _exec_started before the replacement waits for capacity, it keeps its captured deadline. The timeout and reap records name the enforced deadline.
  • At execution start the run also resolves one ACP prompt budget through agent_sdk.capture_prompt_timeout: the larger of the live transport wait and the captured deadline plus 60 seconds. It forwards that budget on the original prompt and on every transient retry, model fallback, stop-recovery and dependency-recovery prompt. AcpProvider and AcpSessionProvider declare prompt_timeout_for_deadline. An undeclared provider gets None and keeps its one-argument stream(message) call, with no keyword probe and no TypeError retry.
  • resolve_prompt_timeout() is unchanged from main: the ordinary timeout=None path (dashboard turns, review runs) still resolves live config, bounded by agent.subagent_timeout_secs with its 0 sentinel.
Case Before After
Config reload while a run executes 🟨 the reaper, the follow-up watcher, a recovery respawn and the timeout record read the new value 🟩 the run keeps its captured deadline and the next run reads the new value
Spawn-approval wait 🟨 counted against the run's deadline by the reaper 🟩 the captured deadline is measured from execution start, and a run not yet executing keeps the registration-age bound against the live default
Config reload during a cancel-recovery capacity wait 🟨 the reaper compared the run's registration age with the new value 🟩 compared with the run's captured deadline
Retry, fallback or recovery prompt after a lowering reload 🟥 transport wait resolved from the lowered config 🟩 reuses the budget captured at execution start
Deadline exactly at the 4-hour transport floor 🟨 transport and run expire together 🟩 transport expires 60 seconds after the run
Timeout and reap records 🟨 name the manager default 🟩 name the deadline the run enforced
Legacy provider 🟦 stream(message) 🟦 stream(message)
Dashboard turns and other timeout=None callers 🟦 live config 🟦 live config

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Changed area Purpose
src/kiro_crew/subagent.py, src/kiro_crew/subagent_manager/run.py SubagentInfo.timeout_secs and its capture at execution start, plus the transport budget captured once per run and forwarded on every prompt.
src/kiro_crew/subagent_manager/monitoring.py, continuation.py, terminal.py The reaper enforces the captured deadline from _exec_started and measures a run that is not executing on its registration age, against the live default before its first start and against its captured deadline between recovery attempts. The follow-up watcher waits for the capture, and the reap record names the enforced deadline.
src/kiro_crew/acp/client.py, src/kiro_crew/acp/session_provider.py, src/kiro_crew/providers/, src/kiro_crew/agent_sdk/ resolve_prompt_timeout_for_deadline, prompt_timeout_for_deadline and capture_prompt_timeout, plus the optional timeout on stream(message, timeout=None, *, allow_image=True), the shape AcpSessionHandle.prompt and AcpClient.stream_events already use.
docs/system-specs/modules/, src/kiro_crew/docs/subagents.md Document the captured deadline, the reaper's clock and the per-run transport budget.
.github/black-baseline.txt test/test_prompt_timeout_follows_ceiling.py is black-clean and leaves the baseline.
test/test_acp_provider.py, test/test_acp_session_provider.py, test/test_agent_sdk_capabilities.py, test/test_prompt_timeout_follows_ceiling.py, test/test_session_sharing.py, test/test_spawn_followup.py, test/test_subagent_turn_resilience.py Lock in the captured deadline, the reaper's clock for executing and non-executing runs, the per-run transport budget, provider forwarding and legacy compatibility.

Removed from earlier revisions: the adaptive learner (src/kiro_crew/subagent_timeout.py and test/test_subagent_timeout.py), the agent.subagent_timeout_auto and agent.subagent_timeout_max_secs keys with their constants, config-baseline.json rows, loader parsing and dashboard editors, the persisted member-memory-bindings/subagent-timeout.json record with its write, read-back and shutdown drain, and the adaptive branch of resolve_prompt_timeout(). Also removed: the dashboard editor for agent.subagent_timeout_secs (its _EDITABLE_CONFIG row, the two config.loader imports and the write-gate parity row), and the docs/architecture/resource-protection.md edit, so that file is main's text again. The diff touches no file under src/kiro_crew/security/, no src/kiro_crew/dashboard/ file and no sandbox*.py.

Backwards compatibility

Compatible: every key main accepts is still accepted, and agent.subagent_timeout_secs keeps its range, its 0 sentinel and its CLI and config.json editing paths. A reload now applies to the next run instead of an executing one. A run parked before its first start (at spawn approval, or approved and waiting for admission into startup) is reaped on the registration age against the live default, as before this change. A run between the two attempts of a context-overflow cancel recovery is measured on the same registration age against its captured deadline, and an executing run on its captured deadline from _exec_started. Providers that declare nothing keep stream(message), and timeout=None callers resolve live config as before.

Removes nothing: the diff against main deletes no key, file, record or public function, and the learner and its keys were never on main.

Tests

  • Candidate 0be9319eadcc is one commit on main d3f462f23b4c (2026-10-10, test: use _floor_monkeypatch in poll-gate autouse fixture #18696), 24 files, +894/-34.
  • The seven changed test modules pass 392/392. New or ported there: the reaper enforces each run's captured deadline from execution start, reaps a long-parked start on its registration age while leaving a fresh one alone, and measures a run between two recovery attempts against its captured deadline. A reload that cuts the default to 60 seconds during a context-overflow recovery's capacity wait leaves the run alive through one real reaper sweep, and the replacement then runs on the captured deadline. The reap and timeout records name the enforced deadline, the follow-up watcher waits for the capture, the captured deadline survives a config cut across retry, stall-recovery and cancel-recovery prompts, a fresh run captures the lowered value, the dedicated path publishes its cleanup identity before the budget capture, and a legacy provider keeps stream(message).
  • The 24 new or changed tests pass in 5 of 5 serial runs.
  • Mutation proofs on an archive copy, each anchor matched once and the file restored afterwards: a capture that re-reads the live default fails 3 tests, a reaper that enforces the manager default fails 2, a reaper that measures a run that is not executing on the live default fails 2, a reaper that skips such a run fails 1, a transport budget that ignores the captured deadline fails 5 (2 of them in test_prompt_timeout_follows_ceiling.py), and a run that stops forwarding the budget fails 4.
  • The owning set (245 files: the changed modules plus test_subagent*, test_acp*, test_config*, test_spawn_*, test_providers*, test_dashboard_config*, test_conductor_board_contract_parity, every suite that drives the reaper or reads _EDITABLE_CONFIG, and the session-health, source-pin-budget and flake-ratchet suites) passes 12,432 with 11 skipped and 0 failed. The flake ratchet, source-pin budget and config baseline also pass 117/117 serially with FLAKE_RATCHET_BASE=origin/main.
  • The broad selection (587 test files: the owning set plus every test file importing a changed module) passes 34,939 with 76 skipped and 13 failed. 11 of the failures fail identically on git archive copies of main and of this candidate: 10 TestDoctor* in test_cli.py and one in test_model_served_default.py. The other 2 pass on both archive copies. test_dashboard_server_composition_contract.py depends on the built SPA this checkout links at src/kiro_crew/static/dist, and main's archive fails it too once that link is added. The vault-owned KAS row in test_cli_doctor.py passes alone and as a whole file in the same checkout.
  • Black (baselined gate), isort, flake8, mypy on the 11 changed modules, the agent-SDK boundary, subprocess encoding, comment history and brand name (both diff-scoped against main), harness parity, docs lint and git diff --check pass.
  • Not run on this candidate: the full backend suite, the frontend suites and Electron TAP. The diff touches no file under website/.

Manual verification

N/A: the captured deadline, the reaper's clock, the follow-up watcher, transport forwarding on shared and dedicated providers, and legacy provider compatibility are regression-tested above, each with a mutation that fails by name.

Related Issues

N/A: this change was proposed directly in the pull request.

Pattern harvest

Not generalizable: the run's timeout was read from the manager default at each point of use instead of once per run, a lifecycle choice specific to this manager that no line pattern separates from a deliberate live read.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

N/A — placeholder wording remains pending OSPO; this PR does not invent CLA text.

@rubencu
rubencu requested review from a team and pepmach August 31, 2026 03:28
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 31, 2026
@rubencu

rubencu commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author
  • Reaper diagnostic may display the mutable manager default — rebutted as disproportionate to this PR.

The advisory holds only for the diagnostic number; enforcement, persistence, and captured deadlines remain correct. The owning upstream module currently contains an unbaselined backend import, so any edit fails the shrink-only SDK-boundary gate. Correcting the string therefore requires a separate process-supervisor boundary migration rather than widening this timeout-policy PR.

@rubencu

rubencu commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author
  • Concurrent user stop could be learned as a timeout — fixed in 6fb9c2b3b.

The reaper now observes timeout usage only after execution started and only when user_stopped remains false after force-reap returns. Parameterized one-sweep coverage locks normal timeout growth, concurrent-stop neutrality, and pre-execution neutrality; the focused verifier returned PASS.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 31, 2026
@bolichen97 bolichen97 mentioned this pull request Sep 4, 2026
5 tasks done
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #5277 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5277: MERGE_DISCUSSION. Independent goals in the same constructor; sequence them rather than choose between them. Files: src/kiro_crew/subagent.py. The two independent directions used different labels; the matrix conservatively retains OVERLAPPING for coordination.
  • PR #6713 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #6713: KEEP. Different user goals with no behavioral collision; only ordinary rebase friction in run.py and subagent.md. Files: src/kiro_crew/subagent_manager/run.py, docs/system-specs/modules/subagent.md.
  • This PR is OVERLAPPING with PR #7304. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7195: REBASE. 7304 merged at 2026-09-02T03:57:34Z and the merge conflict label was applied to 7195 twenty seconds later at 2026-09-02T03:57:54Z. This is a pure relocation conflict in one file, not a behavior conflict. Files: src/kiro_crew/config/loader.py.
  • This PR is OVERLAPPING with PR #8003. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7195: REBASE. Different user goals — an enforced adaptive deadline versus credit-usage reporting — that complement each other. Only textual conflict in shared hunks; whichever merges second rebases. Files: src/kiro_crew/subagent.py, src/kiro_crew/slack/gateway.py, docs/system-specs/modules/subagent.md.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@dwu96 dwu96 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 7, 2026
@NicholasRBowers NicholasRBowers added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 7, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: The maintainer audit already mapped the rebase file-by-file (pure relocation conflict in config/loader.py from #7304, textual hunks in subagent.py/slack/gateway.py/docs from #8003, judged non-behavioral). The stale-base CI reds are in files this diff never touches and clear on rebase; the fresh head then lets the skipped review lanes stamp. Prior GPT/Opus findings are already dispositioned.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@bolichen97

Copy link
Copy Markdown
Contributor

Hi @rubencu, thanks for this one. Two things before it can land.

Still worth doing, but the numbers need reworking. The motivation, a fixed 30-minute deadline killing healthy long runs, is already handled on main by a different mechanism: merged #8891 raised the static default in src/kiro_crew/constants.py to SUBAGENT_TIMEOUT_SECS = 10800 with a 60..86400 load-time clamp. The adaptive part is genuinely absent from main: there is no src/kiro_crew/subagent_timeout.py, no subagent_timeout_auto or subagent_timeout_max_secs, and both enforcement sites still read the single mutable manager value (src/kiro_crew/subagent_manager/run.py, src/kiro_crew/subagent_manager/monitoring.py). So the learned per-run deadline is the remaining scope, and it is not a rebase-only change. As written it would regress: your 1800s floor and 7200s learning ceiling both sit below main's 10800s default. Please rebase on current main, derive the floor from constants.SUBAGENT_TIMEOUT_SECS and put the ceiling above it, and move the config hunks from src/kiro_crew/config/loader.py to src/kiro_crew/config/sections.py plus constants.py, where AgentConfig now lives. One more thing to declare: main documents agent.subagent_timeout_secs as deliberately not dashboard-settable in docs/architecture/resource-protection.md, and your _EDITABLE_CONFIG change in src/kiro_crew/dashboard/handlers/core.py makes both new keys editable.

Overlap with #8003 (credit usage reporting, cc @welikoiwanenko). Shared files: src/kiro_crew/subagent.py (both add fields to SubagentInfo), src/kiro_crew/subagent_manager/run.py, src/kiro_crew/subagent_manager/monitoring.py, src/kiro_crew/slack/gateway.py, and the same run-lifecycle bullet in docs/system-specs/modules/subagent.md. The goals do not collide, the conflict is textual. #8003 is already based on current main, so let it land first and rebase this PR on top.

Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong.

@rubencu
rubencu force-pushed the feat/adaptive-subagent-timeouts branch from 6fb9c2b to 740c27c Compare September 13, 2026 19:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 13, 2026
@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 6d633a1709c9f1b397257b0a13cb28a7e13d71eb via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check one thing before merge: a run that is not executing still uses its registration age against the live default. A long spawn-approval park can still be reaped before the run starts, same as on main.

[DESIGN-REVIEWED] 6d633a1

Description read

[DESCRIPTION-READ] 0bb20212ac9dca6d98485308f1566f7d52b0dd47e8c599ae6990d4beed059a03

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 6d633a1709c9f1b397257b0a13cb28a7e13d71eb via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The provider capability seam guards a "legacy duck provider" that does not exist in src/: both real providers return the same value, so the None branch serves only test doubles.

Not justified as shipped

  • Item 4 — one consumer, generalized: LLMProvider.prompt_timeout_for_deadline (default None), agent_sdk.capture_prompt_timeout (reads it from the type) and run.py's if prompt_timeout is None branch are there so that "an undeclared provider keeps its one-argument stream(message) call". But src/ has exactly 2 LLMProvider subclasses (grep class \w+\(LLMProvider\)): AcpProvider and AcpSessionProvider. Both override the method with the same return resolve_prompt_timeout_for_deadline(deadline). The None path has 0 production consumers.
    Subtraction: delete prompt_timeout_for_deadline (base and both overrides), capture_prompt_timeout and its exports, and the two-way stream call. In _run_inner, call resolve_prompt_timeout_for_deadline(info.timeout_secs) once and always pass timeout=, since the base stream signature now accepts it.
    Clears when: the seam is removed, or a provider in src/ is named that subclasses LLMProvider without the override.
  • Item 2 — rides along: this changes the reaper's clock, which the Goal does not cover. The Goal is about config reloads; the spawn-approval wait counting against the deadline is a separate defect.

What this change ships

Inventory (8 items) — 5 justified

Intent: keep a running subagent's deadline and transport budget fixed when config reloads mid-run. FIX.

  1. A running subagent keeps the deadline it started with when config reloads — justified
  2. The reaper times an executing run from execution start, not registration — rides along, separate defect
  3. The follow-up watcher waits for the run's deadline capture, then times itself on it — justified
  4. Retry, fallback and recovery prompts reuse one transport budget set when the run starts — one consumer, generalized
  5. A deadline exactly at the 4-hour floor gets a 60-second transport margin — justified
  6. Timeout and reap messages name the deadline the run actually enforced — justified
  7. A reload during a cancel-recovery capacity wait keeps the captured deadline — justified
  8. Test fixtures that close managers, plus black reformatting and baseline removal — rides along, harmless

[FIRST-PRINCIPLES-REVIEWED] 6d633a1

Description read

[DESCRIPTION-READ] 562aaadad3f6e0e0fb01055b368ef3edb8b7d41228e6b3bdfb0bd718dc5906e6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 19, 2026
@rubencu
rubencu force-pushed the feat/adaptive-subagent-timeouts branch from 2f15a75 to a04b834 Compare September 19, 2026 10:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
@rubencu

rubencu commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

Maintainer rerun handoff for current head a04b834d4dcddf5ab5e97b34529ba6ff9424d798:

  • All substantive CI/build/test checks have settled successfully (A0/P0/F3/pass89).
  • GPT 5.6 Review job 105880533854 and Opus 4.8 Review job 105880523053 failed without executing any steps, produced zero annotations, and posted no current-head markers or findings.
  • PR Readiness is the resulting cascade.
  • Independent exact-SHA local GPT 5.6 Sol and Opus 4.8 reviews both returned no findings.

Please rerun the failed GPT and Opus reviewer checks for this exact head. No code change is indicated by the available evidence.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 20, 2026

Copy link
Copy Markdown
Contributor

Open-PR triage (2026-10-05): DISTINCT, needs a rebase

What was checked: head a04b834d4d, merge base 1a8b9f7750, and origin/main af1eb612da. I read the full merge-base diff (33 files, +1942/−69).

Verdict: DISTINCT. There is no breaking change and I found no P0 bug.

Not on main yet. None of the adaptive learner exists on main:

  • no subagent_timeout.py, subagent_timeout_auto, subagent_timeout_max_secs, capture_prompt_timeout or SubagentInfo.timeout_secs;
  • subagent_manager/run.py:791 and monitoring.py:1055 still enforce the single mutable _default_timeout.

PR #8891 (merged) only raised the static default to 10800s. This PR already builds on that: the floor is the configured or 3h value and the default ceiling is 21600s. The 2026-09-08 audit points are addressed: config is in sections.py/constants.py, and the doc "known gap" for dashboard editing is removed.

Other open PRs: the file overlaps are textual only. #12551, #11794, #10171 and #13320/#14861 share hunks but have different goals. No open PR implements adaptive or captured subagent deadlines.

Compatibility:

  • The feature is default-off.
  • LLMProvider.stream gains an optional timeout= argument. Callers pass it only when the provider type declares prompt_timeout_for_deadline, so legacy providers keep stream(message).
  • New dashboard-editable keys use the same load-time bounds.

Blocker: the PR is 2279 commits behind main (mergeable_state=dirty). git merge-tree reports conflicts in 11 files:

  • subagent.py, subagent_manager/run.py
  • providers/base.py, providers/acp.py, acp/session_provider.py
  • test/test_subagent_turn_resilience.py
  • four docs/system-specs/architecture docs, plus src/kiro_crew/docs/subagents.md

Recommendation: REBASE onto current main, then KEEP. Main's run.py timeout block has moved to around line 791, so re-anchor the captured-deadline change there. The open "needs-a-decision" (production-usage evidence before shipping) is still a maintainer call. Minor, non-blocking: in the run.py timeout message, _observe_timeout_usage returns the visible default, so "future runs use N minutes" is usually omitted until the detached write publishes.

@rubencu
rubencu force-pushed the feat/adaptive-subagent-timeouts branch from a04b834 to 31e8ae1 Compare October 5, 2026 22:37
@rubencu
rubencu requested a review from a team as a code owner October 5, 2026 22:37
@rubencu
rubencu requested a review from CrysisDeu October 5, 2026 22:37
@rubencu

rubencu commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Rebased as asked in the 2026-10-05 triage (#issuecomment-5993023490). New head 31e8ae1514f9, one commit directly on main 0ec74ff5d8f3, same 33 files plus subagent_manager/terminal.py (see below), commit message unchanged.

The anchors from the triage, as landed

  • subagent_manager/run.py: the deadline is captured once at the moved timeout block (~L791) and survives the same-record cancel respawn; the success observation runs after main's reap arm; timeout learning sits inside main's not reaped and not done guard.
  • LLMProvider.stream(message, timeout=None, *, allow_image=True): the optional budget takes the shape main already uses for AcpSessionHandle.prompt and AcpClient.stream_events; the ACP providers chain functools.partial(timeout=...) after main's allow_image partial (and _claim_shared_turn on the session provider). An undeclared legacy provider keeps the exact stream(message) call.
  • Default-off, same load-time bounds for the three keys; main's startup_timeout override and _startup_deadline property are kept.
  • Docs: main's #10521 stall wording and the startup-watchdog row are kept; the PR's rows are added beside them.

Local review before this push (CI's own review contracts, run on the exact head each round; Opus 5.5 = the CI pin; the GPT lane ran as 5.6 because the pinned 6.1 is not available locally, so CI's GPT verdict is the authoritative one):

  • Round 1 (rebase): no blocking finding. Two Opus advisories fixed before publishing: (a) the new capture_prompt_timeout await had landed between session acquisition and the dedicated path's cleanup-identity publish, so a cancel there would have left that session's provider files unreclaimable; moved below the identity block, with a test. (b) _force_reap's record printed the manager default where the reaper now enforces the per-run deadline; it now prints info.timeout_secs or _default_timeout (this is why terminal.py joins the diff), with a test.
  • Rounds 2-3: all four test modules that build SubagentManager inline now opt into the repo's close_subagent_managers fixture (no-test-side-effects).
  • Rounds 4-5: three rationale sentences (ACP spec, resolver docstring, a test docstring, plus the _TIMEOUT_SECS row) still described a pre-persistence in-memory raise; they now describe the actual persist-then-publish order.
  • Final pair on the reviewed head: no findings from either lane. The publish is a patch-identical replay of that head onto today's main (patch-id ab5bbceb267e, 428/428 on the PR's test modules).

Left as they are, deliberately

  • Triage nit on the run.py timeout message ("future runs use N minutes" appears only once a higher level is already saved): that is the pinned behaviour the PR's tests assert (publish only after the write is read back); changing it is the author's call, not a rebase's.
  • Local from kiro_crew.agent_sdk import capture_prompt_timeout in _run_inner_impl: that body already carries main's local imports under the "runs on kiro_crew.subagent's globals" comment; the PR follows the base pattern.
  • Below-bar advisories from the local lanes, listed so they are not lost; none blocks: a live reconfigure that lowers the enabled ceiling and then raises it keeps the clamped level until restart instead of re-reading the persisted one (apply_limits; a design call: re-adopt vs re-learn); approval wait time inside a run still counts toward a learned raise (_maybe_flag_stall exempts it, _observe_timeout_usage does not); HARD_WEDGE_CEILING_SECS is still derived from the 3 h constant (safe today because spawn_sub_agents caps itself at 7200 s); the cancel_all learned-timeout drain has no wait_for bound; _observe_timeout_usage(completed=False) does not compare elapsed against the deadline; _read_timeout_state_bytes could set fd = -1 right after os.fdopen; the approval_parked/release_parked arms in _force_reap are no longer reachable from the deadline reap.
  • Whether production-usage evidence is needed before shipping stays the maintainers' decision, as the triage says.

PR body: the frozen Problem / Why sections are unchanged; the template's **Goal:** line and ## Not a goal section were added from statements already in them (the description check was red without those fields), and everything from "What changed" down was rewritten to this diff.

@rubencu

rubencu commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

New head 9d81b8f2dab4, one commit on main 97fdb002dfb1 (2026-10-09), 27 files, +802/-46. Title: fix(subagent): capture each run's deadline at start and forward it to the ACP transport.

What the reviews on 31e8ae15 said

  • First Principles (BLOCK): the learner raised one shared _default_timeout that every later run captured, wedged or not, which is the harm the description gave for not raising the configured timeout. No linked report shows a healthy run hitting three hours. Items 2 and 3 (the ceiling key, the persisted level) existed only to serve it, item 7 (the adaptive branch of resolve_prompt_timeout) duplicated resolve_prompt_timeout_for_deadline, and item 5 (the reaper's clock) was undeclared.
  • Design (CONCERNS): any timeout with stream activity trained the level, so a looping run passed the test, and nothing lowered the level again. The reaper change contradicted the stated non-goal.

The learner is removed, not defended. Gone: subagent_timeout.py and its 37 tests, agent.subagent_timeout_auto, agent.subagent_timeout_max_secs with their constants, baseline rows, loader parsing and dashboard editors, the persisted member-memory-bindings/subagent-timeout.json record with its write, read-back and shutdown drain, and the adaptive branch of resolve_prompt_timeout(), which is now byte-identical to main. The diff touches no file under security/ and no sandbox*.py. The dashboard editor for agent.subagent_timeout_secs (item 4, "justified, keep") stays, with the loader's clamp constants as its range and a row in the write-gate parity ratchet.

What remains, and why each piece holds without a learner

  • Captured deadline (items 6, 9, 10): SubagentInfo.timeout_secs is set when execution starts, so a reload changes the next run, not the reaper, follow-up watcher or recovery respawn of one in flight, and the timeout and reap records name the deadline that was enforced.
  • Transport budget (item 8): each run resolves one ACP prompt budget, at least 60 seconds past its captured deadline, and reuses it for every retry and recovery prompt, so a lowering reload cannot cut a later prompt below the run's own deadline.
  • Legacy providers: an undeclared provider keeps its exact stream(message) call.
  • Reaper clock (item 5, Design Watch 2): an executing run is measured on its captured deadline from _exec_started. A run that has not started executing (parked at approval, or approved and waiting for admission) keeps main's bound, the registration age against the live default, instead of being skipped as the earlier revision did: a skipped start that waits for admission would have had no wall-clock bound at all. The reaper test covers both: a start parked for 4,000 s against a 60 s default is reaped, one parked for 30 s is not, and the skip as a mutation fails it by name.

Description: the Problem and Why sections are rewritten as well as Goal and Not a goal, because the earlier text described the learner and no longer matched the change; this comment is the record of that edit.

Rebase: onto 97fdb002dfb1. Conflicts in docs/architecture/resource-protection.md, docs/system-specs/modules/config.md, docs/system-specs/modules/subagent.md, src/kiro_crew/config/loader.py and src/kiro_crew/docs/subagents.md, each resolved to main's text, with only the captured-deadline and transport-budget sentences added back. Every loader hunk was learner-only, and main already carries the 0-sentinel clamp.

Tests: the eight changed test modules pass 435/435. Five mutations on an archive copy fail by name: a capture that re-reads the live default (2), a reaper on the manager default (1), a reaper that skips parked starts (1), a transport budget that ignores the captured deadline (5), a run that stops forwarding it (4). The owning set (214 files) passes 10,935 with 0 failed. The broad selection (1,144 files) passes 71,392 with 23 failed, and all 23 also fail without this commit: 16 on a git archive copy of main, 7 in this checkout detached at the base.

@bolichen97: rebased onto current main as your triage asked, and the learner it described as absent from main is now absent from this PR too.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 6d633a1709c9f1b397257b0a13cb28a7e13d71eb touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@iamwhatever

Copy link
Copy Markdown
Contributor

This PR fixes the bug where a config reload changes the deadline of a subagent run that is already executing. Each run now captures agent.subagent_timeout_secs on SubagentInfo.timeout_secs when it starts, and a captured transport budget is passed to the ACP stream. I read the run, monitoring and provider diff. The capture and reuse look correct, and legacy providers keep the one-argument stream(message) call. bolichen97's earlier requests are addressed: the learner was removed and the floor no longer sits below main's 10800s default. I am holding for two reasons. First, src/kiro_crew/dashboard/handlers/core.py adds agent.subagent_timeout_secs to _EDITABLE_CONFIG. That reverses the documented resource-protection decision that this key is not settable from the dashboard, and the PR edits docs/architecture/resource-protection.md to match. That is a protection-policy change a maintainer must accept explicitly. Second, the current head 485584c was last reviewed on 2026-09-13. Design and First Principles are CONCERNS on that head, and the author's dispositions name older heads (5427ea4 and earlier). Please rebase onto current main, rerun the lanes, and answer each current-head item. If the dashboard-editable key is not essential to the deadline fix, splitting it out would let the fix land on its own. — crew-pr-reviewer

@rubencu

rubencu commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

New head 0be9319eadcc, one commit on main d3f462f23b4c (2026-10-10), 24 files, +894/-34. The rebase replayed with no conflicts.

Hold 1, the dashboard key: removed from this PR. src/kiro_crew/dashboard/handlers/core.py, docs/architecture/resource-protection.md and test/test_config_load_bounds_parity.py are byte-identical to main, so known gap 1 reads as main documents it. The parity row pinned the _EDITABLE_CONFIG write gate, not the load-time clamp (_subagent_timeout_from, src/kiro_crew/config/loader.py:2777, untouched), so it left with the editor. Following the hold, ## Not a goal gains a line naming the key as a resource-protection decision for a maintainer, and its first bullet drops "now also editable from the dashboard". This comment records that edit.

Hold 2, the current-head items on 485584c89fc4:

  • Design, recovery gap: fixed. src/kiro_crew/subagent_manager/monitoring.py:1083-1098 measures a run with no _exec_started on its registration age against info.timeout_secs or _default_timeout. A run between the two attempts of a context-overflow recovery keeps its captured deadline, and a run parked before its first start, which has captured nothing, keeps the live default. test_reload_during_the_recovery_capacity_wait_keeps_the_captured_deadline drives the real recovery into its capacity wait, cuts the default to 60 s and runs one real reaper sweep: no reap, and the replacement then runs on the captured 21,600 s. The one-sweep reaper test gains a row on each side of a captured deadline. Restoring the live default in that branch fails both tests by name.
  • Design, dashboard claim: accepted. The editor row is gone (hold 1) and the description no longer claims dashboard editing.
  • First Principles item 7 (editor row outside the Goal): accepted, removed with its imports and parity row.
  • First Principles item 8 (known gap deleted): void, the gap is main's text again.
  • First Principles item 5 (one consumer, generalized): kept, on two facts. src/kiro_crew/subagent_manager/run.py is outside the agent-SDK boundary and absent from .github/agent-sdk-boundary-baseline.txt, so it cannot import resolve_prompt_timeout_for_deadline from kiro_crew.acp.client, and the budget reaches it through capture_prompt_timeout (src/kiro_crew/agent_sdk/capabilities.py:258). Always passing timeout=, the proposed subtraction, fails 65 tests in 8 files on an archive copy of this head, each with unexpected keyword argument 'timeout': their provider doubles take no timeout, like main's LLMProvider.stream. The type-declared capability keeps their call unchanged without a signature probe, the property the hold confirmed ("legacy providers keep the one-argument stream(message) call").
  • First Principles items 1 to 4 and 6 were judged justified and are unchanged.

Tests: the seven PR test modules pass 392/392, the 24 new or changed tests pass in 5 of 5 serial runs, and six mutations fail by name. The owning set (245 files) passes 12,432 with 0 failed. Of 13 failures in the broad selection (587 files), 11 fail identically on a git archive copy of main, and the other 2 pass on both archive copies (the description names them).

… the ACP transport

A subagent run fixes its deadline on SubagentInfo.timeout_secs when
execution starts, and asyncio.wait_for, the reaper and the follow-up
watcher all enforce that one value, so a config reload applies to the
next run only. The reaper measures an executing run from
_exec_started. A run that is not executing is measured on its
registration age: against the live default while it is parked before
its first start, and against its captured deadline between the two
attempts of a cancel recovery.

At execution start the run also resolves one ACP prompt budget at
least 60s past that deadline and forwards it on every original, retry,
fallback and recovery prompt through the providers that declare
prompt_timeout_for_deadline. A provider that declares nothing keeps
its one-argument stream(message) call.
@rubencu

rubencu commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main at d8430e9156 (0be9319ead -> 6d633a1709), same single commit. One merge with #18992 in subagent_manager/monitoring.py: main moved the reaper's durations to the monotonic clock, so this change's captured-deadline block now reads _exec_started_mono and _started_mono against mono_now instead of the wall-clock twins, and its reaper test registers each run's age on both clocks. This change's seven test files plus main's test_continuable_followups.py, test_subagent_coverage.py, test_subagent_startup_pressure.py and test_subagent_turn_resilience.py pass: 853 tests. black (test_session_sharing.py is on main's baseline), isort, flake8, docs-lint, comment-history gate and git diff --check pass.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stops a config reload from changing the deadline of a subagent run that is already executing. _run_impl captures agent.subagent_timeout_secs once on SubagentInfo.timeout_secs, and wait_for, the reaper, the follow-up watcher and both error messages all read that captured value. Both blockers from my last comment are resolved on 6d633a1. The _EDITABLE_CONFIG and resource-protection changes are gone from the diff, and the branch is rebased with fresh lanes on this head. I read the run, monitoring, continuation, terminal and provider hunks. The reaper now times an executing run from _exec_started_mono, matching the wait_for clock, and keeps the registration age for a parked run. The watcher's new wait for the capture is bounded, because the reaper ends a parked run. Providers that do not declare prompt_timeout_for_deadline keep the one-argument stream(message) call, and both ACP stream_events/prompt signatures already accept timeout. The tests cover a lowered config across retries and recovery, the captured deadline from execution start, and the legacy one-argument provider. First Principles' remark that the provider seam has no production legacy consumer is advisory and unanswered. It does not affect correctness. — crew-pr-reviewer (deep re-check)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants