Repository navigation
fix(subagent): leave startup on the first stream event, not at stream open - #14020
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify every waker that resolves [DESIGN-REVIEWED] 7c3b94a |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify that no selectable backend can reach the first-answer edge with a slow first token: for a PID-less provider the startup deadline is now a first-output budget, and a healthy start that opens its stream but thinks past that deadline is reaped where stream-open once saved it. The diff's own mitigation is checked and holds in-repo — every What this change shipsInventory (10 items) — 10 justifiedIntent: FIX — a subagent run must leave startup only when its own session has answered its prompt, and the release path must record only what actually happened.
[FIRST-PRINCIPLES-REVIEWED] 7c3b94a |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
Intent: A sub-agent leaves startup only on evidence that its own session answered its prompt, so the startup watchdog and the in-startup bound cannot be satisfied by an opened stream or by a co-tenant's traffic, and the approval-release path writes truthful spawn records. |
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
8e8729f to
651cf7a
Compare
|
Fan-out measurement behind the "Manual verification" table (120 items through the real {"config": {"n": 120, "cap": 12, "gate_width": 2, "startup_cap": 4, "session_new_secs": 5.0, "jitter_secs": 2.0, "run_body_secs": 45.0}, "elapsed_secs": 538.3, "success": 120, "reaped_by_startup_watchdog": 0, "never_started": 0, "max_in_startup_observed": 4, "spawn_queue_wait_secs": {"p50": 249.19, "p90": 435.11, "p99": 480.42, "max": 484.47}, "gate_wait_secs": {"p50": 0.0, "p90": 2.95, "p99": 6.16, "max": 6.16}, "time_to_pid_from_spawn_secs": {"p50": 252.88, "p99": 487.11}, "end_to_end_secs": {"p50": 297.88, "p99": 532.11, "max": 536.55}, "mean_gate_wait_secs": 0.91}
{"config": {"n": 120, "cap": 32, "gate_width": 2, "startup_cap": 4, "session_new_secs": 5.0, "jitter_secs": 2.0, "run_body_secs": 45.0}, "elapsed_secs": 351.7, "success": 120, "reaped_by_startup_watchdog": 0, "never_started": 0, "max_in_startup_observed": 4, "spawn_queue_wait_secs": {"p50": 150.28, "p90": 269.98, "p99": 294.39, "max": 296.0}, "gate_wait_secs": {"p50": 5.06, "p90": 6.59, "p99": 6.97, "max": 7.0}, "time_to_pid_from_spawn_secs": {"p50": 161.45, "p99": 302.75}, "end_to_end_secs": {"p50": 206.45, "p99": 347.75, "max": 350.73}, "mean_gate_wait_secs": 5.01}
{"config": {"n": 120, "cap": 32, "gate_width": 2, "startup_cap": 4, "session_new_secs": 20.0, "jitter_secs": 5.0, "run_body_secs": 20.0}, "elapsed_secs": 1242.4, "success": 120, "reaped_by_startup_watchdog": 0, "never_started": 0, "max_in_startup_observed": 4, "spawn_queue_wait_secs": {"p50": 595.77, "p90": 1068.31, "p99": 1181.58, "max": 1184.55}, "gate_wait_secs": {"p50": 20.48, "p90": 24.19, "p99": 24.97, "max": 25.0}, "time_to_pid_from_spawn_secs": {"p50": 639.39, "p99": 1218.06}, "end_to_end_secs": {"p50": 659.39, "p99": 1238.06, "max": 1241.3}, "mean_gate_wait_secs": 20.02}Reading: zero startup reaps at any queue depth (the clock never runs before admission and is frozen at the gate); the in-startup population never exceeds the bound; the tail is the running cap when the cap is small and the gate ( Not measured here, and not measurable offline: the real distribution of |
|
Fan-out measurement behind the body's "Manual verification" table — 120 items through this branch's real admission path, with a modelled provider. Supersedes the earlier measurement comment on this PR, which used a plain semaphore for the gate and no cold-start admission. Real code: Modelled: Reading:
Per-scenario summaries ( {"scenario": "s1-healthy", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:2,5", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 30.0, "session_new": 90.0}, "completed": 120, "success_rate": 1.0, "reaped_startup_timeout": 0, "reap_stages": {}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 339.8, "p90": 597.1, "p99": 649.1, "max": 651.7}, "spawn_to_session_ready_s": {"p50": 367.7, "p90": 611.8, "max": 720.0}, "makespan_s": 764.9}
{"scenario": "s2-healthy-stagger2", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 2.0, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:2,5", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 30.0, "session_new": 90.0}, "completed": 120, "success_rate": 1.0, "reaped_startup_timeout": 0, "reap_stages": {}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 322.8, "p90": 584.4, "p99": 636.0, "max": 642.6}, "spawn_to_session_ready_s": {"p50": 348.4, "p90": 598.6, "max": 667.3}, "makespan_s": 742.1}
{"scenario": "s3b-throttled-init", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:30,85", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 90.0, "session_new": 90.0}, "completed": 117, "success_rate": 0.975, "reaped_startup_timeout": 3, "reap_stages": {"spawn+initialize": 3}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 1626.9, "p90": 3027.8, "p99": 3345.1, "max": 3370.3}, "spawn_to_session_ready_s": {"p50": 1744.7, "p90": 3168.0, "max": 3501.0}, "makespan_s": 3586.2}
{"scenario": "s4-throttled-init-G3", "n": 120, "cap": 64, "gate_G": 3, "in_startup_bound": 6, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:30,85", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 90.0, "session_new": 90.0}, "completed": 68, "success_rate": 0.567, "reaped_startup_timeout": 52, "reap_stages": {"spawn+initialize": 52}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 1348.5, "p90": 2488.7, "p99": 2788.8, "max": 2789.1}, "spawn_to_session_ready_s": {"p50": 1459.3, "p90": 2616.3, "max": 2965.5}, "makespan_s": 3011.3}
{"scenario": "s4b-throttled-init-G3", "n": 120, "cap": 64, "gate_G": 3, "in_startup_bound": 6, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:30,85", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 90.0, "session_new": 90.0}, "completed": 73, "success_rate": 0.608, "reaped_startup_timeout": 47, "reap_stages": {"spawn+initialize": 47}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 1348.5, "p90": 2496.3, "p99": 2792.8, "max": 2825.2}, "spawn_to_session_ready_s": {"p50": 1523.1, "p90": 2720.3, "max": 2970.7}, "makespan_s": 3058.0}
{"scenario": "s5-large-mcp-fleet", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:2,5", "session_new": "uniform:100,160", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 30.0, "session_new": 300.0}, "completed": 56, "success_rate": 0.467, "reaped_startup_timeout": 64, "reap_stages": {"session/new with permit": 64}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 3508.8, "p90": 6269.4, "p99": 6989.6, "max": 6989.9}, "spawn_to_session_ready_s": {"p50": 2766.8, "p90": 6491.1, "max": 7221.6}, "makespan_s": 7295.5}Harness (stdlib + the repo's own modules; run from the worktree)#!/usr/bin/env python3
"""Wide sub-agent fan-out through the REAL SubagentManager admission path.
Real code exercised (this worktree's src/):
* SubagentManager.spawn -> admission pump (stagger, running cap, the in-startup
bound 2G) -> _run -> _run_inner, with a real durable task store;
* the real per-loop SessionStartGate (acp.runtime.session_start_gate) and the
real _gate_wait_mark / _gate_exit_reset clock callbacks;
* the real per-loop cold-start admission (acp.runtime._cold_start_admission,
limit _COLD_START_MAX_CONCURRENT);
* the real startup watchdog: _is_startup_stalled + _force_reap(reason=
"startup_timeout"), evaluated on a sweep at the production interval.
Modeled: the provider. get_or_create reproduces the DEDICATED path's permit
sequence (session_allocation._start_sem -> AcpRuntime.spawn's cold-start
admission -> spawn + initialize -> SessionStartGate -> session/new), with each
stage's own budget; a PID is published once get_or_create returns, as
AcpSessionProvider._pid does. stream() waits the time-to-first-token, emits one
chunk, waits the rest of the turn and ends it.
Every duration -- latencies, stagger, startup deadline, reaper interval, stage
budgets -- is multiplied by SCALE, so a 120-item wave runs in minutes; the
admission logic only compares these against each other. Reported numbers are in
PRODUCTION seconds (divided back by SCALE).
"""
from __future__ import annotations
import argparse
import asyncio
import json
import os
import random
import statistics
import sys
import tempfile
import time
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
WT = os.environ.get("KIROCREW_WORKTREE", os.getcwd()) # the worktree under test
sys.path.insert(0, os.path.join(WT, "src"))
sys.path.insert(0, os.path.join(WT, "test"))
_HOME = tempfile.mkdtemp(prefix="kc-fanout-home-")
os.environ["KIROCREW_HOME"] = _HOME
os.environ["KIROCREW_TELEMETRY"] = "0"
import kiro_crew.acp.runtime as rt_mod # noqa: E402
import kiro_crew.resource_status as resource_status # noqa: E402
import kiro_crew.subagent as subagent_mod # noqa: E402
from kiro_crew.providers.base import EVENT_COMPLETE, EVENT_TEXT_CHUNK # noqa: E402
from kiro_crew.subagent import SubagentManager # noqa: E402
def _pin_host_memory() -> None:
subagent_mod.check_memory_available = lambda min_gb=None, path=None: (True, 64.0)
subagent_mod.cached_admission_check = lambda: resource_status.AdmissionDecision(
admitted=True, posture=resource_status.POSTURE_AMPLE, available_gb=64.0
)
class Dist:
"""A latency distribution in production seconds."""
def __init__(self, spec: str, rng: random.Random) -> None:
self.spec = spec
self._rng = rng
kind, _, rest = spec.partition(":")
self.kind = kind
self.args = [float(x) for x in rest.split(",")] if rest else []
def draw(self) -> float:
a = self.args
if self.kind == "const":
return a[0]
if self.kind == "uniform":
return self._rng.uniform(a[0], a[1])
if self.kind == "lognormal": # median, sigma, lo, hi
import math
v = self._rng.lognormvariate(math.log(a[0]), a[1])
return min(max(v, a[2]), a[3])
raise ValueError(self.spec)
class Rec(SimpleNamespace):
pass
async def run(args: argparse.Namespace) -> dict:
_pin_host_memory()
rng = random.Random(args.seed)
S = args.scale
spawn_init = Dist(args.spawn_init, rng)
session_new = Dist(args.session_new, rng)
ttft = Dist(args.ttft, rng)
turn = Dist(args.turn, rng)
t0 = time.monotonic()
t_wall0 = time.time()
def now() -> float:
return (time.monotonic() - t0) / S # production seconds since the wave began
loop = asyncio.get_running_loop()
# The real gate, sized for this scenario (the manager's bound follows it).
rt_mod._session_start_gates[loop] = rt_mod.SessionStartGate(args.gate)
start_sem = asyncio.Semaphore(args.start_sem) # SessionManager._start_sem
recs: dict[str, Rec] = {}
pids: dict[str, int] = {}
next_pid = [90_000_000]
def rec_for(key: str) -> Rec:
if key not in recs:
recs[key] = Rec(key=key)
return recs[key]
def make_provider(key: str):
provider = MagicMock()
provider.start = AsyncMock()
provider.shutdown = AsyncMock()
provider.context_usage_pct = lambda: 0.0
provider.context_used_tokens = MagicMock(return_value=0)
provider.context_window_tokens = MagicMock(return_value=0)
provider.mcp_session_report = MagicMock(return_value=None)
provider.last_infra_error = None
provider.session_id = f"sess-{key}"
def _stream(msg, *_a, **_k):
async def _gen():
r = rec_for(key)
r.t_stream = now()
await asyncio.sleep(ttft.draw() * S)
r.t_first = now()
yield SimpleNamespace(kind=EVENT_TEXT_CHUNK, text="x", runtime_global=False)
await asyncio.sleep(turn.draw() * S)
r.t_done = now()
yield SimpleNamespace(
kind=EVENT_COMPLETE, stop_reason="end_turn", runtime_global=False
)
return _gen()
provider.stream = MagicMock(side_effect=_stream)
return provider
async def get_or_create(session_key, *_a, on_gate_acquired=None, on_gate_queued=None, **_kw):
r = rec_for(session_key)
r.t_enter = now()
async with start_sem:
r.t_start_sem = now()
adm = rt_mod._cold_start_admission()
await adm.acquire()
r.t_cold = now()
try:
d = spawn_init.draw()
r.spawn_init = d
if d > args.init_budget:
await asyncio.sleep(args.init_budget * S)
r.fail_stage = "initialize budget"
raise RuntimeError("initialize went unanswered (typed start failure)")
await asyncio.sleep(d * S)
finally:
adm.release()
r.t_spawned = now()
gate = await rt_mod.session_start_gate()
if on_gate_queued is not None:
on_gate_queued()
permit = await gate.acquire()
if on_gate_acquired is not None:
on_gate_acquired(permit.queue_wait_ms)
r.t_gate = now()
try:
d = session_new.draw()
r.session_new = d
if d > args.session_budget:
await asyncio.sleep(args.session_budget * S)
r.fail_stage = "session/new budget"
raise RuntimeError("session/new timed out (typed start failure)")
await asyncio.sleep(d * S)
finally:
permit.release()
r.t_session = now()
next_pid[0] += 1
pids[session_key] = next_pid[0]
return make_provider(session_key), True, False
sessions = MagicMock()
sessions.get_or_create = AsyncMock(side_effect=get_or_create)
sessions.get_pid = MagicMock(side_effect=lambda key: pids.get(key))
sessions.get_approval_policy = MagicMock(return_value="auto")
sessions.get_agent = MagicMock(return_value="")
sessions.get_agent_selection = MagicMock(return_value=("template", ""))
sessions.has_session = MagicMock(return_value=True)
sessions.release = MagicMock()
sessions.reset = AsyncMock()
sessions.record_success = MagicMock()
ctx = MagicMock()
ctx.hooks.auto_approve_subagent_spawn = True
ctx.hooks.auto_approve_subagent_tools = False
ctx.build_message = MagicMock(side_effect=lambda msg, *a, **k: (msg, None))
mgr = SubagentManager(
sessions=sessions,
ctx_builder=ctx,
max_concurrent=args.cap,
startup_timeout=120,
default_timeout=int(args.wall_clock),
)
mgr._startup_deadline = args.deadline * S
mgr._default_timeout = args.wall_clock * S
mgr._session_start_concurrency = args.gate
mgr._spawn_stagger_secs = args.stagger * S
mgr._should_use_session_sharing = MagicMock(return_value=False)
mgr._fire_event = AsyncMock()
await mgr.wait_taskq_ready()
reaped_at: dict[str, dict] = {}
stop = asyncio.Event()
async def sweep() -> None:
# The production watchdog body (monitoring._reaper_loop), on its interval.
phase = rng.uniform(0, args.reaper_interval) * S
await asyncio.sleep(phase)
while not stop.is_set():
t = time.time()
for agent_id, info in list(mgr._agents.items()):
if info.done:
continue
if mgr._is_startup_stalled(info, t):
key = f"subagent:{agent_id}"
r = rec_for(key)
stage = (
"queued for start_sem" if not hasattr(r, "t_start_sem") else
"queued for cold-start admission" if not hasattr(r, "t_cold") else
"spawn+initialize" if not hasattr(r, "t_spawned") else
"queued at SessionStartGate" if not hasattr(r, "t_gate") else
"session/new with permit" if not hasattr(r, "t_session") else
"after session/new"
)
reaped_at[agent_id] = {"stage": stage, "t": now()}
try:
await mgr._force_reap(
agent_id, info, t - (info._exec_started or t), reason="startup_timeout"
)
except Exception as exc: # noqa: BLE001
reaped_at[agent_id]["reap_error"] = repr(exc)
await asyncio.sleep(args.reaper_interval * S)
sweeper = asyncio.ensure_future(sweep())
spawned: list[tuple[str, float]] = []
infos = []
for i in range(args.n):
info = mgr.spawn(f"item-{i}")
infos.append(info)
spawned.append((info.id, now()))
t_spawn = {aid: t for aid, t in spawned}
deadline = time.monotonic() + args.max_wall
ids = [aid for aid, _ in spawned]
seen: dict[str, object] = {}
progress_at = time.monotonic()
while time.monotonic() < deadline:
for aid, info in list(mgr._agents.items()):
seen.setdefault(aid, info)
settled = sum(1 for aid in ids if aid in seen and seen[aid].done)
if settled >= args.n:
break
if time.monotonic() - progress_at > 30:
progress_at = time.monotonic()
print(
f"[progress t={now():.0f}s] settled={settled} seen={len(seen)} "
f"queue={len(mgr._queue)} running={mgr._running_count} "
f"in_startup={mgr._startup_population()}",
file=sys.stderr,
)
await asyncio.sleep(0.05)
mgr_seen = seen
stop.set()
sweeper.cancel()
await asyncio.gather(sweeper, return_exceptions=True)
rows = []
for aid, ts in spawned:
info = mgr_seen.get(aid) or mgr._agents.get(aid)
r = recs.get(f"subagent:{aid}", Rec())
rows.append(
{
"id": aid,
"t_spawn": ts,
"t_enter": getattr(r, "t_enter", None),
"t_start_sem": getattr(r, "t_start_sem", None),
"t_cold": getattr(r, "t_cold", None),
"t_spawned": getattr(r, "t_spawned", None),
"spawn_init": getattr(r, "spawn_init", None),
"session_new": getattr(r, "session_new", None),
"exec_started_rel": (
(info._exec_started - t_wall0) / S
if info is not None and getattr(info, "_exec_started", None)
else None
),
"t_gate": getattr(r, "t_gate", None),
"t_session": getattr(r, "t_session", None),
"t_first": getattr(r, "t_first", None),
"t_done": getattr(r, "t_done", None),
"done": bool(info and info.done),
"outcome": getattr(info, "outcome", "") if info else "missing",
"error": (getattr(info, "error", "") or "")[:160] if info else "",
"reaped": aid in reaped_at,
"reap_stage": reaped_at.get(aid, {}).get("stage", ""),
"fail_stage": getattr(r, "fail_stage", ""),
}
)
def pct(xs, p):
xs = sorted(xs)
if not xs:
return None
k = min(len(xs) - 1, max(0, int(round(p / 100 * (len(xs) - 1)))))
return round(xs[k], 1)
queue_wait = [r["t_enter"] - r["t_spawn"] for r in rows if r["t_enter"] is not None]
to_start = [r["t_session"] - r["t_spawn"] for r in rows if r["t_session"] is not None]
completed = [r for r in rows if r["t_done"] is not None and not r["reaped"]]
summary = {
"scenario": args.name,
"n": args.n,
"cap": args.cap,
"gate_G": args.gate,
"in_startup_bound": mgr._startup_cap(),
"cold_start_admission": rt_mod._COLD_START_MAX_CONCURRENT,
"start_sem": args.start_sem,
"stagger_s": args.stagger,
"startup_deadline_s": args.deadline,
"dists": {
"spawn_init": args.spawn_init,
"session_new": args.session_new,
"ttft": args.ttft,
"turn": args.turn,
},
"budgets": {"initialize": args.init_budget, "session_new": args.session_budget},
"completed": len(completed),
"success_rate": round(len(completed) / args.n, 3),
"reaped_startup_timeout": sum(r["reaped"] for r in rows),
"reap_stages": {
s: sum(1 for r in rows if r["reap_stage"] == s)
for s in sorted({r["reap_stage"] for r in rows if r["reap_stage"]})
},
"typed_start_failures": {
s: sum(1 for r in rows if r["fail_stage"] == s and not r["reaped"])
for s in sorted({r["fail_stage"] for r in rows if r["fail_stage"]})
},
"not_finished": sum(1 for r in rows if not r["done"]),
"queue_wait_s": {
"p50": pct(queue_wait, 50),
"p90": pct(queue_wait, 90),
"p99": pct(queue_wait, 99),
"max": round(max(queue_wait), 1) if queue_wait else None,
},
"spawn_to_session_ready_s": {
"p50": pct(to_start, 50),
"p90": pct(to_start, 90),
"max": round(max(to_start), 1) if to_start else None,
},
"makespan_s": round(max((r["t_done"] or 0) for r in rows), 1),
"wall_seconds": round(time.monotonic() - t0, 1),
}
if args.rows:
with open(args.rows, "w") as fh:
json.dump(rows, fh, indent=1)
await mgr.shutdown() if hasattr(mgr, "shutdown") else None
return summary
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--name", default="scenario")
ap.add_argument("--n", type=int, default=120)
ap.add_argument("--cap", type=int, default=64)
ap.add_argument("--gate", type=int, default=2)
ap.add_argument("--start-sem", type=int, default=4)
ap.add_argument("--stagger", type=float, default=0.25)
ap.add_argument("--deadline", type=float, default=120.0)
ap.add_argument("--reaper-interval", type=float, default=60.0)
ap.add_argument("--wall-clock", type=float, default=1800.0)
ap.add_argument("--init-budget", type=float, default=30.0)
ap.add_argument("--session-budget", type=float, default=90.0)
ap.add_argument("--spawn-init", default="uniform:2,5")
ap.add_argument("--session-new", default="lognormal:10,0.5,2,80")
ap.add_argument("--ttft", default="uniform:2,10")
ap.add_argument("--turn", default="uniform:30,90")
ap.add_argument("--scale", type=float, default=0.05)
ap.add_argument("--seed", type=int, default=1)
ap.add_argument("--max-wall", type=float, default=1800.0)
ap.add_argument("--rows", default="")
args = ap.parse_args()
summary = asyncio.run(run(args))
print(json.dumps(summary, indent=1))
if __name__ == "__main__":
main() |
|
self-added: yes
|
|
|
505ce68 to
d5c0e98
Compare
|
Adversarial review of head 1. P3 — two fanned-out event kinds still carry no provenance ( 2. P3 — two "red-first" claims are not true ( 3. P3 — one spec paragraph left unqualified ( Verified OK (so nobody re-checks): |
Independent adversarial review — 🟡 CONCERNS (no blocker)Reviewed head Five advisory findings, ordered by weight. None blocks the merge on its own, but 1–3 are real interleavings the new tests do not reach. 1. A reap during the admission wait can still lose the "waiting to be admitted" text; the new
|
a9e24b6 to
947e587
Compare
947e587 to
d2dd9db
Compare
|
Rebased: main moved in Reply to the macOS review, item by item, against head 1. Reap loses the "waiting to be admitted" text. Fixed. 2. 3. Compaction / clear / agent-switch without 4. Agent folder timing. Stated rather than changed. The body's compatibility section now says a start ended before admission leaves no agent folder (a persistent one leaves only its tombstone), which is the same record a declined prompt leaves, so it is not orphan-visible after a restart. The spec's release-path paragraph says the same. 5. Stale wording. Fixed at every site listed:
The "with a permit held" contradiction is gone from both Nits.
|
Review: leaving startup on the first owned frameVerdict: in favour. This fixes a real false negative that I hit repeatedly today, on this exact code path. My one substantive request is not about the design but about diagnosability — see point 2. Where these opinions come from, and what they are notA Windows opinion is worth little unless you know which host, which start path, and what went untested, so:
Why the platform matters rather than being incidental: this PR's own safety argument is that "every AcpRuntime-backed start records one [PID] before its stream opens", i.e. PID-backed starts leave startup on the PID and 1. The false negative this closes is one I measuredBefore this change, The third conjunct was satisfied early, so a run that opened a stream and then wedged forever could never be caught by the startup watchdog. That matches what I saw: in one wave nine agents each sat ~9000s with no startup reap whatsoever and needed manual stops; the ones that were eventually caught in later waves were caught by the much later idle-based detector at ~5400s. "The stream opened" never did imply "it answered" — which is your title — and on this host that gap was worth 90–150 minutes per affected agent. 2. The failure direction flips, and the premise behind it is conditionalAfter this change, a no-PID run's only way out of startup is a frame with The direction of failure has changed: before, a stall was invisible (user waits); after, a healthy run can be killed (user loses work). The second is the worse one to get wrong. And the premise that bounds the risk is conditional in the code: # run.py:2916 (shared-session path)
if runtime.pid:
info._pid = runtime.pid
# run.py:1271
pid = self._manager._sessions.get_pid(session_key)
if pid:
info._pid = pidBoth are This is not hypothetical on my host: I had agents legitimately working 90+ minutes on a single item (two hit the 180-minute cap wedged on file I/O). Under 27-way fan-out on one shared runtime, a misattributed frame turns exactly that population into false reaps. Concrete, cheap request (no semantic change): when a startup reap fires, record how many frames this start saw and how many were 3. The three-valued
|
A run without a runtime PID leaves the in-startup population (_in_startup, _is_startup_stalled, the adaptive controller's progress floor) when _first_stream_started is set. Stamping it when the provider's stream() was entered let a stream that opens and never yields read as progress: the watchdog stopped watching a start that was not starting, and _note_startup_progress woke the pump for it. SubagentManager._leave_startup now stamps the marker, once per execution, at the two moments the backend is shown to have answered the prompt: the first frame out of the stream that is addressed to this session -- taken on the raw frame, so a completion withheld for in-place recovery counts -- and a dependency verdict on the prompt, the step that also takes the durable row from starting to running. A runtime_global frame, which a shared runtime fans out to every co-tenant, ends neither the startup state nor the stream's running mark: both now key on one provenance test. The shared-runtime dispatch loop marks the compaction, clear and agent-switch notices and the steer echoes it builds from a fanned-out frame runtime_global, as it already did for the roster and the MCP notices, so the flag covers every notice kind such a frame yields; a session/update names its session by protocol and is routed. A start parked on a provider throttle before its first frame is therefore out of startup for the whole wait, which its dependency scope bounds. Every AcpRuntime-backed start records a runtime PID before its stream opens (get_pid on the dedicated path, runtime.pid at _bind_shared_handle), so for those the startup window closes at the PID; a test pins that ordering, and the spec names the backends whose first frame is model output. _spawn_with_approval records a run as spawned (_log_spawned: metrics, audit, agent folder) only once _admit_released_start has admitted it, so a start refused or ended while it waited for the in-startup bound is never counted as started. _admit_released_start answers one of three outcomes instead of a boolean: "admitted" (run), "admission_closed" (gateway admission closed at release -- the caller writes the rejection, releases the slot, audits rejected / admission_closed and announces), or "ended" (a user stop or a reap landed while it waited -- the caller writes nothing, because _force_reap owns that record: a stop is neutral and a reap names the interrupted wait). A stop or a reap is therefore never announced as "spawn rejected". A start ended while it waited for admission was never recorded, so no live-run state or state.json marks it incognito or temporary; _write_tombstone decides from the run's own memory mode, and a non-persistent run ended before admission or at a declined prompt leaves nothing on disk.
d2dd9db to
7c3b94a
Compare
|
Thanks. Replies against head 2. Diagnosable startup reap: done. The startup reap's error and the reaper's warning now name the On the misattribution you describe, where a frame that is really this session's arrives with PID wording: aligned with the code. The field comment and the spec now say an AcpRuntime-backed start records the PID its runtime reports before the stream opens. A runtime reports one from spawn until its process is reset, and a session is created only on a spawned runtime. The 1. A correction to the measurement's attribution. Your ~660 runs were all on the session-sharing path. On that path 3, 4, 5. Agreed; no change. The approval-prompt path is covered by the release-path tests, including the two added for the macOS review (a reap mid-pump and a stop with admission closed). |
|
Checks on the two PASS headlines at head Design: does every waker that resolves
The First principles: can a selectable backend reach the first-answer edge with a slow first token? Not on any shipped path. Every selectable backend (kiro-cli, KAS, claude-agent-acp) runs through |
macOS re-review at head
|
| # | Item | Status | Evidence |
|---|---|---|---|
| 1 | Reap mid-pump lost the "waiting to be admitted" text | Fixed | terminal.py:493 sets _reap_started, :505/:508 snapshot both parked flags, no await before :557/:571; text rendered from the snapshot at :614. test_a_reap_names_the_admission_wait_even_when_the_pump_ends_it_mid_reap drives the pump inside the patched reset and asserts _start_release was cleared mid-reap plus the exact error. |
| 2 | admission_closed read before the end markers |
Fixed | pump.py:854-855 early "ended", :859-863 gate read after, :917-918 re-check after await fut. test_a_stop_during_the_prompt_is_not_turned_into_a_rejection would fail with the old order. |
| 3 | Compaction / clear / agent-switch without runtime_global |
Fixed | session_handle.py:4073-4083, :4110-4136, :4142, :4147-4151. Remaining unmarked constructors (:3921, :5086, :5155, :5164-5169) are on routed paths and correctly stay unmarked. |
| 4 | Agent-folder timing undisclosed | Stated | Body + subagent.md release-path bullet. Only create_agent_folder( call is pump.py:1013 inside _log_spawned_impl, reached only after "admitted" (:825); tombstone-only folders are skipped by list_orphans. |
| 5 | Stale wording | Fixed | All eight listed sites; grep -i "permit held" over src/kiro_crew and docs/system-specs hits only two unrelated files. |
Tests, run serially one file at a time: test_subagent_startup_pressure.py 56 passed, test_subagent_startup_watchdog.py 13 passed, test_subagent_stall_activity_scope.py 16 passed, test_runloop_integration.py -k 'provenance or cotenant or startup' 1 passed. 0 failures.
The two PASS headlines
Design (every False-resolving waker also reaps): holds. Enumerated every writer of a terminal record for a release-parked run (_force_reap, the approval-path admission_closed branch, cancel_impl, boundary revocation, _unqueue/_settle_boundary_queue) across seven interleavings; none produces two terminal records or admits a start after _reap_started. rejected + reaped cannot both be written because both sequences are synchronous on either side of the marker.
First principles (PID before stream on every shipped start): the conclusion holds, but the supporting premise in the reply is wrong. ACP_BACKENDS_ACP_RUNTIME = {kiro, kas, codex} (agent_sdk/backends.py:1355); claude-agent-acp, opencode, pi, goose and deepseek are AcpClient-backed, not AcpRuntime-backed. They still record a PID at run.py:1537-1539 because get_or_create runs provider.start() -> ensure_ready() -> _spawn() before returning. So the real invariant is "every start whose provider spawned in start()", not "AcpRuntime-backed". Also confirmed: the reviewer's ~660 shared-session runs were out of startup at bind (run.py:3299-3300, _is_startup_stalled requires _pid is None), so the correction in the reply is right and the stream-open -> first-answer change is inert for that population.
ADVISORY (wording; none changes behaviour)
- "AcpRuntime-backed" under-describes the invariant (
subagent.py:1881-1885,subagent.md:1008, the commit message's unconditional form). A future provider that spawns lazily fromstream()is exactly the one the first-answer edge governs; the wording should let the next reader recognise it. Theif pid:/if runtime.pid:guards are effectively dead for shipped providers (AcpRuntime._pidis assigned once atruntime.py:2763and never cleared); a log line when they skip would make the no-PID population visible. - "Session-named
_kiro.devnotices are routed" is only true of_kiro.dev/session/update. The router reads onlyparams.sessionId(runtime.py:4245-4251), and compaction / clear / agent-switch notices and the steer echoes are sessionless, so under fan-out a session's own notice is flaggedruntime_globaland counted in_startup_cotenant_frames. By design, and fail-safe (a healthy run'ssession/updatechunks still end startup), but the counter's comment should say the count can include the run's own sessionless notices. - Counter predicate vs. wording.
_startup_cotenant_framesincrements while_first_stream_started is None(run.py:1657-1661), not while_in_startup(info), so a PID-backed start keeps counting after leaving startup at bind. Never rendered (a startup reap requires_pid is None), so the shown number is always correct, butsubagent.py:1886andsubagent.md:1008say "while still in startup". Either addinfo._pid is Noneto the guard or word it "before its first own frame". Frames dropped by the pre-prompt stale drain (session_handle.py:1487-1497) are also never counted. - Two deadline values in one reap.
terminal.py:617interpolates the live_startup_deadline;monitoring.py:952uses_stamped_startup_deadline(info)(merged in by the rebase). After a config write the error and the warning for one reap can name differentNs. Pre-existing on the base, but this PR rewrote line 617 and did not align it.
NIT
- The reaper's warning text (
monitoring.py:947-954) is not asserted by any test; only theinfo.errorrendering is (test_subagent_startup_watchdog.py:167). - A reap landing in the single loop tick between
"admitted"(pump.py:919) and_exec_startedfinds both parked flags False and uses the generic deadline text. One record, one tick, pre-existing on main.
Windows
No platform-conditional code in the nine changed source files and the delta adds none. Both PID sites see a CreateProcess id synchronously on the Proactor loop; the Windows-only dead-process branch in ensure_ready (client.py:10646-10655) clears the client's _pid after info._pid was already copied, so a run never re-enters startup. Nothing in this delta changes what Windows does for the shared-session population.
Nothing here was reproduced under a live gateway; interleavings are traced from code. The base was not executed.
Problem / Motivation
A run leaves startup when
_first_stream_startedis stamped: the startup watchdog stops watching it, the in-startup bound stops counting it, and a held spawn is woken. Onmainthat stamp lands when the provider'sstream()is entered, before anything has arrived, so a stream that opens and never yields reads as progress. The release path for approved spawns writes two wrong records:_spawn_with_approvallogs a run as spawned before it is admitted, and a user Stop or a reap that lands while the start waits is announced as "spawn rejected".Why it matters
The stamp is the evidence the watchdog and the bound act on. Stamped at stream open, a hang that begins after the open is invisible to both. The release-path records are what an operator diagnoses from: a spawn metric and audit for a run that never ran, and a neutral stop surfaced to the parent as a rejection.
What changed (motivation → approach → change)
src/kiro_crew/subagent.py_leave_startup, called fromsubagent_manager/run.pystream()is enteredrunningmark,subagent_manager/run.pysrc/kiro_crew/acp/session_handle.pyruntime_global, like the roster and MCP notices_log_spawnedon the release path,subagent_manager/admission/pump.py_admit_released_startadmits_admit_released_start"admitted"/"admission_closed"/"ended"; a stop or reap already under way answers"ended"before the admission gate is readsubagent_manager/terminal.py_reap_started, before the first await, so a pump pass inside the teardown cannot erase itsubagent_manager/terminal.pyandmonitoring.pyruntime_globalframes the start received while in startup (_startup_cotenant_frames), so a start that saw only co-tenant traffic reads apart from a silent one_write_tombstone,subagent.pyThe startup predicates' docstrings (
subagent.py,subagent_manager/monitoring.py) name the same exit.A run's startup ends when its own session answers its prompt.
_leave_startupstamps the marker once per execution at the two moments that is shown. The first is the first frame out of the stream addressed to this session, read on the raw frame so a completion held back for in-place recovery counts. The second is a dependency verdict on the prompt, the same step that takes the durable row fromstartingtorunning.A
runtime_globalframe is ownerless traffic that a shared runtime fans out to every co-tenant. It ends neither the startup state nor the durablerunningmark, because both read one provenance test. The dispatch loop sets that flag on every notice kind such a frame yields: the roster, the MCP notices, the compaction, clear and agent-switch notices and the steer echoes. Asession/updatenames its session by protocol and is routed, so its events stay clear. A start parked on a provider throttle before its first frame is out of startup for the whole wait; the dependency scope bounds that wait and ends it with its own error, so the watchdog never reaps it as "failed to start" and it never holds an in-startup slot while parked.A runtime PID ends startup on its own, and every AcpRuntime-backed start records one before its stream opens:
get_pidafterget_or_createon the dedicated path,runtime.pidin_bind_shared_handleon the shared one. For those starts the window closes at the PID. The first-answer edge is the exit only for a provider that publishes no PID.docs/system-specs/modules/subagent.mdstates that scope and names the recorded backends whose turns open with model output, which makes the window a first-output budget for such a provider.src/kiro_crew/docs/dynamic-subagent-sizing.mdnames the same wake edge.The release path records a start as spawned only once it is admitted. It writes the refusal only for
"admission_closed". For"ended"it writes nothing, because_force_reapowns that record: a stop is neutral, and a reap names the interrupted wait. Such a start was never recorded, so no live-run state orstate.jsonsays it is incognito or temporary;_write_tombstonedecides from the run's own memory mode, and a non-persistent run ended before admission (or at a declined prompt) leaves nothing on disk. Such a start has no agent folder, so, like a declined prompt, it is not orphan-visible after a restart.Backwards compatibility
Compatible: no config, API or wire change.
_admit_released_startis private; its return type changes frombooltostrand its only caller is updated. The durable row'sstarting -> runningedge waits for a frame addressed to this session, so a co-tenant's broadcast does not advance it; every laterrunningwriter (stop recovery, the dependency park, late adoption) is unchanged. A start ended before admission leaves no agent folder (a persistent one leaves only its tombstone), the same record a declined prompt leaves. The in-startup bound's wake edges are unchanged in kind: a PID, the first answer, a terminal.Tests
test_subagent_startup_pressure.py::TestOnlyThisSessionsFrameIsItsAnswerdrives a realAcpRuntime._reader_loop, which fans an ownerless frame -- the roster, an MCP registration or a compaction notice naming no session -- out to two sessions; the subagent's realAcpSessionHandleandAcpSessionProviderdeliver it to_run_innerasruntime_global. The marker stays clear and norunningmark is issued; the session's ownagent_message_chunkdoes both. Red while the stamp sits on the first event the loop consumes, whatever its provenance.TestOnlyThisSessionsFrameIsItsAnswer::test_a_routed_roster_frame_is_the_sessions_own_answer: the same roster kind through a routed frame (the KAS lifecycle path) ends startup and issues therunningmark, so an event-kind exclusion in place of the provenance test fails.TestStreamOpenIsNotProgress::test_a_completion_withheld_for_recovery_is_an_answer: a recoverable first completion takes the run out of startup before the recovery wait. Red while the stamp is taken only from events the run loop consumes, since the withheld completion never reaches it. The same class pins the hung-stream and first-event cases.test_subagent_stall_activity_scope.py::test_dispatch_carries_provenance_onto_every_fanned_out_notice: through the real reader loop and dispatch, a fanned-out compaction, clear, agent-switch or steer notice isruntime_global, and the same frame to a lone session is not.test_runloop_integration.py::test_a_throttle_before_the_first_frame_ends_startup_at_the_park(real durable store and coordinator): a PID-less start parked on a throttle is out of the bound, not reapable an hour past its deadline, and wakes the queue once. Red without the park stamp in_yield_for_dependency.TestAPidEndsStartupBeforeTheStream: the dedicated start has its PID and is out of startup when its stream opens; the shared start is out of startup at bind.user_stopped.test_a_reap_names_the_admission_wait_even_when_the_pump_ends_it_mid_reap: asessions.resetthat yields runs a pump pass that clears_start_release; the reap's error still names the admission wait and_log_spawnednever fires for the reaped start. Red with the snapshot taken after the teardown.test_a_stop_during_the_prompt_is_not_turned_into_a_rejection: admission closes and the prompt resolves inside a user stop's teardown; norejectedoutcome is written and nothing starts. Red with the end-marker check after the admission gate.test_subagent_startup_watchdog.py::test_startup_reap_names_the_co_tenant_frames_it_received, and the real-runtime co-tenant test asserting the count is 1 after the fanned-out frame and stays 1 once the run's own frame ends startup.test_a_temporary_start_reaped_while_waiting_leaves_nothing_on_disk: a temporary-mode start reaped while it waits for the bound, through the real_write_tombstone, leaves no agent folder and notombstone.json; red without the memory-mode check.Ran on this branch against current
main: the backend related-test set (run_scoped_tests.py --surface backend) and the backend gate floor (black, isort, flake8, mypy, comment-history, docs lint and the othercheck_*gates).Manual verification
The co-tenant case runs through a real runtime reader loop and session handle, the throttle park through the real dependency coordinator and durable store, and the release path through the real
_spawn_with_approval.120-item fan-out through this branch's real admission path (pump, in-startup bound,
SessionStartGate, cold-start admission, startup watchdog on its 60s sweep) with a modelled dedicated-path provider, time-scaled x0.1 (--scale 0.1; the harness default is 0.05); harness and raw output in #14020 (comment):initializesession/newinitializesession_start_concurrency = 3session_start_timeout_secs = 300, slow MCP fleetsession/newin flight)Every start here publishes a PID, so it leaves startup at the PID and this PR's first-answer edge moves no row. A deep queue costs latency, not starts. The reaps in the last three rows come from two limits outside this PR, reported on the thread: the startup clock runs while a dedicated start waits for the 2-wide cold-start admission ahead of the gate, and a startup deadline shorter than a raised
session/newbudget;mainnow sizes that deadline from the start budgets, and this branch is rebased on it.Related Issues
no linked issue: follow-ups on #13890's final head, whose advisory findings were carried forward after that PR merged.
Pattern harvest
Rule candidate: review-prompt
Pattern: "a progress marker stamped when an operation is STARTED (stream opened, request sent), or from a multiplexed source without provenance, instead of when THIS operation first produces something -- every watchdog keyed on it goes blind to a hang, or is satisfied by another tenant's traffic".
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)docs/system-specs/modules/subagent.md,docs/system-specs/modules/acp-client.md,src/kiro_crew/docs/dynamic-subagent-sizing.md