Skip to content

fix(subagent): leave startup on the first stream event, not at stream open - #14020

Merged
bolichen97 merged 1 commit into
mainfrom
fix/subagent-startup-progress-stamp
Sep 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/subagent-startup-progress-stamp

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A run leaves startup when _first_stream_started is stamped: the startup watchdog stops watching it, the in-startup bound stops counting it, and a held spawn is woken. On main that stamp lands when the provider's stream() is entered, before anything has arrived, so a stream that opens and never yields reads as progress. The release path for approved spawns writes two wrong records: _spawn_with_approval logs a run as spawned before it is admitted, and a user Stop or a reap that lands while the start waits is announced as "spawn rejected".

Why it matters

The stamp is the evidence the watchdog and the bound act on. Stamped at stream open, a hang that begins after the open is invisible to both. The release-path records are what an operator diagnoses from: a spawn metric and audit for a run that never ran, and a neutral stop surfaced to the parent as a rejection.

What changed (motivation → approach → change)

Where Before After
Startup exit: src/kiro_crew/subagent.py _leave_startup, called from subagent_manager/run.py stamped when stream() is entered first frame addressed to this session, or a dependency verdict on the prompt
Stream's durable running mark, subagent_manager/run.py any frame the same provenance test as the startup exit
Fanned-out compaction, clear and agent-switch notices and steer echoes, src/kiro_crew/acp/session_handle.py no provenance on the event runtime_global, like the roster and MCP notices
_log_spawned on the release path, subagent_manager/admission/pump.py before admission after _admit_released_start admits
_admit_released_start one boolean "admitted" / "admission_closed" / "ended"; a stop or reap already under way answers "ended" before the admission gate is read
Reap's "waiting to be admitted" wording, subagent_manager/terminal.py read after the session teardown awaits read at _reap_started, before the first await, so a pump pass inside the teardown cannot erase it
Startup reap record, subagent_manager/terminal.py and monitoring.py no frame evidence names the runtime_global frames the start received while in startup (_startup_cotenant_frames), so a start that saw only co-tenant traffic reads apart from a silent one
_write_tombstone, subagent.py reads only the folder's recorded mode the run's own memory mode, so an unrecorded non-persistent run writes nothing

The startup predicates' docstrings (subagent.py, subagent_manager/monitoring.py) name the same exit.

A run's startup ends when its own session answers its prompt. _leave_startup stamps the marker once per execution at the two moments that is shown. The first is the first frame out of the stream addressed to this session, read on the raw frame so a completion held back for in-place recovery counts. The second is a dependency verdict on the prompt, the same step that takes the durable row from starting to running.

A runtime_global frame is ownerless traffic that a shared runtime fans out to every co-tenant. It ends neither the startup state nor the durable running mark, because both read one provenance test. The dispatch loop sets that flag on every notice kind such a frame yields: the roster, the MCP notices, the compaction, clear and agent-switch notices and the steer echoes. A session/update names its session by protocol and is routed, so its events stay clear. A start parked on a provider throttle before its first frame is out of startup for the whole wait; the dependency scope bounds that wait and ends it with its own error, so the watchdog never reaps it as "failed to start" and it never holds an in-startup slot while parked.

A runtime PID ends startup on its own, and every AcpRuntime-backed start records one before its stream opens: get_pid after get_or_create on the dedicated path, runtime.pid in _bind_shared_handle on the shared one. For those starts the window closes at the PID. The first-answer edge is the exit only for a provider that publishes no PID. docs/system-specs/modules/subagent.md states that scope and names the recorded backends whose turns open with model output, which makes the window a first-output budget for such a provider. src/kiro_crew/docs/dynamic-subagent-sizing.md names the same wake edge.

The release path records a start as spawned only once it is admitted. It writes the refusal only for "admission_closed". For "ended" it writes nothing, because _force_reap owns that record: a stop is neutral, and a reap names the interrupted wait. Such a start was never recorded, so no live-run state or state.json says it is incognito or temporary; _write_tombstone decides from the run's own memory mode, and a non-persistent run ended before admission (or at a declined prompt) leaves nothing on disk. Such a start has no agent folder, so, like a declined prompt, it is not orphan-visible after a restart.

Backwards compatibility

Compatible: no config, API or wire change. _admit_released_start is private; its return type changes from bool to str and its only caller is updated. The durable row's starting -> running edge waits for a frame addressed to this session, so a co-tenant's broadcast does not advance it; every later running writer (stop recovery, the dependency park, late adoption) is unchanged. A start ended before admission leaves no agent folder (a persistent one leaves only its tombstone), the same record a declined prompt leaves. The in-startup bound's wake edges are unchanged in kind: a PID, the first answer, a terminal.

Tests

  • test_subagent_startup_pressure.py::TestOnlyThisSessionsFrameIsItsAnswer drives a real AcpRuntime._reader_loop, which fans an ownerless frame -- the roster, an MCP registration or a compaction notice naming no session -- out to two sessions; the subagent's real AcpSessionHandle and AcpSessionProvider deliver it to _run_inner as runtime_global. The marker stays clear and no running mark is issued; the session's own agent_message_chunk does both. Red while the stamp sits on the first event the loop consumes, whatever its provenance.
  • TestOnlyThisSessionsFrameIsItsAnswer::test_a_routed_roster_frame_is_the_sessions_own_answer: the same roster kind through a routed frame (the KAS lifecycle path) ends startup and issues the running mark, so an event-kind exclusion in place of the provenance test fails.
  • TestStreamOpenIsNotProgress::test_a_completion_withheld_for_recovery_is_an_answer: a recoverable first completion takes the run out of startup before the recovery wait. Red while the stamp is taken only from events the run loop consumes, since the withheld completion never reaches it. The same class pins the hung-stream and first-event cases.
  • test_subagent_stall_activity_scope.py::test_dispatch_carries_provenance_onto_every_fanned_out_notice: through the real reader loop and dispatch, a fanned-out compaction, clear, agent-switch or steer notice is runtime_global, and the same frame to a lone session is not.
  • test_runloop_integration.py::test_a_throttle_before_the_first_frame_ends_startup_at_the_park (real durable store and coordinator): a PID-less start parked on a throttle is out of the bound, not reapable an hour past its deadline, and wakes the queue once. Red without the park stamp in _yield_for_dependency.
  • TestAPidEndsStartupBeforeTheStream: the dedicated start has its PID and is out of startup when its stream opens; the shared start is out of startup at bind.
  • The release-path tests: a reap while waiting is announced as a reap, a closed-admission release is refused on its own, and a stop while waiting keeps user_stopped.
  • test_a_reap_names_the_admission_wait_even_when_the_pump_ends_it_mid_reap: a sessions.reset that yields runs a pump pass that clears _start_release; the reap's error still names the admission wait and _log_spawned never fires for the reaped start. Red with the snapshot taken after the teardown.
  • test_a_stop_during_the_prompt_is_not_turned_into_a_rejection: admission closes and the prompt resolves inside a user stop's teardown; no rejected outcome is written and nothing starts. Red with the end-marker check after the admission gate.
  • test_subagent_startup_watchdog.py::test_startup_reap_names_the_co_tenant_frames_it_received, and the real-runtime co-tenant test asserting the count is 1 after the fanned-out frame and stays 1 once the run's own frame ends startup.
  • test_a_temporary_start_reaped_while_waiting_leaves_nothing_on_disk: a temporary-mode start reaped while it waits for the bound, through the real _write_tombstone, leaves no agent folder and no tombstone.json; red without the memory-mode check.

Ran on this branch against current main: the backend related-test set (run_scoped_tests.py --surface backend) and the backend gate floor (black, isort, flake8, mypy, comment-history, docs lint and the other check_* gates).

Manual verification

The co-tenant case runs through a real runtime reader loop and session handle, the throttle park through the real dependency coordinator and durable store, and the release path through the real _spawn_with_approval.

120-item fan-out through this branch's real admission path (pump, in-startup bound, SessionStartGate, cold-start admission, startup watchdog on its 60s sweep) with a modelled dedicated-path provider, time-scaled x0.1 (--scale 0.1; the harness default is 0.05); harness and raw output in #14020 (comment):

Scenario G / bound spawn + initialize session/new Success Startup reaps (stage) Queue wait p50 / max
healthy 2 / 4 2-5s median 10s 100% 0 340s / 652s
healthy, stagger 2s 2 / 4 2-5s median 10s 100% 0 323s / 643s
memory-throttled initialize 2 / 4 30-85s median 10s 97.5% 3 (before the gate) 1627s / 3370s
same, session_start_concurrency = 3 3 / 6 30-85s median 10s 57-61% (2 runs) 47-52 (before the gate) 1349s / 2825s
session_start_timeout_secs = 300, slow MCP fleet 2 / 4 2-5s 100-160s 46.7% 64 (session/new in flight) 3509s / 6990s

Every start here publishes a PID, so it leaves startup at the PID and this PR's first-answer edge moves no row. A deep queue costs latency, not starts. The reaps in the last three rows come from two limits outside this PR, reported on the thread: the startup clock runs while a dedicated start waits for the 2-wide cold-start admission ahead of the gate, and a startup deadline shorter than a raised session/new budget; main now sizes that deadline from the start budgets, and this branch is rebased on it.

Related Issues

no linked issue: follow-ups on #13890's final head, whose advisory findings were carried forward after that PR merged.

Pattern harvest

Rule candidate: review-prompt
Pattern: "a progress marker stamped when an operation is STARTED (stream opened, request sent), or from a multiplexed source without provenance, instead of when THIS operation first produces something -- every watchdog keyed on it goes blind to a hang, or is satisfied by another tenant's traffic".

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) -- docs/system-specs/modules/subagent.md, docs/system-specs/modules/acp-client.md, src/kiro_crew/docs/dynamic-subagent-sizing.md
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and hoang-phan98 September 25, 2026 23:20
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 7c3b94a15be767b152a63617e3eb61b67ac5964b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify every waker that resolves _start_release False also runs _force_reap — an unowned "ended" outcome now writes no terminal record at all.

[DESIGN-REVIEWED] 7c3b94a

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 7c3b94a15be767b152a63617e3eb61b67ac5964b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify that no selectable backend can reach the first-answer edge with a slow first token: for a PID-less provider the startup deadline is now a first-output budget, and a healthy start that opens its stream but thinks past that deadline is reaped where stream-open once saved it. The diff's own mitigation is checked and holds in-repo — every AcpRuntime-backed start records runtime.pid before its stream opens (_bind_shared_handle_impl, run.py:3299, pinned by TestAPidEndsStartupBeforeTheStream), and the live fixtures for kiro-cli 2.21.4, KAS 0.63.3 and claude-agent-acp 0.76.0 open turns with model output — so the confirmation needed is only that no non-AcpRuntime provider path exists or is planned to select without a PID. The reversed spec line ("startup ends when the first provider stream begins") is named, argued against with tests red on base, and rewritten in the same commit, so the pin reversal carries its evidence. _admit_released_start's three-way return has one caller consuming all three variants (grep: 1 caller, pump.py:793).

What this change ships

Inventory (10 items) — 10 justified

Intent: FIX — a subagent run must leave startup only when its own session has answered its prompt, and the release path must record only what actually happened.

  1. A stream that opens and never yields keeps the run in startup, watchdog-reapable and bound-counted — justified
  2. A co-tenant's fanned-out frame no longer ends startup or marks the durable row running — justified
  3. Fanned-out compaction, clear, agent-switch and steer notices now carry runtime_global provenance — justified
  4. A start parked on a provider throttle before its first frame leaves startup at the park — justified
  5. An approved spawn is logged as spawned only after admission, never for a refused or ended start — justified
  6. A stop or reap during the admission wait is announced as itself, not as "spawn rejected" — justified
  7. The reap's "waiting to be admitted" wording survives a pump pass that ends the wait mid-reap — justified
  8. Startup reap error and reaper warning name the co-tenant frame count received in startup — justified
  9. A non-persistent run ended before admission leaves no tombstone on disk — justified
  10. Specs and docs restated to name the new startup exit, same commit — justified

[FIRST-PRINCIPLES-REVIEWED] 7c3b94a

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 7c3b94a15be767b152a63617e3eb61b67ac5964b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7c3b94a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7c3b94a15be767b152a63617e3eb61b67ac5964b: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 7c3b94a15be767b152a63617e3eb61b67ac5964b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7c3b94a

Verdict parsed from the review's SHA-scoped output markers for commit 7c3b94a15be767b152a63617e3eb61b67ac5964b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7c3b94a15be767b152a63617e3eb61b67ac5964b: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: A sub-agent leaves startup only on evidence that its own session answered its prompt, so the startup watchdog and the in-startup bound cannot be satisfied by an opened stream or by a co-tenant's traffic, and the approval-release path writes truthful spawn records.
Not a goal: Re-deriving the 120s startup deadline, freezing the clock for pre-gate cold-start permits, resizing the in-startup bound, or changing any config key.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the startup exit and the stream's running mark key on frame provenance at the raw stream frame

  • span=3b96d9d7798e Global runtime events falsely end startup -- fixed in 651cf7a

Fixed: _leave_startup (subagent.py) and ensure_running_marked run in run.py's inner stream loop only for _ev.runtime_global False, so a fanned-out roster or MCP frame ends neither startup nor the durable starting state; the pump wake rides _leave_startup.
Test: TestOnlyThisSessionsFrameIsItsAnswer feeds an ownerless roster frame through a real AcpRuntime._reader_loop fan-out and a real AcpSessionHandle/AcpSessionProvider into _run_inner; fails on 8e8729f.
Reach note: _in_startup/_is_startup_stalled read the marker only while _pid is None, and every AcpRuntime-backed start records runtime.pid before its stream (run.py _bind_shared_handle, the dedicated get_pid record; pinned by TestAPidEndsStartupBeforeTheStream), so the reaper/bound harm needed a PID-less start; the predicate is fixed regardless.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the startup exit and the stream's running mark key on frame provenance at the raw stream frame

  • span=01601ed122ca a runtime_global frame stamps _first_stream_started -- fixed in 651cf7a

Fixed: the stamp (_leave_startup) and the running mark require not _ev.runtime_global, taken in the inner stream loop in run.py, so a co-tenant's roster or MCP fan-out ends neither.
Test: TestOnlyThisSessionsFrameIsItsAnswer (real runtime reader loop, real session handle and provider); fails on 8e8729f.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a dependency verdict on the prompt ends startup, in the step that takes the row starting -> running

  • span=01601ed122ca a run parked before its first event stays inside the startup window -- fixed in 651cf7a

Fixed: _yield_for_dependency calls _leave_startup before the park -- the step whose _dependency_report_db already takes the row starting -> running, because the provider answered the prompt. The wait is then bounded by its scope's deadline and error, never the startup watchdog, and holds no in-startup slot.
A withheld stop-recovery completion is an answer too: the stamp is read on the raw frame, ahead of the withholding (test_a_completion_withheld_for_recovery_is_an_answer).
Test: test_a_throttle_before_the_first_frame_ends_startup_at_the_park (real durable store and coordinator); fails without the park stamp.
Mechanism correction: _bind_shared_handle sets info._pid = runtime.pid, so a shared-session run has a PID before its stream; only a PID-less start could be reaped this way.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=eac5666cf65e the first-event defence is asserted, not pinned -- fixed in 651cf7a

Fixed: subagent.md states which starts the first-answer edge governs. Every AcpRuntime-backed start records a runtime PID before its stream opens (dedicated: get_pid after get_or_create; shared: runtime.pid at bind), pinned by TestAPidEndsStartupBeforeTheStream, so for them the window closes at the PID and never reaches the first token.
For a PID-less provider the window IS a first-output budget: the recorded live turns in test/fixtures/acp_frames/ (kiro-cli 2.21.4, KAS 0.63.3, claude-agent-acp 0.76.0) open with model output and carry no turn-start frame, and _kiro.dev/metadata yields no event; the spec names this.

@bolichen97
bolichen97 force-pushed the fix/subagent-startup-progress-stamp branch from 8e8729f to 651cf7a Compare September 26, 2026 03:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Fan-out measurement behind the "Manual verification" table (120 items through the real SubagentManager admission machinery -- durable task store, spawn_async, stagger queue, _startup_cap, pump, the real _gate_wait_mark / _gate_exit_reset callbacks around an asyncio.Semaphore(G) standing in for the SessionStartGate, the real _is_startup_stalled predicate swept every second; synthetic provider that holds a permit for session/new seconds, records a PID and runs a body). Measured on the first commit's tree; the second commit changes no admission or gate code path.

{"config": {"n": 120, "cap": 12, "gate_width": 2, "startup_cap": 4, "session_new_secs": 5.0, "jitter_secs": 2.0, "run_body_secs": 45.0}, "elapsed_secs": 538.3, "success": 120, "reaped_by_startup_watchdog": 0, "never_started": 0, "max_in_startup_observed": 4, "spawn_queue_wait_secs": {"p50": 249.19, "p90": 435.11, "p99": 480.42, "max": 484.47}, "gate_wait_secs": {"p50": 0.0, "p90": 2.95, "p99": 6.16, "max": 6.16}, "time_to_pid_from_spawn_secs": {"p50": 252.88, "p99": 487.11}, "end_to_end_secs": {"p50": 297.88, "p99": 532.11, "max": 536.55}, "mean_gate_wait_secs": 0.91}
{"config": {"n": 120, "cap": 32, "gate_width": 2, "startup_cap": 4, "session_new_secs": 5.0, "jitter_secs": 2.0, "run_body_secs": 45.0}, "elapsed_secs": 351.7, "success": 120, "reaped_by_startup_watchdog": 0, "never_started": 0, "max_in_startup_observed": 4, "spawn_queue_wait_secs": {"p50": 150.28, "p90": 269.98, "p99": 294.39, "max": 296.0}, "gate_wait_secs": {"p50": 5.06, "p90": 6.59, "p99": 6.97, "max": 7.0}, "time_to_pid_from_spawn_secs": {"p50": 161.45, "p99": 302.75}, "end_to_end_secs": {"p50": 206.45, "p99": 347.75, "max": 350.73}, "mean_gate_wait_secs": 5.01}
{"config": {"n": 120, "cap": 32, "gate_width": 2, "startup_cap": 4, "session_new_secs": 20.0, "jitter_secs": 5.0, "run_body_secs": 20.0}, "elapsed_secs": 1242.4, "success": 120, "reaped_by_startup_watchdog": 0, "never_started": 0, "max_in_startup_observed": 4, "spawn_queue_wait_secs": {"p50": 595.77, "p90": 1068.31, "p99": 1181.58, "max": 1184.55}, "gate_wait_secs": {"p50": 20.48, "p90": 24.19, "p99": 24.97, "max": 25.0}, "time_to_pid_from_spawn_secs": {"p50": 639.39, "p99": 1218.06}, "end_to_end_secs": {"p50": 659.39, "p99": 1238.06, "max": 1241.3}, "mean_gate_wait_secs": 20.02}

Reading: zero startup reaps at any queue depth (the clock never runs before admission and is frozen at the gate); the in-startup population never exceeds the bound; the tail is the running cap when the cap is small and the gate (G starts per session/new latency) when it is large -- at cap 32 the running population peaked around 24 because admission, not the cap, was the limit. Two harness notes for anyone repeating it: the sync SubagentManager.spawn() must not be used from inside a running loop (the on-loop DB guard refuses some row writes and 27 of 120 rows were then never dispatched -- a harness artifact, not a product defect; spawn_async dispatched 120/120), and the run body must be short enough that the wave finishes inside the sweep's own budget.

Not measured here, and not measurable offline: the real distribution of session/new under provider throttling. What the second commit does about that is make the deadline follow agent.session_start_timeout_secs instead of guessing it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Fan-out measurement behind the body's "Manual verification" table — 120 items through this branch's real admission path, with a modelled provider. Supersedes the earlier measurement comment on this PR, which used a plain semaphore for the gate and no cold-start admission.

Real code: SubagentManager.spawn -> the pump (stagger, cap 64, in-startup bound 2G) -> _run -> _run_inner, a durable task store, the real SessionStartGate with _gate_wait_mark / _gate_exit_reset, the real per-loop cold-start admission (_COLD_START_MAX_CONCURRENT), and the real _is_startup_stalled + _force_reap(reason="startup_timeout") swept at the production 60s interval.

Modelled: get_or_create runs the dedicated path's permit sequence -- the 4-wide SessionManager start semaphore, the 2-wide cold-start admission, spawn + initialize (budget 30s, or 90s under throttle), the gate, session/new (its budget) -- then publishes a PID; stream() waits a first-token delay, yields a chunk, waits the turn. Every duration is scaled x0.1 (x0.1 and x0.2 agree within 3% on a 40-item calibration run); numbers below are production seconds.

Reading:

  • A deep queue costs latency, not starts: healthy starts complete 120/120 with the last admitted about 650s in (gate-bound: G / session/new).
  • Every start here publishes a PID, so it leaves startup at the PID and this PR's first-answer edge moves no row.
  • The reaps come from two limits this PR does not touch. (1) Before the gate, the startup clock runs while a dedicated start waits for the 2-wide cold-start admission: the three reaped starts in the throttled row had waited 68-73s for it before their own 82-85s initialize, and a gate of 3 (bound 6 > admission 2) turns that into 47-52 reaps. (2) The 120s deadline is shorter than a raised session/new budget: with session_start_timeout_secs = 300 every reap (64) hit a session/new still inside its own budget.

Per-scenario summaries (scenario, knobs, success, reaps by stage, queue wait, spawn-to-session-ready):

{"scenario": "s1-healthy", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:2,5", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 30.0, "session_new": 90.0}, "completed": 120, "success_rate": 1.0, "reaped_startup_timeout": 0, "reap_stages": {}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 339.8, "p90": 597.1, "p99": 649.1, "max": 651.7}, "spawn_to_session_ready_s": {"p50": 367.7, "p90": 611.8, "max": 720.0}, "makespan_s": 764.9}
{"scenario": "s2-healthy-stagger2", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 2.0, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:2,5", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 30.0, "session_new": 90.0}, "completed": 120, "success_rate": 1.0, "reaped_startup_timeout": 0, "reap_stages": {}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 322.8, "p90": 584.4, "p99": 636.0, "max": 642.6}, "spawn_to_session_ready_s": {"p50": 348.4, "p90": 598.6, "max": 667.3}, "makespan_s": 742.1}
{"scenario": "s3b-throttled-init", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:30,85", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 90.0, "session_new": 90.0}, "completed": 117, "success_rate": 0.975, "reaped_startup_timeout": 3, "reap_stages": {"spawn+initialize": 3}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 1626.9, "p90": 3027.8, "p99": 3345.1, "max": 3370.3}, "spawn_to_session_ready_s": {"p50": 1744.7, "p90": 3168.0, "max": 3501.0}, "makespan_s": 3586.2}
{"scenario": "s4-throttled-init-G3", "n": 120, "cap": 64, "gate_G": 3, "in_startup_bound": 6, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:30,85", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 90.0, "session_new": 90.0}, "completed": 68, "success_rate": 0.567, "reaped_startup_timeout": 52, "reap_stages": {"spawn+initialize": 52}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 1348.5, "p90": 2488.7, "p99": 2788.8, "max": 2789.1}, "spawn_to_session_ready_s": {"p50": 1459.3, "p90": 2616.3, "max": 2965.5}, "makespan_s": 3011.3}
{"scenario": "s4b-throttled-init-G3", "n": 120, "cap": 64, "gate_G": 3, "in_startup_bound": 6, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:30,85", "session_new": "lognormal:10,0.5,2,80", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 90.0, "session_new": 90.0}, "completed": 73, "success_rate": 0.608, "reaped_startup_timeout": 47, "reap_stages": {"spawn+initialize": 47}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 1348.5, "p90": 2496.3, "p99": 2792.8, "max": 2825.2}, "spawn_to_session_ready_s": {"p50": 1523.1, "p90": 2720.3, "max": 2970.7}, "makespan_s": 3058.0}
{"scenario": "s5-large-mcp-fleet", "n": 120, "cap": 64, "gate_G": 2, "in_startup_bound": 4, "cold_start_admission": 2, "start_sem": 4, "stagger_s": 0.25, "startup_deadline_s": 120.0, "dists": {"spawn_init": "uniform:2,5", "session_new": "uniform:100,160", "ttft": "uniform:2,10", "turn": "uniform:30,90"}, "budgets": {"initialize": 30.0, "session_new": 300.0}, "completed": 56, "success_rate": 0.467, "reaped_startup_timeout": 64, "reap_stages": {"session/new with permit": 64}, "typed_start_failures": {}, "not_finished": 0, "queue_wait_s": {"p50": 3508.8, "p90": 6269.4, "p99": 6989.6, "max": 6989.9}, "spawn_to_session_ready_s": {"p50": 2766.8, "p90": 6491.1, "max": 7221.6}, "makespan_s": 7295.5}
Harness (stdlib + the repo's own modules; run from the worktree)
#!/usr/bin/env python3
"""Wide sub-agent fan-out through the REAL SubagentManager admission path.

Real code exercised (this worktree's src/):
  * SubagentManager.spawn -> admission pump (stagger, running cap, the in-startup
    bound 2G) -> _run -> _run_inner, with a real durable task store;
  * the real per-loop SessionStartGate (acp.runtime.session_start_gate) and the
    real _gate_wait_mark / _gate_exit_reset clock callbacks;
  * the real per-loop cold-start admission (acp.runtime._cold_start_admission,
    limit _COLD_START_MAX_CONCURRENT);
  * the real startup watchdog: _is_startup_stalled + _force_reap(reason=
    "startup_timeout"), evaluated on a sweep at the production interval.

Modeled: the provider. get_or_create reproduces the DEDICATED path's permit
sequence (session_allocation._start_sem -> AcpRuntime.spawn's cold-start
admission -> spawn + initialize -> SessionStartGate -> session/new), with each
stage's own budget; a PID is published once get_or_create returns, as
AcpSessionProvider._pid does. stream() waits the time-to-first-token, emits one
chunk, waits the rest of the turn and ends it.

Every duration -- latencies, stagger, startup deadline, reaper interval, stage
budgets -- is multiplied by SCALE, so a 120-item wave runs in minutes; the
admission logic only compares these against each other. Reported numbers are in
PRODUCTION seconds (divided back by SCALE).
"""

from __future__ import annotations

import argparse
import asyncio
import json
import os
import random
import statistics
import sys
import tempfile
import time
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock

WT = os.environ.get("KIROCREW_WORKTREE", os.getcwd())  # the worktree under test
sys.path.insert(0, os.path.join(WT, "src"))
sys.path.insert(0, os.path.join(WT, "test"))
_HOME = tempfile.mkdtemp(prefix="kc-fanout-home-")
os.environ["KIROCREW_HOME"] = _HOME
os.environ["KIROCREW_TELEMETRY"] = "0"

import kiro_crew.acp.runtime as rt_mod  # noqa: E402
import kiro_crew.resource_status as resource_status  # noqa: E402
import kiro_crew.subagent as subagent_mod  # noqa: E402
from kiro_crew.providers.base import EVENT_COMPLETE, EVENT_TEXT_CHUNK  # noqa: E402
from kiro_crew.subagent import SubagentManager  # noqa: E402


def _pin_host_memory() -> None:
    subagent_mod.check_memory_available = lambda min_gb=None, path=None: (True, 64.0)
    subagent_mod.cached_admission_check = lambda: resource_status.AdmissionDecision(
        admitted=True, posture=resource_status.POSTURE_AMPLE, available_gb=64.0
    )


class Dist:
    """A latency distribution in production seconds."""

    def __init__(self, spec: str, rng: random.Random) -> None:
        self.spec = spec
        self._rng = rng
        kind, _, rest = spec.partition(":")
        self.kind = kind
        self.args = [float(x) for x in rest.split(",")] if rest else []

    def draw(self) -> float:
        a = self.args
        if self.kind == "const":
            return a[0]
        if self.kind == "uniform":
            return self._rng.uniform(a[0], a[1])
        if self.kind == "lognormal":  # median, sigma, lo, hi
            import math

            v = self._rng.lognormvariate(math.log(a[0]), a[1])
            return min(max(v, a[2]), a[3])
        raise ValueError(self.spec)


class Rec(SimpleNamespace):
    pass


async def run(args: argparse.Namespace) -> dict:
    _pin_host_memory()
    rng = random.Random(args.seed)
    S = args.scale
    spawn_init = Dist(args.spawn_init, rng)
    session_new = Dist(args.session_new, rng)
    ttft = Dist(args.ttft, rng)
    turn = Dist(args.turn, rng)
    t0 = time.monotonic()
    t_wall0 = time.time()

    def now() -> float:
        return (time.monotonic() - t0) / S  # production seconds since the wave began

    loop = asyncio.get_running_loop()
    # The real gate, sized for this scenario (the manager's bound follows it).
    rt_mod._session_start_gates[loop] = rt_mod.SessionStartGate(args.gate)
    start_sem = asyncio.Semaphore(args.start_sem)  # SessionManager._start_sem
    recs: dict[str, Rec] = {}
    pids: dict[str, int] = {}
    next_pid = [90_000_000]

    def rec_for(key: str) -> Rec:
        if key not in recs:
            recs[key] = Rec(key=key)
        return recs[key]

    def make_provider(key: str):
        provider = MagicMock()
        provider.start = AsyncMock()
        provider.shutdown = AsyncMock()
        provider.context_usage_pct = lambda: 0.0
        provider.context_used_tokens = MagicMock(return_value=0)
        provider.context_window_tokens = MagicMock(return_value=0)
        provider.mcp_session_report = MagicMock(return_value=None)
        provider.last_infra_error = None
        provider.session_id = f"sess-{key}"

        def _stream(msg, *_a, **_k):
            async def _gen():
                r = rec_for(key)
                r.t_stream = now()
                await asyncio.sleep(ttft.draw() * S)
                r.t_first = now()
                yield SimpleNamespace(kind=EVENT_TEXT_CHUNK, text="x", runtime_global=False)
                await asyncio.sleep(turn.draw() * S)
                r.t_done = now()
                yield SimpleNamespace(
                    kind=EVENT_COMPLETE, stop_reason="end_turn", runtime_global=False
                )

            return _gen()

        provider.stream = MagicMock(side_effect=_stream)
        return provider

    async def get_or_create(session_key, *_a, on_gate_acquired=None, on_gate_queued=None, **_kw):
        r = rec_for(session_key)
        r.t_enter = now()
        async with start_sem:
            r.t_start_sem = now()
            adm = rt_mod._cold_start_admission()
            await adm.acquire()
            r.t_cold = now()
            try:
                d = spawn_init.draw()
                r.spawn_init = d
                if d > args.init_budget:
                    await asyncio.sleep(args.init_budget * S)
                    r.fail_stage = "initialize budget"
                    raise RuntimeError("initialize went unanswered (typed start failure)")
                await asyncio.sleep(d * S)
            finally:
                adm.release()
            r.t_spawned = now()
            gate = await rt_mod.session_start_gate()
            if on_gate_queued is not None:
                on_gate_queued()
            permit = await gate.acquire()
            if on_gate_acquired is not None:
                on_gate_acquired(permit.queue_wait_ms)
            r.t_gate = now()
            try:
                d = session_new.draw()
                r.session_new = d
                if d > args.session_budget:
                    await asyncio.sleep(args.session_budget * S)
                    r.fail_stage = "session/new budget"
                    raise RuntimeError("session/new timed out (typed start failure)")
                await asyncio.sleep(d * S)
            finally:
                permit.release()
            r.t_session = now()
        next_pid[0] += 1
        pids[session_key] = next_pid[0]
        return make_provider(session_key), True, False

    sessions = MagicMock()
    sessions.get_or_create = AsyncMock(side_effect=get_or_create)
    sessions.get_pid = MagicMock(side_effect=lambda key: pids.get(key))
    sessions.get_approval_policy = MagicMock(return_value="auto")
    sessions.get_agent = MagicMock(return_value="")
    sessions.get_agent_selection = MagicMock(return_value=("template", ""))
    sessions.has_session = MagicMock(return_value=True)
    sessions.release = MagicMock()
    sessions.reset = AsyncMock()
    sessions.record_success = MagicMock()
    ctx = MagicMock()
    ctx.hooks.auto_approve_subagent_spawn = True
    ctx.hooks.auto_approve_subagent_tools = False
    ctx.build_message = MagicMock(side_effect=lambda msg, *a, **k: (msg, None))

    mgr = SubagentManager(
        sessions=sessions,
        ctx_builder=ctx,
        max_concurrent=args.cap,
        startup_timeout=120,
        default_timeout=int(args.wall_clock),
    )
    mgr._startup_deadline = args.deadline * S
    mgr._default_timeout = args.wall_clock * S
    mgr._session_start_concurrency = args.gate
    mgr._spawn_stagger_secs = args.stagger * S
    mgr._should_use_session_sharing = MagicMock(return_value=False)
    mgr._fire_event = AsyncMock()
    await mgr.wait_taskq_ready()

    reaped_at: dict[str, dict] = {}
    stop = asyncio.Event()

    async def sweep() -> None:
        # The production watchdog body (monitoring._reaper_loop), on its interval.
        phase = rng.uniform(0, args.reaper_interval) * S
        await asyncio.sleep(phase)
        while not stop.is_set():
            t = time.time()
            for agent_id, info in list(mgr._agents.items()):
                if info.done:
                    continue
                if mgr._is_startup_stalled(info, t):
                    key = f"subagent:{agent_id}"
                    r = rec_for(key)
                    stage = (
                        "queued for start_sem" if not hasattr(r, "t_start_sem") else
                        "queued for cold-start admission" if not hasattr(r, "t_cold") else
                        "spawn+initialize" if not hasattr(r, "t_spawned") else
                        "queued at SessionStartGate" if not hasattr(r, "t_gate") else
                        "session/new with permit" if not hasattr(r, "t_session") else
                        "after session/new"
                    )
                    reaped_at[agent_id] = {"stage": stage, "t": now()}
                    try:
                        await mgr._force_reap(
                            agent_id, info, t - (info._exec_started or t), reason="startup_timeout"
                        )
                    except Exception as exc:  # noqa: BLE001
                        reaped_at[agent_id]["reap_error"] = repr(exc)
            await asyncio.sleep(args.reaper_interval * S)

    sweeper = asyncio.ensure_future(sweep())
    spawned: list[tuple[str, float]] = []
    infos = []
    for i in range(args.n):
        info = mgr.spawn(f"item-{i}")
        infos.append(info)
        spawned.append((info.id, now()))
    t_spawn = {aid: t for aid, t in spawned}

    deadline = time.monotonic() + args.max_wall
    ids = [aid for aid, _ in spawned]
    seen: dict[str, object] = {}
    progress_at = time.monotonic()
    while time.monotonic() < deadline:
        for aid, info in list(mgr._agents.items()):
            seen.setdefault(aid, info)
        settled = sum(1 for aid in ids if aid in seen and seen[aid].done)
        if settled >= args.n:
            break
        if time.monotonic() - progress_at > 30:
            progress_at = time.monotonic()
            print(
                f"[progress t={now():.0f}s] settled={settled} seen={len(seen)} "
                f"queue={len(mgr._queue)} running={mgr._running_count} "
                f"in_startup={mgr._startup_population()}",
                file=sys.stderr,
            )
        await asyncio.sleep(0.05)
    mgr_seen = seen
    stop.set()
    sweeper.cancel()
    await asyncio.gather(sweeper, return_exceptions=True)

    rows = []
    for aid, ts in spawned:
        info = mgr_seen.get(aid) or mgr._agents.get(aid)
        r = recs.get(f"subagent:{aid}", Rec())
        rows.append(
            {
                "id": aid,
                "t_spawn": ts,
                "t_enter": getattr(r, "t_enter", None),
                "t_start_sem": getattr(r, "t_start_sem", None),
                "t_cold": getattr(r, "t_cold", None),
                "t_spawned": getattr(r, "t_spawned", None),
                "spawn_init": getattr(r, "spawn_init", None),
                "session_new": getattr(r, "session_new", None),
                "exec_started_rel": (
                    (info._exec_started - t_wall0) / S
                    if info is not None and getattr(info, "_exec_started", None)
                    else None
                ),
                "t_gate": getattr(r, "t_gate", None),
                "t_session": getattr(r, "t_session", None),
                "t_first": getattr(r, "t_first", None),
                "t_done": getattr(r, "t_done", None),
                "done": bool(info and info.done),
                "outcome": getattr(info, "outcome", "") if info else "missing",
                "error": (getattr(info, "error", "") or "")[:160] if info else "",
                "reaped": aid in reaped_at,
                "reap_stage": reaped_at.get(aid, {}).get("stage", ""),
                "fail_stage": getattr(r, "fail_stage", ""),
            }
        )

    def pct(xs, p):
        xs = sorted(xs)
        if not xs:
            return None
        k = min(len(xs) - 1, max(0, int(round(p / 100 * (len(xs) - 1)))))
        return round(xs[k], 1)

    queue_wait = [r["t_enter"] - r["t_spawn"] for r in rows if r["t_enter"] is not None]
    to_start = [r["t_session"] - r["t_spawn"] for r in rows if r["t_session"] is not None]
    completed = [r for r in rows if r["t_done"] is not None and not r["reaped"]]
    summary = {
        "scenario": args.name,
        "n": args.n,
        "cap": args.cap,
        "gate_G": args.gate,
        "in_startup_bound": mgr._startup_cap(),
        "cold_start_admission": rt_mod._COLD_START_MAX_CONCURRENT,
        "start_sem": args.start_sem,
        "stagger_s": args.stagger,
        "startup_deadline_s": args.deadline,
        "dists": {
            "spawn_init": args.spawn_init,
            "session_new": args.session_new,
            "ttft": args.ttft,
            "turn": args.turn,
        },
        "budgets": {"initialize": args.init_budget, "session_new": args.session_budget},
        "completed": len(completed),
        "success_rate": round(len(completed) / args.n, 3),
        "reaped_startup_timeout": sum(r["reaped"] for r in rows),
        "reap_stages": {
            s: sum(1 for r in rows if r["reap_stage"] == s)
            for s in sorted({r["reap_stage"] for r in rows if r["reap_stage"]})
        },
        "typed_start_failures": {
            s: sum(1 for r in rows if r["fail_stage"] == s and not r["reaped"])
            for s in sorted({r["fail_stage"] for r in rows if r["fail_stage"]})
        },
        "not_finished": sum(1 for r in rows if not r["done"]),
        "queue_wait_s": {
            "p50": pct(queue_wait, 50),
            "p90": pct(queue_wait, 90),
            "p99": pct(queue_wait, 99),
            "max": round(max(queue_wait), 1) if queue_wait else None,
        },
        "spawn_to_session_ready_s": {
            "p50": pct(to_start, 50),
            "p90": pct(to_start, 90),
            "max": round(max(to_start), 1) if to_start else None,
        },
        "makespan_s": round(max((r["t_done"] or 0) for r in rows), 1),
        "wall_seconds": round(time.monotonic() - t0, 1),
    }
    if args.rows:
        with open(args.rows, "w") as fh:
            json.dump(rows, fh, indent=1)
    await mgr.shutdown() if hasattr(mgr, "shutdown") else None
    return summary


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--name", default="scenario")
    ap.add_argument("--n", type=int, default=120)
    ap.add_argument("--cap", type=int, default=64)
    ap.add_argument("--gate", type=int, default=2)
    ap.add_argument("--start-sem", type=int, default=4)
    ap.add_argument("--stagger", type=float, default=0.25)
    ap.add_argument("--deadline", type=float, default=120.0)
    ap.add_argument("--reaper-interval", type=float, default=60.0)
    ap.add_argument("--wall-clock", type=float, default=1800.0)
    ap.add_argument("--init-budget", type=float, default=30.0)
    ap.add_argument("--session-budget", type=float, default=90.0)
    ap.add_argument("--spawn-init", default="uniform:2,5")
    ap.add_argument("--session-new", default="lognormal:10,0.5,2,80")
    ap.add_argument("--ttft", default="uniform:2,10")
    ap.add_argument("--turn", default="uniform:30,90")
    ap.add_argument("--scale", type=float, default=0.05)
    ap.add_argument("--seed", type=int, default=1)
    ap.add_argument("--max-wall", type=float, default=1800.0)
    ap.add_argument("--rows", default="")
    args = ap.parse_args()
    summary = asyncio.run(run(args))
    print(json.dumps(summary, indent=1))


if __name__ == "__main__":
    main()

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a routed-roster test pins provenance over event kind at the startup exit

  • span=10162e9148b0 weakening the runtime_global test to an event-kind check survives the tests -- fixed in d5c0e98

Fixed: test_a_routed_roster_frame_is_the_sessions_own_answer yields the roster kind with runtime_global False (the routed KAS lifecycle shape) and asserts the run leaves startup and issues the running mark.
Verified by mutation: with the inner-loop predicate replaced by _ev.kind != EVENT_SUBAGENT_LIST, that test fails and the fanned-out-frame test still passes, which is the gap the finding named.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=7f9fc3c4dbaf the description promises fixes this head does not contain -- fixed (description)

Fixed: a parallel session rewrote the description to describe an implementation that was never pushed here. The description now describes only this diff: the first-answer startup exit, the provenance-gated running mark, the park stamp, and the release-path records.
The deadline derivation and reap-phase naming are not in this PR; the fan-out measurement (issuecomment-5843528861) shows why they matter, and they are left to a follow-up.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=4450cd03d0a0 item 7: the description claims deadline-derivation, reap-phase and freeze/thaw changes the diff lacks -- fixed (description)

Fixed as the Subtraction asks: the description no longer claims them. It covers only the five fixes this diff ships; the deadline and reap-phase gaps stay open and are reported on the thread with the fan-out numbers, not listed as shipped.

@bolichen97
bolichen97 force-pushed the fix/subagent-startup-progress-stamp branch from 505ce68 to d5c0e98 Compare September 26, 2026 05:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Adversarial review of head d5c0e984867f — run from a native Windows host (Windows Server, Python 3.12, KiroCrew dashboard). Five parallel review angles (invariant correctness, asyncio races, Windows/cross-platform semantics, test strength, spec consistency) plus an arbiter that re-verified every finding against the source; I then independently confirmed each one below, including the two test claims by mutation on this machine. Verdict: not merge-blocking. No Windows-specific defect: the diff touches only in-memory startup bookkeeping (no subprocess, path, encoding or signal surface), and every clock it reads is time.time() on both sides of each comparison. Four non-blocking items:

1. P3 — two fanned-out event kinds still carry no provenance (src/kiro_crew/acp/session_handle.py:3850 and :3852)
EVENT_COMPACTION_STATUS and EVENT_CLEAR_STATUS are yielded without runtime_global=, so they default to False (acp/types.py), while the roster (:3891) and the three MCP kinds (:3994/:4006/:4028) propagate msg.fanout_no_owner / not _owns_mcp_frame(msg). A co-tenant's sessionless compaction/clear broadcast therefore reaches the new gate at run.py (if not _ev.runtime_global:) as if it were this session's own answer and calls ensure_running_marked + _leave_startup. Verified unreachable as a correctness defect on this head — a frame is marked fanout_no_owner only on a >1-queue runtime (runtime.py:4370), every co-tenant on such a runtime has _pid = runtime.pid set at _bind_shared_handle before its stream, and both _in_startup and _is_startup_stalled require _pid is None — so it is a consistency gap, not a bug. Suggested: runtime_global=msg.fanout_no_owner on both yields (the branch already computes owns_frame from the same field), so every fanned-out kind sits on the one provenance test this PR introduces, and the gap cannot open if a PID-less start ever co-tenants.

2. P3 — two "red-first" claims are not true (test/test_runloop_integration.py::test_a_throttle_before_the_first_frame_ends_startup_at_the_park; test/test_subagent_startup_pressure.py::TestStreamOpenIsNotProgress::test_a_completion_withheld_for_recovery_is_an_answer)
The PR body says the second "fails on the previous head" and the Opus disposition says the first "fails without the park stamp". Reproduced here: with only the src/ hunks of this PR reverted to the merge-base (694f61983e) and the tests kept, both stay green (2 passed), while test_a_stream_that_opens_and_yields_nothing_is_still_reaped correctly goes red in the same run. Mechanism: the old head's unconditional stamp at the former run.py:1818 fired before the async for, i.e. before the throttle is raised inside the generator and before the withheld completion is consumed, so every assertion those two tests make already held. The behaviour they cover is correct; only the claims are overstated. Either reword them as coverage, or add a per-hunk negative control (remove only the _leave_startup call in _yield_for_dependency / only the raw-frame stamp) so each test is red against its own hunk.

3. P3 — one spec paragraph left unqualified (docs/system-specs/modules/subagent.md:970, "Durable row timing")
Still says the row becomes running at "the run's FIRST stream event (ensure_running_marked, …)". The code now gates ensure_running_marked on not _ev.runtime_global, and this PR's own Reaper-Loop bullet and the ensure_running_marked docstring already say so; this paragraph is the one place in the PR that still describes the unqualified pre-change behaviour. Suggested: "…at the run's first stream event ADDRESSED TO THIS SESSION (a runtime_global co-tenant broadcast does not)".

Verified OK (so nobody re-checks): _admit_released_start three-outcome refactor has no await between the post-future guard and the return, and "ended" is exactly the set of sticky flags the release-phase pump sets before set_result(False); _log_spawned now runs only on "admitted" and the auto-approve path's own _log_spawned call is mutually exclusive with it; slot release stays on the one-shot token (no leak, no double decrement); _leave_startup is idempotent and reset per execution, so stop-recovery and cancel-recovery re-enter and re-exit startup correctly; the raw-frame stamp precedes the withhold; PID precedes the stream on both AcpRuntime paths (matches TestAPidEndsStartupBeforeTheStream); no history narration on added lines.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Independent adversarial review — 🟡 CONCERNS (no blocker)

Reviewed head d5c0e984867f00f9cd18e582dce2e3fafe6fab1c against base 694f6198. This branch was authored and its tests were run on Windows; this review ran on macOS (Python 3.12), six independent read-only lanes (startup exit, admission release, watchdog/bound, concurrency + platform, test vacuity, body truth) plus a synthesis pass, each finding re-checked by hand on the head. The 12 new/changed tests were run individually and serially on macOS: 12 pass. No crash, hang, leak, security issue or wrong SEL row was found; the in-startup bound, slot/task/queue/future accounting and the runtime_global guard all hold as described.

Five advisory findings, ordered by weight. None blocks the merge on its own, but 1–3 are real interleavings the new tests do not reach.

1. A reap during the admission wait can still lose the "waiting to be admitted" text; the new ended branch is not executed by any test

src/kiro_crew/subagent_manager/terminal.py:493, 556-558, 577 · src/kiro_crew/subagent_manager/admission/pump.py:897-901, 945-957

_force_reap sets _reap_started, then awaits sessions.reset, and only after that await reads release_parked = info._start_release is not None and info._exec_started is None (terminal.py:577). reset suspends on owner._lock whenever another session's reset holds it across record_session_ended (routine under fan-out). In that window the 1 s repump or any drain runs the pump's ended branch and resolves the waiter's future False; the waiter's finally clears _start_release (pump.py:901). When the reap resumes, release_parked is False and the record falls through to the generic Reaped after Ns (exceeded Ns deadline) at terminal.py:617-618 — not the "never started, still waiting to be admitted" text the body, the spec (subagent.md, "a reap names the interrupted wait") and the comment at pump.py:819-820 promise. SEL row (reaped), slot and queue accounting stay correct; user stop is unaffected. On main the same interleaving also wrote "spawn rejected", so this PR narrows the race but states the outcome as a guarantee.

Why the tests miss it: in test_a_reap_while_waiting_is_announced_as_a_reap_not_a_rejection (test_subagent_startup_pressure.py:544) sessions.reset is an AsyncMock, which never suspends under asyncio.wait_for on 3.12, so no pump pass can run inside the reap and the assertions are satisfied by the pre-existing cancel path — on head and base alike. Line coverage of the three release-path tests shows pump.py:822 (return on outcome != "admitted"), :913 (return "ended") and :957 (fut.set_result(False)) never execute.

Minimal fix: capture release_parked (and approval_parked) next to info._reap_started = True at terminal.py:493, before the first await — both predicates only go True→False afterwards, the same argument the existing comment at terminal.py:566-575 makes for approval_parked. Test: give reset one real yield (async def reset(*_): await asyncio.sleep(0)) and call _drain_queue() while it is suspended, then assert the error text and that _log_spawned was not called.

2. admission_closed is checked before the end markers, so a stopped/reaped run can still be announced as a rejection

src/kiro_crew/subagent_manager/admission/pump.py:854-858 vs :912

_admit_released_start_impl returns "admission_closed" at :858 without consulting done / user_stopped / reaped / _reap_started; that check exists only at :912, after the wait. Scenario: user Stop while the run sits on the approval prompt → cancel_impl sets user_stopped, _force_reap sets _reap_started and suspends in reset (as in 1); the approval resolves in that window with admission closed → caller writes done=True, error="spawn rejected: ..." and a rejected SEL row (pump.py:807-813); _force_reap then writes its own reaped row. One run carries both rejected and reaped, and a completion with user_stopped=True plus a rejection error, which cancellation.py:1001-1005 says a neutral stop must not have. Pre-existing on main, but the tri-state this PR introduces is the natural place to close it.

Minimal fix: move the info.done or info.user_stopped or info.reaped or info._reap_started → "ended" check above the admission_closed check.

3. The provenance predicate is narrower than the property claimed: three fanned-out frame kinds still read as this session's own answer

src/kiro_crew/acp/session_handle.py:3828, 3850, 3852, 3858 vs :3891, 3994, 4006, 4028 · consumer src/kiro_crew/subagent_manager/run.py:1650-1654

The compaction branch computes owns_frame = not msg.fanout_no_owner (:3828) but the yields for EVENT_COMPACTION_STATUS (:3850), EVENT_CLEAR_STATUS (:3852) and EVENT_AGENT_SWITCHED (:3858) do not carry runtime_global, while the roster (:3891, runtime_global=msg.fanout_no_owner) and MCP yields (:3994/4006/4028) do. On a shared runtime, a co-tenant's sessionless _kiro.dev/compaction/status fanned out before this subagent's first own frame therefore hits run.py:1650 with runtime_global=False, marks the durable row running and calls _leave_startup. The watchdog/bound are shielded because the shared path already holds runtime.pid, and main marked running on every first frame anyway — so this is an incomplete narrowing, not a regression — but the body ("a co-tenant's broadcast does not advance it") and the new spec sentence equating runtime_global with fanout_no_owner are false for these kinds. Unverified: whether the backend emits these frames without sessionId on the wire (the fixture notifications.jsonl carries sessionId on all three and is labelled synthesized; the owns_frame guard at :3813-3835 was written for the sessionless case).

Minimal fix: pass runtime_global=msg.fanout_no_owner on the three yields, and add compaction to the fanned_out parametrize list of test_a_co_tenants_fanned_out_frame_neither_starts_nor_marks_the_run (test_subagent_startup_pressure.py:1099, currently roster and mcp_registration only).

4. Moving _log_spawned also moves the on-disk agent folder — an unlisted behaviour change

src/kiro_crew/subagent_manager/admission/pump.py:825, 1005-1008 · src/kiro_crew/subagent_persistence.py:889-906, 1038-1055

create_agent_folder is the first act of _log_spawned_impl. A run reaped or stopped while waiting for the in-startup bound now has no agent.json when _force_reap calls _write_tombstone (terminal.py:626): write_tombstone builds from read_state(agent_id) or {}, so the folder holds only a tombstone.json with empty task, agent, parent_session. Second face: before this PR such a run was visible across a gateway restart as a never-run running folder (list_orphans); after it the wait is simply gone, like an unapproved spawn. Probably an improvement, but the body describes _log_spawned as metric+audit only and the compatibility section does not list it. The release-path tests mock both _log_spawned and _write_tombstone and cannot observe either face.

Minimal fix: pass task and parent_session from info into _write_tombstone (it already accepts extras), or state in the body that the agent folder now exists only for admitted starts and what that means for orphan visibility.

5. Docs and comments still describe the old startup exit in places the diff did not touch

  • src/kiro_crew/docs/dynamic-subagent-sizing.md:164 — "no first provider stream" is the exit this PR removes; the edited line is 25 lines below.
  • docs/system-specs/modules/subagent.md:970 — "becomes running at the run's FIRST stream event"; head gates this on not _ev.runtime_global.
  • docs/system-specs/modules/subagent.md:1004 and src/kiro_crew/subagent_manager/monitoring.py:1019-1020 — "the deadline measures time spent starting with a permit held": the permit is released when session/new answers (acp/runtime.py:6185); for a PID-less start the clock now runs to the first session-addressed frame, which the new bullet at :1003 itself calls "a first-output budget". Bullets 1003 and 1004 now contradict each other.
  • docs/system-specs/modules/subagent.md:1003 — "a stream that opens and hangs keeps the run in startup" holds only before the first answer or dependency park; after the park stamps (run.py:2690) and the retry re-issues the prompt, a hang on the second stream is outside _in_startup and only _default_timeout catches it.
  • Stale "first stream" comments the body says "name the same exit": pump.py:513-514, 873, subagent.py:1968, 2818, admission/gate.py:1294, admission/waits.py:96, run.py:962.
Nits and body divergences (no action required)
  • Adaptive-controller progress floor (adaptive/controller.py:714, 723-725) now sees one stamp per execution instead of per-turn; at most one tick per turn boundary, and on a coarse clock the stamp (run.py:1654) and first _touch_activity (run.py:1861) can tie and defer the first tick by one frame. Unlisted under compatibility; test_adaptive_controller.py uses SimpleNamespace fixtures and cannot see it.
  • Reap text Failed to start within {N}s (no runtime launched, no turn produced) (terminal.py:615-616, monitoring.py:945-948) asserts more than the predicate knows (_pid is None and _first_stream_started is None and turns == 0); no shipped provider reaches it.
  • _leave_startup docstring (subagent.py:3942-3947) names _run_inner as the second call site; it is _yield_for_dependency_impl (run.py:2690).
  • Every PID-backed start pumps the queue twice: _leave_startup (subagent.py:3957-3958) does not test _pid. Harmless.
  • acp/runtime.py:4365-4370 marks a sessionless broadcast fanout_no_owner only when len(_queues) > 1, so a lone session's ownerless roster frame counts as "addressed to this session". Wording only.
  • test_subagent_startup_pressure.py:566 asserts a value the test itself set at :558; :886 reads _first_stream_started after _run_inner returns, so it pins presence, not timing.
  • Body: "fails on the previous head" for test_a_completion_withheld_for_recovery_is_an_answer and test_a_throttle_before_the_first_frame_ends_startup_at_the_park is true only against this PR's own prior push; both pass on main by source reading (main stamped before the stream loop). Say "prior push of this PR".
  • Body: "Documentation updated" is incomplete (item 5). "Backwards compatibility" omits the controller floor and the agent-folder timing (item 4). The headline blind spot is not reachable by any shipped provider (both publish a PID before the stream); the body concedes this only in the manual-verification footnote.
  • Body: the manual-verification table names "time-scaled x0.1" while the pasted harness defaults to 0.05; the numbers match the pasted JSON to rounding.
Windows vs macOS
  • None of the new tests carries a skipif, IS_WINDOWS/IS_POSIX guard or path-separator assertion; all 12 pass serially on macOS.
  • PID-before-stream holds on every OS: AcpClient._spawn assigns self._pid = self._process.pid right after create_subprocess_exec (acp/client.py:9274); the Windows CREATE_SUSPENDED flag (:9259) does not delay Process.pid. The Windows-only ensure_ready dead-process branch (:10333-10342) clears _pid via _reset_state() but respawns and re-assigns before AcpProvider.start returns, so get_pid at run.py:1535 never sees the cleared value.
  • The co-tenant tests drive a real AcpRuntime._reader_loop over an in-memory StreamReader with a mock process (test :1030-1055): no subprocess, pipe or signal path, so they cannot expose Proactor-vs-selector differences. The PID-timing claims rest on code reading, not on these tests.
  • Item 1's window is owner._lock contention on both platforms (3.12 wait_for awaits inline; the project floor is 3.12).
  • All startup deadlines use one wall clock (time.time()); a sleep/resume or NTP step shifts them equally on both platforms. Pre-existing.

Method: six independent read-only review lanes plus one synthesis pass; each ADVISORY item above was then re-verified by hand against the head source. "Passes on main" statements are derived from reading the base source, not from executing the tests against it.

@bolichen97
bolichen97 force-pushed the fix/subagent-startup-progress-stamp branch from a9e24b6 to 947e587 Compare September 26, 2026 23:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-progress-stamp branch from 947e587 to d2dd9db Compare September 26, 2026 23:15
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/subagent.md, src/kiro_crew/docs/dynamic-subagent-sizing.md, src/kiro_crew/subagent.py and subagent_manager/monitoring.py (the startup deadline is now sized from the start budgets). The one conflict, the startup-deadline bullet in subagent.md, keeps main's budget sentences and this branch's exit wording. Head d2dd9dbc73 on 962247d7a5.

Reply to the macOS review, item by item, against head d2dd9dbc73:

1. Reap loses the "waiting to be admitted" text. Fixed. _force_reap now captures approval_parked and release_parked right after _reap_started = True, before the first await; both only go True→False after that point. New test test_a_reap_names_the_admission_wait_even_when_the_pump_ends_it_mid_reap: sessions.reset really yields and runs _drain_queue(), and the test asserts that the pump cleared _start_release inside the reap. The error still names the admission wait, and _log_spawned never fires for the reaped start. This test drives the pump's "ended" branch.

2. admission_closed checked before the end markers. Fixed. _admit_released_start answers "ended" for done / user_stopped / reaped / _reap_started before it reads the admission gate. New test test_a_stop_during_the_prompt_is_not_turned_into_a_rejection: admission closes and the prompt resolves inside a user stop's teardown. No rejected outcome is written, no "spawn rejected" error, and nothing starts.

3. Compaction / clear / agent-switch without runtime_global. Already fixed on the head you reviewed after d5c0e984. The three yields carry runtime_global, and so do the steer echoes. test_dispatch_carries_provenance_onto_every_fanned_out_notice covers every kind through the real reader loop. This push also adds compaction to the fanned_out parametrize of the real-runtime co-tenant test.

4. Agent folder timing. Stated rather than changed. The body's compatibility section now says a start ended before admission leaves no agent folder (a persistent one leaves only its tombstone), which is the same record a declined prompt leaves, so it is not orphan-visible after a restart. The spec's release-path paragraph says the same. test_a_temporary_start_reaped_while_waiting_leaves_nothing_on_disk runs the real _write_tombstone.

5. Stale wording. Fixed at every site listed:

  • dynamic-subagent-sizing.md:164
  • subagent.md durable-row timing
  • pump.py:513-514, 873
  • subagent.py:1968, 2818
  • admission/gate.py:1294
  • admission/waits.py:96
  • run.py:962

The "with a permit held" contradiction is gone from both subagent.md and monitoring.py. The clock now reads "from gate exit until the start's exit (a runtime PID, or for a start that publishes none its first answer)". The hung-stream sentence is now scoped: once per execution, and after a first answer or a park a later hung stream is bounded by the ordinary deadline.

Nits.

  • The _leave_startup docstring now names _yield_for_dependency as the second call site.
  • The body's scale note now says --scale 0.1 (the harness default is 0.05).
  • The in-startup bound's wake edges are unchanged in kind, and the compatibility section now states that.
  • Controller floor, the reap text's wording, and the double pump for PID-backed starts: no change. The reap text is unreachable by shipped providers, and the double pump is harmless, as you note.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Review: leaving startup on the first owned frame

Verdict: in favour. This fixes a real false negative that I hit repeatedly today, on this exact code path. My one substantive request is not about the design but about diagnosability — see point 2.

Where these opinions come from, and what they are not

A Windows opinion is worth little unless you know which host, which start path, and what went untested, so:

Basis Environment What it covers What it does not
Measured (strong) Windows Server 2025, build 26100, LongPathsEnabled=1, single-host gateway, host slot cap 12 ~660 spawn_run dispatches today at batch sizes 7→12→18→27→36→60, all on the session-sharing path (no model / reasoning_effort override, so no dedicated process) any POSIX platform
Measured (strong) same real stall shapes: a pipe deadlock wedged ~9000s with no reap at all; an MCP call wedged ~5400s and was reaped as 3/3 in-place recovery attempts exhausted; 180-minute platform timeouts; user stops; queue-starvation refusals at 60-wide the approval-prompt path — auto-approve was on throughout, so no run of mine ever parked on a spawn-approval prompt
Code reading (moderate) local main + this PR at d2dd9db _is_startup_stalled's conjunction, both _pid assignment sites, the _leave_startup call sites I have not executed this branch

Why the platform matters rather than being incidental: this PR's own safety argument is that "every AcpRuntime-backed start records one [PID] before its stream opens", i.e. PID-backed starts leave startup on the PID and _first_stream_started is not load-bearing for them. So the flag is load-bearing precisely on the no-PID paths — and on Windows there is no fork, every kiro-cli child is a real CreateProcess, while shared-session dispatch (~200ms, no new process) is exactly the path those ~660 runs took. That is why I think this evidence lands on the lines you changed rather than being a general "I use subagents a lot".

1. The false negative this closes is one I measured

Before this change, info._first_stream_started = time.time() was assigned unconditionally ahead of async for event in _stream_with_transient_retry(): — so a run counted as out of startup the moment the stream opened, with zero frames received. _is_startup_stalled (monitoring.py:803-808) needs all four of:

turns == 0  and  _pid is None  and  _first_stream_started is None  and  past deadline

The third conjunct was satisfied early, so a run that opened a stream and then wedged forever could never be caught by the startup watchdog.

That matches what I saw: in one wave nine agents each sat ~9000s with no startup reap whatsoever and needed manual stops; the ones that were eventually caught in later waves were caught by the much later idle-based detector at ~5400s. "The stream opened" never did imply "it answered" — which is your title — and on this host that gap was worth 90–150 minutes per affected agent.

2. The failure direction flips, and the premise behind it is conditional

After this change, a no-PID run's only way out of startup is a frame with not _ev.runtime_global. So if frame provenance is ever wrong in the other direction — a frame that really is this session's arrives carrying fanout_no_owner — _leave_startup never fires, _first_stream_started stays None, and a healthy agent gets reaped at _startup_deadline.

The direction of failure has changed: before, a stall was invisible (user waits); after, a healthy run can be killed (user loses work). The second is the worse one to get wrong.

And the premise that bounds the risk is conditional in the code:

# run.py:2916  (shared-session path)
if runtime.pid:
    info._pid = runtime.pid

# run.py:1271
pid = self._manager._sessions.get_pid(session_key)
if pid:
    info._pid = pid

Both are if <truthy>. Whenever runtime.pid / get_pid() yields 0/None, the run has no PID and now stakes its life entirely on frame provenance. The docstring says "every AcpRuntime-backed start records one"; the code says "records one if the runtime reports one". Those are different claims — worth either aligning the wording with the code's actual strength, or making the invariant genuinely hold.

This is not hypothetical on my host: I had agents legitimately working 90+ minutes on a single item (two hit the 180-minute cap wedged on file I/O). Under 27-way fan-out on one shared runtime, a misattributed frame turns exactly that population into false reaps.

Concrete, cheap request (no semantic change): when a startup reap fires, record how many frames this start saw and how many were runtime_global. Today a false reap and a genuine wedge are indistinguishable after the fact. I spent eleven fan-out waves chasing a stall whose real cause was a single pointless first tool call, and what made it slow was precisely that the failure records did not carry enough to separate the competing hypotheses. One counter in the reap reason would have collapsed that.

3. The three-valued _admit_released_start is right, but I cannot back it with measurement

Returning "admitted" / "admission_closed" / "ended", and checking info.done or user_stopped or reaped or _reap_started up front, fixes something real: a stop or reap landing while an approved start waits for the in-startup bound previously produced spawn rejected: the gateway closed admission — a reason that is simply untrue, and a terminal record competing with _force_reap's own.

I want to be explicit that I did not verify this: auto-approve was on for all ~660 runs, so none of them ever parked on a spawn-approval prompt and none traversed this path. The 18 runs a user stopped today all reported Stopped by the user before completing, which is the correct text — evidence that they took a different path, not evidence that this one is sound.

Separately: moving _log_spawned(info) behind outcome == "admitted" so the funnel counts only starts that will actually run is a real improvement for anyone measuring a fan-out. I derived progress today by counting output files on disk precisely because I did not trust any "how many were dispatched" number; this change addresses that class of mistrust.

4. _write_tombstone's early return — I can't judge this one

The reasoning for if info.memory_mode != "persistent": return (a run that ended before its folder was seeded has neither live state nor a recorded mode, so the run's own mode has to decide) reads as self-consistent. But all my runs passed include_memory=false, which is a different axis from memory_mode, and I never established which mode they resolved to. So: reads fine, not verified.

5. Propagating runtime_global to more event kinds — in favour, and my configuration is the reason

Marking the steer echoes, compaction, clear and agent_switched notices is clearly right: those are exactly what a shared runtime fans out to co-tenant sessions. My configuration was 27 subagents on one shared runtime, and in that shape treating another tenant's compaction notice as your own activity directly pollutes the activity clock. The added types.py note — that a session/update names its session by protocol and is routed, so its events leave the flag clear — is the load-bearing distinction and is worth keeping as prose.

Summary

In favour of merging. The false negative it closes is one I measured, at a cost of 90–150 minutes per affected agent.

The only thing I would like addressed first is point 2 — not the design, but leaving a diagnosable trace on the reap path now that a no-PID run's survival depends on frame provenance.

Reviewed at head d2dd9db. Empirical claims come from a Windows Server 2025 (build 26100) single-host gateway running shared-session dispatch; the approval-prompt path and all POSIX behaviour are untested by me and flagged as such above.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
A run without a runtime PID leaves the in-startup population
(_in_startup, _is_startup_stalled, the adaptive controller's progress
floor) when _first_stream_started is set. Stamping it when the
provider's stream() was entered let a stream that opens and never yields
read as progress: the watchdog stopped watching a start that was not
starting, and _note_startup_progress woke the pump for it.

SubagentManager._leave_startup now stamps the marker, once per
execution, at the two moments the backend is shown to have answered the
prompt: the first frame out of the stream that is addressed to this
session -- taken on the raw frame, so a completion withheld for in-place
recovery counts -- and a dependency verdict on the prompt, the step that
also takes the durable row from starting to running. A runtime_global
frame, which a shared runtime fans out to every co-tenant, ends neither
the startup state nor the stream's running mark: both now key on one
provenance test. The shared-runtime dispatch loop marks the compaction,
clear and agent-switch notices and the steer echoes it builds from a
fanned-out frame runtime_global, as it already did for the roster and
the MCP notices, so the flag covers every notice kind such a frame
yields; a session/update names its session by protocol and is routed. A
start parked on a provider throttle before its first frame is therefore
out of startup for the whole wait, which its dependency scope bounds.

Every AcpRuntime-backed start records a runtime PID before its stream
opens (get_pid on the dedicated path, runtime.pid at
_bind_shared_handle), so for those the startup window closes at the PID;
a test pins that ordering, and the spec names the backends whose first
frame is model output.

_spawn_with_approval records a run as spawned (_log_spawned: metrics,
audit, agent folder) only once _admit_released_start has admitted it, so
a start refused or ended while it waited for the in-startup bound is
never counted as started.

_admit_released_start answers one of three outcomes instead of a
boolean: "admitted" (run), "admission_closed" (gateway admission closed
at release -- the caller writes the rejection, releases the slot, audits
rejected / admission_closed and announces), or "ended" (a user stop or a
reap landed while it waited -- the caller writes nothing, because
_force_reap owns that record: a stop is neutral and a reap names the
interrupted wait). A stop or a reap is therefore never announced as
"spawn rejected".

A start ended while it waited for admission was never recorded, so no
live-run state or state.json marks it incognito or temporary;
_write_tombstone decides from the run's own memory mode, and a
non-persistent run ended before admission or at a declined prompt leaves
nothing on disk.
@bolichen97
bolichen97 force-pushed the fix/subagent-startup-progress-stamp branch from d2dd9db to 7c3b94a Compare September 26, 2026 23:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Thanks. Replies against head 7c3b94a15b, which is rebased on 4d6490860c (main moved in CI and conftest only; no overlap).

2. Diagnosable startup reap: done. The startup reap's error and the reaper's warning now name the runtime_global frames the start received while still in startup (_startup_cotenant_frames, reset with the marker for each execution). The error reads Failed to start within Ns (no runtime launched, no turn produced; K co-tenant frame(s) received, none addressed to this session). That separates a start that saw only another tenant's traffic from one whose stream stayed silent. Pinned by test_startup_reap_names_the_co_tenant_frames_it_received, and the real-runtime co-tenant test asserts the count is 1 after the fanned-out frame and stays 1 once the run's own frame ends startup.

On the misattribution you describe, where a frame that is really this session's arrives with fanout_no_owner: the reader loop sets that flag only when a frame names no session it can route and fans out to more than one queue. A frame carrying this session's sessionId is routed to it alone and never gets the flag. That covers every session/update (the session is named by protocol) and the session-named _kiro.dev notices. So the frames that could be misread are the sessionless ones, and those carry no evidence of which tenant they belong to. The new count is there to make that case visible if it ever turns up.

PID wording: aligned with the code. The field comment and the spec now say an AcpRuntime-backed start records the PID its runtime reports before the stream opens. A runtime reports one from spawn until its process is reset, and a session is created only on a spawned runtime. The if pid: guards are what the wording now describes.

1. A correction to the measurement's attribution. Your ~660 runs were all on the session-sharing path. On that path _bind_shared_handle sets info._pid = runtime.pid at bind, before the stream opens, and _is_startup_stalled requires _pid is None. So neither main nor this branch had the startup watchdog governing those runs once they were bound. The 9000s wedges you saw were not a false negative of the stream-open stamp. They were runs already out of startup at the PID, so only the ordinary deadline and the idle/stall detector could catch them, as the ~5400s catches did. This PR doesn't change that path: TestAPidEndsStartupBeforeTheStream pins that a shared start is out of startup at bind. The stream-open false negative it closes is real, but it matters only for a start with no PID, which no shipped AcpRuntime path produces. The body says that under "Which starts this governs". If those wedges were pipe or MCP stalls mid-turn, they belong in their own issue against the stall detector's thresholds.

3, 4, 5. Agreed; no change. The approval-prompt path is covered by the release-path tests, including the two added for the macOS review (a reap mid-pump and a stop with admission closed).

@bolichen97

Copy link
Copy Markdown
Contributor Author

Checks on the two PASS headlines at head 7c3b94a15b:

Design: does every waker that resolves _start_release with False also run _force_reap? Yes. There is one such waker, the pump's ended branch in _release_admitted_start_impl. It resolves False only when info.done, user_stopped, reaped or _reap_started is set, and each of those has an owner that writes the record:

  • reaped and _reap_started are set only inside _force_reap.
  • user_stopped is set in cancel_impl, which then calls _force_reap, and in the stage-boundary revocation, whose caller cancels every run it marks through cancel → _force_reap.
  • done means a terminal record already exists.

The info is None / fut is None / fut.done() cases resolve nothing. The waiter also re-checks the same markers after the await before it answers "ended", so an unowned "ended" cannot be returned.

First principles: can a selectable backend reach the first-answer edge with a slow first token? Not on any shipped path. Every selectable backend (kiro-cli, KAS, claude-agent-acp) runs through AcpRuntime, which records the runtime PID at bind (shared path) or after get_or_create (dedicated path), before the stream opens (TestAPidEndsStartupBeforeTheStream). The first-answer edge is the startup exit only for a provider that publishes no PID. Such a provider would add its own PID record at acquisition, and the spec states that scope.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

macOS re-review at head 7c3b94a15b (base 4d6490860c)

Second pass on the same branch (authored on a Windows host; this review ran on macOS / Python 3.12). Scope: the range-diff from the previously reviewed head d5c0e984 to 7c3b94a15b, the item-by-item replies above, and the two PASS headlines. Two independent lanes: one verifying every claimed fix against the head and running the new tests, one attacking the frame-routing and PID premises. Every line number is on 7c3b94a15b.

Verdict: no blocker. All five items from the first review are closed on the head; the fixes are real and the new tests exercise the branches they claim to.

Earlier items, re-checked

# Item Status Evidence
1 Reap mid-pump lost the "waiting to be admitted" text Fixed terminal.py:493 sets _reap_started, :505/:508 snapshot both parked flags, no await before :557/:571; text rendered from the snapshot at :614. test_a_reap_names_the_admission_wait_even_when_the_pump_ends_it_mid_reap drives the pump inside the patched reset and asserts _start_release was cleared mid-reap plus the exact error.
2 admission_closed read before the end markers Fixed pump.py:854-855 early "ended", :859-863 gate read after, :917-918 re-check after await fut. test_a_stop_during_the_prompt_is_not_turned_into_a_rejection would fail with the old order.
3 Compaction / clear / agent-switch without runtime_global Fixed session_handle.py:4073-4083, :4110-4136, :4142, :4147-4151. Remaining unmarked constructors (:3921, :5086, :5155, :5164-5169) are on routed paths and correctly stay unmarked.
4 Agent-folder timing undisclosed Stated Body + subagent.md release-path bullet. Only create_agent_folder( call is pump.py:1013 inside _log_spawned_impl, reached only after "admitted" (:825); tombstone-only folders are skipped by list_orphans.
5 Stale wording Fixed All eight listed sites; grep -i "permit held" over src/kiro_crew and docs/system-specs hits only two unrelated files.

Tests, run serially one file at a time: test_subagent_startup_pressure.py 56 passed, test_subagent_startup_watchdog.py 13 passed, test_subagent_stall_activity_scope.py 16 passed, test_runloop_integration.py -k 'provenance or cotenant or startup' 1 passed. 0 failures.

The two PASS headlines

Design (every False-resolving waker also reaps): holds. Enumerated every writer of a terminal record for a release-parked run (_force_reap, the approval-path admission_closed branch, cancel_impl, boundary revocation, _unqueue/_settle_boundary_queue) across seven interleavings; none produces two terminal records or admits a start after _reap_started. rejected + reaped cannot both be written because both sequences are synchronous on either side of the marker.

First principles (PID before stream on every shipped start): the conclusion holds, but the supporting premise in the reply is wrong. ACP_BACKENDS_ACP_RUNTIME = {kiro, kas, codex} (agent_sdk/backends.py:1355); claude-agent-acp, opencode, pi, goose and deepseek are AcpClient-backed, not AcpRuntime-backed. They still record a PID at run.py:1537-1539 because get_or_create runs provider.start() -> ensure_ready() -> _spawn() before returning. So the real invariant is "every start whose provider spawned in start()", not "AcpRuntime-backed". Also confirmed: the reviewer's ~660 shared-session runs were out of startup at bind (run.py:3299-3300, _is_startup_stalled requires _pid is None), so the correction in the reply is right and the stream-open -> first-answer change is inert for that population.

ADVISORY (wording; none changes behaviour)

  1. "AcpRuntime-backed" under-describes the invariant (subagent.py:1881-1885, subagent.md:1008, the commit message's unconditional form). A future provider that spawns lazily from stream() is exactly the one the first-answer edge governs; the wording should let the next reader recognise it. The if pid: / if runtime.pid: guards are effectively dead for shipped providers (AcpRuntime._pid is assigned once at runtime.py:2763 and never cleared); a log line when they skip would make the no-PID population visible.
  2. "Session-named _kiro.dev notices are routed" is only true of _kiro.dev/session/update. The router reads only params.sessionId (runtime.py:4245-4251), and compaction / clear / agent-switch notices and the steer echoes are sessionless, so under fan-out a session's own notice is flagged runtime_global and counted in _startup_cotenant_frames. By design, and fail-safe (a healthy run's session/update chunks still end startup), but the counter's comment should say the count can include the run's own sessionless notices.
  3. Counter predicate vs. wording. _startup_cotenant_frames increments while _first_stream_started is None (run.py:1657-1661), not while _in_startup(info), so a PID-backed start keeps counting after leaving startup at bind. Never rendered (a startup reap requires _pid is None), so the shown number is always correct, but subagent.py:1886 and subagent.md:1008 say "while still in startup". Either add info._pid is None to the guard or word it "before its first own frame". Frames dropped by the pre-prompt stale drain (session_handle.py:1487-1497) are also never counted.
  4. Two deadline values in one reap. terminal.py:617 interpolates the live _startup_deadline; monitoring.py:952 uses _stamped_startup_deadline(info) (merged in by the rebase). After a config write the error and the warning for one reap can name different Ns. Pre-existing on the base, but this PR rewrote line 617 and did not align it.

NIT

  • The reaper's warning text (monitoring.py:947-954) is not asserted by any test; only the info.error rendering is (test_subagent_startup_watchdog.py:167).
  • A reap landing in the single loop tick between "admitted" (pump.py:919) and _exec_started finds both parked flags False and uses the generic deadline text. One record, one tick, pre-existing on main.

Windows

No platform-conditional code in the nine changed source files and the delta adds none. Both PID sites see a CreateProcess id synchronously on the Proactor loop; the Windows-only dead-process branch in ensure_ready (client.py:10646-10655) clears the client's _pid after info._pid was already copied, so a run never re-enters startup. Nothing in this delta changes what Windows does for the shared-session population.

Nothing here was reproduced under a live gateway; interleavings are traced from code. The base was not executed.

@bolichen97
bolichen97 merged commit a3b8f6c into main Sep 27, 2026
98 of 106 checks passed
@bolichen97
bolichen97 deleted the fix/subagent-startup-progress-stamp branch September 27, 2026 02:23
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants