Repository navigation
fix(subagent): mark returned results only after the reply is written - #27508
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: when the ledger is unsure it re-delivers the result rather than dropping it, the new wire fields are optional, and the main reader that loses suppression is listed with its test. Notes
[DESIGN-REVIEWED] 8e35878 Description read[DESCRIPTION-READ] a3b69b2993457defc33b6098c5c15934e9d5648b0cc9da75e567e01b8a1e6cb0 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every item comes from the reported defect (#23434) or its no-loss constraint (#23478), and each new surface has a counted consumer. Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: cron and channel parents stop paying one redundant model turn for each child a blocking
[FIRST-PRINCIPLES-REVIEWED] 8e35878 Description read[DESCRIPTION-READ] b9c5e85b68b21003199b125939ba6c569cf169231f8719e844c26cf8869b3e64 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] 8e35878 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head 05230ad. One blocking issue, which agrees with the GPT and Opus lanes.
🔴 A parked inline completion loses the leaked-process flag (slack/gateway_runtime/subagent_injection.py, _park_inline). When the result was returned inline and the parent turn still holds it, _park_inline builds the owed delivery as SubagentDelivery(info.id, info.elapsed, info.credits) and sets _delivery_queued = True. The queued tab path in the same file (around line 553) passes leaked_process=getattr(info, "_leaked_process", False) is True, but this one does not. _delivery_queued also makes terminal.py skip its own leaked_process tombstone branch. So if the child's teardown could not end its process, the eventual delivered tombstone has no flag, list_leaked_processes never returns that run, and the process keeps running with nothing tracking it.
Suggestion: build own the same way _defer_queued_delivery does, passing leaked_process=getattr(info, "_leaked_process", False) is True, and add a test where a parked inline completion carries _leaked_process through to its settled tombstone.
This PR changes 94 files, so it also needs a human maintainer's approval in any case. I have not yet read the logs of the other red checks on this head, so I am not claiming anything about them here.
05230ad to
3d89a7a
Compare
|
GPT 6.1 Review on 05230ad — disposition (fixed in 3d89a7a)
First-pass check: R1-R8 OK. |
|
Opus 5.5 Review on 05230ad — disposition (fixed in 3d89a7a)
First-pass check: R1-R8 OK. |
|
Design Review on 05230ad (PASS) — notes
|
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head 3d89a7a against 05230ad with range-diff.
Fixed: _park_inline now builds the owed delivery with leaked_process=getattr(info, "_leaked_process", False) is True, with a comment saying why. The new test_a_parked_completion_keeps_its_leaked_process_flag checks both values of the flag on the parked completion. No other hunk changed. The GPT and Opus lanes now report no blocking findings at this head.
Still red: Backend Tests (macOS) (2) fails three tests that this PR adds, all in test/test_spawn_sub_agents_spawning_attempt.py:
test_a_queued_member_that_starts_after_its_spawning_attempt_ended_holds_nothingtest_queued_members_of_one_call_are_confirmed_with_no_completion_turntest_a_queued_member_stopped_before_it_starts_is_delivered_not_lost
Each one times out in async_wait_until after 30 s, two of them waiting on _start_the_queue.<locals>._all_started. The same shard was red on the previous head too, so this looks like a real problem on macOS, not a one-off. Please check why the queued members never all start there.
This PR changes 94 files, so it also needs a human maintainer's approval.
3d89a7a to
cf36544
Compare
|
macOS Backend Tests (2) on 3d89a7a — fixed in cf36544 Three tests in |
A parent whose turn blocks in spawn_sub_agents got one redundant
completion turn per child the call had already returned, unless it was
a dashboard tab: mark-collected recorded ids only on a dashboard slot,
and the cron and channel routes asked nothing before injecting.
The tool now sends mark-collected from DeferredTool.on_answered, which
the stdio dispatcher runs after answering, with written=True only when
respond() wrote the whole frame into the client's own pipe, the
descriptor captured before any tool ran. A refused or uncaptured pipe,
a substituted error frame and a pooled backend's answer never count.
The mark names only the members whose whole block survives the
MAX_RESPONSE_LEN cut, and a mark without written=True changes nothing.
Bytes in kiro-cli's pipe are not a landed turn: a cron deadline, a stop,
a reap or a failure can end the turn before its answer is recorded, and
a turn cut without telling kiro-cli keeps its call running. So for every
parent that is not dashboard-born (parent_turns.settles_at_release), a
tab open on it or not, the gateway holds each mark, pending, only under
the attempt that spawned the child. An attempt is one turn inside one
holding of the parent's permit; its token is the session key, the
holding task and an epoch that no other attempt has. POST /api/spawn
from the parent's own tool records on the child the attempt that held
the session, with its prompt in flight
(SessionManager.turn_prompt_in_flight), when the request arrived: it is
read before the handler awaits anything, so a turn cut while the handler
awaits never lends the spawn its successor's attempt. Every member of
one spawn_sub_agents call carries that call's call_id, and a member
records an attempt only when it equals the one the call's first member
named (ParentTurnLedger.call_attempt). A call_id that is not ASCII
letters and digits, at most 32 characters, is a 400 coded
invalid_call_id. A member the admission gate queues carries the attempt,
in memory only, to the record the pump builds when it starts it
(carry_spawn_attempt). GET /api/spawn/{id} stamps a done read only under
the spawning attempt, and a mark is held only when the spawn and the
stamp both name the attempt holding now. A read or mark under any other
attempt, or after the spawning one ended, holds nothing, so the
completion is delivered: a duplicate at worst.
SessionManager.settle_turn ends an attempt: an injection's in-claim
retry and its session-not-found reload end the failed one there, and a
cron turn's one-shot CONTINUE re-prompt ends the interrupted prompt's.
Release ends the last one. Each engine notes landing where it already
decides it (note_turn_landed); a completion's own injected turn notes it
only when its stop reason landed, and the dashboard runner only for a
turn that ended on end_turn, so a person's turn in a cron job's or a
channel conversation's tab plays no second turn for its children. An
attempt's end confirms its marks when it noted landing and revokes them
otherwise, and an attempt a finished task left open is ended by the next
one opened.
The cron, channel and tab routes ask _inline_verdict; a dashboard-born
tab keeps _delivered_inline. A confirmed announce is acknowledged with
no turn; a pending one is parked with its tombstones owed and _on_done
returns at once; anything else is delivered. At the release
_settle_inline runs a parked completion through its route again: a tab
completion through the same steps _subagent_done takes, so a closed tab
hands it to the parent's own route, with its held wave siblings owed.
A parked completion whose parent was retired takes no route:
_finish_parked asks suppress_retired_delivery, the teardown check the
terminal report asks before _on_done, by its own id and every member it
carries or holds, and writes only its own marks. A landed release also
settles its children on a tab that shows the parent.
A re-routed completion of a cron parent is counted before the cron
turn's cleanup decides its reset. That cleanup
(_reset_cron_parent_when_idle) decides by who holds the session: a cron
run's own task, held in _cron_runs_holding from the start of
_cron_callback to its end, is left to that run's finalizer; another turn
keeps the session; an idle session is reset. A key is retired only when
the job's live run did not register it or is done with it
(CronService.retire_finished_session_key): each cron engine marks the
key it registered as it releases the session (finish_session_key), and
registering the key again takes the mark off, so a newer run waiting for
the permit keeps its own. A parked tab completion whose tab cannot be
read takes the parent's own route, and a counted re-route gives its
count back however it ends.
A dashboard-born tab still acknowledges on the mark, a window tracked
in #23495.
Fixes #23478
cf36544 to
8e35878
Compare
|
Opus 5.5 Review on cf36544 — disposition (fixed in 8e35878)
First-pass check: R1-R8 OK. |
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 8e35878.
The compaction-replay blocker is fixed. Before the reset in the transient-compaction branch, settle_turn(sessions, session_key) now ends the abandoned attempt, so its inline marks are revoked and the replay runs as a new attempt. The fix covers both paths:
messaging/dispatch.py, beforesessions.reset(session_key)(about line 1116).slack/handler_runtime/turn.py, before both resets in_run_turn(the compaction replay and the session-not-found replay).
New tests: test/test_channel_turns.py, test/test_session_not_found_surfaces.py, test/test_slack_handler_turn_outcomes.py. The macOS fix from cf36544 (healthy_host_memory fixture) is still present.
CI: at 02:04 UTC every AI lane failed closed on a GitHub API rate limit while reading the merge-base. These were infrastructure failures with no findings, and I reran them. On rerun, Opus 5.5, GPT 6.1, Design, First Principles, UX, Sensitive Change and Security Scope all pass. Test shards and PR Readiness were still running when I checked.
Both of my earlier change requests are now addressed. The PR changes 94 files, which is above our 20-file limit for agent approval, so a human maintainer needs to approve and merge it.
bolichen97
left a comment
There was a problem hiding this comment.
Approving at 8e35878.
Both earlier change requests are fixed.
- macOS timeouts:
test/test_spawn_sub_agents_spawning_attempt.pynow uses thehealthy_host_memoryfixture. - Compaction replay:
settle_turn(sessions, session_key)now runs before the reset inmessaging/dispatch.pyand before both resets inslack/handler_runtime/turn.py, so the abandoned attempt's inline marks are revoked and the replay starts a new attempt. New tests intest/test_channel_turns.py,test/test_session_not_found_surfaces.pyandtest/test_slack_handler_turn_outcomes.pycover it.
How the 96 files were covered (above our 20-file review limit):
- Every hunk in
src/anddocs/was read line by line. That includesparent_turns.py,session_parts/claims.py,slack/gateway_runtime/subagent_injection.py,dashboard/messaging_api/run_control.py,run_views.py,spawn.py,mcp_shared_wire.py,mcp_shared_dispatch.py,mcp_shared_calls.py,mcp_tools/spawn_parts/wave.py, the cron run registry, and every engine that now callsnote_turn_landed. - Every test hunk over 5 lines was read, including the new files
test_spawn_sub_agents_duplicate_never_loss.py,test_spawn_sub_agents_settle_at_release.py,test_spawn_sub_agents_spawning_attempt.py,test_cron_run_holding.py,test_chat_runner_notes_landing.pyandtest_subagent_injection_session_reload.py. - The remaining test edits of 1 to 5 lines were checked mechanically. Every changed line is one of these: a
_cron_runs_holding = set()fixture line, askip_if_busy=Trueassertion update, awritten: Truemark body, theon_answered(True)call, or a composition-contract signature row.
Trust boundaries checked:
mark-collectednow flags only the claimed parent's own children, and only when the mark says the frame was written.call_idis validated (ASCII letters and digits, at most 32 characters) before any spawn happens.- An attempt is named only for an internal caller whose
X-Session-Keyis the parent session, while that session's prompt is in flight. - No approval, permission or tool-allowlist path is touched.
CI: 0 red. Opus 5.5, GPT 6.1, Design, First Principles, UX, Sensitive Change and Security Scope all pass (the first run failed closed on a GitHub API rate limit and was rerun). The macOS shards, Coverage Combine and PR Readiness were still running when I checked. Auto-merge waits for them.
Cherry-picks (git cherry-pick -x), in this order: - fix(i18n): raise the i18next floor to 26.4.4 so an unclosed nesting marker cannot hang the tab (#22326) (cherry picked from commit 7efc766) - fix(desktop): make the Linux packages depend on libgbm (#28980) (cherry picked from commit 068b96b) - fix(platform): read a macOS all-zombie process group as empty in pgroup_exists (#27615) (cherry picked from commit e1b9965) - fix(platform): read a macOS all-zombie group's killpg EPERM as an emptied group (#27734) (cherry picked from commit 7d4da6b) - fix(governance): judge cron turns by their session and fail closed on unattended turns (#27745) (cherry picked from commit 3fa3e80) Conflict, hand-resolved: - #27745, src/kiro_crew/slack/gateway_runtime/cron_turn.py (both stream calls): on main the added session_key=/unattended=True lines sit next to on_resume=, which #27508 added and 0.12 lacks. Took the two added keyword arguments at both sites and left on_resume= out, so #27508 is not pulled in. Every other file matches the source.
…29386) Cherry-picks (git cherry-pick -x), in this order: - fix(i18n): raise the i18next floor to 26.4.4 so an unclosed nesting marker cannot hang the tab (#22326) (cherry picked from commit 7efc766) - fix(desktop): make the Linux packages depend on libgbm (#28980) (cherry picked from commit 068b96b) - fix(platform): read a macOS all-zombie process group as empty in pgroup_exists (#27615) (cherry picked from commit e1b9965) - fix(platform): read a macOS all-zombie group's killpg EPERM as an emptied group (#27734) (cherry picked from commit 7d4da6b) - fix(governance): judge cron turns by their session and fail closed on unattended turns (#27745) (cherry picked from commit 3fa3e80) Conflict, hand-resolved: - #27745, src/kiro_crew/slack/gateway_runtime/cron_turn.py (both stream calls): on main the added session_key=/unattended=True lines sit next to on_resume=, which #27508 added and 0.12 lacks. Took the two added keyword arguments at both sites and left on_resume= out, so #27508 is not pulled in. Every other file matches the source.
A dashboard tab whose turn called spawn_sub_agents acknowledged each returned child on the mark, so a turn the user stopped before it landed lost the result. Every parent now holds the mark pending under the spawning attempt and settles it at that attempt's release, as cron and channel parents already do (#27508): a landed turn confirms it, any other end delivers it. The specs and the brokered-delivery RFC no longer describe the removed dashboard-born exception. Builds on #28576 by @cruisercohen. Closes #23495 Co-authored-by: Matt Cohen <mattcohe@amazon.com>
…32387) A dashboard tab whose turn called spawn_sub_agents acknowledged each returned child on the mark, so a turn the user stopped before it landed lost the result. Every parent now holds the mark pending under the spawning attempt and settles it at that attempt's release, as cron and channel parents already do (#27508): a landed turn confirms it, any other end delivers it. The specs and the brokered-delivery RFC no longer describe the removed dashboard-born exception. Builds on #28576 by @cruisercohen. Closes #23495 Co-authored-by: Matt Cohen <mattcohe@amazon.com>
…32387) A dashboard tab whose turn called spawn_sub_agents acknowledged each returned child on the mark, so a turn the user stopped before it landed lost the result. Every parent now holds the mark pending under the spawning attempt and settles it at that attempt's release, as cron and channel parents already do (#27508): a landed turn confirms it, any other end delivers it. The specs and the brokered-delivery RFC no longer describe the removed dashboard-born exception. Builds on #28576 by @cruisercohen. Closes #23495 (cherry picked from commit 2a4fb38) Original author: Bolin Chen (#32387). docs/system-specs/modules/subagent.md hand-resolved: removed the dashboard-born-tab sentence this PR makes untrue and kept the rest of release's paragraph; main's #25261 sentence belongs to a main change not on release.
Moved from #17684 (fork) to a branch on this repo so CI runs on the CodeBuild fleet, per CONTRIBUTING.md. Earlier review threads and dispositions are on #17684.
Problem / Motivation
Goal: A cron or channel parent whose turn already got its children's results back from a blocking
spawn_sub_agentscall gets no second turn for them, whether or not a dashboard tab shows it, and no returned result is acknowledged until the parent turn that received it has landed.On main a cron run that called
spawn_sub_agentswith three children took four turns: turns 2-4 were each a[Subagent completion event]for a child the call had already returned (#23434).Why it matters
Every cron job and every channel conversation that uses
spawn_sub_agentspays one redundant model turn per child. Skipping them must not lose a result: a deadline, a stop, a reap or a failure can end the turn after the reply reached kiro-cli and before its answer is recorded (#23478).Not a goal
spawn_rundelivery.What changed (motivation → approach → change)
On main,
mark-collectedrecorded returned ids only on a dashboard tab's slot, so cron and channel routes injected every completion, and it was sent before the write, so a torn or cut reply still suppressed its completions._commit_settled(mcp_shared_dispatch.py) answers, then runson_answered(written)(mcp_shared_calls.py): True only whenrespond()(mcp_shared_wire.py) wrote the whole frame into the client's pipe, never for a pooled backend.wave.pymarks the members whose block survives theMAX_RESPONSE_LENcut.settles_at_release) a mark is pending only under the attempt (turn_token: key, task, epoch) that spawned the child.api_spawnrecords the attempt that held the session, with its prompt in flight (turn_prompt_in_flight), when the request arrived, read before the handler awaits anything; onecall_idper call ties each member to its first member's attempt, a queued member carries it to the record the pump builds, andGET /api/spawn/{id}stamps a done read only under it.note_turn_landed);runtime_charge.pyonly on a landed stop reason, the dashboard runner (turn_stats.py) only onend_turn.settle_turnends an attempt (a failed in-claim one before its retry or reload),releasethe last; noted confirms, anything else revokes._inline_verdict(subagent_injection.py), asked by the cron, channel and tab routes: confirmed is acknowledged, pending is parked; the rest is delivered. At the release a parked completion takes its route again, unless its parent was retired (suppress_retired_delivery). The cron cleanup leaves the run's own task (_cron_runs_holding) to its finalizer, lets another turn keep the session, resets an idle one, and retires only a key the job's live run did not register or is done with (retire_finished_session_key; each cron engine marks its key withfinish_session_keyas it releases the session).Since
0592d79154: the spawn reads its attempt when the request arrives, so a call's first member posted across a cut never records the successor's attempt; thecall_idrefusal carriescode: invalid_call_id; the dashboard runner notes a turn that ended onend_turn, so a person's turn in a cron or channel tab plays no duplicate (main played none there); a cron engine marks its key finished as it releases the session, so the last cleanup retires it even while the run's claim is live (main retired it there); a parked tab completion whose tab cannot be read takes the parent's own route instead of waiting for the next start.Residual, one duplicate turn and never a loss: a mark processed after its spawning attempt ended; an engine that notes no landing (a monitor nudge, a
hook:session, side threads, workflow agents); on a cron or channel tab, a child that finishes before the reply is written, as on main; a member spawned while no prompt of its turn is in flight; a queued member after a gateway restart or one that ends unstarted. A channel member holds its permit while it listens and notes no landing, so its parked completions wait until it ends: a delay, never a loss. A spawn with nocall_id(the CLI,spawn_run, a pre-upgradekirocrew-core) is judged by its own holding; none of them sends awrittenmark. Two windows can still lose a result, and both need kiro-cli to stop waiting for a call the server is not told to drop: a cut turn's call whose first member's POST reaches the gateway only after the successor's prompt is in flight, when the backend keeps that call running aftersession/cancel; and a call kiro-cli gives up on withoutnotifications/cancelled, answered into the open pipe while the same turn ends onend_turn.Backwards compatibility
Compatible:
respond()returns a bool where it returnedNone;mark-collectedtakes an optionalwritten, and one withoutwritten: truechanges nothing;POST /api/spawntakes an optionalcall_id. A pre-upgradekirocrew-coregets its members' completion turns (duplicates, never a loss) until it restarts.Reader: src/kiro_crew/slack/gateway_runtime/subagent_injection.py:_route_subagent_to_tab -- chat -- a brokered or pre-upgrade tab's unflagged members are injected: one duplicate turn per child (test_a_pooled_backends_written_reply_marks_nothing)
Reader: src/kiro_crew/dashboard/messaging_api/run_control.py:api_spawn_mark_collected -- chat -- an unwritten or foreign mark leaves a queued tab completion to play (test_a_queued_tab_completion_stays_without_its_written_mark)
Tests
test_spawn_sub_agents_duplicate_never_loss.py(new, 84): a whole write before any mark; the real cron callback andSessionManager; the channel route; sequence steps;!agent offwith a parked completion; the parked tab route's teardown gate and an unreadable tab.test_spawn_sub_agents_settle_at_release.py(new, 36): tabs; unlanded injected turns; late marks under a later attempt; a cut turn's call; the cron cleanup per holder.test_spawn_sub_agents_spawning_attempt.py(new, 32): the spawning attempt through the realapi_spawnand the real admission queue; the in-flight gate;call_id; a first member posted across a cut; a queued member stopped before it starts.test_cron_run_holding.py(new, 9) andtest_subagent_injection_session_reload.py(new, 1): the run's task holder, a newer run's key, a live run's finished key, the reload's settle.test_chat_runner_notes_landing.py(new, 6): the real dashboard runner notes landing from the claiming task onend_turnonly.d9a20ee8a7. Main's newer seams are reused, not replaced: the mark-collected retry (_mark_collected, now called fromon_answeredwithwritten: true),_skip_inline_collectedfor each carried member,admitted_cron_dispatcharound the cron run's holding, and the_reported_to_parentfilter inside_settle_collected_on_slot. Main tests set up for the mark this PR moves are updated in setup only, with their assertions unchanged:test_spawn_mark_collected_retry.pywrites before the mark and expectswritten: true; four synthesis-arm tests intest_gateway_subagent_injection.pyand one intest_subagent_queued_visibility.pyset the record's_collected_inlineand parent, which the slot set alone no longer decides;test_gateway_completion_skill_trigger.pyandtest_subagent_injection_channel_drain.pypasscarried.on_resume,finish_session_keybefore each release, the inventories, 26 cron gateways.fast-gate.yml(the source-only ratchets, the error-code contract among them, and the flake ratchet); black, isort, flake8, mypy; semgrep 0 on our lines; the wholetest/directory: 215,274 passed, and every other id fails the same way on main on this host or passes when run alone.Proof: PROVEN (prove.py, head
d8a4ba9fd1, rebased on mainea38bdd5e5; not re-run on thed9a20ee8a7rebase): with this PR's src/ hunks reverted, the changed tests fail on an assertion.Manual verification
Real kiro-cli, unbrokered, an isolated gateway; one
spawn_sub_agentscall with three children per parent:d2417a48c(2.33.1), cron jobd799f0daeb(2.35.1), cron jobd799f0daeb, cron job shown in its tabd799f0daeb, dashboard tabd799f0daeb, a person's turn in that cron job's tabThe cron run's own end-of-run reset cut the early child's tab turn once (
Connection lost — retrying…), as it does on main, and the retry delivered it. The head adds a test-harness answer, docs and a comment tod799f0daeb.Related Issues
Closes #23434
Fixes #23478
Sibling sweep
Swept:
git grep -nE "_inject_with_retry|mark-collected|_subagents_inline_collected|rollback_skill_bodies\(|settles_at_release|_finish_parked" -- src/Sibling: src/kiro_crew/slack/gateway_runtime/subagent_injection.py -- changed here (tab, channel and cron routes)
Sibling: src/kiro_crew/dashboard/chat_turn/turn_stats.py -- the dashboard runner, changed here (notes a landed turn); a dashboard-born tab's own acknowledgement is not changed (#23495)
Pattern harvest
Rule candidate: review-prompt
Pattern: a delivery acknowledged before the turn that received it landed.
Checklist
Contribution License Agreement
Apache-2.0 (see CONTRIBUTING.md).