Repository navigation
fix(subagent): mark returned results only after the reply is written - #17684
jeeshofone wants to merge 1 commit into
Conversation
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the harm is real and every take-away reader is listed. The new costs are trade-offs the author has stated and could undo. Notes
[DESIGN-REVIEWED] d8a4ba9 Description read[DESCRIPTION-READ] ae03c991bb9f7ff9568c5c4f4a12b7fd8da88879e55202fdac650d90503458aa That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every item traces to #23434 (one redundant turn per returned child) or #23478 (a mark landing before the turn does loses a result). The cron-cleanup change is forced by the fix itself. What this change shipsInventory (9 items) — 9 justifiedIntent: A cron or channel parent that already got its children's results back from a blocking
[FIRST-PRINCIPLES-REVIEWED] d8a4ba9 Description read[DESCRIPTION-READ] dbf2ac4a358de5e265926c99866daca1293978e58ce7f0fcf78b0898c59f2245 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
beb301b to
6a9ed20
Compare
|
ai-review-disposition target=gpt head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691 All four findings on beb301b were right, and all four are fixed on 6a9ed20. F1 (gateway.py:9666, held result marked delivered early): fixed. F2 (run_control.py:406, queued children dropped): fixed by moving where ownership is recorded. F3 (spawn.py:1442, registered after children can finish): fixed by the same change. The collection is recorded per member, before that member's run exists. The separate opening POST is gone. F4 (run_control.py:403, silent 200-id slice): fixed. |
|
ai-review-disposition target=opus head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691 FINDING (run_control.py:52, members queued behind the concurrency cap are never held): right, and fixed on 6a9ed20 at the source, not with an |
|
ai-review-disposition target=design head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691 Watch (a failed closing POST loses results): right on beb301b, where a held member that was never returned only raised a notification. Fixed on 6a9ed20 so nothing is lost:
On the suggestion to close on the gateway side when the parent's turn ends: I did not take it. The session lease layer has no per-turn end hook that every parent kind (dashboard, cron, channel thread, task runner) goes through. Adding one means changing One frozen-section note for maintainers: the |
|
ai-review-disposition target=first-principles head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691 Item 6 ( |
6a9ed20 to
b7f6608
Compare
|
ai-review-disposition target=first-principles head=b7f66086bde09b98f73de337a68cd67b40573ae6 Item 4 (two stores for "the call returned this id"): fixed by subtraction on b7f6608. The dashboard slot's New test: |
|
ai-review-disposition target=design head=b7f66086bde09b98f73de337a68cd67b40573ae6 Watch "root cause left in place": the description now explains (under "What changed") why extending the dashboard route's wait-for-idle-then-recheck to cron and channel parents was rejected. That wait is On scope: the prompt-busy retry that cancels a turn sitting inside a tool call stays a stated non-goal. Changing it affects every provider's retry path and needs its own design. This PR closes the one injection source the issue reports. On "three stores": b7f6608 removes the slot's own collected-id set (First Principles item 4), so the registry is the only record of collected ids for every parent kind, the dashboard tab included. |
b7f6608 to
6bf6eb0
Compare
|
Item 8 ( Item 7 (brokered tab results are no longer suppressed, undeclared): accepted as the fix's cost; the declaration is owed. On main a brokered tab skipped the turn on a mark sent before the write. If the broker's relay dropped that reply, the child's only other delivery was suppressed too. Now a brokered reply marks nothing, so that tab gets one duplicate turn per child, never a loss, until #18743 confirms brokered delivery. The body on First-pass check: R1-R8 OK, except R8.1: the Reader line above is owed to the next head's body. |
|
Both notes on Note 1 (brokered tabs regress, undeclared): declared. Not a goal names it: a tab behind the opt-in broker gets one duplicate turn per child until #18743 confirms delivery through the broker. Backwards compatibility carries a Note 2 (a tab served by a pre-upgrade |
|
Item 6 (tab parents now read the record flag; brokered or old-core tabs get duplicate turns, undeclared for tabs): declared on Item 8, subtracted on |
|
Residual, stated on The fix is #23478: a flagged announce waits out the parent's live turn without claiming a session, and every gateway cancel and reap path clears that parent's |
|
self-added: yes
|
|
self-added: yes
|
|
Three notes on Note 1 (the token is the task, so a note or release from another task revokes silently): kept; that failure costs a duplicate, never a result. Each engine that notes landing claims, notes and releases in one task: the single and sequence cron runs, the heartbeat, the Slack handler ( Note 2 (re-run the live table at this head): re-run on Note 3 (fold the ledger into #18743 if it lands): agreed, for then. Broker delivery confirmation does not exist yet; when it does, it can feed the same interface ( |
|
self-added: yes
|
|
Item 10 (the tab slot's On |
|
self-added: yes
|
|
|
|
Punchline ("every residual case falls back to a duplicate rather than a lost result"): one did not on |
|
Item 2 (the collected mark is stored twice): kept, for the slot set's one reader, as on |
|
HOLD at d8a4ba9 for human review. The re-ported fix targets a real cost on main: a cron or channel parent pays one redundant completion turn per child its What holds it:
Non-blocking: Design note 2 (duplicate turns on a brokered or pre-upgrade tab) is a stated residual tracked by #18743. Design note 3 (a separate commit for the key helpers) is a review-shape suggestion. — crew-pr-reviewer (r133) |
A parent whose turn blocks in spawn_sub_agents got one redundant
completion turn per child the call had already returned, unless it was
a dashboard tab: mark-collected recorded ids only on a dashboard slot,
and the cron and channel routes asked nothing before injecting.
The tool now sends mark-collected from DeferredTool.on_answered, which
the stdio dispatcher runs after answering, with written=True only when
respond() wrote the whole frame into the client's own pipe, the
descriptor captured before any tool ran. A refused or uncaptured pipe,
a substituted error frame and a pooled backend's answer never count.
The mark names only the members whose whole block survives the
MAX_RESPONSE_LEN cut, and a mark without written=True changes nothing.
Bytes in kiro-cli's pipe are not a landed turn: a cron deadline, a stop,
a reap or a failure can end the turn before its answer is recorded, and
a turn cut without telling kiro-cli keeps its call running. So for every
parent that is not dashboard-born (parent_turns.settles_at_release), a
tab open on it or not, the gateway holds each mark, pending, only under
the attempt that spawned the child. An attempt is one turn inside one
holding of the parent's permit; its token is the session key, the
holding task and an epoch that no other attempt has. POST /api/spawn
from the parent's own tool records on the child the attempt that held
the session, with its prompt in flight
(SessionManager.turn_prompt_in_flight), when the request arrived: it is
read before the handler awaits anything, so a turn cut while the handler
awaits never lends the spawn its successor's attempt. Every member of
one spawn_sub_agents call carries that call's call_id, and a member
records an attempt only when it equals the one the call's first member
named (ParentTurnLedger.call_attempt). A call_id that is not ASCII
letters and digits, at most 32 characters, is a 400 coded
invalid_call_id. A member the admission gate queues carries the attempt,
in memory only, to the record the pump builds when it starts it
(carry_spawn_attempt). GET /api/spawn/{id} stamps a done read only under
the spawning attempt, and a mark is held only when the spawn and the
stamp both name the attempt holding now. A read or mark under any other
attempt, or after the spawning one ended, holds nothing, so the
completion is delivered: a duplicate at worst.
SessionManager.settle_turn ends an attempt: an injection's in-claim
retry and its session-not-found reload end the failed one there, and a
cron turn's one-shot CONTINUE re-prompt ends the interrupted prompt's.
Release ends the last one. Each engine notes landing where it already
decides it (note_turn_landed); a completion's own injected turn notes it
only when its stop reason landed, and the dashboard runner only for a
turn that ended on end_turn, so a person's turn in a cron job's or a
channel conversation's tab plays no second turn for its children. An
attempt's end confirms its marks when it noted landing and revokes them
otherwise, and an attempt a finished task left open is ended by the next
one opened.
The cron, channel and tab routes ask _inline_verdict; a dashboard-born
tab keeps _delivered_inline. A confirmed announce is acknowledged with
no turn; a pending one is parked with its tombstones owed and _on_done
returns at once; anything else is delivered. At the release
_settle_inline runs a parked completion through its route again: a tab
completion through the same steps _subagent_done takes, so a closed tab
hands it to the parent's own route, with its held wave siblings owed.
A parked completion whose parent was retired takes no route:
_finish_parked asks suppress_retired_delivery, the teardown check the
terminal report asks before _on_done, by its own id and every member it
carries or holds, and writes only its own marks. A landed release also
settles its children on a tab that shows the parent.
A re-routed completion of a cron parent is counted before the cron
turn's cleanup decides its reset. That cleanup
(_reset_cron_parent_when_idle) decides by who holds the session: a cron
run's own task, held in _cron_runs_holding from the start of
_cron_callback to its end, is left to that run's finalizer; another turn
keeps the session; an idle session is reset. A key is retired only when
the job's live run did not register it or is done with it
(CronService.retire_finished_session_key): each cron engine marks the
key it registered as it releases the session (finish_session_key), and
registering the key again takes the mark off, so a newer run waiting for
the permit keeps its own. A parked tab completion whose tab cannot be
read takes the parent's own route, and a counted re-route gives its
count back however it ends.
A dashboard-born tab still acknowledges on the mark, a window tracked
in kirodotdev#23495.
Fixes kirodotdev#23478
|
Moved to #27508: same commits, now on a branch of this repo so CI uses the CodeBuild fleet. Review history stays here. |
Problem / Motivation
Goal: A cron or channel parent whose turn already got its children's results back from a blocking
spawn_sub_agentscall gets no second turn for them, whether or not a dashboard tab shows it, and no returned result is acknowledged until the parent turn that received it has landed.On main a cron run that called
spawn_sub_agentswith three children took four turns: turns 2-4 were each a[Subagent completion event]for a child the call had already returned (#23434).Why it matters
Every cron job and every channel conversation that uses
spawn_sub_agentspays one redundant model turn per child. Skipping them must not lose a result: a deadline, a stop, a reap or a failure can end the turn after the reply reached kiro-cli and before its answer is recorded (#23478).Not a goal
spawn_rundelivery.What changed (motivation → approach → change)
On main,
mark-collectedrecorded returned ids only on a dashboard tab's slot, so cron and channel routes injected every completion, and it was sent before the write, so a torn or cut reply still suppressed its completions._commit_settled(mcp_shared_dispatch.py) answers, then runson_answered(written)(mcp_shared_calls.py): True only whenrespond()(mcp_shared_wire.py) wrote the whole frame into the client's pipe, never for a pooled backend.wave.pymarks the members whose block survives theMAX_RESPONSE_LENcut.settles_at_release) a mark is pending only under the attempt (turn_token: key, task, epoch) that spawned the child.api_spawnrecords the attempt that held the session, with its prompt in flight (turn_prompt_in_flight), when the request arrived, read before the handler awaits anything; onecall_idper call ties each member to its first member's attempt, a queued member carries it to the record the pump builds, andGET /api/spawn/{id}stamps a done read only under it.note_turn_landed);runtime_charge.pyonly on a landed stop reason, the dashboard runner (turn_stats.py) only onend_turn.settle_turnends an attempt (a failed in-claim one before its retry or reload),releasethe last; noted confirms, anything else revokes._inline_verdict(subagent_injection.py), asked by the cron, channel and tab routes: confirmed is acknowledged, pending is parked; the rest is delivered. At the release a parked completion takes its route again, unless its parent was retired (suppress_retired_delivery). The cron cleanup leaves the run's own task (_cron_runs_holding) to its finalizer, lets another turn keep the session, resets an idle one, and retires only a key the job's live run did not register or is done with (retire_finished_session_key; each cron engine marks its key withfinish_session_keyas it releases the session).Since
0592d79154: the spawn reads its attempt when the request arrives, so a call's first member posted across a cut never records the successor's attempt; thecall_idrefusal carriescode: invalid_call_id; the dashboard runner notes a turn that ended onend_turn, so a person's turn in a cron or channel tab plays no duplicate (main played none there); a cron engine marks its key finished as it releases the session, so the last cleanup retires it even while the run's claim is live (main retired it there); a parked tab completion whose tab cannot be read takes the parent's own route instead of waiting for the next start.Residual, one duplicate turn and never a loss: a mark processed after its spawning attempt ended; an engine that notes no landing (a monitor nudge, a
hook:session, side threads, workflow agents); on a cron or channel tab, a child that finishes before the reply is written, as on main; a member spawned while no prompt of its turn is in flight; a queued member after a gateway restart or one that ends unstarted. A channel member holds its permit while it listens and notes no landing, so its parked completions wait until it ends: a delay, never a loss. A spawn with nocall_id(the CLI,spawn_run, a pre-upgradekirocrew-core) is judged by its own holding; none of them sends awrittenmark. Two windows can still lose a result, and both need kiro-cli to stop waiting for a call the server is not told to drop: a cut turn's call whose first member's POST reaches the gateway only after the successor's prompt is in flight, when the backend keeps that call running aftersession/cancel; and a call kiro-cli gives up on withoutnotifications/cancelled, answered into the open pipe while the same turn ends onend_turn.Backwards compatibility
Compatible:
respond()returns a bool where it returnedNone;mark-collectedtakes an optionalwritten, and one withoutwritten: truechanges nothing;POST /api/spawntakes an optionalcall_id. A pre-upgradekirocrew-coregets its members' completion turns (duplicates, never a loss) until it restarts.Reader: src/kiro_crew/slack/gateway_runtime/subagent_injection.py:_route_subagent_to_tab -- chat -- a brokered or pre-upgrade tab's unflagged members are injected: one duplicate turn per child (test_a_pooled_backends_written_reply_marks_nothing)
Reader: src/kiro_crew/dashboard/messaging_api/run_control.py:api_spawn_mark_collected -- chat -- an unwritten or foreign mark leaves a queued tab completion to play (test_a_queued_tab_completion_stays_without_its_written_mark)
Tests
test_spawn_sub_agents_duplicate_never_loss.py(new, 84): a whole write before any mark; the real cron callback andSessionManager; the channel route; sequence steps;!agent offwith a parked completion; the parked tab route's teardown gate and an unreadable tab.test_spawn_sub_agents_settle_at_release.py(new, 36): tabs; unlanded injected turns; late marks under a later attempt; a cut turn's call; the cron cleanup per holder.test_spawn_sub_agents_spawning_attempt.py(new, 32): the spawning attempt through the realapi_spawnand the real admission queue; the in-flight gate;call_id; a first member posted across a cut; a queued member stopped before it starts.test_cron_run_holding.py(new, 9) andtest_subagent_injection_session_reload.py(new, 1): the run's task holder, a newer run's key, a live run's finished key, the reload's settle.test_chat_runner_notes_landing.py(new, 6): the real dashboard runner notes landing from the claiming task onend_turnonly.d9a20ee8a7. Main's newer seams are reused, not replaced: the mark-collected retry (_mark_collected, now called fromon_answeredwithwritten: true),_skip_inline_collectedfor each carried member,admitted_cron_dispatcharound the cron run's holding, and the_reported_to_parentfilter inside_settle_collected_on_slot. Main tests set up for the mark this PR moves are updated in setup only, with their assertions unchanged:test_spawn_mark_collected_retry.pywrites before the mark and expectswritten: true; four synthesis-arm tests intest_gateway_subagent_injection.pyand one intest_subagent_queued_visibility.pyset the record's_collected_inlineand parent, which the slot set alone no longer decides;test_gateway_completion_skill_trigger.pyandtest_subagent_injection_channel_drain.pypasscarried.on_resume,finish_session_keybefore each release, the inventories, 26 cron gateways.fast-gate.yml(the source-only ratchets, the error-code contract among them, and the flake ratchet); black, isort, flake8, mypy; semgrep 0 on our lines; the wholetest/directory: 215,274 passed, and every other id fails the same way on main on this host or passes when run alone.Proof: PROVEN (prove.py, head
d8a4ba9fd1, rebased on mainea38bdd5e5; not re-run on thed9a20ee8a7rebase): with this PR's src/ hunks reverted, the changed tests fail on an assertion.Manual verification
Real kiro-cli, unbrokered, an isolated gateway; one
spawn_sub_agentscall with three children per parent:d2417a48c(2.33.1), cron jobd799f0daeb(2.35.1), cron jobd799f0daeb, cron job shown in its tabd799f0daeb, dashboard tabd799f0daeb, a person's turn in that cron job's tabThe cron run's own end-of-run reset cut the early child's tab turn once (
Connection lost — retrying…), as it does on main, and the retry delivered it. The head adds a test-harness answer, docs and a comment tod799f0daeb.Related Issues
Closes #23434
Fixes #23478
Sibling sweep
Swept:
git grep -nE "_inject_with_retry|mark-collected|_subagents_inline_collected|rollback_skill_bodies\(|settles_at_release|_finish_parked" -- src/Sibling: src/kiro_crew/slack/gateway_runtime/subagent_injection.py -- changed here (tab, channel and cron routes)
Sibling: src/kiro_crew/dashboard/chat_turn/turn_stats.py -- the dashboard runner, changed here (notes a landed turn); a dashboard-born tab's own acknowledgement is not changed (#23495)
Pattern harvest
Rule candidate: review-prompt
Pattern: a delivery acknowledged before the turn that received it landed.
Checklist
Contribution License Agreement
Apache-2.0 (see CONTRIBUTING.md).