Skip to content

fix(subagent): mark returned results only after the reply is written - #17684

Closed
jeeshofone wants to merge 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/17626-cron-inline-collected
Closed

jeeshofone wants to merge 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/17626-cron-inline-collected

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A cron or channel parent whose turn already got its children's results back from a blocking spawn_sub_agents call gets no second turn for them, whether or not a dashboard tab shows it, and no returned result is acknowledged until the parent turn that received it has landed.

On main a cron run that called spawn_sub_agents with three children took four turns: turns 2-4 were each a [Subagent completion event] for a child the call had already returned (#23434).

Why it matters

Every cron job and every channel conversation that uses spawn_sub_agents pays one redundant model turn per child. Skipping them must not lose a result: a deadline, a stop, a reap or a failure can end the turn after the reply reached kiro-cli and before its answer is recorded (#23478).

Not a goal

What changed (motivation → approach → change)

On main, mark-collected recorded returned ids only on a dashboard tab's slot, so cron and channel routes injected every completion, and it was sent before the write, so a torn or cut reply still suppressed its completions.

Step Change
Answer, then report _commit_settled (mcp_shared_dispatch.py) answers, then runs on_answered(written) (mcp_shared_calls.py): True only when respond() (mcp_shared_wire.py) wrote the whole frame into the client's pipe, never for a pooled backend. wave.py marks the members whose block survives the MAX_RESPONSE_LEN cut.
Hold the mark under its spawning attempt For every parent but a dashboard-born one (settles_at_release) a mark is pending only under the attempt (turn_token: key, task, epoch) that spawned the child. api_spawn records the attempt that held the session, with its prompt in flight (turn_prompt_in_flight), when the request arrived, read before the handler awaits anything; one call_id per call ties each member to its first member's attempt, a queued member carries it to the record the pump builds, and GET /api/spawn/{id} stamps a done read only under it.
Settle at the attempt's end Engines note landing where they decide it (note_turn_landed); runtime_charge.py only on a landed stop reason, the dashboard runner (turn_stats.py) only on end_turn. settle_turn ends an attempt (a failed in-claim one before its retry or reload), release the last; noted confirms, anything else revokes.
Park, never wait _inline_verdict (subagent_injection.py), asked by the cron, channel and tab routes: confirmed is acknowledged, pending is parked; the rest is delivered. At the release a parked completion takes its route again, unless its parent was retired (suppress_retired_delivery). The cron cleanup leaves the run's own task (_cron_runs_holding) to its finalizer, lets another turn keep the session, resets an idle one, and retires only a key the job's live run did not register or is done with (retire_finished_session_key; each cron engine marks its key with finish_session_key as it releases the session).

Since 0592d79154: the spawn reads its attempt when the request arrives, so a call's first member posted across a cut never records the successor's attempt; the call_id refusal carries code: invalid_call_id; the dashboard runner notes a turn that ended on end_turn, so a person's turn in a cron or channel tab plays no duplicate (main played none there); a cron engine marks its key finished as it releases the session, so the last cleanup retires it even while the run's claim is live (main retired it there); a parked tab completion whose tab cannot be read takes the parent's own route instead of waiting for the next start.

Residual, one duplicate turn and never a loss: a mark processed after its spawning attempt ended; an engine that notes no landing (a monitor nudge, a hook: session, side threads, workflow agents); on a cron or channel tab, a child that finishes before the reply is written, as on main; a member spawned while no prompt of its turn is in flight; a queued member after a gateway restart or one that ends unstarted. A channel member holds its permit while it listens and notes no landing, so its parked completions wait until it ends: a delay, never a loss. A spawn with no call_id (the CLI, spawn_run, a pre-upgrade kirocrew-core) is judged by its own holding; none of them sends a written mark. Two windows can still lose a result, and both need kiro-cli to stop waiting for a call the server is not told to drop: a cut turn's call whose first member's POST reaches the gateway only after the successor's prompt is in flight, when the backend keeps that call running after session/cancel; and a call kiro-cli gives up on without notifications/cancelled, answered into the open pipe while the same turn ends on end_turn.

Backwards compatibility

Compatible: respond() returns a bool where it returned None; mark-collected takes an optional written, and one without written: true changes nothing; POST /api/spawn takes an optional call_id. A pre-upgrade kirocrew-core gets its members' completion turns (duplicates, never a loss) until it restarts.

Reader: src/kiro_crew/slack/gateway_runtime/subagent_injection.py:_route_subagent_to_tab -- chat -- a brokered or pre-upgrade tab's unflagged members are injected: one duplicate turn per child (test_a_pooled_backends_written_reply_marks_nothing)
Reader: src/kiro_crew/dashboard/messaging_api/run_control.py:api_spawn_mark_collected -- chat -- an unwritten or foreign mark leaves a queued tab completion to play (test_a_queued_tab_completion_stays_without_its_written_mark)

Tests

  • test_spawn_sub_agents_duplicate_never_loss.py (new, 84): a whole write before any mark; the real cron callback and SessionManager; the channel route; sequence steps; !agent off with a parked completion; the parked tab route's teardown gate and an unreadable tab.
  • test_spawn_sub_agents_settle_at_release.py (new, 36): tabs; unlanded injected turns; late marks under a later attempt; a cut turn's call; the cron cleanup per holder.
  • test_spawn_sub_agents_spawning_attempt.py (new, 32): the spawning attempt through the real api_spawn and the real admission queue; the in-flight gate; call_id; a first member posted across a cut; a queued member stopped before it starts.
  • test_cron_run_holding.py (new, 9) and test_subagent_injection_session_reload.py (new, 1): the run's task holder, a newer run's key, a live run's finished key, the reload's settle.
  • test_chat_runner_notes_landing.py (new, 6): the real dashboard runner notes landing from the claiming task on end_turn only.
  • Rebased on main d9a20ee8a7. Main's newer seams are reused, not replaced: the mark-collected retry (_mark_collected, now called from on_answered with written: true), _skip_inline_collected for each carried member, admitted_cron_dispatch around the cron run's holding, and the _reported_to_parent filter inside _settle_collected_on_slot. Main tests set up for the mark this PR moves are updated in setup only, with their assertions unchanged: test_spawn_mark_collected_retry.py writes before the mark and expects written: true; four synthesis-arm tests in test_gateway_subagent_injection.py and one in test_subagent_queued_visibility.py set the record's _collected_inline and parent, which the slot set alone no longer decides; test_gateway_completion_skill_trigger.py and test_subagent_injection_channel_drain.py pass carried.
  • Updated pins: the cron cleanup, the cron retries, the sequence on_resume, finish_session_key before each release, the inventories, 26 cron gateways.
  • Each mechanism this round adds has a mutant that turns a named test red; the new files pass 5 repeats and 2 shuffled orders.
  • Every Fast Gate step, replayed from fast-gate.yml (the source-only ratchets, the error-code contract among them, and the flake ratchet); black, isort, flake8, mypy; semgrep 0 on our lines; the whole test/ directory: 215,274 passed, and every other id fails the same way on main on this host or passes when run alone.

Proof: PROVEN (prove.py, head d8a4ba9fd1, rebased on main ea38bdd5e5; not re-run on the d9a20ee8a7 rebase): with this PR's src/ hunks reverted, the changed tests fail on an assertion.

Manual verification

Real kiro-cli, unbrokered, an isolated gateway; one spawn_sub_agents call with three children per parent:

parent turns completion turns
main d2417a48c (2.33.1), cron job 4 3
d799f0daeb (2.35.1), cron job 1 0
d799f0daeb, cron job shown in its tab 1 1 tab turn, for an early child (the Residual)
d799f0daeb, dashboard tab 1 0
d799f0daeb, a person's turn in that cron job's tab 1 0

The cron run's own end-of-run reset cut the early child's tab turn once (Connection lost — retrying…), as it does on main, and the retry delivered it. The head adds a test-harness answer, docs and a comment to d799f0daeb.

Related Issues

Closes #23434
Fixes #23478

Sibling sweep

Swept: git grep -nE "_inject_with_retry|mark-collected|_subagents_inline_collected|rollback_skill_bodies\(|settles_at_release|_finish_parked" -- src/
Sibling: src/kiro_crew/slack/gateway_runtime/subagent_injection.py -- changed here (tab, channel and cron routes)
Sibling: src/kiro_crew/dashboard/chat_turn/turn_stats.py -- the dashboard runner, changed here (notes a landed turn); a dashboard-born tab's own acknowledgement is not changed (#23495)

Pattern harvest

Rule candidate: review-prompt
Pattern: a delivery acknowledged before the turn that received it landed.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Apache-2.0 (see CONTRIBUTING.md).

@jeeshofone
jeeshofone requested a review from a team as a code owner October 7, 2026 10:04
@jeeshofone
jeeshofone requested a review from CrysisDeu October 7, 2026 10:04
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of d8a4ba9fd1b64f2ff1fb5cf4af9e30d86427d2e4 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the harm is real and every take-away reader is listed. The new costs are trade-offs the author has stated and could undo.

Notes

  • Cron and channel parents lose nothing on main today, because every completion gets injected. This PR adds two windows where a result can be lost ("Two windows can still lose a result"), and both depend on kiro-cli behaviour after session/cancel that is unverified. Pin a test or a tracking issue on each window so the cost of saving a turn doesn't quietly become a lost result.
  • A dashboard-born tab on a pooled backend, or on a pre-upgrade kirocrew-core, loses main's skip and gets one duplicate turn per child. That duplicate turn can push the [OPTIONS:] buttons from the synthesis message out of view. The PR states this regression as accepted, and it needs Broker<->stub delivery confirmation for brokered MCP responses #18743 to close it.
  • retire_finished_session_key and finish_session_key change the lifecycle of cron session keys for every cron engine. The PR needs them to make parking work, but they would be easier to review and to revert as a separate commit.

[DESIGN-REVIEWED] d8a4ba9

Description read

[DESCRIPTION-READ] ae03c991bb9f7ff9568c5c4f4a12b7fd8da88879e55202fdac650d90503458aa

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of d8a4ba9fd1b64f2ff1fb5cf4af9e30d86427d2e4 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item traces to #23434 (one redundant turn per returned child) or #23478 (a mark landing before the turn does loses a result). The cron-cleanup change is forced by the fix itself.

What this change ships

Inventory (9 items) — 9 justified

Intent: A cron or channel parent that already got its children's results back from a blocking spawn_sub_agents call should not pay another model turn per child, and no result should be lost. FIX.

  1. Cron and channel parents no longer get a [Subagent completion event] turn for a child the call already returned — justified
  2. A returned child counts as delivered only after the whole reply frame reached the client's pipe; a torn or brokered reply marks nothing — justified
  3. For parents that are not dashboard-born, the mark stays pending until the spawning turn ends; a turn that never landed has its children delivered normally — justified
  4. A completion that arrives while its mark is pending is parked, then re-routed or acknowledged when the parent's turn releases — justified
  5. Engines and the dashboard runner record that a turn landed (the dashboard runner only on end_turn) — justified
  6. POST /api/spawn accepts an optional call_id and refuses a malformed one with 400 invalid_call_id — justified
  7. mark-collected without written: true changes nothing, so a pre-upgrade core gets duplicate turns, never a loss — justified
  8. A dashboard tab's mark now flags only the caller's own children; other parents' ids are ignored — justified
  9. The cron "last subagent done" cleanup no longer resets a session a live turn holds, and hands key retirement to the run's finalizer or finish_session_key — justified (base subagent_injection.py:853 reset unconditionally, which would kill the turn that item 4 now leaves running)

[FIRST-PRINCIPLES-REVIEWED] d8a4ba9

Description read

[DESCRIPTION-READ] dbf2ac4a358de5e265926c99866daca1293978e58ce7f0fcf78b0898c59f2245

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed d8a4ba9fd1b64f2ff1fb5cf4af9e30d86427d2e4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d8a4ba9

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed d8a4ba9fd1b64f2ff1fb5cf4af9e30d86427d2e4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d8a4ba9

@jeeshofone
jeeshofone force-pushed the fix/17626-cron-inline-collected branch from beb301b to 6a9ed20 Compare October 7, 2026 12:22
@jeeshofone

Copy link
Copy Markdown
Contributor Author

ai-review-disposition target=gpt head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691

All four findings on beb301b were right, and all four are fixed on 6a9ed20.

F1 (gateway.py:9666, held result marked delivered early): fixed. inline_collection_owns now sets _delivery_queued on a held completion, so _report_terminal_impl writes no delivered tombstone and the reaper's retention clock does not start. The registry keeps the held record. When the call's closing mark-collected lists it as collected, api_spawn_mark_collected settles it through settle_queued_delivery. If the call never returns it, it is delivered as an ordinary completion, and marked delivered after that. Test: test_first_child_finishing_does_not_touch_the_blocked_parent_turn asserts the held completion is undelivered, and that exactly one settle (for that member) runs at the close.

F2 (run_control.py:406, queued children dropped): fixed by moving where ownership is recorded. /api/spawn with inline_collect mints the run id through SubagentManager.reserve_inline_member and records it before calling spawn with _preassigned_id, so a member queued behind the cap is held from its first moment, with no _agents row needed. The closing mark no longer filters released ids by get(), because ending a collection only touches ids the registry already holds. Test: test_the_minted_id_is_held_before_spawn_starts_it.

F3 (spawn.py:1442, registered after children can finish): fixed by the same change. The collection is recorded per member, before that member's run exists. The separate opening POST is gone.

F4 (run_control.py:403, silent 200-id slice): fixed. subagent_inline_collection.MAX_IDS_PER_PARENT (1000) is the one bound. The dashboard slot's _COLLECTED_IDS_CAP is now defined from it, and the handler slices by it. A spawn that would exceed it is refused before spawning with 429 inline_collection_full, which the tool reports as that member's spawn error, so no accepted member is ever left unheld. Test: test_a_full_collection_refuses_before_spawning.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

ai-review-disposition target=opus head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691

FINDING (run_control.py:52, members queued behind the concurrency cap are never held): right, and fixed on 6a9ed20 at the source, not with an is_queued filter. Ownership is now recorded by /api/spawn itself. It mints the member's id (SubagentManager.reserve_inline_member), records it in inline_collections, and only then calls spawn with _preassigned_id. So the id is held before the member is queued or started, whatever admission does with it. The opening collecting POST, and the get() filter it went through, are gone. Test: test_the_minted_id_is_held_before_spawn_starts_it asserts the id is already held when spawn is called.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

ai-review-disposition target=design head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691

Watch (a failed closing POST loses results): right on beb301b, where a held member that was never returned only raised a notification. Fixed on 6a9ed20 so nothing is lost:

  • A held completion is now kept undelivered on the registry, not just as an id. Every held member the call did not return is delivered as an ordinary completion, through the same _subagent_done route, and marked delivered after that. "Did not return" means a cancelled call (ids: [], released), or a collection whose close never arrives, which expires after max_wait plus 300 s. This also covers the still_running members you named, if the close is lost. Tests: test_a_held_member_the_cancelled_call_never_returned_is_delivered, test_a_collection_whose_close_never_arrives_delivers_what_it_held.
  • The close is retried up to three times (_close_collection, test_a_failed_close_is_retried), so the expiry is a last backstop and does not stand in for the close.

On the suggestion to close on the gateway side when the parent's turn ends: I did not take it. The session lease layer has no per-turn end hook that every parent kind (dashboard, cron, channel thread, task runner) goes through. Adding one means changing session.py's release contract, which is wider than this fix. With the retried close plus delivery at expiry, a lost close now costs a delay, bounded by the call's own wait, and no longer loses the result.

One frozen-section note for maintainers: the Not a goal bullet "Re-injecting a result whose collecting call was cancelled before returning it" describes beb301b. Resolving this Watch is what changed it. It is a defect in code this PR adds, so it is in scope. I have left the frozen section untouched, as the template requires.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

ai-review-disposition target=first-principles head=6a9ed20005823edb41a8ad5ac8e5d8e1e3de8691

Item 6 (is_collecting has no production consumer): right. Deleted on 6a9ed20. The tests now assert through hold() and consume_collected(). The registry's public surface is what production calls: reserve (from /api/spawn), hold and consume_collected (from inline_collection_owns), finish and record_collected (from mark-collected).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@jeeshofone
jeeshofone force-pushed the fix/17626-cron-inline-collected branch from 6a9ed20 to b7f6608 Compare October 7, 2026 13:34
@jeeshofone

Copy link
Copy Markdown
Contributor Author

ai-review-disposition target=first-principles head=b7f66086bde09b98f73de337a68cd67b40573ae6

Item 4 (two stores for "the call returned this id"): fixed by subtraction on b7f6608. The dashboard slot's _subagents_inline_collected set is deleted (dashboard/state.py, attribute and __slots__ entry). mark-collected now records a tab's returned ids with InlineCollections.record_collected, the same call every other parent kind uses, so one store, one TTL (COLLECTED_TTL_SECS) and one cap (MAX_IDS_PER_PARENT) cover every parent. inline_collection_owns checks only the registry, and a tab's synthesis turn is disarmed when has_collected(parent) goes false. The synthesis-arm guard in _subagent_done drops its slot-set clause: a returned member is consumed by inline_collection_owns before the arm is reached.

New test: test_a_tabs_returned_ids_live_on_the_registry_like_any_parent drives the real mark-collected handler for a tab, then both completions, and asserts that neither is injected or queued and that synthesis is disarmed only after the second. The tests that read the slot set (test_subagent_queued_visibility.py, test_handlers_messaging_coverage.py) now assert against the registry.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

ai-review-disposition target=design head=b7f66086bde09b98f73de337a68cd67b40573ae6

Watch "root cause left in place": the description now explains (under "What changed") why extending the dashboard route's wait-for-idle-then-recheck to cron and channel parents was rejected. That wait is asyncio.wait_for(shield(slot.task), timeout=INJECTION_TIMEOUT), 900 s by default, while spawn_sub_agents waits up to 7200 s. A member that finishes more than 900 s before its call returns gets past the recheck while the call is still collecting it. On a tab that becomes a queued completion row, and on a cron or channel parent it is the cancelling prompt again. The recheck can only see "the call already returned it", never "the call is still collecting it", and the second is the case this issue is about.

On scope: the prompt-busy retry that cancels a turn sitting inside a tool call stays a stated non-goal. Changing it affects every provider's retry path and needs its own design. This PR closes the one injection source the issue reports. On "three stores": b7f6608 removes the slot's own collected-id set (First Principles item 4), so the registry is the only record of collected ids for every parent kind, the dashboard tab included.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@jeeshofone
jeeshofone force-pushed the fix/17626-cron-inline-collected branch from b7f6608 to 6bf6eb0 Compare October 7, 2026 14:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Item 8 ("flagged": flagged has zero consumers): subtracted in a9d5d45eb. The key is gone from all three mark-collected replies, which are back to main's shape: {"status": "no_slot"} and {"status": "ok", "marked": N}. _flag_collected_inline now returns nothing. The coverage pins in test_handlers_messaging_coverage.py are back to main's exact replies.

Item 7 (brokered tab results are no longer suppressed, undeclared): accepted as the fix's cost; the declaration is owed. On main a brokered tab skipped the turn on a mark sent before the write. If the broker's relay dropped that reply, the child's only other delivery was suppressed too. Now a brokered reply marks nothing, so that tab gets one duplicate turn per child, never a loss, until #18743 confirms brokered delivery. The body on a9d5d45eb still says "Removes nothing". The next head's body replaces it with a Reader: line for _route_subagent_to_tab naming this. The body is not edited on this head, because an edit after the push would cancel the lanes running on it.

First-pass check: R1-R8 OK, except R8.1: the Reader line above is owed to the next head's body.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Both notes on a9d5d45eb are declared on 77ba3235f.

Note 1 (brokered tabs regress, undeclared): declared. Not a goal names it: a tab behind the opt-in broker gets one duplicate turn per child until #18743 confirms delivery through the broker. Backwards compatibility carries a Reader: line for _route_subagent_to_tab in place of "Removes nothing".

Note 2 (a tab served by a pre-upgrade kirocrew-core): declared, and now duplicate-only. The same Not-a-goal bullet and Reader: line name it. On a9d5d45eb such a mark, sent before its write and without written, still removed a completion already queued on the tab and settled its delivery, so a write that then tore lost that member. On 77ba3235f a mark without written: true is answered before any slot bookkeeping and changes nothing (test_a_queued_tab_completion_stays_without_its_written_mark), so that tab gets its completion turns: duplicates, never a loss.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Item 6 (tab parents now read the record flag; brokered or old-core tabs get duplicate turns, undeclared for tabs): declared on 77ba3235f. Not a goal names the tab route behind the opt-in broker or a pre-upgrade kirocrew-core, and Backwards compatibility replaces "Removes nothing" with Reader: lines for _route_subagent_to_tab and api_spawn_mark_collected. Such a tab gets one duplicate turn per child, never a loss: a mark without written: true now changes nothing, a completion already queued on the tab included.

Item 8, subtracted on a9d5d45eb, still holds for the replies, which keep main's shape. _flag_collected_inline now returns the ids it flagged; only the tab-slot bookkeeping below it reads them, so a queued completion is settled only for the claim's own children.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Residual, stated on 77ba3235f in the spec's Residual line and in the body: written means the reply's bytes reached kiro-cli's pipe, not that the parent's model finished with them. If the gateway itself cancels or reaps a cron or channel parent's turn after that write, the results that turn collected get no completion turn: the mark normally lands before each child's completion, and the route then skips it. Tabs already behave this way on main.

The fix is #23478: a flagged announce waits out the parent's live turn without claiming a session, and every gateway cancel and reap path clears that parent's _collected_inline flags, so the held announce is injected into the resumed session.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a returned result is acknowledged only at the release of a parent turn that landed

  • span=ffbb335430d2 Cron completion is acknowledged before its parent turn lands: fixed in 81bf50538 as the Fix line describes.

Upheld, and the channel route had the same window (Slack threads, Telegram, task runners), for a deadline, a stop, a reap or any failure after the write.
A cron or channel mark is now pending, held under the turn that holds the parent's permit when it arrives (SessionManager.turn_token, kiro_crew.parent_turns). Each engine notes landing where it already decides it (note_turn_landed: cron single and sequence steps, TurnBracket.landed, Slack, Telegram, the task runner, the heartbeat, _inject_with_retry). SessionManager.release settles that holding before anything else, an absorbed release after a reaper's reset included: a landed turn confirms its marks, any other end revokes them.
The routes never wait inside _on_done. A completion whose mark is still pending is parked (_park_inline): it owes its deliveries like a queued tab completion, so _delivery_queued keeps the run loop from writing the delivered tombstone, _on_done returns at once, and _ON_DONE_TIMEOUT never resets a live parent for it. A restart while parked finds the folder in list_orphans(), and the pruner never touches an untombstoned folder.
At the release, _settle_inline acknowledges a confirmed completion with no second turn, and re-routes a revoked one as an ordinary completion. A re-routed cron completion is counted in _cron_injecting inside the release, before the cron finally decides its reset, so it injects into the same session.
Tests that fail on 77ba3235f: test_a_cron_turn_that_does_not_land_still_delivers_the_returned_result[deadline|failure|reap] (the real cron callback over a real SessionManager) and test_a_channel_route_does_not_acknowledge_a_returned_result_under_a_live_turn[slack-thread|telegram|task-runner]. test_a_cron_turn_that_lands_plays_no_second_turn_for_its_returned_result keeps the PR's goal. The PR closes #23478 with it.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a mark is held only under the holding that read its child done

  • span=9ecf7a13276d Bind collected results to their receiving turn, not a successor: fixed in f07811b8db1c5379b669f721454c813b72a3351d, keyed to the receiving holding's own read of the child rather than a new field on the tool call.

Upheld. GET /api/spawn/{id} from the parent's own tool (internal auth, X-Session-Key equal to the run's parent) stamps SubagentInfo._seen_done_under with turn_token on the first read that reports the child done. The holding is read in the same step as done and kept only if the same holding still answers when the read ends. spawn_sub_agents reads every child through that route just before it builds its reply, so the stamp names the holding whose reply carries the child.
_flag_collected_inline holds a mark only when the holding of the permit now equals that stamp. A mark processed after the receiving turn released, while a successor turn, a sibling's injection or a compaction holds the permit, flags nothing, so the completion is delivered: one duplicate turn, never a loss. A compaction's holding ends with a raw semaphore.release(), so a mark held there would never have settled.
No wire change: the tool already sends that read. A call that went on reading after its turn was cancelled could name the next turn; the stdio loop drops a parked spawn_sub_agents call on its notifications/cancelled, and the body's Residual names that dependency.
Red with the check reverted: test_the_next_messages_landed_turn_does_not_confirm_a_stopped_turns_marks (this trigger, on a Telegram parent), test_a_late_mark_is_not_confirmed_by_a_siblings_landed_injection (the real cron route and _inject_with_retry) and test_a_late_mark_under_a_compaction_is_still_delivered (the real _compact_in_place). The stamp itself: test_only_the_parent_tools_first_done_read_names_the_holding_a_mark_is_held_under.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Three notes on 81bf50538, answered on f07811b8db1c5379b669f721454c813b72a3351d.

Note 1 (the token is the task, so a note or release from another task revokes silently): kept; that failure costs a duplicate, never a result. Each engine that notes landing claims, notes and releases in one task: the single and sequence cron runs, the heartbeat, the Slack handler (handle_message awaits _run_turn, then _deliver_answer, which releases), the Slack transport loop, Telegram's _run_turn, the task runner, ChannelTurns through TurnBracket, and the cron and channel routes around _inject_with_retry. A note from another task leaves the real holding un-noted, so its marks are revoked and each child replays one turn, as for an engine that notes nothing; it can never confirm a turn that did not land. Real-engine tests hold the token for the cron callback (test_a_cron_turn_that_lands_plays_no_second_turn_for_its_returned_result) and for _inject_with_retry (test_a_completion_turn_that_lands_confirms_what_it_received_inline), and the census test_every_engine_that_settles_a_turn_notes_its_landing pins that each settling engine notes. End-to-end tests for every other engine are not added, since a regression there costs a duplicate turn.

Note 2 (re-run the live table at this head): re-run on f07811b8db1c5379b669f721454c813b72a3351d. Real kiro-cli 2.35.1, unbrokered, an isolated gateway: a cron job with no tab ran 1 turn and injected no completion; a dashboard tab ran 1 turn. A cron job shown in its own tab, the parent this head adds, ran 1 cron turn and then 1 tab turn for the one child that finished before the reply was written; the other two played none. That early child reaches the tab before any mark exists, as on main, and the body's Residual and Manual verification say so.

Note 3 (fold the ledger into #18743 if it lands): agreed, for then. Broker delivery confirmation does not exist yet; when it does, it can feed the same interface (turn_token, note_turn_landed, settle_turn and the release hook) instead of a second way to confirm delivery.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: an injected completion turn confirms its inline marks only when its stop reason landed

  • span=0d961c1d22c5 Do not confirm inline results after a cancelled injection turn: fixed in f07811b8db1c5379b669f721454c813b72a3351d as the Fix line describes.

Upheld. _inject_with_retry now passes on_complete to stream_and_collect, records the stop reason the way the cron engine does (None until a completion is seen), and calls note_turn_landed only when stop_reason_landed says the turn landed. A user stop, a refusal, a tool stall and a stream with no completion each return normally and now revoke the turn's marks at the release, so those children's completions are delivered.
The in-claim AcpError retry is a second turn inside the same holding, so it now settles the failed attempt first (SessionManager.settle_turn, the call release itself makes): what attempt 1 received is revoked there, never confirmed by attempt 2's landing.
Red with either half reverted: test_a_completion_turn_that_did_not_land_does_not_confirm_what_it_received_inline[stopped|refused|stalled|no-completion] (the real _inject_with_retry over the real stream_and_collect) and test_a_failed_attempt_does_not_hand_its_inline_marks_to_the_retry_that_lands. test_a_completion_turn_that_lands_confirms_what_it_received_inline keeps the landed case.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Item 10 (the tab slot's _subagents_inline_collected duplicates the record flag): kept, for its one reader. The two answer different questions. SubagentInfo._collected_inline says a child was returned inline. The slot set says which of the tab's returned children still have a completion to process, and the synthesis disarm (if not slot._subagents_inline_collected) waits for it to empty. Deciding the disarm from record flags would need the manager to list a parent's finished records, and it has no in-memory query for that (running_agents_for lists running ones only), on a delivery-path step kept await-free on purpose.

On f07811b8db1c5379b669f721454c813b72a3351d the set is written only where a mark settles a tab: a dashboard-born parent's mark, and, for a tab that shows a cron job or a channel conversation, the landed release of the turn that received the children. Both go through one function (_settle_collected_on_slot), so the disarm behaves the same on every tab.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the token names one attempt, and an attempt's end gives the next turn a new token

  • span=8eac717aa925 Retry attempts reuse the same result-delivery token: fixed in 060ca46f22.

Upheld. The token is (key, task, epoch). ParentTurnLedger keeps the open attempt of each (key, task) holding and draws each epoch from one counter for the whole ledger, so no two attempts share a token. The parent tool's done read and the landing note open the attempt. SessionManager.settle_turn ends it, which the in-claim AcpError retry calls before its backoff, and release ends the last one. The retry, and the same task's next claim, each open a new attempt with a new epoch.
_flag_collected_inline only looks the current attempt up (turn_token(..., open_attempt=False)). A mark the failed attempt's tool stamped, processed in the backoff or later, is held under nothing and flags nothing, so the completion is delivered: one duplicate turn at most, never a loss. The same rule covers the cancelled-turn successor from 81bf50538, and a successor in the same task as well.
Red on f07811b8db, green here: test_a_failed_attempts_mark_processed_in_the_retry_backoff_is_not_held_under_the_retry (this trigger, the real _inject_with_retry over the real stream_and_collect) and test_a_late_mark_is_not_held_under_the_same_tasks_next_turn. A mutant that keeps the epoch across an attempt's end, and one that settles without ending the attempt, each turn them red.
The cron engine's one-shot CONTINUE re-prompt is a second turn inside one claim as well, so the dispatcher ends the interrupted prompt's attempt before its backoff (on_resume, which calls settle_turn): test_a_cron_turn_resumed_after_a_post_token_error_confirms_nothing_its_first_prompt_got.
An attempt that a finished task never ended (a compaction gives its permit back with a raw semaphore.release()) is ended by the next attempt opened, so the ledger keeps no attempt of a finished task: test_an_attempt_its_finished_task_left_open_is_ended_by_the_next_open.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • span=8089d85560e1 The last sub-agent's cleanup keeps the cron run's key registered when the session is already gone: fixed in 060ca46f22.

Upheld. _reset_cron_parent_when_idle resets with skip_if_busy, then decides by who holds the session. When nothing holds it, the session is reset, or was already gone (the timeout arm's reset, the idle reaper, another path), and the key is retired either way; the log line says which. When the cron run's own turn holds it (the cron engine marks that turn in _cron_runs_holding once its acquire returns, until its release), the session and the key are left to that run's finalizer, so a reap or cancel of that live run still finds its key. When any other turn holds it (a tab that shows the job, a person's own), that turn keeps the session, left to the idle sweep, and the finished run's key is retired. A reset that raises still leaves the key registered.
Red on f07811b8db, green here: test_a_cron_run_whose_last_completion_timed_out_still_retires_its_key (this trigger: the real cron route, its injection timing out, the timeout arm's reset), test_the_idle_cleanup_retires_the_key_of_a_cron_parent_already_gone, and test_a_tab_or_user_turn_keeps_a_finished_runs_session_and_the_key_is_retired (a real CronService; the key leaves active_session_keys()). The run's own turn is pinned by test_the_real_cron_turn_keeps_its_key_while_a_confirmed_completion_is_cleaned_up (the real cron callback) and test_the_cron_runs_own_turn_keeps_its_key_for_its_finalizer. A mutant that keeps another holder's key, one that retires the run's own, and one where the engine never marks its turn each turn one of them red.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • span=ffbb335430d2 Honor parent teardown before rerouting parked completions: fixed in 0592d79154.

Upheld. The teardown check and its suppression now live in one function, suppress_retired_delivery (subagent_manager/terminal.py). The terminal report calls it before _on_done, with its behaviour and log line unchanged, and _finish_parked calls it before any route. A parked completion is retired when its own id, or any member it carries or holds, is in _teardown_cancelled_ids, so a flush-only record with a synthetic id is caught too. A suppressed park calls no route (no get_or_create, no prompt, no tab queue, no cron injection) and writes only its own marks. Its held siblings stay undelivered, as on the fresh path. A counted cron re-route gives its count back and runs the cron cleanup.
Every path that delivers a completion now reaches the parent only after that function: fresh, CONTINUE and the digest flush go through the terminal report; a parked completion goes through _finish_parked; and the cron, tab and channel routes are reached only from those two.
Red on 060ca46f22, green here: test_agent_off_while_a_completion_is_parked_delivers_nothing (the real !agent off handler, then the real release, _settle_inline and _finish_parked: no claim, no inject, only the own marks written), both cases of test_a_retired_member_suppresses_a_completion_whose_own_id_is_live, and test_a_suppressed_cron_re_route_gives_its_count_back_and_cleans_up. A mutant without the call, one that checks only the own id, one without the own-marks settle, and one without the count give-back each turn at least one of them red.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Punchline ("every residual case falls back to a duplicate rather than a lost result"): one did not on 060ca46f22, and it is fixed in 0592d79154. A turn cut without telling kiro-cli (a cron wake budget) left its spawn_sub_agents call running. A child that call first read done under a sibling's injection was stamped with that attempt, and the sibling's landing confirmed it: acknowledged, never delivered.
A mark is now held only under the attempt that spawned the child. api_spawn records it while the holder's own prompt is in flight, and one call_id per call ties every member to the attempt its first member named. The done read and the mark both compare against it. A call that outlived its attempt holds nothing under another one, whatever kiro-cli does with the cancel. Pinned by test_a_cut_turns_call_never_confirms_a_child_under_a_siblings_injection, red on 060ca46f22. The body's Residual drops the stdio-loop reliance and lists duplicates only.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Item 2 (the collected mark is stored twice): kept, for the slot set's one reader, as on 81bf50538. The two answer different questions. SubagentInfo._collected_inline says a child was returned inline. The tab slot's _subagents_inline_collected says which of that tab's returned children still have a completion to process. The synthesis disarm (_pending_synthesis) waits for that set to empty. Deciding the disarm from record flags would need the manager to list a parent's finished records. It has no in-memory query for that (running_agents_for lists running ones only), and this step of the delivery path stays await-free on purpose.
On 0592d79154 the set is still filled in one place, _settle_collected_on_slot, and emptied as each completion is processed. It is filled for a dashboard-born parent's mark, and for the landed release of the turn that received the children on a tab that shows a cron job or a channel conversation. This round does not change that.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at d8a4ba9 for human review.

The re-ported fix targets a real cost on main: a cron or channel parent pays one redundant completion turn per child its spawn_sub_agents call already returned. CI is green, the branch merges cleanly, and all AI lanes pass.

What holds it:

  1. Sensitive paths. The cron cleanup now resets the parent session only when no turn holds it and otherwise retires the run's key while another turn keeps the session (subagent_injection.py#L1374-L1453, run_registry.py#L69-L95). That changes what the reaper and job ownership can target for every cron engine, so a human must review it.
  2. Result-loss windows, no reply on this head. Design note 1 points out that the body names two windows where a cron or channel result can now be lost, where main injects every completion. Both depend on kiro-cli behaviour after session/cancel that nobody has verified. Please pin each window with a test, show it cannot happen, or open one issue per window and link it in the description.
  3. Maintainer scope ruling. The size question from our r100 hold and the 23 Oct triage (is removing the duplicate turns worth about 7k lines now that fix(acp): let a session's keepalive defer its tool-stall watchdog #17780 fixed the loss) has no recorded answer.

Non-blocking: Design note 2 (duplicate turns on a brokered or pre-upgrade tab) is a stated residual tracked by #18743. Design note 3 (a separate commit for the key helpers) is a review-shape suggestion.

— crew-pr-reviewer (r133)

A parent whose turn blocks in spawn_sub_agents got one redundant
completion turn per child the call had already returned, unless it was
a dashboard tab: mark-collected recorded ids only on a dashboard slot,
and the cron and channel routes asked nothing before injecting.

The tool now sends mark-collected from DeferredTool.on_answered, which
the stdio dispatcher runs after answering, with written=True only when
respond() wrote the whole frame into the client's own pipe, the
descriptor captured before any tool ran. A refused or uncaptured pipe,
a substituted error frame and a pooled backend's answer never count.
The mark names only the members whose whole block survives the
MAX_RESPONSE_LEN cut, and a mark without written=True changes nothing.

Bytes in kiro-cli's pipe are not a landed turn: a cron deadline, a stop,
a reap or a failure can end the turn before its answer is recorded, and
a turn cut without telling kiro-cli keeps its call running. So for every
parent that is not dashboard-born (parent_turns.settles_at_release), a
tab open on it or not, the gateway holds each mark, pending, only under
the attempt that spawned the child. An attempt is one turn inside one
holding of the parent's permit; its token is the session key, the
holding task and an epoch that no other attempt has. POST /api/spawn
from the parent's own tool records on the child the attempt that held
the session, with its prompt in flight
(SessionManager.turn_prompt_in_flight), when the request arrived: it is
read before the handler awaits anything, so a turn cut while the handler
awaits never lends the spawn its successor's attempt. Every member of
one spawn_sub_agents call carries that call's call_id, and a member
records an attempt only when it equals the one the call's first member
named (ParentTurnLedger.call_attempt). A call_id that is not ASCII
letters and digits, at most 32 characters, is a 400 coded
invalid_call_id. A member the admission gate queues carries the attempt,
in memory only, to the record the pump builds when it starts it
(carry_spawn_attempt). GET /api/spawn/{id} stamps a done read only under
the spawning attempt, and a mark is held only when the spawn and the
stamp both name the attempt holding now. A read or mark under any other
attempt, or after the spawning one ended, holds nothing, so the
completion is delivered: a duplicate at worst.

SessionManager.settle_turn ends an attempt: an injection's in-claim
retry and its session-not-found reload end the failed one there, and a
cron turn's one-shot CONTINUE re-prompt ends the interrupted prompt's.
Release ends the last one. Each engine notes landing where it already
decides it (note_turn_landed); a completion's own injected turn notes it
only when its stop reason landed, and the dashboard runner only for a
turn that ended on end_turn, so a person's turn in a cron job's or a
channel conversation's tab plays no second turn for its children. An
attempt's end confirms its marks when it noted landing and revokes them
otherwise, and an attempt a finished task left open is ended by the next
one opened.

The cron, channel and tab routes ask _inline_verdict; a dashboard-born
tab keeps _delivered_inline. A confirmed announce is acknowledged with
no turn; a pending one is parked with its tombstones owed and _on_done
returns at once; anything else is delivered. At the release
_settle_inline runs a parked completion through its route again: a tab
completion through the same steps _subagent_done takes, so a closed tab
hands it to the parent's own route, with its held wave siblings owed.
A parked completion whose parent was retired takes no route:
_finish_parked asks suppress_retired_delivery, the teardown check the
terminal report asks before _on_done, by its own id and every member it
carries or holds, and writes only its own marks. A landed release also
settles its children on a tab that shows the parent.

A re-routed completion of a cron parent is counted before the cron
turn's cleanup decides its reset. That cleanup
(_reset_cron_parent_when_idle) decides by who holds the session: a cron
run's own task, held in _cron_runs_holding from the start of
_cron_callback to its end, is left to that run's finalizer; another turn
keeps the session; an idle session is reset. A key is retired only when
the job's live run did not register it or is done with it
(CronService.retire_finished_session_key): each cron engine marks the
key it registered as it releases the session (finish_session_key), and
registering the key again takes the mark off, so a newer run waiting for
the permit keeps its own. A parked tab completion whose tab cannot be
read takes the parent's own route, and a counted re-route gives its
count back however it ends.

A dashboard-born tab still acknowledges on the mark, a window tracked
in kirodotdev#23495.

Fixes kirodotdev#23478
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Moved to #27508: same commits, now on a branch of this repo so CI uses the CodeBuild fleet. Review history stays here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

2 participants