Skip to content

fix(subagent): mark returned results only after the reply is written - #27508

Merged
bolichen97 merged 1 commit into
mainfrom
fix/17626-cron-inline-collected
Nov 2, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/17626-cron-inline-collected

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Moved from #17684 (fork) to a branch on this repo so CI runs on the CodeBuild fleet, per CONTRIBUTING.md. Earlier review threads and dispositions are on #17684.

Problem / Motivation

Goal: A cron or channel parent whose turn already got its children's results back from a blocking spawn_sub_agents call gets no second turn for them, whether or not a dashboard tab shows it, and no returned result is acknowledged until the parent turn that received it has landed.

On main a cron run that called spawn_sub_agents with three children took four turns: turns 2-4 were each a [Subagent completion event] for a child the call had already returned (#23434).

Why it matters

Every cron job and every channel conversation that uses spawn_sub_agents pays one redundant model turn per child. Skipping them must not lose a result: a deadline, a stop, a reap or a failure can end the turn after the reply reached kiro-cli and before its answer is recorded (#23478).

Not a goal

What changed (motivation → approach → change)

On main, mark-collected recorded returned ids only on a dashboard tab's slot, so cron and channel routes injected every completion, and it was sent before the write, so a torn or cut reply still suppressed its completions.

Step Change
Answer, then report _commit_settled (mcp_shared_dispatch.py) answers, then runs on_answered(written) (mcp_shared_calls.py): True only when respond() (mcp_shared_wire.py) wrote the whole frame into the client's pipe, never for a pooled backend. wave.py marks the members whose block survives the MAX_RESPONSE_LEN cut.
Hold the mark under its spawning attempt For every parent but a dashboard-born one (settles_at_release) a mark is pending only under the attempt (turn_token: key, task, epoch) that spawned the child. api_spawn records the attempt that held the session, with its prompt in flight (turn_prompt_in_flight), when the request arrived, read before the handler awaits anything; one call_id per call ties each member to its first member's attempt, a queued member carries it to the record the pump builds, and GET /api/spawn/{id} stamps a done read only under it.
Settle at the attempt's end Engines note landing where they decide it (note_turn_landed); runtime_charge.py only on a landed stop reason, the dashboard runner (turn_stats.py) only on end_turn. settle_turn ends an attempt (a failed in-claim one before its retry or reload), release the last; noted confirms, anything else revokes.
Park, never wait _inline_verdict (subagent_injection.py), asked by the cron, channel and tab routes: confirmed is acknowledged, pending is parked; the rest is delivered. At the release a parked completion takes its route again, unless its parent was retired (suppress_retired_delivery). The cron cleanup leaves the run's own task (_cron_runs_holding) to its finalizer, lets another turn keep the session, resets an idle one, and retires only a key the job's live run did not register or is done with (retire_finished_session_key; each cron engine marks its key with finish_session_key as it releases the session).

Since 0592d79154: the spawn reads its attempt when the request arrives, so a call's first member posted across a cut never records the successor's attempt; the call_id refusal carries code: invalid_call_id; the dashboard runner notes a turn that ended on end_turn, so a person's turn in a cron or channel tab plays no duplicate (main played none there); a cron engine marks its key finished as it releases the session, so the last cleanup retires it even while the run's claim is live (main retired it there); a parked tab completion whose tab cannot be read takes the parent's own route instead of waiting for the next start.

Residual, one duplicate turn and never a loss: a mark processed after its spawning attempt ended; an engine that notes no landing (a monitor nudge, a hook: session, side threads, workflow agents); on a cron or channel tab, a child that finishes before the reply is written, as on main; a member spawned while no prompt of its turn is in flight; a queued member after a gateway restart or one that ends unstarted. A channel member holds its permit while it listens and notes no landing, so its parked completions wait until it ends: a delay, never a loss. A spawn with no call_id (the CLI, spawn_run, a pre-upgrade kirocrew-core) is judged by its own holding; none of them sends a written mark. Two windows can still lose a result, and both need kiro-cli to stop waiting for a call the server is not told to drop: a cut turn's call whose first member's POST reaches the gateway only after the successor's prompt is in flight, when the backend keeps that call running after session/cancel; and a call kiro-cli gives up on without notifications/cancelled, answered into the open pipe while the same turn ends on end_turn.

Backwards compatibility

Compatible: respond() returns a bool where it returned None; mark-collected takes an optional written, and one without written: true changes nothing; POST /api/spawn takes an optional call_id. A pre-upgrade kirocrew-core gets its members' completion turns (duplicates, never a loss) until it restarts.

Reader: src/kiro_crew/slack/gateway_runtime/subagent_injection.py:_route_subagent_to_tab -- chat -- a brokered or pre-upgrade tab's unflagged members are injected: one duplicate turn per child (test_a_pooled_backends_written_reply_marks_nothing)
Reader: src/kiro_crew/dashboard/messaging_api/run_control.py:api_spawn_mark_collected -- chat -- an unwritten or foreign mark leaves a queued tab completion to play (test_a_queued_tab_completion_stays_without_its_written_mark)

Tests

  • test_spawn_sub_agents_duplicate_never_loss.py (new, 84): a whole write before any mark; the real cron callback and SessionManager; the channel route; sequence steps; !agent off with a parked completion; the parked tab route's teardown gate and an unreadable tab.
  • test_spawn_sub_agents_settle_at_release.py (new, 36): tabs; unlanded injected turns; late marks under a later attempt; a cut turn's call; the cron cleanup per holder.
  • test_spawn_sub_agents_spawning_attempt.py (new, 32): the spawning attempt through the real api_spawn and the real admission queue; the in-flight gate; call_id; a first member posted across a cut; a queued member stopped before it starts.
  • test_cron_run_holding.py (new, 9) and test_subagent_injection_session_reload.py (new, 1): the run's task holder, a newer run's key, a live run's finished key, the reload's settle.
  • test_chat_runner_notes_landing.py (new, 6): the real dashboard runner notes landing from the claiming task on end_turn only.
  • Rebased on main d9a20ee8a7. Main's newer seams are reused, not replaced: the mark-collected retry (_mark_collected, now called from on_answered with written: true), _skip_inline_collected for each carried member, admitted_cron_dispatch around the cron run's holding, and the _reported_to_parent filter inside _settle_collected_on_slot. Main tests set up for the mark this PR moves are updated in setup only, with their assertions unchanged: test_spawn_mark_collected_retry.py writes before the mark and expects written: true; four synthesis-arm tests in test_gateway_subagent_injection.py and one in test_subagent_queued_visibility.py set the record's _collected_inline and parent, which the slot set alone no longer decides; test_gateway_completion_skill_trigger.py and test_subagent_injection_channel_drain.py pass carried.
  • Updated pins: the cron cleanup, the cron retries, the sequence on_resume, finish_session_key before each release, the inventories, 26 cron gateways.
  • Each mechanism this round adds has a mutant that turns a named test red; the new files pass 5 repeats and 2 shuffled orders.
  • Every Fast Gate step, replayed from fast-gate.yml (the source-only ratchets, the error-code contract among them, and the flake ratchet); black, isort, flake8, mypy; semgrep 0 on our lines; the whole test/ directory: 215,274 passed, and every other id fails the same way on main on this host or passes when run alone.

Proof: PROVEN (prove.py, head d8a4ba9fd1, rebased on main ea38bdd5e5; not re-run on the d9a20ee8a7 rebase): with this PR's src/ hunks reverted, the changed tests fail on an assertion.

Manual verification

Real kiro-cli, unbrokered, an isolated gateway; one spawn_sub_agents call with three children per parent:

parent turns completion turns
main d2417a48c (2.33.1), cron job 4 3
d799f0daeb (2.35.1), cron job 1 0
d799f0daeb, cron job shown in its tab 1 1 tab turn, for an early child (the Residual)
d799f0daeb, dashboard tab 1 0
d799f0daeb, a person's turn in that cron job's tab 1 0

The cron run's own end-of-run reset cut the early child's tab turn once (Connection lost — retrying…), as it does on main, and the retry delivered it. The head adds a test-harness answer, docs and a comment to d799f0daeb.

Related Issues

Closes #23434
Fixes #23478

Sibling sweep

Swept: git grep -nE "_inject_with_retry|mark-collected|_subagents_inline_collected|rollback_skill_bodies\(|settles_at_release|_finish_parked" -- src/
Sibling: src/kiro_crew/slack/gateway_runtime/subagent_injection.py -- changed here (tab, channel and cron routes)
Sibling: src/kiro_crew/dashboard/chat_turn/turn_stats.py -- the dashboard runner, changed here (notes a landed turn); a dashboard-born tab's own acknowledgement is not changed (#23495)

Pattern harvest

Rule candidate: review-prompt
Pattern: a delivery acknowledged before the turn that received it landed.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Apache-2.0 (see CONTRIBUTING.md).

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners November 1, 2026 23:12
@jeeshofone
jeeshofone requested a review from bolichen97 November 1, 2026 23:12
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8e35878890df4c95060462a6f5cea26a1422b8aa touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8e35878890df4c95060462a6f5cea26a1422b8aa — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: when the ledger is unsure it re-delivers the result rather than dropping it, the new wire fields are optional, and the main reader that loses suppression is listed with its test.

Notes

  • Every turn engine has to call note_turn_landed itself. An engine that never does (monitor nudges, hook: sessions, workflow agents) silently keeps the duplicate turns. A ledger-level count of attempts revoked without a landing note would show which engines are missing it.
  • The description admits two windows where a result can still be lost, so the Goal's "no returned result is acknowledged until … landed" claims more than the code delivers. Track both windows in an issue, each with a pinning test.
  • The take-away check covered mark-collected without written and the slot-set reader _route_subagent_to_tab. It skipped the cron cleanup's retire path (retire_finished_session_key), because the Backwards-compatibility section may sit past the description's truncation notice.

[DESIGN-REVIEWED] 8e35878

Description read

[DESCRIPTION-READ] a3b69b2993457defc33b6098c5c15934e9d5648b0cc9da75e567e01b8a1e6cb0

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8e35878890df4c95060462a6f5cea26a1422b8aa — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item comes from the reported defect (#23434) or its no-loss constraint (#23478), and each new surface has a counted consumer.

Notes

  • Item 2 — oversized: the stated bar is "Skipping them must not lose a result". Yet the description says "Two windows can still lose a result", and main has neither, because it injects every completion. Subtraction: keep the skip off on the two named paths (a call whose first member posts after a cut, and a call kiro-cli drops without notifications/cancelled) so they deliver as they do on main, or defer those paths.
  • Item 6 — oversized: ParentTurnLedger.call_attempt with CALL_ANCHOR_LIMIT = 1024 and the call_id 400 validation is a second identity for one call, used only by wave.py (1 consumer). Subtraction: shrink it to a bounded dict and drop the coded invalid_call_id refusal; an unparseable id can be treated as absent.

What this change ships

Inventory (8 items) — 6 justified

Intent: cron and channel parents stop paying one redundant model turn for each child a blocking spawn_sub_agents call already returned, without losing results — a FIX.

  1. Cron and channel parents get no completion turn for a child whose result was already returned — justified
  2. A returned result is acknowledged only after the parent turn that received it lands; otherwise it is delivered — oversized (adds two loss windows)
  3. A child is marked collected only after the whole reply frame is written, never for a pooled backend — justified
  4. A completion that arrives while its mark is pending is parked and settled at release, never waited for in _on_done — justified
  5. The last sub-agent's cron cleanup no longer resets a session a live turn holds — justified
  6. POST /api/spawn takes an optional call_id, and mark-collected takes written — oversized (generalized anchor map)
  7. Cron engines mark their key finished as they release the session (finish_session_key) — justified
  8. The dashboard runner and every channel engine note that a turn landed (note_turn_landed) — justified

[FIRST-PRINCIPLES-REVIEWED] 8e35878

Description read

[DESCRIPTION-READ] b9c5e85b68b21003199b125939ba6c569cf169231f8719e844c26cf8869b3e64

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8e35878890df4c95060462a6f5cea26a1422b8aa and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8e35878

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8e35878890df4c95060462a6f5cea26a1422b8aa false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8e35878890df4c95060462a6f5cea26a1422b8aa — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • cron_fire.py:242: when fair admission refuses a fire, admit_and_run closes the unstarted _held(...) coroutine but never closes the dispatch coroutine inside it, so Python logs a "coroutine was never awaited" RuntimeWarning. The skipped run behaves the same either way.

[OPUS-REVIEWED] 8e35878

Verdict parsed from the review's SHA-scoped output markers for commit 8e35878890df4c95060462a6f5cea26a1422b8aa.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8e35878890df4c95060462a6f5cea26a1422b8aa false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 1, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 05230ad. One blocking issue, which agrees with the GPT and Opus lanes.

🔴 A parked inline completion loses the leaked-process flag (slack/gateway_runtime/subagent_injection.py, _park_inline). When the result was returned inline and the parent turn still holds it, _park_inline builds the owed delivery as SubagentDelivery(info.id, info.elapsed, info.credits) and sets _delivery_queued = True. The queued tab path in the same file (around line 553) passes leaked_process=getattr(info, "_leaked_process", False) is True, but this one does not. _delivery_queued also makes terminal.py skip its own leaked_process tombstone branch. So if the child's teardown could not end its process, the eventual delivered tombstone has no flag, list_leaked_processes never returns that run, and the process keeps running with nothing tracking it.

Suggestion: build own the same way _defer_queued_delivery does, passing leaked_process=getattr(info, "_leaked_process", False) is True, and add a test where a parked inline completion carries _leaked_process through to its settled tombstone.

This PR changes 94 files, so it also needs a human maintainer's approval in any case. I have not yet read the logs of the other red checks on this head, so I am not claiming anything about them here.

@jeeshofone
jeeshofone force-pushed the fix/17626-cron-inline-collected branch from 05230ad to 3d89a7a Compare November 2, 2026 00:16
@jeeshofone

Copy link
Copy Markdown
Contributor Author

GPT 6.1 Review on 05230ad — disposition (fixed in 3d89a7a)

  • F1 BLOCKING subagent_injection.py:171 (parked completion drops leaked_process) — Fixed. _park_inline now builds its owed mark with leaked_process=getattr(info, "_leaked_process", False) is True, the same expression _defer_queued_delivery uses. span=src/kiro_crew/slack/gateway_runtime/subagent_injection.py:174-179. Pinned by test_a_parked_completion_keeps_its_leaked_process_flag (test/test_spawn_sub_agents_settle_at_release.py); with the flag forced to False the [True] case fails. The flag is final when _park_inline reads it: the terminal report waits for teardown before _on_done runs.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Opus 5.5 Review on 05230ad — disposition (fixed in 3d89a7a)

  • BLOCKING subagent_injection.py:2258 (same defect) (parked completion drops leaked_process) — Fixed. _park_inline now builds its owed mark with leaked_process=getattr(info, "_leaked_process", False) is True, the same expression _defer_queued_delivery uses. span=src/kiro_crew/slack/gateway_runtime/subagent_injection.py:174-179. Pinned by test_a_parked_completion_keeps_its_leaked_process_flag (test/test_spawn_sub_agents_settle_at_release.py); with the flag forced to False the [True] case fails. The flag is final when _park_inline reads it: the terminal report waits for teardown before _on_done runs.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design Review on 05230ad (PASS) — notes

  • Loss windows on cron/channel routes: the PR body's residual names both, and each needs kiro-cli to stop waiting for a call the server is not told to drop (a backend that keeps a cancelled call running, or a call given up on without notifications/cancelled). Closing them needs a kiro-cli change, so a tracking issue is the right home; no code change here.
  • Engines opting into note_turn_landed: an engine that never notes landing revokes the mark and delivers, so it falls back to a duplicate, never a loss. Naming the note in the engine checklist in subagent.md is a docs follow-up outside this fix; no change in this PR.
  • subagent_injection.py size: 1,487 lines after this push, under the 1,600 split threshold. Split deferred to keep this diff to the fix.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 2, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 3d89a7a against 05230ad with range-diff.

Fixed: _park_inline now builds the owed delivery with leaked_process=getattr(info, "_leaked_process", False) is True, with a comment saying why. The new test_a_parked_completion_keeps_its_leaked_process_flag checks both values of the flag on the parked completion. No other hunk changed. The GPT and Opus lanes now report no blocking findings at this head.

Still red: Backend Tests (macOS) (2) fails three tests that this PR adds, all in test/test_spawn_sub_agents_spawning_attempt.py:

  • test_a_queued_member_that_starts_after_its_spawning_attempt_ended_holds_nothing
  • test_queued_members_of_one_call_are_confirmed_with_no_completion_turn
  • test_a_queued_member_stopped_before_it_starts_is_delivered_not_lost

Each one times out in async_wait_until after 30 s, two of them waiting on _start_the_queue.<locals>._all_started. The same shard was red on the previous head too, so this looks like a real problem on macOS, not a one-off. Please check why the queued members never all start there.

This PR changes 94 files, so it also needs a human maintainer's approval.

@jeeshofone
jeeshofone force-pushed the fix/17626-cron-inline-collected branch from 3d89a7a to cf36544 Compare November 2, 2026 01:11
@jeeshofone

Copy link
Copy Markdown
Contributor Author

macOS Backend Tests (2) on 3d89a7a — fixed in cf36544

Three tests in test/test_spawn_sub_agents_spawning_attempt.py start queued members through the real admission gate. That gate reads the host's free memory, and on the macOS runner the reading was below the spawn floor, so the members stayed queued and each async_wait_until timed out. Reproduced locally by pinning check_memory_available low (the same 3 fail). The module now uses the shared healthy_host_memory fixture (span=test/test_spawn_sub_agents_spawning_attempt.py:32-34), as other suites that drive the real gate do. With the low pin still in place, all 32 tests pass. No src change.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 2, 2026
A parent whose turn blocks in spawn_sub_agents got one redundant
completion turn per child the call had already returned, unless it was
a dashboard tab: mark-collected recorded ids only on a dashboard slot,
and the cron and channel routes asked nothing before injecting.

The tool now sends mark-collected from DeferredTool.on_answered, which
the stdio dispatcher runs after answering, with written=True only when
respond() wrote the whole frame into the client's own pipe, the
descriptor captured before any tool ran. A refused or uncaptured pipe,
a substituted error frame and a pooled backend's answer never count.
The mark names only the members whose whole block survives the
MAX_RESPONSE_LEN cut, and a mark without written=True changes nothing.

Bytes in kiro-cli's pipe are not a landed turn: a cron deadline, a stop,
a reap or a failure can end the turn before its answer is recorded, and
a turn cut without telling kiro-cli keeps its call running. So for every
parent that is not dashboard-born (parent_turns.settles_at_release), a
tab open on it or not, the gateway holds each mark, pending, only under
the attempt that spawned the child. An attempt is one turn inside one
holding of the parent's permit; its token is the session key, the
holding task and an epoch that no other attempt has. POST /api/spawn
from the parent's own tool records on the child the attempt that held
the session, with its prompt in flight
(SessionManager.turn_prompt_in_flight), when the request arrived: it is
read before the handler awaits anything, so a turn cut while the handler
awaits never lends the spawn its successor's attempt. Every member of
one spawn_sub_agents call carries that call's call_id, and a member
records an attempt only when it equals the one the call's first member
named (ParentTurnLedger.call_attempt). A call_id that is not ASCII
letters and digits, at most 32 characters, is a 400 coded
invalid_call_id. A member the admission gate queues carries the attempt,
in memory only, to the record the pump builds when it starts it
(carry_spawn_attempt). GET /api/spawn/{id} stamps a done read only under
the spawning attempt, and a mark is held only when the spawn and the
stamp both name the attempt holding now. A read or mark under any other
attempt, or after the spawning one ended, holds nothing, so the
completion is delivered: a duplicate at worst.

SessionManager.settle_turn ends an attempt: an injection's in-claim
retry and its session-not-found reload end the failed one there, and a
cron turn's one-shot CONTINUE re-prompt ends the interrupted prompt's.
Release ends the last one. Each engine notes landing where it already
decides it (note_turn_landed); a completion's own injected turn notes it
only when its stop reason landed, and the dashboard runner only for a
turn that ended on end_turn, so a person's turn in a cron job's or a
channel conversation's tab plays no second turn for its children. An
attempt's end confirms its marks when it noted landing and revokes them
otherwise, and an attempt a finished task left open is ended by the next
one opened.

The cron, channel and tab routes ask _inline_verdict; a dashboard-born
tab keeps _delivered_inline. A confirmed announce is acknowledged with
no turn; a pending one is parked with its tombstones owed and _on_done
returns at once; anything else is delivered. At the release
_settle_inline runs a parked completion through its route again: a tab
completion through the same steps _subagent_done takes, so a closed tab
hands it to the parent's own route, with its held wave siblings owed.
A parked completion whose parent was retired takes no route:
_finish_parked asks suppress_retired_delivery, the teardown check the
terminal report asks before _on_done, by its own id and every member it
carries or holds, and writes only its own marks. A landed release also
settles its children on a tab that shows the parent.

A re-routed completion of a cron parent is counted before the cron
turn's cleanup decides its reset. That cleanup
(_reset_cron_parent_when_idle) decides by who holds the session: a cron
run's own task, held in _cron_runs_holding from the start of
_cron_callback to its end, is left to that run's finalizer; another turn
keeps the session; an idle session is reset. A key is retired only when
the job's live run did not register it or is done with it
(CronService.retire_finished_session_key): each cron engine marks the
key it registered as it releases the session (finish_session_key), and
registering the key again takes the mark off, so a newer run waiting for
the permit keeps its own. A parked tab completion whose tab cannot be
read takes the parent's own route, and a counted re-route gives its
count back however it ends.

A dashboard-born tab still acknowledges on the mark, a window tracked
in #23495.

Fixes #23478
@jeeshofone
jeeshofone force-pushed the fix/17626-cron-inline-collected branch from cf36544 to 8e35878 Compare November 2, 2026 02:03
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Opus 5.5 Review on cf36544 — disposition (fixed in 8e35878)

  • BLOCKING messaging/turn_bracket.py:229 (a compaction-failure replay reuses the abandoned attempt, so its landing confirms marks only the abandoned turn received) — Fixed in the replay branch, where the attempt was left open. Before the reset, the abandoned turn's attempt now ends with settle_turn, the same seam the cron on_resume and _inject_with_retry use. Its marks are revoked and the replay opens a new attempt. span=src/kiro_crew/messaging/dispatch.py:1114-1116.
  • Sibling sweep: the Slack handler replays in the same task in two places, a transient compaction failure and a backend session loss, and both had the same gap. Both now settle before their reset. span=src/kiro_crew/slack/handler_runtime/turn.py:996-998 and span=src/kiro_crew/slack/handler_runtime/turn.py:1179-1181. Telegram, the task runner, heartbeat and cron do not replay a turn inside one claim after a reset, or they already settle first.
  • Tests: test_a_replay_ends_the_abandoned_attempt_before_its_reset (test/test_channel_turns.py, test/test_slack_handler_turn_outcomes.py) and test_slack_lost_session_ends_the_lost_attempt_before_its_reset (test/test_session_not_found_surfaces.py). Each fails with its settle_turn call removed.

First-pass check: R1-R8 OK.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 2, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 8e35878.

The compaction-replay blocker is fixed. Before the reset in the transient-compaction branch, settle_turn(sessions, session_key) now ends the abandoned attempt, so its inline marks are revoked and the replay runs as a new attempt. The fix covers both paths:

  • messaging/dispatch.py, before sessions.reset(session_key) (about line 1116).
  • slack/handler_runtime/turn.py, before both resets in _run_turn (the compaction replay and the session-not-found replay).

New tests: test/test_channel_turns.py, test/test_session_not_found_surfaces.py, test/test_slack_handler_turn_outcomes.py. The macOS fix from cf36544 (healthy_host_memory fixture) is still present.

CI: at 02:04 UTC every AI lane failed closed on a GitHub API rate limit while reading the merge-base. These were infrastructure failures with no findings, and I reran them. On rerun, Opus 5.5, GPT 6.1, Design, First Principles, UX, Sensitive Change and Security Scope all pass. Test shards and PR Readiness were still running when I checked.

Both of my earlier change requests are now addressed. The PR changes 94 files, which is above our 20-file limit for agent approval, so a human maintainer needs to approve and merge it.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 8e35878.

Both earlier change requests are fixed.

  • macOS timeouts: test/test_spawn_sub_agents_spawning_attempt.py now uses the healthy_host_memory fixture.
  • Compaction replay: settle_turn(sessions, session_key) now runs before the reset in messaging/dispatch.py and before both resets in slack/handler_runtime/turn.py, so the abandoned attempt's inline marks are revoked and the replay starts a new attempt. New tests in test/test_channel_turns.py, test/test_session_not_found_surfaces.py and test/test_slack_handler_turn_outcomes.py cover it.

How the 96 files were covered (above our 20-file review limit):

  • Every hunk in src/ and docs/ was read line by line. That includes parent_turns.py, session_parts/claims.py, slack/gateway_runtime/subagent_injection.py, dashboard/messaging_api/run_control.py, run_views.py, spawn.py, mcp_shared_wire.py, mcp_shared_dispatch.py, mcp_shared_calls.py, mcp_tools/spawn_parts/wave.py, the cron run registry, and every engine that now calls note_turn_landed.
  • Every test hunk over 5 lines was read, including the new files test_spawn_sub_agents_duplicate_never_loss.py, test_spawn_sub_agents_settle_at_release.py, test_spawn_sub_agents_spawning_attempt.py, test_cron_run_holding.py, test_chat_runner_notes_landing.py and test_subagent_injection_session_reload.py.
  • The remaining test edits of 1 to 5 lines were checked mechanically. Every changed line is one of these: a _cron_runs_holding = set() fixture line, a skip_if_busy=True assertion update, a written: True mark body, the on_answered(True) call, or a composition-contract signature row.

Trust boundaries checked:

  • mark-collected now flags only the claimed parent's own children, and only when the mark says the frame was written.
  • call_id is validated (ASCII letters and digits, at most 32 characters) before any spawn happens.
  • An attempt is named only for an internal caller whose X-Session-Key is the parent session, while that session's prompt is in flight.
  • No approval, permission or tool-allowlist path is touched.

CI: 0 red. Opus 5.5, GPT 6.1, Design, First Principles, UX, Sensitive Change and Security Scope all pass (the first run failed closed on a GitHub API rate limit and was rerun). The macOS shards, Coverage Combine and PR Readiness were still running when I checked. Auto-merge waits for them.

@bolichen97
bolichen97 enabled auto-merge (squash) November 2, 2026 02:41
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 2, 2026
@bolichen97
bolichen97 merged commit b053285 into main Nov 2, 2026
101 of 110 checks passed
@bolichen97
bolichen97 deleted the fix/17626-cron-inline-collected branch November 2, 2026 02:43
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 2, 2026
buluoray added a commit that referenced this pull request Nov 3, 2026
Cherry-picks (git cherry-pick -x), in this order:

- fix(i18n): raise the i18next floor to 26.4.4 so an unclosed nesting marker cannot hang the tab (#22326)
  (cherry picked from commit 7efc766)
- fix(desktop): make the Linux packages depend on libgbm (#28980)
  (cherry picked from commit 068b96b)
- fix(platform): read a macOS all-zombie process group as empty in pgroup_exists (#27615)
  (cherry picked from commit e1b9965)
- fix(platform): read a macOS all-zombie group's killpg EPERM as an emptied group (#27734)
  (cherry picked from commit 7d4da6b)
- fix(governance): judge cron turns by their session and fail closed on unattended turns (#27745)
  (cherry picked from commit 3fa3e80)

Conflict, hand-resolved:

- #27745, src/kiro_crew/slack/gateway_runtime/cron_turn.py (both
  stream calls): on main the added session_key=/unattended=True lines
  sit next to on_resume=, which #27508 added and 0.12 lacks. Took the
  two added keyword arguments at both sites and left on_resume= out, so
  #27508 is not pulled in. Every other file matches the source.
buluoray added a commit that referenced this pull request Nov 4, 2026
…29386)

Cherry-picks (git cherry-pick -x), in this order:

- fix(i18n): raise the i18next floor to 26.4.4 so an unclosed nesting marker cannot hang the tab (#22326)
  (cherry picked from commit 7efc766)
- fix(desktop): make the Linux packages depend on libgbm (#28980)
  (cherry picked from commit 068b96b)
- fix(platform): read a macOS all-zombie process group as empty in pgroup_exists (#27615)
  (cherry picked from commit e1b9965)
- fix(platform): read a macOS all-zombie group's killpg EPERM as an emptied group (#27734)
  (cherry picked from commit 7d4da6b)
- fix(governance): judge cron turns by their session and fail closed on unattended turns (#27745)
  (cherry picked from commit 3fa3e80)

Conflict, hand-resolved:

- #27745, src/kiro_crew/slack/gateway_runtime/cron_turn.py (both
  stream calls): on main the added session_key=/unattended=True lines
  sit next to on_resume=, which #27508 added and 0.12 lacks. Took the
  two added keyword arguments at both sites and left on_resume= out, so
  #27508 is not pulled in. Every other file matches the source.
bolichen97 added a commit that referenced this pull request Nov 7, 2026
A dashboard tab whose turn called spawn_sub_agents acknowledged each
returned child on the mark, so a turn the user stopped before it landed
lost the result. Every parent now holds the mark pending under the
spawning attempt and settles it at that attempt's release, as cron and
channel parents already do (#27508): a landed turn confirms it, any
other end delivers it. The specs and the brokered-delivery RFC no
longer describe the removed dashboard-born exception.

Builds on #28576 by @cruisercohen.

Closes #23495

Co-authored-by: Matt Cohen <mattcohe@amazon.com>
iamwhatever pushed a commit that referenced this pull request Nov 7, 2026
…32387)

A dashboard tab whose turn called spawn_sub_agents acknowledged each
returned child on the mark, so a turn the user stopped before it landed
lost the result. Every parent now holds the mark pending under the
spawning attempt and settles it at that attempt's release, as cron and
channel parents already do (#27508): a landed turn confirms it, any
other end delivers it. The specs and the brokered-delivery RFC no
longer describe the removed dashboard-born exception.

Builds on #28576 by @cruisercohen.

Closes #23495

Co-authored-by: Matt Cohen <mattcohe@amazon.com>
buluoray added a commit that referenced this pull request Nov 8, 2026
…32387)

A dashboard tab whose turn called spawn_sub_agents acknowledged each
returned child on the mark, so a turn the user stopped before it landed
lost the result. Every parent now holds the mark pending under the
spawning attempt and settles it at that attempt's release, as cron and
channel parents already do (#27508): a landed turn confirms it, any
other end delivers it. The specs and the brokered-delivery RFC no
longer describe the removed dashboard-born exception.

Builds on #28576 by @cruisercohen.

Closes #23495

(cherry picked from commit 2a4fb38)

Original author: Bolin Chen (#32387). docs/system-specs/modules/subagent.md hand-resolved: removed the dashboard-born-tab sentence this PR makes untrue and kept the rest of release's paragraph; main's #25261 sentence belongs to a main change not on release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants