Skip to content

fix(subagent): tell a restart orphan's parent how to resume the run - #13317

Merged
bolichen97 merged 1 commit into
mainfrom
fix/orphan-resume-hint
Sep 24, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/orphan-resume-hint

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

After a gateway restart, a sub-agent run the restart caught before its first token is announced to its parent as:

Agent `6077b1bf` ❌ lost to gateway restart
No result was captured before the restart.

That is true of result.txt and false of the run. Its conversation — every turn and tool call kiro-cli persisted under ~/.kiro/sessions/cli/{sid}.json — is a file reconciliation keeps on purpose (retain-by-default), and spawn_continue(conversation="<id>", …) re-seeds the session map from the run's state.json and resumes it after a restart. The notice never says so.

Why it matters

Three such orphans landed in one digest during the 2026-09-23 memory-crash reports; each had run 20–60 minutes of tool calls. Told "no result", the parent re-spawns from scratch and pays for the same work twice, when one spawn_continue would have picked the run up where it stopped.

What changed (motivation → approach → change)

Before After
❌ lost to gateway restart … No result was captured before the restart. the same line, plus It had completed 6 turn(s); its last tool call was \shell`. Its conversation survived the restart: `spawn_continue(conversation="", task=...)` resumes it …` — only when the conversation is resumable

orphan_resume_hint (subagent_manager/monitoring.py) builds the extra line from the orphan's state.json (session_id, provider, conversation_key, turns, last_tool). The handle names the conversation's owner — the conversation_key's subagent id when the orphan was itself minted by spawn_continue, else the run's own id — so continuing never seeds a second session-map key onto a sid another run already owns. Whether the handle is offered is decided by the one rule SessionMap.get already applies before it hands a sid out, now hoisted into session_files_resumable(sid, provider) in src/kiro_crew/session_map.py: for kiro-cli the {sid}.json present and the {sid}.jsonl holding at least a turn (_RESUMABLE_JSONL_MIN_BYTES, the single definition get() reads too, replacing its bare literal); for any other backend session/load decides, so the handle is offered and spawn_continue refuses typed if the session is gone. A pruned, released or never-started kiro-cli conversation is never advertised. The hint never raises; a notice that cannot be decorated is still a notice. subagent.py imports the helper so the rebound _impl resolves it; docs/system-specs/modules/subagent.md, session.md and docs/system-specs/common/injected-messages.md (the owning spec of the [Subagent completion event] text) document the hint and the shared rule.

Backwards compatibility

Compatible: the notice keeps every existing line and gains one; SessionMap.get keeps its exact behaviour (its 10-byte JSONL bar is now the named constant it shares with the hint). No field, file or API changes.

Tests

  • test/test_subagent_persistence.py::TestOrphanReconciliation::test_a_lost_orphan_with_a_surviving_conversation_says_how_to_resume: no session file → no hint; .json with a nine-byte .jsonl → no hint (one under the bar SessionMap.get prunes at); a .jsonl with a turn → the hint with turns, last tool and the run id; a non-kiro provider → the hint without a file check, as SessionMap.get behaves; a conversation_key of subagent:orig0001 → the handle names orig0001, not the orphan's own id. Removing the msg += resume line turns it red.

Manual verification

N/A — unit coverage drives _notify_orphan end to end with the real session-map directory seam; the live gateway on this host must not be restarted to stage an orphan.

Screenshots / video

N/A — the change is one line of an injected completion notice; no frontend surface.

Issue link

No linked issue: surfaced while investigating the 2026-09-23 sub-agent death reports (ticket P519578361, internal); companion to #13301, which fixes the stop attribution on the same notices.

Pattern harvest

Not generalizable: the defect is a notice that reports one artifact's absence (result.txt) as the absence of the work, when a second artifact (the conversation) still carries it. Review-prompt candidate: when a terminal notice says "nothing was captured", check every artifact the run leaves behind, not the one the notice was written around.

@bolichen97
bolichen97 requested review from a team and patrigao September 24, 2026 09:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: A sub-agent orphaned by a gateway restart whose conversation still exists on disk is announced with its progress and the spawn_continue handle that resumes it, instead of only "no result was captured".
Not a goal: Changing what reconciliation keeps or deletes, resuming the run automatically, or touching the partial/complete result notices.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 2685c6f9a51cd468fadcaac086ddcae3674234ba and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2685c6f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2685c6f9a51cd468fadcaac086ddcae3674234ba: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 2685c6f9a51cd468fadcaac086ddcae3674234ba — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify once against a real restart orphan that spawn_continue with the advertised owner id resumes it — the hint now sends parents down that pre-existing path.

[DESIGN-REVIEWED] 2685c6f

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 2685c6f9a51cd468fadcaac086ddcae3674234ba — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The hint decorates only the no-result orphan; the mid-turn partial_result orphan pays the same re-spawn-twice cost and still gets no resume handle.

Not justified as shipped

  • Item 1 — symptom-level: the harm named ("the parent re-spawns from scratch and pays for the same work twice") holds equally for the partial_result restart notice, whose conversation survives by the same retain-by-default rule; 1 counted unfixed sibling among the three branches of _notify_orphan_impl (src/kiro_crew/subagent_manager/monitoring.py:533).
    Clears when: the partial_result branch carries the same hint, or the description states that branch is deliberately left and why.
  • Item 7 — undeclared: the notice path now stats session files in an executor before sending; harm-free rider, inventory only.

What this change ships

Inventory (8 items) — 6 justified

Intent: FIX — a restart orphan's parent is told "no result" and re-spawns, when the surviving conversation could be resumed with spawn_continue; provenance is the added test (red on base) and the in-code retain-by-default comment the notice contradicted.

  1. "Lost to gateway restart" notice adds turns, last tool, and the spawn_continue handle when resumable — symptom-level: one sibling branch left unfixed
  2. The handle names the continued conversation's original owner, not the orphan's own id — justified
  3. Last tool call in the notice is flattened, redacted, capped at 80 chars — justified
  4. Non-kiro backends always get the handle; spawn_continue itself refuses if gone — justified
  5. New session_map.session_files_resumable predicate, one rule shared with SessionMap.get — justified
  6. SessionMap.get's bare 10-byte bar becomes the named constant, behaviour unchanged — justified
  7. Orphan notify stats session files off the event loop first — undeclared, harm-free rider
  8. Three owning specs updated in the same commit — justified

[FIRST-PRINCIPLES-REVIEWED] 2685c6f

@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from 0aa1955 to 50f4c0a Compare September 24, 2026 09:38
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Hint ignores the provider branch SessionMap.get takes span=4590fc7d0bb9

fixed in 50f4c0a: the hint calls session_map.session_files_resumable(sid, provider), which answers True for a non-default provider exactly as get() hands the sid out without a file check (the resume decides; spawn_continue refuses typed if the session is gone). Pinned by the new provider="other-backend" arm of the test.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Hoist the 10-byte bar into session_map.py as one named constant span=5f2ba329b977

fixed in 50f4c0a: _RESUMABLE_JSONL_MIN_BYTES lives in session_map.py, SessionMap.get reads it through _jsonl_holds_a_turn (its bare 10 is gone), and the hint reads the same rule through session_files_resumable; session.md names the single definition.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 2 — a second spelling of the resumability bar span=8f69bd1ddb57

fixed in 50f4c0a: exactly one definition remains (session_map._RESUMABLE_JSONL_MIN_BYTES + session_files_resumable); both SessionMap.get and orphan_resume_hint read it, and monitoring.py no longer declares a threshold of its own.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 2685c6f9a51cd468fadcaac086ddcae3674234ba — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2685c6f

Verdict parsed from the review's SHA-scoped output markers for commit 2685c6f9a51cd468fadcaac086ddcae3674234ba.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2685c6f9a51cd468fadcaac086ddcae3674234ba: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 24, 2026
buluoray
buluoray previously approved these changes Sep 24, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 50f4c0a.

Verified: SessionMap.get behaves exactly as before (the 10-byte prune is now _RESUMABLE_JSONL_MIN_BYTES through a shared helper), and session_files_resumable applies the same rule per provider. The offered spawn_continue(conversation=<agent_id>) handle resolves, because _continue_prelude_impl seeds subagent:<id> from state.json, and a session that is gone gets a typed conversation_gone rather than a crash. The hint is additive and wrapped in try/except. Removing the resumability gate reddens test_a_lost_orphan_with_a_surviving_conversation_says_how_to_resume. 306 + 40 tests passed.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Comment says every reader goes through session_files_resumable, but get() calls _jsonl_holds_a_turn span=3c51f33c3bb7

fixed in d6f2af0 (wording, not code): the constant's comment and session.md now say exactly what the code does -- the single bar _RESUMABLE_JSONL_MIN_BYTES is read only through _jsonl_holds_a_turn, by get() (which prunes on it and needs the JSONL verdict separately from the .json check) and by session_files_resumable (the whole-rule predicate for a reader outside the module). Routing get() through the predicate would fold its two distinct prune branches into one and lose the "empty JSONL" log line, so the code stays and the claim is corrected.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Comment and session.md overstate that every reader goes through session_files_resumable span=2ae802583a75

fixed in d6f2af0: both texts now name _jsonl_holds_a_turn as the shared reader of the one constant and describe get() and session_files_resumable as its two callers; the code is unchanged and the drift the comment guards against remains impossible (one constant, one reader).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • injected-messages.md still enumerates the lost variant without the new line span=744ce9a19522

fixed in d6f2af0: the owning spec's ❌ lost to gateway restart bullet now describes the conditional resume line (turns, last tool, the spawn_continue handle, offered only when session_files_resumable holds) and links to the subagent spec's owning section.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 24, 2026
@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from d6f2af0 to e13c47e Compare September 24, 2026 11:26
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Changing _RESUMABLE_JSONL_MIN_BYTES to 1 leaves the test green span=3c51f33c3bb7

fixed in e13c47e: the test's not-resumable transcript is now nine bytes -- one under the bar -- so lowering the constant turns the assertion red.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 24, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 24, 2026 19:00
buluoray
buluoray previously approved these changes Sep 24, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at e13c47e.

Verified the new handle: subagent_id_from_conversation_key(state["conversation_key"]) or agent_id names the conversation's owner, which is the key _continue_prelude_impl builds (subagent:{conv_id}), so the hint resumes the same session-map key instead of putting a second key on the same sid. Reverting to {agent_id} or setting _RESUMABLE_JSONL_MIN_BYTES to 9 each turns the new test red. 157 passed.

Non-blocking:

  • The partial_result ("cut off mid-turn by gateway restart") notice gets no resume hint although its conversation survives the same way; either extend it or say in the body that only the "lost" notice is covered.
  • The branch conflicts with main in docs/system-specs/modules/subagent.md, so the rebase will need a fresh approval.

@bolichen97
bolichen97 dismissed stale reviews from buluoray and chenmingwei23 via 43a74bc September 24, 2026 19:11
@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from e13c47e to 43a74bc Compare September 24, 2026 19:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 24, 2026
@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from 43a74bc to 677c555 Compare September 24, 2026 19:42
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • session-file probes block the gateway event loop span=e9fb8da2b635

fixed in 677c555: _notify_orphan_impl now runs orphan_resume_hint through run_in_executor(maintenance_executor(), ...), the same offload seam this module already uses for its other file reads, so the exists()/stat() probe under a possibly network-backed KIRO_HOME no longer runs on the loop. No behaviour change otherwise; test_a_lost_orphan_with_a_surviving_conversation_says_how_to_resume still passes.

@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from 677c555 to 23b78bc Compare September 24, 2026 20:35
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • the orphan notice interpolates an unbounded, multi-line last_tool span=ff99f4bd81d6

fixed in 23b78bc: orphan_resume_hint now flattens and caps last_tool at the interpolation (" ".join(...split())[:80]), so a shell heredoc can neither put a blank line into the notice (the completion card's head/body split) nor ship the whole script; the test feeds a 500-char multi-line heredoc and asserts the hint is one line, bounded, and keeps the command's head. Mutation-checked: the assertion fails without the cap.

@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from 23b78bc to be89ffc Compare September 24, 2026 21:22
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • the 80-char cap runs before redaction, so a straddling credential ships as an unredacted prefix span=ff99f4bd81d6

fixed in be89ffc: the cap is now redact_and_truncate(" ".join(...), 80), so redaction runs over the whole command before the slice and the later whole-message redaction is no longer the only guard. The import is local to the helper because orphan_resume_hint is a module-level function, not an _impl method bind_component_globals rebinds onto subagent's namespace, so the module's TYPE_CHECKING-only _redact_and_truncate is not resolvable there. Test feeds a PAT straddling the 80-char boundary; mutation-checked (truncate-first fails it).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
After a gateway restart, a run the restart caught before its first
token is reported as "lost to gateway restart -- No result was captured
before the restart." That is true of result.txt and false of the run:
its conversation -- every turn and tool call kiro-cli persisted under
~/.kiro/sessions/cli/{sid}.json -- is a file reconciliation keeps on
purpose, and spawn_continue re-seeds the session map from the run's
state.json to resume it. The notice never said so, and parents
re-spawned from scratch, paying for the same tool calls twice.

The lost notice now carries the run's progress (turns, last tool) and
the spawn_continue handle, offered only when the conversation is
resumable by the bar SessionMap.get applies -- the .json present and
the .jsonl holding a turn -- read through the same directory it
consults, so a pruned, released or never-started conversation is never
advertised.
@bolichen97
bolichen97 force-pushed the fix/orphan-resume-hint branch from be89ffc to 2685c6f Compare September 24, 2026 21:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 24, 2026
@bolichen97
bolichen97 merged commit 82c9a58 into main Sep 24, 2026
133 of 138 checks passed
@bolichen97
bolichen97 deleted the fix/orphan-resume-hint branch September 24, 2026 22:37
@buluoray

Copy link
Copy Markdown
Contributor

Post-merge record. This landed (82c9a58, merged by @bolichen97 at 22:37Z) while the verification below was still running, so it carries no gating effect — the verdict would have been an approve. The premise holds and every decisive claim in the description checks out against source at 2685c6f9a51cd468fadcaac086ddcae3674234ba.

What I verified

The hoisted rule is behaviour-preserving. session_files_resumable applies exactly what SessionMap.get applies: the same provider normalization ((provider or PROVIDER_LABEL_DEFAULT) != PROVIDER_LABEL_DEFAULT → hand the sid out unchecked, session_map.py:90 vs :904), then {sid}.json present and _jsonl_holds_a_turn. The removed bare literal was jsonl_size < 10; the helper answers size >= _RESUMABLE_JSONL_MIN_BYTES with _RESUMABLE_JSONL_MIN_BYTES = 10, and not (size >= 10) is size < 10. The FileNotFoundError → 0 branch is preserved. 320 tests pass across the seven test_session_map_*.py files plus test_subagent_persistence.py.

The noqa: F401 import in subagent.py is load-bearing, not cargo cult. bind_component_globals (subagent_manager/_component.py:37) rebinds each _impl onto subagent's namespace, and its own docstring states an import at the defining module's top is inert for the rebound function. orphan_resume_hint is loaded as a global by _notify_orphan_impl, so it has to resolve in that namespace — same reason tombstone_recovery_action was already imported there. maintenance_executor is already present (subagent.py:75).

The handle's owner resolution is correct. subagent_id_from_conversation_key (subagent_persistence.py:326) returns the id after SUBAGENT_CONVERSATION_PREFIX or None, so or agent_id falls back to the run's own id for a run that was not minted by spawn_continue. One session-map key per sid holds.

Redaction order. redact_and_truncate(text, 80) redacts the whole string before the slice by its own contract (security/__init__.py:2834), so passing the unflattened-then-flattened command in full and letting the helper cap it is the right shape — the cap cannot cut a PAT into an unmatched prefix.

The test is load-bearing. Four independent mutations, each red on test_a_lost_orphan_with_a_surviving_conversation_says_how_to_resume, tree restored clean after each:

Mutation Result
drop msg += f"\n{resume}" FAILED
owner = agent_id (ignore conversation_key) FAILED
drop " ".join(...split()) flatten FAILED
redact_and_truncate(text[:80]) (cap before redact) FAILED

flake8 clean on all four changed source/test files. CI at this head: 79 success, 10 skipped, 0 failures; all three blocking lanes green and Design Review PASS.

Follow-up on main

First Principles' open CONCERNS (Item 1) is fair and undispositioned at this head: _notify_orphan_impl has three branches and only the else (no result) one carries the hint. The partial_result and whole-has_result branches describe runs whose conversation survived by the same retain-by-default rule, and their parents can re-spawn for the same reason.

This was never grounds to hold the PR. The branch fixed is the one that actively misinforms — No result was captured before the restart. reads as no work happened, whereas the other two already point the parent at a file with content, so the omission there costs a resume opportunity rather than asserting something false. Extending the hint is additive and belongs in its own change. Worth either a follow-up issue or one line in the description saying those branches are deliberately left.

Not verified

No mypy run (no venv in the review worktree); macOS suite still gated on Decide whether the macOS suite runs. Manual staging of a real restart orphan was not attempted, matching the PR's own note.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…irodotdev#13317)

After a gateway restart, a run the restart caught before its first
token is reported as "lost to gateway restart -- No result was captured
before the restart." That is true of result.txt and false of the run:
its conversation -- every turn and tool call kiro-cli persisted under
~/.kiro/sessions/cli/{sid}.json -- is a file reconciliation keeps on
purpose, and spawn_continue re-seeds the session map from the run's
state.json to resume it. The notice never said so, and parents
re-spawned from scratch, paying for the same tool calls twice.

The lost notice now carries the run's progress (turns, last tool) and
the spawn_continue handle, offered only when the conversation is
resumable by the bar SessionMap.get applies -- the .json present and
the .jsonl holding a turn -- read through the same directory it
consults, so a pruned, released or never-started conversation is never
advertised.

Co-authored-by: w <w@k.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants