Skip to content

fix(session): reach process groups after leader reap - #10431

Closed
satyam-thakur wants to merge 1 commit into
kirodotdev:mainfrom
satyam-thakur:fix/preserve-agent-process-group
Closed

satyam-thakur wants to merge 1 commit into
kirodotdev:mainfrom
satyam-thakur:fix/preserve-agent-process-group

Conversation

@satyam-thakur

@satyam-thakur satyam-thakur commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Agent cleanup keeps only the leader PID. If that leader exits and is reaped first, getpgid(pid) fails. A child process that ignores the graceful stop signal (SIGTERM) can then survive cleanup.

Why it matters

The surviving process tree keeps its memory, file descriptors, and background work. Repeated sessions can leave several trees behind until the host slows down or the user kills them by hand.

What changed (motivation → approach → change)

Linux/macOS process groups are now captured when the agent starts. The capture records the leader's start identity and every group member seen while that leader is still proven current. ACP clients refresh these member witnesses before teardown so late child processes are included.

Cleanup signals the saved group only when it can prove that the group is still ours. A live leader must match its saved identity. After the leader is reaped, one saved member must still be the same process and still belong to that group. Recycled, moved, missing, malformed, or unreadable identities are refused. A refused group never falls back to the same number as a PID.

Direct ACP teardown uses the same saved-group check for SIGTERM and SIGKILL. This includes a leader that exits during the graceful stop wait and a leader that was already reaped before cleanup starts. Already-reaped cleanup also runs the saved escaped-child sweep after group signaling, so a child that moved out of the group is still removed. LLMProvider.spawned_process_group() exposes the saved group through one public capability. Pool and dashboard cleanup preserve that group after the leader PID is cleared. Potentially long process scans run outside the async event loop.

flowchart LR
  subgraph Before
    A1[spawn leader]:::ctx --> B1[save leader PID]:::removed --> C1[leader reaped]:::ctx --> D1[child survives]:::removed
  end
  subgraph After
    A2[spawn leader]:::ctx --> B2[save group witnesses]:::added --> C2[leader reaped]:::ctx --> D2[check surviving member]:::changed --> E2[signal saved group]:::added
  end
  classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px
  classDef changed fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:2px
  classDef removed fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-dasharray:4 3
  classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E
  linkStyle 0,1,2 stroke:#DC2626,stroke-dasharray:4 3
  linkStyle 3,4,5,6 stroke:#16A34A,stroke-width:2px
Loading

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Cleanup now keeps proof of the original process group, so it can remove child processes after the leader is reaped.

Tests

  • Added a real Linux/macOS process-cleanup test that reaps the leader before cleaning up a child process that ignores the graceful stop signal (SIGTERM). Its fixture now holds the leader until the child witness is captured, then releases and reaps it.
  • Added deterministic coverage for recycled, unreadable, empty, moved, and leaderless groups; per-signal authorization; cleared leader PIDs; and rejected-group behavior.
  • Added direct ACP teardown coverage for leaders that exit during the graceful stop wait or are already reaped, already-reaped escaped-child cleanup, terminal group rejection, no-group behavior, off-loop authorization, and import-cycle safety.
  • Updated three POSIX process-group tests to enable their Linux/macOS branches explicitly when they run on Windows, while isolating unrelated Windows file locking.
  • Removed process-global garbage collection from the bare AsyncMock capability test; direct call and await counts now prove the capability was not invoked.
  • Affected teardown, provider, dashboard-reset, and import-cycle suite: 1063 passed and 1 skipped.
  • The complete backend suite was attempted but did not pass on this host because existing host-environment failures occur outside the affected tests.
  • Every other resolved gate passed, including formatting, lint, type checks, docs, policy checks, frontend builds and tests, Electron tests, and bundle analysis.

Manual verification

A controlled Linux process tree confirmed the failure mode and fix. The leader exited and was reaped. The child ignored SIGTERM. Saved-group escalation removed it. Windows behavior has deterministic unit coverage.

Related Issues

Fixes #10245

Pattern harvest

Rule candidate: review-prompt
Pattern: A saved PID or process group used after a reap must carry process-incarnation proof, and failed proof must not fall back to the same numeric ID.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@satyam-thakur
satyam-thakur requested review from a team and iamwhatever September 12, 2026 21:49
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 12, 2026
@satyam-thakur

Copy link
Copy Markdown
Contributor Author

Intent: Keep agent process groups reachable through teardown after their leader exits and is reaped, without signaling a recycled group.
Not a goal: Change Windows process cleanup or add general process supervision.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of e417445bdfc043f82e8e3bd19a8cad79e7d83663 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the new real-child fixtures in test_pid_lifecycle stay flake-free on macOS/CI runners, per testing-conventions' real-spawn and reap rows.

[DESIGN-REVIEWED] e417445

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed e417445bdfc043f82e8e3bd19a8cad79e7d83663 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/acp/client.py:3813 -- function-local "from kiro_crew.session_pid import _signal_saved_group" lacks the required circular-import annotation -> Fix: add a # circular import: comment explaining the cycle.
[GPT-REVIEWED] e417445

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of e417445bdfc043f82e8e3bd19a8cad79e7d83663 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm the new teardown-time witness scan (unbounded /proc reads inside shutdown) cannot push pooled discards past _POOL_DISCARD_TIMEOUT on wide process trees.

Not justified as shipped

  • Item 9 — rides along: black-baseline removal and whole-file reformat hunks in two test files the description never mentions; tooling-mandated once the files were touched, harm-free.

What this change ships

Inventory (10 items) — 9 justified

Intent: FIX — a child that ignores SIGTERM survives cleanup once its group leader is reaped first (linked issue #10245, real failing-on-base process test).

  1. A SIGTERM-ignoring child now dies at teardown after its leader is reaped — justified
  2. A group failing its identity gate is signalled nothing, never falling back to the pid — justified
  3. Already-reaped client teardown now signals the saved group and sweeps escaped children — justified
  4. Runtime teardown now group-SIGKILLs even after a graceful leader exit — justified
  5. Providers answer one public spawned_process_group() capability (2 counted consumers: _sync_kill_provider, pool discard) — justified
  6. platform_compat gains the incarnation-proved group toolkit (capture, record, gate, kill_pgroup; AcpRuntime._child_pids is genuinely never populated, so no existing record serves it) — justified
  7. Dashboard reset-all runs the force-kill off the event loop, awaited — justified
  8. Pool discard re-reads the group after shutdown and can hard-kill a leaderless tree — justified
  9. Two test files reformatted and dropped from the black baseline — rides along
  10. Specs updated in the same commit (platform-compat row, session.md section) — justified

[FIRST-PRINCIPLES-REVIEWED] e417445

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed e417445bdfc043f82e8e3bd19a8cad79e7d83663 via the fork AI-review pipeline; updated in place on each push.

Review details

I've now examined the candidate against the actual code and the platform helpers.

Analysis of Candidate 1 (_kill_process force-fallback skipped when group is not None):

The candidate's leak requires the incarnation gate to refuse while our child is genuinely alive and ours. Tracing the gate (pgroup_matches_incarnation) for a live, unreaped leader:

  • The new code only reaches the force block when self._process.returncode is None, i.e. asyncio has not reaped the child — so pgid is not recyclable and the leader is alive and in its own group.
  • With the leader alive: pgroup_exists(pgid) and pid_exists(pgid) are both True, so the gate lands in the third branch and reads get_process_start_id(pgid).
  • For a genuinely-alive owned process, that read (/proc/<pid>/stat field 22 on Linux; libproc on macOS) succeeds and returns leader_start_id → the gate accepts and killpg fires (sent=True). No leak.

The only path to a refusal for a still-alive leader is get_process_start_id transiently returning empty — which the candidate itself concedes it "could not verify" and rates only "medium," qualifying it as "most plausible on macOS." That is a "could/might" trigger, not a concrete input that occurs in practice. For a live process the identity read does not transiently fail on Linux, and the "recycled to a stranger" case (the whole reason self._process.kill() is withheld) is impossible while the child is unreaped.

The withholding is also a deliberate, spec-documented invariant ("A captured group never degrades to the leader PID"), and even under a hypothetical transient refusal _kill_escaped_children still runs and later discard/orphan sweeps retry. So (a) is speculative and (c) is not a durable observable failure. It fails the Step-1 bar.

No new Step-2 finding grounds to 80+: the reaped-leader path, the non-force SIGKILL escalation, and the group-None fallback preservation all check out against the diff.

No findings.

[OPUS-REVIEWED] e417445

@satyam-thakur

Copy link
Copy Markdown
Contributor Author
  • Group authorization and signaling remain racy — needs-a-decision

The leaderless path has a check-to-signal window: the last witness could exit and the numeric process-group ID could be reused after authorization but before killpg.
Linux/macOS provide no atomic process-group handle or signal-with-incarnation operation. Signaling only witnessed members would no longer guarantee cleanup of every surviving descendant, weakening this PR's intent.
The current code minimizes the unavoidable window by authorizing immediately before each signal and treats an empty group as no signal sent.

Maintainer decision: accept this platform residual to retain group-complete cleanup, or require a larger design that keeps a dedicated leader or supervisor alive through teardown?

@satyam-thakur
satyam-thakur force-pushed the fix/preserve-agent-process-group branch from 9f49739 to 9382a91 Compare September 13, 2026 07:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 13, 2026
@satyam-thakur
satyam-thakur force-pushed the fix/preserve-agent-process-group branch from 9382a91 to b0ee622 Compare September 13, 2026 20:37
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 13, 2026
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 14, 2026
Keep the process-group identity captured at spawn so cleanup can reach
descendants after the group leader has been reaped. Deriving the group
from the leader pid at signal time fails exactly when it matters:
os.getpgid raises once the leader is gone, so a descendant that ignores
SIGTERM survives teardown unsignalled.

Both spawn sites record the group while the leader is provably alive and
expose it through one public LLMProvider capability, bound to the
leader's start identity so a recycled pgid is refused rather than
signalled. Preserve that group across pooled shutdown, and resolve it
before the pid guards in _sync_kill_provider: shutdown clears the client
pid while the captured group remains, so a pid-first return left the
group unsignalled and leaked the resistant descendant. A cleared pid
over a live group now escalates group-only, with the incarnation
rechecked before every signal and no pid fallback.

A leaderless group needs an original member to authorize it. Liveness
alone cannot separate our own surviving tree from a stranger that
inherited the pgid and then lost its own leader, because both reach the
same populated-but-leaderless shape. Carry ProcessGroupMember witnesses
recorded while the leader is still alive to vouch for the group, and
authorize a reaped-leader group only when one witness is still alive,
still the same incarnation and still in this group. An unreadable
identity, a witness that left the group, and a group carrying no
witnesses all refuse.

Cover the reaped-leader, leaderless-witness and cleared-pid paths with
deterministic lifecycle tests.
@satyam-thakur
satyam-thakur force-pushed the fix/preserve-agent-process-group branch from d109970 to e417445 Compare September 16, 2026 17:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 16, 2026
@satyam-thakur

Copy link
Copy Markdown
Contributor Author

Superseded by #11309, which landed the same fix: teardown now signals the process group authorized by group ownership (_pgroup_still_ours), holds the leader's zombie until the last group signal, and reaps it only after.

One residual remained — a leader already reaped when teardown starts leaves no readable group id, so no group signal is sent at all. That is now #11456, at 51 production lines instead of this PR's 3,188. Closing in favour of those two.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refused speculative session leaks the agent process tree (single-PID kill)

1 participant