Repository navigation
docs(rfc): decide how leaked agent runtimes are reclaimed - #15748
Conversation
4c98fda to
8c0e5b2
Compare
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- docs/request-for-change/rfc-leaked-runtime-reclaim.md:171 -- False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Description says claims verified at [DESIGN-REVIEWED] c4dfa6b Description read[DESCRIPTION-READ] 9dc94864e0f7f33b638789fe26d768d0c9f8d28c74f3fdb6480cc12790fdb53f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsThree statements the new RFC adds disagree with the code it cites at FINDING — docs/request-for-change/rfc-leaked-runtime-reclaim.md:84 — Decision 3 derives the spawn record's shape from "the lease and tenancy tables already key by pid and start id", but FINDING — docs/request-for-change/rfc-leaked-runtime-reclaim.md:100 — in a table the same paragraph calls main's behaviour ("On main FINDING — docs/request-for-change/rfc-leaked-runtime-reclaim.md:78,98 — both places the RFC positions [OPUS-REVIEWED] c4dfa6b Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
8c0e5b2 to
ffb5d1e
Compare
|
[operator: iamwhatever] CI reds on
Rebased onto |
buluoray
left a comment
There was a problem hiding this comment.
Reviewed at ffb5d1e9fa78eb3891a3bc63d70b4fb00a0046dc against origin/main (35171e238b; none of runtime_reconcile.py, session_scope_reap.py, session_pid.py, acp/runtime.py or apps/backend_runtime/pidfile.py has changed since 2db531ec89). Docs-only, and scripts/docs_lint.py passes.
Most of what the RFC says about current code is right. The reconciler's five conditions, DEFAULT_MAX_KILLS = 5, DEFAULT_MIN_AGE_SECS = 300, the reconcile_max_kills ceiling and its observe-at-0 behaviour, instance_slice_pids returning the empty set when the slice cannot be resolved, the report-only _is_untracked_managed_agent_orphan, _process_instance and the _record_app_pid precedent all match source. Protected pids reach the reconciler through _collect_active_pids → _protected_pids(), so "tracked or protected" really is on main today.
Four things need fixing before this is the decision record the First Principles lane reads.
Blocking
1. "The only arm that kills an unowned runtime" is false on main. The README row says runtime_reconcile.py "is the only arm that signals an unowned runtime", and Decision 1 says it is "the only code that kills an unowned runtime". That leaves out session_scope_reap.reap_abandoned_agent_scopes. It is wired into every cleanup tick (session.py:1353) and stops whole scopes in the same per-instance slice. Its first condition is that no member is tracked or active (session_scope_reap.py:26-40), so the scopes it ends are ones no record names. The Non-goals list "scope reaping" among paths that "end what their records name", which is the opposite of what it does. This matters because the RFC's goal is "one kill path for unowned runtimes, with one set of refusals", and main already has two. Fix: add the scope reaper to the current-state table. Then either bring it under Decision 1, or state that it keeps its own four-part check and explain why the new leader refusals (Decision 4) do not apply to it.
2. Step A gives the non-cgroup source a kill before it has install-scoped proof. The report arm's candidates come from _our_orphan_pids(), which covers every init-reparented process of this uid (session_pid.py:3796-3826). The marker it checks is the constant KIROCREW_SPAWNED=1 (constants.py:22), so these candidates are not limited to this install. The report arm's own docstring names a second install's runtime as a known false positive. Even so, step A's exit criterion says that on a non-cgroup host "a confirmed Reclaim ends it", and that is before step B adds the per-data-home record. This contradicts the Non-goal ("A process another data home spawned is never ours") and Decision 5 ("proven by Decision 3"). Decision 6's reason ("the user … knows no other install shares the data home") does not cover a second install with a different data home under the same uid. The security table misses that case too. Fix: either limit step A's confirmed Reclaim to slice candidates and leave non-cgroup candidates report-only until step B, or add a "same uid, different data home" row with a mitigation that actually holds before step B.
Non-blocking
3. Windows has no report-arm reading at all. _our_orphan_pids() returns [] on Windows (session_pid.py:3803), and _env_has_kirocrew_marker always returns false there (:3989). The Decision 5 Windows row ("report arm reading") and the #11991 Resolves row ("gains the dashboard reading") therefore promise a count that would always be 0 while the leak stays. Mark Windows as having no reading until it gets a native candidate source, and change #11991 to unresolved. macOS is fine: KERN_PROCARGS2 gives it a marker check, and the report arm's "Linux-only" docstring is out of date.
4. A recycled sid or pgid would refuse reclaim forever. Decision 4 refuses when "the candidate's sid/pgid names a live process other than the candidate". The launcher of a measured leak is dead, so as soon as its pid number is reused by any long-lived process, that leak can never be reclaimed. That is exactly the class of leak this RFC is meant to end. Main already handles this in _work_orphan_session_leader_alive (session_pid.py:4493-4519, leader_sid == sid). State the refusals as "names a live process that is still that session's / group's leader". This is the Fable suggestion, and it holds.
5. Frontmatter. implementation-prs: [14510] names the reconciler PR, which merged on 2026-09-28, before this RFC. It is not an implementation of this RFC. List the step A PR once it exists, or leave the field empty and keep #14510 under Related.
Bot dispositions
- Opus, "a second arm already kills": verified, and it is finding 1.
- Opus, "Windows reading always zero": verified, and it is finding 3.
- Fable, recycled-leader suggestion: verified, and it is finding 4.
- GPT, no findings: noted. Findings 1 and 2 hold regardless.
- CI reds: the author says they come from main, and the current head is a rebase. This diff touches no code, so the reds do not gate it.
ffb5d1e to
1f2fc03
Compare
|
[operator: iamwhatever] @buluoray thanks. All five addressed in
|
|
Read at 1. The record's lifecycle decides where it lives, not the file format. A leak is by definition a runtime that has already dropped out of the tracked records. Neither existing record keeps a row that long:
So "new columns on the existing PID file" only works if that row's removal rule changes. The rule Step B needs: write the row at spawn, before 2. Key it by runtime root, not by session. With chat runtime sharing (#15325), one root process serves many sessions, and the lease/tenancy tables already key by pid + start id. A per-session row would give N rows for one root, and the reconciler would have to agree with all of them. Root pid + start id + 3. Restart path. Step B's exit criterion covers "after a restart the reading still names the previous process's leaks". Worth adding the converse: a root that a new gateway adopts or re-leases after restart must be refused by the lease gate even though its spawn row is from the previous gateway. Decision 4 already gives the lease the veto; one sentence stating that ordering (lease veto first, spawn record second) would close it. No objection to Decisions 2, 4, 5, 6. |
1f2fc03 to
db4c9d1
Compare
|
Thanks, @buluoray — this was a careful read and every point held up against main. New head is 1 (blocking) — "the only arm that kills an unowned runtime" is false. Confirmed: 2 (blocking) — step A kills before install-scoped proof. Confirmed: 3 (non-blocking) — Windows reading always 0. Fixed. The Decision 5 Windows row is now "none / no reading" ( 4 (non-blocking) — recycled sid/pgid refuses forever. Fixed, as you (and Fable) prescribed. Decision 4 now reads the two leader refusals as "names a live process … that is still that session's / group's leader", matching 5 (non-blocking) — frontmatter. Fixed.
|
buluoray
left a comment
There was a problem hiding this comment.
Re-review at dc12c803ce6e73fcbb69b1b55bc5804926ebf7df against origin/main 57d71bb54e: requesting changes
The five items from my last round are answered. But this head adds a fold of the scope reaper, and it carries several statements about main that are not true. This document is the record the First Principles lane will read when step C gives the gateway new default kill authority, so those statements have to be right before it lands. Auto-merge is armed and there is another approval, so I'm holding it here.
I checked each item below against source myself. Most match Opus's findings on this SHA, which were filed as non-blocking.
Blocking
1. The Decision 4 table does not cover a detached setsid child, but the text says it does. After setsid, sid == pid and pgid == pid. _session_leader_alive returns sid != pid and … (runtime_reconcile.py:289) and _group_leader_alive returns pgid != pid and … (:298), so both answer false for that child. The refusals that actually protect it are in _why_not_reclaimable: "no readable spawn instance" (:918) and "another live process shares its spawn instance" (:926). Neither is in the table. "These four refusals cover each live shape seen so far" and Decision 1's claim that the refusals "subsume" the scope reaper's check both depend on this. Step C builds default reclaim from this table, so as written it would kill a live runtime's detached child. Fix: add the two spawn-instance rows to the table and drop the "four refusals cover each live shape" sentence and the subsumption claim (or restate them with the instance rows).
2. Not every live ACP spawn path is stamped. "Each ACP runtime spawn … stamps it beside KIROCREW_SPAWN_HOME" is true only of acp/runtime.py:2200-2204. AcpClient._spawn (acp/client.py:7214, env at :8075) sets only KIROCREW_SPAWNED. That path is live: providers/acp.py:473 and knowledge/llm_pool.py:354. _why_not_reclaimable refuses every unstamped candidate, so those runtimes can never be reclaimed and step C's exit criterion ("a non-cgroup host's leaked runtimes are reclaimed … with no manual pass") cannot be met. Fix: limit the sentence to AcpRuntime, add stamping AcpClient._spawn to step B, and scope step C's criterion to stamped spawn paths.
3. The owner gate is described the wrong way round. Step A says "owner-only GET /api/system/leaked-runtimes". api_leaked_runtimes (dashboard/handlers_system.py:816) has no _require_owner and returns per-pid RSS to any authenticated session. Only the reclaim POST (:837) is owner-gated. A security pass that trusts this row would believe the pid/RSS inventory is owner-only. Fix: "owner-only, confirm-required POST …/reclaim; GET …/leaked-runtimes authenticated, not owner-gated". Or name adding that gate as follow-up work.
4. The scope-reaper fold is decided but no step ships it. Last round I offered two options: keep its own check and explain why, or bring it under Decision 1. The 1f2fc03315 reply chose the first, and db4c9d1837 switched to the second. Either is fine. But Decision 1 now changes a shipped default kill arm, and steps A–C never carry that change. That also conflicts with the PR's frozen Not-a-goal ("Changing … scope reaping"), which Design flags. Fix: either give the fold its own rollout step with an exit criterion and a named pinning test (a scope that predates boot; a live group leader as candidate), plus a PR comment about the goal-line divergence. Or go back to "keeps its own four-part check" and say why Decision 4 does not apply.
Also fix (factual, small)
- The Decision 6 dashboard already shipped.
LeakedRuntimesCard.tsxmerged in #15764 (00b57642a) and is rendered atServicesTab.tsx:311, with count, RSS and a confirmed reclaim button. Problem 4 and the step A note ("The dashboard button of Decision 6 is not built yet") are false at the PR's base5a70ce7d3a. Add 15764 toimplementation-prs, and re-audit:audited-at: 4224aa2336predates that merge. - The reason for rejecting the signed mapping is wrong. Decision 3 says
_prune_stale_session_pid_files"drops it within one cleanup tick of the session's release". Its docstring and body (session_pid.py:2880) prune only a mapping whose pid is dead or recycled, and I found no unlink at release time. The mapping is keyed on the runtime root pid, which is the lifecycle Decision 3 asks for. Correct the parenthetical, and re-weigh that option in Open question 1.
Bot dispositions
- Opus (5 findings, "nothing blocks"): all five verified. Three are blocking items 1–3, two are under "Also fix". I disagree with the non-blocking framing: in a decision record, a wrong statement about what a kill gate refuses is the defect.
- Design CONCERNS: verified, item 4.
- GPT, no findings: noted.
CI: 82 success, 12 skipped, 0 failures. Docs only, so CI does not settle any of the above.
Records the decision before the code: one reclaim arm (the runtime reconciler), default-on with its per-pass budget and the reconcile_max_kills=0 observe switch, ownership proved by a persisted install-scoped spawn record that can only veto, four refusals that always win, Linux first, and a user-confirmed Reclaim on the dashboard. Step A (report + confirmed reclaim) adds no default authority. Steps B and C extend it, and the First Principles lane reads that decision from the base branch, so this document lands on its own. Refs #11899
buluoray: name reap_abandoned_agent_scopes as a separate arm that keeps its own four-part check; step A (#15755, #15764) refuses a candidate whose KIROCREW_SPAWN_HOME is not this data home, with a same-uid other-home security row; the reading GET is authenticated, only the reclaim POST is owner-gated; Decision 4 lists the two spawn-instance refusals that stop a detached setsid child; only AcpRuntime stamps the spawn instance and home, so step B stamps AcpClient._spawn and step C covers stamped paths; Windows has no reading and #11991 stays unresolved; the leader refusals read "still that session's / group's leader". chenmingwei23: the spawn record is one row per runtime root (root pid, start id, spawn instance), written at spawn before session/new and removed only once the root is gone, never on session release or lease drop; the lease veto comes before the spawn record, and step B's exit criteria cover a root a new gateway re-leases after a restart. Co-authored-by: cixuuz <26782336+cixuuz@users.noreply.github.com>
dc12c80 to
c4dfa6b
Compare
|
[operator: iamwhatever] @buluoray thanks, every point checked against Blocking
Also fix
@chenmingwei23 your three points are unchanged in Decision 3, Decision 4 and step B; the only scope change this round is removing the fold. |
buluoray
left a comment
There was a problem hiding this comment.
Re-review at c4dfa6bbba against origin/main: approving. All four blocking items and both small factual fixes from my last round are resolved, and I checked each new statement against main's source myself.
Previous blocking items
- Detached
setsidchild: fixed. Decision 4's table now carries the two spawn-instance rows (no readable spawn instance,spawn instance still live). Both exist on main in_why_not_reclaimable(runtime_reconcile.py~918 and ~926). The text now says plainly that the leader rows do not protect asetsidchild and that the instance rows do. The "four refusals cover each live shape" sentence and the subsumption claim are gone. - Unstamped spawn path: fixed. Current state names
AcpClient._spawnas a second live path that stamps onlyKIROCREW_SPAWNED, which matches main (acp/client.py~8075). Step B now stamps it, and step C's exit criterion is scoped to stamped spawn paths, so it can be met. - Owner gate: fixed. Step A now reads "owner-only, confirm-required
POST …/reclaim;GET …/leaked-runtimesauthenticated but not owner-gated", which matcheshandlers_system.py:_require_ownerappears only on the POST. - Scope-reaper fold: resolved by going back. The reaper stays a separate arm on its own four-part check, with a reason I agree with: its unit is a whole cgroup scope the kernel placed at spawn, and
setsidchanges session and group, not cgroup. That also brings Decision 1 back in line with the PR's frozen Not-a-goal ("Changing … scope reaping").
Previous small fixes
- #15764 credited: it is in
implementation-prsand in step A, andaudited-atmoved to6bf7a81920, which is on main. - Signed-mapping reason corrected: the parenthetical now says
_prune_stale_session_pid_filesremoves a mapping only when its pid is dead or recycled, and Open question 1 now weighs that mapping as a third option.
Follow-up (non-blocking)
GET /api/system/leaked-runtimesreturning per-pid RSS to any authenticated session is now described accurately. Whether it should be owner-gated is worth a decision in step B; it's a one-line_require_owner.
Docs only. CI was still running when I reviewed, so it does not settle any of the above.
Background sessions (kirocrew-lite: title gen, navigation, folder assignment, suggestions) run on a shared codex-acp adapter. Each session/new causes codex to start its per-session MCP fleet (~44 processes) from CODEX_HOME/config.toml. session/close evicts the ACP session but does not terminate those processes; they accumulate as children of the live codex app-server until the background runtime hits its 6-hour age ceiling and killpg reclaims the whole group. Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB after 4 hours. Fix: when spawning the kirocrew-lite background runtime, set private_state_env = 'CODEX_HOME' to redirect the whole config directory to a per-process private scratch. The new seed function _seed_background_codex_home prepares the private dir: - Writes a stripped config.toml via _codex_config_no_mcp (an allowlist filter identical to _provider_config_only in test_codex_session_mcp.py) that keeps model/provider settings but removes every process-spawning entry (mcp_servers in any TOML spelling, hooks, notify). The mcp_servers key in codex config uses named sub-tables ([mcp_servers.<name>]), not array-of-tables ([[mcp_servers]]), so a denylist regex cannot reliably match all forms; an allowlist is the right instrument. - Symlinks auth.json from the real CODEX_HOME (not a copy) so an OAuth token refresh updates only the real file with no divergence. Falls back to shutil.copy2 on platforms without symlink support. - Seeds the SQLite state database (same as seed_private_state) so rollout history is available immediately. Foreground runtimes keep the full operator CODEX_HOME unchanged. If the operator has already set CODEX_HOME, the redirect is skipped and the existing CODEX_SQLITE_HOME isolation path runs instead. Not in this PR: - RFC #15748 step B (persisted spawn record for AcpRuntime) - RFC #15748 step C (default reclaim on non-cgroup Linux) - AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping (step B second req) Fixes #13324
Background sessions (kirocrew-lite: title gen, navigation, folder, suggestions) run on a shared codex-acp adapter. Each session/new causes codex to start its per-session MCP fleet (~44 processes) from CODEX_HOME/config.toml. session/close evicts the ACP session but does not terminate those processes; they accumulate as children of the live codex app-server until the background runtime hits its 6-hour age ceiling and killpg reclaims the whole group. Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB after 4 hours. Fix: when spawning the kirocrew-lite background runtime (lazily imported from session.BACKGROUND_AGENT -- same circular-import avoidance as llm_helpers.py), set private_state_env='CODEX_HOME' to redirect the whole config directory to a per-process private scratch. The seed function _seed_background_codex_home prepares the private dir: - Writes a stripped config.toml via _codex_config_no_mcp. This is an allowlist filter that keeps only the scalar keys model, model_provider, forced_login_method, model_reasoning_effort, chatgpt_base_url, check_for_update_on_startup and all [model_providers.*] table sections. Everything else -- mcp_servers in any TOML spelling ([mcp_servers.name], inline tables, dotted keys), hooks, notify -- is excluded by not being on the list. Multiline string values for allowed keys are silently dropped (safe default vs corrupt TOML). Logic is identical to what test_codex_session_mcp.py previously called _provider_config_only; that helper is now removed and replaced with a production import. - Symlinks auth.json from the real CODEX_HOME (not a copy) so no credential copy ends up under the per-process scratch dir. On platforms without symlink support, fails closed (leaves auth.json absent in the private home). - Seeds the SQLite state database so rollout history is available. Foreground runtimes keep the full operator CODEX_HOME unchanged. If the operator has already set CODEX_HOME, the redirect is skipped and the existing CODEX_SQLITE_HOME isolation path runs instead. Deduplication: removes _provider_config_only from test_codex_session_mcp.py (was a copy of the same allowlist algorithm) and replaces its single usage with _codex_config_no_mcp from production code. Not in this PR: - RFC #15748 step B (persisted spawn record for AcpRuntime) - RFC #15748 step C (default reclaim on non-cgroup Linux) - AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping Fixes #13324
Background sessions (kirocrew-lite: title gen, navigation, folder, suggestions) run on a shared codex-acp adapter. Each session/new causes codex to start its per-session MCP fleet (~44 processes) from CODEX_HOME/config.toml. session/close evicts the ACP session but does not terminate those processes; they accumulate as children of the live codex app-server until the background runtime hits its 6-hour age ceiling and killpg reclaims the whole group. Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB after 4 hours. Fix: when spawning the kirocrew-lite background runtime (BACKGROUND_AGENT lazy-imported from session to avoid the circular import at module scope), set private_state_env='CODEX_HOME' to redirect the whole config directory to a per-process private scratch. The seed function _seed_background_codex_home prepares the private dir: - Writes a stripped config.toml via _codex_config_no_mcp. This is an allowlist filter that keeps only the scalar keys model, model_provider, forced_login_method, model_reasoning_effort, chatgpt_base_url, check_for_update_on_startup and all [model_providers.*] table sections. Those tables carry provider-specific auth. Everything else -- mcp_servers in any TOML spelling ([mcp_servers.name], inline tables, dotted keys), hooks, notify, any future process-spawning key -- is excluded by not being on the list. Multiline string values for allowed keys are silently dropped (safe vs corrupt TOML). auth.json is deliberately NOT included: provider auth lives in [model_providers.*] (kept by the filter) and API keys reach the child through the ambient environment (OPENAI_API_KEY, ANTHROPIC_API_KEY). This matches the live prompt test _codex_config_no_mcp approach. - Seeds the SQLite state database (same as seed_private_state) so rollout history is available immediately. Foreground runtimes keep the full operator CODEX_HOME unchanged. If the operator has already set CODEX_HOME, the redirect is skipped and the existing CODEX_SQLITE_HOME isolation path runs instead. Deduplication: removes _provider_config_only from test_codex_session_mcp.py (was a copy of the same allowlist algorithm) and replaces its single usage with the production _codex_config_no_mcp. Not in this PR: - RFC #15748 step B (persisted spawn record for AcpRuntime) - RFC #15748 step C (default reclaim on non-cgroup Linux) - AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping Fixes #13324
Background sessions (kirocrew-lite: title gen, navigation, folder, suggestions) run on a shared codex-acp adapter. Each session/new causes codex to start its per-session MCP fleet (~44 processes) from CODEX_HOME/config.toml. session/close evicts the ACP session but does not terminate those processes; they accumulate as children of the live codex app-server until the background runtime hits its 6-hour age ceiling and killpg reclaims the whole group. Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB after 4 hours. Fix: when spawning the kirocrew-lite background runtime (BACKGROUND_AGENT lazy-imported from session -- same pattern as llm_helpers.py), set private_state_env='CODEX_HOME' to redirect the config directory to a per-process private scratch. The seed function _seed_background_codex_home prepares the private dir: - Copies auth.json from the real CODEX_HOME (if present) so both API-key and ChatGPT-OAuth auth work in background sessions. A copy (not a symlink) because the background runtime is short-lived (max 6h then killpg recycles it), bounding copy staleness. - Writes a stripped config.toml via _codex_config_no_mcp. This is an allowlist filter that keeps only safe scalar keys (model, model_provider, forced_login_method, model_reasoning_effort, chatgpt_base_url, check_for_update_on_startup) and all [model_providers.*] table sections. Everything else -- mcp_servers in any TOML spelling ([mcp_servers.name], inline tables, dotted keys), hooks, notify, any future process-spawning key -- is excluded by not being on the list. Multiline string values for allowed keys are silently dropped (safe vs corrupt TOML). - Seeds the SQLite state database (same as seed_private_state) so rollout history is available immediately. Foreground runtimes keep the full operator CODEX_HOME unchanged. If the operator has already set CODEX_HOME, the redirect is skipped and the existing CODEX_SQLITE_HOME isolation path runs instead. Deduplication: removes _provider_config_only from test_codex_session_mcp.py (was a copy of the same allowlist algorithm) and replaces its usage with the production _codex_config_no_mcp. Not in this PR: - RFC #15748 step B (persisted spawn record for AcpRuntime) - RFC #15748 step C (default reclaim on non-cgroup Linux) - AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping Fixes #13324
Background sessions (kirocrew-lite: title gen, navigation, folder, suggestions) run on a shared codex-acp adapter. Each session/new causes codex to start its per-session MCP fleet (~44 processes) from CODEX_HOME/config.toml. session/close evicts the ACP session but does not terminate those processes; they accumulate as children of the live codex app-server until the background runtime hits its 6-hour age ceiling and killpg reclaims the whole group. Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB after 4 hours. Fix: when spawning the kirocrew-lite background runtime (BACKGROUND_AGENT lazy-imported from session to avoid the circular import at module scope -- same pattern as llm_helpers.py), set private_state_env='CODEX_HOME' to redirect the config directory to a per-process private scratch. The seed function _seed_background_codex_home prepares the private dir: - Copies auth.json from the real CODEX_HOME (if present) so both API-key and ChatGPT-OAuth (codex login) auth work. A copy, not a symlink: the background runtime is short-lived (max 6h then killpg recycles it), bounding copy staleness. - Writes a stripped config.toml via _codex_config_no_mcp -- an allowlist filter that keeps only safe scalar keys (model, model_provider, forced_login_method, model_reasoning_effort, chatgpt_base_url, check_for_update_on_startup) and all [model_providers.*] table sections. Everything else -- mcp_servers in any TOML spelling ([mcp_servers.name], inline tables, dotted keys), hooks, notify -- is excluded by not being on the list. Multiline string values for allowed keys are silently dropped. - Seeds the SQLite state database so rollout history is available. Foreground runtimes keep the full operator CODEX_HOME unchanged. If the operator has already set CODEX_HOME, the redirect is skipped. Deduplication: removes _provider_config_only from test_codex_session_mcp.py (was a copy of the same allowlist algorithm) and replaces its usage with the production _codex_config_no_mcp. Also updates providers/mirrors/codex.py to note that Crew now writes to a private CODEX_HOME for background sessions (the previously-recorded 'create-or-decline' gap this fills for the hooks channel). Not in this PR: - RFC #15748 step B (persisted spawn record for AcpRuntime) - RFC #15748 step C (default reclaim on non-cgroup Linux) - AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping Fixes #13324
…ONFIG
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter. Each session/new causes
codex to start its per-session MCP fleet (~44 processes) from the
mcp_servers entries in CODEX_HOME/config.toml. session/close evicts the
ACP session but does not terminate those MCP children; they accumulate
until the background runtime hits its 6-hour age ceiling and killpg
reclaims the group. Reporter hugoncosta observed ~855 processes / 21 GiB
after 4 hours.
Fix: at background-runtime spawn, read CODEX_HOME/config.toml (read-only),
enumerate the mcp_servers keys, and inject CODEX_CONFIG with every named
server set to enabled=false. McpServerConfig.enabled is documented in
codex-rs/config/src/mcp_types.rs line 234:
"When false, Codex skips initializing this MCP server."
Mechanism: CODEX_CONFIG is a JSON object merged into the Codex session
config (codex-acp README). Setting mcp_servers.<name>.enabled=false deep-
merges correctly (codex-rs/config/src/merge.rs merge_toml_values_at_path --
merging {enabled:false} into an existing server table adds the key). Server
names containing dots are safe: they travel as JSON object keys, not as TOML
dotted paths.
Shared CODEX_HOME, no auth.json copy or symlink: auth reaches the background
runtime through the ambient environment unchanged. CODEX_SQLITE_HOME
isolation is unchanged.
If the operator has already set CODEX_CONFIG with an explicit mcp_servers
key, that value is left as-is (operator override takes precedence).
_read_codex_mcp_server_names tries tomllib (3.11+) / tomli for structural
accuracy; falls back to a regex scan on older interpreters without tomli.
Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping
Fixes #13324
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter. Each session/new
causes codex to start its per-session MCP fleet (~44 processes) from
the mcp_servers entries in CODEX_HOME/config.toml. session/close evicts
the ACP session but does not terminate those MCP children; they accumulate
until the background runtime hits its 6-hour age ceiling and killpg
reclaims the group. Reporter hugoncosta observed ~855 processes / 21 GiB
after 4 hours.
Fix: at background-runtime spawn, read CODEX_HOME/config.toml (read-only,
off-loop) via tomllib, enumerate the mcp_servers keys, and build a
CODEX_CONFIG env var with every named server set to enabled=false.
McpServerConfig.enabled is documented in
codex-rs/config/src/mcp_types.rs line 234:
"When false, Codex skips initializing this MCP server."
Mechanism: CODEX_CONFIG is a JSON object merged into the Codex session
config (codex-acp README). Setting mcp_servers.<name>.enabled=false
deep-merges correctly (codex-rs/config/src/merge.rs
merge_toml_values_at_path: merging {enabled:false} into an existing
server table adds the key). Server names with dots are safe: they travel
as JSON object keys, not TOML dotted paths.
CODEX_CONFIG is placed in the new SpawnPlan.extra_env field. extra_env
is applied via setdefault semantics in launch.py so that an operator-set
CODEX_CONFIG in agent.env or os.environ takes precedence. It is
delivered unconditionally -- not gated on CODEX_SQLITE_HOME -- so it
works even when the operator has already set that variable.
Shared CODEX_HOME unchanged: no credential copy, no symlink.
CODEX_CONFIG skip condition: any existing CODEX_CONFIG in the spawn
environment (os.environ or agent.env) is left as-is.
Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping
Fixes #13324
Background sessions (kirocrew-lite: title gen, navigation, folder, suggestions) run on a shared codex-acp adapter. Each session/new causes codex to start its per-session MCP fleet (~44 processes) from the mcp_servers entries in CODEX_HOME/config.toml. session/close evicts the ACP session but does not terminate those MCP children; they accumulate until the background runtime hits its 6-hour age ceiling and killpg reclaims the group. Reporter hugoncosta observed ~855 processes / 21 GiB after 4 hours. Fix: at background-runtime spawn, read CODEX_HOME/config.toml via tomllib (stdlib 3.11+; pyproject.toml requires >= 3.12), enumerate the mcp_servers keys, and inject CODEX_CONFIG into the child environment with every named server set to enabled=false. McpServerConfig.enabled is documented in codex-rs/config/src/mcp_types.rs line 234: "When false, Codex skips initializing this MCP server." CODEX_CONFIG is a JSON object merged into the Codex session config per the codex-acp README. The deep-merge applies enabled:false to each existing server table (codex-rs/config/src/merge.rs). Implementation follows the opencode.py/pi.py pattern: - resolve_spawn (off-loop worker thread) computes the CODEX_CONFIG value and stores it in self._bg_codex_config. - apply_spawn_env consumes self._bg_codex_config after the full env (including agent.env) is assembled; it sets CODEX_CONFIG only when the env does not already carry one, so operator values take precedence. No changes to SpawnPlan, launch.py precedence, or CODEX_HOME. CODEX_SQLITE_HOME isolation is unchanged. CODEX_CONFIG skip condition: any existing CODEX_CONFIG in the fully assembled spawn env (os.environ or agent.env) is preserved. Not in this PR: - RFC #15748 step B / step C (persisted spawn record + auto-reclaim) - AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping Fixes #13324
Problem / Motivation
Goal: Record, before any code lands, who may kill a leaked agent runtime, on what proof, and in what order that authority rolls out.
A runtime that outlives its session and is in no record is ended by no sweep. On busy hosts these pile up until the host runs out of memory and the gateway restarts (#11899). Main has two arms that see this population: the runtime reconciler kills unowned processes in the agent cgroup slice by default, and the untracked-runtime report in
session_pid.pysees the non-cgroup population and only logs, its docstring reserving a kill for a maintainer ruling. The last attempt to add that kill (#12612) built a second kill path beside the reconciler and closed without a recorded decision.Why it matters
Non-cgroup Linux hosts have no reclaim at all, and every leak spawned before a gateway restart loses its proof of ownership, because the per-spawn token lives only in memory. The next PRs that extend the default kill authority need a decision on the base branch for the First Principles lane to read.
Not a goal
What changed (motivation → approach → change)
Adds
docs/request-for-change/rfc-leaked-runtime-reclaim.md(statusdraft) and its index row indocs/request-for-change/README.md. The RFC decides:runtime_reconcile.pyreconciler. The report arm feeds itswould_killreading; no second kill path.DEFAULT_MAX_KILLS;session.reconcile_max_kills=0stays the observe switch.KIROCREW_SPAWN_INSTANCEtoken plus spawn start time. Subtractive only: missing, mismatched or unreadable means no kill.authorize_runtime_kill), session leader alive, group leader alive, tracked or protected pid. sid/ppid never grant.Every claim about main is verified at
2db531ec89and named in the index row.Backwards compatibility
Compatible: docs only.
Tests
N/A — docs only.
python3 scripts/docs_lint.pypasses locally.Manual verification
N/A — docs only.
Related Issues
Refs #11899, #11789, #8133, #13324, #11991. Follows #14510 and the closed #12612.
Checklist