Skip to content

docs(rfc): decide how leaked agent runtimes are reclaimed - #15748

Merged
bolichen97 merged 2 commits into
mainfrom
docs/rfc-leaked-runtime-reclaim
Oct 3, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
docs/rfc-leaked-runtime-reclaim

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Record, before any code lands, who may kill a leaked agent runtime, on what proof, and in what order that authority rolls out.

A runtime that outlives its session and is in no record is ended by no sweep. On busy hosts these pile up until the host runs out of memory and the gateway restarts (#11899). Main has two arms that see this population: the runtime reconciler kills unowned processes in the agent cgroup slice by default, and the untracked-runtime report in session_pid.py sees the non-cgroup population and only logs, its docstring reserving a kill for a maintainer ruling. The last attempt to add that kill (#12612) built a second kill path beside the reconciler and closed without a recorded decision.

Why it matters

Non-cgroup Linux hosts have no reclaim at all, and every leak spawned before a gateway restart loses its proof of ownership, because the per-spawn token lives only in memory. The next PRs that extend the default kill authority need a decision on the base branch for the First Principles lane to read.

Not a goal

  • Any code change. Step A (report + confirmed reclaim) is a separate PR and adds no default authority.
  • Changing tracked teardown, scope reaping or app-backend stale reap.
  • Default reclaim on macOS or Windows.

What changed (motivation → approach → change)

Adds docs/request-for-change/rfc-leaked-runtime-reclaim.md (status draft) and its index row in docs/request-for-change/README.md. The RFC decides:

  1. One reclaim arm: the existing runtime_reconcile.py reconciler. The report arm feeds its would_kill reading; no second kill path.
  2. Default-on with the per-pass budget DEFAULT_MAX_KILLS; session.reconcile_max_kills=0 stays the observe switch.
  3. Ownership proof is install-scoped: a persisted per-data-home record of the minted KIROCREW_SPAWN_INSTANCE token plus spawn start time. Subtractive only: missing, mismatched or unreadable means no kill.
  4. Refusals that always win: lease (authorize_runtime_kill), session leader alive, group leader alive, tracked or protected pid. sid/ppid never grant.
  5. Linux (cgroup and non-cgroup) first; macOS and Windows report-only until each has a native proof.
  6. Dashboard leaked count + RSS and a user-confirmed Reclaim that runs the same gates once.
  7. Rollout A (report + confirmed reclaim) → B (persisted record) → C (non-cgroup default reclaim), each with exit criteria.

Every claim about main is verified at 2db531ec89 and named in the index row.

Backwards compatibility

Compatible: docs only.

Tests

N/A — docs only. python3 scripts/docs_lint.py passes locally.

Manual verification

N/A — docs only.

Related Issues

Refs #11899, #11789, #8133, #13324, #11991. Follows #14510 and the closed #12612.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner October 1, 2026 06:54
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-leaked-runtime-reclaim branch from 4c98fda to 8c0e5b2 Compare October 1, 2026 06:55
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of c4dfa6bbba930e424232129686c3b1d77c28eb3b and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- docs/request-for-change/rfc-leaked-runtime-reclaim.md:171 -- reclaimed with their root is false: reclaim_untracked → _kill_pid_tree skips non-harness MCP descendants, leaving them orphaned after killing the root -> Fix: mark #13324 unresolved.
[GPT-REVIEWED] c4dfa6b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c4dfa6bbba930e424232129686c3b1d77c28eb3b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of c4dfa6bbba930e424232129686c3b1d77c28eb3b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Description says claims verified at 2db531ec89, diff audits at 6bf7a81920 — confirm the description reflects this revision's audit.

[DESIGN-REVIEWED] c4dfa6b

Description read

[DESCRIPTION-READ] 9dc94864e0f7f33b638789fe26d768d0c9f8d28c74f3fdb6480cc12790fdb53f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed c4dfa6bbba930e424232129686c3b1d77c28eb3b — this comment is updated in place on each push.

Review details

Three statements the new RFC adds disagree with the code it cites at 6bf7a81920.

FINDING — docs/request-for-change/rfc-leaked-runtime-reclaim.md:84 — Decision 3 derives the spawn record's shape from "the lease and tenancy tables already key by pid and start id", but runtime_ownership._Entry is keyed by lease handle (leases_on_pid scans every entry comparing entry.pid), _Claim's docstring says the opposite outright ("It rides on the claim rather than in a table keyed by pid"), and neither dataclass carries any start-time/start-id field → Fix: state what is there — both tables are keyed by handle/claim and merely carry a pid, with no start identity persisted — and justify the per-root row and its start-id field on its own terms rather than as an existing table shape.

FINDING — docs/request-for-change/rfc-leaked-runtime-reclaim.md:100 — in a table the same paragraph calls main's behaviour ("On main _why_not_reclaimable applies all six"), the Group-leader row claims the pgid must name a live process "that is still that group's leader", but _group_leader_alive (runtime_reconcile.py:292-298) returns pgid != pid and pid_liveness(pgid) != PID_DEAD with no getpgid(pgid) == pgid re-test — which the RFC itself states 7 lines later ("reads liveness only and gains the same getpgid(pgid) == pgid test in step B") → Fix: write the row as main behaves and mark the still-leader test as step-B state.

FINDING — docs/request-for-change/rfc-leaked-runtime-reclaim.md:78,98 — both places the RFC positions authorize_runtime_kill contradict main's deliberate "gate LAST" ordering: Decision 2's observe switch says "evaluate every gate" where runtime_reconcile.py:1191-1200 holds at budget 0 with the ownership gate explicitly not asked, and Decision 4's row says authorize_runtime_kill is "checked before the spawn record" where the pre-record veto is _leases_on/_claims_on (:899) and self._authorize runs after all of _why_not_reclaimable (:870-874, matching runtime-ownership.md:317); since the gate's allow path unconditionally calls note_runtime_kill, an implementation built from either sentence writes kill-authorization records for pids no kill follows → Fix: say "evaluate every refusal, audit would_kill, ask no ownership gate" in Decision 2, and split the Decision 4 row into the lease/claim veto before the spawn record and authorize_runtime_kill last.

[OPUS-REVIEWED] c4dfa6b

Verdict parsed from the review's SHA-scoped output markers for commit c4dfa6bbba930e424232129686c3b1d77c28eb3b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c4dfa6bbba930e424232129686c3b1d77c28eb3b: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-leaked-runtime-reclaim branch from 8c0e5b2 to ffb5d1e Compare October 1, 2026 08:02
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] CI reds on 8c0e5b2a47 were main-side, not this docs-only diff:

Red Cause Proof
Backend Lint (flake8 F821 _row_stem_tokens_for_scan, vector_memory_runtime/lessons.py:1206) main flake8 on lessons.py at main 6b033ab03a reports the same F821
Backend Tests 3.12 + Windows shards 4/5/6 (test_fold_catalogue, test_lesson_turn_injection, test_vector_memory_composition_contract) main detached main 6b033ab03a: 28 failed; main d8fa6a527e (after #15744): 167 passed
Coverage Gate, PR Readiness cascade —

Rebased onto d8fa6a527e; new head ffb5d1e9fa. No unrelated code touched.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at ffb5d1e9fa78eb3891a3bc63d70b4fb00a0046dc against origin/main (35171e238b; none of runtime_reconcile.py, session_scope_reap.py, session_pid.py, acp/runtime.py or apps/backend_runtime/pidfile.py has changed since 2db531ec89). Docs-only, and scripts/docs_lint.py passes.

Most of what the RFC says about current code is right. The reconciler's five conditions, DEFAULT_MAX_KILLS = 5, DEFAULT_MIN_AGE_SECS = 300, the reconcile_max_kills ceiling and its observe-at-0 behaviour, instance_slice_pids returning the empty set when the slice cannot be resolved, the report-only _is_untracked_managed_agent_orphan, _process_instance and the _record_app_pid precedent all match source. Protected pids reach the reconciler through _collect_active_pids → _protected_pids(), so "tracked or protected" really is on main today.

Four things need fixing before this is the decision record the First Principles lane reads.

Blocking

1. "The only arm that kills an unowned runtime" is false on main. The README row says runtime_reconcile.py "is the only arm that signals an unowned runtime", and Decision 1 says it is "the only code that kills an unowned runtime". That leaves out session_scope_reap.reap_abandoned_agent_scopes. It is wired into every cleanup tick (session.py:1353) and stops whole scopes in the same per-instance slice. Its first condition is that no member is tracked or active (session_scope_reap.py:26-40), so the scopes it ends are ones no record names. The Non-goals list "scope reaping" among paths that "end what their records name", which is the opposite of what it does. This matters because the RFC's goal is "one kill path for unowned runtimes, with one set of refusals", and main already has two. Fix: add the scope reaper to the current-state table. Then either bring it under Decision 1, or state that it keeps its own four-part check and explain why the new leader refusals (Decision 4) do not apply to it.

2. Step A gives the non-cgroup source a kill before it has install-scoped proof. The report arm's candidates come from _our_orphan_pids(), which covers every init-reparented process of this uid (session_pid.py:3796-3826). The marker it checks is the constant KIROCREW_SPAWNED=1 (constants.py:22), so these candidates are not limited to this install. The report arm's own docstring names a second install's runtime as a known false positive. Even so, step A's exit criterion says that on a non-cgroup host "a confirmed Reclaim ends it", and that is before step B adds the per-data-home record. This contradicts the Non-goal ("A process another data home spawned is never ours") and Decision 5 ("proven by Decision 3"). Decision 6's reason ("the user … knows no other install shares the data home") does not cover a second install with a different data home under the same uid. The security table misses that case too. Fix: either limit step A's confirmed Reclaim to slice candidates and leave non-cgroup candidates report-only until step B, or add a "same uid, different data home" row with a mitigation that actually holds before step B.

Non-blocking

3. Windows has no report-arm reading at all. _our_orphan_pids() returns [] on Windows (session_pid.py:3803), and _env_has_kirocrew_marker always returns false there (:3989). The Decision 5 Windows row ("report arm reading") and the #11991 Resolves row ("gains the dashboard reading") therefore promise a count that would always be 0 while the leak stays. Mark Windows as having no reading until it gets a native candidate source, and change #11991 to unresolved. macOS is fine: KERN_PROCARGS2 gives it a marker check, and the report arm's "Linux-only" docstring is out of date.

4. A recycled sid or pgid would refuse reclaim forever. Decision 4 refuses when "the candidate's sid/pgid names a live process other than the candidate". The launcher of a measured leak is dead, so as soon as its pid number is reused by any long-lived process, that leak can never be reclaimed. That is exactly the class of leak this RFC is meant to end. Main already handles this in _work_orphan_session_leader_alive (session_pid.py:4493-4519, leader_sid == sid). State the refusals as "names a live process that is still that session's / group's leader". This is the Fable suggestion, and it holds.

5. Frontmatter. implementation-prs: [14510] names the reconciler PR, which merged on 2026-09-28, before this RFC. It is not an implementation of this RFC. List the step A PR once it exists, or leave the field empty and keep #14510 under Related.

Bot dispositions

  • Opus, "a second arm already kills": verified, and it is finding 1.
  • Opus, "Windows reading always zero": verified, and it is finding 3.
  • Fable, recycled-leader suggestion: verified, and it is finding 4.
  • GPT, no findings: noted. Findings 1 and 2 hold regardless.
  • CI reds: the author says they come from main, and the current head is a rebase. This diff touches no code, so the reds do not gate it.

@iamwhatever
iamwhatever force-pushed the docs/rfc-leaked-runtime-reclaim branch from ffb5d1e to 1f2fc03 Compare October 1, 2026 18:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] @buluoray thanks. All five addressed in 1f2fc03315 (rebased on 4224aa2336):

  1. Scope reaper added to the current-state table and the README row; Non-goals no longer files it under record-driven paths. Option chosen: it keeps its own four-part check (matches the code), and Decision 1 says why Decision 4's leader refusals do not apply (its unit is a kernel-placed scope; a live runtime's detached child stays in that scope, so condition 1 refuses).
  2. Checked merged feat(runtime): read leaked untracked runtimes, reclaim on confirm #15755: reclaim_untracked → _why_not_reclaimable already refuses unless KIROCREW_SPAWN_HOME equals this data home, plus live-leader and shared-instance refusals. Step A now states exactly that; "same uid, different data home" row added to the security table.
  3. Windows: "none / no reading" in Decision 5; Windows: Node-hosted adapter orphans are unreclaimable (image-name-only identity) #11991 moved to unresolved.
  4. Refusals restated as "a live process that is still that session's / group's leader"; the RFC notes the reclaim's group check reads liveness only and gains getpgid(pgid) == pgid in step B.
  5. implementation-prs: [15755]; fix(session): ask who owns a pid before every sweep signals it, and reconcile both ways #14510 moved to Related.

@iamwhatever
iamwhatever requested a review from buluoray October 1, 2026 18:32
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Read at 1f2fc03315 against main 4d24fde5ad. The direction matches the runtime-ownership work (lease + tenancy, one authorize_runtime_kill gate, reconciler budget 5 with reconcile_max_kills=0 as observe-only). Notes on Decision 3 / Open question 1, from the side that owns the lease and tenancy tables:

1. The record's lifecycle decides where it lives, not the file format. A leak is by definition a runtime that has already dropped out of the tracked records. Neither existing record keeps a row that long:

  • kiro_pids.txt rows are removed on tracked teardown (_untrack_pid), and the root row is a bare pid (only descendant rows carry pid:parent:start).
  • the signed session_pid_<pid> mapping is pruned within one cleanup tick of its session being released (_prune_stale_session_pid_files).

So "new columns on the existing PID file" only works if that row's removal rule changes. The rule Step B needs: write the row at spawn, before session/new; remove it only once the root is confirmed gone (pid absent, or its start id no longer matches). Never remove it on session release or lease drop. Suggest writing that rule into Decision 3 so Step B does not inherit the PID file's teardown semantics by accident.

2. Key it by runtime root, not by session. With chat runtime sharing (#15325), one root process serves many sessions, and the lease/tenancy tables already key by pid + start id. A per-session row would give N rows for one root, and the reconciler would have to agree with all of them. Root pid + start id + KIROCREW_SPAWN_INSTANCE, one row per spawn, matches the shape the kill gate already reads.

3. Restart path. Step B's exit criterion covers "after a restart the reading still names the previous process's leaks". Worth adding the converse: a root that a new gateway adopts or re-leases after restart must be refused by the lease gate even though its spawn row is from the previous gateway. Decision 4 already gives the lease the veto; one sentence stating that ordering (lease veto first, spawn record second) would close it.

No objection to Decisions 2, 4, 5, 6.

@iamwhatever
iamwhatever force-pushed the docs/rfc-leaked-runtime-reclaim branch from 1f2fc03 to db4c9d1 Compare October 1, 2026 22:56
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Thanks, @buluoray — this was a careful read and every point held up against main. New head is db4c9d1837cf9b4ff9bd3a5a7cf5e71e84981e2c. Point by point:

1 (blocking) — "the only arm that kills an unowned runtime" is false. Confirmed: reap_abandoned_agent_scopes is wired into every cleanup tick (session.py, via reap_agent_scopes) and its first condition is "no member tracked or active", so it ends scopes no record names — a second kill arm. I added it to the current-state table (now three arms) and, rather than let it keep a parallel check, folded it under Decision 1: a scope it would stop must pass Decision 4's refusals before any kill, so the slice source, the non-cgroup source and the scope reaper answer to one list of refusals instead of three. Its scope-level reads survive as a candidate source and pre-filter, not a second verdict. One rule set is the RFC's whole point, so main's two half-overlapping checks become one. The Non-goal that listed scope reaping among paths that "end what their records name" is gone.

2 (blocking) — step A kills before install-scoped proof. Confirmed: _our_orphan_pids() enumerates every init-reparented process of this uid and the marker is the install-agnostic KIROCREW_SPAWNED=1. Step A is now scoped to install-stamped candidates: the merged #15755's reclaim_untracked refuses through _why_not_reclaimable when KIROCREW_SPAWN_HOME is absent or differs from this data home. The rollout table, Decision 6 and Decision 7 now say so, and I added a "Same uid, different data home" row to the security table naming that exact case and its mitigation.

3 (non-blocking) — Windows reading always 0. Fixed. The Decision 5 Windows row is now "none / no reading" (_our_orphan_pids returns [] and there's no environ oracle there), and the macOS row notes its KERN_PROCARGS2 marker read. #11991 is now Unresolved in the Resolves table.

4 (non-blocking) — recycled sid/pgid refuses forever. Fixed, as you (and Fable) prescribed. Decision 4 now reads the two leader refusals as "names a live process … that is still that session's / group's leader", matching _work_orphan_session_leader_alive (leader_sid == sid).

5 (non-blocking) — frontmatter. Fixed. implementation-prs: [15755] (step A, the actual implementation); #14510 moved to Related as the pre-existing reconciler kill arm.

scripts/docs_lint.py passes. Re-requesting your review.

chenmingwei23
chenmingwei23 previously approved these changes Oct 2, 2026
@chenmingwei23
chenmingwei23 enabled auto-merge (squash) October 2, 2026 15:51

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at dc12c803ce6e73fcbb69b1b55bc5804926ebf7df against origin/main 57d71bb54e: requesting changes

The five items from my last round are answered. But this head adds a fold of the scope reaper, and it carries several statements about main that are not true. This document is the record the First Principles lane will read when step C gives the gateway new default kill authority, so those statements have to be right before it lands. Auto-merge is armed and there is another approval, so I'm holding it here.

I checked each item below against source myself. Most match Opus's findings on this SHA, which were filed as non-blocking.

Blocking

1. The Decision 4 table does not cover a detached setsid child, but the text says it does. After setsid, sid == pid and pgid == pid. _session_leader_alive returns sid != pid and … (runtime_reconcile.py:289) and _group_leader_alive returns pgid != pid and … (:298), so both answer false for that child. The refusals that actually protect it are in _why_not_reclaimable: "no readable spawn instance" (:918) and "another live process shares its spawn instance" (:926). Neither is in the table. "These four refusals cover each live shape seen so far" and Decision 1's claim that the refusals "subsume" the scope reaper's check both depend on this. Step C builds default reclaim from this table, so as written it would kill a live runtime's detached child. Fix: add the two spawn-instance rows to the table and drop the "four refusals cover each live shape" sentence and the subsumption claim (or restate them with the instance rows).

2. Not every live ACP spawn path is stamped. "Each ACP runtime spawn … stamps it beside KIROCREW_SPAWN_HOME" is true only of acp/runtime.py:2200-2204. AcpClient._spawn (acp/client.py:7214, env at :8075) sets only KIROCREW_SPAWNED. That path is live: providers/acp.py:473 and knowledge/llm_pool.py:354. _why_not_reclaimable refuses every unstamped candidate, so those runtimes can never be reclaimed and step C's exit criterion ("a non-cgroup host's leaked runtimes are reclaimed … with no manual pass") cannot be met. Fix: limit the sentence to AcpRuntime, add stamping AcpClient._spawn to step B, and scope step C's criterion to stamped spawn paths.

3. The owner gate is described the wrong way round. Step A says "owner-only GET /api/system/leaked-runtimes". api_leaked_runtimes (dashboard/handlers_system.py:816) has no _require_owner and returns per-pid RSS to any authenticated session. Only the reclaim POST (:837) is owner-gated. A security pass that trusts this row would believe the pid/RSS inventory is owner-only. Fix: "owner-only, confirm-required POST …/reclaim; GET …/leaked-runtimes authenticated, not owner-gated". Or name adding that gate as follow-up work.

4. The scope-reaper fold is decided but no step ships it. Last round I offered two options: keep its own check and explain why, or bring it under Decision 1. The 1f2fc03315 reply chose the first, and db4c9d1837 switched to the second. Either is fine. But Decision 1 now changes a shipped default kill arm, and steps A–C never carry that change. That also conflicts with the PR's frozen Not-a-goal ("Changing … scope reaping"), which Design flags. Fix: either give the fold its own rollout step with an exit criterion and a named pinning test (a scope that predates boot; a live group leader as candidate), plus a PR comment about the goal-line divergence. Or go back to "keeps its own four-part check" and say why Decision 4 does not apply.

Also fix (factual, small)

  • The Decision 6 dashboard already shipped. LeakedRuntimesCard.tsx merged in #15764 (00b57642a) and is rendered at ServicesTab.tsx:311, with count, RSS and a confirmed reclaim button. Problem 4 and the step A note ("The dashboard button of Decision 6 is not built yet") are false at the PR's base 5a70ce7d3a. Add 15764 to implementation-prs, and re-audit: audited-at: 4224aa2336 predates that merge.
  • The reason for rejecting the signed mapping is wrong. Decision 3 says _prune_stale_session_pid_files "drops it within one cleanup tick of the session's release". Its docstring and body (session_pid.py:2880) prune only a mapping whose pid is dead or recycled, and I found no unlink at release time. The mapping is keyed on the runtime root pid, which is the lifecycle Decision 3 asks for. Correct the parenthetical, and re-weigh that option in Open question 1.

Bot dispositions

  • Opus (5 findings, "nothing blocks"): all five verified. Three are blocking items 1–3, two are under "Also fix". I disagree with the non-blocking framing: in a decision record, a wrong statement about what a kill gate refuses is the defect.
  • Design CONCERNS: verified, item 4.
  • GPT, no findings: noted.

CI: 82 success, 12 skipped, 0 failures. Docs only, so CI does not settle any of the above.

iamwhatever and others added 2 commits October 2, 2026 22:05
Records the decision before the code: one reclaim arm (the runtime
reconciler), default-on with its per-pass budget and the
reconcile_max_kills=0 observe switch, ownership proved by a persisted
install-scoped spawn record that can only veto, four refusals that always
win, Linux first, and a user-confirmed Reclaim on the dashboard.

Step A (report + confirmed reclaim) adds no default authority. Steps B and
C extend it, and the First Principles lane reads that decision from the
base branch, so this document lands on its own.

Refs #11899
buluoray: name reap_abandoned_agent_scopes as a separate arm that keeps
its own four-part check; step A (#15755, #15764) refuses a candidate whose
KIROCREW_SPAWN_HOME is not this data home, with a same-uid other-home
security row; the reading GET is authenticated, only the reclaim POST is
owner-gated; Decision 4 lists the two spawn-instance refusals that stop a
detached setsid child; only AcpRuntime stamps the spawn instance and home,
so step B stamps AcpClient._spawn and step C covers stamped paths; Windows
has no reading and #11991 stays unresolved; the leader refusals read
"still that session's / group's leader".

chenmingwei23: the spawn record is one row per runtime root (root pid,
start id, spawn instance), written at spawn before session/new and
removed only once the root is gone, never on session release or lease
drop; the lease veto comes before the spawn record, and step B's exit
criteria cover a root a new gateway re-leases after a restart.

Co-authored-by: cixuuz <26782336+cixuuz@users.noreply.github.com>
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] @buluoray thanks, every point checked against origin/main 6bf7a81920. Fixed in c4dfa6bbba (rebased on that main, still 2 commits, docs only):

Blocking

  1. Verified: _session_leader_alive returns sid != pid and … (runtime_reconcile.py:289), _group_leader_alive returns pgid != pid and … (:298), so both are false for a setsid child; the refusals that stop it are "no readable spawn instance" (:918) and "another live process shares its spawn instance" (:926). Decision 4 now lists all six rows, says the leader rows do not cover a detached child and the instance rows do, and the "four refusals cover each live shape" sentence is gone. The security row names the instance refusals.
  2. Verified: only AcpRuntime stamps (acp/runtime.py:2201 instance, :2205 home); AcpClient._spawn (acp/client.py:7214) sets only KIROCREW_SPAWNED (:8075), live through providers/acp.py:489 and knowledge/llm_pool.py:354. Current state now names both paths; step B adds stamping AcpClient._spawn; step C's criterion is scoped to stamped spawn paths; step A's note says unstamped runtimes stay report-only.
  3. Verified: api_leaked_runtimes (dashboard/handlers_system.py:816) has no _require_owner; only api_leaked_runtimes_reclaim is gated (:837) and needs confirm (:844). Step A now reads "owner-only, confirm-required POST; GET authenticated but not owner-gated".
  4. Took your first option and dropped the fold. Decision 1 again says the scope reaper keeps its own four-part check and this RFC does not change it. Why Decision 4 does not apply: its unit is a kernel-placed cgroup scope; setsid changes session and group, not cgroup, so a live runtime's detached child stays in that runtime's scope and condition 1 (no member tracked or active, session_scope_reap.py:26-40) refuses the whole scope. The Not-a-goal "Changing the scope reaper" is back, so the frozen goal line and the RFC agree. Summary, Decision 4 text and the README row no longer mention a fold.

Also fix

  • Dashboard: LeakedRuntimesCard.tsx from feat(ui): show leaked agent runtimes on Services, reclaim on confirm #15764 (00b57642a0), rendered at ServicesTab.tsx:311. Problem 4 and the "not built yet" note are removed; Decision 6 and step A cite it; implementation-prs: [15755, 15764]; audited-at: 6bf7a81920.
  • Signed mapping: _prune_stale_session_pid_files (session_pid.py:2880) removes a mapping only when its pid is dead or no longer that session; I found no unlink at release. Decision 3 now says so, and Open question 1 lists the mapping as a third home: root-pid key and the right removal rule, but no instance or home fields today.

@chenmingwei23 your three points are unchanged in Decision 3, Decision 4 and step B; the only scope change this round is removing the fold.

@iamwhatever
iamwhatever requested a review from buluoray October 2, 2026 22:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 2, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at c4dfa6bbba against origin/main: approving. All four blocking items and both small factual fixes from my last round are resolved, and I checked each new statement against main's source myself.

Previous blocking items

  1. Detached setsid child: fixed. Decision 4's table now carries the two spawn-instance rows (no readable spawn instance, spawn instance still live). Both exist on main in _why_not_reclaimable (runtime_reconcile.py ~918 and ~926). The text now says plainly that the leader rows do not protect a setsid child and that the instance rows do. The "four refusals cover each live shape" sentence and the subsumption claim are gone.
  2. Unstamped spawn path: fixed. Current state names AcpClient._spawn as a second live path that stamps only KIROCREW_SPAWNED, which matches main (acp/client.py ~8075). Step B now stamps it, and step C's exit criterion is scoped to stamped spawn paths, so it can be met.
  3. Owner gate: fixed. Step A now reads "owner-only, confirm-required POST …/reclaim; GET …/leaked-runtimes authenticated but not owner-gated", which matches handlers_system.py: _require_owner appears only on the POST.
  4. Scope-reaper fold: resolved by going back. The reaper stays a separate arm on its own four-part check, with a reason I agree with: its unit is a whole cgroup scope the kernel placed at spawn, and setsid changes session and group, not cgroup. That also brings Decision 1 back in line with the PR's frozen Not-a-goal ("Changing … scope reaping").

Previous small fixes

  • #15764 credited: it is in implementation-prs and in step A, and audited-at moved to 6bf7a81920, which is on main.
  • Signed-mapping reason corrected: the parenthetical now says _prune_stale_session_pid_files removes a mapping only when its pid is dead or recycled, and Open question 1 now weighs that mapping as a third option.

Follow-up (non-blocking)

  • GET /api/system/leaked-runtimes returning per-pid RSS to any authenticated session is now described accurately. Whether it should be owner-gated is worth a decision in step B; it's a one-line _require_owner.

Docs only. CI was still running when I reviewed, so it does not settle any of the above.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@chenmingwei23
chenmingwei23 added this pull request to the merge queue Oct 2, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 3, 2026 01:30
@bolichen97
bolichen97 merged commit 807e10e into main Oct 3, 2026
97 checks passed
@bolichen97
bolichen97 deleted the docs/rfc-leaked-runtime-reclaim branch October 3, 2026 01:31
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 3, 2026
@bolichen97 bolichen97 mentioned this pull request Oct 6, 2026
5 tasks done
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder
assignment, suggestions) run on a shared codex-acp adapter.  Each
session/new causes codex to start its per-session MCP fleet (~44
processes) from CODEX_HOME/config.toml.  session/close evicts the ACP
session but does not terminate those processes; they accumulate as
children of the live codex app-server until the background runtime hits
its 6-hour age ceiling and killpg reclaims the whole group.

Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB
after 4 hours.

Fix: when spawning the kirocrew-lite background runtime, set
private_state_env = 'CODEX_HOME' to redirect the whole config directory
to a per-process private scratch.  The new seed function
_seed_background_codex_home prepares the private dir:

- Writes a stripped config.toml via _codex_config_no_mcp (an allowlist
  filter identical to _provider_config_only in test_codex_session_mcp.py)
  that keeps model/provider settings but removes every process-spawning
  entry (mcp_servers in any TOML spelling, hooks, notify).  The
  mcp_servers key in codex config uses named sub-tables
  ([mcp_servers.<name>]), not array-of-tables ([[mcp_servers]]), so a
  denylist regex cannot reliably match all forms; an allowlist is the
  right instrument.

- Symlinks auth.json from the real CODEX_HOME (not a copy) so an OAuth
  token refresh updates only the real file with no divergence.  Falls
  back to shutil.copy2 on platforms without symlink support.

- Seeds the SQLite state database (same as seed_private_state) so
  rollout history is available immediately.

Foreground runtimes keep the full operator CODEX_HOME unchanged.  If the
operator has already set CODEX_HOME, the redirect is skipped and the
existing CODEX_SQLITE_HOME isolation path runs instead.

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping (step B second req)

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new causes
codex to start its per-session MCP fleet (~44 processes) from
CODEX_HOME/config.toml.  session/close evicts the ACP session but does
not terminate those processes; they accumulate as children of the live
codex app-server until the background runtime hits its 6-hour age
ceiling and killpg reclaims the whole group.

Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB
after 4 hours.

Fix: when spawning the kirocrew-lite background runtime (lazily imported
from session.BACKGROUND_AGENT -- same circular-import avoidance as
llm_helpers.py), set private_state_env='CODEX_HOME' to redirect the
whole config directory to a per-process private scratch.  The seed
function _seed_background_codex_home prepares the private dir:

- Writes a stripped config.toml via _codex_config_no_mcp.  This is an
  allowlist filter that keeps only the scalar keys model, model_provider,
  forced_login_method, model_reasoning_effort, chatgpt_base_url,
  check_for_update_on_startup and all [model_providers.*] table sections.
  Everything else -- mcp_servers in any TOML spelling ([mcp_servers.name],
  inline tables, dotted keys), hooks, notify -- is excluded by not being
  on the list.  Multiline string values for allowed keys are silently
  dropped (safe default vs corrupt TOML).  Logic is identical to what
  test_codex_session_mcp.py previously called _provider_config_only;
  that helper is now removed and replaced with a production import.

- Symlinks auth.json from the real CODEX_HOME (not a copy) so no
  credential copy ends up under the per-process scratch dir.  On
  platforms without symlink support, fails closed (leaves auth.json
  absent in the private home).

- Seeds the SQLite state database so rollout history is available.

Foreground runtimes keep the full operator CODEX_HOME unchanged.  If the
operator has already set CODEX_HOME, the redirect is skipped and the
existing CODEX_SQLITE_HOME isolation path runs instead.

Deduplication: removes _provider_config_only from test_codex_session_mcp.py
(was a copy of the same allowlist algorithm) and replaces its single usage
with _codex_config_no_mcp from production code.

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new causes
codex to start its per-session MCP fleet (~44 processes) from
CODEX_HOME/config.toml.  session/close evicts the ACP session but does
not terminate those processes; they accumulate as children of the live
codex app-server until the background runtime hits its 6-hour age
ceiling and killpg reclaims the whole group.

Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB
after 4 hours.

Fix: when spawning the kirocrew-lite background runtime (BACKGROUND_AGENT
lazy-imported from session to avoid the circular import at module scope),
set private_state_env='CODEX_HOME' to redirect the whole config directory
to a per-process private scratch.  The seed function
_seed_background_codex_home prepares the private dir:

- Writes a stripped config.toml via _codex_config_no_mcp.  This is an
  allowlist filter that keeps only the scalar keys model, model_provider,
  forced_login_method, model_reasoning_effort, chatgpt_base_url,
  check_for_update_on_startup and all [model_providers.*] table sections.
  Those tables carry provider-specific auth.  Everything else --
  mcp_servers in any TOML spelling ([mcp_servers.name], inline tables,
  dotted keys), hooks, notify, any future process-spawning key -- is
  excluded by not being on the list.  Multiline string values for allowed
  keys are silently dropped (safe vs corrupt TOML).
  auth.json is deliberately NOT included: provider auth lives in
  [model_providers.*] (kept by the filter) and API keys reach the child
  through the ambient environment (OPENAI_API_KEY, ANTHROPIC_API_KEY).
  This matches the live prompt test _codex_config_no_mcp approach.

- Seeds the SQLite state database (same as seed_private_state) so
  rollout history is available immediately.

Foreground runtimes keep the full operator CODEX_HOME unchanged.  If the
operator has already set CODEX_HOME, the redirect is skipped and the
existing CODEX_SQLITE_HOME isolation path runs instead.

Deduplication: removes _provider_config_only from test_codex_session_mcp.py
(was a copy of the same allowlist algorithm) and replaces its single usage
with the production _codex_config_no_mcp.

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new causes
codex to start its per-session MCP fleet (~44 processes) from
CODEX_HOME/config.toml.  session/close evicts the ACP session but does
not terminate those processes; they accumulate as children of the live
codex app-server until the background runtime hits its 6-hour age
ceiling and killpg reclaims the whole group.

Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB
after 4 hours.

Fix: when spawning the kirocrew-lite background runtime (BACKGROUND_AGENT
lazy-imported from session -- same pattern as llm_helpers.py), set
private_state_env='CODEX_HOME' to redirect the config directory to a
per-process private scratch.  The seed function
_seed_background_codex_home prepares the private dir:

- Copies auth.json from the real CODEX_HOME (if present) so both
  API-key and ChatGPT-OAuth auth work in background sessions.  A copy
  (not a symlink) because the background runtime is short-lived (max 6h
  then killpg recycles it), bounding copy staleness.

- Writes a stripped config.toml via _codex_config_no_mcp.  This is an
  allowlist filter that keeps only safe scalar keys (model,
  model_provider, forced_login_method, model_reasoning_effort,
  chatgpt_base_url, check_for_update_on_startup) and all
  [model_providers.*] table sections.  Everything else -- mcp_servers in
  any TOML spelling ([mcp_servers.name], inline tables, dotted keys),
  hooks, notify, any future process-spawning key -- is excluded by not
  being on the list.  Multiline string values for allowed keys are
  silently dropped (safe vs corrupt TOML).

- Seeds the SQLite state database (same as seed_private_state) so
  rollout history is available immediately.

Foreground runtimes keep the full operator CODEX_HOME unchanged.  If the
operator has already set CODEX_HOME, the redirect is skipped and the
existing CODEX_SQLITE_HOME isolation path runs instead.

Deduplication: removes _provider_config_only from test_codex_session_mcp.py
(was a copy of the same allowlist algorithm) and replaces its usage with
the production _codex_config_no_mcp.

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new causes
codex to start its per-session MCP fleet (~44 processes) from
CODEX_HOME/config.toml.  session/close evicts the ACP session but does
not terminate those processes; they accumulate as children of the live
codex app-server until the background runtime hits its 6-hour age
ceiling and killpg reclaims the whole group.

Reporter hugoncosta (codex-acp 1.11.0) observed ~855 processes / 21 GiB
after 4 hours.

Fix: when spawning the kirocrew-lite background runtime (BACKGROUND_AGENT
lazy-imported from session to avoid the circular import at module scope --
same pattern as llm_helpers.py), set private_state_env='CODEX_HOME' to
redirect the config directory to a per-process private scratch.  The seed
function _seed_background_codex_home prepares the private dir:

- Copies auth.json from the real CODEX_HOME (if present) so both API-key
  and ChatGPT-OAuth (codex login) auth work.  A copy, not a symlink:
  the background runtime is short-lived (max 6h then killpg recycles it),
  bounding copy staleness.

- Writes a stripped config.toml via _codex_config_no_mcp -- an allowlist
  filter that keeps only safe scalar keys (model, model_provider,
  forced_login_method, model_reasoning_effort, chatgpt_base_url,
  check_for_update_on_startup) and all [model_providers.*] table sections.
  Everything else -- mcp_servers in any TOML spelling ([mcp_servers.name],
  inline tables, dotted keys), hooks, notify -- is excluded by not being on
  the list.  Multiline string values for allowed keys are silently dropped.

- Seeds the SQLite state database so rollout history is available.

Foreground runtimes keep the full operator CODEX_HOME unchanged.  If the
operator has already set CODEX_HOME, the redirect is skipped.

Deduplication: removes _provider_config_only from test_codex_session_mcp.py
(was a copy of the same allowlist algorithm) and replaces its usage with
the production _codex_config_no_mcp.

Also updates providers/mirrors/codex.py to note that Crew now writes to a
private CODEX_HOME for background sessions (the previously-recorded
'create-or-decline' gap this fills for the hooks channel).

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
…ONFIG

Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new causes
codex to start its per-session MCP fleet (~44 processes) from the
mcp_servers entries in CODEX_HOME/config.toml.  session/close evicts the
ACP session but does not terminate those MCP children; they accumulate
until the background runtime hits its 6-hour age ceiling and killpg
reclaims the group.  Reporter hugoncosta observed ~855 processes / 21 GiB
after 4 hours.

Fix: at background-runtime spawn, read CODEX_HOME/config.toml (read-only),
enumerate the mcp_servers keys, and inject CODEX_CONFIG with every named
server set to enabled=false.  McpServerConfig.enabled is documented in
codex-rs/config/src/mcp_types.rs line 234:
  "When false, Codex skips initializing this MCP server."

Mechanism: CODEX_CONFIG is a JSON object merged into the Codex session
config (codex-acp README).  Setting mcp_servers.<name>.enabled=false deep-
merges correctly (codex-rs/config/src/merge.rs merge_toml_values_at_path --
merging {enabled:false} into an existing server table adds the key).  Server
names containing dots are safe: they travel as JSON object keys, not as TOML
dotted paths.

Shared CODEX_HOME, no auth.json copy or symlink: auth reaches the background
runtime through the ambient environment unchanged.  CODEX_SQLITE_HOME
isolation is unchanged.

If the operator has already set CODEX_CONFIG with an explicit mcp_servers
key, that value is left as-is (operator override takes precedence).

_read_codex_mcp_server_names tries tomllib (3.11+) / tomli for structural
accuracy; falls back to a regex scan on older interpreters without tomli.

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new
causes codex to start its per-session MCP fleet (~44 processes) from
the mcp_servers entries in CODEX_HOME/config.toml.  session/close evicts
the ACP session but does not terminate those MCP children; they accumulate
until the background runtime hits its 6-hour age ceiling and killpg
reclaims the group.  Reporter hugoncosta observed ~855 processes / 21 GiB
after 4 hours.

Fix: at background-runtime spawn, read CODEX_HOME/config.toml (read-only,
off-loop) via tomllib, enumerate the mcp_servers keys, and build a
CODEX_CONFIG env var with every named server set to enabled=false.
McpServerConfig.enabled is documented in
codex-rs/config/src/mcp_types.rs line 234:
  "When false, Codex skips initializing this MCP server."

Mechanism: CODEX_CONFIG is a JSON object merged into the Codex session
config (codex-acp README).  Setting mcp_servers.<name>.enabled=false
deep-merges correctly (codex-rs/config/src/merge.rs
merge_toml_values_at_path: merging {enabled:false} into an existing
server table adds the key).  Server names with dots are safe: they travel
as JSON object keys, not TOML dotted paths.

CODEX_CONFIG is placed in the new SpawnPlan.extra_env field.  extra_env
is applied via setdefault semantics in launch.py so that an operator-set
CODEX_CONFIG in agent.env or os.environ takes precedence.  It is
delivered unconditionally -- not gated on CODEX_SQLITE_HOME -- so it
works even when the operator has already set that variable.

Shared CODEX_HOME unchanged: no credential copy, no symlink.

CODEX_CONFIG skip condition: any existing CODEX_CONFIG in the spawn
environment (os.environ or agent.env) is left as-is.

Not in this PR:
- RFC #15748 step B (persisted spawn record for AcpRuntime)
- RFC #15748 step C (default reclaim on non-cgroup Linux)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
CrysisDeu pushed a commit that referenced this pull request Oct 14, 2026
Background sessions (kirocrew-lite: title gen, navigation, folder,
suggestions) run on a shared codex-acp adapter.  Each session/new
causes codex to start its per-session MCP fleet (~44 processes) from
the mcp_servers entries in CODEX_HOME/config.toml.  session/close evicts
the ACP session but does not terminate those MCP children; they accumulate
until the background runtime hits its 6-hour age ceiling and killpg
reclaims the group.  Reporter hugoncosta observed ~855 processes / 21 GiB
after 4 hours.

Fix: at background-runtime spawn, read CODEX_HOME/config.toml via
tomllib (stdlib 3.11+; pyproject.toml requires >= 3.12), enumerate the
mcp_servers keys, and inject CODEX_CONFIG into the child environment
with every named server set to enabled=false.

McpServerConfig.enabled is documented in codex-rs/config/src/mcp_types.rs
line 234: "When false, Codex skips initializing this MCP server."

CODEX_CONFIG is a JSON object merged into the Codex session config
per the codex-acp README. The deep-merge applies enabled:false to each
existing server table (codex-rs/config/src/merge.rs).

Implementation follows the opencode.py/pi.py pattern:
- resolve_spawn (off-loop worker thread) computes the CODEX_CONFIG value
  and stores it in self._bg_codex_config.
- apply_spawn_env consumes self._bg_codex_config after the full env
  (including agent.env) is assembled; it sets CODEX_CONFIG only when
  the env does not already carry one, so operator values take precedence.

No changes to SpawnPlan, launch.py precedence, or CODEX_HOME.
CODEX_SQLITE_HOME isolation is unchanged.

CODEX_CONFIG skip condition: any existing CODEX_CONFIG in the fully
assembled spawn env (os.environ or agent.env) is preserved.

Not in this PR:
- RFC #15748 step B / step C (persisted spawn record + auto-reclaim)
- AcpClient._spawn KIROCREW_SPAWN_INSTANCE/HOME stamping

Fixes #13324
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants