Skip to content

feat(ui): show leaked agent runtimes on Services, reclaim on confirm - #15764

Merged
iamwhatever merged 1 commit into
mainfrom
feat/leaked-runtime-reclaim-ui
Oct 2, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/leaked-runtime-reclaim-ui

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: an operator sees how many agent runtimes have leaked and how much memory they hold, and can end them with an explicit two-step confirm.

Leaked runtimes (#11899, #11789) hold memory until the host OOMs, and the dashboard shows nothing about them. #15755 (merged) added the reading and an owner-only reclaim route; this PR is the surface for both.

Why it matters

Without a number on screen, a leak is found only after the gateway restarts on OOM. About 23 users hit this.

Not a goal

What changed (motivation → approach → change)

The RFC proposing how leaked runtimes are reclaimed is #15748 (open). This is the UI for its first step. The routes it calls landed in #15755 (merged); this PR is rebased onto main, so both exist on this head.

System > Services gains a Leaked agent runtimes card (website/src/pages/system/LeakedRuntimesCard.tsx):

  • Count and memory held from GET /api/system/leaked-runtimes, refetched every 30s. Hidden only when the route is absent (404, a gateway older than feat(runtime): read leaked untracked runtimes, reclaim on confirm #15755), the gateway reports supported: false (non-Linux), or nothing leaks. Any other read failure renders through ErrorNotice with the hand-off, so a broken reading never looks healthy.
  • Reclaim reuses the shared useArmedDelete arm → confirm → decay machine rather than a new one. A muted line under the numbers says no session uses these runtimes and reclaiming loses no work. The button is neutral at rest. First click only arms, turns it red and relabels it "Reclaim now? No work is lost" (visible label is the accessible name, no aria-label). Second click sends one POST /api/system/leaked-runtimes/reclaim with {"confirm": true}. The arm decays after 3s.
  • Result line "Ended N · kept M". A refusal renders through ErrorNotice with the agent hand-off on (the card has no unsaved input, so nothing is lost); a 403 owner_only says only the dashboard owner can reclaim, and a failed read uses a catalog sentence rather than raw backend text.
  • Rows: the error row holds hand-off + dismiss; Reclaim sits in its own row. Single-column dl, so it renders at 390px and 320px.
  • API client: leakedRuntimes() and reclaimLeakedRuntimes() in website/src/api/client/system.ts.
  • Ten catalog keys under pages.servicesTab in every locale, en-XA regenerated. Feature-map row updated.

Backwards compatibility

Compatible: a new card that self-hides when the route is absent; no existing component changes behaviour.

Tests

website/src/test/LeakedRuntimesCard.test.tsx: count and memory render; hidden at zero; hidden on 404 but a 500 shows through ErrorNotice; the first click only arms and the second sends exactly one reclaim and shows the result; a rejected reclaim shows through ErrorNotice.

Manual verification

Local: vitest (card, src/pages/system, src/i18n — 733 tests), tsc -p tsconfig.app.json, eslint, i18n:check with I18N_BASE_REF, feature-map, comment-history and brand gates. Frames below from the built SPA via website/scripts/capture-leaked-runtimes.mjs (gateway-free, /api/** stubbed, light theme).

Screenshots / video

Rest

Rest

Tip open

Tip open

Armed

Armed

Reclaimed

Reclaimed

Refused, read failed, 390px, armed at 320px

Refused (owner only)

Refused (owner only)

Read failed (500)

Read failed (500)

390px

390px

Armed at 320px

Armed at 320px

Related Issues

Refs #11899, #11789. RFC: #15748. Backend: #15755.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (feature map)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@iamwhatever
iamwhatever requested a review from a team October 1, 2026 07:31
@iamwhatever
iamwhatever requested a review from a team as a code owner October 1, 2026 07:31
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Screenshots from the built SPA (website/scripts/capture-leaked-runtimes.mjs): rest, armed, reclaimed, refused (403 through ErrorNotice), 390px, armed at 320px.

leaked-runtimes-rest

leaked-runtimes-armed

leaked-runtimes-reclaimed

leaked-runtimes-refused

leaked-runtimes-narrow

leaked-runtimes-narrow-320-armed

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Post-reclaim numbers are stale for up to a tick: the card's refetch rereads a last_reading that reclaim never updates, contradicting its own result line.

Watch

  • The card's onSettled invalidation refetches GET /api/system/leaked-runtimes, but that route serves reconciler.last_reading (handlers_system.py:819), which only run_once on the cleanup tick refreshes (runtime_reconcile.py:759, ceiling 300s in session_cleanup.py:288) — reclaim_untracked never touches it. So after a confirmed reclaim the operator sees "Ended 2 · kept 1" under an unchanged "Leaked 3 · 1.8GB" and a still-live Reclaim button for up to ~5 minutes. The "Reclaimed" screenshot shows the count dropping immediately only because capture-leaked-runtimes.mjs stubs the refetch with the next reading ("the reading the card refetches is the one the gateway would publish next") — the shipped gateway does not publish it then, so the evidence depicts behavior the system doesn't deliver in that window.
    Clears when: the card derives the post-reclaim display from the reclaim result (it has data.runtimespids andresult.killed), or a backend follow-up refreshes the reading on reclaim, or the card states the reading updates at the next pass.

Suggestions

  • Subtracting result.killed from the displayed count/RSS client-side closes the stale window inside this PR without violating its "no backend change" boundary.

[DESIGN-REVIEWED] 860ff2b

Description read

[DESCRIPTION-READ] 3818860154c5fcf74a7629eb13f425a273c2f2fc33ab7ac8567b2a606b477e32

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The description ties this UI's shape to RFC #15748, still open — a maintainer should confirm shipping ahead of it is intended.

What this change ships

Inventory (10 items) — 10 justified

Intent: let an operator see leaked agent runtimes (count, memory) and end them with a two-step confirm — an ADDITION.

  1. Services gains a "Leaked agent runtimes" card with count and memory, refreshed every 30s — justified
  2. The card hides when nothing leaks, the route is absent, or the host can't read — justified
  3. Any other read failure shows an error instead of a healthy-looking blank — justified
  4. A Reclaim button that only acts on a second, red, relabeled click (3s decay) — justified
  5. A result line "Ended N · kept M" after reclaiming — justified
  6. A refused reclaim explains itself; 403 says only the owner may reclaim — justified
  7. Two new API client calls for the existing backend routes — justified
  8. Ten translated strings in every locale — justified
  9. Feature-map row now names the card — justified
  10. A screenshot capture script for the card's states — justified

[FIRST-PRINCIPLES-REVIEWED] 860ff2b

Description read

[DESCRIPTION-READ] 8d483f4e999c7d2a139545c9da24d922bff966d9403eebd0d65716c1d330fbcb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- website/src/pages/system/LeakedRuntimesCard.tsx:64 -- after a successful read, a later 404 retains cached data, showing stale leaks instead of hiding -> Fix: return null when isNotFoundError(loadError).
FINDING -- website/src/pages/system/LeakedRuntimesCard.tsx:43 -- onSettled refetches the unchanged last_reading, restoring pre-reclaim count and memory for up to five minutes -> Fix: update the query cache from the reclaim result instead. (origin: validation)
[GPT-REVIEWED] 860ff2b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the feat/leaked-runtime-reclaim-ui branch from a2718c6 to c51a48d Compare October 1, 2026 08:22
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] New frame for head c51a48d: a failed read (500) now renders through ErrorNotice instead of hiding the card. Only a 404 (route absent) hides it.

leaked-runtimes-load-failed

@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] GPT findings on a2718c6:

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — 🟡 CONCERNS

UX-level review of 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

"Kept 1" gives no reason the card already holds, directly under copy promising "loses no work" — the blind reader could not tell whether to worry.

Watch

  • leaked_reclaim_result renders only result.refused.length while the API hands back refused[].reason ("its group leader is alive"), and the always-visible leaked_no_loss sentence keeps asserting nothing uses these runtimes → after every partial reclaim the receipt contradicts the promise; reader: "I cannot tell why one survived or whether I should worry about it. No idea." Fix: render the refusal reason beside "kept" (inline or InfoTip).
  • After an owner_only 403 the Reclaim button stays enabled and neutral below the refusal strip (shot-05); reader: "I can't tell if it's disabled, and I wouldn't click it again." Rare, but persistent while shown. Fix: disable the button when isOwnerOnly(reclaim.error).
  • Any non-owner_only reclaim failure renders raw reclaim.error.message (LeakedRuntimesCard.tsx:552) — backend exception text where the read path deliberately uses a catalog sentence. Fix: catalog fallback like leaked_load_failed.

Suggestions

  • Shot-06's load-error state offers no visible retry though the query re-polls every 30s (reader: "I don't know how I would try again"); add "Retries automatically" to leaked_load_failed or a Retry action on the ErrorNotice.

[UX-REVIEWED] 860ff2b

Description read

[DESCRIPTION-READ] fe313a1e334da7f0c7ed5e73134c2d2f7424abb5f0ec292d332ace381078aa3f

That sha256 names everything this verdict read: the PR description, plus the 11 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de — this comment is updated in place on each push.

Review details

A failed background refetch replaces the live card, so a reading that is sitting right there — and a just-confirmed reclaim's result — disappears for up to 30s.

FINDING — website/src/pages/system/LeakedRuntimesCard.tsx:51 — if (loadFailed) returns the error-only card while React Query still holds the last good data (query-core keeps data on a failed refetch and only sets status: 'error', the reason SessionSummaryTab.tsx:442 guards with error && !data), so one 502/timeout on the 30s poll — or on the onSettled invalidateQueries refetch fired right after a reclaim — swaps "Leaked 3 / Memory held 1.84GB" plus the armed Reclaim for leaked_load_failed, and when it is the post-reclaim refetch that failed the result paragraph the operator was waiting for never renders at all → Fix: take the branch only with nothing to show (if (loadFailed && !data)) and render the notice above the existing reading otherwise, as TasksCapacityCard does.

FINDING — website/src/pages/system/LeakedRuntimesCard.tsx:94 — a reclaim that ended nothing renders only as the muted leaked_reclaim_result line: with session.reconcile_max_kills: 0, or when the reported pids exit between the reading and the confirm (refused = [(pid, "no longer detected") …] in runtime_reconcile.reclaim_untracked), the route answers 200 {killed: [], refused: [...]}, reclaim.error stays unset, and result.refused[].reason — the only text saying why ("kill budget spent", "refused by the ownership gate") — is read for nothing but refused.length, so the operator sees success-shaped status and has no next move → Fix: render the distinct refused[].reason values beside the counts (machine text, translate="no", as tasks-capacity-degrade does).

[OPUS-REVIEWED] 860ff2b

Verdict parsed from the review's SHA-scoped output markers for commit 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 860ff2b44b8ed6aff47bc04b5a2f0a440f7c64de: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the feat/leaked-runtime-reclaim-ui branch from c51a48d to 95e386e Compare October 1, 2026 09:21
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] Frames for head 95e386e (built SPA, capture-leaked-runtimes.mjs): rest, tip open, armed, reclaimed with the post-reclaim reading, refused (owner_only), read failed, 390px, armed at 320px.

leaked-runtimes-rest

leaked-runtimes-tip-open

leaked-runtimes-armed

leaked-runtimes-reclaimed

leaked-runtimes-refused

leaked-runtimes-load-failed

leaked-runtimes-narrow

leaked-runtimes-narrow-320-armed

@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] Dispositions for c51a48d (head now 95e386e):

  • UX blocker (cost not stated at the action): fixed. A visible line under the numbers says no session uses these runtimes and reclaiming loses no work; the button is neutral at rest and red only when armed; the armed label reads "Reclaim now? No work is lost".
  • UX watch, raw backend text: a 403 owner_only now says only the dashboard owner can reclaim; a failed read uses the catalog sentence.
  • UX watch, reclaimed frame: the harness now serves the post-reclaim reading (1 left) after confirm. Tip-open frame added.
  • UX watch, why one was kept: the per-runtime reasons are backend English (its group leader is alive); rendering them would ship untranslated text. Left for a follow-up that maps reason codes to catalog strings.
  • First Principles, runtimes has no consumer: it is part of the route's response type; the card shows totals only by design (no per-runtime kill in this step).
  • GPT F1 (routes not on main): still true until feat(runtime): read leaked untracked runtimes, reclaim on confirm #15755 merges; this PR is rebased after it lands.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] Opus blocker on 95e386e (UI without routes) and GPT F1 are the same sequencing fact: the routes land in #15755, which was split out to keep each PR reviewable. This PR stays open and unmerged until #15755 merges, then it is rebased onto main, at which point both routes exist on the head and both findings are moot. No change here until then.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 1, 2026
System > Services gains a "Leaked agent runtimes" card: how many managed
runtimes no session tracks, and the memory they hold, read from
GET /api/system/leaked-runtimes. A line under the numbers says no session
uses them and reclaiming loses no work. The card hides when the route is
absent (404), on a non-Linux host, or when nothing leaks; any other read
failure shows through ErrorNotice.

Reclaim is the shared useArmedDelete machine: the first click only arms
and relabels the button "Reclaim now? No work is lost", the second sends
the one owner-only POST /api/system/leaked-runtimes/reclaim with
confirm: true. The result line says how many ended and how many were
kept; an owner_only refusal names who can reclaim, and every failure
renders through ErrorNotice with the agent hand-off (no draft here).

Ten catalog keys under pages.servicesTab in every locale, en-XA
regenerated. capture-leaked-runtimes.mjs photographs rest, the open
tip, armed, reclaimed (with the post-reclaim reading), refused, a failed
read, 390px and armed at 320px from the built SPA.
@iamwhatever
iamwhatever force-pushed the feat/leaked-runtime-reclaim-ui branch from 95e386e to 860ff2b Compare October 1, 2026 18:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

[operator: iamwhatever] Dispositions for 860ff2b (Design and UX are CONCERNS, not blocking; no push for these, so the lanes are not re-run):

  • Design, stale count after a reclaim: real. The read route serves the last reconcile pass, so the numbers lag up to one cleanup tick. Follow-up: have reclaim_untracked refresh last_reading on the backend, which fixes every reader, rather than subtracting on the client.
  • UX, why one was kept: the reasons are backend English strings; showing them needs reason codes mapped to catalog keys. Follow-up, same issue.
  • UX, button stays enabled after owner_only: a second click gives the same clear refusal and no signal. Follow-up.
  • UX, raw text on a non-owner reclaim failure: those are rare (not_ready, unsupported); fold into the reason-code follow-up.
  • UX, no visible retry on a failed read: the card re-polls every 30s on its own. Follow-up copy tweak.

@github-actions github-actions Bot removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 1, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 1, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 2, 2026 05:38
@iamwhatever
iamwhatever merged commit 00b5764 into main Oct 2, 2026
104 of 112 checks passed
@iamwhatever
iamwhatever deleted the feat/leaked-runtime-reclaim-ui branch October 2, 2026 05:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
iamwhatever added a commit that referenced this pull request Oct 2, 2026
buluoray: name reap_abandoned_agent_scopes as a separate arm that keeps
its own four-part check; step A (#15755, #15764) refuses a candidate whose
KIROCREW_SPAWN_HOME is not this data home, with a same-uid other-home
security row; the reading GET is authenticated, only the reclaim POST is
owner-gated; Decision 4 lists the two spawn-instance refusals that stop a
detached setsid child; only AcpRuntime stamps the spawn instance and home,
so step B stamps AcpClient._spawn and step C covers stamped paths; Windows
has no reading and #11991 stays unresolved; the leader refusals read
"still that session's / group's leader".

chenmingwei23: the spawn record is one row per runtime root (root pid,
start id, spawn instance), written at spawn before session/new and
removed only once the root is gone, never on session release or lease
drop; the lease veto comes before the spawn record, and step B's exit
criteria cover a root a new gateway re-leases after a restart.

Co-authored-by: cixuuz <26782336+cixuuz@users.noreply.github.com>
bolichen97 pushed a commit that referenced this pull request Oct 3, 2026
* docs(rfc): decide how leaked agent runtimes are reclaimed

Records the decision before the code: one reclaim arm (the runtime
reconciler), default-on with its per-pass budget and the
reconcile_max_kills=0 observe switch, ownership proved by a persisted
install-scoped spawn record that can only veto, four refusals that always
win, Linux first, and a user-confirmed Reclaim on the dashboard.

Step A (report + confirmed reclaim) adds no default authority. Steps B and
C extend it, and the First Principles lane reads that decision from the
base branch, so this document lands on its own.

Refs #11899

* docs(rfc): answer the leaked-runtime reclaim review rounds

buluoray: name reap_abandoned_agent_scopes as a separate arm that keeps
its own four-part check; step A (#15755, #15764) refuses a candidate whose
KIROCREW_SPAWN_HOME is not this data home, with a same-uid other-home
security row; the reading GET is authenticated, only the reclaim POST is
owner-gated; Decision 4 lists the two spawn-instance refusals that stop a
detached setsid child; only AcpRuntime stamps the spawn instance and home,
so step B stamps AcpClient._spawn and step C covers stamped paths; Windows
has no reading and #11991 stays unresolved; the leader refusals read
"still that session's / group's leader".

chenmingwei23: the spawn record is one row per runtime root (root pid,
start id, spawn instance), written at spawn before session/new and
removed only once the root is gone, never on session release or lease
drop; the lease veto comes before the spawn record, and step B's exit
criteria cover a root a new gateway re-leases after a restart.

Co-authored-by: cixuuz <26782336+cixuuz@users.noreply.github.com>

---------

Co-authored-by: cixuuz <26782336+cixuuz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants