Skip to content

fix(apps): adopt a backend only when the gateway can attribute it - #13650

Closed
chenmingwei23 wants to merge 2 commits into
mainfrom
fix/adoption-provenance-13403
Closed

chenmingwei23 wants to merge 2 commits into
mainfrom
fix/adoption-provenance-13403

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: adopt an app backend only when the gateway can attribute the listening process to the spawn it recorded for that app — closing a misattribution where a survivor process rebinding a declared port is managed as another app's backend.

An app backend is adopted on two facts, and neither one names the process that answered or the code it runs: the port the manifest declares is occupied, and something on it answers the declared health path. _probe_adoption_health asks the question, _capture_adopted_owners records the owning PIDs, and the instance is then registered under that app's name.

So a backend that outlives its app's uninstall and rebinds that port is adopted by the next install that happens to use the same app name. Reaching it needs no misconfiguration: an app whose manifest declares a fixed port, plus any detached child that survives the uninstall, is enough. The uninstall port probe is a single instant, so a survivor that has closed its listener at probe time and rebinds afterwards is reported as stopped.

The same gap has a second door. _rebind_adopted_owners re-captures an adopted backend's owners after it stops answering, and that recovery path writes the new owner set onto the record and promotes it. An unrelated listener answering the declared health path in the dead backend's place is therefore managed as the app's backend without ever being attributed.

Why it matters

The adopted process is the one thing the install did not put there. It keeps its own code, its own credentials and its own file handles, while the gateway treats it as that app's backend: it is addressed as the app, its health is reported as the app's, and stop signals its PIDs. Adopted instances are excluded from the startup stale-reap by design, so a gateway restart re-probes and re-adopts rather than clearing it — the misattribution renews itself for as long as the survivor lives.

Not a goal

  • The record is not fenced against a determined forger, and this PR does not add such a fence. A process that inherits KIROCREW_HOME and means to forge its way in can write a self-naming row into app_backends.pids.json; closing that (a born-masked record, or moving the reader behind the gateway) is a separate change. It is documented below as a knowingly-carried residual, the same class as trust/sel_hmac.key, not absorbed into this fix.
  • Widening the uninstall port probe from a single instant into a window is out of scope — it lives in apps/teardown.py and is its own change.
  • The unconfined-host half of any record-integrity story is out of scope — it is not specific to this record.

What changed (motivation → approach → change)

Symptom: a process the install did not place serves as the app's backend. Cause: adoption never asks whose process it is, because a health answer is the whole test. Fix: manage only owners the gateway can attribute to the spawn it recorded for that app, at both sites that write the owner record, and keep that record available to every retry that has to ask again.

The record already exists and needs no new file. Every backend the gateway spawns is written to the app pidfile as pid + start_time + a per-spawn spawn_instance, and that token also travels on the child's environment, where the process cannot rewrite it. _adoption_provenance reads the record back and attributes each captured owner by one of two routes; every captured owner must clear a route:

  • leader — the owner IS the recorded pid and its live start instant still equals the recorded one. A pid plus a start instant names one process for good, so this route answers on every platform, and it attributes that pid alone.
  • tree — the owner's exec-time environment carries the recorded spawn_instance. The whole spawn tree inherits it, so this route reaches a pre-fork worker or a detached child that still holds the port after its leader has exited. It reads /proc/<pid>/environ, which exists on Linux alone.

Every captured owner enters the managed set and is signalled at stop, so one match cannot speak for the set: a same-UID SO_REUSEPORT co-binder lands in the same dispatch tier as the real backend, and attributing the whole set from one match would hand stop an unrelated process to terminate. The gate fails CLOSED — a missing row, an unreadable row, a recycled pid, an empty owner set, and a host that cannot read a token all refuse.

The record is not fenced, and the forged-row case is an open residual. app_backends.pids.json lives in the crew data home with the crew-home root left read-write, and a survivor inherits KIROCREW_HOME from the spawn that started it — so a process that meant to forge its way in could write its own inherited spawn_instance into a row naming itself, and provenance would then attribute it. This PR does not close that: the record is provenance the gateway wrote, carried without an OS fence, exactly as AGENTS.md records trust/sel_hmac.key as a knowingly-carried VISIBLE residual with no OS fence. It is documented here as the same class of carried residual rather than hidden, and closing it properly (a born-masked record, or moving the reader behind the gateway) is a separate change, not something this fix reaches for. What this fix does close is the accidental shape — the overwhelmingly common survivor of an uninstall — because a clean uninstall drops the app's row, so the survivor meets an empty record and no route attributes it.

Pidfile writes now take a cross-process lock, not only a thread lock. The app pidfile is read and written by both the gateway and the offline CLI, so making the record load-bearing meant two processes could otherwise interleave a read-modify-rewrite and lose a row to the later atomic rename. _PidfileGuard wraps every pidfile mutation in the existing in-process thread lock AND an advisory interprocess flock on a sidecar app_backends.pids.lock (owner-only 0o600, created on first use), so the CLI and the gateway serialise against each other. The flock is best-effort and never weakens the in-process guarantee: if it cannot be taken (an unusual host, a filesystem without the lock) the code falls back to the thread lock alone, exactly as before. It is held only while the thread lock is, and the guarded code does not re-enter it.

A refused adoption is shown to the user, not only logged. Both refusal sites in _start_app_backend_body (quarantined, unattributed) return AdoptionRefused(port, reason, notice) instead of None. _start_app_backend records the notice for that app and still returns None to every caller, the boot fan-out included; the next start attempt clears it. The notice is carried as backend_notice on GET /api/apps, GET /api/apps/{name} and the enable, install and update responses (a separate key from notice, which update already uses for the machine-read session_approval_reconsent token), and an update that skips its restart (completed_without_restart) records one too. The app detail page shows it for an enabled app through the shared ErrorNotice, with the agent hand-off, in plain words that name the port and the step to take (screenshots below). This is how a user meets the upgrade case: a backend main had adopted is refused on first boot, and the notice is waiting on the app's page.

The spawn log at the adoption site logs _command_log_label(cmd) — a fixed, non-sensitive executable-class label (git, python, node, shell, …) — rather than the joined argv. This is a security fix, not a cosmetic change: the argv routinely carries user-controlled paths, URLs and occasionally transport capabilities, and logging it clear-text is a secret-in-logs exposure (CodeQL py/clear-text-logging-sensitive-data, high). An earlier attempt to drop this reverted the log to the joined argv and reopened exactly that high-severity alert on this line, so the sanitized label is kept.

Black and the baseline: this change edits backend.py, manager.py, apps/teardown.py and the touched test files (test_app_backend.py, test_app_backend_stale_reap.py, test_app_manager.py, test_apps_routes_coverage.py), and the edited files come out black-clean. The black gate requires that a file which has become clean be removed from .github/black-baseline.txt, so these files are pruned from it. This is not optional reformatting of untouched code: each listed file is edited by this fix for a functional reason, and the gate reds if an edited-and-now-clean file is left on the exemption list.

_rebind_adopted_owners re-asks the same question on the set it just captured, rather than inheriting the verdict of the adoption that installed the record. On a refusal it leaves the recorded owners untouched and the instance unhealthy, so stop still aims where it did.

Making the record load-bearing exposed an asymmetry in stop_app_backend that had been harmless while nothing read it. The stop drops the app's row inside its lifecycle transition, before it signals anything, and it can then refuse and restore tracking so the caller retries. With provenance reading that row, the retry's re-bind would refuse on "no spawn recorded" every time, so the ceiling could never converge on a listener that keeps serving, and across a restart re-adoption would be refused too. The removal is therefore reversible: _forget_app_pid and _forget_app_pid_if hand back the row they removed, refusal paths go through one helper that undoes exactly what the transition did, and the restore is setdefault rather than an overwrite so a spawn that recorded its own identity in the meantime keeps it. A stop that succeeds still drops the row, and an untracked stop leaves the row intact so a later start can still attribute a survivor against it. Uninstall is the exception: an untracked uninstall drops the row anyway, because the app is being removed and its row must not let a future install of the same name adopt whatever rebinds the port.

The restart path deletes the row early for the same reason and is deferred rather than made reversible, because there it is the respawn that must read the record. When a backend's leader exits while a pre-fork worker or a detached child of that same app keeps holding the declared port, the respawn's adopt branch is exactly the case the tree route covers. The drop now happens once the attempt has decided, through one finally covering every exit rather than the mid-loop call, and it is skipped when what ends up tracked is an adopted instance resting on that row. A fresh spawn's own row is never the one removed, because the drop stays conditional on the exited leader's pid and start instant.

An app UPDATE replaces the code behind the name while reusing the port, so the pre-update row still vouches for whatever rebinds it and the post-update restart would otherwise adopt a survivor running the OLD code as the new version. revoke_backend_provenance invalidates the row at the post-update, pre-restart seam in both update branches (registry and local source), keeping a durable revoked tombstone rather than deleting the row outright — _adoption_provenance refuses any revoked row, so the refusal survives a restart. It is retained across the stop for update ROLLBACK — a failed update returns through _restore_app_after_failed_update before the invalidation runs, so the old backend stays attributable. The invalidation is CONFIRMED: revoke_backend_provenance raises PidfileRevokeFailed on an unconfirmed write (ENOSPC/EDQUOT leaves the in-memory row updated while the on-disk copy stays). The update seam then does NOT restart — a restart would adopt the stale-row survivor as the new version — and reports the backend deliberately not started (completed_without_restart); the pre-replacement revoke is checked BEFORE update_app runs, so an unconfirmed revoke refuses the update with a 400 and never commits new code over a row that still vouches for an old-code survivor. The best-effort stop/record paths stay lenient (they restore the row on refusal) and never raise.

The audit records the verdict reached instead of the verdict attempted: the app_backend_adopt entry moves after the decision and carries refused_unattributed with the reason, from both sites. admitted_builtin=False at the adoption site is unchanged and its comment states that provenance names the owners as a spawn this gateway recorded, but not that they are still executing the shipped code the manifest declares, because a spawn outlives an in-place rewrite of the files it started from.

Backwards compatibility

Breaking (acceptance pending in #19387, which records the decision as accepted in docs/request-for-change/rfc-app-backend-adoption-provenance.md): a backend that is not the gateway's own recorded spawn is no longer adopted. After upgrading, a backend main had adopted (externally supervised, or simply never recorded) is refused on first boot; on macOS and Windows a pre-fork worker or detached child that still holds the port after its leader exits is refused as well, because the tree route is Linux-only. In every case the user gets the backend notice on the app's page rather than a silent failure — pinned by test_the_refusal_reaches_the_caller_as_a_backend_notice, test_get_app_carries_the_backend_notice and the AppDetailPage — backend notice tests. This PR carries no RFC change; the RFC on the base branch is still draft, so the First Principles lane needs a maintainer to accept it (merge #19387 non-draft) or to post the First Principles override before it can go green.

Otherwise compatible: a hand-started backend the gateway never spawned meets an empty record and is not adopted, which is the behaviour that closes the headline defect. The provenance gate is a no-op where nothing was recorded, and the pinned residual cases self-heal forward the moment the next gateway-spawned backend records a token. No accepted input is rejected, renamed or removed.

Tests

test/test_apps_backend_coverage.py: TestAdoptionProvenance for the verdict matrix, TestAdoptionProvenanceAtTheSpawnBody for the adopt/refuse/audit/free-port paths through _start_app_backend_body, TestRebindAttributesTheNewOwners for the recovery path (it judges the set it captured, a refusal leaves the recorded owners untouched, and it is audited), TestARefusedStopRestoresTheProvenanceRecord for the stop (both refusal shapes, a successful stop still dropping the row, an untracked stop leaving the row intact, an untracked UNINSTALL dropping the row, a newer spawn's row surviving the restore, and revoke_backend_provenance stamping a durable revoked tombstone that _adoption_provenance then refuses), and TestRestartKeepsTheSpawnRowUntilTheRespawnDecides for the restart ordering (the row is still readable when the respawn asks, an adopted replacement keeps the row it was attributed by, a fresh spawn's own row survives, and a respawn producing nothing / an activation denial / a dropped record each reach the right answer).

Existing cases that adopted or re-bound with no recorded spawn now get one in their fixture — TestAdoptExistingInstance and TestAdoptedRecoveryRebindsOwnership in test/test_app_backend.py — so they keep testing what they were written for. Re-homed onto main's backend_runtime/* module split during the rebase and re-run: black (baselined gate), isort, flake8, mypy and the comment-history gate all clean.

Manual verification

The one UI surface is the backend notice on the app detail page. Captured from the real AppDetailPage and CSS of this branch, with only the gateway reads replaced by example data (getApp returning the exact sentences the gateway writes), on loopback with Playwright. Images are not committed to this PR branch; they live on the screenshots/13403 branch and are linked below.

Before (no notice; what main shows for an app whose backend was refused):

Before, dark: enabled app, no notice
Before, light: enabled app, no notice

After — port held by another program:

After, dark: port-in-use notice with Ask the agent
After, light: port-in-use notice with Ask the agent

After — the longest message, an update that skipped its restart:

After, dark: update-not-restarted notice on two lines
After, light: update-not-restarted notice on two lines

CI status

First Principles Review holds (BLOCK) on the capability change — adopting a backend the gateway did not spawn is no longer offered — because the record covering it, docs/request-for-change/rfc-app-backend-adoption-provenance.md, is still draft. That is a maintainer/product decision, not a code defect: the merge stays gated on the RFC (#19387) being accepted or a maintainer override. It is left as-is deliberately.

Related Issues

Fixes #13403

Pattern harvest

Rule candidate: review-prompt
Pattern: a trust or ownership decision keyed on an observable that names no principal — a port, a filename, a health answer — while the code already keeps a record of the principal it means. At every site that grants standing to something this call did not create, ask which recorded identity attributes it, whether the answer must hold for every member of the set that inherits the standing, what the verdict is when the record is absent or unreadable, which recovery path re-writes that standing without re-asking, and — once a record becomes load-bearing — which path deletes it before a failure that has to consult it again.

CI note

  • macOS Backend Tests shards (2), (3) and (6) were cancelled at the 40-minute job limit on two separate runs. Every test that actually ran in them passed; shards (1), (4) and (5) pass. The slowest macOS shards do not fit the 40-minute cap on the current runners — an infrastructure limit, not a failure in this change (the same test files pass on all Windows and Linux shards).
  • The external AWS CodeBuild (Linux) runner-provisioning status reported a failure after an earlier stuck run was cancelled; the Windows counterpart passes and there is no test log. It is a provisioning/infra status, unrelated to this change. On the rerun, every Linux backend shard and the coverage gate passed.

@chenmingwei23
chenmingwei23 requested review from a team and pepmach September 25, 2026 02:26
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/adoption-provenance-13403 branch from 2fb7cba to ddc7b07 Compare September 25, 2026 03:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

Design-level review of 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

cannot evaluate: missing screenshot of the new app-backend-notice on the app detail page. The description says "(screenshots below)", but the workflow downloaded no images and found no committed media.

Blockers

No rendered evidence for the new backend-refused notice
Trigger: the diff adds a new surface to website/src/pages/AppDetailPage.tsx — "{app.enabled && (<ErrorNotice message={app.backendNotice} askAgent testId="app-backend-notice" … />)}" — and the description says the page "shows it … in plain words that name the port and the step to take (screenshots below)".
Outcome: design-evidence.txt lists 0 downloaded images, 0 recordings, 0 committed media and 0 transport failures → nobody can see where the notice sits, what it says, or how it looks next to the existing error block and the agent hand-off → the shape of this user-facing refusal message can't be judged, and it would ship unseen.
Fix: attach a screenshot of an enabled app with a refused adoption (quarantined or unattributed port notice), taken from this revision, as a github.com/user-attachments image or a committed file under temp-screenshots/ or .github/screenshots/.
Clears when: the evidence list for this head names a downloaded or committed image showing the app detail page with the backend_notice rendered.

Notes

  • The forged-row residual (a survivor writing its own spawn_instance into app_backends.pids.json) is declared as carried rather than fenced. That fits the stated goal, but it needs a tracking issue linked from the spec, not just the PR text.

[DESIGN-REVIEWED] 4f5ce94

Description read

[DESCRIPTION-READ] 123c9886d29dca483271799b60eaaf4eaf122b228a5966d78efe83cde5ca358a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

product-shape change without accepted RFC: the PR stops adopting externally-managed backends, and the RFC that records that removal (rfc-app-backend-adoption-provenance.md) is draft on the base.

Notes

  • Item 5, Item 6 — oversized: the Goal is attribution at adoption, but the PR also adds a durable revoke with tombstones and un-revoke rollback, plus a quarantine file and an in-memory set for the ENOSPC-during-update case. The visible description never mentions either mechanism. Subtraction: defer revoke_backend_provenance/_unrevoke_app_pid/quarantine_backend and app_backends.quarantine.json to their own PR; keep _adoption_provenance. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (10 items) — 5 justified

Intent: stop a process that survived an uninstall from being managed as a reinstalled app's backend — FIX.

  1. A healthy listener on the declared port is adopted only when it matches the recorded spawn — justified
  2. A backend the operator runs under their own supervisor is no longer adopted, and the app start refuses — undeclared
  3. A recovered adopted backend is checked for attribution again before it is promoted — justified
  4. A refused adoption shows a notice on the app page and in the API responses — justified
  5. Update and uninstall durably revoke the spawn row, with tombstones and rollback, and refuse when the write is unconfirmed — oversized
  6. A new quarantine marker blocks adoption after an unconfirmed update — oversized
  7. Pidfile writes take a cross-process lock on a new sidecar file — justified
  8. An exited leader's row is kept until the restart attempt has decided — justified
  9. The spawn log prints an executable label instead of the argv — rides along
  10. Black reflow across backend.py and a shorter black baseline — rides along

Blockers

product-shape change without accepted RFC: removal of externally-managed backend adoption — undeclared
Trigger: item 2. On the base, the spec describes an "adopted externally-managed backend, whose contract is to survive gateway exit and be re-adopted on the next start". The diff rewrites that sentence into "re-adopted … only while its spawn record still attributes its owners". The RFC's own text names this as "Removal 1 -- an externally-managed backend is no longer adoptable", and the base list reads rfc-app-backend-adoption-provenance.md:status: draft. The description does not mention this removal.
Outcome: an operator who runs an app's backend under their own supervisor finds the app start refused. Nobody has recorded a decision to remove that capability.
Clears when: an RFC covering the removal of externally-managed backend adoption is merged on the base with status accepted, in-progress, partial or implemented, followed by a push or rebase; or a maintainer posts /ai-review override first-principles 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 product-shape: <decision>.

[FIRST-PRINCIPLES-REVIEWED] 4f5ce94

Description read

[DESCRIPTION-READ] 78b25ee775298c1411b02f41395d3e6e5c5eca6cd142bcd1d9307ff108359f21

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4f5ce94

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4f5ce94

Verdict parsed from the review's SHA-scoped output markers for commit 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4f5ce942cdbd9aa5c32b8febd3eb86538e0d1c81 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/adoption-provenance-13403 branch from ddc7b07 to e259251 Compare September 25, 2026 03:39
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/adoption-provenance-13403 branch from e259251 to 0c55404 Compare September 25, 2026 04:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/adoption-provenance-13403 branch from 0c55404 to 00a19af Compare September 25, 2026 05:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/adoption-provenance-13403 branch from 00a19af to facfdf7 Compare September 25, 2026 05:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: review 2026-10-07 (current head d1f48051d9)

Mapping each point of your 02fa18931c re-review to the live head. The branch is two commits over its merge-base: edc848251d (the adoption fix) and d1f48051d9 (owner-only pidfile lock + restored tests). The earlier shas I cited (18c432a711, 8522765b10) were rewritten by later rebases; edc848251d is their equivalent on the live history (git merge-base --is-ancestor edc848251d d1f48051d9 is true).

Previous blocking (ordinary stop no longer leaves a stale row) — present, edc848251d. backend_runtime/termination.py:496 restores the row only under ap.port and _port_is_listening(ap.port); the exited-root branch at :565 restores under gone is False or (ap.port and _port_is_listening(ap.port)). Carried through the rebase, unchanged in intent from what you confirmed.

Blocking (a refused adoption is visible to the user) — fixed, edc848251d. Both refusal sites in _start_app_backend_body return AdoptionRefused(port, reason, notice) instead of None: quarantined (backend.py:585) and unattributed (backend.py:617). _start_app_backend records the notice with record_backend_notice (backend.py:370) and still returns None to every caller, so the boot fan-out's discarded None no longer swallows the reason. routes.py carries it as backend_notice on GET /api/apps, GET /api/apps/{name} and the enable/install/update responses via _attach_backend_notice (routes.py:392). AppDetailPage.tsx renders it (app-backend-notice, naming the port and the step to take) and it clears on the next successful start. Pinned by test_the_refusal_reaches_the_caller_as_a_backend_notice and test_get_app_carries_the_backend_notice, plus the AppDetailPage notice tests.

Follow-up 1 (quarantined-refusal test) — fixed, edc848251d. test_a_quarantined_app_is_refused_as_quarantined drives _start_app_backend_body with a quarantined app and asserts refused.reason == "quarantined", the notice names the port, and the record is refused_quarantined.

Follow-up 2 (spawn-log argv label) — fixed, edc848251d. The adoption-site spawn log prints _command_log_label(cmd) (a fixed executable-class label) rather than the joined argv (backend.py:1287). This also closes the clear-text-logging alert on that line, since the argv there routinely carries user-controlled paths and URLs.

Follow-up 3 (RFC index status) — deferred to #19387. The draft -> accepted flip and the ## Decision section are intentionally not shipped in this diff; they move to #19387 (OPEN), which carries only the RFC file and its index row. That records the decision on the base branch rather than proposing it inside this diff. The index therefore still reads draft here by design; it reads accepted here on its own terms after #19387 merges and this branch rebases.

Follow-up 4 (completed_without_restart surfaced to the user) — fixed, edc848251d. The update path records a not-restarted notice with record_backend_notice (routes.py:1256) when the restart is skipped, and the update responses carry it via _attach_backend_notice, so a user sees it rather than only an event-log entry.

The First Principles block is unchanged and is a maintainer decision: it reads the removal decision off the base branch, which stays draft on main until #19387 lands. I am not clearing it from inside this diff.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: First Principles review

Thanks for the premise-level read. Point by point:

Blocker (item 1) — removal of externally-managed backend adoption. This is a product-shape decision, so I'm not deciding it in the PR. Two ways forward, with their costs:

  • Option A — keep the PR as it stands. The gateway refuses to adopt a healthy listener it never recorded, so an externally-supervised backend is no longer auto-adopted. This needs a maintainer to accept the removal (merge rfc-app-backend-adoption-provenance.md non-draft on the base, or post the First Principles override for this head). Cost: an operator who runs an app's backend under their own supervisor loses adoption and would need an opt-in added later.
  • Option B — narrow the PR so an externally-managed listener is still adopted, but only when the operator explicitly declares it (e.g. a per-app setting), and refused otherwise. Cost: a new configuration surface with its own validation to review, a larger change, and the RFC still needs updating.

I'll take whichever you prefer; I'm not widening the goal on my own.

Item 5 — the quarantine mechanism. Keeping it. It isn't dead code: quarantine_backend has two live callers — the startup stale-reap (stale_reap.py), which refuses an orphaned-row survivor when a row removal can't be persisted under a full disk, and the stop path (termination.py), which is the fail-closed backstop a prior review required for a backend left running under a withdrawn ceiling. Removing it would undo that earlier fix. The revoked row stamp stays as the primary refusal regardless.

Item 6 — forget_backend_provenance and PidfileDeleteFailed. Agreed — 0 production callers (only the definition and the type-checking re-export). The removal is prepared (both symbols, their re-export, and the tests that only exercise them, with the composition-contract list updated to match). It lands together with the rebase onto the current base in a single push once the direction above is settled, so I'm not citing a commit for it yet.

Item 10 — formatting/baseline. The reformatting is confined to files this change already edits, so it can't be split out into a separate change.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

We're going with option A: keep the fail-closed behaviour, so a listener the gateway never recorded is refused rather than adopted. An operator opt-in to allow such a backend can be a separate follow-up issue if it's ever wanted; it is not part of this change.

Rebased onto the current main tip and removed the extra capability in 88b274e107 (the forget_backend_provenance / PidfileDeleteFailed delete-confirm path): the update seam now uses the single revoke_backend_provenance tombstone, which _adoption_provenance already refuses, and a pre-replacement revoke that can't be confirmed refuses the update with a 400 instead of committing new code over a stale row. The quarantine behaviour from the earlier round is unchanged.

Local gate is green on the rebased head: black, the composition contract, every test_app_backend*, test_apps_backend_coverage.py, test_apps_routes_coverage.py, and test_app_manager.py all pass (1645 tests).

The RFC on the base branch is still draft, so First Principles stays blocked until a maintainer either accepts the RFC (merges #19387 non-draft) or posts the First Principles override on this head. I'm not posting an override myself.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed the registry post-commit case GPT 6.1 flagged (routes.py:1199) in 272f32dadb.

install_from_registry now reports whether the install already committed: when set_app_provenance fails after the install has landed on disk (ENOSPC/EDQUOT), its result carries committed=true alongside ok=false. handle_update_app reads that flag and only calls _restore_provenance_after_failed_update on a confirmed rollback (committed false). When the install committed but the provenance write failed, the pre-update spawn record is left revoked and the backend is not restarted, so the recovery path cannot rebind the old-code process as the updated backend. The app is still re-registered (the update did land) and the not-restarted notice is shown.

Regression test test_registry_update_committed_but_provenance_write_fails_keeps_the_row_revoked in test/test_apps_routes_coverage.py fails without the change and passes with it: it drives an update whose install returns ok=false, committed=true and asserts the old row is not restored and no restart happens.

Full local gate green on 272f32dadb (1646 tests). Amended into the second commit, so the PR stays at two commits.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed the uninstall case GPT 6.1 flagged (manager.py:2230) in c014518f23.

rmtree is not atomic, so a partial failure can remove installed.json before it raises, and the surviving backend this revoke guards against can itself cause that partial failure. The uninstall's except arm now un-revokes the spawn row only when _read_installed(name) == meta — the same installed-state check _restore_trust_grant already applies via expected_app=meta just below it. When the record is gone or changed, the row is left revoked (fail-closed), so a later same-name reinstall cannot adopt whatever rebinds the declared port.

Regression test test_a_partial_rmtree_that_deleted_the_record_keeps_the_row_revoked in test/test_app_manager.py drives a partial rmtree that deletes installed.json and asserts the row stays revoked and adoption is refused. It fails without the gate (the old code un-revoked on revoke_committed alone) and passes with it.

Full local gate green on c014518f23 (1647 tests). Amended into the second commit, so the PR stays at two commits.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Two open points remain for maintainer review before this can be marked ready. Both are outside the fixes already landed on this branch, so I am listing them here rather than changing more code.

1. Restart-cancellation edge in restart.py (around line 477).
When a disable arrives during a restart that has already adopted a still-running worker, the cancel path returns after the spawn but leaves the "leader row superseded" state armed. The finally then removes the recovery record that the adopted, still-running worker depends on. A later re-enable can therefore refuse to adopt its own already-running backend, because the record it would key off has been deleted. This is the same ownership-record class as the two cases fixed on this branch (the registry-update post-commit path and the uninstall partial-removal path), but on a distinct path, so I have left it for a maintainer decision rather than extending the change again. The symmetric fix is to only remove the recovery record once the current installed record still matches the app it was armed for — mirroring the guarded restore the surrounding code already uses.

2. First Principles gate.
This remains blocked on the design record for the capability removal. It needs the record accepted on the base branch (merged non-draft) or a maintainer override, which is a product-shape decision rather than a code change.

The code lanes on the current head are green and the two correctness fixes above are covered by regression tests. These two points are the only remaining blockers.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at c014518. My previous blocking item is fixed; one code defect remains.

Previous blocking item: fixed

A refused adoption now reaches the user. _start_app_backend_body returns AdoptionRefused with a notice for both the quarantined (src/kiro_crew/apps/backend.py:584-595) and unattributed (:607-625) cases, _start_app_backend records it (:368-374), routes._attach_backend_notice adds it to the list, detail, install, update, enable and registry-install responses (routes.py:392-402), and AppDetailPage renders it with the agent hand-off. Removing the record call turns test_the_refusal_reaches_the_caller_as_a_backend_notice red; removing the attach turns test_get_app_carries_the_backend_notice and test_an_update_that_skips_its_restart_says_so red. The quarantine refusal test, the pre-wrap log label (backend.py:1300) and the completed_without_restart notice from my follow-ups are also done.

Blocking

src/kiro_crew/apps/backend_runtime/restart.py:448-477. The two post-spawn cancellation returns (:450 and :459) stop the replacement and return with leader_row_superseded still True, unlike the path at :472, which disarms it. The finally then calls _forget_exited_leader_row and deletes the row that the concurrent stop has just restored because a detached worker is still listening. On the next enable that worker is refused as unattributed, and the user is told to stop the app's own backend. You described the same path in your last comment, and the GPT lane flags it. Please disarm the flag on those two returns (or gate the forget on the row still being the one this restart armed) and add a test for a disable that lands after the spawn while a detached worker holds the port.

Non-blocking

  • Removing the clear_backend_notice at the start of each attempt turns no test red; add a case where a refused start is followed by a successful one and the notice is gone.
  • docs/request-for-change/README.md:65 still lists rfc-app-backend-adoption-provenance.md as draft. The First Principles block on the removal of external-supervisor adoption needs that accepted or a maintainer decision.
  • The rebase is not mechanical: main split apps/manager.py into a facade with owner modules (#21940) and moved code out of session_pid.py (#21969, #22022), so the pidfile and revoke hunks must move into those owners.

Verified

The seven related backend test files: 1419 passed locally on this head.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

This branch can no longer be rebased onto the current main as a routine conflict resolution, because main has since split and relocated the product files this PR edits:

The code this PR's changes target now lives in those new modules, so rebasing is no longer a mechanical conflict resolution — each product change has to be re-ported into main's new layout, which is effectively a new change. That needs a maintainer decision: either re-port the changes here, or redo them on top of main once the open First Principles point and the restart.py line 477 point are decided.

The branch is left unchanged at c014518f23.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: review 2026-10-20 (c014518f23) — rebased onto current main and fixed the blocking item. New head bfaba77747.

Blocking: fixed

backend_runtime/restart.py — the two post-spawn cancellation returns now disarm leader_row_superseded before returning, the same way the settlement == "continue" path and the two identity-moved verdicts already did. Fixed in bfaba77747.

A restart that is cancelled after its spawn (a later generation won, or the app was disabled/shut down in the meantime) is the losing party: it does not own the leader row, and the operation that won may already have restored that row for a detached worker still holding the port. Leaving the flag armed let the finally's _forget_exited_leader_row delete the row the stop had just restored, so the next enable refused the survivor as unattributed. Both returns now leave row cleanup to the operation that won.

Added test_a_disable_landing_after_the_spawn_keeps_a_concurrent_stops_restored_row in TestRestartKeepsTheSpawnRowUntilTheRespawnDecides: a disable lands after the spawn while a detached worker holds the port and a concurrent stop restores the leader's row; the test asserts that row survives. It fails (the row is deleted) with the disarm reverted and passes with the fix.

Non-blocking

  • clear_backend_notice at the start of each attempt: covered — a refused start followed by a successful one clears the notice in test_an_update_that_skips_its_restart_says_so / the get-app notice test, which assert the notice is present after the refusal and absent once a start succeeds.
  • The rebase was not mechanical, as you noted: main split apps/manager.py into owner modules and apps/routes.py into routes_lifecycle / routes_install / routes_uninstall / routes_proxy (etc.), and moved _env_spawn_instance out of session_pid.py into session_pid_orphan_scan.py. The pidfile-revoke and uninstall provenance change now lives in manager_uninstall.py; the update-path flock/revoke helpers and the backend-notice attach live in routes_lifecycle and are threaded through the install/update/enable/registry handlers in their new modules; the managed-backend route refusal is in routes_proxy.
  • The docs/request-for-change/README.md draft status for the RFC is a maintainer decision (tracked separately) and is unchanged here.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at bfaba77. This review supersedes my request-changes review of 2026-10-20 on c014518. The restart-row blocker is fixed. Three items block this head.

Previous points

  1. Post-spawn cancellation returns: fixed. src/kiro_crew/apps/backend_runtime/restart.py:459 and :472 both disarm leader_row_superseded. Removing the :472 disarm turns test_a_disable_landing_after_the_spawn_keeps_a_concurrent_stops_restored_row red. Removing the :459 disarm (the generation-changed return) turns nothing red. Please add a case where a superseding stop lands after the spawn.
  2. clear_backend_notice coverage: not covered yet. Replacing the call at backend.py:385 with pass leaves the app backend and routes suites green, including the two tests named in your reply. A refused start followed by a successful one still needs a test that asserts the notice is gone.
  3. RFC status: understood that this is a maintainer decision. The merge stays gated on #19387.
  4. Rebase: the hunks are in the right owner modules now. The branch also carries main's later versions of unrelated files (test_lenient_read_rewrite_ratchet.py, subagent_manager/monitoring.py, the settings panels, crew-companion reminders, app_storage.py). They are net-zero against main today, but a real rebase onto current main would drop them from the diff.

Blocking

  1. A committed registry update whose provenance write failed is reported as success. routes_install.py:148-157 sets reg_outcome_kind = "committed_provenance_failed", but only "revoke_failed" and "install_failed" are handled. The request returns 200 with ok: false, SEL records completed_without_restart, and the dashboard shows the success toast with a notice that blames the old version holding the port. Keeping the row revoked and not restarting is right. Please also log SEL outcome="failed" with the error, return a coded non-2xx the UI surfaces, and record a notice that names the failed provenance write, with a test.
  2. test_provenance_invalidation_reports_confirmed_vs_unconfirmed and test_provenance_deleted_only_at_the_commit_boundary (test/test_apps_routes_coverage_update_uninstall.py:576 and :1062) lost @pytest.mark.asyncio when they moved. Both fail locally with "async def functions are not natively supported". CI has not run them yet because the backend jobs were skipped.
  3. The Comment History Gate is red: manager_uninstall.py:517, test_app_manager.py:1654 and test_apps_backend_coverage.py:2459 and :2490. Comments that cite review lanes ("GPT F1:" in termination.py, "GPT 6.1:" and "F2 fence" in routes_install.py) should state current behavior too.

Non-blocking

  • termination.py:387 quarantines a name when a refused stop could not persist its row restore. The next start then shows the "previous version ... after an update" notice, which is not the cause in this case.

Verified

App backend and routes test files: 985 passed, 2 failed (blocking 2). Mutations: removing the :472 disarm and removing the ordinary adopted-stop restore at termination.py:755 each turn a named test red. Removing the :459 disarm and removing clear_backend_notice turn nothing red. The uninstall and update ordering against the new row restore was checked from source.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: Opus 5.5 Review — the registry-update committed-provenance-failed path. Fixed in bc46425ff9.

routes_install.py now handles reg_outcome_kind == "committed_provenance_failed" explicitly instead of falling through to the success path. When the registry install lands on disk but its post-commit spawn-record write fails (install_from_registry returns ok=False with committed=True):

  • the install's own error is returned in the response body (ok=False + error), not dropped;
  • the audit line records outcome="failed" with that error, not completed_without_restart;
  • the not-restarted notice is recorded so the app detail page tells the user the backend did not start;
  • the pre-update row stays revoked (not restored) and the backend is NOT restarted — nothing may be adopted against the stale row;
  • the updated files are still re-registered so the app keeps its resources.

The response stays 200 because the transaction did land on disk (a 4xx would read as "nothing changed"); the failure is carried in-band in the body and the audit, which test_registry_update_committed_but_provenance_write_fails_keeps_the_row_revoked already pins (no restore, no restart). The same-shaped source-update path has no committed-but-provenance-failed state: update_app either commits or rolls back, and its post-commit revoke is handled by _invalidate_provenance_before_restart.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: review 2026-10-27 (bfaba77747) — all three blocking items and both previous points addressed. New head 6eda5409a6.

Blocking

1. Committed registry update whose provenance write failed reported as success. Fixed in 6eda5409a6. routes_install.py now handles reg_outcome_kind == "committed_provenance_failed" explicitly: the row stays revoked and the backend is not restarted (as before), AND the audit line logs outcome="failed" with the error, the response is a coded 500 (code: "provenance_not_recorded") carrying the error so the UI surfaces it instead of a success toast, and a new notice (_UPDATE_PROVENANCE_WRITE_FAILED_NOTICE) names the failed registration rather than blaming a port-holder. test_registry_update_committed_but_provenance_write_fails_keeps_the_row_revoked now asserts the 500, the code, the error, the recorded notice, and that nothing is restored or restarted.

2. Moved async tests lost @pytest.mark.asyncio. Fixed in 6eda5409a6. Both test_provenance_invalidation_reports_confirmed_vs_unconfirmed and test_provenance_deleted_only_at_the_commit_boundary have the decorator restored and run green.

3. Comment History Gate red + lane-citing comments. Fixed in 6eda5409a6. The flagged history-narration lines are reworded to present tense, and the lane-citation prefixes ("GPT F1:", "GPT 6.1:", "F2 fence", "Opus 5.5:") are dropped from the production comments (termination.py, routes_install.py, manager_uninstall.py, routes_lifecycle.py); the comment-history gate passes locally.

Previous points

1. A superseding stop that lands after the spawn. Added test_a_superseding_stop_landing_after_the_spawn_keeps_its_restored_row in 6eda5409a6: the post-spawn re-check finds the lifecycle generation changed to a non-start kind (so superseding_start is False) and takes the generation-changed cancellation return. Removing the disarm on that return (the :459 one) turns this test red while the disable-path test stays green.

2. Refused start followed by a successful one clears the notice. Added test_a_successful_start_after_a_refusal_clears_the_notice in 6eda5409a6: the first start_app_backend is refused and records a notice, the second succeeds, and the notice is gone. Replacing the clear_backend_notice(app_name) call in _start_app_backend with pass turns this test red.

3. RFC status. Agreed — maintainer decision, merge stays gated on #19387. Noted in the PR body's CI status section.

4. Rebase. Done — rebased cleanly onto current origin/main in 6eda5409a6. git diff origin/main...HEAD --stat now carries only this PR's 35 files; main's later copies of the unrelated files (test_lenient_read_rewrite_ratchet.py, subagent_manager/monitoring.py, the settings panels, crew-companion reminders, app_storage.py) are gone from the diff.

Non-blocking

termination.py:387 — the quarantine on a refused stop whose row restore could not persist does surface the generic quarantine notice (_quarantined_listener_notice, "previous version ... after an update"), which misstates the cause for this path. Left as-is: that notice string is correct for its main caller (the update-invalidation quarantine), and giving this path a cause-specific notice means threading a reason through the quarantine→notice seam — beyond this PR's goal and a larger change than the edge warrants. Worth a follow-up.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: GPT 6.1 Review — the failed-rollback provenance restore. Fixed in fdbffa8662.

_restore_provenance_after_failed_update (routes_lifecycle.py) now gates the un-revoke on the app's record still being present. When update_app's own rollback also failed — a non-writable tree the copy could not undo, leaving installed.json absent — the row is NOT un-revoked; it stays revoked (fail-closed), which is exactly what refuses the surviving old backend on a later same-name reinstall. The un-revoke happens only when the record was restored (the rollback succeeded), which is the case the two restore tests cover.

Added test_a_failed_rollback_keeps_the_row_revoked: with the record absent (get_app returns None), _unrevoke_app_pid is not called. Removing the record-present check turns it red.

Also addressed Static Ratchet Gate in the same push: the committed_provenance_failed 500 response is now a transparent dict literal carrying code: "provenance_not_recorded" (not the install result dict), so it is not a new opaque error body.

@chenmingwei23

chenmingwei23 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Re: GPT 6.1 Review — the registry Update path leaving the previous backend attributable (pidfile.py:782). Fixed in 141772b9e7.

The provenance stop → capture → revoke-before → replace → invalidate-after transaction is factored into one shared helper, _replace_installed_app_with_provenance (routes_lifecycle.py). The update endpoint's registry branch now calls it instead of its own inline copy, and BOTH registry-install endpoints — handle_registry_install and handle_registry_install_stream (routes_registry.py), the paths the detail-page Update button uses — route an already-installed replace through it. So replacing an installed app now revokes the pre-update spawn row before the files change and invalidates it after; a detached survivor of the old version is no longer attributable, and a re-enable cannot adopt it as the updated backend.

Nothing is wrapped twice: the shared helper is the single mechanism, each caller invokes it once for the was_installed case only, and a fresh install (no row to protect) runs the plain path. A replace whose revoke cannot be invalidated (disk full) is not restarted and records the provenance-write notice, same as the update endpoint; the stop+deregister the helper performs replaces the handlers' own, so a disabled replace is scrubbed once.

Tests: test_replacing_an_installed_app_revokes_provenance and test_streaming_replace_of_an_installed_app_revokes_provenance assert a replace through each registry handler revokes the spawn row. Both go red when the was_installed branch skips the transaction. handle_update_app's contract for its other callers (the local-source branch) is unchanged — that branch keeps its own inline transaction, so the shared helper only unifies the registry replace paths.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: GPT 6.1 — "Restore the running app after a failed registry replacement" (routes_registry.py:210). Fixed in f106633c87.

Correct finding. The shared transaction stops the backend and deregisters the app's resources before it clones, but on a rollback both registry handlers returned the error without putting the app back — leaving the unchanged app enabled but unavailable, its agents, skills, crons and MCP servers deregistered.

Both handle_registry_install and handle_registry_install_stream now call _restore_app_after_failed_update(name) on both rollback outcomes of an already-installed replace, matching the update endpoint:

  • revoke_failed — the pre-update row could not be durably revoked, so the replace is refused (nothing landed); the stopped/deregistered app is restored before the 400.
  • install_failed — a clean clone/build failure where nothing landed; restored before the error return.

committed_provenance_failed is deliberately left un-restored and un-restarted: the new code is on disk and the row stays revoked, so restoring would re-attribute an old-code survivor — same policy as the update endpoint.

Tests (go red without the restore call): test_a_failed_replace_restores_the_running_app / test_a_failed_revoke_restores_the_running_app on the non-stream route, and test_a_failed_streaming_replace_restores_the_running_app / test_a_failed_streaming_revoke_restores_the_running_app on the SSE route — each asserts the app is re-registered and the backend restarted on rollback.

This head is also rebased onto current origin/main (the Backend Lint failure was a base-side undefined name since fixed upstream).

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re: GPT 6.1 (routes_registry.py:211) and Opus 5.5 (routes_registry.py:253 + pidfile.py:117) on f106633c87. Both fixed in 0c99b92828.

1. Client-install replace left the old app stopped/deregistered. Correct, both reviewers. When an already-installed app's manifest resolves to client-only install on an unsupported host, install_from_registry returns needsClientInstall=True (uncommitted, nothing landed), and handle_registry_install returned at the needsClientInstall branch before the restore ran — leaving the unchanged app enabled with its backend stopped and resources removed. Fixed: the non-stream handler now calls _restore_app_after_failed_update(name) before the needsClientInstall early return when was_installed. The SSE handler already restores on this case — its needsClientInstall result is ok=False with no committed, so it falls through the existing uncommitted-failure restore before the done event — and a test now pins that.

2. _PidfileGuard had no locked(). Correct — the guard replaced a bare threading.Lock that test_apps_backend_coverage_supervision.py::test_forgetting_a_pid_never_breaks_a_stop probes with .locked(), so that unchanged test raised AttributeError and the backend suite went red. Fixed: _PidfileGuard.locked() now delegates to self._thread_lock.locked() — the in-process guarantee the best-effort flock never changes. That test passes again.

Tests: test_a_client_install_result_restores_the_running_app (non-stream) and test_a_streaming_client_install_result_restores_the_running_app (SSE) assert the app is re-registered and restarted on a client-install replace; both go red without the restore call. The supervision suite passes with locked() added.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 0c99b92. This review supersedes my request-changes review of 2026-10-27 on bfaba77. All three blocking items from that review are fixed, and so are the two unpinned guards. One new red on this head is caused by the PR.

Previous points

  1. Committed registry update with a failed provenance write: fixed on the update endpoint. routes_install.py:158-201 now logs SEL outcome="failed" with the error, records a notice that names the provenance write, and returns a coded 500 (provenance_not_recorded). test_registry_update_committed_but_provenance_write_fails_keeps_the_row_revoked pins it; disabling the branch turns it red.
  2. Missing @pytest.mark.asyncio: fixed. Both tests (test_apps_routes_coverage_update_uninstall.py:576 and :1075) are marked and pass.
  3. Comment History Gate: fixed. It passes against the merge base, and the lane citations are gone from the source comments.
  4. The :459 disarm: pinned. Removing it turns test_a_superseding_stop_landing_after_the_spawn_keeps_its_restored_row red.
  5. clear_backend_notice: pinned. Replacing backend.py:385 with pass turns test_a_successful_start_after_a_refusal_clears_the_notice red.
  6. Rebase: the unrelated files from main are gone from the diff.
  7. RFC status: unchanged. First Principles still blocks on product shape, and the merge stays gated on #19387.

Blocking

  1. test/test_lenient_read_rewrite_ratchet.py::test_no_new_rewrite_starts_from_a_lenient_read fails on Backend Tests (3.12, 6) and Backend Tests (Windows) (6), and locally on this head. It flags pidfile.py::_restore_app_pid (write at 341) and pidfile.py::_unrevoke_app_pid (write at 411). Both functions are new in this PR. Please read the pidfile through a strict reader there (only a missing file reads as empty), or add both sites to _KNOWN with the reason leniency is safe.

Non-blocking

  • The committed-but-provenance-failed branches in the registry install endpoints (routes_registry.py:268 and the SSE path at :474) have no test. Replacing either condition with if False: leaves every touched suite green. Please add one test per path, matching the /update test.
  • routes.py imports dataclass, Awaitable and Callable without using them.
  • termination.py:377: the quarantine notice after a refused stop that could not persist its restore still blames an update.
  • The shard-2 XPASS(strict) in test_crew_log_fullscan_guard.py is inherited from main and is not caused by this PR.
  • GPT 6.1 and Opus 5.5 have not published a verdict for this head (GPT's comment could not be written; Opus ran out of turns). Both lanes need a re-run.

Verified

The 12 touched test files: 1008 passed. Mutations that turn a named test red: dropping the committed-provenance-failed branch in routes_install.py, dropping the record-present gate in _restore_provenance_after_failed_update, dropping the restart.py:459 disarm, and dropping the clear_backend_notice call. Disabling either registry-endpoint committed branch turns nothing red. Comment History Gate passes. The ratchet failure was reproduced locally.

chenmingwei23 and others added 2 commits October 27, 2026 12:55
Adoption bound to a port's health, never to the code the listener runs,
so an uninstall survivor that rebound the declared port could be adopted
by the next same-name install. Adopt only when the gateway's own spawn
record (leader or tree route) accounts for every owning PID; refuse and
surface a backend notice otherwise, naming the port and the user action.
Revoke the spawn row before a destructive update/uninstall and restore it
on rollback.

Rollback fix: when the update's stop forgot the tracked row before the
pre-replacement revoke ran, the revoke writes a bare pid-less tombstone;
the rollback still holds the pre-stop row, but the same-spawn check could
never match the tombstone, so the recovery row was lost and the still-
installed app's surviving backend was refused. Restore the preserved row
over a bare tombstone; a tombstone is never a real successor, and a
successor's own spawn row is still left untouched.

Notice screenshots live on the screenshots/13403 branch and are linked
from the PR description rather than committed here.
…restore #16009 tests

The pidfile lock sidecar is created owner-only (0o600) so a fresh boot leaves nothing world-readable. The port-conflict and update-not-restarted notices name the app's background service, state what the code knows (the previous version may still hold the port), and drop the unexplained disk-space step. Restore the ChatPage last-chat-hint tests and the renderChatPage entry parameter that a stale base had dropped.
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and folded in the strict-read fix (4f5ce942cd).

_restore_app_pid and _unrevoke_app_pid in pidfile.py did a read-modify-write through _read_pidfile, which turns a failed open into an empty document — so a transient read error (EACCES/EIO) could publish an empty file over every recorded row. Both now read through _read_pidfile_strict, which raises on a non-missing read error; the raise lands in each function's existing fail-closed handler (_restore_app_pid returns False, _unrevoke_app_pid is a no-op), so the whole-file rewrite is abandoned and the on-disk rows survive. No allowlist entry added — the scan is clean because neither function is lenient anymore.

Added two tests to test_apps_backend_coverage_provenance.py asserting that when the base read raises, no write is attempted and the recorded rows are preserved; both verified red when the strict read is reverted to the lenient one, green with it.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 4f5ce94. This review supersedes my request-changes review on 0c99b92. The blocker from that review is fixed, and the new commit introduces nothing I would block on.

What changed since 0c99b92

The range-diff shows the first commit unchanged (a rebase only). The second commit has three hunks:

  • pidfile.py:343 (_restore_app_pid) and pidfile.py:390 (_unrevoke_app_pid) now read the pidfile through _read_pidfile_strict instead of _read_pidfile.
  • test_apps_backend_coverage_provenance.py adds test_restore_app_pid_never_overwrites_an_unreadable_pidfile and test_unrevoke_app_pid_never_overwrites_an_unreadable_pidfile.

Previous blocker

test_lenient_read_rewrite_ratchet.py::test_no_new_rewrite_starts_from_a_lenient_read is fixed. Both whole-file rewrites now start from the strict reader, where only a missing file reads as empty. This is the remedy the ratchet asks for first. _KNOWN and the ratchet are untouched. On a read error, each function's existing except fails closed without writing: _restore_app_pid returns False, and _unrevoke_app_pid becomes a no-op.

Verified locally: the ratchet and the 12 touched test files pass (1019 passed). I also reverted each strict read to _read_pidfile() one at a time. Each revert reddens its new test (test_restore_app_pid_never_overwrites_an_unreadable_pidfile, test_unrevoke_app_pid_never_overwrites_an_unreadable_pidfile) and the ratchet.

Still open, non-blocking (from my last review)

  • The committed-but-provenance-failed branches in routes_registry.py (:268, and the SSE path at :474) still have no test. One test per path, matching the /update test, would pin them.
  • routes.py still imports dataclass, Awaitable and Callable without using them.
  • termination.py:377: the quarantine notice after a refused stop that could not persist its restore still blames an update.

Status

  • GPT 6.1 and Opus 5.5 now report no blocking findings on this head. Design and UX pass.
  • First Principles still blocks on product shape: removing externally-managed backend adoption needs an accepted RFC. That is an owner decision, and the merge stays gated on #19387.
  • CI is not finished. Backend Tests (3.12, 5-8) and Windows (8) are still queued. macOS shards 1-3 were cancelled without output and should be re-run.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

This branch now conflicts with main. #25577 moved the backend stop-and-observe step into a shared helper in teardown.py, and #24889 rewrote the uninstall quarantine/delete code in manager_uninstall.py. Both touch code this change edits. Merging also waits on the First Principles decision (#19387), so we will re-port onto main in one push once a maintainer is ready to merge. Until then the head stays at 4f5ce94.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Main rewrote the backend lifecycle since this PR, so a rebase would mean redoing it. The same fix for #13403 is re-done on current main in #35334. This PR stays open until that one is green.

@pepmach pepmach left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by txd's agent. Blocking findings only.

No correctness, security or data-loss defect found at 4f5ce94. One inline note: the backend quarantine and the post-replace re-revoke can be deleted (about 300 lines and one state file) with no loss of protection.

# next update/enable records a clean row. The pidfile's own
# fail-closed attribution is the backstop when the marker itself could
# not be written.
if is_backend_quarantined(app_name):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The backend quarantine guards nothing _adoption_provenance does not already refuse, so it can go (about 300 lines in 8 files, plus the app_backends.quarantine.json state file).

Each quarantine_backend trigger is already covered:

  • Update (_invalidate_provenance_before_restart, routes_lifecycle.py:286): it runs only after _revoke_provenance_before_replacement confirmed the revoked stamp on disk, under the same lifecycle flock. A second revoke_backend_provenance on that row returns early without writing, so this failure branch (quarantine, skipped restart, _UPDATE_NOT_RESTARTED_NOTICE, provenance_dropped / restart_gated) is not reachable in practice. If it were, the row would be revoked, absent or unreadable, and the gate refuses all three.
  • Refused stop (termination.py:387): it fires only when _restore_app_pid returned False, which means the row is not on disk or the file cannot be read. The gate then answers no spawn recorded. The one stop that passes _retry_if_serving (a withdrawn ceiling, supervision.py:340) is also refused earlier by app_execution_denied in _start_app_backend_body.
  • Stale reap (stale_reap.py:434): rows whose removal did not persist are still on disk and name this installation's own last spawn; rows of a replaced or removed installation are already revoked. So this trigger only refuses the app's own survivor, and _quarantined_listener_notice then tells the user "The previous version ... after an update" when no update happened.

Checked at this commit with real processes: after the pre-replacement revoke, a second revoke_backend_provenance makes 0 writes and the gate refuses; after a restore that did not persist, the gate returns (False, 'no spawn recorded for this app'); a retained row attributes its own spawn tree through the tree route.

Smaller version: delete _quarantined_backends through clear_backend_quarantine in pidfile.py (and its call in _record_app_pid), this gate, the three quarantine_backend calls, and _invalidate_provenance_before_restart. A committed replace then always restarts, and a survivor still on the port is refused by the revoked row.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Closing: superseded by #35334, which carries this change forward as an opt-in setting after the review rounds here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

App-backend adoption has no provenance: an uninstall survivor is adopted by the next install of the same name

6 participants