Repository navigation
fix(apps): adopt a backend only when the gateway can attribute it - #13650
chenmingwei23 wants to merge 2 commits into
Conversation
2fb7cba to
ddc7b07
Compare
Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)Design-level review of Design-Verdict: PASS cannot evaluate: missing screenshot of the new BlockersNo rendered evidence for the new backend-refused notice Notes
[DESIGN-REVIEWED] 4f5ce94 Description read[DESCRIPTION-READ] 123c9886d29dca483271799b60eaaf4eaf122b228a5966d78efe83cde5ca358a That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK product-shape change without accepted RFC: the PR stops adopting externally-managed backends, and the RFC that records that removal ( Notes
What this change shipsInventory (10 items) — 5 justifiedIntent: stop a process that survived an uninstall from being managed as a reinstalled app's backend — FIX.
Blockersproduct-shape change without accepted RFC: removal of externally-managed backend adoption — undeclared [FIRST-PRINCIPLES-REVIEWED] 4f5ce94 Description read[DESCRIPTION-READ] 78b25ee775298c1411b02f41395d3e6e5c5eca6cd142bcd1d9307ff108359f21 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
ddc7b07 to
e259251
Compare
e259251 to
0c55404
Compare
0c55404 to
00a19af
Compare
00a19af to
facfdf7
Compare
Re: review 2026-10-07 (current head
|
|
Re: First Principles review Thanks for the premise-level read. Point by point: Blocker (item 1) — removal of externally-managed backend adoption. This is a product-shape decision, so I'm not deciding it in the PR. Two ways forward, with their costs:
I'll take whichever you prefer; I'm not widening the goal on my own. Item 5 — the quarantine mechanism. Keeping it. It isn't dead code: Item 6 — Item 10 — formatting/baseline. The reformatting is confined to files this change already edits, so it can't be split out into a separate change. |
|
We're going with option A: keep the fail-closed behaviour, so a listener the gateway never recorded is refused rather than adopted. An operator opt-in to allow such a backend can be a separate follow-up issue if it's ever wanted; it is not part of this change. Rebased onto the current Local gate is green on the rebased head: The RFC on the base branch is still |
|
Fixed the registry post-commit case GPT 6.1 flagged (routes.py:1199) in
Regression test Full local gate green on |
|
Fixed the uninstall case GPT 6.1 flagged (manager.py:2230) in
Regression test Full local gate green on |
|
Two open points remain for maintainer review before this can be marked ready. Both are outside the fixes already landed on this branch, so I am listing them here rather than changing more code. 1. Restart-cancellation edge in 2. First Principles gate. The code lanes on the current head are green and the two correctness fixes above are covered by regression tests. These two points are the only remaining blockers. |
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at c014518. My previous blocking item is fixed; one code defect remains.
Previous blocking item: fixed
A refused adoption now reaches the user. _start_app_backend_body returns AdoptionRefused with a notice for both the quarantined (src/kiro_crew/apps/backend.py:584-595) and unattributed (:607-625) cases, _start_app_backend records it (:368-374), routes._attach_backend_notice adds it to the list, detail, install, update, enable and registry-install responses (routes.py:392-402), and AppDetailPage renders it with the agent hand-off. Removing the record call turns test_the_refusal_reaches_the_caller_as_a_backend_notice red; removing the attach turns test_get_app_carries_the_backend_notice and test_an_update_that_skips_its_restart_says_so red. The quarantine refusal test, the pre-wrap log label (backend.py:1300) and the completed_without_restart notice from my follow-ups are also done.
Blocking
src/kiro_crew/apps/backend_runtime/restart.py:448-477. The two post-spawn cancellation returns (:450 and :459) stop the replacement and return with leader_row_superseded still True, unlike the path at :472, which disarms it. The finally then calls _forget_exited_leader_row and deletes the row that the concurrent stop has just restored because a detached worker is still listening. On the next enable that worker is refused as unattributed, and the user is told to stop the app's own backend. You described the same path in your last comment, and the GPT lane flags it. Please disarm the flag on those two returns (or gate the forget on the row still being the one this restart armed) and add a test for a disable that lands after the spawn while a detached worker holds the port.
Non-blocking
- Removing the clear_backend_notice at the start of each attempt turns no test red; add a case where a refused start is followed by a successful one and the notice is gone.
- docs/request-for-change/README.md:65 still lists rfc-app-backend-adoption-provenance.md as draft. The First Principles block on the removal of external-supervisor adoption needs that accepted or a maintainer decision.
- The rebase is not mechanical: main split apps/manager.py into a facade with owner modules (#21940) and moved code out of session_pid.py (#21969, #22022), so the pidfile and revoke hunks must move into those owners.
Verified
The seven related backend test files: 1419 passed locally on this head.
|
This branch can no longer be rebased onto the current
The code this PR's changes target now lives in those new modules, so rebasing is no longer a mechanical conflict resolution — each product change has to be re-ported into The branch is left unchanged at |
|
Re: review 2026-10-20 ( Blocking: fixed
A restart that is cancelled after its spawn (a later generation won, or the app was disabled/shut down in the meantime) is the losing party: it does not own the leader row, and the operation that won may already have restored that row for a detached worker still holding the port. Leaving the flag armed let the Added Non-blocking
|
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at bfaba77. This review supersedes my request-changes review of 2026-10-20 on c014518. The restart-row blocker is fixed. Three items block this head.
Previous points
- Post-spawn cancellation returns: fixed. src/kiro_crew/apps/backend_runtime/restart.py:459 and :472 both disarm leader_row_superseded. Removing the :472 disarm turns test_a_disable_landing_after_the_spawn_keeps_a_concurrent_stops_restored_row red. Removing the :459 disarm (the generation-changed return) turns nothing red. Please add a case where a superseding stop lands after the spawn.
- clear_backend_notice coverage: not covered yet. Replacing the call at backend.py:385 with
passleaves the app backend and routes suites green, including the two tests named in your reply. A refused start followed by a successful one still needs a test that asserts the notice is gone. - RFC status: understood that this is a maintainer decision. The merge stays gated on #19387.
- Rebase: the hunks are in the right owner modules now. The branch also carries main's later versions of unrelated files (test_lenient_read_rewrite_ratchet.py, subagent_manager/monitoring.py, the settings panels, crew-companion reminders, app_storage.py). They are net-zero against main today, but a real rebase onto current main would drop them from the diff.
Blocking
- A committed registry update whose provenance write failed is reported as success. routes_install.py:148-157 sets reg_outcome_kind = "committed_provenance_failed", but only "revoke_failed" and "install_failed" are handled. The request returns 200 with ok: false, SEL records completed_without_restart, and the dashboard shows the success toast with a notice that blames the old version holding the port. Keeping the row revoked and not restarting is right. Please also log SEL outcome="failed" with the error, return a coded non-2xx the UI surfaces, and record a notice that names the failed provenance write, with a test.
- test_provenance_invalidation_reports_confirmed_vs_unconfirmed and test_provenance_deleted_only_at_the_commit_boundary (test/test_apps_routes_coverage_update_uninstall.py:576 and :1062) lost @pytest.mark.asyncio when they moved. Both fail locally with "async def functions are not natively supported". CI has not run them yet because the backend jobs were skipped.
- The Comment History Gate is red: manager_uninstall.py:517, test_app_manager.py:1654 and test_apps_backend_coverage.py:2459 and :2490. Comments that cite review lanes ("GPT F1:" in termination.py, "GPT 6.1:" and "F2 fence" in routes_install.py) should state current behavior too.
Non-blocking
- termination.py:387 quarantines a name when a refused stop could not persist its row restore. The next start then shows the "previous version ... after an update" notice, which is not the cause in this case.
Verified
App backend and routes test files: 985 passed, 2 failed (blocking 2). Mutations: removing the :472 disarm and removing the ordinary adopted-stop restore at termination.py:755 each turn a named test red. Removing the :459 disarm and removing clear_backend_notice turn nothing red. The uninstall and update ordering against the new row restore was checked from source.
|
Re: Opus 5.5 Review — the registry-update committed-provenance-failed path. Fixed in
The response stays |
|
Re: review 2026-10-27 ( Blocking1. Committed registry update whose provenance write failed reported as success. Fixed in 2. Moved async tests lost 3. Comment History Gate red + lane-citing comments. Fixed in Previous points1. A superseding stop that lands after the spawn. Added 2. Refused start followed by a successful one clears the notice. Added 3. RFC status. Agreed — maintainer decision, merge stays gated on #19387. Noted in the PR body's CI status section. 4. Rebase. Done — rebased cleanly onto current Non-blocking
|
|
Re: GPT 6.1 Review — the failed-rollback provenance restore. Fixed in
Added Also addressed Static Ratchet Gate in the same push: the |
|
Re: GPT 6.1 Review — the registry Update path leaving the previous backend attributable (pidfile.py:782). Fixed in The provenance stop → capture → revoke-before → replace → invalidate-after transaction is factored into one shared helper, Nothing is wrapped twice: the shared helper is the single mechanism, each caller invokes it once for the Tests: |
|
Re: GPT 6.1 — "Restore the running app after a failed registry replacement" (routes_registry.py:210). Fixed in Correct finding. The shared transaction stops the backend and deregisters the app's resources before it clones, but on a rollback both registry handlers returned the error without putting the app back — leaving the unchanged app enabled but unavailable, its agents, skills, crons and MCP servers deregistered. Both
Tests (go red without the restore call): This head is also rebased onto current origin/main (the Backend Lint failure was a base-side undefined name since fixed upstream). |
|
Re: GPT 6.1 (routes_registry.py:211) and Opus 5.5 (routes_registry.py:253 + pidfile.py:117) on 1. Client-install replace left the old app stopped/deregistered. Correct, both reviewers. When an already-installed app's manifest resolves to client-only install on an unsupported host, 2. Tests: |
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at 0c99b92. This review supersedes my request-changes review of 2026-10-27 on bfaba77. All three blocking items from that review are fixed, and so are the two unpinned guards. One new red on this head is caused by the PR.
Previous points
- Committed registry update with a failed provenance write: fixed on the update endpoint. routes_install.py:158-201 now logs SEL outcome="failed" with the error, records a notice that names the provenance write, and returns a coded 500 (
provenance_not_recorded). test_registry_update_committed_but_provenance_write_fails_keeps_the_row_revoked pins it; disabling the branch turns it red. - Missing @pytest.mark.asyncio: fixed. Both tests (test_apps_routes_coverage_update_uninstall.py:576 and :1075) are marked and pass.
- Comment History Gate: fixed. It passes against the merge base, and the lane citations are gone from the source comments.
- The :459 disarm: pinned. Removing it turns test_a_superseding_stop_landing_after_the_spawn_keeps_its_restored_row red.
- clear_backend_notice: pinned. Replacing backend.py:385 with
passturns test_a_successful_start_after_a_refusal_clears_the_notice red. - Rebase: the unrelated files from main are gone from the diff.
- RFC status: unchanged. First Principles still blocks on product shape, and the merge stays gated on #19387.
Blocking
test/test_lenient_read_rewrite_ratchet.py::test_no_new_rewrite_starts_from_a_lenient_readfails on Backend Tests (3.12, 6) and Backend Tests (Windows) (6), and locally on this head. It flagspidfile.py::_restore_app_pid(write at 341) andpidfile.py::_unrevoke_app_pid(write at 411). Both functions are new in this PR. Please read the pidfile through a strict reader there (only a missing file reads as empty), or add both sites to_KNOWNwith the reason leniency is safe.
Non-blocking
- The committed-but-provenance-failed branches in the registry install endpoints (routes_registry.py:268 and the SSE path at :474) have no test. Replacing either condition with
if False:leaves every touched suite green. Please add one test per path, matching the /update test. - routes.py imports
dataclass,AwaitableandCallablewithout using them. - termination.py:377: the quarantine notice after a refused stop that could not persist its restore still blames an update.
- The shard-2 XPASS(strict) in test_crew_log_fullscan_guard.py is inherited from main and is not caused by this PR.
- GPT 6.1 and Opus 5.5 have not published a verdict for this head (GPT's comment could not be written; Opus ran out of turns). Both lanes need a re-run.
Verified
The 12 touched test files: 1008 passed. Mutations that turn a named test red: dropping the committed-provenance-failed branch in routes_install.py, dropping the record-present gate in _restore_provenance_after_failed_update, dropping the restart.py:459 disarm, and dropping the clear_backend_notice call. Disabling either registry-endpoint committed branch turns nothing red. Comment History Gate passes. The ratchet failure was reproduced locally.
Adoption bound to a port's health, never to the code the listener runs, so an uninstall survivor that rebound the declared port could be adopted by the next same-name install. Adopt only when the gateway's own spawn record (leader or tree route) accounts for every owning PID; refuse and surface a backend notice otherwise, naming the port and the user action. Revoke the spawn row before a destructive update/uninstall and restore it on rollback. Rollback fix: when the update's stop forgot the tracked row before the pre-replacement revoke ran, the revoke writes a bare pid-less tombstone; the rollback still holds the pre-stop row, but the same-spawn check could never match the tombstone, so the recovery row was lost and the still- installed app's surviving backend was refused. Restore the preserved row over a bare tombstone; a tombstone is never a real successor, and a successor's own spawn row is still left untouched. Notice screenshots live on the screenshots/13403 branch and are linked from the PR description rather than committed here.
…restore #16009 tests The pidfile lock sidecar is created owner-only (0o600) so a fresh boot leaves nothing world-readable. The port-conflict and update-not-restarted notices name the app's background service, state what the code knows (the previous version may still hold the port), and drop the unexplained disk-space step. Restore the ChatPage last-chat-hint tests and the renderChatPage entry parameter that a stale base had dropped.
|
Rebased onto current main and folded in the strict-read fix (
Added two tests to |
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at 4f5ce94. This review supersedes my request-changes review on 0c99b92. The blocker from that review is fixed, and the new commit introduces nothing I would block on.
What changed since 0c99b92
The range-diff shows the first commit unchanged (a rebase only). The second commit has three hunks:
- pidfile.py:343 (
_restore_app_pid) and pidfile.py:390 (_unrevoke_app_pid) now read the pidfile through_read_pidfile_strictinstead of_read_pidfile. - test_apps_backend_coverage_provenance.py adds
test_restore_app_pid_never_overwrites_an_unreadable_pidfileandtest_unrevoke_app_pid_never_overwrites_an_unreadable_pidfile.
Previous blocker
test_lenient_read_rewrite_ratchet.py::test_no_new_rewrite_starts_from_a_lenient_read is fixed. Both whole-file rewrites now start from the strict reader, where only a missing file reads as empty. This is the remedy the ratchet asks for first. _KNOWN and the ratchet are untouched. On a read error, each function's existing except fails closed without writing: _restore_app_pid returns False, and _unrevoke_app_pid becomes a no-op.
Verified locally: the ratchet and the 12 touched test files pass (1019 passed). I also reverted each strict read to _read_pidfile() one at a time. Each revert reddens its new test (test_restore_app_pid_never_overwrites_an_unreadable_pidfile, test_unrevoke_app_pid_never_overwrites_an_unreadable_pidfile) and the ratchet.
Still open, non-blocking (from my last review)
- The committed-but-provenance-failed branches in routes_registry.py (:268, and the SSE path at :474) still have no test. One test per path, matching the /update test, would pin them.
- routes.py still imports
dataclass,AwaitableandCallablewithout using them. - termination.py:377: the quarantine notice after a refused stop that could not persist its restore still blames an update.
Status
- GPT 6.1 and Opus 5.5 now report no blocking findings on this head. Design and UX pass.
- First Principles still blocks on product shape: removing externally-managed backend adoption needs an accepted RFC. That is an owner decision, and the merge stays gated on #19387.
- CI is not finished. Backend Tests (3.12, 5-8) and Windows (8) are still queued. macOS shards 1-3 were cancelled without output and should be re-run.
|
This branch now conflicts with main. #25577 moved the backend stop-and-observe step into a shared helper in teardown.py, and #24889 rewrote the uninstall quarantine/delete code in manager_uninstall.py. Both touch code this change edits. Merging also waits on the First Principles decision (#19387), so we will re-port onto main in one push once a maintainer is ready to merge. Until then the head stays at 4f5ce94. |
pepmach
left a comment
There was a problem hiding this comment.
Review by txd's agent. Blocking findings only.
No correctness, security or data-loss defect found at 4f5ce94. One inline note: the backend quarantine and the post-replace re-revoke can be deleted (about 300 lines and one state file) with no loss of protection.
| # next update/enable records a clean row. The pidfile's own | ||
| # fail-closed attribution is the backstop when the marker itself could | ||
| # not be written. | ||
| if is_backend_quarantined(app_name): |
There was a problem hiding this comment.
The backend quarantine guards nothing _adoption_provenance does not already refuse, so it can go (about 300 lines in 8 files, plus the app_backends.quarantine.json state file).
Each quarantine_backend trigger is already covered:
- Update (
_invalidate_provenance_before_restart,routes_lifecycle.py:286): it runs only after_revoke_provenance_before_replacementconfirmed therevokedstamp on disk, under the same lifecycle flock. A secondrevoke_backend_provenanceon that row returns early without writing, so this failure branch (quarantine, skipped restart,_UPDATE_NOT_RESTARTED_NOTICE,provenance_dropped/restart_gated) is not reachable in practice. If it were, the row would be revoked, absent or unreadable, and the gate refuses all three. - Refused stop (
termination.py:387): it fires only when_restore_app_pidreturned False, which means the row is not on disk or the file cannot be read. The gate then answersno spawn recorded. The one stop that passes_retry_if_serving(a withdrawn ceiling,supervision.py:340) is also refused earlier byapp_execution_deniedin_start_app_backend_body. - Stale reap (
stale_reap.py:434): rows whose removal did not persist are still on disk and name this installation's own last spawn; rows of a replaced or removed installation are alreadyrevoked. So this trigger only refuses the app's own survivor, and_quarantined_listener_noticethen tells the user "The previous version ... after an update" when no update happened.
Checked at this commit with real processes: after the pre-replacement revoke, a second revoke_backend_provenance makes 0 writes and the gate refuses; after a restore that did not persist, the gate returns (False, 'no spawn recorded for this app'); a retained row attributes its own spawn tree through the tree route.
Smaller version: delete _quarantined_backends through clear_backend_quarantine in pidfile.py (and its call in _record_app_pid), this gate, the three quarantine_backend calls, and _invalidate_provenance_before_restart. A committed replace then always restarts, and a survivor still on the port is refused by the revoked row.
|
Closing: superseded by #35334, which carries this change forward as an opt-in setting after the review rounds here. |
Problem / Motivation
Goal: adopt an app backend only when the gateway can attribute the listening process to the spawn it recorded for that app — closing a misattribution where a survivor process rebinding a declared port is managed as another app's backend.
An app backend is adopted on two facts, and neither one names the process that answered or the code it runs: the port the manifest declares is occupied, and something on it answers the declared health path.
_probe_adoption_healthasks the question,_capture_adopted_ownersrecords the owning PIDs, and the instance is then registered under that app's name.So a backend that outlives its app's uninstall and rebinds that port is adopted by the next install that happens to use the same app name. Reaching it needs no misconfiguration: an app whose manifest declares a fixed port, plus any detached child that survives the uninstall, is enough. The uninstall port probe is a single instant, so a survivor that has closed its listener at probe time and rebinds afterwards is reported as stopped.
The same gap has a second door.
_rebind_adopted_ownersre-captures an adopted backend's owners after it stops answering, and that recovery path writes the new owner set onto the record and promotes it. An unrelated listener answering the declared health path in the dead backend's place is therefore managed as the app's backend without ever being attributed.Why it matters
The adopted process is the one thing the install did not put there. It keeps its own code, its own credentials and its own file handles, while the gateway treats it as that app's backend: it is addressed as the app, its health is reported as the app's, and stop signals its PIDs. Adopted instances are excluded from the startup stale-reap by design, so a gateway restart re-probes and re-adopts rather than clearing it — the misattribution renews itself for as long as the survivor lives.
Not a goal
KIROCREW_HOMEand means to forge its way in can write a self-naming row intoapp_backends.pids.json; closing that (a born-masked record, or moving the reader behind the gateway) is a separate change. It is documented below as a knowingly-carried residual, the same class astrust/sel_hmac.key, not absorbed into this fix.apps/teardown.pyand is its own change.What changed (motivation → approach → change)
Symptom: a process the install did not place serves as the app's backend. Cause: adoption never asks whose process it is, because a health answer is the whole test. Fix: manage only owners the gateway can attribute to the spawn it recorded for that app, at both sites that write the owner record, and keep that record available to every retry that has to ask again.
The record already exists and needs no new file. Every backend the gateway spawns is written to the app pidfile as
pid+start_time+ a per-spawnspawn_instance, and that token also travels on the child's environment, where the process cannot rewrite it._adoption_provenancereads the record back and attributes each captured owner by one of two routes; every captured owner must clear a route:spawn_instance. The whole spawn tree inherits it, so this route reaches a pre-fork worker or a detached child that still holds the port after its leader has exited. It reads/proc/<pid>/environ, which exists on Linux alone.Every captured owner enters the managed set and is signalled at stop, so one match cannot speak for the set: a same-UID
SO_REUSEPORTco-binder lands in the same dispatch tier as the real backend, and attributing the whole set from one match would hand stop an unrelated process to terminate. The gate fails CLOSED — a missing row, an unreadable row, a recycled pid, an empty owner set, and a host that cannot read a token all refuse.The record is not fenced, and the forged-row case is an open residual.
app_backends.pids.jsonlives in the crew data home with the crew-home root left read-write, and a survivor inheritsKIROCREW_HOMEfrom the spawn that started it — so a process that meant to forge its way in could write its own inheritedspawn_instanceinto a row naming itself, and provenance would then attribute it. This PR does not close that: the record is provenance the gateway wrote, carried without an OS fence, exactly asAGENTS.mdrecordstrust/sel_hmac.keyas a knowingly-carried VISIBLE residual with no OS fence. It is documented here as the same class of carried residual rather than hidden, and closing it properly (a born-masked record, or moving the reader behind the gateway) is a separate change, not something this fix reaches for. What this fix does close is the accidental shape — the overwhelmingly common survivor of an uninstall — because a clean uninstall drops the app's row, so the survivor meets an empty record and no route attributes it.Pidfile writes now take a cross-process lock, not only a thread lock. The app pidfile is read and written by both the gateway and the offline CLI, so making the record load-bearing meant two processes could otherwise interleave a read-modify-rewrite and lose a row to the later atomic rename.
_PidfileGuardwraps every pidfile mutation in the existing in-process thread lock AND an advisory interprocessflockon a sidecarapp_backends.pids.lock(owner-only0o600, created on first use), so the CLI and the gateway serialise against each other. The flock is best-effort and never weakens the in-process guarantee: if it cannot be taken (an unusual host, a filesystem without the lock) the code falls back to the thread lock alone, exactly as before. It is held only while the thread lock is, and the guarded code does not re-enter it.A refused adoption is shown to the user, not only logged. Both refusal sites in
_start_app_backend_body(quarantined, unattributed) returnAdoptionRefused(port, reason, notice)instead ofNone._start_app_backendrecords the notice for that app and still returnsNoneto every caller, the boot fan-out included; the next start attempt clears it. The notice is carried asbackend_noticeonGET /api/apps,GET /api/apps/{name}and the enable, install and update responses (a separate key fromnotice, which update already uses for the machine-readsession_approval_reconsenttoken), and an update that skips its restart (completed_without_restart) records one too. The app detail page shows it for an enabled app through the sharedErrorNotice, with the agent hand-off, in plain words that name the port and the step to take (screenshots below). This is how a user meets the upgrade case: a backendmainhad adopted is refused on first boot, and the notice is waiting on the app's page.The spawn log at the adoption site logs
_command_log_label(cmd)— a fixed, non-sensitive executable-class label (git,python,node,shell, …) — rather than the joined argv. This is a security fix, not a cosmetic change: the argv routinely carries user-controlled paths, URLs and occasionally transport capabilities, and logging it clear-text is a secret-in-logs exposure (CodeQLpy/clear-text-logging-sensitive-data, high). An earlier attempt to drop this reverted the log to the joined argv and reopened exactly that high-severity alert on this line, so the sanitized label is kept.Black and the baseline: this change edits
backend.py,manager.py,apps/teardown.pyand the touched test files (test_app_backend.py,test_app_backend_stale_reap.py,test_app_manager.py,test_apps_routes_coverage.py), and the edited files come out black-clean. The black gate requires that a file which has become clean be removed from.github/black-baseline.txt, so these files are pruned from it. This is not optional reformatting of untouched code: each listed file is edited by this fix for a functional reason, and the gate reds if an edited-and-now-clean file is left on the exemption list._rebind_adopted_ownersre-asks the same question on the set it just captured, rather than inheriting the verdict of the adoption that installed the record. On a refusal it leaves the recorded owners untouched and the instance unhealthy, so stop still aims where it did.Making the record load-bearing exposed an asymmetry in
stop_app_backendthat had been harmless while nothing read it. The stop drops the app's row inside its lifecycle transition, before it signals anything, and it can then refuse and restore tracking so the caller retries. With provenance reading that row, the retry's re-bind would refuse on "no spawn recorded" every time, so the ceiling could never converge on a listener that keeps serving, and across a restart re-adoption would be refused too. The removal is therefore reversible:_forget_app_pidand_forget_app_pid_ifhand back the row they removed, refusal paths go through one helper that undoes exactly what the transition did, and the restore issetdefaultrather than an overwrite so a spawn that recorded its own identity in the meantime keeps it. A stop that succeeds still drops the row, and an untracked stop leaves the row intact so a later start can still attribute a survivor against it. Uninstall is the exception: an untracked uninstall drops the row anyway, because the app is being removed and its row must not let a future install of the same name adopt whatever rebinds the port.The restart path deletes the row early for the same reason and is deferred rather than made reversible, because there it is the respawn that must read the record. When a backend's leader exits while a pre-fork worker or a detached child of that same app keeps holding the declared port, the respawn's adopt branch is exactly the case the
treeroute covers. The drop now happens once the attempt has decided, through onefinallycovering every exit rather than the mid-loop call, and it is skipped when what ends up tracked is an adopted instance resting on that row. A fresh spawn's own row is never the one removed, because the drop stays conditional on the exited leader's pid and start instant.An app UPDATE replaces the code behind the name while reusing the port, so the pre-update row still vouches for whatever rebinds it and the post-update restart would otherwise adopt a survivor running the OLD code as the new version.
revoke_backend_provenanceinvalidates the row at the post-update, pre-restart seam in both update branches (registry and local source), keeping a durable revoked tombstone rather than deleting the row outright —_adoption_provenancerefuses any revoked row, so the refusal survives a restart. It is retained across the stop for update ROLLBACK — a failed update returns through_restore_app_after_failed_updatebefore the invalidation runs, so the old backend stays attributable. The invalidation is CONFIRMED:revoke_backend_provenanceraisesPidfileRevokeFailedon an unconfirmed write (ENOSPC/EDQUOT leaves the in-memory row updated while the on-disk copy stays). The update seam then does NOT restart — a restart would adopt the stale-row survivor as the new version — and reports the backend deliberately not started (completed_without_restart); the pre-replacement revoke is checked BEFOREupdate_appruns, so an unconfirmed revoke refuses the update with a 400 and never commits new code over a row that still vouches for an old-code survivor. The best-effort stop/record paths stay lenient (they restore the row on refusal) and never raise.The audit records the verdict reached instead of the verdict attempted: the
app_backend_adoptentry moves after the decision and carriesrefused_unattributedwith the reason, from both sites.admitted_builtin=Falseat the adoption site is unchanged and its comment states that provenance names the owners as a spawn this gateway recorded, but not that they are still executing the shipped code the manifest declares, because a spawn outlives an in-place rewrite of the files it started from.Backwards compatibility
Breaking (acceptance pending in #19387, which records the decision as accepted in
docs/request-for-change/rfc-app-backend-adoption-provenance.md): a backend that is not the gateway's own recorded spawn is no longer adopted. After upgrading, a backendmainhad adopted (externally supervised, or simply never recorded) is refused on first boot; on macOS and Windows a pre-fork worker or detached child that still holds the port after its leader exits is refused as well, because thetreeroute is Linux-only. In every case the user gets the backend notice on the app's page rather than a silent failure — pinned bytest_the_refusal_reaches_the_caller_as_a_backend_notice,test_get_app_carries_the_backend_noticeand theAppDetailPage — backend noticetests. This PR carries no RFC change; the RFC on the base branch is stilldraft, so the First Principles lane needs a maintainer to accept it (merge #19387 non-draft) or to post the First Principles override before it can go green.Otherwise compatible: a hand-started backend the gateway never spawned meets an empty record and is not adopted, which is the behaviour that closes the headline defect. The provenance gate is a no-op where nothing was recorded, and the pinned residual cases self-heal forward the moment the next gateway-spawned backend records a token. No accepted input is rejected, renamed or removed.
Tests
test/test_apps_backend_coverage.py:TestAdoptionProvenancefor the verdict matrix,TestAdoptionProvenanceAtTheSpawnBodyfor the adopt/refuse/audit/free-port paths through_start_app_backend_body,TestRebindAttributesTheNewOwnersfor the recovery path (it judges the set it captured, a refusal leaves the recorded owners untouched, and it is audited),TestARefusedStopRestoresTheProvenanceRecordfor the stop (both refusal shapes, a successful stop still dropping the row, an untracked stop leaving the row intact, an untracked UNINSTALL dropping the row, a newer spawn's row surviving the restore, andrevoke_backend_provenancestamping a durable revoked tombstone that_adoption_provenancethen refuses), andTestRestartKeepsTheSpawnRowUntilTheRespawnDecidesfor the restart ordering (the row is still readable when the respawn asks, an adopted replacement keeps the row it was attributed by, a fresh spawn's own row survives, and a respawn producing nothing / an activation denial / a dropped record each reach the right answer).Existing cases that adopted or re-bound with no recorded spawn now get one in their fixture —
TestAdoptExistingInstanceandTestAdoptedRecoveryRebindsOwnershipintest/test_app_backend.py— so they keep testing what they were written for. Re-homed onto main'sbackend_runtime/*module split during the rebase and re-run: black (baselined gate), isort, flake8, mypy and the comment-history gate all clean.Manual verification
The one UI surface is the backend notice on the app detail page. Captured from the real
AppDetailPageand CSS of this branch, with only the gateway reads replaced by example data (getAppreturning the exact sentences the gateway writes), on loopback with Playwright. Images are not committed to this PR branch; they live on thescreenshots/13403branch and are linked below.Before (no notice; what
mainshows for an app whose backend was refused):After — port held by another program:
After — the longest message, an update that skipped its restart:
CI status
First Principles Review holds (BLOCK) on the capability change — adopting a backend the gateway did not spawn is no longer offered — because the record covering it,
docs/request-for-change/rfc-app-backend-adoption-provenance.md, is stilldraft. That is a maintainer/product decision, not a code defect: the merge stays gated on the RFC (#19387) being accepted or a maintainer override. It is left as-is deliberately.Related Issues
Fixes #13403
Pattern harvest
Rule candidate: review-prompt
Pattern: a trust or ownership decision keyed on an observable that names no principal — a port, a filename, a health answer — while the code already keeps a record of the principal it means. At every site that grants standing to something this call did not create, ask which recorded identity attributes it, whether the answer must hold for every member of the set that inherits the standing, what the verdict is when the record is absent or unreadable, which recovery path re-writes that standing without re-asking, and — once a record becomes load-bearing — which path deletes it before a failure that has to consult it again.
CI note