Skip to content

fix: wait out the port release before reporting a backend alive - #30595

Merged
bolichen97 merged 1 commit into
mainfrom
fix/adoption-teardown-one-shot-port-probe
Nov 6, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/adoption-teardown-one-shot-port-probe

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Disabling, un-trusting or uninstalling an app whose backend runs in the sandbox no longer reports a false "backend still running ... not ours to stop" warning when the backend did stop.

Found by the #26967 former built-in smoke run and reproduced on a plain harness gateway with the wheel's file-explorer (no adoption involved): 7 of 11 enable then disable cycles answered 200 with "warnings": ["backend still running on port 9103 — the gateway stopped every process it was tracking, so this one is not ours to stop"], while a TCP connect from the client right after each response found the port already free.

Why it matters

The warning tells the operator a process the gateway cannot stop is still running, so they go hunting for a leaked server that does not exist. On trust withdrawal the same observation is a teardown failure, so DELETE /api/security/trusted-apps/<app> answers 409 teardown_incomplete and leaves the grant in place for a backend that had in fact exited. This hits every sandboxed app backend, built-in or installed from a repository, on the disable, revoke and uninstall paths.

Not a goal

  • Changing how stop_app_backend signals or waits (the kill path in backend_runtime/termination.py), or the ceiling sweep's own post-stop health checks.
  • Reporting less: a listener that is still bound after the grace is reported exactly as before, and trust withdrawal still refuses on it.
  • Any change to sandbox scope, grants, approvals or what counts as gateway-owned port evidence.

What changed (motivation → approach → change)

Symptom → cause: teardown.stop_backend_and_observe probed the port once, right after stop_app_backend returned. That stop signals the backend's process group but waits only on the root it spawned. For a sandboxed backend the root is the sandbox wrapper, which exits before the server inside it has closed its listening socket, so the single probe often caught a socket that was about to close.

Change: when the first probe finds the port listening, stop_backend_and_observe probes it again every 50 ms for up to 2 s (PORT_RELEASE_GRACE_SECONDS, PORT_RELEASE_POLL_SECONDS). It returns the port only if it is still listening when that time runs out. A port that is already free costs one probe and no wait, the same as before. Disable, trust withdrawal and uninstall all go through this one function, so all three are fixed together. The same port hint as before (or the manifest fallback) is used on every probe. The uninstall step in app-kit-platform.md now describes the grace.

Release needs evidence: once a probe has seen the port listening, a later probe that misses it ends the wait only when the new teardown.port_released confirms the port is free. The probe also answers "not listening" on a connect timeout, and a listener that stops accepting turns repeated probes into timeouts once its backlog fills, so a miss alone is not a release. port_released counts a refused connect, or, when the connect times out (a full backlog, or Windows, which reports a refused loopback connect only after retries), a port->PID lookup that completes with no LISTEN socket. A lookup that cannot complete keeps the port reported.

Security reading: the evidence and the decision stay the same, so no trust is widened. Each probe looks at the same gateway-recorded port, and a listener that outlives the grace still lands in failures, which trust withdrawal refuses on. Enforcement under a withdrawn ceiling is still the supervision sweep's job.

Backwards compatibility

Compatible: the return shape and every message are unchanged; the only new behaviour is up to 2 s of extra probing, and only when a port is still listening after the stop.

Removes nothing: no path, key, record or check is removed or tightened; a surviving listener is still reported and still refuses trust withdrawal.

Tests

  • test/test_app_teardown_port_release_grace.py (new) uses an injected ManualClock on the teardown module, so nothing sleeps on the wall clock, and a stand-in for port_released. It covers: a port that frees during the grace is not reported, the hint is kept on every re-probe, and the release is confirmed once on that port; a probe that loses sight of the listener without a confirmed release still reports it after the whole grace; a listener that outlives the grace is still reported; a free port gets one probe, no sleep and no confirmation; the manifest fallback gets the same grace; and teardown_app_runtime for both disable and trust withdrawal. Withdrawal does not fail on a port that frees, and it still fails with backend still running on port 9137 on one that never frees and on one the probe stops seeing unconfirmed.
  • TestPortReleased in the same file: an accepting loopback listener is not released; a port bound with no listener is (bind 0, no literal port); and on a connect timeout the answer follows the PID lookup: a LISTEN socket keeps it, a completed empty lookup releases it, a failed lookup keeps it.
  • test_trusted_apps_api.py and test_apps_uninstall_backend_stop.py: the tests whose probe always reports a live port now run the grace on a ManualClock. Without it each would take 2 real seconds. test_revoke_probes_the_port_even_when_the_stop_reported_success now asserts the probe ran, and only for this app, instead of exactly once, because re-probing is the new contract. What it guards (the probe still runs when the stop returns True) is unchanged.
  • Determinism: the grace test file passed 20/20 at -n0, 10/10 at -n 4 --dist loadgroup alongside its two neighbour files, and in a shuffled node-id order. test_flake_pattern_ratchet.py passes. The 543 tests in the three changed test files plus test_app_backend_composition_contract.py pass.
  • Mutation: restoring the re-probe loop to take any probe miss as the release (keeping port_released defined) turns 4 tests red, the new unconfirmed-listener test and both trust-withdrawal cases among them. prove.py cannot express that mutation: reverting the loop hunk also deletes port_released, which the tests patch, so the run ends inconclusive.

Proof: not provable -- other: prove.py reverting the loop hunk also deletes port_released, which the tests patch, so its run is inconclusive; the hand mutation above turns 4 tests red

Manual verification

End to end on an isolated harness gateway (spawn_feature_gateway("minimal"), throwaway KIROCREW_HOME, loopback only, stub ACP), driving POST /api/apps/file-explorer/enable then POST /api/apps/file-explorer/disable in a loop:

  • origin/main @ 16ff876: 3/5 and then 4/6 disables carried the "not ours to stop" warning. In every cycle a client TCP connect right after the response found the port already free.
  • This branch: 0/8 disables carried the warning, and every port was free right after the response.

Not run locally: semgrep (not installed on this host; CI's SAST job runs the same rules). The model-pinned local review lanes were not dispatched from this session either.

Related Issues

Part of #26967

Sibling sweep

Swept: git grep -nE "_port_is_listening(|unstopped_backend_port(|_health_probe(ap.port" -- src/ and git grep -n "stop_app_backend(" -- src/kiro_crew
Sibling: src/kiro_crew/apps/teardown.py:175 -- changed here
Sibling: src/kiro_crew/apps/routes_lifecycle.py:190 -- not affected -- delegates to teardown.stop_backend_and_observe, so the uninstall path gets the grace and the confirmation
Sibling: src/kiro_crew/apps/backend_runtime/termination.py:376 -- not affected -- an HTTP health GET that a server already shutting down on SIGTERM does not answer, unlike a bare TCP connect the kernel backlog accepts; a refusal there also re-sweeps on the next interval
Sibling: src/kiro_crew/apps/backend_runtime/ports.py:115 -- not affected -- spawn survival check while the process is starting, not a probe after a stop; the post-stop first probe at :404 in the same file is unchanged, its None documented as "nothing observed", and only the re-probes this PR adds can fill a backlog

Pattern harvest

Rule candidate: review-prompt
Pattern: liveness is judged from one probe taken right after a stop that waited only on the root process, while the process group it signalled may still be exiting

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 5, 2026 17:27
@bolichen97
bolichen97 requested a review from mbriones98 November 5, 2026 17:27
@bolichen97
bolichen97 enabled auto-merge (squash) November 5, 2026 17:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4fca18f3c7407db78e102417b7285034ab7f427e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4fca18f3c7407db78e102417b7285034ab7f427e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this fixes the cause (the stop waits only on the sandbox wrapper), the wait is bounded, and the port is reported free only on positive evidence. Any uncertainty still reports the port.

[DESIGN-REVIEWED] 4fca18f

Description read

[DESCRIPTION-READ] 9efe533668d0ce451b43d6141a76b026706e00cfae7943c3f42e981bd9c9b390

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 4fca18f3c7407db78e102417b7285034ab7f427e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix is aimed at the observer, not at the stop that returns early; the description names that cause and declares it out of goal, so this is a note.

Notes

  • Item 1 — symptom-level: the description says "That stop signals the backend's process group but waits only on the root it spawned", and the fix re-probes from the observing side. Waiting for the process group inside stop_app_backend (apps/backend_runtime/termination.py) would fix the cause for every caller of the stop. ## Not a goal excludes that change, so this PR stays at symptom level and the cause remains open. Subtraction: none in scope; the cause-level fix would replace _observe_until_released later.

What this change ships

Inventory (4 items) — 3 justified

Intent: let an operator disable, un-trust or uninstall a sandboxed app without a false "backend still running" warning or a false 409 teardown_incomplete. This is a FIX.

  1. Disable, revoke and uninstall no longer report a backend whose port frees within 2 s — symptom-level: the stop's root-only wait is the cause
  2. Those paths can take up to 2 s longer, only when the port is still listening after the stop — justified
  3. A port that stops answering counts as freed only with positive evidence (a refused connect, or a completed PID lookup that finds no listener) — justified
  4. The uninstall step in the app-kit-platform spec now describes the grace — justified

[FIRST-PRINCIPLES-REVIEWED] 4fca18f

Description read

[DESCRIPTION-READ] ae6070979507f9abc955bf3fe452136a6a827180b443f70bd7e3a45b542278be

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4fca18f3c7407db78e102417b7285034ab7f427e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4fca18f

Verdict parsed from the review's SHA-scoped output markers for commit 4fca18f3c7407db78e102417b7285034ab7f427e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4fca18f3c7407db78e102417b7285034ab7f427e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4fca18f3c7407db78e102417b7285034ab7f427e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4fca18f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4fca18f3c7407db78e102417b7285034ab7f427e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 5, 2026
@bolichen97
bolichen97 force-pushed the fix/adoption-teardown-one-shot-port-probe branch from 7b0df40 to 07f9d5e Compare November 5, 2026 19:05
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed: Repeated probes can misclassify a surviving listener as stopped span=3ad3793c24aa

Fixed in 07f9d5e: once a probe has seen the port listening, a later miss ends the release wait only when teardown.port_released confirms the port is free.
port_released counts a refused connect, or, after a connect timeout (a full backlog, or Windows' slow refusal), a port->PID lookup (probe_port_listeners) that completes with no LISTEN socket; a lookup that cannot complete keeps the port reported, so withdrawal still refuses.
Class swept: the only post-stop re-probe is _observe_until_released; the single first probe in unstopped_backend_port is unchanged by this PR, and _wait_for_spawn_survival reads False as "keep polling", never "stopped".
Pinned by test_a_probe_losing_sight_of_an_unconfirmed_listener_still_reports_it, the trust-withdrawal probe_loses_sight cases and TestPortReleased; reverting the loop condition reds 4 tests.
self-added: yes
mechanism: port_released confirms a release before a probe miss ends the post-stop grace

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 5, 2026
@bolichen97
bolichen97 force-pushed the fix/adoption-teardown-one-shot-port-probe branch from 07f9d5e to a14561b Compare November 5, 2026 22:03
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/apps/backend.py, docs/system-specs/modules/app-kit-platform.md (and same-dir specs) since this head went green (d775742 -> b764e20); CI re-running.

The rate-limited lanes on 07f9d5e (installation API rate limit exceeded) were rerun and all came back clean on that head (GPT, Opus, Design, First Principles: no blocking findings). main's backend.py change (#30460) is in the npm-install start path, not the stop/probe code this PR uses; rebased tree passes the 1,236 related backend/teardown tests locally plus black/isort/flake8/mypy/comment-history/brand/line-cap/flake-ratchet/docs-lint.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 5, 2026
Disable, trust withdrawal and uninstall probed the backend's port once,
right after stop_app_backend returned. That stop signals the whole
process group but waits only on the root it spawned, and a sandboxed
backend's root is the sandbox wrapper, which exits before the server
inside it closes its listening socket. The single probe caught the
socket still bound and reported "backend still running ... not ours to
stop" for a backend that was already exiting; on trust withdrawal that
turned into a 409 refusal.

stop_backend_and_observe now re-probes a still-listening port for a
bounded grace (2 s, 50 ms polls) and returns a port only when it is
still listening at the end. A listener that outlives the grace is
reported exactly where it was, so withdrawn trust still refuses.

Part of #26967
@bolichen97
bolichen97 force-pushed the fix/adoption-teardown-one-shot-port-probe branch from a14561b to 4fca18f Compare November 5, 2026 23:34
@bolichen97

bolichen97 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/app-kit-platform.md, src/kiro_crew/apps/registry.py, src/kiro_crew/apps/registry_pipeline/catalog.py and 13 other same-dir/import neighbours since this head went green (b764e20 -> 7e52ec8); CI re-running.

The five AI review lanes on a14561b (Design, First Principles, GPT 6.1, Opus 5.5, UX) produced no verdict: each failed closed on API rate limit exceeded for installation (repo-wide GitHub App quota), not on the diff. The rebase onto 7e52ec8 re-runs all of them on 4fca18f; the diff itself is unchanged and the related tests (584 across the teardown importers) pass locally on the new base.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 5, 2026
@bolichen97
bolichen97 merged commit 7443c94 into main Nov 6, 2026
98 checks passed
@bolichen97
bolichen97 deleted the fix/adoption-teardown-one-shot-port-probe branch November 6, 2026 01:31
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 6, 2026
buluoray pushed a commit that referenced this pull request Nov 8, 2026
Disable, trust withdrawal and uninstall probed the backend's port once,
right after stop_app_backend returned. That stop signals the whole
process group but waits only on the root it spawned, and a sandboxed
backend's root is the sandbox wrapper, which exits before the server
inside it closes its listening socket. The single probe caught the
socket still bound and reported "backend still running ... not ours to
stop" for a backend that was already exiting; on trust withdrawal that
turned into a 409 refusal.

stop_backend_and_observe now re-probes a still-listening port for a
bounded grace (2 s, 50 ms polls) and returns a port only when it is
still listening at the end. A listener that outlives the grace is
reported exactly where it was, so withdrawn trust still refuses.

Part of #26967

(cherry picked from commit 7443c94)
buluoray pushed a commit that referenced this pull request Nov 8, 2026
Disable, trust withdrawal and uninstall probed the backend's port once,
right after stop_app_backend returned. That stop signals the whole
process group but waits only on the root it spawned, and a sandboxed
backend's root is the sandbox wrapper, which exits before the server
inside it closes its listening socket. The single probe caught the
socket still bound and reported "backend still running ... not ours to
stop" for a backend that was already exiting; on trust withdrawal that
turned into a 409 refusal.

stop_backend_and_observe now re-probes a still-listening port for a
bounded grace (2 s, 50 ms polls) and returns a port only when it is
still listening at the end. A listener that outlives the grace is
reported exactly where it was, so withdrawn trust still refuses.

Part of #26967

(cherry picked from commit 7443c94)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants