Repository navigation
fix(gateway): spare a booting peer on port reclaim; start app backends in one wave - #29277
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The holder-age check stays inside Notes
[DESIGN-REVIEWED] e3bcbdf Description read[DESCRIPTION-READ] 887a25361740d5980ad1817afa523830123cfd96348bfaae1d10d3afffbbf8f6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Both tasks fix a mechanism tied to issue #11633, and the deferred wave is deleted rather than patched. One test-only injection seam is a note. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: a second gateway on the same port must not kill a peer that is still starting, and app backends start in a single boot wave. FIX (task 1) plus a deletion (task 2).
[FIRST-PRINCIPLES-REVIEWED] e3bcbdf Description read[DESCRIPTION-READ] a2a13566dc69016af3015a00f59a9d694bc9d06ac3be4834e1fe265f0ce69dab That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
cecd2b9 to
b9924af
Compare
b9924af to
1235842
Compare
…s in one wave A gateway reserves its dashboard port (bound and listening, accept deferred) for its whole boot pass, so the reclaim probe could not tell it from a wedged gateway and a second gateway racing for the port could SIGTERM it. A silent holder younger than BOOTING_PEER_MAX_AGE_SECS (120 s) is now reported as BOOTING_PEER and never signalled, unless it is a single-threaded fork. Since that reservation exports KIROCREW_BOUND_PORT before the boot wave, the deferred second wave for bound-port apps (Dev Fleet) is no longer needed: start_enabled_app_backends now spawns every admitted app, and start_deferred_app_backends and its ordering tests are retired. Refs #11633
1235842 to
e3bcbdf
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approving e3bcbdf.
What I checked on this head:
- All 98 checks are green, none pending.
- Port reclaim (
dashboard/port_reclaim.py): a silent holder is now spared when it is a Kiro Crew process younger than 120 seconds and not a single-threaded fork, and the newBOOTING_PEERoutcome is handled next toHEALTHY_PEERin both listener paths. This only makes reclaim less eager. An unreadable age is not read as young, an age check that times out returnsUNAVAILABLEand kills nothing, and a wedged gateway is still reclaimed by the next relaunch once it is older than the window. - One boot wave (
apps/backend_runtime/startup.py): the deferred second wave and its module state are removed, and bound-port apps spawn with the rest. I checked thatstart_enabled_app_backendshas a single caller,start_dashboard, which reserves the port and exportsKIROCREW_BOUND_PORTfirst. The headless entrypoint exports after listen and never starts app backends, so removing the second wave leaves no path without the port. - Admission and governance: every app is still vetted before the spawn list is built. The old deferred wave re-checked enablement and
_app_activation_deniedbecause the rest ofstart_dashboardran between vetting and spawn. That window no longer exists for bound-port apps. The PR adds one fail-closed enablement re-read for all apps right before the spawn (an unreadable state does not spawn), so no app that was started before is refused now except one the operator just disabled. builtin_host_grants.pyhas comment and docstring edits only.- The tests cover the age and thread-count branches, the new outcome in both listeners, the single wave and the spawn environment. No regex is added.
Security checklist: no change to auth, the user allowlist, a sandbox or an approval gate, and no new input parsing. The change loosens nothing and tightens nothing beyond the disabled-app re-read above. I left the existing auto-merge alone.
Problem / Motivation
Goal: A second gateway that starts on the same port no longer kills a gateway that is still starting, and every app backend starts in one boot wave.
Since #6599 a gateway reserves its dashboard port for its whole boot pass: the socket is bound and listening, but nothing accepts until the runner starts. During that window the gateway does not answer HTTP.
reclaim_stale_gateway_porttreats "a Kiro Crew process holds the port and does not answer HTTP" as a wedged gateway and terminates it. So a second gateway that races for the same port while the first is booting can SIGTERM/SIGKILL a healthy peer (issue #11633, task 1).The same reservation exports
KIROCREW_BOUND_PORTbefore the app-backend boot wave. The second, deferred wave thatstart_dashboardran after the listener served (added for #10330 so Dev Fleet would get the bound port) no longer has a reason to exist (task 2).Why it matters
Two gateways booting on one port at the same time is narrow, but when it happens the loser kills a healthy gateway and orphans its backends until the next boot's stale reap. The deferred wave is dead weight on the boot path and keeps a second spelling of "when does a backend start" alive in the code, the specs and the tests.
Not a goal
SO_EXCLUSIVEADDRUSE/ TIME_WAIT verification, the_bind_oncecomment, retiring Dev Fleet'sKIROCREW_BOUND_PORTpass-through). It needs a real Windows run, so the issue stays open:Refs #11633.What changed (motivation → approach → change)
Task 1, reclaim. Symptom: a booting peer is killed. Cause: the only signals were "is a Kiro Crew process" and "answers HTTP", and a booting gateway fails the second one by design. Of the two remedies the issue names, a reservation-time responder would change the reservation contract and every reader of it; a holder-age check stays inside
port_reclaim.pyand spares the case the issue describes. Change:src/kiro_crew/dashboard/port_reclaim.py: new outcomeBOOTING_PEER. After the probe fails and the identity re-check passes, a holder younger thanBOOTING_PEER_MAX_AGE_SECS(120 s) is read as booting and no holder is signalled. A single-threaded holder is still reclaimed: it is a wedged fork, never a running gateway (the same signal_describe_holdersalready uses). An unreadable age is not read as young. Age comes fromsession_pid._pid_age_seconds(Linux/proc, macOS start id, Windows creation FILETIME). Both collaborators are injectable like the others.src/kiro_crew/dashboard/server_runtime/listener.py(_start_site,_reserve_dashboard_port):BOOTING_PEERis handled likeHEALTHY_PEER(a live holder: no misleading "waiting for release" line, no Windows TIME_WAIT stretch).server.pybinds the new name on the facade.Task 2, one wave.
start_dashboardcalls_reserve_dashboard_portand_export_reserved_bind_evidencebefore_start_app_backends, soKIROCREW_BOUND_PORTis set when every backend spawns. The headless entrypoint starts no app backends, so the "shared with the headless entrypoint" reason in_start_bound_port_app_backends' docstring did not hold. Change:src/kiro_crew/apps/backend_runtime/startup.py:start_enabled_app_backendsspawns every admitted app.start_deferred_app_backendsand_DEFERRED_BOUND_PORT_APPSare removed. The fail-closed enablement re-read the deferred wave did (_app_enabled_state(name) is True) now runs for every app right before the one wave spawns, becauselist_apps()is a snapshot taken before the reconcile passes and an operator can disable an app in between.src/kiro_crew/dashboard/server_runtime/app_platform.py/server.py:_start_bound_port_app_backendsand its call after the listener are removed.src/kiro_crew/apps/backend.pyfacade,builtin_host_grants.pycomments, and the specsdocs/system-specs/modules/app-kit-platform.mdanddocs/system-specs/modules/dev-fleet.mdnow describe one wave.PointerUnavailable, which the fleet view already renders as "Live state unavailable" and the removal guards refuse on; it is not cached, and the next read retries.dev-fleet.mdsays this.Backwards compatibility
Compatible: boot behaviour for every app is the same except Dev Fleet starts earlier in the same boot; no config, route or stored record changes.
Reader: src/kiro_crew/dashboard/server.py:start_dashboard -- release -- the call to
_start_bound_port_app_backendsand thestart_deferred_app_backendsimport are removed with the functions;test_every_app_backend_starts_in_one_wave_before_setupReader: src/kiro_crew/dashboard/server_runtime/app_platform.py:_start_bound_port_app_backends -- release -- removed; it was the only caller of
start_deferred_app_backendsReader: src/kiro_crew/apps/backend.py:facade re-exports -- release --
start_deferred_app_backendsand_DEFERRED_BOUND_PORT_APPSdropped from the facade;test_app_backend_composition_contract.pyupdatedReader: test/campaign_facade_inventories/kiro_crew.dashboard.server.json -- release --
start_deferred_app_backendsleaves the server facade deliberately;test_campaign_facades_base_public_names.pypassesReader: docs/system-specs/modules/app-kit-platform.md, docs/system-specs/modules/dev-fleet.md -- release -- rewritten for one wave
git grepon origin/main finds no other reader (no app bundle, cron or website code names these symbols).Tests
test/test_port_reclaim.py::test_a_booting_peer_holding_its_reserved_port_is_never_killed: a silent, young, multi-threaded holder givesBOOTING_PEERand nothing is terminated.test/test_port_reclaim.py::test_a_young_holder_with_unknown_thread_count_is_spared: off Linux age alone decides.test/test_port_reclaim.py::test_a_silent_holder_that_is_not_booting_is_still_reclaimed(3 cases): old holder, single-threaded young fork, unreadable age are all still reclaimed.test/test_app_backend_spawn_env.py::TestBootAdmissionRevet::test_bound_port_backends_spawn_in_the_one_boot_wave: Dev Fleet spawns in the boot wave and seesKIROCREW_BOUND_PORT.test/test_builtin_host_grants.py::test_the_boot_wave_spawns_the_bound_port_apps_with_the_rest: a bound-port app is not held back.test/test_app_backend_spawn_env.py::TestBootAdmissionRevet::test_an_app_disabled_after_the_listing_is_not_spawned(False,None): an app disabled, or with an unreadable state, after the listing is not spawned.test/test_dashboard_server_startup_coverage.py::test_every_app_backend_starts_in_one_wave_before_setup: one wave, beforerunner.setup().test_dashboard_server_composition_contract*.py,test_app_backend_composition_contract.py) updated for the removed names and the boot trace.Proof: not provable -- other: a whole-hunk revert drops the BOOTING_PEER constant the new tests import, so prove.py only sees a collection error. Behavioural mutations were run by hand instead:
if booting:->if False:reddens test_a_booting_peer_holding_its_reserved_port_is_never_killed and test_a_young_holder_with_unknown_thread_count_is_spared; holding bound-port apps back in start_enabled_app_backends reddens test_bound_port_backends_spawn_in_the_one_boot_wave and test_the_boot_wave_spawns_the_bound_port_apps_with_the_rest; spawningadmittedinstead ofstill_enabledreddens test_an_app_disabled_after_the_listing_is_not_spawned (head b9924af).Locally: 956 passed (every test file naming
start_enabled_app_backendsorport_reclaim, plus the composition contracts) across the reclaim, app-backend, host-grants, composition-contract and startup-coverage suites; flake8 and mypy clean on the changed modules;scripts/check_comment_history.pypasses against origin/main.Manual verification
Not run on a live pair of gateways. The race needs two gateways booting on one port at once; the decision logic is covered through injected collaborators. Windows and macOS were not run: the age read on those platforms (
_pid_age_seconds) is existing code, checked against source only.Related Issues
Refs #11633 (tasks 1 and 2; task 3 needs a Windows host and stays open)
Sibling sweep
Swept: git grep -nE "reclaim_stale_gateway_port|_probe_gateway_healthy" -- src/
Sibling: src/kiro_crew/dashboard/server_runtime/listener.py:_start_site -- changed here
Sibling: src/kiro_crew/dashboard/server_runtime/listener.py:_reserve_dashboard_port -- changed here
Pattern harvest
Not generalizable: the "silent means wedged" inference is specific to this one probe and the reservation state #6599 introduced.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.