Skip to content

fix(gateway): spare a booting peer on port reclaim; start app backends in one wave - #29277

Merged
buluoray merged 1 commit into
mainfrom
fix/11633-booting-peer-reclaim
Nov 5, 2026
Merged

buluoray merged 1 commit into
mainfrom
fix/11633-booting-peer-reclaim

Conversation

@buluoray

@buluoray buluoray commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A second gateway that starts on the same port no longer kills a gateway that is still starting, and every app backend starts in one boot wave.

Since #6599 a gateway reserves its dashboard port for its whole boot pass: the socket is bound and listening, but nothing accepts until the runner starts. During that window the gateway does not answer HTTP. reclaim_stale_gateway_port treats "a Kiro Crew process holds the port and does not answer HTTP" as a wedged gateway and terminates it. So a second gateway that races for the same port while the first is booting can SIGTERM/SIGKILL a healthy peer (issue #11633, task 1).

The same reservation exports KIROCREW_BOUND_PORT before the app-backend boot wave. The second, deferred wave that start_dashboard ran after the listener served (added for #10330 so Dev Fleet would get the bound port) no longer has a reason to exist (task 2).

Why it matters

Two gateways booting on one port at the same time is narrow, but when it happens the loser kills a healthy gateway and orphans its backends until the next boot's stale reap. The deferred wave is dead weight on the boot path and keeps a second spelling of "when does a backend start" alive in the code, the specs and the tests.

Not a goal

  • Task 3 of Post-#6599 follow-ups: reclaim probe vs booting peer; collapse the two-wave backend spawn #11633 (Windows SO_EXCLUSIVEADDRUSE / TIME_WAIT verification, the _bind_once comment, retiring Dev Fleet's KIROCREW_BOUND_PORT pass-through). It needs a real Windows run, so the issue stays open: Refs #11633.
  • A reservation-time HTTP responder. This PR keeps the reservation contract as it is and reads "booting" from the holder's age.
  • Changing the reclaim ladder's timing or the Windows TIME_WAIT budget.

What changed (motivation → approach → change)

Task 1, reclaim. Symptom: a booting peer is killed. Cause: the only signals were "is a Kiro Crew process" and "answers HTTP", and a booting gateway fails the second one by design. Of the two remedies the issue names, a reservation-time responder would change the reservation contract and every reader of it; a holder-age check stays inside port_reclaim.py and spares the case the issue describes. Change:

  • src/kiro_crew/dashboard/port_reclaim.py: new outcome BOOTING_PEER. After the probe fails and the identity re-check passes, a holder younger than BOOTING_PEER_MAX_AGE_SECS (120 s) is read as booting and no holder is signalled. A single-threaded holder is still reclaimed: it is a wedged fork, never a running gateway (the same signal _describe_holders already uses). An unreadable age is not read as young. Age comes from session_pid._pid_age_seconds (Linux /proc, macOS start id, Windows creation FILETIME). Both collaborators are injectable like the others.
  • The cost of a wrong guess is bounded and stated in the code: a young gateway that really is wedged is reclaimed by the first relaunch after it is 120 s old.
  • src/kiro_crew/dashboard/server_runtime/listener.py (_start_site, _reserve_dashboard_port): BOOTING_PEER is handled like HEALTHY_PEER (a live holder: no misleading "waiting for release" line, no Windows TIME_WAIT stretch). server.py binds the new name on the facade.

Task 2, one wave. start_dashboard calls _reserve_dashboard_port and _export_reserved_bind_evidence before _start_app_backends, so KIROCREW_BOUND_PORT is set when every backend spawns. The headless entrypoint starts no app backends, so the "shared with the headless entrypoint" reason in _start_bound_port_app_backends' docstring did not hold. Change:

  • src/kiro_crew/apps/backend_runtime/startup.py: start_enabled_app_backends spawns every admitted app. start_deferred_app_backends and _DEFERRED_BOUND_PORT_APPS are removed. The fail-closed enablement re-read the deferred wave did (_app_enabled_state(name) is True) now runs for every app right before the one wave spawns, because list_apps() is a snapshot taken before the reconcile passes and an operator can disable an app in between.
  • src/kiro_crew/dashboard/server_runtime/app_platform.py / server.py: _start_bound_port_app_backends and its call after the listener are removed.
  • src/kiro_crew/apps/backend.py facade, builtin_host_grants.py comments, and the specs docs/system-specs/modules/app-kit-platform.md and docs/system-specs/modules/dev-fleet.md now describe one wave.
  • Dev Fleet's background warm refresh may call the gateway's live-target route while the gateway is still booting. That call waits in the reserved socket's backlog (the same path every other app's early callback already takes). If it outlasts the broker's 5 s timeout it is a PointerUnavailable, which the fleet view already renders as "Live state unavailable" and the removal guards refuse on; it is not cached, and the next read retries. dev-fleet.md says this.

Backwards compatibility

Compatible: boot behaviour for every app is the same except Dev Fleet starts earlier in the same boot; no config, route or stored record changes.

Reader: src/kiro_crew/dashboard/server.py:start_dashboard -- release -- the call to _start_bound_port_app_backends and the start_deferred_app_backends import are removed with the functions; test_every_app_backend_starts_in_one_wave_before_setup
Reader: src/kiro_crew/dashboard/server_runtime/app_platform.py:_start_bound_port_app_backends -- release -- removed; it was the only caller of start_deferred_app_backends
Reader: src/kiro_crew/apps/backend.py:facade re-exports -- release -- start_deferred_app_backends and _DEFERRED_BOUND_PORT_APPS dropped from the facade; test_app_backend_composition_contract.py updated
Reader: test/campaign_facade_inventories/kiro_crew.dashboard.server.json -- release -- start_deferred_app_backends leaves the server facade deliberately; test_campaign_facades_base_public_names.py passes
Reader: docs/system-specs/modules/app-kit-platform.md, docs/system-specs/modules/dev-fleet.md -- release -- rewritten for one wave

git grep on origin/main finds no other reader (no app bundle, cron or website code names these symbols).

Tests

  • test/test_port_reclaim.py::test_a_booting_peer_holding_its_reserved_port_is_never_killed: a silent, young, multi-threaded holder gives BOOTING_PEER and nothing is terminated.
  • test/test_port_reclaim.py::test_a_young_holder_with_unknown_thread_count_is_spared: off Linux age alone decides.
  • test/test_port_reclaim.py::test_a_silent_holder_that_is_not_booting_is_still_reclaimed (3 cases): old holder, single-threaded young fork, unreadable age are all still reclaimed.
  • Both reclaim test files pin fabricated pids as old with an autouse fixture, so a real young process that happens to own pid 1234 on a CI host cannot flip a case.
  • test/test_app_backend_spawn_env.py::TestBootAdmissionRevet::test_bound_port_backends_spawn_in_the_one_boot_wave: Dev Fleet spawns in the boot wave and sees KIROCREW_BOUND_PORT.
  • test/test_builtin_host_grants.py::test_the_boot_wave_spawns_the_bound_port_apps_with_the_rest: a bound-port app is not held back.
  • test/test_app_backend_spawn_env.py::TestBootAdmissionRevet::test_an_app_disabled_after_the_listing_is_not_spawned (False, None): an app disabled, or with an unreadable state, after the listing is not spawned.
  • test/test_dashboard_server_startup_coverage.py::test_every_app_backend_starts_in_one_wave_before_setup: one wave, before runner.setup().
  • Retired with the second wave: the deferred-wave re-check, disable-between-waves, governance-between-waves and no-re-arm tests. Composition contracts (test_dashboard_server_composition_contract*.py, test_app_backend_composition_contract.py) updated for the removed names and the boot trace.

Proof: not provable -- other: a whole-hunk revert drops the BOOTING_PEER constant the new tests import, so prove.py only sees a collection error. Behavioural mutations were run by hand instead: if booting: -> if False: reddens test_a_booting_peer_holding_its_reserved_port_is_never_killed and test_a_young_holder_with_unknown_thread_count_is_spared; holding bound-port apps back in start_enabled_app_backends reddens test_bound_port_backends_spawn_in_the_one_boot_wave and test_the_boot_wave_spawns_the_bound_port_apps_with_the_rest; spawning admitted instead of still_enabled reddens test_an_app_disabled_after_the_listing_is_not_spawned (head b9924af).

Locally: 956 passed (every test file naming start_enabled_app_backends or port_reclaim, plus the composition contracts) across the reclaim, app-backend, host-grants, composition-contract and startup-coverage suites; flake8 and mypy clean on the changed modules; scripts/check_comment_history.py passes against origin/main.

Manual verification

Not run on a live pair of gateways. The race needs two gateways booting on one port at once; the decision logic is covered through injected collaborators. Windows and macOS were not run: the age read on those platforms (_pid_age_seconds) is existing code, checked against source only.

Related Issues

Refs #11633 (tasks 1 and 2; task 3 needs a Windows host and stays open)

Sibling sweep

Swept: git grep -nE "reclaim_stale_gateway_port|_probe_gateway_healthy" -- src/
Sibling: src/kiro_crew/dashboard/server_runtime/listener.py:_start_site -- changed here
Sibling: src/kiro_crew/dashboard/server_runtime/listener.py:_reserve_dashboard_port -- changed here

Pattern harvest

Not generalizable: the "silent means wedged" inference is specific to this one probe and the reservation state #6599 introduced.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@buluoray buluoray added the tier:T1 Bug: behaviour does not match intent; fix keeps the design label Nov 3, 2026
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 3, 2026 19:04
@buluoray
buluoray requested a review from CrysisDeu November 3, 2026 19:04
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e3bcbdf1a5c602759ce652162806b51325195de2 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e3bcbdf1a5c602759ce652162806b51325195de2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The holder-age check stays inside port_reclaim.py. An unreadable age falls back to the old reclaim behaviour. The one boot wave runs after _export_reserved_bind_evidence, so every backend starts with KIROCREW_BOUND_PORT set.

Notes

  • The check reads age from process start. A gateway that re-execs in place keeps its pid and start time, so it still looks old while booting and stays reclaimable. That gap existed before this PR and is not new, but one sentence in port_reclaim.py naming it would stop a reader from assuming every booting peer is spared.

[DESIGN-REVIEWED] e3bcbdf

Description read

[DESCRIPTION-READ] 887a25361740d5980ad1817afa523830123cfd96348bfaae1d10d3afffbbf8f6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e3bcbdf1a5c602759ce652162806b51325195de2 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Both tasks fix a mechanism tied to issue #11633, and the deferred wave is deleted rather than patched. One test-only injection seam is a note.

Notes

  • Item 6 — one consumer, generalized: holder_age and thread_count on reclaim_stale_gateway_port have 0 callers in src/ (grepped holder_age=|thread_count=); only tests pass them. Subtraction: drop both parameters and monkeypatch _holder_age_secs and platform_compat.process_thread_count in the tests.

What this change ships

Inventory (7 items) — 6 justified

Intent: a second gateway on the same port must not kill a peer that is still starting, and app backends start in a single boot wave. FIX (task 1) plus a deletion (task 2).

  1. A silent gateway holding the port for less than 120 s is spared, unless it is a single-threaded fork — justified
  2. The relaunch reports "still starting — not reclaiming" with no wait line and no Windows TIME_WAIT stretch — justified
  3. Dev Fleet now starts earlier, in the same wave as every other app, before runner.setup() — justified
  4. The second wave, start_deferred_app_backends and _DEFERRED_BOUND_PORT_APPS are removed from the facade — justified
  5. Every app's enabled state is re-read just before it spawns, so an app disabled mid-boot does not start — justified
  6. New injectable age and thread-count parameters on the reclaim function — one consumer, generalized (tests only)
  7. The governance re-check between waves is retired along with its tests; governance is still checked in the main wave (startup.py:111,177) — justified

[FIRST-PRINCIPLES-REVIEWED] e3bcbdf

Description read

[DESCRIPTION-READ] a2a13566dc69016af3015a00f59a9d694bc9d06ac3be4834e1fe265f0ce69dab

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e3bcbdf1a5c602759ce652162806b51325195de2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e3bcbdf

Verdict parsed from the review's SHA-scoped output markers for commit e3bcbdf1a5c602759ce652162806b51325195de2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e3bcbdf1a5c602759ce652162806b51325195de2 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e3bcbdf1a5c602759ce652162806b51325195de2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e3bcbdf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e3bcbdf1a5c602759ce652162806b51325195de2 false-positive: <reason, 20 to 500 characters>

@buluoray
buluoray force-pushed the fix/11633-booting-peer-reclaim branch from cecd2b9 to b9924af Compare November 3, 2026 19:26
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 3, 2026
@buluoray
buluoray force-pushed the fix/11633-booting-peer-reclaim branch from b9924af to 1235842 Compare November 3, 2026 22:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 3, 2026
@buluoray
buluoray enabled auto-merge (squash) November 3, 2026 23:34
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 4, 2026
…s in one wave

A gateway reserves its dashboard port (bound and listening, accept deferred)
for its whole boot pass, so the reclaim probe could not tell it from a wedged
gateway and a second gateway racing for the port could SIGTERM it. A silent
holder younger than BOOTING_PEER_MAX_AGE_SECS (120 s) is now reported as
BOOTING_PEER and never signalled, unless it is a single-threaded fork.

Since that reservation exports KIROCREW_BOUND_PORT before the boot wave, the
deferred second wave for bound-port apps (Dev Fleet) is no longer needed:
start_enabled_app_backends now spawns every admitted app, and
start_deferred_app_backends and its ordering tests are retired.

Refs #11633
@buluoray
buluoray force-pushed the fix/11633-booting-peer-reclaim branch from 1235842 to e3bcbdf Compare November 4, 2026 15:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 4, 2026
@buluoray

buluoray commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 5) red is #29457, fails on main too (same 3 cases of test_decisions_end_to_end.py::test_jev_response_reaches_the_assembled_message fail on main 7e94bdf). Fix is open in #29458. Not re-running; will rebase once #29458 merges to main.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 5, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving e3bcbdf.

What I checked on this head:

  • All 98 checks are green, none pending.
  • Port reclaim (dashboard/port_reclaim.py): a silent holder is now spared when it is a Kiro Crew process younger than 120 seconds and not a single-threaded fork, and the new BOOTING_PEER outcome is handled next to HEALTHY_PEER in both listener paths. This only makes reclaim less eager. An unreadable age is not read as young, an age check that times out returns UNAVAILABLE and kills nothing, and a wedged gateway is still reclaimed by the next relaunch once it is older than the window.
  • One boot wave (apps/backend_runtime/startup.py): the deferred second wave and its module state are removed, and bound-port apps spawn with the rest. I checked that start_enabled_app_backends has a single caller, start_dashboard, which reserves the port and exports KIROCREW_BOUND_PORT first. The headless entrypoint exports after listen and never starts app backends, so removing the second wave leaves no path without the port.
  • Admission and governance: every app is still vetted before the spawn list is built. The old deferred wave re-checked enablement and _app_activation_denied because the rest of start_dashboard ran between vetting and spawn. That window no longer exists for bound-port apps. The PR adds one fail-closed enablement re-read for all apps right before the spawn (an unreadable state does not spawn), so no app that was started before is refused now except one the operator just disabled.
  • builtin_host_grants.py has comment and docstring edits only.
  • The tests cover the age and thread-count branches, the new outcome in both listeners, the single wave and the spawn environment. No regex is added.

Security checklist: no change to auth, the user allowlist, a sandbox or an approval gate, and no new input parsing. The change loosens nothing and tightens nothing beyond the disabled-app re-read above. I left the existing auto-merge alone.

@buluoray
buluoray merged commit 7b44c15 into main Nov 5, 2026
153 of 155 checks passed
@buluoray
buluoray deleted the fix/11633-booting-peer-reclaim branch November 5, 2026 00:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tier:T1 Bug: behaviour does not match intent; fix keeps the design

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants