Skip to content

fix(service): let a live-holder lock refusal stand the unit down - #14525

Merged
iamwhatever merged 1 commit into
mainfrom
fix/gateway-live-holder-refusal-terminal-exit
Sep 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/gateway-live-holder-refusal-terminal-exit

Conversation

@pepmach

@pepmach pepmach commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A kirocrew gateway service unit that finds its home already served by another running gateway stands down once, with the reason in the journal, instead of relaunching against that refusal.

Bug fix. When a systemd-supervised kirocrew gateway starts against a KIROCREW_HOME whose gateway.lock is held by a live gateway that also holds the dashboard port, it prints 👻 <home>/gateway.lock is held by pid <PID> (… holds port 5476, answering HTTP) -- another gateway already owns <home>; … and exits 1. Restart=always relaunches it every RestartSec, and each relaunch boots most of the stack only to meet the same refusal. On the shipped unit StartLimitBurst=3 parks the unit failed after three rounds; on a unit without StartLimit* the loop is unbounded (the reporting host counted 26,316 restarts). The healthy sibling keeps serving throughout, so nothing ever recovers the looping unit.

Why it matters

Every operator whose host ends up with two controllers for one home — a hand-run gateway beside the service, a second supervisor, a post-update relaunch into a home the previous gateway still holds — gets a unit that is either parked failed after three noisy boots or churning forever, with status=1/FAILURE telling them nothing about why. The refusal itself is correct; the supervisor just cannot tell a permanent "this home is already served" apart from a crash worth retrying.

Not a goal

  • Making the launchd (macOS) gateway plist stand down on the same refusal. The exit status changes on every platform, but service/macos.py keeps KeepAlive=true and relaunches every exit, so a launchd-supervised gateway meets this refusal exactly as it does today. Deferred: the plist relaunches exit 0 on purpose (the stale-asset watchdog relies on it) and needs its own success/failure design, as the pod's launchd_exit_code translation shows.
  • The desktop/Electron supervisor's fight over the port after a side-by-side update (issue desktop: external 'kirocrew restart' races the supervisor's crash recovery, resurrecting the old-version gateway after a side-by-side update #11811), and the stale-asset watchdog's status=75 exit that triggered the reporting host's loop (issue Stale-asset watchdog shuts down the gateway mid-update and reaps its own installer, leaving a venv with no console scripts #13521). Both are separate mechanisms with their own issues.
  • Re-rendering an already-installed unit. service install writes the unit once and no upgrade re-renders it, so a unit written by an earlier build does not carry RestartPreventExitStatus= until it is re-rendered. A currency check for the gateway unit (the pod's unit_is_current shape) is new install/status scope, not part of this fix; the gap is stated in the spec.
  • Making the orphaned-flock refusal terminal. When the acquirer is dead and an unknown inheritor holds the descriptor, a relaunch is what brings the gateway back once that process dies, so that refusal deliberately stays exit 1.

What changed (motivation → approach → change)

Observed symptom: every lock refusal exits 1 (cli.py, the GatewayLockError catch on the gateway path), and the rendered unit has Restart=always with no RestartPreventExitStatus, so systemd retries a refusal the sibling keeps permanent.

Root cause: the exit status carried no information about whether a retry could help. The pod plane already has the convention (pod/config.py terminal boot exit codes fed into RestartPreventExitStatus= by pod/unit.py); the gateway unit had no counterpart.

Change, copying that convention:

  • gateway_lock.py defines LIVE_HOLDER_EXIT_CODE = 78 (EX_CONFIG: two supervisors pointed at one home is a host configuration; the remedy is to change it, never to try again), distinct from 69 (listener lost) and 75 (stale assets), which the running gateway exits with on purpose to be relaunched. It lives beside the refusal because cli.py returns it and service/linux.py exempts it — one definition, no drift.
  • GatewayLockError gains a structured live_holder flag, and ONE predicate decides it: GatewayLock._serving_verdict, asked only about the process /proc/locks positively identifies as the lock's acquirer (lock file, or home anchor when the file was deleted or replaced), never about the pid the lock file merely records. Four conjuncts, each measured once: alive; holding the configured dashboard port (one listener enumeration, platform_compat.find_port_listeners, filtered to the owner's own sockets); holding it AT the address the probe reaches — this gateway's configured bind (KIROCREW_BIND when an IP; none or 0.0.0.0 → 127.0.0.1, :: → ::1 because the dashboard binds it IPV6_V6ONLY), wildcards covering by family only, an unreported address counting as unknowable; and answering HTTP there. The address conjunct ties port ownership and HTTP health to ONE process, so a stranger answering at the probe address is never credited to an owner bound elsewhere. Only all four make live_holder True. Everything else stays False and exits 1: an identity race, an unmeasurable home anchor, an orphaned flock, a live acquirer without the port, one whose own socket at the probed address is silent (a wedged gateway), one holding the port only elsewhere or at an unreported address (the residual row, unasserted by design; the message names what was measured, and probing the holder's own address is #14525 follow-up: service unit staleness self-heal for RestartPreventExitStatus #14527's follow-up), and every refusal with no owner surface (no /proc/locks, or a filesystem the lock table never matches), where the predicate is never asked. With no port to weigh (--port auto, --slack-only) every refusal stays restartable.
  • cli.py exits LIVE_HOLDER_EXIT_CODE when exc.live_holder is set and keeps sys.exit(1) for every other refusal; it hands the lock the configured bind (bind_address_for) beside the port.
  • service/linux.py::render_unit() emits RestartPreventExitStatus=78 right after RestartSec=10, in both the system and the --user scope. Restart=always, RestartSec=10 and both StartLimit* lines are untouched, so a clean self-exit, the watchdog's 75, the listener guard's 69 and a transient lock failure are all still relaunched.
  • docs/system-specs/modules/cli.md (Service Management) documents the predicate by name, its three conjuncts, the residual row, what stays restartable, and the installed-unit gap with its remedy (kirocrew service install); docs/system-specs/modules/slack-gateway.md's exit-status table gains the 78 row.

Backwards compatibility

Compatible: nothing that works today stops working. The only observable differences are a different non-zero exit status (78 instead of 1) for the one refusal where the process /proc/locks positively identifies as holding the lock is running, holds the port and answers HTTP on it, a hedge sentence added to the refusal message when no surface can identify the holder or when the identified holder listens without answering HTTP (same exit 1 as before), and one added RestartPreventExitStatus= line in units rendered from now on. A unit written by an earlier build keeps its current behaviour (exit 78 is as non-zero as exit 1 was, so Restart=always still relaunches it) until it is re-rendered. GatewayLockError's constructor gains a keyword-only argument with a default; every existing construction site is unchanged.

Tests

Red-first, by file, run with -n0 against this branch's source and then against main's source (git stash of src/ with the tests kept):

  • test/test_cli.py::TestGatewayLockRefusalExit (32 tests): test_live_holder_truth_table[...] is the table-driven pin of the serving-holder predicate — 18 rows judged by the REAL diagnosis and the real kirocrew gateway exit: identified acquirer alive, its own socket at the probed address, answering HTTP (78), the same acquirer silent (1), without the port (1), dead (1), with no port configured (1); the address conjunct — a stranger answering at the probe address while the owner holds the port on another address (1), an owner whose socket address the platform did not report (1); a configured 0.0.0.0 bind probed at loopback (78), a configured :: bind probed at ::1 (78), a configured ::1 bind probed there (78); the residual row, an owner holding the port only at ::1 while this gateway binds loopback (1); no owner surface with the recorded pid alive and on the port, alive without the port, or dead (1 each, and the probe is never made); and the deleted-lock-file path for an answering acquirer (78), a silent one (1), a stranger at the probe address (1) and no identified owner (1). test_diagnostic_bind_address_is_the_bind_the_gateway_resolves[...] pins the lock's bind to dashboard.urls.bind_address_for for nine override values. The named regressions stay: a hand-made live-holder refusal exits 78 and never 0 or 1; every other refusal exits 1; an unidentified owner on the port exits 1; an identified owner on the port exits 78 only when it answers HTTP. Red on main: ImportError: cannot import name 'LIVE_HOLDER_EXIT_CODE' / AttributeError … 'live_holder'; the silent, residual, stranger and unidentified rows are also red against a diagnosis that decides on port ownership alone.
  • test/test_service.py::TestLinuxUnitRendering::test_render_unit_exempts_the_live_holder_refusal_from_restart[False|True]: the rendered unit carries exactly one RestartPreventExitStatus= line, in [Service], whose set is exactly {LIVE_HOLDER_EXIT_CODE}; the constant is pinned to 78 (it is baked into installed units); the set never includes 0, 1, 69 or 75; Restart=always, RestartSec=10 and both StartLimit* lines are still present. Red on main: ImportError: cannot import name 'LIVE_HOLDER_EXIT_CODE'.
  • test/test_gateway_lock_diagnosis.py: live_holder pinned per diagnosis branch — /proc/locks acquirer alive, holding the port and answering HTTP → True; live acquirer holding the port but not answering → False, message names not answering HTTP at 127.0.0.1 (the probed address), wedged and not treated as permanent, never offers kill, and the probe runs exactly once (test_a_live_acquirer_listening_but_not_answering_http_is_not_a_live_holder); probe_host_for_bind maps no override and 0.0.0.0 to 127.0.0.1, :: to ::1, and passes a specific address through, bracketed IPv6 included (test_the_predicate_probes_the_configured_bind_with_wildcards_mapped_by_family); _listener_reaches is pinned row by row — a specific bind reaches only itself (v4-mapped spelling included), 0.0.0.0 every v4 host, :: every v6 host and no v4 one, lsof's * by its reported family, and an unreported address or family never (test_a_listener_reaches_the_probe_host_only_when_its_bind_covers_it); live acquirer without the port → False (also covers the no-port case); dead acquirer with a single inheritor → False; no owner surface with the recorded pid running and holding the port → False, with the message hedging that the holder cannot be confirmed and the refusal is not treated as permanent, never naming systemd (the branch is the non-Linux path too) and never wording it as held by pid (test_refusal_hedges_when_a_live_port_holding_pid_has_no_owner_surface); recorded pid without the port, no port supplied, recorded pid gone → False. Red on main: AttributeError … 'live_holder'; the no-owner-surface and not-answering tests are also red against a diagnosis that decides on port ownership alone.
  • test/test_gateway_lock.py: two Linux-only real-process tests, each parametrized over the HTTP answer (/proc/locks names this process as the acquirer of a REAL held lock; the listener enumeration is pinned to this pid at loopback so the host's lsof never decides the verdict, and a bound loopback socket supplies the port number) — a refusal by the live acquirer on the port is a live holder only when it answers HTTP, otherwise the message says wedged and the refusal stays restartable, and the same lock without a port is never a live holder; a home still anchored by that acquirer after the lock file is deleted follows the same rule — plus the identity-race refusal (being replaced faster) pinned as not a live holder. Red on main: AttributeError … 'live_holder'; the not-answering variants are also red against a diagnosis that decides on port ownership alone.
  • Existing render_unit tests (test_service.py, test_service_selinux.py) stay green: test_service.py 408 passed / 3 skipped, test_service_selinux.py 50 passed, test_gateway_lock.py 64 passed / 1 skipped, test_gateway_lock_diagnosis.py 63 passed, test_cli.py::TestGatewayLockRefusalExit 32 passed, each file run alone with -n0.

The two TestGatewayLockRefusalExit tests patch faulthandler.enable because the gateway path arms it on the real stderr descriptor, which a capsys stream lacks; cli.py and crash_guard.py both call it through the one faulthandler module object, so the single patch covers both arm sites (the class passes when run alone, with a cold crash guard).

Manual verification

  • Rendered unit parses on systemd 252: systemd-analyze verify on render_unit(user_scope=True) written to a file exits 0 with no complaint about the new directive, while a deliberately misspelled RestartPreventExitStatusX= in the same file draws Unknown key name … in section 'Service', ignoring — so the check is real.
  • Real binary, isolated scratch KIROCREW_HOME, kirocrew gateway --no-open --no-crons --port 5599, against a scratch holder process that acquires the home's lock and occupies 5599: holder ANSWERING HTTP (a real http.server returning 200) → exit code 78, stderr 👻 <home>/gateway.lock is held by pid <PID> (2 threads, holds port 5599, answering HTTP at 127.0.0.1) -- another gateway already owns <home>; stop it first (kirocrew stop) or set KIROCREW_HOME to an isolated directory; holder LISTENING but never accepting (the wedged shape) → exit code 1, stderr 👻 <home>/gateway.lock is held by pid <PID> (1 thread, holds port 5599, not answering HTTP at 127.0.0.1) -- it is listening on port 5599 but not answering HTTP at 127.0.0.1 (the address this gateway would bind), so it looks like a wedged gateway rather than one serving this home; this refusal is not treated as permanent, and a supervisor, if one manages this gateway, will retry it. Stop it first (kirocrew stop) or set KIROCREW_HOME to an isolated directory; holder that holds the lock only → exit code 1, stderr … (1 thread, does not hold port 5599) ….
  • Not verified here: the supervised loop itself under a live systemd --user manager (this sandbox has no user bus). The exemption rests on systemd's RestartPreventExitStatus= semantics, the same mechanism the pod unit already relies on.
  • Gates: black ratchet, subprocess-encoding ratchet, isort, flake8, mypy (Success: no issues found in 1738 source files), comment-history gate, scripts/docs-lint.sh all clean. scripts/local-gate.py (backend related set, 52,286 tests): 52,285 passed, 1 failed — test_home_pin_survives_monkeypatch_undo.py::test_the_guard_accepts_a_scratch_home, which fails identically on an untouched main checkout in this sandbox (the guard sees the sandbox's live KIROCREW_HOME environment), so it is inherited, not caused here.
  • Local review round (one pass, one lane at a time) on the code of head f77aeb3082 (the head the server lanes judged; round 1 below changed gateway_lock.py, two test files and the two spec pages after it):
    • GPT lane (gpt-5.6-sol, CI-extracted brief): No findings.
    • Opus lane (claude-opus-5, CI-extracted brief): nothing blocks, three advisory findings — (1) the faulthandler patch covers only one of two arm sites: rebutted, both sites call the same module attribute and the class passes alone with a cold crash guard; (2) the added spec table row ended mid-clause: fixed in this head; (3) installed units are not re-rendered: declined as out of goal, stated under Not a goal and in the spec.
    • First Principles lane (claude-fable-5.1, base-ref prompt): CONCERNS once for an undeclared unfixed sibling (the launchd plist), cleared by stating the systemd-only scope and the deferred launchd half in the description; re-run on this head: PASS, 6 of 6 inventory items justified.
    • Round 1 (server lanes on f77aeb3082): GPT FINDING src/kiro_crew/gateway_lock.py:473 — fixed: _describe_unidentified_owner can no longer return a live-holder verdict; the truth table now reads "live_holder True only for a /proc/locks-identified acquirer that is alive and on the port; every unidentified-owner shape (no /proc/locks, or a filesystem it never matches) is False, keeps its hedge and exits 1", with the docstrings and cli.md updated to match. Opus FINDING src/kiro_crew/service/linux.py:333 (installed units are not re-rendered) — accepted and deferred to #14525 follow-up: service unit staleness self-heal for RestartPreventExitStatus #14527 (deferred-finding, Due 2026-10-09); cli.md names the interim remedy.
    • Round 2 (server lanes on ffbdca652f): Opus FINDING src/kiro_crew/gateway_lock.py:504 — fixed: the unidentified-owner hedge no longer names systemd (the branch is reached on macOS and Windows on every refusal); it reads "…so this refusal is not treated as permanent; a supervisor, if one manages this gateway, will retry it", the test asserts the platform-neutral phrase, and cli.md describes the message the same way. First Principles item 3 ("rides along: the hedge rewording") — declared and kept: the message states exactly what the exit code asserts. Rebased onto f57c1dee40 (fix(ci): stop three gates from reddening unrelated pull requests #14518, which also clears the three main-inherited CI reds this PR cited).
    • Round 3 (server lanes on 4c6d29ca24): GPT BLOCKING src/kiro_crew/gateway_lock.py:572 (an unresponsive listener permanently disables recovery) — fixed: live_holder requires the identified acquirer to hold the port AND answer HTTP, in both _describe_live_owner and _diagnose_replaced_lock_file; one listener scan and at most one HTTP probe per diagnosis (_port_verdict) feed both the message and the verdict; the listening-but-silent shape says wedged … not treated as permanent and exits 1; truth tables in the docstrings, cli.md and slack-gateway.md updated; red-first tests in all three test files, and the manual verification redone with an answering holder (78) and a silent one (1). Opus FINDING test/test_gateway_lock.py:270 (the two real-process tests let the host's lsof decide the verdict) — fixed in the same tests: the listener scan is pinned to this pid, the bound socket only supplies the port number.
    • Round 4 (restructure, on c6e5e8eb01): three truth-table rows on three heads met the restructure bar, so the verdict now lives in ONE predicate, GatewayLock._serving_verdict (a frozen _ServingVerdict: alive, holds the configured port, answers HTTP at probe_host, serving = all three), called by both _describe_live_owner and _diagnose_replaced_lock_file and never for an unidentified owner; the probe goes to the address this gateway is configured to bind (cli.py::_diagnostic_bind_address → probe_host_for_bind, wildcard → loopback), which makes the residual row explicit in code: a holder bound to another address is not asserted, exit 1 by design (GPT r3 gateway_lock.py:695, overridden on that head and tracked as #14525 follow-up: service unit staleness self-heal for RestartPreventExitStatus #14527 entry 2). Same rows, same verdicts as round 3 for a loopback or wildcard bind; a specific KIROCREW_BIND is now probed at that address instead of at loopback. Table-driven test test_live_holder_truth_table (14 rows, real diagnosis + real exit); manual verification re-run: answering holder 78, silent holder 1. Rebased onto 61e100dedd.
    • Round 5 (server lanes on 8b0e1d42a3): GPT BLOCKING src/kiro_crew/gateway_lock.py:808 (port ownership and HTTP health can belong to different processes) — fixed inside the predicate: the port conjunct is address-bound, the owner's OWN socket must be the one the probe reaches (_listener_reaches: wildcards cover by family, :: never covers v4 because the dashboard binds it IPV6_V6ONLY, an unreported address is unknowable → False), so a stranger's answer at the probe address is never credited to the owner; GPT FINDING :243 and Opus FINDING :225 (:: probed at IPv4 loopback) — fixed: :: → ::1; Opus FINDING :263 (the one-probe claim is false on the orphaned-lock path) — the claim is qualified to the identified-acquirer paths in the docstring and cli.md; First Principles item 7 (_diagnostic_bind_address duplicated bind_address_for) — fixed: the copy is deleted and the lock receives bind_address_for(local_only=True). Table rows added: stranger at the probe address, owner address unknowable, :: incumbent answering on ::1, and the stranger row on the deleted-lock-file path (18 rows). Same rows, same verdicts otherwise; CI red Integration (in-process gateway) is main's (test_route_sweep.py, red on main since de25db109a); rebased onto 972675c207.

Screenshots / video

no-visual-delta

Why no screenshot: backend-only change — an exit status, a systemd unit line and two spec documents; no user-visible frontend surface is touched.

Related Issues

Fixes #13914

Pattern harvest

Rule candidate: review-prompt
Pattern: a refusal caught on a supervised entry point exits with the generic status (sys.exit(1)) without classifying whether a retry can heal it, so a Restart= supervisor churns on a standing condition; the fix shape is the pod's — a terminal-code constant beside the refusal, fed into RestartPreventExitStatus=.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@pepmach
pepmach requested a review from a team as a code owner September 27, 2026 19:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 09960dc13ce4e2bd84626c62f15d542ed2855aa9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 09960dc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 09960dc13ce4e2bd84626c62f15d542ed2855aa9: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 09960dc13ce4e2bd84626c62f15d542ed2855aa9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify follow-up issue #14527 exists — the residual bound-elsewhere row deliberately keeps restart-looping on the strength of that tracked deferral.

What this change ships

Inventory (9 items) — 9 justified

Intent: a supervised gateway that finds its home already served by a live sibling stands down once instead of restart-looping — a FIX (reproduction in description: 26,316 counted restarts; red-first tests).

  1. A live-holder lock refusal exits 78 instead of 1 — justified
  2. Newly rendered systemd units (both scopes) gain RestartPreventExitStatus=78 — justified
  3. Terminal verdict requires the acquirer's OWN socket at the probed address, so a stranger's HTTP answer never parks the unit — justified
  4. HTTP probe targets the configured bind (:: probed at ::1) instead of always 127.0.0.1 — justified
  5. Unidentified-owner refusal message hedges instead of asserting "held by pid X" — rides along, declared; base's own comment ("weigh its own facts rather than presenting it as the holder") supports the reversed pin
  6. Wedged/bound-elsewhere refusals gain "not treated as permanent" wording — rides along, declared
  7. GatewayLockError.live_holder flag (1 consumer: cli.py) — justified
  8. bind_address parameter / probe_host_for_bind (1 passing caller; second constructor site apps/bridges.py:3590 unchanged by defaults) — justified
  9. cli.md + slack-gateway.md updates — justified (same-commit doc invariant)

Mechanism check: copies, not duplicates, the pod convention — pod/config.py TERMINAL_BOOT_EXIT_CODES is pod-plane-only and gateway_lock cannot import pod. Premises verified on base: find_port_listeners/PortListener (platform_compat.py:5724), bind_address_for (dashboard/urls.py:267), IPV6_V6ONLY (dashboard/server.py:2308). Unverified-premise directions all fail toward exit 1 (status quo). Siblings (launchd, installed units) declared and deferred in ## Not a goal.

[FIRST-PRINCIPLES-REVIEWED] 09960dc

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 09960dc13ce4e2bd84626c62f15d542ed2855aa9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify once under a live systemd user manager that exit 78 parks the unit failed without relaunch — the only premise no run exercised.

[DESIGN-REVIEWED] 09960dc

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 09960dc13ce4e2bd84626c62f15d542ed2855aa9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 09960dc

Verdict parsed from the review's SHA-scoped output markers for commit 09960dc13ce4e2bd84626c62f15d542ed2855aa9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 09960dc13ce4e2bd84626c62f15d542ed2855aa9: <one-sentence reason>

@pepmach
pepmach force-pushed the fix/gateway-live-holder-refusal-terminal-exit branch from f77aeb3 to ffbdca6 Compare September 27, 2026 20:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@pepmach
pepmach force-pushed the fix/gateway-live-holder-refusal-terminal-exit branch from ffbdca6 to 4c6d29c Compare September 27, 2026 21:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@pepmach
pepmach force-pushed the fix/gateway-live-holder-refusal-terminal-exit branch from 4c6d29c to c6e5e8e Compare September 27, 2026 22:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

AI-review override not recorded: keep the reason to 500 characters or fewer.

@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt c6e5e8e: The terminal exit rests on positive evidence only -- identified owner, alive, holding the port and answering HTTP at 127.0.0.1, the default bind (0.0.0.0 covers it); a holder reachable only elsewhere is not asserted live and keeps the pre-PR exit 1, capped by StartLimitBurst=3/StartLimitIntervalSec=300, nothing written or lost; guessing more addresses would widen the assertion this PR narrows on purpose; probing the holder's real listener address is tracked as #14527 entry 2 (Due 2026-10-09)

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@pepmach marked the gpt AI finding as false positive, not applicable, or explicitly accepted for c6e5e8eb014d2ce1cc7a7e00bb8af218a61297b1.

The terminal exit rests on positive evidence only -- identified owner, alive, holding the port and answering HTTP at 127.0.0.1, the default bind (0.0.0.0 covers it); a holder reachable only elsewhere is not asserted live and keeps the pre-PR exit 1, capped by StartLimitBurst=3/StartLimitIntervalSec=300, nothing written or lost; guessing more addresses would widen the assertion this PR narrows on purpose; probing the holder's real listener address is tracked as #14527 entry 2 (Due 2026-10-09)

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
@pepmach
pepmach force-pushed the fix/gateway-live-holder-refusal-terminal-exit branch from c6e5e8e to 8b0e1d4 Compare September 27, 2026 23:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@pepmach
pepmach force-pushed the fix/gateway-live-holder-refusal-terminal-exit branch from 8b0e1d4 to 1a7587d Compare September 28, 2026 01:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@pepmach

pepmach commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 1a7587d: The lock endpoint is resolved from the environment the gateway itself is about to bind with; a mise-injected port or bind the process env does not carry makes the verdict fall to exit 1 -- the documented residual row, the pre-PR behaviour, bounded by StartLimitBurst -- never to a false 78, and activating mise ahead of the lock resolution is a startup-ordering improvement tracked as #14527 entry 3 (Due 2026-10-09), not a correctness condition of this fix

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@pepmach marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 1a7587defa897320a72f8288818fd32b93254b2f.

The lock endpoint is resolved from the environment the gateway itself is about to bind with; a mise-injected port or bind the process env does not carry makes the verdict fall to exit 1 -- the documented residual row, the pre-PR behaviour, bounded by StartLimitBurst -- never to a false 78, and activating mise ahead of the lock resolution is a startup-ordering improvement tracked as #14527 entry 3 (Due 2026-10-09), not a correctness condition of this fix

This decision applies only to this commit. A new push requires a new judgment.

A supervised kirocrew gateway that finds its home's gateway.lock held by
a running gateway which also holds the dashboard port cannot do anything
useful: the sibling keeps serving, and a relaunch meets the identical
refusal. cli.py exited 1 for every lock failure, a status Restart=always
relaunches, so the shipped unit churned three times before StartLimit*
parked it failed, and a unit without StartLimit* churned without bound.

Give that one refusal its own terminal status. GatewayLockError now
carries a structured live_holder verdict, True only when a running
process holds both the lock and the port this gateway was about to bind;
cli.py exits gateway_lock.LIVE_HOLDER_EXIT_CODE (78, EX_CONFIG) on it and
keeps exit 1 for every other refusal, and render_unit() emits
RestartPreventExitStatus= with the same constant, the way the pod unit
exempts its terminal boot codes. Restart=always and the StartLimit* lines
are unchanged, so the watchdog's and listener guard's own relaunch
requests (75, 69) and a transient lock failure are still restarted.

A live acquirer that does not hold the port stays restartable on
purpose: it is a sibling still starting or one shutting down, and the
next attempt resolves either.

Linux/systemd only. The exit status changes on every platform, but only
the systemd unit exempts it: the macOS launchd plist keeps KeepAlive=true
and relaunches every exit, so a launchd-supervised gateway meets this
refusal exactly as it does today -- that half is deferred, not attempted
here, because the plist relaunches exit 0 on purpose and needs its own
success/failure design. The desktop supervisor and the stale-asset
watchdog's own exit path are untouched as well. Installed units are
written once by service install and are not re-rendered on upgrade, so a
unit from an earlier build keeps relaunching until it is re-rendered.
@pepmach
pepmach force-pushed the fix/gateway-live-holder-refusal-terminal-exit branch from 1a7587d to 09960dc Compare September 28, 2026 03:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 28, 2026 05:32
@iamwhatever
iamwhatever merged commit 9d45d1e into main Sep 28, 2026
100 checks passed
@iamwhatever
iamwhatever deleted the fix/gateway-live-holder-refusal-terminal-exit branch September 28, 2026 05:33
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gateway: live-holder lock refusal exits 1, so a Restart=always service churns and parks failed instead of standing down

2 participants