Repository navigation
fix(service): let a live-holder lock refusal stand the unit down - #14525
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify follow-up issue #14527 exists — the residual bound-elsewhere row deliberately keeps restart-looping on the strength of that tracked deferral. What this change shipsInventory (9 items) — 9 justifiedIntent: a supervised gateway that finds its home already served by a live sibling stands down once instead of restart-looping — a FIX (reproduction in description: 26,316 counted restarts; red-first tests).
Mechanism check: copies, not duplicates, the pod convention — [FIRST-PRINCIPLES-REVIEWED] 09960dc |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify once under a live systemd user manager that exit 78 parks the unit [DESIGN-REVIEWED] 09960dc |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
f77aeb3 to
ffbdca6
Compare
ffbdca6 to
4c6d29c
Compare
4c6d29c to
c6e5e8e
Compare
|
AI-review override not recorded: keep the reason to 500 characters or fewer. |
|
/ai-review override gpt c6e5e8e: The terminal exit rests on positive evidence only -- identified owner, alive, holding the port and answering HTTP at 127.0.0.1, the default bind (0.0.0.0 covers it); a holder reachable only elsewhere is not asserted live and keeps the pre-PR exit 1, capped by StartLimitBurst=3/StartLimitIntervalSec=300, nothing written or lost; guessing more addresses would widen the assertion this PR narrows on purpose; probing the holder's real listener address is tracked as #14527 entry 2 (Due 2026-10-09) |
Human judgment recorded@pepmach marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
c6e5e8e to
8b0e1d4
Compare
8b0e1d4 to
1a7587d
Compare
|
/ai-review override gpt 1a7587d: The lock endpoint is resolved from the environment the gateway itself is about to bind with; a mise-injected port or bind the process env does not carry makes the verdict fall to exit 1 -- the documented residual row, the pre-PR behaviour, bounded by StartLimitBurst -- never to a false 78, and activating mise ahead of the lock resolution is a startup-ordering improvement tracked as #14527 entry 3 (Due 2026-10-09), not a correctness condition of this fix |
Human judgment recorded@pepmach marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
A supervised kirocrew gateway that finds its home's gateway.lock held by a running gateway which also holds the dashboard port cannot do anything useful: the sibling keeps serving, and a relaunch meets the identical refusal. cli.py exited 1 for every lock failure, a status Restart=always relaunches, so the shipped unit churned three times before StartLimit* parked it failed, and a unit without StartLimit* churned without bound. Give that one refusal its own terminal status. GatewayLockError now carries a structured live_holder verdict, True only when a running process holds both the lock and the port this gateway was about to bind; cli.py exits gateway_lock.LIVE_HOLDER_EXIT_CODE (78, EX_CONFIG) on it and keeps exit 1 for every other refusal, and render_unit() emits RestartPreventExitStatus= with the same constant, the way the pod unit exempts its terminal boot codes. Restart=always and the StartLimit* lines are unchanged, so the watchdog's and listener guard's own relaunch requests (75, 69) and a transient lock failure are still restarted. A live acquirer that does not hold the port stays restartable on purpose: it is a sibling still starting or one shutting down, and the next attempt resolves either. Linux/systemd only. The exit status changes on every platform, but only the systemd unit exempts it: the macOS launchd plist keeps KeepAlive=true and relaunches every exit, so a launchd-supervised gateway meets this refusal exactly as it does today -- that half is deferred, not attempted here, because the plist relaunches exit 0 on purpose and needs its own success/failure design. The desktop supervisor and the stale-asset watchdog's own exit path are untouched as well. Installed units are written once by service install and are not re-rendered on upgrade, so a unit from an earlier build keeps relaunching until it is re-rendered.
1a7587d to
09960dc
Compare
Problem / Motivation
Goal: A
kirocrew gatewayservice unit that finds its home already served by another running gateway stands down once, with the reason in the journal, instead of relaunching against that refusal.Bug fix. When a systemd-supervised
kirocrew gatewaystarts against aKIROCREW_HOMEwhosegateway.lockis held by a live gateway that also holds the dashboard port, it prints👻 <home>/gateway.lock is held by pid <PID> (… holds port 5476, answering HTTP) -- another gateway already owns <home>; …and exits 1.Restart=alwaysrelaunches it everyRestartSec, and each relaunch boots most of the stack only to meet the same refusal. On the shipped unitStartLimitBurst=3parks the unitfailedafter three rounds; on a unit withoutStartLimit*the loop is unbounded (the reporting host counted 26,316 restarts). The healthy sibling keeps serving throughout, so nothing ever recovers the looping unit.Why it matters
Every operator whose host ends up with two controllers for one home — a hand-run gateway beside the service, a second supervisor, a post-update relaunch into a home the previous gateway still holds — gets a unit that is either parked
failedafter three noisy boots or churning forever, withstatus=1/FAILUREtelling them nothing about why. The refusal itself is correct; the supervisor just cannot tell a permanent "this home is already served" apart from a crash worth retrying.Not a goal
service/macos.pykeepsKeepAlive=trueand relaunches every exit, so a launchd-supervised gateway meets this refusal exactly as it does today. Deferred: the plist relaunches exit 0 on purpose (the stale-asset watchdog relies on it) and needs its own success/failure design, as the pod'slaunchd_exit_codetranslation shows.status=75exit that triggered the reporting host's loop (issue Stale-asset watchdog shuts down the gateway mid-update and reaps its own installer, leaving a venv with no console scripts #13521). Both are separate mechanisms with their own issues.service installwrites the unit once and no upgrade re-renders it, so a unit written by an earlier build does not carryRestartPreventExitStatus=until it is re-rendered. A currency check for the gateway unit (the pod'sunit_is_currentshape) is new install/status scope, not part of this fix; the gap is stated in the spec.What changed (motivation → approach → change)
Observed symptom: every lock refusal exits 1 (
cli.py, theGatewayLockErrorcatch on the gateway path), and the rendered unit hasRestart=alwayswith noRestartPreventExitStatus, so systemd retries a refusal the sibling keeps permanent.Root cause: the exit status carried no information about whether a retry could help. The pod plane already has the convention (
pod/config.pyterminal boot exit codes fed intoRestartPreventExitStatus=bypod/unit.py); the gateway unit had no counterpart.Change, copying that convention:
gateway_lock.pydefinesLIVE_HOLDER_EXIT_CODE = 78(EX_CONFIG: two supervisors pointed at one home is a host configuration; the remedy is to change it, never to try again), distinct from 69 (listener lost) and 75 (stale assets), which the running gateway exits with on purpose to be relaunched. It lives beside the refusal becausecli.pyreturns it andservice/linux.pyexempts it — one definition, no drift.GatewayLockErrorgains a structuredlive_holderflag, and ONE predicate decides it:GatewayLock._serving_verdict, asked only about the process/proc/lockspositively identifies as the lock's acquirer (lock file, or home anchor when the file was deleted or replaced), never about the pid the lock file merely records. Four conjuncts, each measured once: alive; holding the configured dashboard port (one listener enumeration,platform_compat.find_port_listeners, filtered to the owner's own sockets); holding it AT the address the probe reaches — this gateway's configured bind (KIROCREW_BINDwhen an IP; none or0.0.0.0→127.0.0.1,::→::1because the dashboard binds itIPV6_V6ONLY), wildcards covering by family only, an unreported address counting as unknowable; and answering HTTP there. The address conjunct ties port ownership and HTTP health to ONE process, so a stranger answering at the probe address is never credited to an owner bound elsewhere. Only all four makelive_holderTrue. Everything else stays False and exits 1: an identity race, an unmeasurable home anchor, an orphaned flock, a live acquirer without the port, one whose own socket at the probed address is silent (a wedged gateway), one holding the port only elsewhere or at an unreported address (the residual row, unasserted by design; the message names what was measured, and probing the holder's own address is #14525 follow-up: service unit staleness self-heal for RestartPreventExitStatus #14527's follow-up), and every refusal with no owner surface (no/proc/locks, or a filesystem the lock table never matches), where the predicate is never asked. With no port to weigh (--port auto,--slack-only) every refusal stays restartable.cli.pyexitsLIVE_HOLDER_EXIT_CODEwhenexc.live_holderis set and keepssys.exit(1)for every other refusal; it hands the lock the configured bind (bind_address_for) beside the port.service/linux.py::render_unit()emitsRestartPreventExitStatus=78right afterRestartSec=10, in both the system and the--userscope.Restart=always,RestartSec=10and bothStartLimit*lines are untouched, so a clean self-exit, the watchdog's 75, the listener guard's 69 and a transient lock failure are all still relaunched.docs/system-specs/modules/cli.md(Service Management) documents the predicate by name, its three conjuncts, the residual row, what stays restartable, and the installed-unit gap with its remedy (kirocrew service install);docs/system-specs/modules/slack-gateway.md's exit-status table gains the 78 row.Backwards compatibility
Compatible: nothing that works today stops working. The only observable differences are a different non-zero exit status (78 instead of 1) for the one refusal where the process
/proc/lockspositively identifies as holding the lock is running, holds the port and answers HTTP on it, a hedge sentence added to the refusal message when no surface can identify the holder or when the identified holder listens without answering HTTP (same exit 1 as before), and one addedRestartPreventExitStatus=line in units rendered from now on. A unit written by an earlier build keeps its current behaviour (exit 78 is as non-zero as exit 1 was, soRestart=alwaysstill relaunches it) until it is re-rendered.GatewayLockError's constructor gains a keyword-only argument with a default; every existing construction site is unchanged.Tests
Red-first, by file, run with
-n0against this branch's source and then against main's source (git stashofsrc/with the tests kept):test/test_cli.py::TestGatewayLockRefusalExit(32 tests):test_live_holder_truth_table[...]is the table-driven pin of the serving-holder predicate — 18 rows judged by the REAL diagnosis and the realkirocrew gatewayexit: identified acquirer alive, its own socket at the probed address, answering HTTP (78), the same acquirer silent (1), without the port (1), dead (1), with no port configured (1); the address conjunct — a stranger answering at the probe address while the owner holds the port on another address (1), an owner whose socket address the platform did not report (1); a configured0.0.0.0bind probed at loopback (78), a configured::bind probed at::1(78), a configured::1bind probed there (78); the residual row, an owner holding the port only at::1while this gateway binds loopback (1); no owner surface with the recorded pid alive and on the port, alive without the port, or dead (1 each, and the probe is never made); and the deleted-lock-file path for an answering acquirer (78), a silent one (1), a stranger at the probe address (1) and no identified owner (1).test_diagnostic_bind_address_is_the_bind_the_gateway_resolves[...]pins the lock's bind todashboard.urls.bind_address_forfor nine override values. The named regressions stay: a hand-made live-holder refusal exits 78 and never 0 or 1; every other refusal exits 1; an unidentified owner on the port exits 1; an identified owner on the port exits 78 only when it answers HTTP. Red on main:ImportError: cannot import name 'LIVE_HOLDER_EXIT_CODE'/AttributeError … 'live_holder'; the silent, residual, stranger and unidentified rows are also red against a diagnosis that decides on port ownership alone.test/test_service.py::TestLinuxUnitRendering::test_render_unit_exempts_the_live_holder_refusal_from_restart[False|True]: the rendered unit carries exactly oneRestartPreventExitStatus=line, in[Service], whose set is exactly{LIVE_HOLDER_EXIT_CODE}; the constant is pinned to 78 (it is baked into installed units); the set never includes 0, 1, 69 or 75;Restart=always,RestartSec=10and bothStartLimit*lines are still present. Red on main:ImportError: cannot import name 'LIVE_HOLDER_EXIT_CODE'.test/test_gateway_lock_diagnosis.py:live_holderpinned per diagnosis branch —/proc/locksacquirer alive, holding the port and answering HTTP → True; live acquirer holding the port but not answering → False, message namesnot answering HTTP at 127.0.0.1(the probed address),wedgedandnot treated as permanent, never offerskill, and the probe runs exactly once (test_a_live_acquirer_listening_but_not_answering_http_is_not_a_live_holder);probe_host_for_bindmaps no override and0.0.0.0to127.0.0.1,::to::1, and passes a specific address through, bracketed IPv6 included (test_the_predicate_probes_the_configured_bind_with_wildcards_mapped_by_family);_listener_reachesis pinned row by row — a specific bind reaches only itself (v4-mapped spelling included),0.0.0.0every v4 host,::every v6 host and no v4 one, lsof's*by its reported family, and an unreported address or family never (test_a_listener_reaches_the_probe_host_only_when_its_bind_covers_it); live acquirer without the port → False (also covers the no-port case); dead acquirer with a single inheritor → False; no owner surface with the recorded pid running and holding the port → False, with the message hedging that the holder cannot be confirmed and the refusal is not treated as permanent, never naming systemd (the branch is the non-Linux path too) and never wording it asheld by pid(test_refusal_hedges_when_a_live_port_holding_pid_has_no_owner_surface); recorded pid without the port, no port supplied, recorded pid gone → False. Red on main:AttributeError … 'live_holder'; the no-owner-surface and not-answering tests are also red against a diagnosis that decides on port ownership alone.test/test_gateway_lock.py: two Linux-only real-process tests, each parametrized over the HTTP answer (/proc/locksnames this process as the acquirer of a REAL held lock; the listener enumeration is pinned to this pid at loopback so the host'slsofnever decides the verdict, and a bound loopback socket supplies the port number) — a refusal by the live acquirer on the port is a live holder only when it answers HTTP, otherwise the message sayswedgedand the refusal stays restartable, and the same lock without a port is never a live holder; a home still anchored by that acquirer after the lock file is deleted follows the same rule — plus the identity-race refusal (being replaced faster) pinned as not a live holder. Red on main:AttributeError … 'live_holder'; the not-answering variants are also red against a diagnosis that decides on port ownership alone.render_unittests (test_service.py,test_service_selinux.py) stay green:test_service.py408 passed / 3 skipped,test_service_selinux.py50 passed,test_gateway_lock.py64 passed / 1 skipped,test_gateway_lock_diagnosis.py63 passed,test_cli.py::TestGatewayLockRefusalExit32 passed, each file run alone with-n0.The two
TestGatewayLockRefusalExittests patchfaulthandler.enablebecause the gateway path arms it on the real stderr descriptor, which a capsys stream lacks;cli.pyandcrash_guard.pyboth call it through the onefaulthandlermodule object, so the single patch covers both arm sites (the class passes when run alone, with a cold crash guard).Manual verification
systemd-analyze verifyonrender_unit(user_scope=True)written to a file exits 0 with no complaint about the new directive, while a deliberately misspelledRestartPreventExitStatusX=in the same file drawsUnknown key name … in section 'Service', ignoring— so the check is real.KIROCREW_HOME,kirocrew gateway --no-open --no-crons --port 5599, against a scratch holder process that acquires the home's lock and occupies 5599: holder ANSWERING HTTP (a realhttp.serverreturning 200) → exit code 78, stderr👻 <home>/gateway.lock is held by pid <PID> (2 threads, holds port 5599, answering HTTP at 127.0.0.1) -- another gateway already owns <home>; stop it first (kirocrew stop) or set KIROCREW_HOME to an isolated directory; holder LISTENING but never accepting (the wedged shape) → exit code 1, stderr👻 <home>/gateway.lock is held by pid <PID> (1 thread, holds port 5599, not answering HTTP at 127.0.0.1) -- it is listening on port 5599 but not answering HTTP at 127.0.0.1 (the address this gateway would bind), so it looks like a wedged gateway rather than one serving this home; this refusal is not treated as permanent, and a supervisor, if one manages this gateway, will retry it. Stop it first (kirocrew stop) or set KIROCREW_HOME to an isolated directory; holder that holds the lock only → exit code 1, stderr… (1 thread, does not hold port 5599) ….systemd --usermanager (this sandbox has no user bus). The exemption rests on systemd'sRestartPreventExitStatus=semantics, the same mechanism the pod unit already relies on.Success: no issues found in 1738 source files), comment-history gate,scripts/docs-lint.shall clean.scripts/local-gate.py(backend related set, 52,286 tests): 52,285 passed, 1 failed —test_home_pin_survives_monkeypatch_undo.py::test_the_guard_accepts_a_scratch_home, which fails identically on an untouched main checkout in this sandbox (the guard sees the sandbox's liveKIROCREW_HOMEenvironment), so it is inherited, not caused here.f77aeb3082(the head the server lanes judged; round 1 below changedgateway_lock.py, two test files and the two spec pages after it):No findings.faulthandlerpatch covers only one of two arm sites: rebutted, both sites call the same module attribute and the class passes alone with a cold crash guard; (2) the added spec table row ended mid-clause: fixed in this head; (3) installed units are not re-rendered: declined as out of goal, stated under Not a goal and in the spec.f77aeb3082): GPT FINDINGsrc/kiro_crew/gateway_lock.py:473— fixed:_describe_unidentified_ownercan no longer return a live-holder verdict; the truth table now reads "live_holderTrue only for a/proc/locks-identified acquirer that is alive and on the port; every unidentified-owner shape (no/proc/locks, or a filesystem it never matches) is False, keeps its hedge and exits 1", with the docstrings andcli.mdupdated to match. Opus FINDINGsrc/kiro_crew/service/linux.py:333(installed units are not re-rendered) — accepted and deferred to #14525 follow-up: service unit staleness self-heal for RestartPreventExitStatus #14527 (deferred-finding, Due 2026-10-09);cli.mdnames the interim remedy.ffbdca652f): Opus FINDINGsrc/kiro_crew/gateway_lock.py:504— fixed: the unidentified-owner hedge no longer names systemd (the branch is reached on macOS and Windows on every refusal); it reads "…so this refusal is not treated as permanent; a supervisor, if one manages this gateway, will retry it", the test asserts the platform-neutral phrase, andcli.mddescribes the message the same way. First Principles item 3 ("rides along: the hedge rewording") — declared and kept: the message states exactly what the exit code asserts. Rebased ontof57c1dee40(fix(ci): stop three gates from reddening unrelated pull requests #14518, which also clears the three main-inherited CI reds this PR cited).4c6d29ca24): GPT BLOCKINGsrc/kiro_crew/gateway_lock.py:572(an unresponsive listener permanently disables recovery) — fixed:live_holderrequires the identified acquirer to hold the port AND answer HTTP, in both_describe_live_ownerand_diagnose_replaced_lock_file; one listener scan and at most one HTTP probe per diagnosis (_port_verdict) feed both the message and the verdict; the listening-but-silent shape sayswedged … not treated as permanentand exits 1; truth tables in the docstrings,cli.mdandslack-gateway.mdupdated; red-first tests in all three test files, and the manual verification redone with an answering holder (78) and a silent one (1). Opus FINDINGtest/test_gateway_lock.py:270(the two real-process tests let the host'slsofdecide the verdict) — fixed in the same tests: the listener scan is pinned to this pid, the bound socket only supplies the port number.c6e5e8eb01): three truth-table rows on three heads met the restructure bar, so the verdict now lives in ONE predicate,GatewayLock._serving_verdict(a frozen_ServingVerdict: alive, holds the configured port, answers HTTP atprobe_host,serving= all three), called by both_describe_live_ownerand_diagnose_replaced_lock_fileand never for an unidentified owner; the probe goes to the address this gateway is configured to bind (cli.py::_diagnostic_bind_address→probe_host_for_bind, wildcard → loopback), which makes the residual row explicit in code: a holder bound to another address is not asserted, exit 1 by design (GPT r3gateway_lock.py:695, overridden on that head and tracked as #14525 follow-up: service unit staleness self-heal for RestartPreventExitStatus #14527 entry 2). Same rows, same verdicts as round 3 for a loopback or wildcard bind; a specificKIROCREW_BINDis now probed at that address instead of at loopback. Table-driven testtest_live_holder_truth_table(14 rows, real diagnosis + real exit); manual verification re-run: answering holder 78, silent holder 1. Rebased onto61e100dedd.8b0e1d42a3): GPT BLOCKINGsrc/kiro_crew/gateway_lock.py:808(port ownership and HTTP health can belong to different processes) — fixed inside the predicate: the port conjunct is address-bound, the owner's OWN socket must be the one the probe reaches (_listener_reaches: wildcards cover by family,::never covers v4 because the dashboard binds itIPV6_V6ONLY, an unreported address is unknowable → False), so a stranger's answer at the probe address is never credited to the owner; GPT FINDING:243and Opus FINDING:225(::probed at IPv4 loopback) — fixed:::→::1; Opus FINDING:263(the one-probe claim is false on the orphaned-lock path) — the claim is qualified to the identified-acquirer paths in the docstring andcli.md; First Principles item 7 (_diagnostic_bind_addressduplicatedbind_address_for) — fixed: the copy is deleted and the lock receivesbind_address_for(local_only=True). Table rows added: stranger at the probe address, owner address unknowable,::incumbent answering on::1, and the stranger row on the deleted-lock-file path (18 rows). Same rows, same verdicts otherwise; CI redIntegration (in-process gateway)is main's (test_route_sweep.py, red on main sincede25db109a); rebased onto972675c207.Screenshots / video
no-visual-delta
Why no screenshot: backend-only change — an exit status, a systemd unit line and two spec documents; no user-visible frontend surface is touched.
Related Issues
Fixes #13914
Pattern harvest
Rule candidate: review-prompt
Pattern: a refusal caught on a supervised entry point exits with the generic status (
sys.exit(1)) without classifying whether a retry can heal it, so aRestart=supervisor churns on a standing condition; the fix shape is the pod's — a terminal-code constant beside the refusal, fed intoRestartPreventExitStatus=.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement