macOS: ocx service repair unconditionally evicts the running LaunchAgent, accepts a no-op launchctl load as success, and leaves the hub down with no log line #4236

Severity: high — a single ocx service repair can take a hub fully offline (public proxy + management ingress + loopback client listener) with no error shown, no rollback, and no automatic recovery. Recovery needs a launchctl command the CLI never prints.

Environment

• opencodex 2.51.0 (source checkout run via bundled Bun 1.4.2)
• macOS 26 / Darwin 27.0.0, arm64
• runtimeRole: "hub", hostname: "100.76.170.81" (Tailscale-only public bind), port: 10100
• hub.managementIngress: { enabled: true, port: 10102 }
• unauthenticatedLoopbackListener: { enabled: true, port: 10104 }
• LaunchAgent com.opencodex.proxy at ~/Library/LaunchAgents/com.opencodex.proxy.plist, KeepAlive: true

All line references are to the source checkout at commit 6d3ad12e3 (main, after PR #4196).

Summary

Four defects, all on the macOS launchd path. (1) is the outage. (2) can send an operator into (1) on a false premise. (3) and (4) turn permanently-fatal config states into silent KeepAlive crash loops / false warnings.

1. installLaunchd() evicts the live job with a domain-explicit bootout, re-registers with the domain-implicit legacy launchctl load -w, and accepts exit-0-with-empty-stderr as success — then writes install state. No pre-check that the job is already healthy, no settle delay, no bootstrap fallback, no plist backup, no rollback. macOS is the only backend with "evict first, verify never"; the Windows branch of the same function has a full preserve/restore protocol.
2. diagnoseService() derives "loaded" from sh("launchctl list | grep <label> || true") — session-relative domain, and || true plus a catch swallow every exit code, so every failure mode collapses into "not loaded". That single bit also drives enabled, running, viable and therefore isServiceViable(), which the update path uses to decide whether to start a competing proxy.
3. An occupied unauthenticatedLoopbackListener.port / hub.managementIngress.port is a fatal startup crash that is then misdiagnosed as a public port conflict, so the retry waits on the wrong port and the job KeepAlive-loops with stdio: "ignore" upstream.
4. ocx status compares client fence ports against the public listen port only, ignoring unauthenticatedLoopbackListener.port, so it warns about the very port opencodex's own startup just wrote — and recommends ocx ensure, which on a non-loopback-hostname hub would repoint local clients at an address they cannot reach.

Repro

Hub with a non-loopback hostname, management ingress enabled, unauthenticatedLoopbackListener enabled, service installed and healthy and serving traffic.

# before: 100.76.170.81:10100, 127.0.0.1:10102, 127.0.0.1:10104 all LISTEN
ocx service repair
ocx start --port 10100     # also attempted; see note

Expected: repair rewrites the plist and leaves the job bootstrapped in gui/$uid and serving. On any failure it restores the previously running registration, or exits non-zero naming the teardown explicitly and the command needed to recover.

Actual: all three listeners gone.

• ~/Library/LaunchAgents/com.opencodex.proxy.plist present and plutil -lint-clean.
• ~/.opencodex/service-api-token rewritten with the same mtime as the plist — i.e. installLaunchd() definitely ran to at least :2370.
• launchctl list shows no com.opencodex.proxy.
• Nothing appended to ~/.opencodex/service.log after the eviction, because the job never started again. The outage leaves no log trace at all, which is what made it undiagnosable.
• ocx status: ❌ Proxy: not running / Service: installed, not loaded (launchd) / re-run 'ocx service repair' — i.e. it recommends the command that caused this.
• Every endpoint refuses connections, including remote clients arriving over Tailscale/cloudflared.

Recovery required exactly one command, which ocx never issues or suggests anywhere:

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.opencodex.proxy.plist

That restored all three listeners from the unmodified plist and the unmodified config, first try, runs = 1, no crash loop. Nothing was wrong with the artifacts repair wrote — only the (re)load verb and the absent verification.

Note on ocx start --port: it is not the tear-down agent. handleStart never touches launchd, and with an explicit --port it sets killOcxHolders: false (src/cli/index.ts:210-221) and refuses to hop. It is implicated only as the command that then fails to recover the evicted state. The kill-then-repair-then-maybe-nothing sequence lives in the update paths — see "Related: update fallback" below.

Defect 1 — installLaunchd(): unconditional teardown, unverified reload, no rollback

ocx service repair on darwin is installLaunchd():

src/service.ts:3214-3217

if (platform === "darwin") {
  (deps.repairLaunchd ?? installLaunchd)();
  return;
}

src/service.ts:2356-2412

2360  recordOwnedConfigPath(getConfigDir(), serviceStatePath());
2362  writeServiceApiTokenFile();
2369  const launcher = stableLauncherEntry();
2370  writeServiceDefinitionFile(p, buildPlist(resolvedProxyEnv(), { launcher }), "utf8");
2385  const bootoutTarget = `${launchdGuiDomain()}/${LABEL}`;
2386  run(["bootout", bootoutTarget]);          // EVICTS the running hub
2387  let loaded = run(["load", "-w", p]);      // legacy, domain-implicit
2388  if (launchctlLoadFailed(loaded.stderr)) { run(["bootout", bootoutTarget]); loaded = run(["load", "-w", p]); }
2396  if (!loaded.ok || launchctlLoadFailed(loaded.stderr)) { throw new Error(...) }
2411  writeServiceInstallState("scheduler", launcher);

1a. Asymmetric domain targeting

bootout is addressed to gui/$uid via launchdGuiDomain() (src/service.ts:968-970). load -w is not addressed at all — legacy load acts on the caller's bootstrap domain. Any invocation from outside the Aqua session (ssh, a user/$uid-bootstrapped parent, a cron/daemon context, a tray helper in another bootstrap context) deletes the gui-domain job and registers nothing there. launchctl bootstrap gui/$uid <plist> is the verb that pairs with the bootout already in use.

The codebase knows these domains are disjoint — src/service-manager-probe.ts:334-352:

"Measured on macOS 27.0: the shipped agent answers 0 under gui/<uid> and 113 under user/<uid>. Asking only one leaves the other free to hold a job"

Confirmed on the affected machine: launchctl print gui/501/com.opencodex.proxy → exit 0; launchctl print user/501/com.opencodex.proxy → exit 113.

1b. Success is inferred from a stderr regex, never verified

launchctlLoadFailed() (src/service.ts:964-966) is /\b(?:Load|Bootstrap) failed\b/i over stderr. Deprecated load can exit 0 with empty stderr and do nothing — which satisfies the guard at :2396 — and it can also emit a diagnostic on a load that did work, which converts a success into a throw-and-leave-it-down.

The correct probe already exists and is already used by the sibling function:

• launchdJobMatchesPlist() (src/service.ts:979-991) runs launchctl print gui/$uid/<label> and compares the live arguments.
• startLaunchd() cross-checks with it before throwing (src/service.ts:2427-2447, call at :2434-2439).
• installLaunchd() never calls it — despite being the only path that evicts first.

Its own header comment at src/service.ts:973-978 says list "only proves domain membership" and print is "the only way to catch a load that silently no-op'd". The install path skips precisely that check.

1c. The teardown is unconditional

repairService() validates supported / conflict / installed (src/service.ts:3015-3026) and then never consults diag.running or diag.viable before delegating to a function whose own comment (:2373-2383) says it "EVICTS the running job". A repair of an already-healthy service is therefore guaranteed to interrupt service, with recovery depending entirely on 1a/1b.

1d. No settle delay, so the retry is useless

bootout is asynchronous. Both load attempts fire back-to-back (:2386-2389), so a job still exiting yields Load failed: 5: Input/output error twice and the "bounded retry" adds nothing. The Windows path has SCHEDULER_SETTLE_DELAYS_MS / settleDelay (src/service.ts:1477, :3162-3169); launchd has neither that nor a kickstart -k fallback.

1e. No plist backup and no rollback — unlike every other backend

writeServiceDefinitionFile (src/service.ts:2470-2496) overwrites in place with no backup, at :2370, before the eviction. If both loads fail, the function throws at :2396: the evicted job is not re-bootstrapped, the previous plist is gone, and writeServiceInstallState at :2411 is skipped. Terminal state = plist on disk, nothing in launchd, nothing listening.

Compare Windows, same file: repairService preserves and restarts the still-registered definition on failure (src/service.ts:3101-3186), with installFreshWindowsSchedulerSafely (:3878-3960) and rollbackWindowsSchedulerTaskOwnedByAttempt (:1301). The comment there states that leaving a previously runnable proxy stopped would make the user "worse off than before the repair". darwin has none of this.

1f. The failure never reaches the verification step

serviceCommand's repair branch (src/service.ts:4581-4589) has no try/catch, so a throw from installLaunchd escapes through src/cli/dispatch.ts:564-570 to the top level and reportServiceServing("repaired") is never reached. Even when it is reached, it only prints and sets process.exitCode = 1 (:790-819) — it never retries or reloads.

1g. The plist is silently rewritten from the caller's PATH

stableLauncherEntry() (src/service.ts:102-124) resolves ocx from the invoking shell's PATH. A repair run from a context without ocx on PATH rewrites a working launcher-form plist into the bun+CLI pair (expectedLaunchdCommand / cliEntry, :614-636, :73-100) — and then boots out the healthy job to load it. Nothing checks the new command is runnable, and there is no rollback if it isn't.

Related latent trap: expectedLaunchdCommand's fallback at :614-636 means that if ~/.opencodex/service-state.json is ever lost, status and start compare a healthy launcher-form job against the bun+CLI pair and report "launchd is running an OLDER plist" (:2444, :4409-4412), with startLaunchd throwing instead of no-op'ing.

1h. The plain install path has the same shape, with a verb that cannot work

serviceCommand install (:4608-4627) → installServiceSafely (:3816) → prepareServiceInstall (:3783-3814) → darwin ServiceInstallCleanupOps (src/service.ts:3628-3637):

status: () => { const listing = sh("launchctl list"); return listing.split("\n").some(line => line.includes(LABEL)) ? listing : null; },
stop:   () => { sh(`launchctl unload "${plistPath()}"`); },

That stop uses legacy unload, which installLaunchd's own comment at :2371-2375 says does not evict a gui-domain job. Teardown first, then installLaunchd with its own bootout; any throw surfaces as ❌ Service install cleanup failed + exitCode = 1 (:4625-4628) with nothing restored.

Suggested fix

1. Skip the bootout entirely when launchdJobMatchesPlist() already reports { loaded: true, matchesPlist: true } and the newly rendered plist is byte-identical — a repair of a healthy service should be a no-op, not an outage.
2. Replace run(["load", "-w", p]) with run(["bootstrap", launchdGuiDomain(), p]) — the verb that matches the bootout target — and add a settle delay between bootout and bootstrap, plus a kickstart -k gui/$uid/<label> fallback.
3. Require launchdJobMatchesPlist(expectedLaunchdCommand(installedServiceListenPort())) to confirm { loaded: true, matchesPlist: true } before writeServiceInstallState(...); treat anything else as failure regardless of stderr.
4. Back up the previous plist before overwriting at :2370; on terminal failure restore it, re-bootstrap, and only then throw — with an error that says the job was evicted and is currently down and names launchctl bootstrap gui/$uid <plist> as the manual remedy.
5. Wrap the repair branch at :4581 so the serving verification runs even on a thrown repair.

Defect 2 — the "loaded" bit is a swallowed, session-relative string grep

src/service.ts:2449

function statusLaunchd(): string { try { return sh(`launchctl list | grep ${LABEL} || true`); } catch { return ""; } }

src/service.ts:4321-4330 (diagnoseService(), darwin branch) — consumed by ocx status at src/cli/status.ts:216-222

const installed = existsSync(plistPath());
const running = installed && Boolean(statusLaunchd());
...
: running ? `installed and loaded (launchd; ${diagnostics})`
          : `installed, not loaded (launchd; ${diagnostics})`;

• Session-relative domain. Bare launchctl list enumerates the caller's own domain, not gui/$uid, while every mutating/inspecting call in the file targets gui/$uid explicitly. From a non-Aqua session a healthy gui-domain job is invisible → installed, not loaded + "re-run ocx service repair" for a hub that is serving traffic. Following that advice triggers Defect 1.
• Every exit code is discarded, twice. || true makes the shell exit 0 whatever launchctl or grep did, and catch { return "" } eats the rest. A bootstrap-server/EPERM error, a missing grep, or an execSync maxBuffer overflow on a host with thousands of agents all collapse into the same empty string as genuine absence. There is no tri-state: diagnoseService has no "unknown", unlike the Windows probe (probeWindowsSchedulerTask → "unknown", src/service.ts:1171) and unlike runLaunchctl, which deliberately preserves the numeric status specifically to separate 112 (no such domain) from 113 (no such service) (src/service.ts:944-956).
• Its own twin fails the opposite way. The install-cleanup probe for the identical question omits || true and therefore throws (src/service.ts:3630-3633). Two probes of one fact with opposite failure semantics.
• Unanchored regex — the inverse false positive. grep ${LABEL} is an unquoted pattern with unescaped dots matched anywhere on the line, so com.opencodex.proxy.helper, or a foreign label matching com?opencodex?proxy, reads as ours → "installed and loaded" for a job that is not the proxy. Domain membership is also not serving: a job bootstrapped from an older plist, or listed but never bound, also reads as "loaded".
• The right probes exist and are not wired in. launchdJobMatchesPlist (:979) is used only for the stale-plist hint line in serviceStatusReport (:4399-4412); inspectLaunchd (src/service-manager-probe.ts:331, queries both domains and is 112/113-aware) is used only for ownership. Neither feeds diagnoseService.
• Not a locale problem. The locale-sensitive decoding in this file is Windows-only (decodeSchtasksOutput :999, windowsSchedulerCsvIncludesTask :1149). Domain scope and swallowed exit status are the causes.

Blast radius beyond the cosmetic string. enabled, running and viable all come from that one bit (:4330), so isServiceViable() (:4175) returns false for a healthy loaded job — and that is what drives the update fallback at src/update/index.ts:372-380, which then treats a successful repair as non-viable and starts a competing proxy.

Why it reads as self-contradictory. src/cli/status.ts:216-222 cross-checks the summary against the live health probe and appends "— registered but NOT serving" only when service.installed && !live. So when the probe succeeds and the grep fails, status prints ✅ Proxy: running immediately above a bare Service: installed, not loaded (launchd) with no reconciling suffix — which is exactly the originally reported symptom.

Honest scoping of this report: in the incident above the job genuinely was not loaded, so that particular message was accurate. The defect is the earlier, reproducible complaint — installed, not loaded printed while launchctl list showed a live pid — which 2a/2b explain, and which is dangerous precisely because the remedy it prints is Defect 1. This is also already recorded as known-but-deferred in the repo: devlog/_plan/260910_post249_round2/_research/4141.md ("statusLaunchd → print gui/<uid>/<label> … changes running / isServiceViable", filed under POLICY (not mechanical)) and devlog/_plan/260910_post249_round2/040_4141_launchctl_bootout.md:58 ("statusLaunchd stays launchctl list | grep").

Suggested fix

Drive the darwin branch from runLaunchctl(["print", ${launchdGuiDomain()}/${LABEL}]) (or inspectLaunchd), keep the 112/113 distinction, query both gui/$uid and user/$uid, and report four states: not installed / not loaded / loaded from a stale plist / loaded from the current plist — plus an explicit unknown that does not recommend ocx service repair and does not make isServiceViable() false.

Defect 3 — an occupied secondary listener port is fatal and misdiagnosed

Both secondary listeners bind inside one startup transaction, after the public listener, with no prior availability probe:

src/server/index.ts:2405-2448

server = Bun.serve({ ...serveOptions, port: listenPort, hostname: bindHost });
if (loopbackListenerPort !== null) { try { loopbackServer = Bun.serve({ ..., port: loopbackListenerPort, hostname: "127.0.0.1" }); } catch (error) { void server.stop(true); throw error; } }
if (managementIngressPort !== null) { try { managementIngressServer = Bun.serve({ ... }); } catch ...

The pre-bind guard only stops the public listener from stealing the loopback port (src/cli/index.ts:193-203, reservedPort in src/server/ports.ts:70-100). Nothing checks that 10104 or 10102 is free. When one is held by a foreign process the transaction rolls back and rethrows, and handleStart cannot tell which listener failed:

src/cli/index.ts:367-380

if (!isAddrInUse(err) || attempt >= 2) throw err;
if (requestedPort !== undefined) { /* waits for the PUBLIC port to free */ }

So with --port (the launchd case) it waits on the wrong port, retries twice, then prints ❌ Port <public> stayed busy; refusing to hop and exits 1 — naming a port that was never the problem. Under KeepAlive: true that is a loaded job that never serves, relaunched forever.

This machine hit exactly that, historically, when the management ingress was configured on 10101 (held by an unrelated local service):

error: Failed to start server. Is port 10101 in use?
 code: "EADDRINUSE"
      at startServer (src/server/index.ts:2433)
      at handleStart (src/cli/index.ts:360)

repeated until the port was changed by hand. ocx status reported only installed, not loaded throughout.

Related: the schema at src/config.ts:1195-1198 is a discriminated union with .optional().catch(undefined), so a malformed hand edit silently disables the loopback listener instead of erroring. The relationship checks (loopbackListenerPortError, src/config.ts:2772-2797; ingress cross-check :2838-2841) are write-time only and cover internal collisions only — a foreign holder is never considered, at write time or at startup.

Suggested fix

Probe unauthenticatedLoopbackListener.port and hub.managementIngress.port for availability in the install / repair / start pre-flight and fail with the holding process named; and tag the rethrown bind error with which listener failed so handleStart stops blaming (and waiting on) the public port. A config error that can never succeed on retry should not be handed to KeepAlive.

Defect 4 — false "fence drift" warning against the loopback listener, with a destructive remedy

With the hub healthy, ocx status prints:

Grok Build config points at port 10104, but the proxy is on 10100;
grok turns will retry against a closed port. Run 'ocx ensure' to repoint it.

10104 is the unauthenticatedLoopbackListener port and it answers GET /v1/models with HTTP 200. opencodex's own startup wrote that value: the same service.log startup block contains both ⚠️ Unauthenticated loopback listener active on http://127.0.0.1:10104 and + Grok Build config updated (~/.grok/config.toml).

• src/cli/status.ts:283-294 calls grokFenceEndpointDrift(readGrokStatus(), health.ok ? listen.port : undefined) — public listen port only.
• src/grok/status.ts:119-133 then flags any mismatch: if (!Number.isFinite(fencePort) || fencePort === livePort) return null;

Neither side knows about config.unauthenticatedLoopbackListener.port. On a hub whose public bind is a Tailscale address, 10100 is unreachable from loopback by design — that is the entire reason the 10104 listener exists — so running the suggested ocx ensure would repoint ~/.grok/config.toml, and likely ~/.codex/config.toml's openai_base_url, at an address local clients cannot connect to.

Suggested fix

Treat the locally reachable set as { listen.port } ∪ { unauthenticatedLoopbackListener.port when enabled } and warn only when the fence port is in neither. Apply the same rule wherever ocx ensure / ocx sync choose a client base_url.

Secondary: assertNotAdminToken under KeepAlive is invisible

Earlier in the same log, the installed plist exported a service token that collided with the admin token, and startup threw ~20 times in a row:

error: OPENCODEX_API_AUTH_TOKEN is a management (admin) token. The service exports it as the
data-plane secret, which fences the whole management API closed and makes every ocx management
command fail with 503. Unset OPENCODEX_API_AUTH_TOKEN, or set it to a distinct data-plane key,
then rerun the install.
      at assertNotAdminToken (...)
      at handleStart (src/cli/index.ts:231)
      at dispatchCommand (src/cli/dispatch.ts)

(token values redacted)

To be precise about what is and is not wrong here:

• The ordering relative to teardown is correct, and this was not the cause of the outage above. serviceCommand runs assertServiceAuthEnvironment() before repairService() (src/service.ts:4582-4583) and before install (:4610), and inside installLaunchd the token write (:2362) precedes the bootout (:2386). A collision aborts before anything is evicted.
• Nothing in the product can mint a colliding data-plane token. writeServiceApiTokenFile (:479-500) only copies process.env.OPENCODEX_API_AUTH_TOKEN; the sole minter is the management token (src/server/management-auth.ts:154), and detection is prefix/byte-equality (src/lib/admin-secrets.ts:28-50). The plist never embeds the token — it cats the file at launch (src/service.ts:591-606).
• Where it does bite is src/cli/index.ts:277-283: a legacy ~/.opencodex/service-api-token holding an admin token makes every supervised ocx start throw at boot. Under KeepAlive that is a crash loop in which the job is loaded, nothing ever listens, ocx status says only installed, not loaded, and — because the update fallback spawns with stdio: "ignore" (src/update/index.ts:408-417) — the message is invisible. src/cli/doctor.ts:188-207 already documents this legacy state; validating the on-disk token during install/repair would surface it in the command the operator is actually running.

Related: the update fallback amplifies all of the above

The "killed the proxy, then failed to bring it back" shape also exists outside repair, and it depends on the broken viability bit from Defect 2:

• src/update/index.ts:349-357 — reclaim with killOcxHolders: capturedListen.oldPid != null, then service repair; on failure serviceViable = isServiceViable() (:376) → either :386-396 "refusing to hop" (nothing started) or :408-417 a fire-and-forget spawn(["start","--port",…], { detached, stdio: "ignore" }) with no health wait, so a child that dies leaves nothing listening and no diagnostics anywhere.
• src/update/job.ts:1111-1122 — same downstream with killOcxHolders: true, killAllOcxOnPort: true.
• src/lib/process-control.ts:205-215 — post-stop reclaim with killOcxHolders: !!(stoppedPid …).

Not a factor: .opencodex-uninstall.json

Worth stating because the file's name and mtime make it look like an uninstall ran. It is the owned-path manifest (CONFIG_UNINSTALL_MANIFEST, src/lib/config-ownership.ts:18; writer recordOwnedConfigPath, :267-289), refreshed by installLaunchd at :2360 — i.e. before the bootout, so a manifest failure aborts before any teardown. The manifest on the affected machine is intact, with ownerId present and 38+ paths. No uninstall path ran.

Impact

On a hub topology (runtimeRole: "hub", non-loopback hostname), ocx service repair — the command ocx status itself recommends, sometimes on the false premise of Defect 2 — can take the proxy, the management ingress and the loopback client listener down simultaneously, report no failure, write no log line, and require a launchctl bootstrap the CLI never prints. Remote clients arriving over Tailscale/cloudflared go down with it, and because ~/.opencodex/service.log gains nothing after the eviction, there is no artifact to diagnose from.