fix(service): wait out the evicted launchd job before repair bootstraps

[vadymhimself]

Summary

On macOS, ocx service repair could take a serving service down and leave it unloaded and stopped.

Incident: a macOS host had a pre-2.66 launchd plist (the old launcher form with KeepAlive set to true) and the proxy was serving under that job. One ocx service repair produced this sequence:

1. service.log printed 🛑 Shutting down opencodex proxy....
2. Repair failed with Bootstrap failed: 5.
3. Repair printed "A proxy is answering on port …, so the existing registration was kept rather than replaced."
4. A few seconds later the old proxy finished draining and exited. After that, nothing was loaded: launchctl kickstart answered "Could not find service". The operator had to run launchctl bootstrap by hand to get the service back.

Root cause. installLaunchd evicts the job with bootout, waits for it to settle, then runs bootstrap. That settle step gave up after 5 × 200 ms. I measured the launchd behaviour with a throwaway label whose process takes 6 s to exit on SIGTERM (macOS 26, Darwin 25.6):

• bootout returns 0 immediately.
• launchctl print keeps answering 0 with state = SIGTERMed until the process exits.
• Every bootstrap in that window fails with 5: Input/output error.
• Only after the exit does print return 113 and bootstrap succeed.

The proxy drains on SIGTERM for up to shutdownTimeoutMs (5 s by default), and the launcher waits for it. So repair's first bootstrap landed inside that window, and so did everything after it:

• The probe saw the still-registered old job as loaded-stale.
• With new plist bytes, the retry skips kickstart and evicts again, which only waits another 1 s.
• The rollback's bootstrap also failed. Its print check still succeeded against the exiting job, so the rollback looked like it worked.
• Once the old process exited, nothing was registered under the label. Because the job had been booted out, KeepAlive did not apply.

Fix: settle for up to 125 × 200 ms = 25 s. launchd's default ExitTimeOut of 20 s, after which it sends SIGKILL, bounds the wait, so a wedged domain still reaches the existing diagnosable error rather than hanging. The wait only runs where bootout actually evicted something (exit 0). The healthy no-op repair and fresh installs never wait.

Not changed, worth a follow-up: in this failure, reportServiceServing says "the existing registration was kept" because the draining old proxy still answers its health probe. It says that even when the registration is gone. With the eviction settled, that message no longer appears for this sequence.

Verification

• New regression test in tests/service/launchd-repair.test.ts ("a job still draining after bootout is waited out, so repair never strands the service"). It uses the existing launchctl/probe/sleepSync/plistPath seams and never touches real launchctl. The fake models the measured behaviour: after bootout the label stays registered for 30 polls (about 6 s) while every bootstrap returns 5.
  • Red: with the old constant (5) it throws launchctl could not bootstrap …: Bootstrap failed: 5.
  • Green: with the fix, one bootstrap, reloaded: true, and the new plist is on disk.
• I updated the existing "settle loop … is bounded" test to the new bound (250 polls across the two domains).
• bun test tests/service/launchd-repair.test.ts: 61 pass, 0 fail.
• bun run typecheck: clean.
• bun run test:changed: 30271 pass, 11 fail. None of the failures come from this change:
  • tests/service/service.test.ts (211 pass), tests/service/service-sqlite-home.test.ts (5 pass) and tests/cli/cli-access-audio-live.test.ts (36 pass) all pass when run on their own. They failed only under the full parallel run.
  • tests/cli/cli-connect-readiness.test.ts ("a rejected preferred runtime falls back…") fails the same way on clean dev (60bf4c7e0).
• I did not run the full bun run test locally because concurrent worktrees were competing for the test lock. Full coverage is left to CI.

Checklist

• Scope stays focused and avoids unrelated cleanup.
• Docs or release notes were updated when needed.
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Service recovery now waits longer for a stopped background service to fully exit before attempting to start it again. This helps avoid failed restarts when the previous service is still shutting down and ensures recovery completes after it exits.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

• Required local validation passed; commands, results, and any full-suite exception are documented.

• I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).

• I resolved all correct Codex and CodeRabbit findings.

• My PR is ready for review.

[vadymhimself]

@coderabbitai review

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: c47a0946-061a-4c0a-8314-f906ad1a977e

📥 Commits

Reviewing files that changed from the base of the PR and between 60bf4c7 and d3075f5.

📒 Files selected for processing: 2

• src/service/launchd.ts
• tests/service/launchd-repair.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

📝 Walkthrough

The launchd settle limit increases from 5 to 125 attempts at the existing 200 ms interval. Repair tests cover the longer wait and a job that remains visible after bootout until it exits.

Changes: Launchd repair

Layer / File(s)	Summary
Settle wait and repair validation src/service/launchd.ts, tests/service/launchd-repair.test.ts	The settle limit increases from 5 to 125 attempts. Tests expect up to 125 polls per evicted domain and verify repair waits for a draining job before bootstrapping once, leaving the new job loaded, and writing the rendered plist.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: lidge-jun

Merge Risk: ⚪ Minimal · up to d3075

This change makes macOS service repair wait for the old launchd job to exit before bootstrapping, which reduces the chance of leaving the service unloaded. No actionable merge risk remains from the supplied evidence.

Pre-merge checks: 5

Check name	Status	Explanation
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and specifically describes the main change: waiting for the evicted launchd job to finish before repair bootstraps. It matches the changes in src/service/launchd.ts and tests/service…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[github-actions]

✅ Deterministic PR hygiene checks passed.

[github-actions]

⏳ DRAFT

• review readiness checklist open (0/4 boxes ticked).

What to do

• Tick all four boxes in the PR description once you're done (currently 0/4).

Review readiness checklist

• ⬜ Required local validation passed; commands, results, and any full-suite exception are documented.
• ⬜ I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
• ⬜ I resolved all correct Codex and CodeRabbit findings.
• ⬜ My PR is ready for review.

0/4 boxes ticked.

This PR stays in draft until every box above is ticked.

[lidge-jun]

Being carried into #7225 with your authorship preserved; this PR will be closed when the bundle lands.

[lidge-jun]

Landed on dev in 300e426 via #7225 (your change is its own commit in that bundle, and the squash keeps your authorship as Co-authored-by). Thank you!