Repository navigation
[Bug]: Interrupted stop during update leaves orphaned pending-teardown receipts, causing subsequent ocx update runs to permanently fail-closed #4897
Description
Client or integration
CLI / Updater / Process Control
Area
Installation or packaging
Summary
When ocx update attempts to stop the running proxy before replacing packages, it records a shared teardown claim via claimPendingTeardown (~/.opencodex/pending-teardown-<nonce>.json). If the stop or drain sequence is interrupted (e.g. timeout, service kill, watchdog kickstart, or external SIGTERM/SIGHUP), the process exits before reaching clearPendingTeardown(nonce).
Because bin/ocx.mjs checks hasPendingTeardownIn(readdirSync, configDir()) without validating process liveness (isProcessAlive(ownerPid)), any subsequent run of ocx update immediately aborts with:
opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.
opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.
If multiple attempts are made while the service is unstable, each attempt claims a new receipt and fails to clean it up. In our environment, this caused 6 orphaned pending-teardown-*.json receipts to accumulate, wedging the updater until files were manually deleted.
Reproduction
- Run
opencodexas a managed service on macOS (gui/<uid>/com.opencodex.proxy). - Trigger an update via
ocx update --tag latestwhile the proxy is under load or handling long requests. - Interrupt or kill the update worker during the stop phase before the final
clearPendingTeardown()call. - The worker process (ownerPid) exits. The receipt
pending-teardown-<nonce>.jsonremains in~/.opencodex/. - Run
ocx updateagain. - Observe:
opencodex v2.56.0 (installed via npm, tag latest)
Verified @bitkyc08/opencodex@2.57.0 integrity metadata sha512-lKEPsKq7DYSg2AhA6…
opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.
opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.
- The updater refuses to proceed due to
teardown-outstanding, even though the recordedownerPidis dead and no proxy is running.
Version
2.56.0 -> 2.57.0
Operating system
macOS 27.0 (Darwin 27.0.0 arm64)
Provider and model
Not provider-specific
Logs or error output
opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.
opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.In update-job.json:
{
"currentVersion": "2.56.0",
"latestVersion": "2.57.0",
"exitCode": 1,
"error": "update command failed (1)",
"log": [
"exit 1 · 505 bytes of output withheld (no recognized diagnostic fields)"
]
}Orphaned receipt files in ~/.opencodex/:
pending-teardown-75280d7e4be191581141ca7863fbc7aa.json
pending-teardown-756b6d207bdfc07862a7969abb7b2c39.json
pending-teardown-a8d6f0e0c1d042ff281c49b7962265bb.json
pending-teardown-b13be77f97d4b87e0f0b6160279dc2b4.json
pending-teardown-b3bee9495dd4b13b3414c3c2013e1281.json
pending-teardown-f9f3728779188403ef2e9179778a0d04.json
Receipt payload example:
{
"ownerPid": 90028,
"nonce": "b3bee9495dd4b13b3414c3c2013e1281",
"createdAt": "2026-09-17T08:52:45.393Z",
"endpoint": { "hostname": "127.0.0.1", "port": 10100 },
"endpointSource": "exact"
}Root cause analysis
- Lack of exit/signal rollback: In
src/cli/index.ts,claimPendingTeardowndrops a durable receipt on disk before stopping the proxy. IfstopProxyor eviction fails or is aborted, there is no signal handler ortry...finallyblock to prune the uncommitted claim. - Stat-only check in update launcher:
hasPendingTeardownIninbin/ocx.mjsonly matches the filename patternpending-teardown-*.json. It does not inspect the receipt payload or verify whetherownerPidis still alive viaisProcessAlive(receipt.ownerPid). - Deadlock accumulation: When
bin/ocx.mjssees an existing receipt, it invokesocx stopto settle it, which claims yet another receipt before verifying the inherited ones. An interrupted chain easily snowballs into multiple orphaned files.
Suggested fix
- In
bin/ocx.mjs/hasPendingTeardownIn: Parse existing receipts and check ifownerPidis still active. IfisProcessAlive(ownerPid)is false and the target port is dead, the abandoned receipt should either be auto-recovered/cleared or safely ignored for the update gate. - In
src/cli/index.ts: WrapstopWithDeferralwith cleanup hooks so that if the process terminates unexpectedly before shared teardown completes, the uncommitted receipt is retracted.
Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
github-actions commentedon Sep 17, 2026
Issue reopened
The report now contains the information required by the automated check. Thanks for updating it.
Vocllum commentedon Sep 17, 2026
Added the required Version and Operating System sections to the issue body.
lidge-jun commentedon Sep 17, 2026
리뷰 · 우선순위 70 / 80
이 이슈는 지금
devHEADfa26404d7(패키지 2.58.0)에서ocx update중 stop/teardown이 끊기면~/.opencodex/pending-teardown-*.json영수증이 남고, 이후 업데이트가 영구 fail-closed 되는 문제를 보고한다. 작성 환경에서는 ownerPid가 이미 죽은 영수증이 6개까지 쌓여 수동 삭제 전까지teardown-outstanding로 막혔다. 버전 기록은 2.56.0→2.57.0, macOS launchd 서비스 경로다.코드 경로를 현재 체크아웃에서 확인했다.
bin/ocx.mjs는hasPendingTeardownIn(readdirSync, configDir())로 파일 존재만 보고 stop을 강제한다.src/config/pending-teardown-names.mjs의hasPendingTeardownIn은 이름 패턴(pending-teardown-+ 32hex +.json또는.unreadable.json)만 검사하고, 페이로드의ownerPid나isProcessAlive는 보지 않는다. 주석도 「Presence is the whole test here — the launcher cannot parse it」라고#3008의도를 명시한다. 즉 런처가 죽은 PID를 자동으로 무시하지 않는 것은 우연 빠진 게 아니라, 예전에 singleton 파일명 불일치로 의무를 놓쳤던 사고를 막으려는 fail-closed 설계다.src/cli/index.ts쪽은claimPendingTeardown으로 nonce를 남긴 뒤stopWithDeferral을 타고, 성공 경로에서clearPendingTeardown(nonce)를 부른다. 중단·시그널·타임아웃으로 clear 전에 죽으면 영수증이 남는다. 그 다음ocx update가 다시 stop을 호출하면 또 claim이 생길 수 있어, 작성자가 말한 눈덩이 축적과 맞다. 수동 복구 문구(ocx restore후 pending-teardown 삭제)도 현재 에러 메시지와 일치한다.그래서 “죽은 ownerPid면 런처가 그냥 무시”는
#3008안전 모델과 충돌할 수 있다. 더 안전한 방향은 (1) stop/recovery 경로에서 영수증을 파싱해 liveness·포트 프로브 후 clear/quarantine을 명시적 회복으로 수행하거나, (2) claim을 try/finally·시그널 훅으로 줄이고, (3) 런처의 존재 검사 자체는 fail-closed로 남기는 쪽이다. 포트가 살아 있는데 PID만 죽은 경우, PID만 보고 지우면 위험하다.types.ts/config.ts 분할과 무관하다. 실사용 fail-closed라 우선순위는 높다. 다만 수정은 “편의 자동삭제”가 아니라
#3008의무 모델을 깨지 않는 회복 경로여야 한다.bin/ocx.mjs (hasPendingTeardown / teardownOutstanding) - 파일 존재만으로 update를 막는다.
#3008의도된 fail-closed다.src/config/pending-teardown-names.mjs hasPendingTeardownIn - 이름 패턴만 보고 ownerPid를 읽지 않는다. 런처에 PID 파싱을 넣는 수정은 설계 변경이다.
src/cli/index.ts claimPendingTeardown / clearPendingTeardown / stopWithDeferral - clear 전 중단이 고아 영수증의 직접 원인.
suggested auto-ignore dead PID - 포트 liveness 없이 PID만 보면 위험. stop 경로의 명시적 회복이 더 맞다.
reproduction - 서비스 불안정 시 claim이 반복되면 영수증이 여러 개 쌓일 수 있다.
메인테이너의 판단이 필요한 지점
#3008fail-closed를 유지한 채 stop/restore에 dead-receipt 회복을 넣을지, 런처 존재 검사에 제한적 PID/포트 판별을 허용할지너의 추천
버그로 받아 수정 PR을 받는다. 런처의
hasPendingTeardownIn을 단순히 죽은 PID ignore로 바꾸지 말고,ocx stop/recovery가 영수증을 파싱·프로브한 뒤 clear하는 쪽을 우선한다. claim 구간은 finally/시그널로 누수를 줄인다. 회귀 테스트에 “죽은 ownerPid + 죽은 포트 → stop이 회복 / 산 포트 → 여전히 차단”을 넣는다.이 댓글은 grok-bot이 작성했습니다
lidge-jun commentedon Sep 17, 2026
This looks fixed on 2.58.0, and the count of receipts in your report is what points at the mechanism. Could you check whether it still reproduces on 2.58.0 or later?
The orphaned receipt was never supposed to block.
handleStopreads an inherited receipt, and whenendpointSourceisexactit probes that endpoint; a dead endpoint is recovered. Yours wereexactand the proxy was not running, so all six should have been discharged.What blocked them is the next branch. If the Codex history preflight refused, nothing was restored, so nothing was discharged — including the receipt the
ocx stophad just claimed. Before #4896 that refusal fired on the mere presence of thehistory_modecolumn, which every current Codex build has, so it fired every time. That makes one update attempt equal one retained receipt, which is why you had six.#4896 (
dfa02fe96a) changed that reason from a refusal into a degraded restore: the configuration half is written, the history relabel stands down, and the receipt is released. It is an ancestor of 2.58.0 and not of 2.57.0, and you were on 2.56.0 → 2.57.0.One note on the first suggestion in your report — cleaning up when
ownerPidis dead. We are not going to implement that as written, because recovery is decided by endpoint liveness rather than by the owner's age: a dead owner does not prove the proxy died (a Task Scheduler wrapper recreates it after the parent exits), and widening the launcher gate that way would also let quarantined receipts through. The second suggestion — reclaiming an unconfirmed claim — is a real remainder, and it is safe for claims made before a deferral was requested.lidge-jun commentedon Sep 18, 2026
Fixed on
devby #4960, squashed asfaa31e207b3b519defa4980c8bc4c1946bbc003b.Pending-teardown ownership is now decided by identity rather than bare PID liveness, so an interrupted stop no longer leaves a receipt that a later
ocx updatereads as a live owner and refuses against forever.The shape of the fix is worth stating, because the guard itself was right: it narrows the precondition rather than opening the gate. A receipt whose owner cannot be identified is still refused; what changed is that a recycled PID no longer counts as proof that the original owner is alive.
Shipping in the next release off
dev.