Repository navigation
[Bug][Windows] Continuation spill failures accumulate behind healthy readiness after #3011 (observed on 2.39.0) #3522
Description
Diagnostic follow-up landed; runtime recovery remains open
#3790 (c2b4dc0696) distinguishes fresh retry timeouts from remembered timeout refusals. The reported long-lived Windows process recovery failure is not declared fixed. The remaining acceptance is same-process evidence and a justified recovery remedy.
Verified against dev 5759d9ea2f1e7281cdc01eb9628f2e0a123fb59c. Original report by @stephen-drew. The attribution record was added in #3811.
Client or integration
Codex App through the local OpenCodex proxy.
Area
Windows response-state persistence, failure diagnostics and recovery.
Summary
During normal concurrent Codex App use on 1 September, an official OpenCodex 2.39.0 process stopped successfully publishing continuation spills while continuing to report healthy/ready. The successful spill count remained at 1,988 and the failure count eventually reached 11,365 in the same process.
This follows the asynchronous Windows spill work in #3011, #3018 and #3044. The new evidence concerns sustained persistence degradation after those changes. The saved measurements do not identify the exact exception thrown inside the affected process.
Reproduction
The incident was intermittent, under concurrent Codex App traffic. A deterministic production trigger has not been established.
- Run the official 2.39.0 Windows package with multiple concurrent Codex tasks.
- Compare successive supported
ocx memory --jsonand health/readiness observations. - Observe a successful spill count that stops advancing while spill failures continue increasing for hours.
- In a separate diagnostic process, verify that the same release's durable spill writer can publish, read back and clean up an isolated disposable spill.
Step 4 succeeded in the captured incident, making a persistent process-local failure a useful investigation direction.
Logs or error output
Saved incident evidence:
Installed/live version: 2.39.0
Official package parity: 923 / 923 files identical
Same process throughout incident: yes
Health/readiness: healthy
Successful spill publications: fixed at 1,988
Spill write failures: 1,582 -> 1,763 -> 3,439 -> 3,599 -> 3,714 -> 11,365
Spill read failures: 0
Earlier degraded-state snapshot: 946 tombstones, 0 spill stubs
Final sampled RSS: approximately 578 MB
Free disk space: approximately 229 GB
Final active turns: 7
Final draining state: false
The resident-memory budget was not exceeded. The spill directory retained a protected owner-only ACL. No ACL timeout override was found in the inspected service or environment configuration.
Separate-process checks of the exact release passed:
Single durable publication, read-back and cleanup: 305 ms
Five sequential publications: 232, 88, 92, 87, 85 ms
Leftover diagnostic files: 0
Source tests also demonstrated the bounded ACL-timeout recovery path and its exhausted-recovery error. This makes exhausted ACL recovery a hypothesis supported by the source and isolated tests. The live incident exposed aggregate counters only, so EACLRETRYEXHAUSTED is not claimed as a captured live exception.
Expected behaviour
Preserve required ACL checks and bounded waits. Sustained failure to persist continuations should be observable as a specific degraded capability, including a privacy-safe cause and an actionable recovery route.
A healthy routing endpoint can coexist with a persistence failure, but operators need to be able to distinguish those states without inferring the cause from thousands of failed saves.
Suggested investigation
- Expose a bounded, sanitised last spill error code, timestamp and failure streak alongside the existing counters.
- Distinguish an isolated failed response from a persistence path that can no longer publish successfully.
- Test an exhausted Windows ACL recovery followed by a healthy ACL runner, and document when publication can resume in the same process.
- If the failure requires a restart, expose that requirement explicitly and preserve the active-turn/drain boundary.
- Preserve fail-closed ACL behaviour, publication ordering and protection against automatic replay of side-effecting tool calls.
Version
Observed live: OpenCodex 2.39.0, official source commit af6113a0381d6fff2e4dce587652825c7eeb6423.
Installed package when reviewing this report on 4 September: 2.42.0. The incident has not been reproduced on 2.42.0. At the 4 September review snapshot, the process had approximately 19 minutes of uptime and reported zero successful spills and zero spill failures, so it had not exercised the relevant path.
Operating system
Windows, x64. The current review host reports 25H2, build 26200.9168.
Provider and model
The suspected failure is in local continuation persistence. No provider-specific root cause is claimed.
Screenshots and supporting files
Evidence was recovered from the saved incident report, package-parity checks, successive runtime snapshots and isolated diagnostic results. Private prompts, conversation contents, usernames, paths and credentials are omitted.
Redacted configuration
{
"client": "Codex App",
"platform": "Windows x64",
"serviceBackend": "Task Scheduler",
"proxyBinding": "local loopback"
}Checks
- Current open spill-related issues were searched before submission.
- Related reports [Bug][Windows] synchronous ACL hardening still stalls /healthz and poisons response spill writes after #612 #3011, fix(windows): keep ACL stalls off the response event loop #3018 and fix(windows): drain response spill publications before shutdown snapshot #3044 were reviewed.
- The recorded incident is distinguished from the current unexercised process.
- The proposed cause is labelled as an inference.
- Private task content and credentials are excluded.
Activity
Ingwannu commentedon Sep 4, 2026
Confirmed as a real current observability gap, but not yet as a reproduced current-version persistence defect.
On current
dev, every ordinary Windows async spill exception reaches the catch inrunPendingResponseSpill, increments onlyspillWriteFailures, and is then discarded.responseStateMetrics()exposes cumulative writes/read failures/write failures, while authenticated/api/system/memoryforwards that scalar snapshot and/api/system/healthremains ordinary process liveness. There is no privacy-safe last error code or timestamp, no consecutive-failure streak, and no success marker that proves the same process recovered. Therefore the reported1,988fixed writes plus11,365failures cannot currently be distinguished from many isolated failures using supported diagnostics.I am keeping two claims separate:
devand is actionable.EACLRETRYEXHAUSTED, and the report has not exercised this path on 2.42+ yet, so I will not label the root cause or weaken/restart the ACL path speculatively.The bounded fix I would accept is:
/healthz.I will leave this open while the current-version reproduction remains unproven. A 2.42/2.43 exercised snapshot showing the counters and the first sanitized failure class after this instrumentation will decide whether a second runtime-recovery fix is needed.
lidge-jun commentedon Sep 4, 2026
리뷰 · 우선순위 62 / 80
이 이슈는 Windows에서 Codex App이 동시에 여러 일을 돌릴 때, 이어쓰기(continuation)를 디스크에 저장하는 spill 쓰기가 계속 실패하는데도 프로세스는 계속 “정상/준비됨”으로 보이는 문제를 말합니다. 보고자는 공식 2.39.0에서 같은 프로세스 안에서 성공한 spill이 1,988에서 멈춘 채 쓰기 실패만 1,582 → 11,365까지 쌓였고, 디스크 여유·RSS·읽기 실패는 정상이었다고 적었습니다. 같은 바이너리로 별도 프로세스에서 한 번 쓰고 읽고 지우는 검사는 통과했기 때문에, “파일 시스템 전체가 죽었다”기보다 “그 프로세스 안의 저장 경로가 오래 막힌 상태”로 보는 편이 맞습니다. 관련 배경은 비동기 Windows spill 작업
#3011/#3018/#3044이후의 지속 저하입니다.지금
devHEAD3aab264e3(패키지 2.43.0, 방금#3510으로 tests 레이아웃 일부 이동)를 보면, 이 진단 구멍은 그대로입니다.src/responses/state.ts의runPendingResponseSpill은 ACL 타임아웃 한 번 재시도 뒤에도 실패하면 catch에서spillCounters.writeFailures만 올리고 예외는 버립니다.responseStateMetrics()가 내보내는 값은 누적spillWrites/spillWriteFailures/spillReadFailures와 개수·바이트 정도뿐이고, “마지막 실패가 언제·무슨 종류였는지”, “연속으로 몇 번 실패했는지”, “같은 프로세스에서 다시 성공했는지”는 없습니다.src/server/management/system-routes.ts의 인증된/api/system/memory는 그 스냅샷을 그대로 실어 주고,/api/system/health와 비인증/healthz는 프로세스 살아 있음 쪽에 가깝습니다. 그래서 운영자가 “라우팅은 되는데 이어쓰기 저장만 죽었다”를 숫자만으로 구분해 내기 어렵습니다. 메인테이너(Ingwannu) 댓글도 같은 선을 확인했고, 2.39.0 원인 단정과 ACL 약화는 보류하는 쪽입니다.보고 버전은 2.39.0이고, 작성 시점의 2.42.0 호스트에서는 spill 경로를 아직 거의 안 탄 상태라 현재 버전 재현은 없습니다. 따라서 “지금 빌드가 같은 방식으로 고착된다”고 단정하면 안 되고, 먼저 안전한 관측(마지막 실패 코드 분류·시각·연속 실패·마지막 성공 시각)을 넣는 쪽이 맞습니다. Windows ACL 쪽 코드는
src/lib/windows-secret-acl.ts에EACLRETRYEXHAUSTED가 있고tests/windows/windows-secret-acl.test.ts에 고갈 경로 테스트가 있지만, 라이브 사고 로그에는 그 코드가 잡혀 있지 않습니다. 가설로는 두고, 계측 후에야 두 번째 런타임 복구(재시작·드레인)를 논의하는 게 안전합니다. 지금dev의 메인 트랙은#3497tests 도메인 이동(#3511/#3513/#3516/#3518스택)이라, 이 이슈는 그 열차와 겹치지 않는 platform 관측 작업으로 따로 잡으면 됩니다.라인 / 심볼 기준으로 보면 아래가 핵심입니다.
src/responses/state.tsrunPendingResponseSpillcatch (대략 383–389) - 예외를 삼키고 writeFailures만 올려서, 연속 실패·마지막 원인 분류·마지막 성공 시각을 남기지 않음src/responses/state.tsResponseStateMetrics/responseStateMetrics()(대략 2099–2153) - 누적 카운터만 노출. streak·lastFailureAt·sanitized code·lastSuccessAt 필드 없음src/server/management/system-routes.ts/api/system/memory- 위 스칼라만 전달. 인증된 진단에 spill health 스냅샷을 더할 자리/api/system/health및/healthz- 라우팅/프로세스 생존과 continuation 저장 가능 여부를 구분하지 않음(의도적으로 분리 유지하는 편이 맞음)src/lib/windows-secret-acl.tsEACLRETRYEXHAUSTED- 소스·단위 테스트에만 있고, 이 사고의 라이브 증거로는 아직 확정되지 않음메인테이너의 판단이 필요한 지점
help wanted로 열어둘지, 메인테이너가 계측 PR을 직접 잡을지너의 추천
Ingwannu가 적어 둔 범위 그대로 진행하는 게 맞습니다. (1) 개인정보 없는 spill-write health 스칼라(연속 실패, 마지막 실패 시각, 닫힌 오류 코드 분류, 마지막 성공 시각)를
responseStateMetrics()와 인증된 memory/CLI에만 추가하고, (2) 누적 카운터는 호환용으로 유지한 채 성공 시에만 streak을 리셋하고, (3) 같은 프로세스 failure→failure→success 회귀와 ACL 고갈 후 건강한 runner로 재개되는 테스트를 넣으세요./healthz에는 넣지 마세요. ACL 강제나 자동 재시작은 현재 버전에서 카운터·첫 sanitized 실패 분류가 잡히기 전까지 손대지 마세요. 이슈는 열어 두되, 작업 범위는 “관측 먼저”로 잘라 주세요.이 댓글은 grok-bot이 작성했습니다
Ingwannu commentedon Sep 4, 2026
Implemented the bounded observability fix in #3525.
The PR keeps cumulative counters and adds only privacy-safe process-local state:
initial/healthy/degraded, consecutive write failures, a closed last-error class, and last failure/success timestamps. A durable publication is the only event that clears the streak. Raw errors, paths, ids, payloads, and causes are not exposed, and neither/healthznor readiness/restart behavior changes.Focused regressions now prove both failure → failure → success and Windows ACL retry exhaustion → healthy-runner recovery in the same process without weakening ACL enforcement. I am leaving this issue open, as planned, until an exercised current-version host captures the new diagnostic snapshot and establishes whether a separate runtime-recovery defect still exists.
stephen-drew commentedon Sep 6, 2026
Exercised Windows 2.43.0 follow-up for #3522, with the diagnostics carried in #3542 present.
Two authenticated memory snapshots from the same running process on 6 September 2026 show continuing failure:
That is 105 additional failures in approximately five minutes with no successful publication. The last-success timestamp stayed unchanged while the last-failure timestamp advanced. Process health remained healthy; 10-12 turns were active, with no drain or restart during this observation. RSS remained below the 4 GiB warning threshold; spill read failures and app-owned over-budget bytes were zero.
Five sequential synthetic spill writes in a separate process, using the installed spill implementation and an isolated home, all passed write/read-back/cleanup in 124-300 ms. Read-only inspection of the live spill directory found protected inheritance and current-user-only FullControl, with no reparse point.
Source disclosure: all installed package files match the verified official 2.43.0 archive except a registered local first-response timeout change in src/server/responses/codex-ws-wire.ts (30 to 300 seconds). Spill-state and Windows ACL code are unmodified.
This captures the previously missing current-version failure class and confirms an ongoing degraded streak. It does not identify the original timeout stage or distinguish a remembered exhausted retry from a second actual timeout. The isolated success does not prove live same-process recovery.
Does this support a separate runtime-recovery investigation, and which additional privacy-safe diagnostic would best distinguish those two paths? No ACL weakening or automatic restart change has been applied.
Ingwannu commentedon Sep 6, 2026
Yes, the same-process streak supports continued runtime-recovery investigation here. The separate-process success does not test the long-lived process's memoized timeout state, so it cannot establish recovery.
The current source confirms why the displayed code is ambiguous.
windows-secret-acl.ts::previousTimeoutError(true)emitsEACLRETRYEXHAUSTEDfor a previously consumed recovery. Separately,responses/state.tssetsexhaustedAclRetrywhen the second spill attempt returnsETIMEDOUT, then passes the sameEACLRETRYEXHAUSTEDoverride tonoteSpillWriteFailure. Thus this field alone cannot count fresh ACL timeouts versus remembered refusals.The smallest useful additional diagnostic would record a fixed origin alongside the existing code:
retry_returned_timeoutversustimeout_memo_refusal, with separate counters. At the ACL layer, an optional fixed step category and whether an OS command was actually launched would distinguish waiting for identity/compliance inspection from grant/inheritance/removal work. No raw paths, SIDs, usernames, stdout or credential material are needed.Current async hardening also attempts
existingAclAlreadyCompliantAsyncbefore checking the timeout memo. Consequently, read-only inspection that the directory looks correct is not yet evidence that the live process's bounded compliance check completed successfully.Keep the existing fail-closed ACL boundary and avoid automatic restart or memo clearing. The next recovery decision should be based on that same-process diagnostic, not on a fresh process passing five writes. No Windows recovery fix is claimed in this reply.
fix(diagnostics): distinguish spill ACL timeout origins [skip ci]
fix(diagnostics): distinguish spill ACL timeout origins [skip ci]
test(responses): pin Windows spill ACL timeout-memo recovery (#3522)
fix(windows-secret-acl): re-arm consumed ACL timeout memo after a qui…
devin-ai-integration commentedon Sep 25, 2026
New matched pull request: #5825 (would resolve this issue) — fix: P0 #5369 spill growth + P1 bug batch [
priority: P0]fix: P0 #5369 spill growth + P1 bug batch (#5421, #5008, #4191, #3522, …
lidge-jun commentedon Sep 25, 2026
Fixed by #5825 (22b22ae): after the one Windows ACL timeout-memo recovery attempt is used, it re-arms 5 minutes later, so the spill directory can publish again without a process restart.