Repository navigation
[Bug]: Intermittent spend-ledger storage safety refusal on macOS with native Codex traffic, including HTTP-only #6314
Description
Client or integration
Codex App
Area
Proxy and routing
Reproduction
Intermittent; these are the performed steps, not a claimed deterministic minimal reproducer:
- Use the portable v2.69.0 executable on Apple Silicon macOS. Prepare separate trial directories for HOME, CODEX_HOME, OPENCODEX_HOME and TMPDIR. The isolated CODEX_HOME starts empty. Configure a localhost listener on port 10100, native openai provider with authMode forward and codexAccountMode direct, and the canonical Codex upstream.
- Start the portable executable with the
startargument under that environment. Confirm schema/startup/health checks pass. - Point Codex desktop's openai_base_url to
http://127.0.0.1:10100/v1with a compatible trial model catalog, then restart the desktop app. - Send ordinary native OpenAI turns. A bounded diagnostic asked for a harmless synthetic readiness file to be read and its fixed line returned; automatic approval-review traffic also occurred. Observe intermittent HTTP 502 entries with the storage error alongside HTTP 200 entries. A completed chat alone did not imply an error-free route.
- In the separate HTTP-only follow-up, keep native account/endpoint routing, set
providers.openai.upstreamWebsocket = false, and disable the four retry controls listed below. Startup and initial requests can still pass, then the same storage errors occur. - Stop after failure and restore the original connection. Inspect stopped ledger/salt metadata: predicates and salt format pass. The precise file/predicate at failure time is not captured.
Version
v2.69.0, source pin 3cc34e1181926b64331490fdcfee162ffb62fe73, embedded Bun 1.4.0.
Operating system
macOS 15.5, Apple Silicon.
Provider and model
Native openai/Codex route. Storage errors observed for ordinary gpt-6-astra and gpt-5.6-sol traffic, plus codex-auto-review. No Qwen inference in these bridge trials.
Summary
OpenCodex v2.69.0 intermittently returns HTTP 502 with:
Provider unreachable: Spend-ledger storage could not be opened safely.
This affects ordinary native OpenAI Codex requests and automatic approval-review requests. Repeating the trial with providers.openai.upstreamWebsocket = false did not resolve it. The bridge is now stopped and the original connection restored.
This is an observed failure, not a claim that the root cause or a fix is established.
Environment and routing
- macOS on Apple Silicon.
- Portable OpenCodex v2.69.0; pinned source
3cc34e1181926b64331490fdcfee162ffb62fe73; embedded Bun 1.4.0. - Codex desktop through a localhost OpenCodex bridge, native
openaiprovider,authMode: forward,codexAccountMode: direct, canonicalhttps://chatgpt.com/backend-api/codexupstream. - Dedicated trial state directory under the user's Documents directory. Storage/sync involvement has not been established.
- An Ollama provider was configured but no Qwen requests were made in these bridge trials.
- No credentials, private request bodies, full logs or machine-specific paths are included here. The exact desktop build was not re-collected for this report.
Observations
- Schema validation, bridge startup and health checks succeeded.
- Initial native-route trial logged 20 requests: 9 HTTP 200 and 11 HTTP 502 carrying the same storage message.
- A separate HTTP-only trial retained the same native account/endpoint and set
providers.openai.upstreamWebsocket = false. Its final log had 28 requests: 6 HTTP 200 and 22 HTTP 502 carrying that message. - Counts include desktop/control-generated traffic, not that many deliberate test prompts. No exact per-chat attribution or hard app-wide request cap is claimed.
- The HTTP-only trial had bridge retry options disabled (
retryOn429,transientRetryOn5xx,retryOnReset,emptyCompletionRetry); this did not prevent additional client/control traffic. - After stopping, the ledger and salt were regular, non-symlink, single-link, process-user-owned files; mode 0600; salt was valid 64-character hexadecimal. Read-only checks using Node 26 and the embedded Bun runtime passed the recorded predicates.
- A separate copied-state reproduction with a fake local upstream/custom provider returned 8/8 successful responses, including four streams. It did not reproduce the error and is NOT equivalent to the native-account desktop route.
Source checks
The message occurs in assertSafeLedgerFile and the invalid-existing-salt branch of loadOrCreateSpendLedgerSalt in src/lib/spend-reservation-ledger.ts. The failed request log does not identify the path or predicate.
Comparing the tested pin with v2.73.0 returned identical source for:
src/lib/spend-reservation-ledger.tssrc/lib/spend-ledger-owner.tssrc/server/responses/request-spend.ts
Related #5157 is already included in the tested version. This is not evidence that all runtime/dependency behavior is unchanged in newer versions. No upgrade has been tested.
Expected behavior
Valid owned ledger storage should allow the request. If storage is refused, a bounded, non-sensitive diagnostic identifying the failed predicate and file role would make the refusal diagnosable without weakening the safety guard.
Questions for maintainers
- Is this a known macOS/portable-Bun or request-lifecycle issue, and is there a version-matched fix or regression test?
- What supported diagnostic can distinguish the ledger/salt file and the exact predicate at the failing instant without logging credentials or request content?
- Is there a documented state-directory constraint relevant to this route that startup/health does not check?
No ledger deletion, guard bypass, permission relaxation, or concurrent shared-home writer workaround has been attempted. We are seeking evidence before another live trial.
Logs or error output
HTTP 502
Provider unreachable: Spend-ledger storage could not be opened safely.
Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
github-actions commentedon Sep 30, 2026
Issue reopened
The report now contains the information required by the automated check. Thanks for updating it.
Ingwannu commentedon Oct 1, 2026
The stopped-file checks and copied-state success do not identify the predicate that failed during the native request. In current dev 6429463 the same message covers the ledger file type/link count/owner predicates and invalid existing salt content, so I cannot responsibly claim a macOS/Bun root cause or a version-fixed resolution yet. The useful next evidence is a bounded diagnostic at the failing operation: file role (journal/salt, not full path), predicate name, and a normalized system error code if there is one; no salt bytes, identities, request bodies, or raw account logs. A synthetic fixture should cover each refusal before adding that diagnostic. Please do not delete the ledger, relax permissions, or bypass the ownership guard to make the error disappear; no further live-account trial is requested here.
github-actions commentedon Oct 1, 2026
Automated translation bookkeeping — detected language: English.
orenhadomi-creator commentedon Oct 1, 2026
Thanks. We already have bounded observations from a locally instrumented diagnostic build based on
3cc34e1181926b64331490fdcfee162ffb62fe73, not a reproduction on dev64294638a69e25ca0c7a4e2102e2349973161f71.At the failing native-request guard, all eight saved observations show:
stat.nlink !== 1.2; required:1.lstatSyncreturned metadata, then the explicit safety guard refused it. The application refusal code is not an OS errno.Saved isolated-function checks cover non-regular file, symlink, multiple links, wrong owner and invalid salt. These are in-memory checks, not upstream application fixtures against current dev. This identifies the refusal, but not who created the extra link, a macOS/Bun root cause, or a fixed version.
Has the relevant journal creation/replacement/compaction or storage-guard code changed between these revisions? Can this existing observation guide investigation on current dev, and what would be the smallest useful offline/synthetic follow-up?
Further live-account trials are paused. No ledger deletion, permission relaxation or ownership-guard bypass is proposed. No raw logs, paths, path hashes, identities, salt bytes or request bodies are attached.
Ingwannu commentedon Oct 1, 2026
The new observation narrows this substantially: in those eight saved runs the journal was refused because nlink was 2, not because lstat failed, the salt was malformed, or the owner/symlink predicates failed. That identifies a correctly triggered guard, not yet the source of the extra directory entry.\n\nI compared 3cc34e1 with current dev a6114b6: src/lib/spend-reservation-ledger.ts, src/lib/spend-ledger-owner.ts and src/server/responses/request-spend.ts have no diff. The journal path uses appendFileSync and exclusive temp-file creation followed by renameSync for compaction; that module does not call link/linkSync. This does not rule out a transient filesystem/runtime or another writer interaction.\n\nThe smallest useful offline follow-up is a current-dev production-journal fixture in a disposable owned home: create/append a journal, create one synthetic second hard link, assert append/compaction refuse with unchanged journal bytes, then remove only that owned test link and check the original storage can proceed. Separately trace nlink around ordinary append/atomic replacement without introducing a link. Existing tests/lib/spend-ledger-owner.test.ts covers hard-linked owner files, which is a different file role; it is not the requested journal-path reproduction. The missing evidence remains who/what introduced link #2 in the observed native route. No new live-account trial, guard bypass, ledger deletion, or permission change is requested.
orenhadomi-creator commentedon Oct 1, 2026
We completed the bounded offline production-journal fixture using unmodified source files pinned to
a6114b62ed1b65dede802359ccd373d913979b24, with Bun 1.4.0 on macOS. It calls the actual journal implementation with real SQLite ownership and real filesystem operations in disposable owned homes, not the owner-file hard-link fixture or extracted/mocked functions.SPEND_LEDGER_OWNER_UNAVAILABLE, leaving journal bytes and directory entries unchanged.The ordinary-operation observations use operation boundaries and existing non-mutating compaction test hooks, not a continuous syscall trace. They do not rule out transient events between observations or another writer/native-route interaction. The fixture ran once and exited 0; this does not identify the creator of the original extra link or claim a version-fixed resolution.
No live-account trial, guard bypass, real-ledger deletion or live-permission change was performed. What is the smallest next bounded evidence that would help identify the source of the original extra directory entry, given these results?
lidge-jun commentedon Oct 1, 2026
Thanks, this narrows it a lot. The refused entry is the journal, the failing condition is
nlink == 2, the stopped files are back atnlink == 1, and your offline fixture in a disposable home never produced a second link. So something outside OpenCodex adds a second directory entry for a short time and then removes it. The journal code itself only appends, creates a temp file exclusively and renames it. It never callslink.Your report puts the trial state directory under the user's Documents directory. If iCloud Drive "Desktop & Documents Folders" is turned on for that account, the iCloud/File Provider daemon (
fileproviderd) can hold an extra hard link to a file in that folder while it stages or uploads a change. That fits what you saw: it is intermittent, it happens with or without WebSocket, it is gone once the server stops, and it does not happen in a disposable home outside Documents. This is a hypothesis we have not confirmed yet.Two bounded questions, with no live-account trial needed:
~/Documentsand once outside it (for example under/tmpor~/Library/Caches), and compare whethernlinkever reads 2. If it does, capture the second name once at that moment:stat -f 'dev=%d inode=%i nlink=%l' <journal>and thenfind <same volume root> -xdev -inum <inode> 2>/dev/null. Please share only whether the second path is inside a sync or File Provider location (for example~/Library/Application Support/FileProvider/...or~/Library/Mobile Documents/...), not the full path.On our side the guard stays strict, because an extra hard link is exactly what it exists to refuse. We are preparing a change that makes this diagnosable and avoidable: the refusal will record the file role and the failed condition (no path, salt or request content), startup will warn when
OPENCODEX_HOMEresolves inside an iCloud Drive or File Provider synced folder, and the docs will say to keep the state directory out of synced folders. Until then, movingOPENCODEX_HOMEoutside Documents should avoid the refusal if the hypothesis holds.orenhadomi-creator commentedon Oct 1, 2026
Replying to the two bounded questions:
Our read-only Google Drive check on September 27 showed Documents selected with "Sync with Google Drive" enabled. The state used in the failing trial was under Documents. We have not established the iCloud Desktop & Documents setting or continuously monitored the sync configuration during the original runs.
We ran the existing current-dev production-journal fixture once in a fresh disposable directory under Documents and once in a fresh directory outside Documents under /tmp. Same a6114b6 production source and existing Bun 1.4.0. The fixture body was unchanged; only its test-root assertion and source-import locations were adapted. No live accounts, model requests, connection/settings changes or guard bypass.
The fixture's deliberate hard-link control was kept separate and excluded from this observation. An independent observer watched only the ordinary-phase journal, which our fixture never hard-links:
Sampling was nominally every 10 ms through execution and for 15 seconds after each fixture completed. On the first unexpected nlink=2, a separate stat confirmed the same device/inode and nlink=2. The requested same-volume exact-inode lookup was bounded to five seconds and timed out without finding an alias; it caused a five-second sampling gap. No second-path category is established, and we are not publishing full paths, numeric file identities or private contents.
This reproduces an unintroduced extra link in Documents but not in the short outside-Documents control, and supports the synced-location hypothesis. It does not yet identify the responsible process/provider or prove the original creator. No real journal was removed or relocated; all production source, prior control files and connection configuration remained unchanged.
Is this enough evidence to justify a separately approved state-home relocation outside synced Documents as the next workaround check, or is a bounded alias-category capture still needed first?
fix(spend): name the refused ledger file and condition, warn on synce…
lidge-jun commentedon Oct 8, 2026
Status on current
dev(a12bb1f): the refusal itself is the intended guard, and we found no OpenCodex code path that creates the second link.nlink !== 1, or has foreign ownership (src/lib/spend-reservation-ledger.ts). The request tracker hits this guard for HTTP and WebSocket alike, which is why disabling WebSocket did not help.renames it, nothing in the module callslink, and a SQLiteBEGIN IMMEDIATElease excludes a second OpenCodex writer.We are not relaxing the guard on this evidence. One useful next step: rerun the same offline comparison on current
devwith the state directory outside any synced folder, recording only file role, predicate, and link-count outcomes. If identifying the producer still matters, a bounded capture of the alias category or the process during an unexpected two-link event would help. Note that the startup warning added in #6398 does not detect every Google Drive configuration, so a missing warning does not rule out sync.fix(spend): warn when Google Drive syncs Desktop or Documents (#6846)
fix(responses): report spend-ledger storage refusals as local 503s (#…
lidge-jun commentedon Oct 10, 2026
Closing after #6890 (squash
fe8864d678). The ledger safety refusal is intended: it fires when an external sync daemon (Google Drive, iCloud and similar) links or replaces the state file. Earlier, #6398 and #6846 made the refusal name the file and condition and warn about synced state directories. #6890 adds the remaining fix: a local storage refusal is now reported as a local 503spend_ledger_storage_unavailable, not a 502 'provider unreachable'. If you see this with the state directory outside any synced folder, please reopen with the refusal message.