Skip to content

[Bug][macOS] Forked-session rollouts permanently fail history restore: readLatestSessionMeta ignores thread id, blocking every ocx stop/update #3026

Closed

Description

@robin-bially

Client or integration

Codex App

Area

Installation or packaging

Summary

snapshotRolloutForRestore validates a rollout using readLatestSessionMeta, which returns the last session_meta line in the file regardless of its id. Forked/branched sessions append the parent thread's session_meta after their own, so meta.record.payload.id === entry.id fails and restore throws history_backup_rollout_postimage_mismatch.

Because preflightRestoreTargets is all-or-nothing, a handful of forked threads permanently blocks the restore of every entry in the manifest. ocx stop then exits 1 on every invocation, and ocx update aborts at the stop.status !== 0 gate in src/update/index.ts. Unlike #3008 (transient, "Codex app holds the DB busy"), this state never clears on retry — the user cannot update at all until the manifest is edited by hand.

On my machine 6 of 389 manifest entries were affected and had blocked every ocx stop / ocx update.

src/codex/history-provider.ts (v2.36.0):

// line 547
function snapshotRolloutForRestore(entry: CodexHistoryBackupEntry): RestoreRolloutSnapshot {
  ...
  const latest = readLatestSessionMeta(entry.rolloutPath);   // <-- id-agnostic
  if (!latest
    || (!rolloutMatchesRestoreTuple(latest, entry, entry.modelProvider, entry.source)
      && !rolloutMatchesExpectedPostImage(latest, entry))) {
    throw new CodexHistoryIntegrityError("history_backup_rollout_postimage_mismatch");
  }

Both matchers require meta.record.payload.id === entry.id, so a foreign trailing session_meta can never satisfy either branch.

The id-aware helper already exists in the same file, 17 lines below the one being used:

  • readLatestSessionMeta — line 663 (used here)
  • readLatestSessionMetaForIdFromText — line 680 (not used here)

assertRestoreReadback (line 620) calls readLatestSessionMeta the same way and would hit the identical problem on the readback path.

Suggested fix: resolve the rollout's own session_meta by id in snapshotRolloutForRestore and assertRestoreReadback, e.g. a readLatestSessionMetaForId(path, entry.id) wrapper around the existing readLatestSessionMetaForIdFromText. Trailing parent-session metadata is normal for forked rollouts and is not an integrity violation.

Expected: a forked child rollout restores normally; a foreign trailing session_meta is not treated as an integrity violation.


Second, related defect: has_user_event drift on a non-relabelled row

rowMatchesExpectedPostImage (line 487) tolerates has_user_event drift only along the opencodex provider branch:

function rowMatchesExpectedPostImage(row: RestoreRowSnapshot, entry: CodexHistoryBackupEntry): boolean {
  if (entry.modelProvider === "openai") {
    const postHasUserEvent = hasFirstUserMessage(row.first_user_message) ? 1 : entry.hasUserEvent;
    return rowMatchesRestoreTuple(row, "opencodex", entry.source, postHasUserEvent);
  }

A thread captured into the manifest as openai / vscode / hasUserEvent=0 that opencodex never relabelled — provider still openai in both the DB and the rollout — but that the user then wrote in, ends up as openai / vscode / has_user_event=1. That matches neither rowMatchesRestoreTuple (drifted has_user_event) nor the post-image branch (provider is not opencodex), so preflightRestoreRows throws history_backup_postimage_mismatch — again blocking the whole manifest.

has_user_event is Codex-owned session state, not provenance opencodex writes. Treating its drift as an integrity violation looks wrong on any branch, and the restore path additionally forces it back to the manifest value.

Reproduction

  1. Run Codex Desktop with opencodex injected, so a codex-history-backup-*.json manifest exists in $OPENCODEX_HOME.
  2. Fork/branch a thread in Codex Desktop. The child's rollout JSONL then contains two session_meta records: its own at line 0 and the parent's at line 1, with distinct session_id/id.
  3. Run ocx stop and check the exit code: it is 1, with Codex resume history could NOT be restored because the backup or restore target failed integrity checks.
  4. Run ocx update: it aborts with could not stop the running proxy even though the proxy is fully stopped.

The abort message is misleading — there is no PID file, no runtime record, and no listener on the configured port. Only the history restore failed.

ocx doctor reports backup manifest or restore target failed integrity checks — manual review required and advises against re-running ocx sync, but offers no way forward.

Verified by replaying the preflight logic read-only against the manifest and ~/.codex/state_5.sqlite: all 389 entries had a matching threads row and a matching rollout_path; the 6 failures were exactly the forked rollouts, each rejected because the trailing session_meta carried the parent's id.

Workaround used: with the Codex app closed, reset the 6 affected threads rows from model_provider='opencodex' back to 'openai' (their rollout JSONLs were already native, so nothing to patch there), drop those entries from the manifest, and correct the drifted hasUserEvent on the one entry described above. ocx stop then exited 0, restored the remaining 383 entries normally, consumed the manifest, and ocx update completed.

Related: #3008 hits the same abort site in src/update/index.ts, but from a transient busy DB. The decoupling proposed there would unblock updates for this bug too, while the forked-rollout mismatch would still make the restore fail forever.

Version

2.36.0 (npm, tag latest; first hit on 2.33.0, code verified unchanged in 2.36.0)

Operating system

macOS 27.0.0 (arm64, Apple Silicon)

Provider and model

Not provider-specific.

Logs or error output

$ ocx stop
🛑 Service manager stopped (won't respawn).
⚠️  opencodex not present in Codex config. ⚠️ Codex resume history could NOT be restored because the backup or restore target failed integrity checks; unverified provider metadata was left unchanged.
$ echo $?
1

$ ocx update
opencodex v2.33.0 (installed via npm, tag latest)
⏹  Stopping the running proxy before updating...
🛑 Service manager stopped (won't respawn).
⚠️  opencodex not present in Codex config. ⚠️ Codex resume history could NOT be restored because the backup or restore target failed integrity checks; unverified provider metadata was left unchanged.
opencodex: could not stop the running proxy; aborting the update. Run 'ocx stop' and retry.

$ ocx doctor
Codex native-write coordinator
  --     backup manifest or restore target failed integrity checks — manual review required

Hints
  - The history manifest or its target is invalid or changed. Preserve both, inspect the
    manifest/database/rollout identity, and do not repeatedly run 'ocx sync' until the
    mismatch is understood. Untracked routed history is not relabeled.

Shape of an affected forked rollout JSONL (two session_meta records, distinct ids):

line 0  id=<child-uuid>   session_id=<child-uuid>   model_provider=openai  source=vscode
line 1  id=<parent-uuid>  session_id=<parent-uuid>  model_provider=openai  source=vscode

Environment details

  • Codex CLI 0.147.0; Codex Desktop rollouts written by 0.151.0-alpha.7.1
  • Node 24.16.0, bundled Bun 1.4.0
  • Codex state DB: state_5.sqlite, threads table
  • Install: npm global, launchd service

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. added
    bugSomething isn't working
    cliCLI, config inject, packaging flags
    platformOS/service/tray/ACL (Windows-heavy, not Windows-only)
    on Aug 30, 2026
  2. github-actions commented on Aug 30, 2026

    @github-actions
    Contributor

    Issue reopened

    The report now contains the information required by the automated check. Thanks for updating it.

  3. lidge-jun commented on Aug 30, 2026

    @lidge-jun
    Owner

    리뷰 · 우선순위 74 / 80

    이 이슈는 macOS(그리고 같은 복원 경로를 쓰는 환경)에서 포크/브랜치된 Codex 세션 때문에 ocx stop과 ocx update가 영구히 실패하는 버그입니다. 지금 dev HEAD(870a2adb6, package 2.37.0)의 src/codex/history-provider.ts를 읽어 보면, 제보자가 가리킨 코드가 그대로 있습니다. 이미 고쳐진 상태가 아닙니다.

    복원 전에 모든 매니페스트 항목을 검사하는 preflightRestoreTargets(약 586줄)는 항목마다 snapshotRolloutForRestore(547줄)를 부릅니다. 그 함수는 롤아웃 JSONL에서 readLatestSessionMeta(663줄)로 마지막 session_meta 한 줄만 고릅니다. 포크된 자식 롤아웃은 자기 session_meta 뒤에 부모 스레드의 session_meta가 붙는 일이 흔합니다. 그러면 마지막 줄의 id가 매니페스트 entry.id와 달라서 rolloutMatchesRestoreTuple / rolloutMatchesExpectedPostImage가 둘 다 실패하고 history_backup_rollout_postimage_mismatch가 납니다.

    문제는 이게 한 스레드만 망가뜨리는 게 아니라 전체를 막는다는 점입니다. 프리플라이트가 all-or-nothing이라 매니페스트 수백 개 중 몇 개만 포크여도 복원 전체가 실패합니다. 그 결과 ocx stop이 비정상 종료하고, src/update/index.ts의 stop.status !== 0 게이트(약 259줄)가 ocx update를 멈춥니다. 프록시는 이미 내려갔는데도 「실행 중인 프록시를 멈출 수 없다」는 메시지가 나와 원인 파악이 더 어렵습니다. 관련 이슈 #3008(Windows, DB busy로 복원이 잠깐 실패)은 재시도하면 풀릴 수 있는 일시 오류인데, 이 이슈는 매니페스트/롤아웃을 손으로 고치기 전까지 영원히 같은 상태입니다. 보고자는 389개 중 6개가 막았다고 했습니다.

    같은 파일에 이미 id를 보는 헬퍼가 있습니다. readLatestSessionMetaForIdFromText(680줄)는 뒤에서부터 읽되 payload.id === expectedId인 줄만 고릅니다. compensateConcurrentSessionMetaAppend(705줄)는 이미 이 헬퍼를 씁니다. 그런데 복원 경로의 snapshotRolloutForRestore와 assertRestoreReadback(631줄)만 id 무관 readLatestSessionMeta를 쓰고 있습니다. 제보자 제안처럼 path+id 래퍼로 바꾸면 부모 메타가 뒤에 붙는 정상 포크를 무결성 위반으로 오인하지 않게 됩니다.

    두 번째 결함도 HEAD에 남아 있습니다. rowMatchesExpectedPostImage(487줄)는 has_user_event 드리프트를 entry.modelProvider === "openai"이면서 실제 행이 opencodex로 다시 찍힌 경우에만 허용합니다. 매니페스트에는 openai/vscode/hasUserEvent=0으로 잡혔는데 앱이 아직 provider를 openai로 두고 사용자가 메시지를 써서 has_user_event=1이 된 행은, 튜플 일치도 실패하고 post-image 분기도 타지 못해 history_backup_postimage_mismatch로 또 전체를 막습니다. has_user_event는 Codex가 소유하는 세션 상태이지 opencodex가 쓰는 출처 표식이 아니므로, 드리프트를 무결성 실패로 다루는 범위가 넓어 보입니다.

    우선순위 74는 설치/업데이트 경로를 영구 차단하는 실사용 버그이고, 수정 방향(이미 있는 id-aware 헬퍼 사용 + has_user_event 허용 범위 재검토)이 명확하며, HEAD에서 재현 경로가 코드로 확인됐기 때문입니다. 다만 복원 무결성 완화는 진짜 교차 롤아웃 손상을 놓칠 수 있어 테스트 케이스가 꼭 필요합니다.

    라인 552 - snapshotRolloutForRestore가 id 무관 readLatestSessionMeta를 써서 포크 롤아웃의 부모 session_meta에 걸림
    라인 631 - assertRestoreReadback도 같은 id 무관 조회로 readback 단계에서 동일 실패 가능
    라인 586-596 - preflightRestoreTargets가 항목 하나 실패 시 전체 복원 중단(all-or-nothing)
    라인 487-490 - rowMatchesExpectedPostImage의 has_user_event 허용이 openai→opencodex 재라벨 분기에만 있음
    src/update/index.ts 약 259-266줄 - stop 비정상 종료를「프록시를 못 멈춤」으로 보여 원인(히스토리 복원)이 가려짐
    #3008 - 같은 stop/update 막힘이지만 일시 DB busy. 이 이슈와 구분해 두고, 영구 포크 케이스는 여기(#3026)로 추적

    메인테이너의 판단이 필요한 지점

    • 포크 롤아웃에서 부모 session_meta를 항상 허용할지, 아니면 같은 파일 내 id 일치 메타만 허용할지
    • has_user_event 드리프트를 모든 provider/source에서 무시할지, openai 미재라벨 케이스만 넓힐지
    • all-or-nothing 프리플라이트를 유지한 채 판정만 고칠지, 불량 항목 skip+경고로 완화할지(업데이트 가용성 vs 이력 완전성)
    • #3008과 한 PR로 묶을지(#3008은 재시도/busy, #3026은 id/드리프트 판정)

    너의 추천
    bug로 받고 빠르게 고치세요. snapshotRolloutForRestore와 assertRestoreReadback에서 readLatestSessionMetaForIdFromText(또는 path+id 래퍼)를 쓰게 바꾸고, 부모 메타가 뒤에 붙은 포크 fixture로 회귀 시험을 추가하세요. 이어서 rowMatchesExpectedPostImage에서 has_user_event 드리프트 허용 범위를 미재라벨 openai 행까지 넓히거나, 출처 튜플 비교에서 그 필드를 빼는 쪽을 검토하세요. 업데이트 실패 메시지는 historyRestoreIncomplete와 구분해 히스토리 복원 실패임을 드러내면 운영이 편해집니다. 라벨은 그대로 두었습니다.

    이 댓글은 grok-bot이 작성했습니다

  4. added 2 commits that reference this issue on Aug 31, 2026
    0e892fc
    25ce1ed
  5. lidge-jun commented on Aug 31, 2026

    @lidge-jun
    Owner

    Fixed on dev in 6123be31f (#3103), carrying @ntdat812's commit from #3056 with authorship preserved.

    Both halves you reported are addressed.

    The id-aware read. readLatestSessionMeta returned the last session_meta line regardless of thread id, so a forked rollout's trailing entry belonged to a sibling thread — and one rejected entry fails the whole manifest, which is why ocx stop and ocx update wedged. Your 6-of-389 measurement matched the three call sites exactly.

    The has_user_event drift. Restore recomputed the expected value from first_user_message as it stands now, while routing had derived it from the message at snapshot time. A row routed while its message was null, then given a first message by you, looked exactly like OpenCodex's own write — so restore put it back to 0. The manifest entry now records whether the message was non-empty at snapshot time (a boolean only, never the text), and ownership is decided by which tuple the row wears plus a crash-recoverable relabel marker.

    One case genuinely cannot be decided — an entry wearing its original tuple with 0 → 1 drift whose route would itself have written a 1 is reachable both by "routing never landed and you typed" and by "routing landed and legacy recovery pulled it back". That refuses rather than guessing, and it now says so specifically instead of reporting a generic integrity failure, because retrying reaches the same answer.

    Manifests written before these fields exist restore exactly as they do today.

    Seven review rounds; verified on repository CI at the merged head across Linux, macOS and Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcliCLI, config inject, packaging flagsinstallInstallation or packagingplatformOS/service/tray/ACL (Windows-heavy, not Windows-only)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions