diff --git a/CONTEXT.md b/CONTEXT.md index ebb6490..98d8c2d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -47,3 +47,7 @@ _Avoid_: Upload error、path error、failed upload **Oversized File**: 本機大小超過 Drive 單檔上限(100 MiB)的檔案;sync 不讀取、不上傳,以 `FILE_TOO_LARGE` 回報,直到檔案縮小、消失或被排除。 _Avoid_: Large file、413 file、upload too large + +**Move Rejection**: +Sync 把本機改名轉成 server move 時,server 以非 retryable、非認證的錯誤拒絕(例如 `409 VERSION_CONFLICT`、`409 PATH_CONFLICT`);該組改名本輪略過並以 path error 回報,不改用 upload+delete,也不讓整輪停止。 +_Avoid_: Move failure、move conflict、409 diff --git a/docs/superpowers/specs/2026-06-21-drive-desktop-cli-sync-v1-design.md b/docs/superpowers/specs/2026-06-21-drive-desktop-cli-sync-v1-design.md index 392f47f..cc4dafa 100644 --- a/docs/superpowers/specs/2026-06-21-drive-desktop-cli-sync-v1-design.md +++ b/docs/superpowers/specs/2026-06-21-drive-desktop-cli-sync-v1-design.md @@ -2,7 +2,7 @@ ## Move/delete 確認契約修訂(2026-09-09) -依 [backend final spec](https://github.com/sadcoderlabs/wspc/blob/bb461ad3299ba16ecd5b9e37f3bc313b25e52595/docs/superpowers/specs/2026-09-09-drive-move-delete-confirmation-design.md) 及使用者授權本次 CLI 發布:rm、sync delete、rename move adapter 必須保存原確認 entry ID/path/Entry Version,不 fresh lookup 替換確認。缺 identity 的舊 sync state 拒絕載入,不執行或標記舊刪除成功。Move 失敗停止,不再 fallback upload+delete;conflict 不自動改用最新確認。 +依 [backend final spec](https://github.com/sadcoderlabs/wspc/blob/bb461ad3299ba16ecd5b9e37f3bc313b25e52595/docs/superpowers/specs/2026-09-09-drive-move-delete-confirmation-design.md) 及使用者授權本次 CLI 發布:rm、sync delete、rename move adapter 必須保存原確認 entry ID/path/Entry Version,不 fresh lookup 替換確認。缺 identity 的舊 sync state 拒絕載入,不執行或標記舊刪除成功。Move 失敗停止,不再 fallback upload+delete;conflict 不自動改用最新確認。(2026-09-25 起,非 retryable 的 move 失敗改為只略過該組並回報 path error,不再讓整輪停止;見 [Move Rejection spec](2026-09-25-drive-move-rejection-wedge-design.md)。不 fallback 與不替換確認兩條不變。) 發布流程仍同步 live OpenAPI,再 generate/驗證/publish。因 CLI 須先於 backend rollout,rm 的 required confirmation 由 handwritten command 設定既有 generated command,避免 release 的 live codegen 還原舊 optional 行為;SDK adapter 額外傳送原 entry ID。這不表示舊 server 已驗證 identity:保護能力仍須以 live required fields 及部署驗收為準。取得 release 後須用已發布 package 對 backend candidate 驗收,再允許 backend merge。版本 bump 使用 minor,因 rm 新增 required flags。 diff --git a/docs/superpowers/specs/2026-09-25-drive-move-rejection-wedge-design.md b/docs/superpowers/specs/2026-09-25-drive-move-rejection-wedge-design.md new file mode 100644 index 0000000..e5292a9 --- /dev/null +++ b/docs/superpowers/specs/2026-09-25-drive-move-rejection-wedge-design.md @@ -0,0 +1,101 @@ +# Drive watch 不再因 move 被拒而永久卡在 HTTP 409 + +## Problem Statement + +身為用 wspc-drive app(背後執行 `wspc drive watch`)同步 library 的使用者,我在本機一次改名或搬移大量檔案後,想要同步在中途被停掉也能自己接續完成;就算某一組改名被 server 拒絕,其他檔案仍照常同步,而且看得出是哪一組、為什麼被拒。 + +2026-09-24,`personal` library(約 5,300 個檔案)在本機大量改名後,wspc-drive `events.log` 從 08:58 UTC 起每次啟動 watch 都在 1–3 秒內以 `cli error: error: HTTP 409` 結束,持續到 23:39 仍未恢復。Workers Logs 顯示: + +- 08:46–08:57 之間 `@wspc/cli/0.13.1` 成功呼叫 `POST /drive/libraries/{id}/files/move` 約 323 次,每次 server wall time 平均 1,980 ms(p95 2,863 ms)。這段期間 `events.log` 只有 `sync started`,沒有任何進度;watch 在 08:52:25 與 08:57:39 兩度被停止,兩次都停在 move 階段。 +- 08:58 之後 move 0 次成功、43 次 `409`,與 `events.log` 的 44 行 `HTTP 409` 對應。 +- 同一時段 manifest 請求依序為 `since_cursor=…20439`(08:52)、`since_cursor=…20811`(08:58)。20811 − 20439 = 372,約等於第一輪 186 次 move × 每次兩筆 coordinator event。 + +根因有兩個,第一個讓 state 過期,第二個讓過期變成永久卡住: + +1. **Manifest cursor 在處理前就寫入。** `runDriveSyncOnce()` 取得 manifest(delta 或 full)後立刻把 `latest_cursor` 寫進 `state.json`,接著才執行 `applyRenamesAsMoves()` 與逐 path 處理。這一輪若在處理完前被中斷,下一輪從新 cursor 取 delta,已被跳過的變動不會再出現;`remoteViewFromState()` 用 state 裡的舊 entry 當作遠端現況。 + - 09-24 的順序:第 1 輪最後一個 move 在 server 已完成、state 未寫入就被停止。第 2 輪的 delta 帶回該 move 的兩筆 event(舊 path 刪除、新 path 出現),本應在逐 path 處理時以 `remove_state`/`state_only` 修正,但它先寫入新 cursor,再花 5 分鐘做剩下的 138 個 move,未走到逐 path 處理就又被停止。第 3 輪的 delta 不含那兩筆 event,state 仍記著舊 path 與舊 Entry Version,本機只有新 path 且內容相同,於是再配成 move。Server 上該 entry 早已在新 path,回 `409 VERSION_CONFLICT`。 + - 同一機制也會讓其他裝置的修改在中斷後永遠不被下載(delta 被跳過、state 仍是舊版)。 +2. **Move 被拒時整輪中止,而且沒有任何狀態改變。** `applyRenamesAsMoves()` 不攔截 `moveFile()` 的錯誤,依 [Drive desktop CLI sync v1 spec](2026-06-21-drive-desktop-cli-sync-v1-design.md) 的「Move 失敗停止,不再 fallback upload+delete」往外丟,watch 視為非 retryable 錯誤而結束 process。state 未變,下一次啟動配出同一組 move,永遠重複。 + +另外兩個問題讓它難以診斷: + +- 最外層只印 `error: HTTP 409`,`DriveHttpError.code`(例如 `VERSION_CONFLICT`、`PATH_CONFLICT`)沒有輸出;`debug.log` 只在 move 成功後記錄,失敗的那組 path 沒有紀錄。 +- `onProgress` 在所有 move 做完之後才第一次呼叫,大量改名時 UI 長時間停在 `sync started`,看起來像卡住,使用者因此停止 watch,觸發了根因 1。 + +限制:`state.json` 維持 `schema_version: 1`,不新增欄位;`drive_sync_once`、`drive_sync_progress`、`drive_watch_retry` 與 `path_errors` 的 event contract 不變。沿用 [backend move/delete confirmation spec](https://github.com/sadcoderlabs/wspc/blob/bb461ad3299ba16ecd5b9e37f3bc313b25e52595/docs/superpowers/specs/2026-09-09-drive-move-delete-confirmation-design.md):move 一律送出 state 保存的原確認(entry ID、path、Entry Version),不以 fresh lookup 替換;move 被拒時不 fallback 成 upload+delete。 + +## Solution + +### Manifest cursor 在整輪處理完才寫入 + +一輪 sync 取得的 manifest cursor,只在該輪所有 move 與逐 path 處理都跑完之後才寫入 `state.json`。以下情況不寫入,下一輪沿用舊 cursor 重新取得同一段 delta: + +- 這一輪丟出錯誤(包含 retryable 的 `DriveRetryableSyncError`、auth failure 與 process 被停止)。 +- 逐 path 處理因 state 寫入失敗而提前停止(`result.stop`)。 + +重播同一段 delta 是安全的:已套用到 state 的 path,delta 內容與 state 相同,決策為 `unchanged`;server 已完成但 state 未寫入的 move,會以 `remove_state`(舊 path)與 `state_only`(新 path)補上。 + +### Move 被拒時的處理(Move Rejection) + +`moveFile()` 丟出的錯誤若是 retryable(429、5xx、network)或 auth failure(401/403),維持現行行為往外丟。其餘錯誤(例如 `409 VERSION_CONFLICT`、`409 PATH_CONFLICT`、其他 4xx)視為 Move Rejection: + +1. **本輪用的是 delta 時**:不再執行剩下的 move,改為忽略 cursor 重新取得 full manifest,以新的遠端現況重新判斷改名配對,再繼續執行。每輪最多重新取得一次。原確認不變:若 full manifest 顯示舊 path 已不在遠端,這組不再被視為改名(由逐 path 處理補 state);若仍配成同一組,送出的仍是 state 裡的原確認。 +2. **本輪用的已經是 full manifest 時**:這組的兩個 path 本輪都不處理(不 upload、不 delete),以一筆 path error 回報在新 path 上,`code` 為 server 回傳的錯誤碼(無則 `DRIVE_PATH_ERROR`),`message` 為 `move from <舊 path> rejected (HTTP )`,`retryable: false`。其他 move 與其他 path 照常處理,`errors` 計入 1。下一輪會再嘗試一次,不會讓 watch 結束。 + +### 進度 + +`onProgress` 在執行任何 move 之前先回報 `(0, total)`,`total` 包含本輪預計執行的 move 數量;每完成(或被拒)一個 move,`processed` 加 1。重新取得 full manifest 後若 `total` 改變,以新的 `total` 再回報一次目前進度。 + +### 診斷 + +- `debug.log`(`--debug` 或 `WSPC_DRIVE_DEBUG=1`)新增: + - `manifest`:`mode`(`delta`/`full`)、`since_cursor`(delta 時)、`latest_cursor`、`entries`(筆數)、`resync_required`(delta 被 server 要求改抓 full 時)。 + - `move_rejected`:`from_path`、`to_path`、`entry_id`、`expected_entry_version`、`status`、`code`、`action`(`refetch_full_manifest` 或 `skip_pair`)。 + - `manifest_cursor`:`action: "persist"` 與寫入的 cursor;或 `action: "keep"` 與原因(`round_interrupted`)。 +- CLI 最外層錯誤輸出:錯誤物件同時帶有數字 `status` 與字串 `code`,且 message 不含該 code 時,輸出 `error: ()`,例如 `error: HTTP 409 (VERSION_CONFLICT)`。其他錯誤輸出不變。 + +## Acceptance Criteria + +所有自動化驗證在本機與 CI 執行 `npm test`,使用 vitest、暫存 library 與 fake Drive API;不需要真實帳號或網路。 + +1. **兩次中斷後可收斂**:以有狀態的 fake server(entry ID、path、Entry Version、coordinator event 與 cursor,確認規則同 server `files-repo.ts`)重現 09-24 時序:四個檔案在本機改名,第 1 輪在 server 完成第 2 個 move 後中斷,第 2 輪在完成第 3 個 move 後中斷。之後連續三輪 `runDriveSyncOnce()` 都不丟錯,最後 state 只有改名後的四個 path,server 上也是這四個 path。改動前三輪皆為 `HTTP 409`。自動化,CI。 +2. **中斷後不遺失遠端修改**:其他裝置修改檔案後,本輪在下載時因 `fetch failed` 中斷;下一輪會下載到新內容。改動前本機仍是舊內容。自動化,CI。 +3. **cursor 寫入時機**:delta 取得新 cursor 後,若處理中丟錯或因 state 寫入失敗提前停止,`state.json` 的 `manifest_cursor` 維持原值;整輪完成時為新值(既有 delta/full/expired cursor 測試的期望值不變)。自動化,CI。 +4. **delta 下 move 被拒會改抓 full manifest**:delta 的遠端現況過期導致 move 回 `409 VERSION_CONFLICT` 時,同一輪呼叫一次 full manifest,並以 full manifest 的結果完成同步,summary `errors === 0`;不呼叫 upload 或 delete 補做那組改名。自動化,CI。 +5. **full manifest 下 move 被拒只影響那一組**:move 回 `409 PATH_CONFLICT`(或其他非 retryable 4xx)時,`runDriveSyncOnce()` 不丟錯;該組兩個 path 沒有 upload/delete;`path_errors` 有一筆 `{ path: <新 path>, code: "PATH_CONFLICT", message: "move from <舊 path> rejected (HTTP 409)", retryable: false }`;同一輪其他檔案正常上傳。自動化,CI。 +6. **retryable 與 auth 維持原行為**:move 回 429 仍以 `DriveRetryableSyncError` 中止;回 403 仍往外丟;都不 upload/delete(既有測試)。自動化,CI。 +7. **進度包含 move**:兩組改名加一個新檔時,`onProgress` 依序收到 `[0,3]`、`[1,3]`、`[2,3]`、`[3,3]`。自動化,CI。 +8. **診斷輸出**:move 被拒時 `debug.log` 有 `move_rejected` event,欄位如上;`dispatch()` 對帶 `status` 與 `code` 的錯誤輸出 `error: HTTP 409 (VERSION_CONFLICT)`。自動化,CI。 +9. **Repository checks**:`npm run typecheck`、`npm test`、`npm run build` 通過。CI。 +10. **Live 驗收(post-release,非 merge gate)**:CLI 發佈且 wspc-drive 使用新版後,在卡住的 `personal` library 啟動 watch,第一輪即完成同步,`events.log` 不再出現 `HTTP 409`。需要使用者本機 wspc-drive,手動。 + +## Implementation Decisions + +- **cursor 延後寫入**:`fetchRemoteManifest()` 仍回傳 `manifestCursor`,但不在取得後立即寫入;改在逐 path 迴圈正常結束(非 `stop`)後、`recordUnresolvedConflicts()` 之前,以同樣的條件(`excludeRules.size === 0`、cursor 有定義且與 state 不同)寫入。 +- **改名配對拆成規劃與執行**:`planRenameMoves()` 回傳 1:1 的 `{ fromPath, toPath }` 清單(純函式,規則不變);`applyRenameMoves()` 依序執行,回傳本輪已 move 或已略過的 path(`settledPaths`)與是否因 delta 下被拒而停下(`stoppedOnRejection`)。Move Rejection 的判斷用 `retry.ts` 既有的 `isRetryableDriveFailure()` 與 `isDriveAuthFailure()`。 +- **重新取得 full manifest**:以 `{ ...state, manifest_cursor: undefined }` 呼叫 `fetchRemoteManifest()`,不寫入 state;本輪結束時寫入的是 full manifest 的 `latest_cursor`。manifest normalization 產生的 path error 由 `recordDrivePathError()` 以 path 去重,不會重複計數。 +- **跳過被拒的組合**:兩個 path 與 Oversized File、Permanent Upload Rejection 一樣不進入逐 path 處理,不計入逐 path 的 `total`。 +- **Oversized File 與 Permanent Upload Rejection 的判斷時機**:在所有 move(含重新取得 full manifest 後的重新規劃)結束之後,以最終的遠端現況與最終實際 move 或略過的組合判斷。重新規劃後不再成組的 path 仍會被檢查,不會把已記錄拒收的內容再傳一次。進度預估用同一個純判斷函式,兩者結果一致。不在任何預計配對中的 path 在 move 開始前就先回報,move 階段因 retryable 錯誤中止時,`drive_watch_retry` 的 `path_errors` 仍帶有這些錯誤。`api.moveFile` 不存在時不規劃任何 move。 +- **為什麼不記成 conflict**:這個 repo 的 conflict 指內容衝突,會觸發 conflict copy 與持久化的 `state.conflicts`;Move Rejection 是改名確認失敗,內容沒有衝突,因此以本輪 path error 回報,下一輪自動再判斷。 +- **錯誤輸出**:在 `src/cli.ts` 的 `dispatch()` 處理;不改 `DriveHttpError.message`,避免 path error 的 `message` 與 wspc-drive dashboard 的 `HTTP 413` 判斷改變。 +- **Rollout/rollback**:隨下一個 CLI release 發佈,wspc-drive 升級內附 CLI 即生效。state 格式不變,回退只會恢復舊行為。 +- **外部服務**:無新增服務、credential 或 server 改動。 + +## Testing Decisions + +| AC | 驗證 | +| --- | --- | +| 1、2 | `test/handwritten/drive/sync-interruption.test.ts`:有狀態 fake server,多輪呼叫 `runDriveSyncOnce()`。 | +| 3、4、5、7、8(debug) | `test/handwritten/drive/sync.test.ts`:沿用 `mkApi` 與暫存 library。 | +| 6 | 既有 `sync.test.ts` move 429/403 測試。 | +| 8(CLI 輸出) | `test/cli-error-message.test.ts`:`cliErrorMessage()` 單元測試,以及經由 `dispatch()` 對本機假 HTTP server 的 `drive sync once`。 | + +既有測試 `stops without upload or delete when move has a permanent failure` 斷言 move 一般錯誤會讓整輪中止;新行為改為回報 path error 並繼續。該斷言的修改獨立成一個 commit。 + +## Out of Scope + +- **持續被拒的組合每輪都會重抓一次 full manifest**:略過該組的那一輪仍會寫入新 cursor,下一輪又是 delta,於是再次被拒、再抓一次 full manifest,每輪多出兩次 move 請求與一次完整清單分頁。只有 server 對同一份確認持續拒絕時才會發生(例如非 409 的 4xx);這個代價可接受,換來的是不會永久卡住。 + +- **server move 延遲**:每次 move 約 2 秒是 server 端(`drive-worker` 的 `files/move`)的 wall time,不在 CLI。大量改名仍會花很久,需在 wspc repo 另外處理(例如批次 move API)。 +- **逐 path 永久錯誤與 cursor**:某 path 本輪記錄 path error 但整輪完成時,cursor 仍會前進;若該錯誤剛好發生在一個遠端變動上(例如非 retryable 的下載失敗),那個變動在下一輪的 delta 中不會再出現。本次只處理「整輪中斷」。 +- **Server 端通知遺失**:`files-service.ts` 發送 coordinator 通知時以 `.catch(() => undefined)` 吞掉錯誤,遺失的 event 不會出現在 delta。這次事件沒有證據顯示發生過,另行追蹤。 diff --git a/src/cli.ts b/src/cli.ts index 8329f04..e06562a 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -120,11 +120,20 @@ export async function dispatch(argv: string[], { allowRetry = true }: { allowRet process.exitCode = 2 return } - process.stderr.write(`error: ${(err as Error).message ?? err}\n`) + process.stderr.write(`error: ${cliErrorMessage(err)}\n`) process.exitCode = 1 } } +// HTTP errors keep the server code out of their message, so print it here. +export function cliErrorMessage(err: unknown): string { + const message = err instanceof Error ? err.message : String(err) + if (typeof err !== "object" || err === null) return message + const { status, code } = err as { status?: unknown; code?: unknown } + if (typeof status !== "number" || typeof code !== "string" || message.includes(code)) return message + return `${message} (${code})` +} + if (isCliEntrypoint()) { dispatch(process.argv) } diff --git a/src/handwritten/commands/drive/sync.ts b/src/handwritten/commands/drive/sync.ts index 0fecbeb..fa9c47d 100644 --- a/src/handwritten/commands/drive/sync.ts +++ b/src/handwritten/commands/drive/sync.ts @@ -15,16 +15,20 @@ import { type DriveScanError, type DriveState, type DriveStateEntry, + type DriveUploadRejection, } from "./state.js" import { render } from "../../output/render.js" import { VERSION } from "../../../version.js" import { DriveRetryableSyncError, + isDriveAuthFailure, isRetryableDriveFailure, + type DrivePathErrorSummary, } from "./retry.js" import { cloneDriveState, drivePathErrorSummary, + errorMessage, executeDrivePathAction, recordDrivePathError, stateEntryFromRemote, @@ -134,116 +138,168 @@ export async function runDriveSyncOnce( } const scanMs = Date.now() - scanStartedMs const manifestStartedMs = Date.now() - let manifest: Awaited> - try { - manifest = await fetchRemoteManifest(root, state, syncApi, summary, blockedPaths, debug) - } catch (error) { - if (isRetryableDriveFailure(error)) { - throw new DriveRetryableSyncError(error, { pathErrors: summary.path_errors ?? [] }) + const loadRemoteView = async (fullManifest: boolean): Promise => { + let manifest: Awaited> + try { + manifest = await fetchRemoteManifest( + root, + fullManifest ? { ...state, manifest_cursor: undefined } : state, + syncApi, + summary, + blockedPaths, + debug, + ) + } catch (error) { + if (isRetryableDriveFailure(error)) { + throw new DriveRetryableSyncError(error, { pathErrors: summary.path_errors ?? [] }) + } + throw error } - throw error - } - const remoteFiles = removeExcludedPaths(manifest.remoteFiles, excludeRules) - if ( - excludeRules.size === 0 && - manifest.manifestCursor !== undefined && - manifest.manifestCursor !== state.manifest_cursor - ) { - state = { ...state, manifest_cursor: manifest.manifestCursor } - await writeDriveState(root, state, clock) + const remoteFiles = removeExcludedPaths(manifest.remoteFiles, excludeRules) + const paths = Array.from( + new Set([...Object.keys(localFiles), ...Object.keys(remoteFiles), ...Object.keys(state.entries)]), + ) + .filter((path) => !blockedPaths.has(path) && !excludeRules.matches(path)) + .sort((left, right) => left.localeCompare(right)) + return { remoteFiles, paths, manifestCursor: manifest.manifestCursor, fromDelta: manifest.fromDelta } } + let view = await loadRemoteView(false) const manifestMs = Date.now() - manifestStartedMs - const paths = Array.from( - new Set([...Object.keys(localFiles), ...Object.keys(remoteFiles), ...Object.keys(state.entries)]), - ) - .filter((path) => !blockedPaths.has(path) && !excludeRules.matches(path)) - .sort((left, right) => left.localeCompare(right)) - let movedPaths: Set - try { - movedPaths = await applyRenamesAsMoves({ - root, - state, - api: syncApi, - paths, - localFiles, - remoteFiles, - summary, - clock, - debug, - onStateChange: (nextState) => { - state = nextState - }, - }) - } catch (error) { - if (isRetryableDriveFailure(error)) { - throw new DriveRetryableSyncError(error, { pathErrors: summary.path_errors ?? [] }) - } - throw error - } + let plannedMoves = + syncApi.moveFile === undefined ? [] : planRenameMoves(state, view.paths, localFiles, view.remoteFiles) - const rejectedPaths = new Set() - for (const path of paths) { + // Uploads this round skips: Oversized Files first, then Permanent Upload + // Rejections that still match this scan. Pure, so progress and the final + // skip use the same answer. + const uploadSkip = (path: string): { op: string; pathError: DrivePathErrorSummary } | undefined => { + const action = decideDriveAction(state.entries[path], localFiles[path], view.remoteFiles[path]) + if (action.type !== "upload_create" && action.type !== "upload_update") return undefined const sizeBytes = localFiles[path]?.size_bytes - if (movedPaths.has(path) || sizeBytes === undefined || sizeBytes <= DRIVE_MAX_FILE_SIZE_BYTES) continue - const action = decideDriveAction(state.entries[path], localFiles[path], remoteFiles[path]) - if (action.type !== "upload_create" && action.type !== "upload_update") continue - rejectedPaths.add(path) - const pathError = { - path, - code: "FILE_TOO_LARGE", - message: `file is ${(sizeBytes / 1_048_576).toFixed(1)} MiB (${sizeBytes} bytes); Drive per-file limit is 100 MiB`, - retryable: false, + if (sizeBytes !== undefined && sizeBytes > DRIVE_MAX_FILE_SIZE_BYTES) { + return { + op: "file_too_large", + pathError: { + path, + code: "FILE_TOO_LARGE", + message: `file is ${(sizeBytes / 1_048_576).toFixed(1)} MiB (${sizeBytes} bytes); Drive per-file limit is 100 MiB`, + retryable: false, + }, + } } + const rejection = state.upload_rejections?.[path] + if (rejection === undefined || !isCurrentUploadRejection(rejection, state.scan_cache?.[path])) return undefined + return { op: "upload_rejected", pathError: { path, code: rejection.code, message: rejection.message, retryable: false } } + } + // decideDriveAction is pure and reads only this path's slices of state, so + // a pre-pass count matches the loop's actions. + const countActionable = (paths: string[], excluded: Set) => + paths.filter( + (path) => + !excluded.has(path) && + uploadSkip(path) === undefined && + isActionableAction(decideDriveAction(state.entries[path], localFiles[path], view.remoteFiles[path])), + ).length + const recordUploadSkip = async (path: string, skip: { op: string; pathError: DrivePathErrorSummary }) => { await recordDrivePathError(summary, undefined, path, undefined, { appendPathResult: true, debug, - op: "file_too_large", - pathError, + op: skip.op, + pathError: skip.pathError, }) } + // Report skips outside the planned pairs now, so a retryable failure during + // the moves still carries them to the watch retry event. + const plannedPairPaths = pairPaths(plannedMoves) + for (const path of view.paths) { + if (plannedPairPaths.has(path)) continue + const skip = uploadSkip(path) + if (skip !== undefined) await recordUploadSkip(path, skip) + } + let processed = 0 + let total = plannedMoves.length + countActionable(view.paths, plannedPairPaths) + onProgress?.(processed, total) - const uploadRejections = { ...state.upload_rejections } - for (const [path, rejection] of Object.entries(uploadRejections)) { - const scanned = state.scan_cache?.[path] - if ( - scanned?.mtime_ms !== rejection.mtime_ms || - scanned.size_bytes !== rejection.size_bytes || - scanned.sha256 !== rejection.sha256 || - rejection.cli_version !== VERSION - ) { - delete uploadRejections[path] - continue + const settledPairPaths = new Set() + let fullManifestFetched = false + for (;;) { + let outcome: DriveRenameMovesOutcome + try { + outcome = await applyRenameMoves({ + root, + state, + api: syncApi, + moves: plannedMoves, + localFiles, + summary, + clock, + debug, + stopOnRejection: view.fromDelta && !fullManifestFetched, + onStateChange: (nextState) => { + state = nextState + }, + onMoveSettled: () => { + processed += 1 + onProgress?.(processed, total) + }, + }) + } catch (error) { + if (isRetryableDriveFailure(error)) { + throw new DriveRetryableSyncError(error, { + remaining: total - processed, + pathErrors: summary.path_errors ?? [], + }) + } + throw error } - const action = decideDriveAction(state.entries[path], localFiles[path], remoteFiles[path]) - if (action.type !== "upload_create" && action.type !== "upload_update") continue - rejectedPaths.add(path) - const pathError = { path, code: rejection.code, message: rejection.message, retryable: false } - await recordDrivePathError(summary, undefined, path, undefined, { - appendPathResult: true, - debug, - op: "upload_rejected", - pathError, - }) + for (const path of outcome.settledPaths) settledPairPaths.add(path) + if (!outcome.stoppedOnRejection) break + // A delta view can be stale after an interrupted round; confirm with a full manifest before giving up on the pair. + fullManifestFetched = true + view = await loadRemoteView(true) + plannedMoves = planRenameMoves(state, view.paths, localFiles, view.remoteFiles) + const nextTotal = + processed + + plannedMoves.length + + countActionable(view.paths, new Set([...settledPairPaths, ...pairPaths(plannedMoves)])) + if (nextTotal !== total) { + total = nextTotal + onProgress?.(processed, total) + } + } + + // The view and the pairs are final here; the skips below must use them. + // recordDrivePathError() ignores paths already reported above. + const skippedUploads = new Set() + for (const path of view.paths) { + if (settledPairPaths.has(path)) continue + const skip = uploadSkip(path) + if (skip === undefined) continue + skippedUploads.add(path) + await recordUploadSkip(path, skip) } + const uploadRejections = Object.fromEntries( + Object.entries(state.upload_rejections ?? {}).filter(([path, rejection]) => + isCurrentUploadRejection(rejection, state.scan_cache?.[path]), + ), + ) if (Object.keys(uploadRejections).length !== Object.keys(state.upload_rejections ?? {}).length) { state = { ...state, upload_rejections: uploadRejections } if (Object.keys(uploadRejections).length === 0) delete state.upload_rejections await writeDriveState(root, state, clock) } - const pendingPaths = paths.filter((path) => !movedPaths.has(path) && !rejectedPaths.has(path)) - // decideDriveAction is pure and reads only this path's slices of the - // initial state, so this pre-pass total matches the loop's actions. - const total = pendingPaths.filter((path) => - isActionableAction(decideDriveAction(state.entries[path], localFiles[path], remoteFiles[path])), - ).length - let processed = 0 - onProgress?.(processed, total) + const pendingPaths = view.paths.filter((path) => !settledPairPaths.has(path) && !skippedUploads.has(path)) + const pendingTotal = processed + countActionable(pendingPaths, settledPairPaths) + if (pendingTotal !== total) { + total = pendingTotal + onProgress?.(processed, total) + } const processStartedMs = Date.now() + let roundCompleted = true for (const path of pendingPaths) { - const remote = remoteFiles[path] + const remote = view.remoteFiles[path] const local = localFiles[path] const action = decideDriveAction(state.entries[path], local, remote) if (isActionableAction(action)) { @@ -288,7 +344,24 @@ export async function runDriveSyncOnce( processed += 1 onProgress?.(processed, total) } - if (result.stop) break + if (result.stop) { + roundCompleted = false + break + } + } + + // The cursor is persisted only after every change it covers is applied; an + // interrupted round replays the same delta next time instead of skipping it. + if (!roundCompleted) { + debug.log("manifest_cursor", { action: "keep", reason: "round_interrupted" }) + } else if ( + excludeRules.size === 0 && + view.manifestCursor !== undefined && + view.manifestCursor !== state.manifest_cursor + ) { + state = { ...state, manifest_cursor: view.manifestCursor } + await writeDriveState(root, state, clock) + debug.log("manifest_cursor", { action: "persist", cursor: view.manifestCursor }) } recordUnresolvedConflicts(summary, state) @@ -377,10 +450,17 @@ async function fetchRemoteManifest( summary: DriveSyncSummary, blockedPaths: Set, debug: DriveDebugLogger, -): Promise<{ remoteFiles: Record; manifestCursor: string | undefined }> { +): Promise<{ remoteFiles: Record; manifestCursor: string | undefined; fromDelta: boolean }> { + let resyncRequired = false if (state.manifest_cursor !== undefined) { const delta = await api.getManifest(state.library_id, undefined, state.manifest_cursor) if (delta.resync_required !== true) { + debug.log("manifest", { + mode: "delta", + since_cursor: state.manifest_cursor, + ...(delta.latest_cursor === undefined ? {} : { latest_cursor: delta.latest_cursor }), + entries: delta.entries.length, + }) const remoteFiles = remoteViewFromState(state) const changed = delta.entries.filter((entry) => entry.deleted_at === undefined) const normalized = normalizeRemoteManifest(root, changed) @@ -395,9 +475,10 @@ async function fetchRemoteManifest( if (entry.deleted_at !== undefined) delete remoteFiles[entry.path] } Object.assign(remoteFiles, normalized.remoteFiles) - return { remoteFiles, manifestCursor: delta.latest_cursor ?? state.manifest_cursor } + return { remoteFiles, manifestCursor: delta.latest_cursor ?? state.manifest_cursor, fromDelta: true } } // resync_required: cursor pruned or invalid, fall back to a full fetch. + resyncRequired = true } const entries: RemoteEntry[] = [] @@ -409,6 +490,12 @@ async function fetchRemoteManifest( if (page.latest_cursor !== undefined) latestCursor = page.latest_cursor cursor = page.next_cursor ?? undefined } while (cursor !== undefined) + debug.log("manifest", { + mode: "full", + ...(latestCursor === undefined ? {} : { latest_cursor: latestCursor }), + entries: entries.length, + ...(resyncRequired ? { resync_required: true } : {}), + }) const normalized = normalizeRemoteManifest(root, entries) for (const pathError of normalized.pathErrors) { @@ -418,7 +505,7 @@ async function fetchRemoteManifest( op: "manifest", }) } - return { remoteFiles: normalized.remoteFiles, manifestCursor: latestCursor } + return { remoteFiles: normalized.remoteFiles, manifestCursor: latestCursor, fromDelta: false } } // Reconstructs the last-known remote view from base state so a manifest delta @@ -440,27 +527,38 @@ function remoteViewFromState(state: DriveState): Record { return remoteFiles } -// Detects local renames (a delete_remote and an upload_create with the same -// content hash) and applies them via the server move API, preserving version -// history and skipping a full re-upload. Only unambiguous 1:1 hash pairs are -// moved; anything else falls back to normal upload + delete processing. -async function applyRenamesAsMoves(args: { - root: string - state: DriveState - api: DriveSyncApi - paths: string[] - localFiles: Record +interface DriveRemoteView { remoteFiles: Record - summary: DriveSyncSummary - clock: DriveClock - debug: DriveDebugLogger - onStateChange: (state: DriveState) => void -}): Promise> { - const { root, api, paths, localFiles, remoteFiles, summary, clock, debug, onStateChange } = args - const movedPaths = new Set() - if (api.moveFile === undefined) return movedPaths + paths: string[] + manifestCursor: string | undefined + fromDelta: boolean +} - let state = args.state +interface DriveRenameMove { + fromPath: string + toPath: string +} + +interface DriveRenameMovesOutcome { + // Pairs that were moved or skipped as rejected; neither path is processed again this round. + settledPaths: Set + stoppedOnRejection: boolean +} + +function pairPaths(moves: DriveRenameMove[]): Set { + return new Set(moves.flatMap((move) => [move.fromPath, move.toPath])) +} + +// Detects local renames (a delete_remote and an upload_create with the same +// content hash) so they can go through the server move API, preserving +// version history and skipping a full re-upload. Only unambiguous 1:1 hash +// pairs are moved; anything else falls back to normal upload + delete processing. +function planRenameMoves( + state: DriveState, + paths: string[], + localFiles: Record, + remoteFiles: Record, +): DriveRenameMove[] { const deletesBySha = new Map() const createsBySha = new Map() for (const path of paths) { @@ -475,15 +573,73 @@ async function applyRenamesAsMoves(args: { } } + const moves: DriveRenameMove[] = [] for (const [sha, fromPaths] of deletesBySha) { const toPaths = createsBySha.get(sha) if (fromPaths.length !== 1 || toPaths === undefined || toPaths.length !== 1) continue - const fromPath = fromPaths[0]! - const toPath = toPaths[0]! + moves.push({ fromPath: fromPaths[0]!, toPath: toPaths[0]! }) + } + return moves +} + +// A rejected move never falls back to upload + delete and never retries with a +// fresh confirmation. On a delta view it stops so the caller can confirm with a +// full manifest; otherwise the pair is skipped for this round as a path error. +async function applyRenameMoves(args: { + root: string + state: DriveState + api: DriveSyncApi + moves: DriveRenameMove[] + localFiles: Record + summary: DriveSyncSummary + clock: DriveClock + debug: DriveDebugLogger + stopOnRejection: boolean + onStateChange: (state: DriveState) => void + onMoveSettled: () => void +}): Promise { + const { root, api, moves, localFiles, summary, clock, debug, stopOnRejection, onStateChange, onMoveSettled } = args + const settledPaths = new Set() + if (api.moveFile === undefined) return { settledPaths, stoppedOnRejection: false } + + let state = args.state + for (const { fromPath, toPath } of moves) { const entry = state.entries[fromPath] const local = localFiles[toPath] if (entry === undefined || local === undefined) continue - const moved = await api.moveFile(state.library_id, fromPath, toPath, entry.entry_version, entry.entry_id) + let moved: Awaited>> + try { + moved = await api.moveFile(state.library_id, fromPath, toPath, entry.entry_version, entry.entry_id) + } catch (error) { + if (isRetryableDriveFailure(error) || isDriveAuthFailure(error)) throw error + const status = structuredField(error, "status") + const code = structuredField(error, "code") + debug.log("move_rejected", { + from_path: fromPath, + to_path: toPath, + entry_id: entry.entry_id, + expected_entry_version: entry.entry_version, + ...(typeof status === "number" ? { status } : {}), + ...(typeof code === "string" ? { code } : {}), + action: stopOnRejection ? "refetch_full_manifest" : "skip_pair", + }) + if (stopOnRejection) return { settledPaths, stoppedOnRejection: true } + settledPaths.add(fromPath) + settledPaths.add(toPath) + await recordDrivePathError(summary, undefined, toPath, error, { + appendPathResult: true, + debug, + op: "move", + pathError: { + path: toPath, + code: typeof code === "string" ? code : "DRIVE_PATH_ERROR", + message: `move from ${fromPath} rejected (${typeof status === "number" ? `HTTP ${status}` : errorMessage(error)})`, + retryable: false, + }, + }) + onMoveSettled() + continue + } const nextState = cloneDriveState(state) delete nextState.entries[fromPath] delete nextState.conflicts[fromPath] @@ -492,12 +648,26 @@ async function applyRenamesAsMoves(args: { await writeDriveState(root, nextState, clock) state = nextState onStateChange(nextState) - movedPaths.add(fromPath) - movedPaths.add(toPath) + settledPaths.add(fromPath) + settledPaths.add(toPath) summary.paths.push({ path: toPath, action: "move" }) debug.log("decision", { path: toPath, action: "move", from_path: fromPath }) + onMoveSettled() } - return movedPaths + return { settledPaths, stoppedOnRejection: false } +} + +function isCurrentUploadRejection(rejection: DriveUploadRejection, scanned: DriveScanCacheEntry | undefined): boolean { + return ( + scanned?.mtime_ms === rejection.mtime_ms && + scanned.size_bytes === rejection.size_bytes && + scanned.sha256 === rejection.sha256 && + rejection.cli_version === VERSION + ) +} + +function structuredField(error: unknown, key: "status" | "code"): unknown { + return typeof error === "object" && error !== null ? (error as Record)[key] : undefined } function recordUnresolvedConflicts(summary: DriveSyncSummary, state: DriveState): void { diff --git a/test/cli-error-message.test.ts b/test/cli-error-message.test.ts new file mode 100644 index 0000000..9487a64 --- /dev/null +++ b/test/cli-error-message.test.ts @@ -0,0 +1,79 @@ +import { createServer, type Server } from "node:http" +import type { AddressInfo } from "node:net" +import { mkdir, mkdtemp, writeFile } from "node:fs/promises" +import { join } from "node:path" +import { tmpdir } from "node:os" +import { afterEach, describe, expect, it, vi } from "vitest" +import { cliErrorMessage, dispatch } from "../src/cli.js" +import { initDriveState } from "../src/handwritten/commands/drive/state.js" +import { DriveHttpError } from "../src/handwritten/commands/drive/retry.js" + +// The config directory is resolved from HOME when the config module loads. +const testHome = await vi.hoisted(async () => { + const { mkdtempSync } = await import("node:fs") + const { tmpdir } = await import("node:os") + const { join } = await import("node:path") + const home = mkdtempSync(join(tmpdir(), "wspc-cli-error-home-")) + process.env.HOME = home + process.env.USERPROFILE = home + return home +}) + +describe("CLI error message", () => { + it("prints the server error code next to the HTTP status", () => { + expect(cliErrorMessage(new DriveHttpError(409, { code: "VERSION_CONFLICT" }))).toBe("HTTP 409 (VERSION_CONFLICT)") + }) + + it("prints only the status when the server sent no code", () => { + expect(cliErrorMessage(new DriveHttpError(409))).toBe("HTTP 409") + }) + + it("leaves errors without an HTTP status unchanged", () => { + const lockHeld = Object.assign(new Error("sync lock already exists"), { code: "WSPC_DRIVE_LOCK_HELD" }) + expect(cliErrorMessage(lockHeld)).toBe("sync lock already exists") + expect(cliErrorMessage("plain failure")).toBe("plain failure") + }) +}) + +describe("CLI error output", () => { + let server: Server | undefined + + afterEach(async () => { + process.exitCode = undefined + vi.restoreAllMocks() + await new Promise((resolve) => (server ? server.close(() => resolve()) : resolve())) + server = undefined + }) + + it("prints the server code when a drive sync request is rejected", async () => { + server = createServer((_request, response) => { + response.writeHead(409, { "content-type": "application/json" }) + response.end(JSON.stringify({ error: { code: "VERSION_CONFLICT", message: "The remote entry changed." } })) + }) + await new Promise((resolve) => server!.listen(0, "127.0.0.1", () => resolve())) + const { port } = server.address() as AddressInfo + await mkdir(join(testHome, ".wspc"), { recursive: true }) + await writeFile( + join(testHome, ".wspc", "config.json"), + JSON.stringify({ + schema_version: 2, + current_env: "prod", + envs: { + prod: { + api_base: `http://127.0.0.1:${port}`, + current_account: "user@example.com", + accounts: { "user@example.com": { email: "user@example.com", api_key: "wspc_test" } }, + }, + }, + }), + ) + const folder = await mkdtemp(join(tmpdir(), "wspc-cli-error-folder-")) + await initDriveState(folder, "lib_1") + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true) + + await dispatch(["node", "wspc", "drive", "sync", "once", folder]) + + expect(stderr).toHaveBeenCalledWith("error: HTTP 409 (VERSION_CONFLICT)\n") + expect(process.exitCode).toBe(1) + }) +}) diff --git a/test/handwritten/drive/sync-interruption.test.ts b/test/handwritten/drive/sync-interruption.test.ts new file mode 100644 index 0000000..d59afe6 --- /dev/null +++ b/test/handwritten/drive/sync-interruption.test.ts @@ -0,0 +1,197 @@ +import { createHash } from "node:crypto" +import { mkdtemp, readFile, rename, writeFile } from "node:fs/promises" +import { join } from "node:path" +import { tmpdir } from "node:os" +import { describe, expect, it } from "vitest" +import { initDriveState, readDriveState } from "../../../src/handwritten/commands/drive/state.js" +import { runDriveSyncOnce, type DriveSyncApi } from "../../../src/handwritten/commands/drive/sync.js" +import { DriveHttpError } from "../../../src/handwritten/commands/drive/retry.js" + +// Stateful fake of the drive worker: entry rows keyed by id, a coordinator event log +// with cursors, and the same move/upload/delete confirmation checks as files-repo.ts. +type Row = { id: string; path: string; version: number; content: string; versionId: string; deleted: boolean } +type Event = { seq: number; reason: "file_uploaded" | "file_deleted"; row: Row; deletedAt?: string } + +function sha(content: string) { + return createHash("sha256").update(content).digest("hex") +} + +function fakeServer() { + const rows = new Map() + const events: Event[] = [] + let nextId = 1 + let nextVersion = 1 + let failNext: { op: string; error: unknown } | undefined + const moveErrors: Array = [] + const active = (path: string) => [...rows.values()].find((r) => r.path === path && !r.deleted) + const publish = (reason: Event["reason"], row: Row, deletedAt?: string) => + events.push({ seq: events.length + 1, reason, row: { ...row }, ...(deletedAt ? { deletedAt } : {}) }) + const toEntry = (row: Row, deletedAt?: string) => ({ + id: row.id, + path: row.path, + kind: "file" as const, + entry_version: row.version, + current_version_id: row.versionId, + content_sha256: sha(row.content), + size_bytes: Buffer.byteLength(row.content), + updated_at: "2026-09-24T00:00:00.000Z", + ...(deletedAt ? { deleted_at: deletedAt } : {}), + }) + const maybeFail = (op: string) => { + if (failNext?.op === op) { + const error = failNext.error + failNext = undefined + throw error + } + } + const library = { id: "lib_1", org_id: "org_1", name: "personal", version: 1, file_count: 0, storage_bytes: 0, created_by_user_id: "usr_1", created_at: 1, updated_at: 2 } + const cursor = () => (events.length === 0 ? undefined : `c${events.length}`) + + const server = { + moveErrors, + activePaths: () => [...rows.values()].filter((r) => !r.deleted).map((r) => r.path).sort(), + failNext(op: string, error: unknown) { + failNext = { op, error } + }, + // Another device / web / MCP writes the file. + remoteWrite(path: string, content: string) { + const row = active(path) + if (row) { + row.version += 1 + row.content = content + row.versionId = `ver_${nextVersion++}` + publish("file_uploaded", row) + } else { + const created: Row = { id: `ent_${nextId++}`, path, version: 1, content, versionId: `ver_${nextVersion++}`, deleted: false } + rows.set(created.id, created) + publish("file_uploaded", created) + } + }, + remoteMove(from: string, to: string) { + const row = active(from)! + row.path = to + row.version += 1 + publish("file_deleted", { ...row, path: from }, "2026-09-24T00:00:00.000Z") + publish("file_uploaded", row) + }, + api: { + async getManifest(_id: string, _cursor?: string, sinceCursor?: string) { + maybeFail("manifest") + if (sinceCursor !== undefined) { + const since = Number(sinceCursor.slice(1)) + const byPath = new Map() + for (const event of events.filter((e) => e.seq > since)) byPath.set(event.row.path, event) + const entries = [...byPath.values()].map((e) => toEntry(e.row, e.reason === "file_deleted" ? e.deletedAt : undefined)) + return { library, entries, next_cursor: null, ...(cursor() ? { latest_cursor: cursor()! } : {}) } + } + const entries = [...rows.values()].filter((r) => !r.deleted).map((r) => toEntry(r)) + return { library, entries, next_cursor: null, ...(cursor() ? { latest_cursor: cursor()! } : {}) } + }, + async uploadFile(_id: string, path: string, body: BodyInit, _digest: string, expected?: number) { + maybeFail("upload") + const content = typeof body === "string" ? body : Buffer.from(await new Response(body).arrayBuffer()).toString("utf8") + const row = active(path) + if ((row?.version ?? 0) !== (expected ?? 0)) throw new DriveHttpError(409, { code: "VERSION_CONFLICT" }) + server.remoteWrite(path, content) + return { entry: toEntry(active(path)!), result: row ? ("updated" as const) : ("created" as const) } + }, + async downloadFile(_id: string, path: string, versionId?: string) { + maybeFail("download") + const row = active(path) + if (!row || (versionId !== undefined && row.versionId !== versionId)) throw new DriveHttpError(404, { code: "NOT_FOUND" }) + return new Response(row.content) + }, + async deleteFile(_id: string, path: string, expected: number, entryId: string) { + maybeFail("delete") + const row = rows.get(entryId) + if (!row || row.deleted || row.path !== path || row.version !== expected) throw new DriveHttpError(409, { code: "VERSION_CONFLICT" }) + row.deleted = true + row.version += 1 + publish("file_deleted", row, "2026-09-24T00:00:00.000Z") + return { entry: toEntry(row, "2026-09-24T00:00:00.000Z"), result: "deleted" } + }, + async moveFile(_id: string, from: string, to: string, expected: number, entryId: string) { + maybeFail("move") + const row = rows.get(entryId) + let code: string | undefined + if (!row || row.deleted || row.path !== from || row.version !== expected) code = "VERSION_CONFLICT" + else if (active(to)) code = "PATH_CONFLICT" + moveErrors.push(code) + if (code) throw new DriveHttpError(409, { code }) + server.remoteMove(from, to) + return { entry: toEntry(row!), result: "moved" as const } + }, + } satisfies DriveSyncApi, + } + return server +} + +async function syncedFolder(files: Record) { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-409-")) + await initDriveState(root, "lib_1") + const server = fakeServer() + for (const [path, content] of Object.entries(files)) server.remoteWrite(path, content) + await runDriveSyncOnce(root, server.api) + return { root, server } +} + +describe("drive sync interrupted mid-round", () => { + it("a remote edit seen by an interrupted sync still reaches the folder on the next sync", async () => { + const { root, server } = await syncedFolder({ "notes.md": "v1\n" }) + server.remoteWrite("notes.md", "v2 from another device\n") + + server.failNext("download", new TypeError("fetch failed")) + await expect(runDriveSyncOnce(root, server.api)).rejects.toThrow("fetch failed") + + await runDriveSyncOnce(root, server.api) + expect(await readFile(join(root, "notes.md"), "utf8")).toBe("v2 from another device\n") + }) + + it("a local rename after an interrupted sync does not wedge every later sync on HTTP 409", async () => { + const { root, server } = await syncedFolder({ "notes.md": "v1\n" }) + server.remoteWrite("notes.md", "v2 from another device\n") + server.failNext("download", new TypeError("fetch failed")) + await expect(runDriveSyncOnce(root, server.api)).rejects.toThrow("fetch failed") + + await rename(join(root, "notes.md"), join(root, "renamed.md")) + + const outcomes: string[] = [] + for (let run = 0; run < 3; run++) { + outcomes.push(await runDriveSyncOnce(root, server.api).then(() => "ok", (error: Error) => error.message)) + } + expect(outcomes).toEqual(["ok", "ok", "ok"]) + }) + + // Production timeline 2026-09-24 (personal library): a bulk local rename made the CLI + // run ~320 moves (~2 s each, no progress shown). The watch was stopped twice mid-moves; + // the first sync after the second stop got HTTP 409 on move and never recovered. + it("a bulk rename interrupted twice mid-move converges instead of wedging on HTTP 409", async () => { + const { root, server } = await syncedFolder({ "a.md": "a\n", "b.md": "b\n", "c.md": "c\n", "d.md": "d\n" }) + for (const name of ["a", "b", "c", "d"]) await rename(join(root, `${name}.md`), join(root, `${name}-renamed.md`)) + + // The server applies a move but the round dies before the CLI records it (watch stopped, or the response is lost). + const moveFile = server.api.moveFile.bind(server.api) + let killAfterMove: string | undefined + server.api.moveFile = async (id, from, to, expected, entryId) => { + const moved = await moveFile(id, from, to, expected, entryId) + if (from === killAfterMove) throw new TypeError("fetch failed") + return moved + } + + killAfterMove = "b.md" + await expect(runDriveSyncOnce(root, server.api)).rejects.toThrow("fetch failed") + killAfterMove = "c.md" + await expect(runDriveSyncOnce(root, server.api)).rejects.toThrow("fetch failed") + killAfterMove = undefined + + const outcomes: string[] = [] + for (let run = 0; run < 3; run++) { + outcomes.push(await runDriveSyncOnce(root, server.api).then(() => "ok", (error: Error) => error.message)) + } + expect(outcomes).toEqual(["ok", "ok", "ok"]) + const renamed = ["a-renamed.md", "b-renamed.md", "c-renamed.md", "d-renamed.md"] + expect(Object.keys((await readDriveState(root)).entries).sort()).toEqual(renamed) + expect(server.activePaths()).toEqual(renamed) + expect(server.moveErrors).toEqual([undefined, undefined, undefined, undefined]) + }) +}) diff --git a/test/handwritten/drive/sync.test.ts b/test/handwritten/drive/sync.test.ts index 98e46e9..69f71b3 100644 --- a/test/handwritten/drive/sync.test.ts +++ b/test/handwritten/drive/sync.test.ts @@ -1,6 +1,6 @@ import { createHash } from "node:crypto" import { spawnSync } from "node:child_process" -import { mkdir, mkdtemp, readFile, readdir, rename, truncate, unlink, utimes, writeFile } from "node:fs/promises" +import { mkdir, mkdtemp, readFile, readdir, rename, stat, truncate, unlink, utimes, writeFile } from "node:fs/promises" import { join } from "node:path" import { tmpdir } from "node:os" import { DateTime } from "luxon" @@ -1509,7 +1509,7 @@ describe("drive sync once", () => { expect(after.entries["new-name.md"]).toMatchObject({ entry_version: 2 }) }) - it("stops without upload or delete when move has a permanent failure", async () => { + it("skips the pair without upload or delete when move has a permanent failure", async () => { const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-fallback-")) const state = await initDriveState(root, "lib_1") state.entries["old-name.md"] = stateEntry("old-name.md", "same content\n", 1) @@ -1520,7 +1520,16 @@ describe("drive sync once", () => { throw new Error("move unsupported") } - await expect(runDriveSyncOnce(root, api)).rejects.toThrow("move unsupported") + const result = await runDriveSyncOnce(root, api) + + expect(result.path_errors).toEqual([ + { + path: "new-name.md", + code: "DRIVE_PATH_ERROR", + message: "move from old-name.md rejected (move unsupported)", + retryable: false, + }, + ]) expect(api.uploads).toEqual([]) expect(api.deletes).toEqual([]) }) @@ -1544,6 +1553,25 @@ describe("drive sync once", () => { expect(api.deletes).toEqual([]) }) + it("keeps reporting an oversized file when a rate-limited move interrupts the round", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-rate-limit-oversized-")) + const state = await initDriveState(root, "lib_1") + state.entries["old-name.md"] = stateEntry("old-name.md", "same content\n", 1) + await writeDriveState(root, state) + await writeFile(join(root, "new-name.md"), "same content\n") + await writeFile(join(root, "big.bin"), "") + await truncate(join(root, "big.bin"), 100 * 1024 * 1024 + 1) + const api = mkApi([{ entries: [entry("old-name.md", "same content\n", 1)] }]) + api.moveFile = async () => { + throw new DriveHttpError(429, { retryAfterMs: 60_000 }) + } + + await expect(runDriveSyncOnce(root, api)).rejects.toMatchObject({ + name: "DriveRetryableSyncError", + pathErrors: [expect.objectContaining({ path: "big.bin", code: "FILE_TOO_LARGE", retryable: false })], + }) + }) + it("does not degrade a forbidden move into upload and delete", async () => { const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-forbidden-")) const state = await initDriveState(root, "lib_1") @@ -2214,6 +2242,243 @@ describe("drive sync once", () => { expect(res.stdout).toContain("once") expect(res.stdout).toContain("Run one Drive sync pass") }) + + it("keeps the stored manifest cursor when the round is interrupted after the manifest fetch", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-cursor-interrupted-")) + const state = await initDriveState(root, "lib_1") + state.manifest_cursor = "000000000000000005" + await writeDriveState(root, state) + const api = mkApi([{ entries: [entry("b.txt", "bbb", 1)], latest_cursor: "000000000000000007" }]) + api.downloadFile = async () => { + throw new TypeError("fetch failed") + } + + await expect(runDriveSyncOnce(root, api)).rejects.toMatchObject({ name: "DriveRetryableSyncError" }) + + expect((await readDriveState(root)).manifest_cursor).toBe("000000000000000005") + }) + + it("keeps the stored manifest cursor when the round stops on a state write failure", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-cursor-stop-")) + const state = await initDriveState(root, "lib_1") + state.manifest_cursor = "000000000000000005" + await writeDriveState(root, state) + const api = mkApi([{ entries: [entry("a.txt", "remote", 3)], latest_cursor: "000000000000000007" }]) + api.downloads.set("a.txt", "remote") + stateWriteControl.failNext = (candidate) => + (candidate as { entries?: Record }).entries?.["a.txt"] ? new Error("state write failed once") : undefined + + const result = await runDriveSyncOnce(root, api) + + expect(result.errors).toBe(1) + expect((await readDriveState(root)).manifest_cursor).toBe("000000000000000005") + }) + + it("refetches a full manifest instead of stopping when a move is rejected on a delta view", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-stale-delta-")) + const state = await initDriveState(root, "lib_1") + // Stale base: the server already moved old-name.md, but the delta that said so was skipped. + state.entries["old-name.md"] = stateEntry("old-name.md", "same content\n", 1) + state.manifest_cursor = "000000000000000005" + await writeDriveState(root, state) + await writeFile(join(root, "new-name.md"), "same content\n") + const moved = { ...entry("new-name.md", "same content\n", 2), id: stateEntry("old-name.md", "same content\n", 1).entry_id } + const api = mkApi([ + { entries: [], latest_cursor: "000000000000000005" }, + { entries: [moved], latest_cursor: "000000000000000009" }, + ]) + const moves: string[] = [] + api.moveFile = async (_id, fromPath, toPath) => { + moves.push(`${fromPath}->${toPath}`) + throw new DriveHttpError(409, { code: "VERSION_CONFLICT" }) + } + + const result = await runDriveSyncOnce(root, api) + + expect(moves).toEqual(["old-name.md->new-name.md"]) + expect(api.deltas).toEqual(["000000000000000005"]) + expect(api.manifests).toEqual([""]) + expect(api.uploads).toEqual([]) + expect(api.deletes).toEqual([]) + expect(result.errors).toBe(0) + const after = await readDriveState(root) + expect(Object.keys(after.entries)).toEqual(["new-name.md"]) + expect(after.entries["new-name.md"]).toMatchObject({ entry_version: 2 }) + expect(after.manifest_cursor).toBe("000000000000000009") + }) + + it("still skips a rejected upload when a full manifest refetch drops its rename pair", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-refetch-rejection-")) + const state = await initDriveState(root, "lib_1") + state.entries["a.md"] = stateEntry("a.md", "same\n", 1) + state.manifest_cursor = "000000000000000005" + await writeFile(join(root, "a2.md"), "same\n") + const scanned = await stat(join(root, "a2.md")) + state.upload_rejections = { + "a2.md": { + mtime_ms: scanned.mtimeMs, + size_bytes: scanned.size, + sha256: sha256("same\n"), + code: "DRIVE_PATH_ERROR", + message: "HTTP 415", + cli_version: VERSION, + rejected_at: "2026-09-25T00:00:00.000Z", + }, + } + await writeDriveState(root, state) + // The full manifest shows a.md edited elsewhere, so a.md -> a2.md is no longer a rename. + const edited = { ...entry("a.md", "edited elsewhere\n", 2), id: stateEntry("a.md", "same\n", 1).entry_id } + const api = mkApi([ + { entries: [], latest_cursor: "000000000000000005" }, + { entries: [edited], latest_cursor: "000000000000000009" }, + ]) + api.downloads.set("a.md", "edited elsewhere\n") + api.downloads.set("a.md@ver_2", "edited elsewhere\n") + api.moveFile = async () => { + throw new DriveHttpError(409, { code: "VERSION_CONFLICT" }) + } + + const result = await runDriveSyncOnce(root, api) + + expect(api.manifests).toEqual([""]) + expect(uploadCount(api, "a2.md")).toBe(0) + expect(result.path_errors).toContainEqual({ path: "a2.md", code: "DRIVE_PATH_ERROR", message: "HTTP 415", retryable: false }) + }) + + it("resends the saved confirmation when the full manifest still pairs the rename", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-refetch-same-pair-")) + const state = await initDriveState(root, "lib_1") + state.entries["old-name.md"] = stateEntry("old-name.md", "same content\n", 1) + state.manifest_cursor = "000000000000000005" + await writeDriveState(root, state) + await writeFile(join(root, "new-name.md"), "same content\n") + const api = mkApi([ + { entries: [], latest_cursor: "000000000000000005" }, + { entries: [entry("old-name.md", "same content\n", 1)], latest_cursor: "000000000000000009" }, + ]) + const calls: unknown[][] = [] + api.moveFile = async (...args) => { + calls.push(args) + throw new DriveHttpError(400, { code: "VALIDATION_ERROR" }) + } + + const result = await runDriveSyncOnce(root, api) + + const confirmation = ["lib_1", "old-name.md", "new-name.md", 1, stateEntry("old-name.md", "same content\n", 1).entry_id] + expect(calls).toEqual([confirmation, confirmation]) + expect(api.uploads).toEqual([]) + expect(api.deletes).toEqual([]) + expect(result.path_errors).toEqual([ + { path: "new-name.md", code: "VALIDATION_ERROR", message: "move from old-name.md rejected (HTTP 400)", retryable: false }, + ]) + }) + + it("logs whether the manifest cursor was persisted or kept", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-cursor-debug-")) + const state = await initDriveState(root, "lib_1") + state.manifest_cursor = "000000000000000005" + await writeDriveState(root, state) + const api = mkApi([ + { entries: [entry("a.txt", "remote", 3)], latest_cursor: "000000000000000007" }, + { entries: [entry("a.txt", "remote", 3)], latest_cursor: "000000000000000007" }, + ]) + api.downloads.set("a.txt", "remote") + const events: Array<{ event: string; fields?: Record }> = [] + const debug = { log: (event: string, fields?: Record) => events.push({ event, fields }) } + stateWriteControl.failNext = (candidate) => + (candidate as { entries?: Record }).entries?.["a.txt"] ? new Error("state write failed once") : undefined + + await runDriveSyncOnce(root, api, undefined, undefined, debug) + await runDriveSyncOnce(root, api, undefined, undefined, debug) + + expect(events.filter((logged) => logged.event === "manifest_cursor")).toEqual([ + { event: "manifest_cursor", fields: { action: "keep", reason: "round_interrupted" } }, + { event: "manifest_cursor", fields: { action: "persist", cursor: "000000000000000007" } }, + ]) + }) + + it("reports a move rejected on a full view as a path error and keeps syncing other files", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-rejected-")) + const state = await initDriveState(root, "lib_1") + state.entries["old-name.md"] = stateEntry("old-name.md", "same content\n", 1) + await writeDriveState(root, state) + await writeFile(join(root, "new-name.md"), "same content\n") + await writeFile(join(root, "other.md"), "other\n") + const api = mkApi([{ entries: [entry("old-name.md", "same content\n", 1)], latest_cursor: "000000000000000003" }]) + api.moveFile = async () => { + throw new DriveHttpError(409, { code: "PATH_CONFLICT" }) + } + + const result = await runDriveSyncOnce(root, api) + + expect(api.manifests).toEqual([""]) + expect(api.uploads.map((upload) => upload.path)).toEqual(["other.md"]) + expect(api.deletes).toEqual([]) + expect(result.errors).toBe(1) + expect(result.path_errors).toEqual([ + { path: "new-name.md", code: "PATH_CONFLICT", message: "move from old-name.md rejected (HTTP 409)", retryable: false }, + ]) + expect((await readDriveState(root)).entries["old-name.md"]).toMatchObject({ entry_version: 1 }) + }) + + it("logs a move_rejected debug event with the pair, confirmation, and server code", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-rejected-debug-")) + const state = await initDriveState(root, "lib_1") + state.entries["old-name.md"] = stateEntry("old-name.md", "same content\n", 1) + await writeDriveState(root, state) + await writeFile(join(root, "new-name.md"), "same content\n") + const api = mkApi([{ entries: [entry("old-name.md", "same content\n", 1)] }]) + api.moveFile = async () => { + throw new DriveHttpError(409, { code: "PATH_CONFLICT" }) + } + const events: Array<{ event: string; fields?: Record }> = [] + const debug = { log: (event: string, fields?: Record) => events.push({ event, fields }) } + + await runDriveSyncOnce(root, api, undefined, undefined, debug) + + expect(events.filter((logged) => logged.event === "move_rejected")).toEqual([ + { + event: "move_rejected", + fields: { + from_path: "old-name.md", + to_path: "new-name.md", + entry_id: stateEntry("old-name.md", "same content\n", 1).entry_id, + expected_entry_version: 1, + status: 409, + code: "PATH_CONFLICT", + action: "skip_pair", + }, + }, + ]) + }) + + it("counts planned moves in progress before running them", async () => { + const root = await mkdtemp(join(tmpdir(), "wspc-drive-sync-move-progress-")) + const state = await initDriveState(root, "lib_1") + state.entries["a.md"] = stateEntry("a.md", "a\n", 1) + state.entries["b.md"] = stateEntry("b.md", "b\n", 1) + await writeDriveState(root, state) + await writeFile(join(root, "a2.md"), "a\n") + await writeFile(join(root, "b2.md"), "b\n") + await writeFile(join(root, "new.md"), "new\n") + const api = mkApi([{ entries: [entry("a.md", "a\n", 1), entry("b.md", "b\n", 1)] }]) + api.moveFile = async (_id, _fromPath, toPath) => ({ + entry: entry(toPath, toPath === "a2.md" ? "a\n" : "b\n", 2), + result: "moved" as const, + }) + const progress: Array<[number, number]> = [] + + await runDriveSyncOnce(root, api, undefined, (processed, total) => { + progress.push([processed, total]) + }) + + expect(progress).toEqual([ + [0, 3], + [1, 3], + [2, 3], + [3, 3], + ]) + }) }) function digestOf(content: string): string { @@ -2231,8 +2496,12 @@ it.each(["delete", "move"])("uses the saved identity for sync %s when another en const fail = async (...args: unknown[]) => { calls.push(args); throw new DriveHttpError(409, { code: "VERSION_CONFLICT" }) } api.deleteFile = fail if (operation === "move") api.moveFile = fail - if (operation === "move") await expect(runDriveSyncOnce(root, api)).rejects.toMatchObject({ code: "VERSION_CONFLICT" }) - else expect((await runDriveSyncOnce(root, api)).deleted).toBe(0) + const result = await runDriveSyncOnce(root, api) + if (operation === "move") { + expect(result.path_errors).toEqual([ + { path: "b.txt", code: "VERSION_CONFLICT", message: "move from a.txt rejected (HTTP 409)", retryable: false }, + ]) + } else expect(result.deleted).toBe(0) expect(calls).toEqual([operation === "move" ? ["lib_1", "a.txt", "b.txt", 1, "original-id"] : ["lib_1", "a.txt", 1, "original-id"]]) expect(api.uploads).toEqual([]) expect((await readDriveState(root)).entries["a.txt"]?.entry_id).toBe("original-id")