diff --git a/docs/code/automated-task-processing.md b/docs/code/automated-task-processing.md index e65e14d..52de858 100644 --- a/docs/code/automated-task-processing.md +++ b/docs/code/automated-task-processing.md @@ -154,6 +154,8 @@ A failed run never ends silently. When processing a task fails after it was move The failure comment will not cause a retry loop: posting it does bump the tracker's update stamp, but the retry gate ignores the harness's own comments and records the attempt, so the ticket is only re-run after you edit the description, post your own comment, or delete the failure comment (see [worker polling](./worker.md#re-running-a-task)). +That covers graceful stops. When the worker itself dies mid-task (power cut, crash, `kill -9`), no comment could be posted at the time — so on its next startup the worker detects the runs left in flight, comments on their tickets with the same failure explanation, and moves them back to To Do. Tickets that moved on after the crash and orphans older than `WORKER_ORPHAN_MAX_AGE_HOURS` (default 168) are left alone. See [Interrupted runs are recovered on startup](./worker.md#interrupted-runs-are-recovered-on-startup). + Pass `--skip-comments` to disable all tracker comments, including failure feedback. ### Linear schedules diff --git a/docs/code/worker.md b/docs/code/worker.md index 0e8bfb6..f5a9761 100644 --- a/docs/code/worker.md +++ b/docs/code/worker.md @@ -138,6 +138,21 @@ If a run completes but you want a different result, move the ticket back to your Retry bookkeeping lives in `.devintern-code/queue.db` next to the worker's cursors. For local one-off runs, `devintern TASK-123 --force` re-runs a task even if nothing on the ticket changed; do not put `--force` in `[defaults].worker_task_args`, since that would disable the gate for every polled task. +### Interrupted runs are recovered on startup + +A graceful stop (Ctrl-C, `SIGTERM`) comments on an in-flight ticket and moves it back to To Do. A hard crash — power cut, kernel panic, `kill -9`, a laptop that died — skips that cleanup, which used to leave the ticket stranded in "In Progress". + +The worker now bridges that gap on startup. Before any new tickets are acquired, it detects task runs left `in_progress` by the previous (dead) worker instance and gives each affected ticket the same treatment a graceful shutdown would have: the processing-failure comment is posted (explaining that the worker exited unexpectedly before a pull request could be created, and how to unlock a retry), and the ticket is moved back to your To Do status so it is no longer stranded. Each recovery is logged. + +Recovery respects the retry gate: the requeued ticket is not re-run on restart (no duplicate execution). It sits in To Do like any other ticket whose last attempt failed, and runs again when the ticket changes — an edited description, a new comment, or `--force`. + +Two guards keep the recovery from making noise or causing harm: + +- **Tickets that moved on are left alone.** If a ticket is no longer in your configured In Progress status (someone closed it, moved it to review, or otherwise handled it after the crash), the worker does not comment on or move it. +- **Very old orphans are not announced.** Runs started more than 7 days ago are marked failed in the database but produce no comment or transition, on the assumption they were already handled manually. Set `WORKER_ORPHAN_MAX_AGE_HOURS` in the workspace `.env` to change the cutoff (`0` disables the feedback for every orphan). + +Runs from scheduled automations and PR reviews are not part of this: automations recover through their own claim machinery (the next occurrence picks up after a stale lease), and PR runs have their own comment flows. + ### Ticket matches the query but is not picked up The worker log is the diagnostic. Look for `[poll:]` (for Jira, `[poll:jira]`): @@ -204,6 +219,7 @@ Mention matching requires a resolvable bot identity, so this team/automation fea ## How events are handled - Events are persisted to a local SQLite queue (`.devintern-code/queue.db`) before processing, so a crash or restart never loses accepted work. +- Runs interrupted by a dead worker are recovered on startup: their tickets get the failure comment and move back to To Do before new work is picked up. - Duplicate webhook deliveries are detected by GitHub's delivery id and skipped. - Review feedback is processed before new task pickup: a human waiting on feedback beats a ticket that can wait a minute. - One task or scheduled automation runs at a time per repository. diff --git a/packages/code/src/lib/orphan-recovery.ts b/packages/code/src/lib/orphan-recovery.ts new file mode 100644 index 0000000..d40ed87 --- /dev/null +++ b/packages/code/src/lib/orphan-recovery.ts @@ -0,0 +1,337 @@ +/** + * Orphaned-run recovery + * + * Bridges crash detection to task resumption. When a worker dies mid-task + * (power cut, crash, `kill -9`), the run row stays `in_progress` in the + * queue database and the ticket stays stranded in its "In Progress" status + * with no comment — the graceful SIGINT/SIGTERM feedback path never ran. + * + * On startup, before any acquirer picks new work, the worker: + * + * 1. Reaps every leftover `in_progress` run (all origins) — the existing + * `reapOrphanedRuns` behavior, kept intact. + * 2. For each orphaned *task* run with a tracker ticket key, approximates + * the graceful-shutdown UX: posts the same processing-failure comment + * (reusing {@link reportTaskFailure}) and moves the ticket back to its + * To Do status, so the normal query can consider it again. + * + * No duplicate execution: the posted comment plus the recorded incomplete + * attempt feed the existing retry gate, so a requeued-but-unchanged ticket + * is skipped on the next pickup exactly like any other reported failure. + * It re-runs only when the ticket changes (edit, comment) or `--force` is + * used — the documented manual behavior is unchanged. + * + * Guards against noisy or harmful notifications: + * + * - **Remote completion** — a ticket that no longer sits in the configured + * "In Progress" status (Done, In Review, or moved by a human) is left + * alone; recovery never clobbers progress that happened after the crash. + * - **Very old orphans** — runs started before the cutoff (default 7 days) + * are reaped silently; they were most likely already handled manually. + * - **Non-ticket runs** — scheduled automations recover via their own lease + * machinery and PR-scoped runs have their own comment flows, so only + * `task`-origin runs with a real ticket key get tracker feedback. + * + * Best-effort throughout: tracker and storage failures degrade to the old + * reap-only behavior with a warning, never a startup failure. + */ + +import { existsSync, readFileSync } from "fs"; +import { join } from "path"; + +import { isMarkdownFilePath } from "@devintern/task-trackers"; + +import type { ProjectSettings } from "../types/settings"; +import type { FailureAttemptRecorder } from "./failure-feedback"; +import { reportTaskFailure } from "./failure-feedback"; +import type { RunRecord, RunStore } from "./run-recorder"; +import type { TaskTrackerClient } from "./task-tracker-client"; + +/** Reason posted on tickets whose run was orphaned by a dead worker. */ +export const ORPHANED_RUN_REASON = + "The worker exited unexpectedly (crash or power loss) while this task was in progress, " + + "before a pull request could be created"; + +/** Orphans older than this are reaped silently (likely handled manually). */ +export const DEFAULT_ORPHAN_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1000; + +/** How many orphaned runs to consider per startup (bound tracker feedback). */ +const MAX_ORPHANS = 50; + +export interface OrphanRecoveryDeps { + runStore: RunStore; + /** + * Tracker client for ticket feedback. Omitted (e.g. tracker credentials + * unavailable) → reap-only with a warning, matching the old behavior. + */ + tracker?: TaskTrackerClient; + /** Tracker identifier stored with the retry state (`TASK_TRACKER`). */ + trackerType?: string; + /** Resolve the tracker project key for a task key (defaults per tracker env). */ + projectKeyFor?: (taskKey: string) => string; + /** Resolve the configured In Progress status for a project (null = unknown). */ + getInProgressStatus?: (projectKey: string) => string | null | undefined; + /** Resolve the configured To Do status for a project (null = unknown). */ + getTodoStatus?: (projectKey: string) => string | null | undefined; + /** Retry-gate recorder (injected for tests; default is the shared store). */ + recordAttempt?: FailureAttemptRecorder; + /** Skip feedback for runs started before `now - maxAgeMs`. */ + maxAgeMs?: number; + /** Present tense "now", overridable in tests. */ + now?: number; + log?: typeof console.log; + warn?: typeof console.warn; +} + +export interface OrphanRecoveryResult { + /** Leftover `in_progress` runs marked failed (all origins). */ + reaped: number; + /** Orphaned task runs that received tracker feedback. */ + recovered: number; + /** Orphaned task runs deliberately left alone (stale, moved on, no key). */ + skipped: number; +} + +/** + * Detect runs abandoned by a previous worker instance, reap them, and give + * the affected tickets the same interrupt feedback a graceful shutdown + * would have. Never throws. + */ +export async function recoverOrphanedTaskRuns( + deps: OrphanRecoveryDeps, +): Promise { + const log = deps.log ?? console.log; + const warn = deps.warn ?? console.warn; + const now = deps.now ?? Date.now(); + const maxAgeMs = deps.maxAgeMs ?? DEFAULT_ORPHAN_MAX_AGE_MS; + + let orphans: RunRecord[] = []; + try { + orphans = deps.runStore.listRuns({ status: "in_progress", origin: "task", limit: MAX_ORPHANS }); + } catch (error) { + warn(`⚠️ Could not list in-progress runs for recovery: ${(error as Error).message}`); + } + + // Reap first (all origins) — the pre-existing behavior this module builds on. + let reaped = 0; + try { + reaped = deps.runStore.reapOrphanedRuns(); + } catch (error) { + warn(`⚠️ Could not reap orphaned runs: ${(error as Error).message}`); + } + + if (reaped === 0) { + return { reaped, recovered: 0, skipped: 0 }; + } + + if (!deps.tracker) { + warn( + `⚠️ ${reaped} orphaned run(s) from the previous worker marked failed, ` + + "but no tracker client is available to notify their tickets", + ); + return { reaped, recovered: 0, skipped: 0 }; + } + + log( + `🔁 ${reaped} in-progress run(s) left behind by the previous worker; ` + + "recovering affected tickets before picking up new work", + ); + + let recovered = 0; + let skipped = 0; + const notified = new Set(); + + for (const run of orphans) { + const taskKey = run.taskKey; + if (!taskKey || notified.has(taskKey)) { + continue; + } + notified.add(taskKey); + + if (isMarkdownFilePath(taskKey)) { + // Markdown "tickets" are local files; automations recover via their + // next occurrence, so there is nothing to notify. + skipped++; + continue; + } + + if (now - run.startedAt > maxAgeMs) { + log( + `⏭️ Orphaned run for ${taskKey} started more than ${Math.round(maxAgeMs / 86_400_000)} day(s) ago; leaving the ticket alone`, + ); + skipped++; + continue; + } + + // Avoid clobbering remote progress: only tickets still sitting in the + // configured In Progress status were stranded by the crash. + const projectKey = (deps.projectKeyFor ?? defaultProjectKeyFor)(taskKey); + const inProgressStatus = deps.getInProgressStatus?.(projectKey)?.trim(); + let movedToInProgress = false; + try { + const task = await deps.tracker.getTask(taskKey); + const currentStatus = task.status?.trim(); + if (inProgressStatus && currentStatus !== inProgressStatus) { + log( + `⏭️ ${taskKey} is now '${currentStatus ?? "unknown"}' (not '${inProgressStatus}'); ` + + "assuming it was handled after the crash and leaving it alone", + ); + skipped++; + continue; + } + movedToInProgress = Boolean(inProgressStatus); + } catch (error) { + warn(`⚠️ Could not fetch ${taskKey} to recover it: ${(error as Error).message}`); + skipped++; + continue; + } + + await reportTaskFailure({ + taskKey, + reason: ORPHANED_RUN_REASON, + tracker: deps.tracker, + trackerType: deps.trackerType ?? process.env.TASK_TRACKER ?? "jira", + projectKey, + movedToInProgress, + getTodoStatus: () => deps.getTodoStatus?.(projectKey), + recordAttempt: deps.recordAttempt, + log, + warn, + }); + recovered++; + } + + if (recovered > 0) { + log( + `✅ Recovered ${recovered} orphaned ticket(s): failure comment posted and moved back to ` + + "To Do. They re-run when the ticket changes (see the posted comment).", + ); + } + return { reaped, recovered, skipped }; +} + +/** + * Best-effort project key for a task key, mirroring the CLI's env-based + * resolution (tracker-specific env wins; otherwise the `PROJ-1` prefix). + */ +export function defaultProjectKeyFor(taskKey: string): string { + const trackerType = (process.env.TASK_TRACKER || "jira").toLowerCase(); + const envKey: Record = { + trello: process.env.TRELLO_DEFAULT_BOARD_ID, + github: process.env.GITHUB_REPO, + gitlab: process.env.GITLAB_PROJECT, + "azure-devops": process.env.AZURE_DEVOPS_PROJECT, + asana: process.env.ASANA_DEFAULT_PROJECT_GID, + }; + return envKey[trackerType] || taskKey.split("-")[0] || taskKey; +} + +// --------------------------------------------------------------------------- +// Settings resolution for status names. +// +// Fleet tasks run as CLI subprocesses in per-repo worktrees, so the pipeline +// reads each repo's `.devintern-code/settings.json` relative to its cwd. The +// recovery path runs in the worker process before any worktree exists for the +// orphaned run, so it reads settings from the repos' persistent base +// worktrees instead (first configured repo wins per project key). +// --------------------------------------------------------------------------- + +const TRACKER_SECTION_KEYS = [ + "jira", + "linear", + "trello", + "azure-devops", + "asana", + "github", + "gitlab", + "markdown", +] as const; + +/** + * Merge the `.devintern-code/settings.json` of the given directories into one + * settings view (earlier directories win per project key). Returns null when + * none of the directories has a readable settings file. + */ +export function loadProjectSettingsFrom(dirs: string[]): ProjectSettings | null { + let merged: ProjectSettings | null = null; + for (const dir of dirs) { + const settingsPath = join(dir, ".devintern-code", "settings.json"); + if (!existsSync(settingsPath)) { + continue; + } + try { + const settings = JSON.parse(readFileSync(settingsPath, "utf8")) as ProjectSettings; + merged = merged ?? {}; + mergeSettingsInto(merged, settings); + } catch { + // Unreadable settings degrade to "no status names configured". + } + } + return merged; +} + +/** First-wins merge of tracker project sections and the legacy top-level map. */ +function mergeSettingsInto(target: ProjectSettings, source: ProjectSettings): void { + for (const key of TRACKER_SECTION_KEYS) { + const projects = source[key]?.projects; + if (!projects) { + continue; + } + const existing = target[key]?.projects ?? {}; + target[key] = { projects: { ...projects, ...existing } }; + } + if (source.projects) { + target.projects = { ...source.projects, ...target.projects }; + } +} + +/** + * Resolve a status name (`inProgressStatus` / `todoStatus` / `prStatus`) for + * a project from settings. Mirrors the CLI's resolution: tracker-specific + * section first, tracker-specific env key fallback for the project key's + * meaning is handled by the caller; legacy top-level `projects` applies to + * Jira only. + */ +export function resolveStatusName( + settings: ProjectSettings | null, + trackerType: string, + projectKey: string, + field: "inProgressStatus" | "todoStatus" | "prStatus", +): string | undefined { + if (!settings) { + return undefined; + } + const tracker = trackerType.toLowerCase(); + const section = settings[tracker as keyof ProjectSettings] as + | { projects?: Record> } + | undefined; + const projectConfig = section?.projects?.[projectKey]; + const fromSection = projectConfig?.[field]; + if (typeof fromSection === "string") { + return fromSection; + } + + // Trello cards key settings by board; a single configured board stands in + // for an unlisted one (same fallback the pipeline uses). + if (tracker === "trello") { + const boardKeys = Object.keys(section?.projects ?? {}); + const fallback = boardKeys.length === 1 ? (boardKeys[0] as string) : undefined; + if (fallback) { + const single = section?.projects?.[fallback]?.[field]; + if (typeof single === "string") { + return single; + } + } + } + + if (tracker === "jira") { + const legacy = (settings.projects?.[projectKey] as Record | undefined)?.[ + field + ]; + if (typeof legacy === "string") { + return legacy; + } + } + + return undefined; +} diff --git a/packages/code/src/lib/workspace/workspace-worker.ts b/packages/code/src/lib/workspace/workspace-worker.ts index 7899b96..0f4fd94 100644 --- a/packages/code/src/lib/workspace/workspace-worker.ts +++ b/packages/code/src/lib/workspace/workspace-worker.ts @@ -8,6 +8,7 @@ * in the central workspace DB. */ +import { existsSync } from "fs"; import { dirname, join, resolve } from "path"; import { LockManager } from "../lock-manager"; @@ -17,6 +18,14 @@ import type { TaskExecutionResult } from "../task-polling-acquirer"; import type { ChangeDetector } from "../change-detector"; import type { WebhookQueue } from "../webhook-queue"; import type { WorkerState } from "../worker-state"; +import { + loadProjectSettingsFrom, + recoverOrphanedTaskRuns, + resolveStatusName, +} from "../orphan-recovery"; +import { RunStore } from "../run-recorder"; +import { RetryStateStore } from "../retry-state"; +import type { TaskTrackerClient } from "../task-tracker-client"; import { findRepo, loadWorkspaceConfig } from "./config"; import type { RepoConfig, WorkspaceConfig } from "./config"; import { buildRepoEnv, parseEnvFile } from "./env"; @@ -25,17 +34,88 @@ import { workspaceConfigPath, workspaceDbPath, workspaceEnvPath, + worktreesDir, } from "./paths"; import { routeTask, toRoutableTask } from "./router"; import type { RoutableTask } from "./router"; import { createRepoRunLock, createWorkspaceLock, openWorkspaceState } from "./state"; import type { RoutingSkipStore } from "./state"; -import { RepoManager } from "./repo-manager"; +import { BASE_WORKTREE_NAME, RepoManager } from "./repo-manager"; import { probePushAccess } from "../github-push-probe"; import { AutomationAcquirer } from "../automation-acquirer"; import type { AutomationConfig } from "../automation-config"; import { flushAnalytics, RUN_ORIGIN_ENV, trackWorkerStarted } from "../analytics"; +/** Orphaned-run feedback cutoff: `WORKER_ORPHAN_MAX_AGE_HOURS`, default 7 days. */ +function orphanMaxAgeMs(): number { + return parseEnvInteger("WORKER_ORPHAN_MAX_AGE_HOURS", 24 * 7, { min: 0 }) * 60 * 60 * 1000; +} + +/** + * Recover task runs left in progress by a previous (dead) worker before any + * acquirer picks up new tickets: reap them and give their tickets the + * graceful-shutdown feedback (failure comment + move back to To Do). + * + * Replaces the old reap-only startup sweep, which also only ran when GitHub + * credentials were configured; recovery here covers every workspace. + */ +export async function recoverOrphanedWorkspaceRuns(options: { + config: WorkspaceConfig; + workspaceDir: string; + dbPath: string; +}): Promise { + const { config, workspaceDir, dbPath } = options; + const runStore = new RunStore(dbPath); + let retryStore: RetryStateStore | null = null; + try { + const hasTaskOrphans = + runStore.listRuns({ status: "in_progress", origin: "task", limit: 1 }).length > 0; + + let tracker: TaskTrackerClient | undefined; + if (hasTaskOrphans) { + try { + const { TaskTrackerManager } = await import("../task-tracker-manager"); + tracker = new TaskTrackerManager().getClient(); + } catch (error) { + console.warn( + `⚠️ [fleet] could not initialize the tracker to recover orphaned tickets: ${ + (error as Error).message + }`, + ); + } + } + + // Fleet tasks run in per-repo worktrees, so their status names come from + // each repo's checked-in settings; the base worktrees hold a checked-out + // copy. Never triggers git work here — only existing directories are read. + const settingsDirs = [ + ...config.repos + .map((repo) => join(worktreesDir(workspaceDir), repo.name, BASE_WORKTREE_NAME)) + .filter(existsSync), + workspaceDir, + ]; + const settings = loadProjectSettingsFrom(settingsDirs); + const trackerType = config.defaults.tracker; + retryStore = new RetryStateStore(dbPath); + + await recoverOrphanedTaskRuns({ + runStore, + tracker, + trackerType, + getInProgressStatus: (projectKey) => + resolveStatusName(settings, trackerType, projectKey, "inProgressStatus"), + getTodoStatus: (projectKey) => + resolveStatusName(settings, trackerType, projectKey, "todoStatus"), + recordAttempt: (taskKey, type, description) => + retryStore?.recordIncompleteAttempt(taskKey, type, description), + maxAgeMs: orphanMaxAgeMs(), + }); + } finally { + retryStore?.close(); + runStore.close(); + } +} + /** Task shape the fleet acquirer needs (structural subset of `Task`). */ export interface FleetTask { key: string; @@ -367,6 +447,13 @@ export async function runWorkspaceWorker(options: RunWorkspaceWorkerOptions): Pr const state = openWorkspaceState(workspaceDir); const repoManager = new RepoManager(workspaceDir); + // Recover what the previous worker left behind before acquiring new work. + await recoverOrphanedWorkspaceRuns({ + config, + workspaceDir, + dbPath: state.dbPath, + }); + for (const repo of config.repos) { const removed = await repoManager.sweepStaleWorktrees( repo.name, @@ -541,14 +628,7 @@ export async function buildFleetEventAcquirers(options: { // Tier 1: the agent's own PRs (central agent_prs registry is repo-keyed, // so one acquirer covers the whole fleet). const { ReviewPollingAcquirer } = await import("../review-polling-acquirer"); - const { RunStore } = await import("../run-recorder"); const runStore = new RunStore(state.dbPath); - const reapedRuns = runStore.reapOrphanedRuns(); - if (reapedRuns > 0) { - console.warn( - `⚠️ Marked ${reapedRuns} in-progress run(s) as failed: previous worker exited before they finished`, - ); - } acquirers.push( new ReviewPollingAcquirer({ intervalSeconds, diff --git a/packages/code/tests/orphan-recovery.test.ts b/packages/code/tests/orphan-recovery.test.ts new file mode 100644 index 0000000..d0c77bf --- /dev/null +++ b/packages/code/tests/orphan-recovery.test.ts @@ -0,0 +1,349 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, rmSync, writeFileSync } from "fs"; +import { join } from "path"; +import { tmpdir } from "os"; + +import { + DEFAULT_ORPHAN_MAX_AGE_MS, + ORPHANED_RUN_REASON, + defaultProjectKeyFor, + loadProjectSettingsFrom, + recoverOrphanedTaskRuns, + resolveStatusName, +} from "../src/lib/orphan-recovery"; +import { RunStore } from "../src/lib/run-recorder"; +import { RetryStateStore, hashDescription } from "../src/lib/retry-state"; +import type { TaskTrackerClient } from "../src/lib/task-tracker-client"; +import type { Task, TaskTrackerCommentContent } from "../src/types/task-tracker"; + +describe("recoverOrphanedTaskRuns", () => { + let dbPath: string; + let store: RunStore; + let retryStore: RetryStateStore; + let posted: Array<{ key: string; body: string }> = []; + let transitions: Array<{ key: string; status: string }> = []; + let tasksByKey: Record = {}; + let getTaskFailures: Record = {}; + + function ticket(key: string, status: string): Task { + return { + key, + summary: `Summary of ${key}`, + issueType: "Task", + status, + reporter: "Alice", + created: "", + updated: "", + labels: [], + components: [], + fixVersions: [], + raw: null, + }; + } + + function fakeTracker(): TaskTrackerClient { + return { + getTask: async (key: string) => { + if (getTaskFailures[key]) { + throw new Error("tracker down"); + } + return tasksByKey[key] ?? ticket(key, "In Progress"); + }, + postComment: async (key: string, content: TaskTrackerCommentContent) => { + posted.push({ key, body: content.body }); + }, + transitionStatus: async (key: string, status: string) => { + transitions.push({ key, status }); + }, + extractDescriptionText: (t: Task): string => `description of ${t.summary}`, + } as unknown as TaskTrackerClient; + } + + interface RecoveryOptions { + inProgressStatus?: string; + todoStatus?: string | null; + tracker?: TaskTrackerClient | null; + now?: number; + maxAgeMs?: number; + } + + function recover({ + inProgressStatus = "In Progress", + todoStatus = "To Do", + tracker = fakeTracker() as TaskTrackerClient | null, + now = Date.now(), + maxAgeMs = DEFAULT_ORPHAN_MAX_AGE_MS, + }: RecoveryOptions = {}) { + return recoverOrphanedTaskRuns({ + runStore: store, + tracker: tracker ?? undefined, + trackerType: "jira", + getInProgressStatus: () => inProgressStatus, + getTodoStatus: () => todoStatus, + recordAttempt: (key, type, description) => + retryStore.recordIncompleteAttempt(key, type, description), + maxAgeMs, + now, + log: () => {}, + warn: () => {}, + }); + } + + beforeEach(() => { + dbPath = join(tmpdir(), `orphan-test-${Date.now()}-${Math.random().toString(36).slice(2)}.db`); + store = new RunStore(dbPath); + retryStore = new RetryStateStore(dbPath); + posted = []; + transitions = []; + tasksByKey = {}; + getTaskFailures = {}; + }); + + afterEach(() => { + store.close(); + retryStore.close(); + for (const suffix of ["", "-wal", "-shm"]) { + rmSync(`${dbPath}${suffix}`, { force: true }); + } + }); + + test("crash mid-run: recovery on next start requeues the ticket and records the attempt", async () => { + // Simulate a crash between the transition to In Progress and completion: + // the previous process recorded an in-progress run and died. + const crashed = new RunStore(dbPath); + crashed.createRun({ origin: "task", taskKey: "PROJ-1", tracker: "jira" }); + crashed.close(); + + const result = await recover(); + + expect(result.reaped).toBe(1); + expect(result.recovered).toBe(1); + expect(result.skipped).toBe(0); + + // The orphaned run is marked failed with the crash reason. + const run = new RunStore(dbPath).listRuns({ taskKey: "PROJ-1" })[0]; + expect(run?.status).toBe("failed"); + expect(run?.outcomeReason).toContain("orphaned"); + expect(run?.finishedAt).toBeGreaterThan(0); + + // The ticket got the failure comment and moved back to To Do. + expect(posted).toHaveLength(1); + expect(posted[0]?.key).toBe("PROJ-1"); + expect(posted[0]?.body).toContain("Automated implementation did not complete"); + expect(posted[0]?.body).toContain("exited unexpectedly"); + expect(transitions).toEqual([{ key: "PROJ-1", status: "To Do" }]); + + // The retry gate records the attempt, so the requeued ticket is not + // double-processed on the next pickup. + const state = retryStore.getRetryState("PROJ-1"); + expect(state).not.toBeNull(); + expect(state?.descriptionHash).toBe(hashDescription("description of Summary of PROJ-1")); + }); + + test("does not clobber a ticket that moved on after the crash", async () => { + store.createRun({ origin: "task", taskKey: "PROJ-2" }); + tasksByKey["PROJ-2"] = ticket("PROJ-2", "Done"); + + const result = await recover(); + + expect(result.recovered).toBe(0); + expect(result.skipped).toBe(1); + expect(posted).toHaveLength(0); + expect(transitions).toHaveLength(0); + expect(retryStore.getRetryState("PROJ-2")).toBeNull(); + }); + + test("skips orphans older than the cutoff instead of notifying", async () => { + store.createRun({ origin: "task", taskKey: "PROJ-3" }); + + const result = await recover({ + now: Date.now() + DEFAULT_ORPHAN_MAX_AGE_MS + 60_000, + }); + + expect(result.recovered).toBe(0); + expect(result.skipped).toBe(1); + expect(posted).toHaveLength(0); + expect(transitions).toHaveLength(0); + }); + + test("respects a custom maxAgeMs", async () => { + const runId = store.createRun({ origin: "task", taskKey: "PROJ-3b" }); + + await recover({ maxAgeMs: 1000, now: Date.now() + 2000 }); + expect(store.getRun(runId)?.status).toBe("failed"); + expect(posted).toHaveLength(0); + }); + + test("reaps every origin but only notifies task runs", async () => { + store.createRun({ origin: "task", taskKey: "PROJ-4" }); + store.createRun({ origin: "pr_mention", repo: "acme/widgets", prNumber: 1 }); + store.createRun({ origin: "conflict_resolution" }); + store.createRun({ origin: "scheduled", automationId: "tidy" }); + + const result = await recover(); + + expect(result.reaped).toBe(4); + expect(result.recovered).toBe(1); + expect(posted.map((p) => p.key)).toEqual(["PROJ-4"]); + }); + + test("skips markdown task keys (local files, recovered by automations themselves)", async () => { + store.createRun({ origin: "task", taskKey: "/tmp/devintern-tasks/note.md" }); + + const result = await recover(); + + expect(result.recovered).toBe(0); + expect(result.skipped).toBe(1); + expect(posted).toHaveLength(0); + }); + + test("comments without transitioning when the ticket was never In Progress", async () => { + const runId = store.createRun({ origin: "task", taskKey: "PROJ-5" }); + + const result = await recover({ inProgressStatus: "" }); + + expect(result.recovered).toBe(1); + expect(posted).toHaveLength(1); + expect(transitions).toHaveLength(0); + expect(store.getRun(runId)?.status).toBe("failed"); + }); + + test("skips the transition when no To Do status is configured", async () => { + store.createRun({ origin: "task", taskKey: "PROJ-6" }); + + const result = await recover({ todoStatus: null }); + + expect(result.recovered).toBe(1); + expect(posted).toHaveLength(1); + expect(transitions).toHaveLength(0); + }); + + test("degrades to reap-only when no tracker client is available", async () => { + const runId = store.createRun({ origin: "task", taskKey: "PROJ-7" }); + + const result = await recover({ tracker: null }); + + expect(result.reaped).toBe(1); + expect(result.recovered).toBe(0); + expect(posted).toHaveLength(0); + expect(store.getRun(runId)?.status).toBe("failed"); + }); + + test("skips a ticket whose details cannot be fetched, without throwing", async () => { + const runId = store.createRun({ origin: "task", taskKey: "PROJ-8" }); + getTaskFailures["PROJ-8"] = true; + + const result = await recover(); + + expect(result.recovered).toBe(0); + expect(result.skipped).toBe(1); + expect(posted).toHaveLength(0); + // The reap still happened. + expect(store.getRun(runId)?.status).toBe("failed"); + }); + + test("notifies each stranded ticket once even with several orphaned attempts", async () => { + store.createRun({ origin: "task", taskKey: "PROJ-9" }); + store.createRun({ origin: "task", taskKey: "PROJ-9" }); + + const result = await recover(); + + expect(result.reaped).toBe(2); + expect(result.recovered).toBe(1); + expect(posted).toHaveLength(1); + }); + + test("does nothing when no runs were orphaned", async () => { + const runId = store.createRun({ origin: "task", taskKey: "PROJ-10" }); + store.finishRun(runId, "succeeded"); + + const result = await recover(); + + expect(result).toEqual({ reaped: 0, recovered: 0, skipped: 0 }); + expect(posted).toHaveLength(0); + }); + + test("reason text explains the crash and the retry path", () => { + expect(ORPHANED_RUN_REASON).toContain("exited unexpectedly"); + expect(ORPHANED_RUN_REASON).toContain("pull request"); + }); +}); + +describe("defaultProjectKeyFor", () => { + const savedEnv = { ...process.env }; + + afterEach(() => { + process.env = { ...savedEnv }; + }); + + test("falls back to the task key prefix", () => { + delete process.env.GITHUB_REPO; + process.env.TASK_TRACKER = "jira"; + expect(defaultProjectKeyFor("PROJ-42")).toBe("PROJ"); + }); + + test("uses the tracker env project when set", () => { + process.env.TASK_TRACKER = "github"; + process.env.GITHUB_REPO = "acme/widgets"; + expect(defaultProjectKeyFor("42")).toBe("acme/widgets"); + }); +}); + +describe("settings resolution", () => { + let dirA: string; + let dirB: string; + + beforeEach(() => { + dirA = join(tmpdir(), `orphan-settings-a-${Date.now()}-${Math.random().toString(36).slice(2)}`); + dirB = join(tmpdir(), `orphan-settings-b-${Date.now()}-${Math.random().toString(36).slice(2)}`); + mkdirSync(join(dirA, ".devintern-code"), { recursive: true }); + mkdirSync(join(dirB, ".devintern-code"), { recursive: true }); + }); + + afterEach(() => { + rmSync(dirA, { recursive: true, force: true }); + rmSync(dirB, { recursive: true, force: true }); + }); + + test("merges tracker sections from repo settings dirs, first dir wins", () => { + writeFileSync( + join(dirA, ".devintern-code", "settings.json"), + JSON.stringify({ + jira: { projects: { PROJ: { inProgressStatus: "Doing", todoStatus: "Backlog" } } }, + }), + ); + writeFileSync( + join(dirB, ".devintern-code", "settings.json"), + JSON.stringify({ + jira: { + projects: { PROJ: { inProgressStatus: "Overridden" }, OTHER: { todoStatus: "To Do" } }, + }, + linear: { projects: { ENG: { inProgressStatus: "Started" } } }, + }), + ); + + const settings = loadProjectSettingsFrom([dirA, dirB]); + expect(resolveStatusName(settings, "jira", "PROJ", "inProgressStatus")).toBe("Doing"); + expect(resolveStatusName(settings, "jira", "PROJ", "todoStatus")).toBe("Backlog"); + expect(resolveStatusName(settings, "jira", "OTHER", "todoStatus")).toBe("To Do"); + expect(resolveStatusName(settings, "linear", "ENG", "inProgressStatus")).toBe("Started"); + }); + + test("falls back to the legacy top-level projects map for Jira", () => { + writeFileSync( + join(dirA, ".devintern-code", "settings.json"), + JSON.stringify({ + projects: { LEG: { inProgressStatus: "In Progress", todoStatus: "To Do" } }, + }), + ); + + const settings = loadProjectSettingsFrom([dirA]); + expect(resolveStatusName(settings, "jira", "LEG", "inProgressStatus")).toBe("In Progress"); + expect(resolveStatusName(settings, "linear", "LEG", "todoStatus")).toBeUndefined(); + }); + + test("returns null when no directory has settings", () => { + expect(loadProjectSettingsFrom([dirA, dirB])).toBeNull(); + expect(resolveStatusName(null, "jira", "PROJ", "todoStatus")).toBeUndefined(); + }); +}); diff --git a/packages/code/tests/workspace-orphan-recovery.test.ts b/packages/code/tests/workspace-orphan-recovery.test.ts new file mode 100644 index 0000000..6590b2f --- /dev/null +++ b/packages/code/tests/workspace-orphan-recovery.test.ts @@ -0,0 +1,88 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, rmSync, writeFileSync } from "fs"; +import { join } from "path"; +import { tmpdir } from "os"; + +import { recoverOrphanedWorkspaceRuns } from "../src/lib/workspace/workspace-worker"; +import { RunStore } from "../src/lib/run-recorder"; +import { BASE_WORKTREE_NAME } from "../src/lib/workspace/repo-manager"; +import type { WorkspaceConfig } from "../src/lib/workspace/config"; + +describe("recoverOrphanedWorkspaceRuns", () => { + let workspaceDir: string; + let dbPath: string; + let store: RunStore; + const savedEnv = { ...process.env }; + + function workspaceConfig(): WorkspaceConfig { + return { + workspace: {}, + defaults: { tracker: "jira", pollIntervalSeconds: 60 }, + repos: [{ name: "web-app", remote: "https://github.com/acme/web-app.git", env: {} }], + routing: [], + automations: [], + } as unknown as WorkspaceConfig; + } + + beforeEach(() => { + workspaceDir = join( + tmpdir(), + `orphan-workspace-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const baseWorktree = join(workspaceDir, "worktrees", "web-app", BASE_WORKTREE_NAME); + mkdirSync(join(baseWorktree, ".devintern-code"), { recursive: true }); + writeFileSync( + join(baseWorktree, ".devintern-code", "settings.json"), + JSON.stringify({ + jira: { projects: { PROJ: { inProgressStatus: "In Progress", todoStatus: "To Do" } } }, + }), + ); + dbPath = join(workspaceDir, "state", "queue.db"); + store = new RunStore(dbPath); + }); + + afterEach(() => { + process.env = { ...savedEnv }; + store.close(); + rmSync(workspaceDir, { recursive: true, force: true }); + }); + + test("reaps orphans and degrades to reap-only when tracker credentials are missing", async () => { + process.env.TASK_TRACKER = "jira"; + delete process.env.JIRA_BASE_URL; + delete process.env.JIRA_EMAIL; + delete process.env.JIRA_API_TOKEN; + + store.createRun({ origin: "task", taskKey: "PROJ-1", tracker: "jira" }); + + // Recovery runs before acquirers in the real startup path; missing + // tracker credentials must never fail the worker start. + await recoverOrphanedWorkspaceRuns({ + config: workspaceConfig(), + workspaceDir, + dbPath, + }); + + const runs = store.listRuns({ taskKey: "PROJ-1" }); + expect(runs).toHaveLength(1); + expect(runs[0]?.status).toBe("failed"); + expect(runs[0]?.outcomeReason).toContain("orphaned"); + }); + + test("leaves terminal history untouched", async () => { + process.env.TASK_TRACKER = "jira"; + delete process.env.JIRA_BASE_URL; + + const done = store.createRun({ origin: "task", taskKey: "PROJ-2" }); + store.finishRun(done, "succeeded"); + + await recoverOrphanedWorkspaceRuns({ + config: workspaceConfig(), + workspaceDir, + dbPath, + }); + + expect(store.getRun(done)?.status).toBe("succeeded"); + expect(store.getRun(done)?.outcomeReason).toBeUndefined(); + }); +});